$\displaystyle=\frac{2\mu}{Z_{p}^{2}}\left(\mathbb{E}_{\mathbb{Q}_{1:2}}[\widehat{I}^{N}\cdot\widehat{Z}_{p}^{N}]-\mathbb{E}_{\mathbb{Q}_{1:2}}[\widehat{I}^{N}\cdot
Z_{p}]-\mathbb{E}_{\mathbb{Q}_{1:2}}[I\cdot\widehat{Z}_{p}^{N}]+IZ_{p}\right)$
$\displaystyle=\underbrace{\frac{2\mu}{Z_{p}^{2}}\cdot\mathbb{E}_{\mathbb{Q}_{1:2}}[\widehat{I}^{N}\cdot\widehat{Z}_{p}^{N}]}_{\stackrel{{\scriptstyle\text{def}}}{{=}}\leavevmode\hbox
to13pt{\vbox to13pt{\pgfpicture\makeatletter\hbox{\hskip
6.49931pt\lower-6.49931pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.29932pt}{0.0pt}\pgfsys@curveto{6.29932pt}{3.47905pt}{3.47905pt}{6.29932pt}{0.0pt}{6.29932pt}\pgfsys@curveto{-3.47905pt}{6.29932pt}{-6.29932pt}{3.47905pt}{-6.29932pt}{0.0pt}\pgfsys@curveto{-6.29932pt}{-3.47905pt}{-3.47905pt}{-6.29932pt}{0.0pt}{-6.29932pt}\pgfsys@curveto{3.47905pt}{-6.29932pt}{6.29932pt}{-3.47905pt}{6.29932pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.52777pt}{-2.39166pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{T}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}-2\mu^{2}$

Expanding T:

T
$\displaystyle=\frac{2\mu}{Z_{p}^{2}}\cdot\left(\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\left(\frac{1}{N}\sum_{n=1}^{N}\frac{f({\boldsymbol{x}}_{1})\widetilde{p}({\boldsymbol{x}}_{1}^{(n)})}{q_{1}({\boldsymbol{x}}_{1}^{(n)})}\right)\left(\frac{1}{N}\sum_{n^{\prime}=1}^{N}\frac{\widetilde{p}({\boldsymbol{x}}_{2}^{(n^{\prime})})}{q_{2}({\boldsymbol{x}}_{2}^{(n^{\prime})})}\right)\right]\right)$
$\displaystyle=\frac{2\mu}{N^{2}}\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\left(\sum_{n=1}^{N}\frac{f({\boldsymbol{x}}_{1}^{(n)})p({\boldsymbol{x}}_{1}^{(n)})}{q_{1}({\boldsymbol{x}}_{1}^{(n)})}\right)\left(\sum_{n^{\prime}=1}^{N}\frac{p({\boldsymbol{x}}_{2}^{(n^{\prime})})}{q_{2}({\boldsymbol{x}}_{2}^{(n^{\prime})})}\right)\right]$
$\displaystyle=\frac{2\mu}{N^{2}}\sum_{n=1}^{N}\sum_{n^{\prime}=1}^{N}\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\frac{f({\boldsymbol{x}}_{1}^{(n)})p({\boldsymbol{x}}_{1}^{(n)})}{q_{1}({\boldsymbol{x}}_{1}^{(n)})}\cdot\frac{p({\boldsymbol{x}}_{2}^{(n^{\prime})})}{q_{2}({\boldsymbol{x}}_{2}^{(n^{\prime})})}\right]$
$\displaystyle=\frac{2\mu}{N^{2}}\left(\sum_{n=n^{\prime}}\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\frac{f({\boldsymbol{x}}_{1}^{(n)})p({\boldsymbol{x}}_{1}^{(n)})}{q_{1}({\boldsymbol{x}}_{1}^{(n)})}\cdot\frac{p({\boldsymbol{x}}_{2}^{(n^{\prime})})}{q_{2}({\boldsymbol{x}}_{2}^{(n^{\prime})})}\right]+\sum_{n\neq
n^{\prime}}\underbrace{\mathbb{E}_{q_{1}}\left[\frac{f({\boldsymbol{x}}_{1}^{(n)})p({\boldsymbol{x}}_{1}^{(n)})}{q_{1}({\boldsymbol{x}}_{1}^{(n)})}\right]}_{=\mu}\underbrace{\mathbb{E}_{q_{2}}\left[\frac{p({\boldsymbol{x}}_{2}^{(n^{\prime})})}{q_{2}({\boldsymbol{x}}_{2}^{(n^{\prime})})}\right]}_{=1}\right)$
$\displaystyle=\frac{2\mu}{N^{{2}}}\left({N}\cdot\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\frac{f({\boldsymbol{x}}_{1})p({\boldsymbol{x}}_{1})}{q_{1}({\boldsymbol{x}}_{1})}\cdot\frac{p({\boldsymbol{x}}_{2})}{q_{2}({\boldsymbol{x}}_{2})}\right]+{N}(N-1)\cdot\mu\right)$
$\displaystyle=\frac{2\mu}{N}\cdot\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\frac{f({\boldsymbol{x}}_{1})p({\boldsymbol{x}}_{1})}{q_{1}({\boldsymbol{x}}_{1})}\cdot\frac{p({\boldsymbol{x}}_{2})}{q_{2}({\boldsymbol{x}}_{2})}\right]+\frac{N-1}{N}\cdot
2\mu^{2}$

Therefore, the third term becomes

$\displaystyle\frac{2\mu}{Z_{p}^{2}}\cdot\operatorname{Cov}_{\mathbb{Q}_{1:2}}[\widehat{I}^{N},\widehat{Z}_{p}^{N}]$
$\displaystyle=\frac{2\mu}{N}\cdot\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\frac{f({\boldsymbol{x}}_{1})p({\boldsymbol{x}}_{1})}{q_{1}({\boldsymbol{x}}_{1})}\cdot\frac{p({\boldsymbol{x}}_{2})}{q_{2}({\boldsymbol{x}}_{2})}\right]+\frac{N-1}{N}\cdot
2\mu^{2}-2\mu^{2}$ (65)
$\displaystyle=\frac{2\mu}{N}\cdot\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\frac{f({\boldsymbol{x}}_{1})p({\boldsymbol{x}}_{1})}{q_{1}({\boldsymbol{x}}_{1})}\cdot\frac{p({\boldsymbol{x}}_{2})}{q_{2}({\boldsymbol{x}}_{2})}\right]-\frac{2\mu^{2}}{N}$
(66)
$\displaystyle=\frac{2\mu}{N}\left(\underbrace{\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\frac{f({\boldsymbol{x}}_{1})p({\boldsymbol{x}}_{1})}{q_{1}({\boldsymbol{x}}_{1})}\cdot\frac{p({\boldsymbol{x}}_{2})}{q_{2}({\boldsymbol{x}}_{2})}\right]}_{\leavevmode\hbox
to13.13pt{\vbox to13.13pt{\pgfpicture\makeatletter\hbox{\hskip
6.56548pt\lower-6.56548pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{6.36548pt}{0.0pt}\pgfsys@curveto{6.36548pt}{3.5156pt}{3.5156pt}{6.36548pt}{0.0pt}{6.36548pt}\pgfsys@curveto{-3.5156pt}{6.36548pt}{-6.36548pt}{3.5156pt}{-6.36548pt}{0.0pt}\pgfsys@curveto{-6.36548pt}{-3.5156pt}{-3.5156pt}{-6.36548pt}{0.0pt}{-6.36548pt}\pgfsys@curveto{3.5156pt}{-6.36548pt}{6.36548pt}{-3.5156pt}{6.36548pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-2.625pt}{-2.39166pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{H}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}-\mu\right)$ (67)

For the case where $f$ can take negative values , we can now expand H as:

H
$\displaystyle=\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\frac{(f_{+}({\boldsymbol{x}}_{1})-f_{-}({\boldsymbol{x}}_{1}))p({\boldsymbol{x}}_{1})}{q_{1}({\boldsymbol{x}}_{1})}\cdot\frac{p({\boldsymbol{x}}_{2})}{q_{2}({\boldsymbol{x}}_{2})}\right]$
(68)
$\displaystyle=\underbrace{\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\frac{f_{+}({\boldsymbol{x}}_{1})p({\boldsymbol{x}}_{1})}{q_{1}({\boldsymbol{x}}_{1})}\cdot\frac{p({\boldsymbol{x}}_{2})}{q_{2}({\boldsymbol{x}}_{2})}\right]}_{\leavevmode\hbox
to15.87pt{\vbox to15.87pt{\pgfpicture\makeatletter\hbox{\hskip
7.93437pt\lower-7.93437pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.73438pt}{0.0pt}\pgfsys@curveto{7.73438pt}{4.27162pt}{4.27162pt}{7.73438pt}{0.0pt}{7.73438pt}\pgfsys@curveto{-4.27162pt}{7.73438pt}{-7.73438pt}{4.27162pt}{-7.73438pt}{0.0pt}\pgfsys@curveto{-7.73438pt}{-4.27162pt}{-4.27162pt}{-7.73438pt}{0.0pt}{-7.73438pt}\pgfsys@curveto{4.27162pt}{-7.73438pt}{7.73438pt}{-4.27162pt}{7.73438pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.375pt}{-2.39166pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{H1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}-\underbrace{\mathbb{E}_{\mathbb{Q}_{1:2}}\left[\frac{f_{-}({\boldsymbol{x}}_{1})p({\boldsymbol{x}}_{1})}{q_{1}({\boldsymbol{x}}_{1})}\cdot\frac{p({\boldsymbol{x}}_{2})}{q_{2}({\boldsymbol{x}}_{2})}\right]}_{\leavevmode\hbox
to15.87pt{\vbox to15.87pt{\pgfpicture\makeatletter\hbox{\hskip
7.93437pt\lower-7.93437pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{7.73438pt}{0.0pt}\pgfsys@curveto{7.73438pt}{4.27162pt}{4.27162pt}{7.73438pt}{0.0pt}{7.73438pt}\pgfsys@curveto{-4.27162pt}{7.73438pt}{-7.73438pt}{4.27162pt}{-7.73438pt}{0.0pt}\pgfsys@curveto{-7.73438pt}{-4.27162pt}{-4.27162pt}{-7.73438pt}{0.0pt}{-7.73438pt}\pgfsys@curveto{4.27162pt}{-7.73438pt}{7.73438pt}{-4.27162pt}{7.73438pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.375pt}{-2.39166pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{H2}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}$ (69)

##  Appendix G Derivation of $q^{\bigstar}_{\text{SNIS}}$

For convenience, let us restate

$\displaystyle\widehat{\mu}_{\text{SNIS}}^{N}\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{\sum_{n=1}^{N}\frac{f({\boldsymbol{x}}^{(n)})\widetilde{p}({\boldsymbol{x}}^{(n)})}{q({\boldsymbol{x}}^{(n)})}}{\sum_{n=1}^{N}\frac{\widetilde{p}({\boldsymbol{x}}^{(n)})}{q({\boldsymbol{x}}^{(n)})}},$
(70)

and:

$\displaystyle\mathbb{V}_{q}^{\delta}[\widehat{\mu}_{\text{SNIS}}^{N}]$
$\displaystyle\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{1}{Z_{p}^{2}}\mathbb{V}_{q}[\widehat{I}^{N}]+\frac{\mu^{2}}{Z_{p}^{2}}\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]-\frac{2\mu}{Z_{p}^{2}}\cdot\operatorname{Cov}_{q}[\widehat{I}^{N},\widehat{Z}_{p}^{N}].$
(71)

The aim is to find the function $q({\boldsymbol{x}})$ that minimizes the
expression above, which is a functional of $q$, with the constraint that it is
a valid pdf. To do this, we will take the functional derivative of Eq. 71 and
equate to zero. First, recalling that
$\widehat{I}^{N}=\frac{1}{N}\sum_{n=1}^{N}\frac{f({\boldsymbol{x}}^{(n)})\widetilde{p}({\boldsymbol{x}}^{(n)})}{q({\boldsymbol{x}}^{(n)})}$
and
$\widehat{Z}_{p}^{N}=\frac{1}{N}\sum_{n=1}^{N}\frac{\widetilde{p}({\boldsymbol{x}}^{(n)})}{q({\boldsymbol{x}}^{(n)})}$,
let us rewrite Eq. 71 more explicitly as:

$\displaystyle\mathcal{F}[q]\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{1}{N}\underbrace{\left(\int\frac{(f({\boldsymbol{x}})p({\boldsymbol{x}}))^{2}}{q({\boldsymbol{x}})}\mathrm{d}{\boldsymbol{x}}-\mu^{2}\right)}_{\mathcal{F}_{1}[q]}+\underbrace{\frac{\mu^{2}}{N}\left(\int\frac{p({\boldsymbol{x}})^{2}}{q({\boldsymbol{x}})}\mathrm{d}{\boldsymbol{x}}-1\right)}_{\mathcal{F}_{2}[q]}$
(72)
$\displaystyle\underbrace{-2\mu\left(\frac{1}{N}\int\frac{p({\boldsymbol{x}})^{2}}{q({\boldsymbol{x}})}\cdot
f({\boldsymbol{x}})\mathrm{d}{\boldsymbol{x}}-\frac{\mu}{N-1}\right)}_{\mathcal{F}_{3}[q]}.$

We now notice that Eq. 72 is a sum of three terms, each of which is a
functional of $q$ in a so-called _integral form_. When looking at an
infinitesimal change in the functional $\mathcal{F}[q]$, by adding a multiple
of some arbitrary test function $\nu({\boldsymbol{x}})$ to $q$, the functional
derivative (first variation) $\frac{\delta\mathcal{F}[q]}{\delta q}$ is
defined as:

$\displaystyle\frac{\delta\mathcal{F}[q]}{\delta
q}\stackrel{{\scriptstyle\text{def}}}{{=}}\lim_{\epsilon\rightarrow
0}\frac{\mathcal{F}[q+\epsilon\nu]-\mathcal{F}[q]}{\epsilon},$ (73)

which exists under some regularity conditions. It can be shown that many
standard rules for derivatives from calculus in $\mathbb{R}$ hold for
functional derivatives also. In particular, our functionals
$\mathcal{F}_{1}[q],\mathcal{F}_{2}[q],\mathcal{F}_{3}[q]$ obtain a
particularly tractable form, that is they are of integral form. Also let
$\mathcal{L}[q]=\lambda\cdot\left(\int
q({\boldsymbol{x}})\mathrm{d}{\boldsymbol{x}}-1\right)$ .The functional
derivative of our functional of interest is then:

$\displaystyle\frac{\delta\mathcal{F}[q]}{\delta q}$
$\displaystyle=\frac{\delta\mathcal{F}_{1}[q]}{\delta
q}+\frac{\delta\mathcal{F}_{2}[q]}{\delta
q}+\frac{\delta\mathcal{F}_{3}[q]}{\delta
q}+\frac{\delta\mathcal{L}[q]}{\delta q}$ (74)
$\displaystyle=-\frac{1}{N}\frac{(f({\boldsymbol{x}})p({\boldsymbol{x}}))^{2}}{q({\boldsymbol{x}})^{2}}-\frac{1}{N}\cdot\mu^{2}\cdot\frac{p({\boldsymbol{x}})^{2}}{q({\boldsymbol{x}})^{2}}+\frac{1}{N}\cdot
2\mu\cdot\frac{p({\boldsymbol{x}})^{2}f({\boldsymbol{x}})}{q({\boldsymbol{x}})^{2}}+\lambda,$
(75)

where we also added a positive Lagrangian multiplier condition to ensure that
$q({\boldsymbol{x}})$ integrates to $1$ (for $\lambda>0$ ). Now equating the
above to $0$, cancelling all $1/N$ and changing sign gives:

$\displaystyle\frac{\delta\mathcal{F}[q]}{\delta q}=0$ (76)
$\displaystyle\Rightarrow~{}~{}$
$\displaystyle\frac{p({\boldsymbol{x}})^{2}(f({\boldsymbol{x}})^{2}+\mu^{2}-2\mu
f({\boldsymbol{x}}))}{q({\boldsymbol{x}})^{2}}=\lambda$ (77)
$\displaystyle\Rightarrow~{}~{}$ $\displaystyle\lambda\cdot
q({\boldsymbol{x}})^{2}=p({\boldsymbol{x}})^{2}(f({\boldsymbol{x}})-\mu)^{2}$
(78) $\displaystyle\Rightarrow~{}~{}$ $\displaystyle
q({\boldsymbol{x}})=\sqrt{\frac{p({\boldsymbol{x}})^{2}(f({\boldsymbol{x}})-\mu)^{2}}{\lambda}}$
(79)

which gives the desired
$q_{\text{SNIS}}^{\bigstar}~{}\propto~{}p({\boldsymbol{x}})|f({\boldsymbol{x}})-\mu|$
(note that the solution is always a positive function). Rather than turn to
complicated second order conditions, we can verify that this is the optimal
solution via the standard Cauchy-Swartz inequality. However, with this
derivation we at least did not have to “guess” the solution beforehand
entirely. This derivation extends that of the optimal $q$ for the UIS
estimator in [63, Chapter 9] to SNIS.

###  G.1 Proof of SNIS asymptotic variance lower bound

Here we give a proof of the lower bound on the asymptotic variance of SNIS
given in Section 2.3. Recall
$q_{\text{SNIS}}^{\bigstar}~{}\propto~{}p({\boldsymbol{x}})|f({\boldsymbol{x}})-\mu|$.
For any PDF $q({\boldsymbol{x}})$, recall that the asymptotic variance of
$\widehat{\mu}_{\text{SNIS}}^{N}$ can be written as:

$\displaystyle\mathbb{V}_{q}^{\infty}[\widehat{\mu}_{\text{SNIS}}^{N}]=\int\frac{p({\boldsymbol{x}})^{2}(f({\boldsymbol{x}})-\mu)^{2}}{q({\boldsymbol{x}})}d{\boldsymbol{x}}.$
(80)

We showed in the previous subsection that $q_{\text{SNIS}}^{\bigstar}$ is the
minimizer of the asymptotic variance. Plugging in $q_{\text{SNIS}}^{\bigstar}$
into the above gives:

$\displaystyle\mathbb{V}_{q_{\text{SNIS}}^{\bigstar}}^{\infty}[\widehat{\mu}_{\text{SNIS}}^{N}]=\int\frac{p({\boldsymbol{x}})^{2}(f({\boldsymbol{x}})-\mu)^{2}}{p({\boldsymbol{x}})|f({\boldsymbol{x}})-\mu|}\cdot\left(\int
p({\boldsymbol{x}})|f({\boldsymbol{x}})-\mu|d{\boldsymbol{x}}\right)d{\boldsymbol{x}}$
(81) $\displaystyle=\int
p({\boldsymbol{x}})|f({\boldsymbol{x}})-\mu|d{\boldsymbol{x}}\cdot\int
p({\boldsymbol{x}})|f({\boldsymbol{x}})-\mu|d{\boldsymbol{x}}$ (82)
$\displaystyle=\left(\int
p({\boldsymbol{x}})|f({\boldsymbol{x}})-\mu|d{\boldsymbol{x}}\right)^{2}.\qed$
(83)

##  Appendix H Delta method MSE of $\widehat{\mu}_{\text{SNIS}}^{N}$ (and
$\widehat{\mu}_{\text{GenSNIS}}^{N}$)

The MSE of $\widehat{\mu}_{\text{SNIS}}$ can be analysed via the delta method,
which obtains a Taylor approximation of 2nd order for the bias, and a 1st
order one for the variance. It is possible to prove that the 1st order Taylor
approximation of the variance coincides with the asymptotic variance, which is
what we use in the main paper. Recall
$\widehat{I}^{N}\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{1}{N}\sum_{n}\frac{f({\boldsymbol{x}}^{(n)})\widetilde{p}({\boldsymbol{x}}^{(n)})}{q({\boldsymbol{x}}^{(n)})},\widehat{Z}_{p}^{N}\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{1}{N}\sum_{n}\frac{\widetilde{p}({\boldsymbol{x}}^{(n)})}{q({\boldsymbol{x}}^{(n)})}$
and that these are unbiased estimators. Note that
$\widehat{\mu}_{\text{SNIS}}^{N}=g(\widehat{I}^{N},\widehat{Z}_{p}^{N})$ with
$g$ being the ratio function, $\widehat{I}^{N}/\widehat{Z}_{p}^{N}$. The delta
method defines first a (here, second-order) Taylor expansion of
$g(\widehat{I},\widehat{Z_{p}})$ around the two-dimensional
$[I,Z_{p}]^{\top}\in\mathbb{R}_{\geq 0}^{2}$:

$\displaystyle g(\widehat{I}^{N},\widehat{Z}_{p}^{N})\approx
g(I,Z_{p})+{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\frac{\partial
g(I,Z_{p})}{\partial\widehat{I}^{N}}}(\widehat{I}^{N}-I)+{\frac{\partial
g(I,Z_{p})}{\partial\widehat{Z}_{p}^{N}}}(\widehat{Z}_{p}^{N}-Z_{p})$ (84)
$\displaystyle+\frac{1}{2}\left({\color[rgb]{0.01953125,0.4609375,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.01953125,0.4609375,0}\frac{\partial
g(I,Z_{p})}{\partial\widehat{I}^{N}\partial\widehat{I}^{N}}}(\widehat{I}^{N}-I)^{2}+2\cdot{\color[rgb]{0.875,0.453125,0}\definecolor[named]{pgfstrokecolor}{rgb}{0.875,0.453125,0}\frac{\partial
g(I,Z_{p})}{\partial\widehat{I}^{N}\partial\widehat{Z}_{p}^{N}}}(\widehat{I}^{N}-I)(\widehat{Z}_{p}^{N}-Z_{p})+{\color[rgb]{1,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,1}\pgfsys@color@cmyk@stroke{0}{1}{0}{0}\pgfsys@color@cmyk@fill{0}{1}{0}{0}\frac{\partial
g(I,Z_{p})}{\partial\widehat{Z}_{p}^{N}\partial\widehat{Z}_{p}^{N}}}(\widehat{Z}_{p}^{N}-Z_{p})^{2}\right)$
(85)
$\displaystyle\stackrel{{\scriptstyle\text{def}}}{{=}}\widehat{g}_{2}(I,Z_{p})$
(86)

where the $2$ in the subscript of $\widehat{g}_{2}$ denotes the order of the
Taylor approximation. Here, $\frac{\partial
g(I,Z_{p})}{\partial\widehat{I}^{N}}$ denotes derivative of
$g(\widehat{I}^{N},\widehat{Z}_{p}^{N})$ w.r.t $\widehat{I}^{N}$ _evaluated
at_ $I$ and $Z_{p}$. Notice that:

$\displaystyle
g(\widehat{I},\widehat{Z_{p}})=\widehat{g}_{2}(I,Z_{p})+\mathcal{O}((\widehat{I}^{N}-I)^{3})+\mathcal{O}((\widehat{Z}_{p}^{N}-Z_{p})^{3})$
(87) $\displaystyle
g(\widehat{I},\widehat{Z_{p}})=\widehat{g}_{1}(I,Z_{p})+\mathcal{O}((\widehat{I}^{N}-I)^{2})+\mathcal{O}((\widehat{Z}_{p}^{N}-Z_{p})^{2}).$
(88)

The big-Oh notation ignores factors that are constant in the differences
$(\widehat{I}^{N}-I)$ and $(\widehat{Z}_{p}^{N}-Z)$. Also remember that
estimators $\widehat{I}^{N}$ and $\widehat{Z}_{p}^{N}$ depend on $N$ and share
the same samples $\\{{\boldsymbol{x}}^{(n)}\\}^{N}_{n=1}$.

Recall that here $I$ and $Z_{p}$ are constants and only $\widehat{I}^{N}$ and
$\widehat{Z}_{p}^{N}$ are r.v.s.

It is now feasible to take expectations to derive bias and variance of
$\widehat{\mu}_{\text{SNIS}}^{N}$. For the bias, we will use
$\widehat{g}_{2}$, while for the variance we use $\widehat{g}_{1}$. To start
with the bias, we look at the expected value

$\displaystyle\mathbb{E}_{q}[\widehat{g}_{2}(I,Z_{p})]=\underbrace{\frac{I}{Z_{p}}}_{\mu}+{\frac{1}{Z_{p}}\cdot
0}-{\frac{I}{Z_{p}^{2}}\cdot
0}-\frac{1}{Z_{p}^{2}}\operatorname{Cov}_{q}[\widehat{I}^{N},\widehat{Z}_{p}^{N}]+\frac{I}{Z_{p}^{3}}\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]$
(89)
$\displaystyle=\mu+\frac{1}{Z_{p}^{2}}\left(\frac{I}{Z_{p}}\cdot\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]-\operatorname{Cov}_{q}[\widehat{I}^{N},\widehat{Z}_{p}^{N}]\right)$
$\displaystyle=\mu+\frac{1}{Z_{p}^{2}}\left(\mu\cdot\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]-\varrho_{\widehat{I}^{N},\widehat{Z}_{p}^{N}}\cdot\sqrt{\mathbb{V}_{q}[\widehat{I}^{N}]\cdot\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]}\right)$

where
$\varrho_{\widehat{I}^{N},\widehat{Z}_{p}^{N}}\stackrel{{\scriptstyle\text{def}}}{{=}}\operatorname{Corr}_{q}[\widehat{I}^{N},\widehat{Z}_{p}^{N}]$.
Note that the expectations become variances here only because the estimators
of $I$ and $Z_{p}$ are unbiased, and would be MSE’s otherwise. A nice feature
of this expression is that it has $\mu$ in front, so the bias of
$\widehat{\mu}_{\text{SNIS}}^{N}$ is immediately seen as the remaining part.

$\displaystyle\operatorname{\text{Bias}}_{q}^{\delta}[\widehat{\mu}_{\text{SNIS}}^{N}]=\frac{1}{Z_{p}^{2}}\left(\mu\cdot\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]-\varrho_{\widehat{I}^{N},\widehat{Z}_{p}^{N}}\cdot\sqrt{\mathbb{V}_{q}[\widehat{I}^{N}]\cdot\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]}\right)$
(90)

What the delta method does is approximating $g$ first by a Taylor
approximation, and then get the _exact_ expectation of that approximation.

#### Variance

We obtain the variance of $\widehat{g}_{1}(I,Z_{p})$, i.e.,

$\displaystyle\mathbb{V}_{q}[\widehat{\mu}_{\text{SNIS}}^{N}]$
$\displaystyle\approx\mathbb{V}_{q}[\widehat{g}_{1}(I,Z_{p})]$ (91)
$\displaystyle=\mathbb{V}_{q}\left[\widehat{g}_{1}(I,Z_{p})\right]=\mathbb{V}_{q}\left[\frac{I}{Z_{p}}+\frac{1}{Z_{p}}(\widehat{I}^{N}-I)-\frac{I}{Z_{p}^{2}}(\widehat{Z}_{p}^{N}-Z_{p})\right]$
(92)

Since $\widehat{I}^{N}$ and $\widehat{Z}_{p}^{N}$ are correlated,
$\frac{I}{Z_{p}}$ is a constant, using
$\mathbb{V}[X-Y]=\mathbb{V}[X]+\mathbb{V}[Y]-2\operatorname{Cov}[X,Y]$ we get

$\displaystyle\mathbb{V}_{q}^{\delta}[\widehat{\mu}_{\text{SNIS}}^{N}]=\frac{1}{Z_{p}^{2}}\mathbb{V}_{q}[\widehat{I}^{N}]+\frac{I^{2}}{Z_{p}^{4}}\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]-\frac{2I}{Z_{p}^{3}}\cdot\operatorname{Cov}_{q}[\widehat{I}^{N},\widehat{Z}_{p}^{N}]$
(93)

Therefore we can get the delta method MSE as the sum

$\displaystyle\operatorname{MSE}_{\delta}[\widehat{\mu}_{\text{SNIS}}^{N}{}]=(\operatorname{\text{Bias}}_{q}^{\delta}[\widehat{\mu}_{\text{SNIS}}^{N}])^{2}+\mathbb{V}_{q}^{\delta}[\widehat{\mu}_{\text{SNIS}}^{N}]$
(94)
$\displaystyle=\frac{1}{Z_{p}^{4}}\left(\mu^{2}\cdot\mathbb{V}_{q}^{2}[\widehat{Z}_{p}^{N}]+\varrho_{\widehat{I}^{N},\widehat{Z}_{p}^{N}}^{2}\mathbb{V}_{q}[\widehat{I}^{N}]\cdot\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]-2\varrho_{\widehat{I}^{N},\widehat{Z}_{p}^{N}}\mu\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]\sqrt{\mathbb{V}_{q}[\widehat{I}^{N}]\cdot\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]}\right)$
(95)
$\displaystyle+\frac{1}{Z_{p}^{2}}\mathbb{V}_{q}[\widehat{I}^{N}]+\frac{I}{Z_{p}^{4}}\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]-\frac{2I}{Z_{p}^{3}}\cdot\varrho_{\widehat{I}^{N},\widehat{Z}_{p}^{N}}\sqrt{\mathbb{V}_{q}[\widehat{I}^{N}]\cdot\mathbb{V}_{q}[\widehat{Z}_{p}^{N}]}$
(96)

##  Appendix I Extension to bridge sampling-type estimators and connections

Here, we connect our method to bridge sampling, a methodology for estimation
of the ratio of normalizing constants [57] (see an excellent explanation in
[63, Chapter 10, Section 9]. Recall notation
$\mu=\mathbb{E}_{p}[f({\boldsymbol{x}})]=I/Z_{p}$. Bridge sampling is a ratio
estimator that also involves sampling from two different distribution,
although there is no joint in an extended space. In fact, bridge sampling
requires us to sample exactly from
$q_{2}^{\bigstar}=p({\boldsymbol{x}})=\widetilde{p}({\boldsymbol{x}})/Z_{p}$
and $q_{1}^{\bigstar}=f({\boldsymbol{x}})p({\boldsymbol{x}})/\mu$. Obtaining
samples
$\\{{\boldsymbol{x}}^{(n_{1})}\\}_{n_{1}=1}^{N_{1}}\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}q_{1}^{\bigstar}({\boldsymbol{x}})$
and
$\\{{\boldsymbol{x}}^{(n_{2})}\\}_{n_{2}=1}^{N_{2}}\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}q_{2}^{\bigstar}({\boldsymbol{x}})$,
the estimator is constructed as

$\widehat{\mu}_{\text{Simple-
bridge}}\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{\frac{1}{N_{2}}\sum_{n_{2}=1}^{N_{2}}f({\boldsymbol{x}}^{(n_{2})})p({\boldsymbol{x}}^{(n_{2})})}{\frac{1}{N_{1}}\sum_{n_{1}=1}^{N_{1}}\widetilde{p}({\boldsymbol{x}}^{(n_{1})})}=\frac{\frac{1}{N_{2}}\sum_{n_{2}=1}^{N_{2}}\widetilde{q}_{1}^{\bigstar}({\boldsymbol{x}}^{(n_{2})})}{\frac{1}{N_{1}}\sum_{n_{1}=1}^{N_{1}}\widetilde{q}_{2}^{\bigstar}({\boldsymbol{x}}^{(n_{1})})},$
(97)

where we used notation for the unnormalized targets
$\widetilde{q}_{1}^{\bigstar}({\boldsymbol{x}})=f({\boldsymbol{x}})p({\boldsymbol{x}})$
and
$\widetilde{q}_{2}^{\bigstar}({\boldsymbol{x}})=\widetilde{p}({\boldsymbol{x}})$.
Notice the use of samples from $q_{2}^{\bigstar}$ in the numerator, while the
samples from $q_{1}^{\bigstar}$ in the denominator, which may be at first
counter-intuitive. Taking expectations of numerator and denominator of Eq. 97
gives

$\frac{\frac{1}{N_{2}}\mathbb{E}_{q_{2}^{\bigstar}}[\sum_{n_{2}=1}^{N_{2}}\widetilde{q}_{1}^{\bigstar}({\boldsymbol{x}}^{(n_{2})})]}{\frac{1}{N_{1}}\mathbb{E}_{q_{1}^{\bigstar}}[\sum_{n_{1}=1}^{N_{1}}\widetilde{q}_{2}^{\bigstar}({\boldsymbol{x}}^{(n_{1})})]}=\frac{Z_{p}^{-1}\int\widetilde{q}_{1}^{\bigstar}({\boldsymbol{x}})\widetilde{q}_{2}^{\bigstar}({\boldsymbol{x}})d{\boldsymbol{x}}}{I^{-1}\int\widetilde{q}_{2}^{\bigstar}({\boldsymbol{x}})\widetilde{q}_{1}^{\bigstar}({\boldsymbol{x}})d{\boldsymbol{x}}}=\frac{Z_{p}^{-1}\cdot
B}{I^{-1}\cdot B}=\frac{I}{Z_{p}}=\mu,$ (98)

where the constant $B$ cancels. Differently to
$\widehat{\mu}_{\text{GenSNIS}}^{N}$, this is a ratio of two unbiased
estimators (almost, they are scaled by the constant $B$) of $1/I$ and
$1/Z_{p}$, instead of $I$ and $Z_{p}$. The asymptotic variance can also be
obtained with the delta method as for $\widehat{\mu}_{\text{GenSNIS}}^{N}$.
Before we do so, in fact the more general version of the estimator is

$\widehat{\mu}_{\text{Bridge}}\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{\frac{1}{N_{2}}\sum_{n_{2}=1}^{N_{2}}\widetilde{q}_{1}^{\bigstar}({\boldsymbol{x}}^{(n_{2})})\alpha({\boldsymbol{x}}^{(n_{2})})}{\frac{1}{N_{1}}\sum_{n_{1}=1}^{N_{1}}\widetilde{q}_{2}^{\bigstar}({\boldsymbol{x}}^{(n_{1})})\alpha({\boldsymbol{x}}^{(n_{1})})}=\frac{\widehat{\eta}_{1}}{\widehat{\eta}_{2}}$
(99)

for some function $\alpha({\boldsymbol{x}})$ which can be chosen to minimize
asymptotic variance.

######  Remark 5.

Keys differences from $\widehat{\mu}_{\text{GenSNIS}}^{N}$ are

  * •

The need to sample exactly from $q_{1}^{\bigstar}$ and $q_{2}^{\bigstar}$,
which is a known disadvantage of bridge sampling that makes it difficult to
apply to certain problems. Quoting [63, Chapter 10, Section 9]: “In the
problems where normalization ratios are most needed, we may not be able to
sample from either $p_{0}$ or $p_{1}$” (the equivalent of our
$q_{1}^{\bigstar}$ and $q_{2}^{\bigstar}$).161616It is possible to use MCMC
for generating such samples, but the analysis of the asymptotic variance and
practical considerations complicate significantly.

  * •

The MSE is lower bounded as in $\widehat{\mu}_{\text{SNIS}}^{N}$, i.e., it is
not theoretically possible to achieve zero MSE.

Therefore the two methodologies, $\widehat{\mu}_{\text{GenSNIS}}^{N}$ and
bridge sampling, can be seen as alternatives depending on the requirements of
the estimation problem. Nonetheless, the core underlying idea behind
$\widehat{\mu}_{\text{GenSNIS}}^{N}$ of improving variance by correlating the
estimates can be applied to give a generalization of bridge sampling.

Delta method for bridge sampling. To do so, let us first derive the asymptotic
variance of bridge sampling. We find the expectations of numerator and
denominator as

$\displaystyle\mathbb{E}_{q_{2}^{\bigstar}}[\widehat{\eta}_{1}]=B\cdot
Z_{p}^{-1}\stackrel{{\scriptstyle\text{def}}}{{=}}\eta_{1},~{}~{}\mathbb{E}_{q_{1}^{\bigstar}}[\widehat{\eta}_{2}]=B\cdot
I^{-1}\stackrel{{\scriptstyle\text{def}}}{{=}}\eta_{2}.$ (100)

Let the ratio of the two estimators
$g(\widehat{\eta}_{1},\widehat{\eta}_{2})=\widehat{\eta}_{1}/\widehat{\eta}_{2}$
be approximated by 1st-order Taylor approximation $\widehat{g}_{1}$ and
evaluated at the true quantities $\eta_{1},\eta_{2}$ (keep in mind,
$\mu=\eta_{1}/\eta_{2}$). The variance of the approximation is

$\displaystyle\mathbb{V}_{\mathbb{Q}_{1:2}^{\bigstar}}\left[\widehat{g}_{1}\left(\eta_{1},\eta_{2}\right)\right]=\mathbb{V}_{\mathbb{Q}_{1:2}^{\bigstar}}\left[\frac{\eta_{1}}{\eta_{2}}+\frac{1}{\eta_{2}}\left(\widehat{\eta_{1}}^{N}-\eta_{1}\right)-\frac{\eta_{1}}{\eta_{2}^{2}}\left(\widehat{\eta}_{2}^{N}-\eta_{2}\right)\right],$
(101)

and since this variance is the asymptotic variance, we have

$\displaystyle\mathbb{V}_{\mathbb{Q}_{1:2}^{\bigstar}}^{\infty}\left[\widehat{\mu}_{\mathrm{Bridge}}^{N}\right]=\frac{1}{\eta_{2}^{2}}\mathbb{V}_{q_{2}^{\bigstar}}\left[\widehat{\eta}_{1}\right]+\frac{\eta_{1}^{2}}{\eta_{2}^{4}}\mathbb{V}_{q_{1}^{\bigstar}}\left[\widehat{\eta}_{2}\right]-\frac{2\eta_{1}}{\eta_{2}^{3}}\cdot\operatorname{Cov}_{q}\left[\widehat{\eta}_{1},\widehat{\eta}_{2}\right],$
(102)

(the coefficient of the third term can also be written as
$-2\mu/\eta_{2}^{2}$) where

$\displaystyle\mathbb{V}_{q_{2}^{\bigstar}}[\widehat{\eta}_{1}]$
$\displaystyle=\frac{1}{N_{2}}\left(\mathbb{E}_{q_{2}^{\bigstar}}[\widetilde{q}_{1}^{\bigstar}({\boldsymbol{x}})^{2}\alpha({\boldsymbol{x}})^{2}]-\left(\int\widetilde{q}_{1}^{\bigstar}({\boldsymbol{x}})\widetilde{q}_{2}^{\bigstar}({\boldsymbol{x}})\alpha({\boldsymbol{x}})\right)^{2}\right)$
(103) $\displaystyle\mathbb{V}_{q_{1}^{\bigstar}}[\widehat{\eta}_{2}]$
$\displaystyle=\frac{1}{N_{1}}\left(\mathbb{E}_{q_{1}^{\bigstar}}[\widetilde{q}_{2}^{\bigstar}({\boldsymbol{x}})^{2}\alpha({\boldsymbol{x}})^{2}]-\left(\int\widetilde{q}_{1}^{\bigstar}({\boldsymbol{x}})\widetilde{q}_{2}^{\bigstar}({\boldsymbol{x}})\alpha({\boldsymbol{x}})\right)^{2}\right).$
(104)

Generalized version of bridge sampling based on couplings. In bridge sampling,
the covariance term of Eq. 102 is zero, as the two sets of samples are
independent. A $\widehat{\mu}_{\text{GenSNIS}}^{N}$ inspired version of bridge
sampling would sample from a joint in extended space to obtain

$\widehat{\mu}_{\text{GenBridge}}\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{\frac{1}{N}\sum_{n=1}^{N}\widetilde{q}_{1}^{\bigstar}({\boldsymbol{x}}_{2}^{(n)})\alpha({\boldsymbol{x}}_{2}^{(n)})}{\frac{1}{N}\sum_{n=1}^{N}\widetilde{q}_{2}^{\bigstar}({\boldsymbol{x}}_{1}^{(n)})\alpha({\boldsymbol{x}}_{1}^{(n)})}=\frac{\widehat{\eta}_{1}}{\widehat{\eta}_{2}},\qquad[{\boldsymbol{x}}_{1}^{(n)},{\boldsymbol{x}}_{2}^{(n)}]\stackrel{{\scriptstyle\text{i.i.d.}}}{{\sim}}\mathbb{Q}_{1:2}^{\bigstar}(d{\boldsymbol{x}}_{1},d{\boldsymbol{x}}_{2})$
(105)

where
$\mathbb{Q}_{1:2}^{\bigstar}(d{\boldsymbol{x}}_{1},d{\boldsymbol{x}}_{2})$ is
a joint with marginals $q_{1}^{\bigstar}({\boldsymbol{x}}_{1})$ and
$q_{2}^{\bigstar}({\boldsymbol{x}}_{2})$. With this estimator, the covariance
term in the asymptotic variance can be controlled as in
$\widehat{\mu}_{\text{GenSNIS}}^{N}$. In $\widehat{\mu}_{\text{GenBridge}}$,
we can control (i) the joint
$\mathbb{Q}_{1:2}^{\bigstar}(d{\boldsymbol{x}}_{1},d{\boldsymbol{x}}_{2})$
with fixed marginals and (ii) the function $\alpha({\boldsymbol{x}})$ used in
the estimator.

##  Appendix J Sample recyling

A straightforward way to use all the samples in the estimation is by having
two UIS estimators in both numerator and denominator, as

$\displaystyle\widehat{\mu}_{\text{GenSNIS-
Recyle}}=\frac{\sum_{n=1}^{N}\frac{f({\boldsymbol{x}}_{1}^{(n)})\cdot\widetilde{p}({\boldsymbol{x}}_{1}^{(n)})}{q_{1}({\boldsymbol{x}}_{1}^{(n)})}+\sum_{n=1}^{N}\frac{f({\boldsymbol{x}}_{2}^{(n)})\cdot\widetilde{p}({\boldsymbol{x}}_{2}^{(n)})}{q_{2}({\boldsymbol{x}}^{(n)}_{2})}}{\sum_{n=1}^{N}\frac{\widetilde{p}({\boldsymbol{x}}^{(n)}_{2})}{q_{2}({\boldsymbol{x}}^{(n)}_{2})}+\sum_{n=1}^{N}\frac{\widetilde{p}({\boldsymbol{x}}_{1}^{(n)})}{q_{1}({\boldsymbol{x}}_{1}^{(n)})}},\qquad[{\boldsymbol{x}}_{1}^{(n)},{\boldsymbol{x}}^{(n)}_{2}]^{\top}\sim\mathbb{Q}_{1:2}({\boldsymbol{x}}_{1},{\boldsymbol{x}}_{2}).$
(106)

In fact, all of the sample recycling variants proposed in TABI [69] can also
be used straightforwardly for $\widehat{\mu}_{\text{GenSNIS}}^{N}$. While such
estimators ahve the advantage of using all the samples in both integrals, the
variance analysis is more complicated, and sample sharing in
numerator/denominator suffers from the same drawback of
$\widehat{\mu}_{\text{SNIS}}^{N}$, in the sense that the implicit CRN coupling
implied by sample sharing is not guaranteed to reduce variance (and could
increase it).


# GDIP: Gated Differentiable Image Processing for Object-Detection in Adverse
Conditions

Sanket Kalwar∗1, Dhruv Patel∗1, Aakash Aanegola1, Krishna Reddy Konda3, Sourav
Garg2, K Madhava Krishna1 * denotes equal contribution1are with RRC, IIIT
Hyderabad, India {sankethkalwar, dhruv.r.patel14<EMAIL_ADDRESS><EMAIL_ADDRESS>with the QUT Centre for Robotics at the Queensland
University of Technology (QUT), Brisbane, Australia<EMAIL_ADDRESS>with
ZF TCI, Hyderabad, India<EMAIL_ADDRESS>page:
https://gatedip.github.io†Code: https://github.com/Gatedip/GDIP-Yolo

###### Abstract

Detecting objects under adverse weather and lighting conditions is crucial for
the safe and continuous operation of an autonomous vehicle, and remains an
unsolved problem. We present a Gated Differentiable Image Processing (GDIP)
block, a domain-agnostic network architecture, which can be plugged into
existing object detection networks (e.g., Yolo) and trained end-to-end with
adverse condition images such as those captured under fog and low lighting.
Our proposed GDIP block learns to enhance images directly through the
downstream object detection loss. This is achieved by learning parameters of
multiple image pre-processing (IP) techniques that operate concurrently, with
their outputs combined using weights learned through a novel gating mechanism.
We further improve GDIP through a multi-stage guidance procedure for
progressive image enhancement. Finally, trading off accuracy for speed, we
propose a variant of GDIP that can be used as a regularizer for training Yolo,
which eliminates the need for GDIP-based image enhancement during inference,
resulting in higher throughput and plausible real-world deployment. We
demonstrate significant improvement in detection performance over several
state-of-the-art methods through quantitative and qualitative studies on
synthetic datasets such as PascalVOC, and real-world foggy (RTTS) and low-
lighting (ExDark) datasets.

##  I INTRODUCTION

Autonomous mobile agents need a high-level understanding of their environment
to plan their trajectories and function effectively. This requires robust
perception which stems from object detection and semantic segmentation like
tasks for recognizing and localizing safety-critical objects such as
pedestrians and vehicles. Most object detection methods are designed for and
trained with images captured under ideal environmental conditions, and often
do not generalize to adverse settings (like foggy and low-light conditions).
Recent attempts to achieve such robustness include domain classification based
invariant detection [2, 3, 4, 5, 6, 7, 8, 9], prior-knowledge based feature
adaptation [10, 11], adversarially-trained image alignment [12, 13], map-
specific domain adaptation [14], and physics-prior based zero-shot learning
[15, 16].

More recently, learnable image pre-processing methods have emerged as a
superior alternative [1, 15, 17, 18, 19, 20, 16, 21, 22, 23]. However, these
methods are either limited to a single preprocessing module [15, 16, 21, 22,
23], require domain-specific architectural variations [18, 19, 1] or use
multiple modules in an arbitrary sequential order [20, 1]. In this work, we
address all these limitations and present a novel approach that learns to
enhance images for object detection under adverse conditions in an end-to-end
manner. This is achieved through a learnable gating-based weighted combination
of concurrent image processing operations, dubbed GDIP (Gated Differentiable
Image Processing). Our proposed GDIP method integrated with Yolo significantly
outperforms the current SOTA, Image Adaptive Yolo (IA-Yolo [1]), which relies
on an arbitrary sequential image preprocessing. On real-word fog (RTTS [24])
and low-light (Ex-Dark [25]) datasets, GDIP-Yolo leads IA-Yolo in mAP by 5.76
and 15.89 respectively. The key contributions of this paper are listed below:

  1. 1.

a novel gating mechanism that enables concurrent relative weighting of
multiple differentiable image processing modules to enhance images for object
detection under adverse environmental conditions;

  2. 2.

a multi-level version of GDIP where an image is progressively enhanced through
multiple GDIP blocks, each guided by a different layer of the image encoder;
and

  3. 3.

an adaptation of GDIP as a training regularizer which directly improves object
detection training for adverse conditions, eliminating the need of GDIP during
inference, thus saving compute time with a minor drop in performance.

##  II RELATED WORK

Object detection is the problem of localizing and classifying objects in the
scene and has seen a recent uptick in popularity due to its applications in
autonomous vehicles and more. There are two primary approaches to the object
detection problem, two-stage detection, and single-stage detection. Two-stage
detectors like FasterRCNN [26] and MaskRCNN [27] utilize a region proposal
network (RPN) which generates proposals of plausible regions of interest, sent
to the downstream network that performs classification. Two-stage approaches
are computationally expensive, reducing their application range. Single-stage
detectors like Yolo [28], RetinaNet [29], SSD [30] and FCOS [31] bypass the
heavy RPN and directly extract objects with their associated labels.
Nevertheless, under adverse conditions, both types of networks fail to detect
objects.

Adverse Conditions: Typical object detection techniques fail in adverse
weather conditions, and transfer learning has proven to be a viable way to
employ object detection in adverse weather conditions. Chen et al. [2]
approach this problem from a domain adaptation perspective and utilize image
and instance-level features to reduce the domain shift. Singadi et al. [10]
use weather-specific knowledge and define a prior-adversarial loss with
feature recovery to mitigate weather effects on detection. Multiscale Domain
Adaptive Yolo (MS-DAYOLO) [32] uses classifiers for each domain at different
scales to learn domain invariant features. Zhang et al. [12] employ image-
level feature alignment to match local and global features.

Differentiable Image Pre-processing: Another popular approach to the problem
is to perform image enhancement before object detection. In Exposure [20], a
deep Reinforcement Learning model learns a policy to apply a sequence of
enhancement operations. AOD-Net [33] dehazes images using a CNN designed on a
re-formulated atmospheric scattering model. Dong et al. [34] use an encoder-
decoder architecture (U-Net) with the strength-operate-subtract boosting
strategy to help dehaze images. GridDehazeNet [18] employs a multi-scale
attention mechanism with pre and post-processing modules to generate better
inputs and reduce artifacts in the final dehazed image. He et al. [35] use a
dark channel prior (one color channel in most pixels will be low) to dehaze
images but do not perform object detection. Some models target only a single
adverse condition, Guo et al. [19] perform light enhancement by estimating
light enhancement curves that are applied iteratively to the input image,
boosting face detection performance. Zeng et al. [36] learn multiple look-up
tables and use CNN predictions to fuse the look-up tables into one and
transform the color and tone of a source image. DSNet [17] uses two subnets
for image restoration and object detection to boost performance in adverse
weather conditions. IA-Yolo [1] uses a CNN to predict differentiable image
processing parameters trained in conjunction with Yolov3 [28] for object
detection to enhance images, and perform object detection in an end-to-end
fashion.

Unlike existing methods, GDIP is a domain agnostic network architecture that
handles multiple image processing operations concurrently. Additionally, it
has a unique advantage with its utility as a training regularizer, which
eliminates image enhancement overhead during inference resulting in higher
throughput.

##  III PROPOSED METHOD

We propose a Gated Differentiable Image Processing (GDIP) framework that
learns to enhance input images for object detection in adverse environmental
conditions. The GDIP block learns parameters for multiple Image Processing
(IP) operations performed concurrently and learns the optimal weights to
combine their output. We use the following IP operations (similar to IA-Yolo
[1]): tone correction ($T$), contrast balance ($C$), sharpening ($S$),
defogging ($DF$), gamma correction ($G$), white balancing ($WB$), and the
identity operation ($I$). Unlike IA-Yolo’s sequential image enhancement, GDIP
enhances images through a weighted combination of concurrent IP operations.

###  III-A Gated Differentiable Image Processing (GDIP) block:

Figure 2: GDIP Block: The peach blocks represent different $Gb_{IP}$ and their
structures can be seen in the expanded general $Gb$ block (top left corner).
Yolo and the object detection output are not shown due to space constraints.

The GDIP block (shown in Fig. 2) consists of multiple gated image processing
modules, referred to as $Gb_{IP}$, that individually enhance images, which are
then combined through the weights predicted by the gates. Each $Gb$ module
contains a linear layer, a differentiable image processing operation, a gate
(shifted tanh function that returns a value between 0 and 1), and a
normalization operation. The linear layer (purple linear block in Fig. 2)
computes two entities: the parameters required by the differentiable IP block
and a scalar value that serves as an input to its corresponding gate. The
individual linear layers of every $Gb$ module are passed a common feature
embedding as input, obtained from a shared vision encoder (described later).
The output of the IP operation (using the predicted parameters) gets
multiplied by the scalar output of the gate. The weighted outputs of
individual $Gb$ blocks are finally aggregated to obtain an enhanced image.
Expressed mathematically, the output of the GDIP block is:

$z=N(\sum_{i}N(f_{i}(x))*w_{i})$ (1)

where $x$ is the input image captured under adverse environmental conditions,
$z$ is the enhanced clear image, $f_{i}(x)$ represents the $i^{th}$ IP
operation (top-right in Fig. 2) weighted by its respective scalar gate output
$w_{i}\in[0,1]$, and $N$ is the min-max normalization operation. Normalization
ensures that the pixel intensity range of the output of all the image
processing operations are the same. The IP operations are expressed
mathematically in Table I, see [1] for a detailed description.

TABLE I: Image Processing (IP) Operations. The parameters in the second column are computed by the linear layer in GDIP. Further details on the equations can be found in IA-Yolo [1] IP Operation | Parameters | Function
---|---|---
Tone | $t_{i}$ |  $I_{tone}=(L_{t_{r}}(r_{i}),L_{t_{g}}(g_{i}),L_{t_{b}}(b_{i}))$
Contrast | $\alpha$ | $I_{contrast}=\alpha*En(I)+(1-\alpha)*I$
Sharpening | $\lambda$ | $I_{sharpen}=I+\lambda*(I-Gaussian(I))$
Defogging | $\omega$ | $I=I_{defog}*t(\omega)+A*(1-t(\omega))$
Gamma | $\gamma$ | $I_{gamma}=I^{\gamma}$
White balance | $W_{r},W_{g},W_{b}$ | $I_{wb}=(W_{r}r_{i},W_{g}g_{i},W_{b}b_{i})$
Identity | - | $I_{identity}=I$

Vision Encoder: Our proposed GDIP block requires latent embeddings to compute
image processing parameters and gate values. For this purpose, we employ a
vision encoder comprising five convolutional layers (each with a kernel size
of three and a stride of one). The number of channels in each layer is double
the previous, starting from 64 in the first layer and 1024 in the final layer.
Each convolution operation is followed by average pooling (with kernel size
three and stride two), while the last layer is followed by global average
pooling, the output of which is a 1x1x1024. This is then projected to a
256-dimensional latent space using a fully connected layer. The GDIP block
takes this 256-dimensional embedding from the vision encoder along with the
adverse input image and performs image enhancement after computing the
necessary parameters.

GDIP-Yolo: To integrate GDIP with Yolo, we use the vision encoder with GDIP to
perform image enhancements (depicted in Fig. 2), and use the enhanced image as
input to Yolo. Integrating GDIP with Yolo in this fashion ensures that our
architecture doesn’t require any additional loss formulation and uses Yolo’s
standard object detection loss [37] (referred to as $L_{obj}$) to train the
network for object detection end-to-end.

Figure 3: MGDIP-Yolo: The GDIP blocks perform enhancements progressively on
the input image starting with $GDIP_{1}$ obtaining intermediate features from
$C1$ (each GDIP block still performs IP operations concurrently). The final
enhanced image is passed to Yolo and the whole pipeline is trained end-to-end.

###  III-B Multi-Level GDIP (MGDIP):

GDIP-Yolo contains a single GDIP block, which is fed with latent embeddings
obtained from the vision encoder. Since, we only use the last layer of the
vision encoder for this purpose, it limits the extent of information available
for GDIP to learn parameters for image processing modules. Thus, we propose
multi-level progressive image enhancement, achieved by integrating a GDIP
block with every layer of the vision encoder, dubbed MGDIP-Yolo. Note that the
individual image processing modules within a single GDIP block still operate
concurrently with their corresponding gates providing relative weightings. As
shown in Fig. 3, MGDIP progressively enhances images by feeding the output
from one GDIP block as input to the next, where individual GDIP blocks are
guided by the features extracted from different layers of the vision encoder.
The final enhanced output from MGDIP is passed to Yolo for object detection.
MGDIP-Yolo is trained in an end-to-end manner using the standard object
detection loss $L_{obj}$, similar to GDIP-Yolo.

We hypothesize that utilizing embeddings from different layers provides GDIP
access to multiple feature scales, each of which can have a varied relevance
for different image processing operations. This is based on the understanding
that earlier layers in CNNs capture lower level information (local information
like edges) and later layers capture high-level (global) information. Thus,
MGDIP gains the ability to use the local/global feature properties to
selectively apply image processing operations.

Figure 4: GDIP as a regularizer: The purple GDIP blocks and the reconstruction
loss help regularize Yolo’s features. During inference, everything apart from
Yolo is removed.

###  III-C GDIP block as a regularizer:

In this section, we demonstrate how GDIP can also be employed as a feature
regularization technique to improve Yolo’s performance while maintaining its
throughput.

The original GDIP block used a vision encoder to obtain feature embeddings.
Alternatively, multiple GDIP blocks can be connected to intermediate layers of
Yolo, bypassing the need for a vision encoder and directly using Yolo’s
embeddings to construct an enhanced output, as shown in Fig. 4. Note that this
enhanced output is not the input to Yolo but rather a byproduct that we use
for training regularization. The reconstruction loss (Eq. 2) is calculated
between this output and the clear version of the input image as a combination
of $L_{1}$ norm and Mean Square Error loss $L_{MSE}$. The overall loss
function used is shown in Eq. 3, where $\alpha$ is the weight of the
reconstruction loss and is empirically set to $1\times 10^{-4}$.  $\small
L_{Reg}=L_{1}+L_{MSE}$ (2) $\small L_{total}=L_{obj}+\alpha L_{Reg}$ (3)

Inclusion of the reconstruction loss in the formulation helps Yolo learn
features that are invariant to adverse conditions, resulting in better
performance when compared to standalone Yolo. Since the GDIP blocks exist
solely to refine Yolo’s features, it is only required during training and can
be removed during inference. This results in an unchanged network architecture
(Yolo) that performs better in adverse weather conditions along with higher
throughput.

##  IV EXPERIMENTAL SETUP

###  IV-A Datasets

Foggy Conditions: We use the RTTS dataset [24], a collection of 4322 natural
foggy images with five annotated classes - person, car, bus, bicycle and
motorcycle - primarily for testing. The PascalVOC train/val datasets (2007 and
2012) [38, 39] have 22136 clear images that we use as a base to create a
synthetic training set. We select images from PascalVOC having objects
belonging to the five classes from RTTS and create two datasets, one with
clear images (VOCNormal) and one with augmented foggy images generated using
the atmospheric scattering model (ASM) [40]. We employ the ASM to generate 10
different levels of fog to include variance in our synthetic training set. We
subsample and prepare a synthetic testing set in a similar fashion from the
PascalVOC 2007 test set with 4952 images (referred to as V_F_Ts). We employ a
hybrid strategy where we use a mix of foggy and clear images (in a 2:1 ratio)
to help our model learn fog-invariant features.

Low-lighting Conditions: The ExDark dataset [25] is a collection of 7363 real-
world images with 10 object classes in low lighting conditions that we use to
evaluate our models. Similar to preparing the foggy dataset, we select images
from PascalVOC having objects from the 10 classes of ExDark and apply a gamma
filter to emulate a low-lighting condition. Mathematically,
$I_{dark}=I^{\gamma}$, where $\gamma$ is sampled uniformly from the range of
1.5 to 5, $I$ is the normalized clear image, and $I_{dark}$ is the synthetic
dark image. Using the same selection and image processing methods, we generate
a synthetic low-light test set (V_D_Ts) from the PascalVOC test set. During
training, we employ a hybrid strategy (similar to the foggy setting) by using
a mix of dark and clear images.

###  IV-B Training Setup

Training for both foggy and low-lighting setting is done by resizing images to
$448\times 448\times 3$ pixels and with a batch size of 6 for 80 epochs. We
use a cosine learning rate scheduler with learning rates ranging from $1\times
10^{-6}$ to $1\times 10^{-4}$ and an SGD optimizer with a weight decay of
$5\times 10^{-4}$.

##  V RESULTS AND ANALYSES

We provide qualitative and quantitative results that establish our proposed
method’s superiority and evaluate design choices through ablation studies. We
also show that GDIP variants provide flexibility to prioritize speed or
accuracy based on application requirements.

###  V-A Qualitative Analysis

We compare the results of GDIP-Yolo with the current SOTA IA-Yolo, shown in
Fig. LABEL:fig:qualitative on real-world data. Unlike IA-Yolo, our method
clears fog and improves lighting conditions without changing underlying color
distributions. Our method is able to detect far-off objects such as cars and
bicycles (see the third column in Fig. LABEL:fig:qualitative), which are
generally missed in extreme foggy conditions by SOTA IA-Yolo (see the second
column). The last column (low-light conditions) clearly indicates GDIP-Yolo
enhances lighting without artifacts, helping the model detect all objects in
the scene (the car in the background, for example). We quantify the
improvement extended by GDIP in the next section.

###  V-B Quantitative Analysis

We compare our proposed method with other SOTA works using the standard object
detection evaluation metric - mean average precision (mAP). All mAP values are
calculated at an IoU (Intersection over Union) of 0.5.

#### Foggy Conditions

In Table II, we compare our proposed variants of the GDIP with other competing
methods on VOCNormal Test set (V_N_Ts), synthetic VOCFoggy Test set (V_F_Ts),
and the real-world foggy dataset RTTS. The second column in the table shows
the training data used by each of the methods, where “Hybrid” implies the use
of both clear and foggy data. We set YoloV3 as the baseline, which is trained
on a mix of foggy and clear images to validate if we can improve performance
by using data augmentation. We also compare our results against a diverse
range of methods based on domain adaption (DA-Yolo [32]), multi-task learning
(DSNet [17]), defogging as pre-processing (MSBDN [34], GridDehaze [18]), and
adaptive image enhancement (IA-Yolo [1]). It can be observed in Table II that
our proposed variants of GDIP establish a new SOTA across different fog
datasets.

TABLE II: Quantitative results for foggy conditions on the V_N_Ts (VOCNormal Test set), V_F_Ts (VOCFoggy Test set) and real-world RTTS dataset. Best and second best mAP scores are bold and italicized, respectively.  Methods | Train Data | V_N_Ts | V_F_Ts | RTTS
---|---|---|---|---
|  |  | (mAP) | (mAP) | (mAP)
Baseline |  Yolov3 [28] | Hybrid | 64.13 | 63.40 | 30.80
|  MSBDN [34] | VOC_Norm | - | 57.38 | 30.20
Defog |  GridDehaze [18] | VOC_Norm | - | 58.23 | 31.42
Domain |  DAYolo [32] | Hybrid | 56.51 | 55.11 | 29.93
Adaptation |  |  |  |  |
Multi-task |  DSNet [17] | Hybrid | 53.29 | 67.40 | 28.91
Image |  IA-Yolo [1] | Hybrid | 73.23 | 72.03 | 37.08
Adaptive |  |  |  |  |
| GDIP-Yolo | Hybrid | 73.70 | 71.92 | 42.42
Proposed | MGDIP-Yolo | Hybrid | 75.36 | 73.37 | 42.84
Method | GDIP | Hybrid | 73.17 | 72.77 | 39.52
| Regularizer |  |  |  |

All GDIP variants perform significantly better than SOTA methods on RTTS,
which tests the generalizability of our method to real-world conditions. Our
basic GDIP-Yolo variant outperforms the SOTA method IA-Yolo by 5.34 mAP. This
can be attributed to the concurrently weighted IP operations unlike IA-Yolo’s
fixed sequential pipeline. MGDIP-Yolo further improves upon the GDIP-Yolo by
0.42 mAP and does so consistently across all datasets. It emerges superior to
all other methods and GDIP variants, as it benefits from multi-scale
information. Our regularizer variant outperforms all SOTA methods, while being
as fast as vanilla Yolo (see Subsection V-E), emerging as an alternative to
GDIP-Yolo and MGDIP-Yolo with an accuracy-speed trade-off.

#### Low-lighting conditions

We compare our proposed variants with other SOTA methods on the real-world
ExDark dataset, synthetic low-lighting VOCDark test set (V_D_Ts), and
VOCNormal test set (V_N_Ts), as shown in Table III. Here once again, we set
YoloV3 as the baseline trained on hybrid data of a mix of dark and clear
images. In addition, we also compare against a diverse range of methods based
on light enhancement as pre-processing (ZeroDCE [19]), domain adaptation (DA-
Yolo [32]), multi-task learning (DSNet [17]) and adaptive image enhancement
(IA-Yolo [1]).

TABLE III: Quantitative results for low-lighting conditions on the V_N_Ts (VOCNormal Test set), V_D_Ts (VOCDark Test set) and real-world ExDark dataset. Best and second best mAP scores are bold and italicized, respectively. Methods | Train Data | V_N_Ts | V_D_Ts | ExDark
---|---|---|---|---
|  |  | (mAP) | (mAP) | (mAP)
Baseline |  Yolov3 [28] | Hybrid | 62.73 | 52.28 | 37.03
Enhance |  ZeroDCE [19] | VOC_Norm | - | 33.57 | 34.41
Domain |  DAYolo [32] | Hybrid | 41.68 | 21.53 | 18.15
Adaptation |  |  |  |  |
Multi-task |  DSNet [17] | Hybrid | 64.08 | 43.75 | 36.97
Image |  IA-Yolo [1] | Hybrid | 56.01 | 48.44 | 26.67
Adaptive |  |  |  |  |
| GDIP-Yolo | Hybrid | 63.23 | 57.85 | 42.56
Proposed | MGDIP-Yolo | Hybrid | 62.86 | 57.91 | 40.96
Method | GDIP | Hybrid | 62.30 | 57.67 | 40.72
| Regularizer |  |  |  |

Our proposed GDIP-Yolo outperforms all the existing methods on the real-world
ExDark dataset and achieves an absolute increase of 16 mAP over the previous
SOTA IA-Yolo. Additionally, MGDIP-Yolo and GDIP as a regularizer variants also
perform superior to other methods in this setting. In the synthetic low-
lighting setting, MGDIP-Yolo approach emerges superior, while the other GDIP
variants also significantly improve the performance over other methods. On the
VOCNormal test set, our proposed method performs comparable to DSNet [17]. To
conclude, our proposed variants are significantly better than existing
approaches for low-light settings (synthetic and real-world).

Figure 5: Alternative Metrics: We indicate the True Positives (TP), False
Negatives (FN) and False Positives (FP) on the RTTS and ExDark datasets for
GDIP-Yolo (blue) and IA-Yolo (red). The x and y axes in the plots represent
the objectness (detection confidence) threshold and the absolute number of
TP/FN/FP. Detection confidence reflects the likelihood of box containing the
object.

###  V-C Detection Statistics

We present True and False Positives (TP, FP) and False Negatives (FN) of the
number of object detections as an interpretable statistical measure as mAP
does not convey the actual detections. The TP (Fig. 5 left) and FN (Fig. 5
middle) plots show substantial improvement at high object detection confidence
thresholds both for RTTS and Ex-Dark datasets for GDIP-Yolo vis a vis SOTA IA-
Yolo. For Autonomous Driving applications, the TP and FN statistics are
critical, as not detecting an object when present can be catastrophic, and on
these vital statistics, the significantly superior performance of GDIP is
evident. GDIP-Yolo evaluates to comparable FP metrics vis a vis SOTA at high
confidence thresholds on which it shows vastly superior performance on the TP,
and FN metrics.

###  V-D Ablation Studies

In this section, we demonstrate experimental support for using the gating
mechanism and normalization layer in our proposed GDIP block. We validate that
incorporating gating and normalization in the GDIP block provides a stable
improvement in the downstream detection task for both foggy and low-light
conditions (as shown in Table IV).

Figure 6: GDIP gate firing pattern: The bar charts show the gates firing for
low lighting and foggy images. GDIP learns the optimum enhancements to apply
based on the input image features.

Single Best vs Weighted Combination: Our proposed gating mechanism helps
combine image processing operations through relative weighting (see Eq. 1). To
illustrate its effect, we remove this mechanism and use a single best image
processing operation based on the highest gate value (referred to as GDIP-max
in Table IV), expressed as $z=N(f_{i^{*}}(x))$ where
$i^{*}=\mathrm{argmax}_{i}\ w_{i}$. Without the proposed gating, performance
reduces by 10.4 mAP for the RTTS dataset (comparing row 1 and 2 in Table IV).
For the ExDark dataset the performance increases by a negligible amount - 0.15
mAP, insubstantial compared to the drop observed in the foggy setting. This
study indicates that incorporating enhancements from multiple image processing
operations is necessary as no single operation is sufficient for dealing with
adverse conditions.

Uniform vs Predicted Weighting: In this experiment, we compare our proposed
relative weighting of image processing operations with a uniform weighting
across all operations (referred to as GDIP w/o gates in Table IV), expressed
mathematically as $z=N(\sum_{i}N(f_{i}(x)))$. We observe that mAP reduces by
0.77 and 0.27 for RTTS and ExDark, respectively. This performance drop can be
attributed to the fact that depending on the environmental conditions of
images, image processing operations need to be weighted differently, achieved
through gating. This is evident from Fig. 6, where activation of the gamma
gate (G) is prominent for low light conditions, and in the case of foggy
conditions, most of the gates remain activated with defogging (DF) being the
highest. The gate activation patterns also help improve the interpretability
of GDIP by indicating which IP operations are performed by what proportions
based on the input image.

With and Without Normalization: We also verify the necessity for the
normalization layer after each image processing operation by removing them
(GDIP-unnormalized in Table IV), expressed as $z=N(\sum_{i}f_{i}(x)*w_{i})$.
This leads to a considerable performance drop of around 1.8 and 2.35 mAP in
RTTS and ExDark, respectively.

Overall, these ablation studies indicate that GDIP with normalization and the
gating mechanism leads to the best overall performance irrespective of
environmental conditions and is a promising solution for the object detection
task.

TABLE IV: Ablation Study

Method |  RTTS (mAP) |  ExDark (mAP)
---|---|---
GDIP | $\mathbf{42.42}$ | 42.56
GDIP-max | 31.99 | $\mathbf{42.71}$
GDIP-unnormalized | 40.61 | 40.2
GDIP w/o gates | 41.65 | 42.29

###  V-E Real-time performance

In Table V, we compare the real-time performance of our proposed GDIP variants
with other techniques. GDIP peforms the fastest as a regularizer at around 68
fps on a Nvidia GTX 1080Ti, which is the same as YoloV3. Our basic variant -
GDIP-Yolo operates at 7 fps higher than IA-Yolo, while achieving SOTA mAP on
real-world fog and night datasets.

TABLE V: Real-time performance on GeForce GTX 1080

Methods | FPS
---|---
Yolo V3 | 68.39 $\pm$ 1.5
IA-Yolo | 22.84 $\pm$ 0.0
GDIP-Yolo | 29.78 $\pm 0.1$
MGDIP-Yolo (top-down) | 11.38 $\pm$ 0.02
MGDIP-Yolo (bottom-up) | 11.25 $\pm$ 0.03
GDIP as regularizer |  68.39 $\pm$ 1.5

##  VI CONCLUSION

We presented GDIP and MGDIP as domain-agnostic network architectures for
object detection in adverse weather conditions, which can be used with
existing object detection networks and trained under different adverse
conditions, as we demonstrated for fog and low lighting. We also presented a
training regularizer variant of GDIP, which improves the baseline Yolo
performance under adverse conditions while maintaining its original
throughput. All our GDIP variants result in a new state-of-the-art on
challenging real-world datasets both under foggy and low-lighting conditions,
while only having trained on synthetic adverse condition data, thus exhibiting
significant generalization capability.

In future, this work can be extended to other adverse condition types (e.g.,
haze, rain, snow, etc.) along with additional relevant image pre-processing
operations which are easy to integrate given their concurrent processing and
relative weighting within GDIP. For long-term autonomy and highly-safe
operations, dealing with adverse conditions is crucial for autonomous
vehicles, and this work pushes the boundaries of robust perception, getting a
step closer to the ubiquity of autonomous vehicles.

## References

  * [1] W. Liu, G. Ren, R. Yu, S. Guo, J. Zhu, and L. Zhang, “Image-adaptive yolo for object detection in adverse weather conditions,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2022.
  * [2] Y. Chen, W. Li, C. Sakaridis, D. Dai, and L. Van Gool, “Domain adaptive faster r-cnn for object detection in the wild,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3339–3348.
  * [3] H.-K. Hsu, C.-H. Yao, Y.-H. Tsai, W.-C. Hung, H.-Y. Tseng, M. Singh, and M.-H. Yang, “Progressive domain adaptation for object detection,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , March 2020.
  * [4] K. Tian, C. Zhang, Y. Wang, S. Xiang, and C. Pan, “Knowledge mining and transferring for domain adaptive object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2021, pp. 9133–9142.
  * [5] A. Wu, R. Liu, Y. Han, L. Zhu, and Y. Yang, “Vector-decomposed disentanglement for domain-invariant object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9342–9351.
  * [6] A. Wu, Y. Han, L. Zhu, and Y. Yang, “Instance-invariant domain adaptive object detection via progressive disentanglement,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [7] C. Lin, Z. Yuan, S. Zhao, P. Sun, C. Wang, and J. Cai, “Domain-invariant disentangled network for generalizable object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 8771–8780.
  * [8] Q. Gu, Q. Zhou, M. Xu, Z. Feng, G. Cheng, X. Lu, J. Shi, and L. Ma, “Pit: Position-invariant transform for cross-fov domain adaptation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 8761–8770.
  * [9] Z. He and L. Zhang, “Domain adaptive object detection via asymmetric tri-way faster-rcnn,” in _European conference on computer vision_. Springer, 2020, pp. 309–324.
  * [10] V. A. Sindagi, P. Oza, R. Yasarla, and V. M. Patel, “Prior-based domain adaptive object detection for hazy and rainy conditions,” in _European Conference on Computer Vision_. Springer, 2020, pp. 763–780.
  * [11] Z. Zhang, L. Zhao, Y. Liu, S. Zhang, and J. Yang, “Unified density-aware image dehazing and object detection in real-world hazy scenes,” in _Proceedings of the Asian Conference on Computer Vision_ , 2020.
  * [12] S. Zhang, H. Tuo, J. Hu, and Z. Jing, “Domain adaptive yolo for one-stage cross-domain detection,” in _Asian Conference on Machine Learning_. PMLR, 2021, pp. 785–797.
  * [13] C. Zhuang, X. Han, W. Huang, and M. Scott, “ifan: Image-instance full alignment networks for adaptive object detection,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 13 122–13 129.
  * [14] C. Sakaridis, D. Dai, and L. Van Gool, “Map-guided curriculum domain adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [15] A. Lengyel, S. Garg, M. Milford, and J. C. van Gemert, “Zero-shot day-night domain adaptation with a physics prior,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 4399–4409.
  * [16] S. Zheng and G. Gupta, “Semantic-guided zero-shot learning for low-light image/video enhancement,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops_ , January 2022, pp. 581–590.
  * [17] S.-C. Huang, T.-H. Le, and D.-W. Jaw, “Dsnet: Joint semantic learning for object detection in inclement weather conditions,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 43, no. 8, pp. 2623–2633, 2020.
  * [18] X. Liu, Y. Ma, Z. Shi, and J. Chen, “Griddehazenet: Attention-based multi-scale network for image dehazing,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 7314–7323.
  * [19] C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong, and R. Cong, “Zero-reference deep curve estimation for low-light image enhancement,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1780–1789.
  * [20] Y. Hu, H. He, C. Xu, B. Wang, and S. Lin, “Exposure: A white-box photo post-processing framework,” _ACM Transactions on Graphics (TOG)_ , vol. 37, no. 2, pp. 1–17, 2018.
  * [21] W. Yang, S. Wang, Y. Fang, Y. Wang, and J. Liu, “From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [22] G. Li, Y. Yang, X. Qu, D. Cao, and K. Li, “A deep learning based image enhancement approach for autonomous driving at night,” _Knowledge-Based Systems_ , vol. 213, p. 106617, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950705120307462
  * [23] Z. Zhang, L. Zhao, Y. Liu, S. Zhang, and J. Yang, “Unified density-aware image dehazing and object detection in real-world hazy scenes,” in _Proceedings of the Asian Conference on Computer Vision (ACCV)_ , November 2020.
  * [24] B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang, “Benchmarking single image dehazing and beyond,” _IEEE Transactions on Image Processing_ , vol. PP, pp. 1–1, 08 2018.
  * [25] Y. P. Loh and C. S. Chan, “Getting to know low-light images with the exclusively dark dataset,” _Computer Vision and Image Understanding_ , vol. 178, pp. 30–42, 2019.
  * [26] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [27] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [28] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [29] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [30] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [31] Z. Tian, C. Shen, H. Chen, and T. He, “Fcos: Fully convolutional one-stage object detection,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 9627–9636.
  * [32] M. Hnewa and H. Radha, “Multiscale domain adaptive yolo for cross-domain object detection,” in _2021 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2021, pp. 3323–3327.
  * [33] B. Li, X. Peng, Z. Wang, J. Xu, and D. Feng, “Aod-net: All-in-one dehazing network,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 4770–4778.
  * [34] H. Dong, J. Pan, L. Xiang, Z. Hu, X. Zhang, F. Wang, and M.-H. Yang, “Multi-scale boosted dehazing network with dense feature fusion,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2157–2167.
  * [35] K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 33, no. 12, pp. 2341–2353, 2010.
  * [36] H. Zeng, J. Cai, L. Li, Z. Cao, and L. Zhang, “Learning image-adaptive 3d lookup tables for high performance photo enhancement in real-time,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [37] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [38] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, “The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results,” http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html.
  * [39] ——, “The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results,” http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html.
  * [40] S. Narasimhan and S. Nayar, “Vision and the atmosphere,” _International Journal of Computer Vision_ , vol. 48, pp. 233–254, 07 2002.