# Heisenberg-Limited Waveform Estimation with Solid-State Spins in Diamond

Yang Dong  Ze-Hao Wang  Hao-Bin Lin  Shao-Chun Zhang  Yu Zheng  Xiang-Dong
Chen  CAS Key Laboratory of Quantum Information, University of Science and
Technology of China, Hefei, 230026, P.R. China  CAS Center for Excellence in
Quantum Information and Quantum Physics, University of Science and Technology
of China, Hefei, 230026, P.R. China  Wei Zhu  Guan-Zhong Wang  Hefei National
Laboratory for Physical Science at Microscale, and Department of Physics,
University of Science and Technology of China, Hefei, Anhui 230026, P. R.
China  Guang-Can Guo  Fang-Wen Sun<EMAIL_ADDRESS>CAS Key Laboratory of
Quantum Information, University of Science and Technology of China, Hefei,
230026, P.R. China  CAS Center for Excellence in Quantum Information and
Quantum Physics, University of Science and Technology of China, Hefei, 230026,
P.R. China

(August 27, 2024)

###### Abstract

The newly established Heisenberg limit in arbitrary waveform estimation is
quite different with parameter estimation and shows a unique characteristic of
a future quantum version of oscilloscope. However, it is still a non-trivial
challenge to generate a large number of exotic quantum entangled states to
achieve this quantum limit. Here, by employing the time-domain quantum
difference detection method, we demonstrate Heisenberg-limited waveform
quantum estimation with diamond spins under ambient condition in the
experiment. Periodic dynamical decoupling is applied to enhance both the
dynamic range and sensitivity by one order of magnitude. Using this quantum-
enhanced estimation scheme, the estimation error of an unknown waveform is
reduced by more than $5$ dB below the standard quantum limit with
$N\sim{\text{2}}\times{\text{1}}{{\text{0}}^{3}}$ resources, where more than
${1\times{\text{1}}{{\text{0}}^{5}}}$ resources would be required to achieve a
similar error level using classical detection. This work provides an essential
step towards realizing quantum-enhanced structure recognition in a continuous
space and time.

Quantum metrology Giovannetti et al. (2011); Pezzè et al. (2018); Braun et al.
(2018); Pirandola et al. (2018); Sun et al. (2008); Higgins et al. (2007);
Dong et al. (2016); Kessler et al. (2014) takes its superpower from
superposition Streltsov et al. (2017); Dong et al. (2018) and entanglement
Nishioka (2018); Dong et al. (2019); Han et al. (2020) to yield higher
statistical precision than pure classical approaches. And the Heisenberg
quantum limit (HQL) measurement ($\delta=O({{N^{-1}}})$) totally outperforms
the standard quantum limit (SQL) ($\delta=O({{N^{-1/2}}})$), where $N$ is the
number of resources. Over the last few decades, lots of experimental systems,
such as multi-photon interferometers Pirandola et al. (2018); Pan et al.
(2012); Hou et al. (2019); Motes et al. (2015), trapped ions Monz et al.
(2011); Omran et al. (2019), superconducting circuits Song et al. (2019), and
solid-spin Bradley et al. (2019); Liu et al. (2015); Dong et al. (2016),
exploited quantum entanglement to demonstrate this non-classical sensitivity.
To extend current results for widely high precision metrology applications, it
is natural to consider the continuous Abadie et al. (2011); Kura and Ueda
(2020); Cooper et al. (2014); Zopes and Degen (2019); Michalet et al. (2005)
nature of the detected signal, which can be categorized as an arbitrary
waveform estimation to further construct a quantum version of oscilloscope.
Unlike scalar and vector estimation Giovannetti et al. (2011); Pezzè et al.
(2018); Braun et al. (2018); Pirandola et al. (2018); Sun et al. (2008);
Higgins et al. (2007); Dong et al. (2016), arbitrary waveform estimation
involves infinite degrees of freedom. Theoretically, the statistical error is
bounded by the HQL of $O\left({{N^{-q/(q+1)}}}\right)$ over the SQL of
$O\left({{N^{-q/(2q+1)}}}\right)$ to demonstrate the quantum superiority Kura
and Ueda (2020), where $q$ is a measure of the degree of smoothness of the
waveform. It is significantly different with quantum-enhanced parameter
estimation Higgins et al. (2007); Giovannetti et al. (2011); Pezzè et al.
(2018); Braun et al. (2018); Pirandola et al. (2018); Sun et al. (2008); Dong
et al. (2016). This is a milestone in quantum metrology which will be further
applied in the efficient detection of functional structures–continuous
physical signal including the detection of the nanoscale nuclear magnetic
resonance Budker and Romalis (2007); Abadie et al. (2011); Kura and Ueda
(2020); Cooper et al. (2014), nano-materials Brida et al. (2010); Samantaray
et al. (2017), event horizons Akiyama et al. (2019a, b), and living cell based
on quantum systems.

In the canonical form of quantum-enhanced metrology, achieving HQL scaling for
such a waveform estimation usually requires the use of large size exotic
quantum entangled states Pezzè et al. (2018); Kura and Ueda (2020); Chen et
al. (2018); Kitagawa and Ueda (1993); Zhang et al. (2017), which is still a
non-trivial challenge with currently available quantum technology Higgins et
al. (2007); Giovannetti et al. (2011); Pezzè et al. (2018); Braun et al.
(2018); Kuzyk and Wang (2018). In this work, we experimentally demonstrate an
HQL arbitrary reproducible waveform estimation based on the time-domain
quantum difference (TDQD) protocol with the electron spin of nitrogen-vacancy
(NV) center in diamond. The basic idea is to use a multi-pass scheme Higgins
et al. (2007) to coherently amplify the unknown detection signal, cancel
unwanted coherent dynamical evolution and suppress quantum decoherence
simultaneously. By combing with periodic dynamical decoupling (PDD) method,
both dynamic range and sensitivity for waveform estimation are improved by one
order of magnitude Zopes and Degen (2019). Finally, the scaling law of HQL for
waveform estimation is achieved in the experiment with such a PDD-enhanced
TDQD protocol, which significantly beats the results of SQL by more than $5$
dB and demonstrates the unique characteristic of the quantum version of
oscilloscope.

Figure 1: Conceptual diagrams for waveform estimation with ${n_{1}}$ samplings
and ${n_{2}}$ independent or correlated quantum resources. The total error
includes the deterministic error (golden shadow) and statistical error (red
error bar).

In general, we consider the estimation of an unknown waveform $b(t)$ defined
over an interval $t\in\left[{0,T}\right]$ with a quantum probe, where $b(t)$
can be stored in a relative phase $\phi(t)$ of its quantum state and sampled
using an impulse train of $n_{1}$ impulses, as shown in Fig. 1. The phase
$\phi({t_{i}})$ at each moment ${t_{i}}$ is measured by ${n_{2}}$ resources.
Finally, we can conveniently generate the waveform estimator $\tilde{\phi}(t)$
with the individual estimators ${\phi(t_{i})}$ by a zero-order hold method
Oppenheim et al. (1997); Cooper et al. (2014); Kura and Ueda (2020):
$\tilde{\phi}(t)=\sum\nolimits_{i=1}^{{n_{1}}}{{\phi(t_{i})}\theta\left({t-{t_{i}}}\right)}$,
where $\theta$ is a smoothing function with $\theta\left(t\right)=1$ for
$\left|t\right|\leqslant T/{2n_{1}}$ and $\theta\left(t\right)=0$ otherwise.
For a given stochastic estimator $\tilde{\phi}$, the waveform estimation error
Kura and Ueda (2020) (see Supplemental Material for details SM (SM)) can be
directly decomposed into two parts

$\displaystyle{\delta^{2}}=\delta_{det}^{2}+\delta_{stat}^{2}\text{,}$ (1)

where ${\delta_{stat}}$ is the statistical error caused by the quantum
projection measurement, and ${\delta_{det}}$ is the deterministic error caused
by the smoothing process, as shown in Fig. 1. When the waveform is $q$th-order
differentiable, ${\delta_{det}}=O\left({n_{1}^{-q}}\right)$ Kura and Ueda
(2020); SM (SM). And standard measurement schemes using each resource
independently lead to a classical phase uncertainty that scales as
${\delta_{stat}}=O({n}_{2}^{-1/2})$. While, by employing quantum correlations
Higgins et al. (2007); Giovannetti et al. (2011); Pezzè et al. (2018); Braun
et al. (2018), the quantum enhanced measurements toward
${\delta_{stat}}=O({n}_{2}^{-1})$ will be available. Hence, for a given total
number of quantum resources $N={n_{1}}{n_{2}}$, the optimal measurement is
determined by the trade-off between these two errors. By employing the
inequality of arithmetic and geometric means, the estimation error of
arbitrary waveform is bounded by the SQL of
$\delta=O\left({{N^{-q/(2q+1)}}}\right)$ and the HQL of
$\delta=O\left({{N^{-q/(q+1)}}}\right)$ Kura and Ueda (2020); SM (SM). For the
present case $q=1$, the optimal allocation of quantum resource for the best
arbitrary waveform estimation can be realized with
${n_{1}}=O\left({{N^{1/3}}}\right)$, ${n_{2}}=O\left({{N^{2/3}}}\right)$ for
SQL scheme and ${n_{1}}=O\left({{N^{1/2}}}\right)$,
${n_{2}}=O\left({{N^{1/2}}}\right)$ for HQL scheme.

Figure 2: (a) PDD-enhanced TDQD protocol for arbitrary reproducible waveform
detection with NV center. (b) The detection of a sinusoidal waveform
$b(t)=b\sin\omega t$ with a period of $T=2.4$ $\mu s$. The integration time is
$t_{\operatorname{s}}=300$ ns and $\pi$-pulse duration is ${t_{\pi}}=50$ ns.
(c) The readout of unknown phase with Rabi oscillation with $k=15$ and
$t_{i}=450$ ns, corresponding to the intersection of white and black dashed
lines in Fig. 2(b). (d) The accumulation phase as a function of $k$ with the
PDD-enhanced (blue circles) and normal Zopes and Degen (2019) (red circles)
TDQD protocols. Lines are theoretical results. With the PDD-enhanced TDQD
protocol, the solid blue line shows a linear scaling. The uncertainty of phase
estimation Dong et al. (2016); Sun et al. (2008); Barry et al. (2020) (error
bars) is calculated in a simple way $\Delta{\phi_{k}}=\frac{\sigma}{A}$, where
$\sigma$ is signal fluctuation and $A$ is the amplitude of Rabi oscillation as
shown in Fig. 2(c). (e) The sensitivity of waveform estimation with the PDD-
enhanced (blue circles) and normal (red circles) TDQD protocol. Without taking
the decoherence into consideration, the sensitivity scales as the dashed gray
line. The best experimental sensitivity (blue dots) is $370$ nT$/\sqrt{Hz}$
for $k=64$.

Here, we experimentally demonstrate the arbitrary reproducible waveform
estimation with the negatively charged NV center in diamond Barry et al.
(2020); Clevenson et al. (2015); Dong et al. (2018); Chen et al. (2019); Dong
et al. (2018); SM (SM), where a substitutional nitrogen atom is next to a
vacancy, forming a spin triplet system in its ground state. A room-temperature
home-built confocal microscopy is employed to image, initialize and read out
the NV center in a single-crystal synthetic diamond sample, which is mounted
on a three-axis closed-loop piezoelectric stage for sub-micrometer-resolution
scanning. Fluorescence photons are collected into a fiber and detected using
single-photon counting module. A copper wire of $10$ $\mu$m diameter above on
the bulk diamond is used for the delivery of microwave (MW) and waveform to
the NV center. The optical and MW pulse sequences are synchronized by a
multichannel pulse generator. Single NV centers are identified by observing
anti-bunching in photon correlation measurements.

We perform the waveform estimation by employing a PDD-enhanced TDQD protocol
which is based on the differential spin-echo detection Zopes and Degen (2019),
as shown in Fig. 2(a). Firstly, we prepare NV center into a probe state
$\frac{{\left|0\right\rangle+\left|1\right\rangle}}{{\sqrt{2}}}$. Then, the
superposition state evolves under the unknown magnetic field $b\left(t\right)$
along the spin’s quantization axis with a relative phase
$\phi(t)=\int_{0}^{t}\gamma_{e}b\left(t\right)dt$, where $\gamma_{e}$ is the
gyromagnetic ratio of the spin. We can selectively acquire the phase from the
time interval $[t_{i},t_{i}+{t_{s}}]$ while canceling out unwanted dynamically
phase evolution by inserting two $\pi$ pulses at times $t_{i}$ and
$t_{i}+{t_{s}}$. For ${t_{s}}/T\ll 1$, we have
$\phi({t_{i}})\approx-2{\gamma_{e}}b({t_{i}}){t_{s}}$ SM (SM). Finally, the
accumulation phase is transferred into the initial phase of Rabi oscillation
and measured with optical method. The protocol can be repeated $k$ times to
linearly increase the accumulation phase (${\phi_{{n_{2}}}}=k\phi({t_{i}})$).
Then, the number of resource used in TDQD protocol is $n_{2}=2k$ for each
sampling. To suppress the quantum dephasing effect of ${}^{13}C$ nuclear spins
in diamond Dong et al. (2016), we exploit the PDD method by adding a time-
delay of $2{t_{s}}$ after the waveform finishing, as shown in Fig. 2(a). In
the present case, the quantum dynamic evolution of ${}^{13}C$ nuclear spins
under the magnetic field $b(t)$ can be neglected
(${\gamma_{{}^{13}C}}b{T_{2}}\ll 1$) and is approximately dominated by the
static magnetic field and hyperfine field conditioned on the electron spin
state Zhao et al. (2012); SM (SM). The PDD-enhanced TDQD protocol, which is
${\left[{{t_{i}}-\pi-\left({T+{t_{s}}}\right)-\pi-\left({T-{t_{i}}+{t_{s}}}\right)}\right]^{k}}$,
consists of a sequence of $\pi$ flips for the sensor evolution. By simply
merging the free-evolution propagator term, it turns out to be
$\left[{{t_{i}}-PD{D_{k-1}}-\pi-\left({T+{t_{s}}}\right)-\pi-\left({T-{t_{i}}+{t_{s}}}\right)}\right]$,
where
$PD{D_{k-1}}={\left[{\pi-\left({T+{t_{s}}}\right)-\pi-\left({T+{t_{s}}}\right)}\right]^{k-1}}$.
So the experimental signal is
$s\sim\exp\left[{-{{\left({\frac{{2k\left({T+{t_{s}}}\right)}}{{{T_{2}}}}}\right)}^{2}}}\right]\cos{\phi_{{n_{2}}}}$
SM (SM).

We first investigate the experimental result by accumulating phase from
${n_{2}}=2k$ consecutive waveform passages. Fig. 2(b) plots the readout of
accumulation phase from a weak sinusoidal test signal recorded with different
values of $k$. With the increase of $k$, a much stronger oscillation response
is observed with a much higher dynamic range Zopes and Degen (2019); Arai et
al. (2018). Fig. 2(c) shows the readout process of an unknown accumulation
phase with $k=15$ and $t_{i}=450$ ns. Furthermore, we explore the sensor
signal as a function of $k$ with a fixed $t_{i}=450$ ns, as shown in Fig.
2(d). For a serial of $k$ values, the accumulation phase is proportional to
$k$. Without correcting the decoherence of NV center, we observe the exact
linear scaling relationship. The overall sensitivity Zopes and Degen (2019) in
the presence of decoherence is also calculated and shown in Fig. 2(e).
Obviously, the PDD-enhanced TDQD protocol extends dynamic range and improved
the sensitivity of waveform estimation by one order of magnitude comparing
with the normal TDQD protocol Zopes and Degen (2019); SM (SM). Besides those
advantages, the multi-pulse is also naturally immune to imperfect pulse error
operation with phase modulation Genov et al. (2017); Wang et al. (2019).

To demonstrate the superiority of HQL in waveform estimation, we choose a
$1$st-order differentiable waveform in the experiment. And more importantly,
the balance between ${\delta_{stat}}$ and ${\delta_{\det}}$ should be
optimized by tuning the width of smoothing ($t_{s}$). Here, we have an
approximate relationship: ${n_{1}}\sim{n_{2}}$ for large $N$, which indicates
$\frac{T}{{{t_{s}}+2{t_{\pi}}}}\sim\frac{{{T_{2}}}}{T}$ or
$T\approx\sqrt{{T_{2}}\left({{t_{s}}+2{t_{\pi}}}\right)}\approx 10$ $\mu s$,
where ${T_{2}=0.66(2)}$ ms SM (SM) is the coherence time of NV center and $T$
is the period of waveform. Here, we choose $T=9.6$ $\mu s$ SM (SM).

Figure 3: SQL and HQL in waveform estimation. (a) The scaling laws (solid
lines) ${\delta_{stat}}=O({n_{2}^{-1/2}})$ for SQL and
${\delta_{stat}}=O({n_{2}^{-1}})$ for HQL, respectively. (b) The deterministic
error. The solid golden line shows ${\delta_{det}}=O({n_{1}^{-1}})$. (c)
Results of SQL and HQL in waveform estimation. The colored dots represent
experimental results. The integration time is ${t_{s}}=\frac{T}{{{n_{2}}}}$
and $T=9.6$ $\mu$s is the period of the sinusoidal waveform. Figure 4: An
example of waveform detection with PDD-enhanced TDQD protocol. (a) Results of
the SQL scheme. (b) Results of the HQL scheme. The total number of resources
is $N={2^{8}}$. The experimental results are shown in stack graph by Y
offsets. The input waveform (solid blue line) and the recorded waveform data
(colored dots) are given by
$b\left(t\right){=b}\left[{{\sin(2}\pi{t/T)+0}{.5\sin(4}\pi{t/T)+0}{.25\sin(8}\pi{t/T)}}\right]$
with $T=9.6$ $\mu s$. (c) Waveform estimation errors (colored dots) with
different resource distributions. When the balance between these two errors is
broken, the overall waveform estimation errors increase SM (SM).

We test the basic scaling laws ${\delta_{stat}}=O(n_{2}^{-1/2})$ for the SQL
with small-interval Ramsey sensing sequence Zopes and Degen (2019); SM (SM)
and ${\delta_{stat}}=O(n_{2}^{-1})$ for the HQL in the estimation of a
sinusoidal waveform. The experimental results are shown in Fig. 3(a), together
with theoretical calculations. The sensitivity of the unknown accumulation
phase is therefore increased by $\sqrt{{n_{2}}}$-fold compared with
uncorrelated quantum resource, yielding the Heisenberg scaling. Results in
Fig. 3(a) also show that as the number of quantum resources further increases
when $n\geqslant 64$, ${\delta_{stat}}$ starts to gradually deviate from
theoretical values of HQL. This is caused by the decoherence effects
(${T_{2}}\sim 64T$) of the NV center SM (SM). However, for the SQL scheme, we
can neglect this decoherence effect. Furthermore, the results of
$\delta_{det}$ are shown in Fig. 3(b). There is an apparent contradiction
between experimental and theoretical results for the SQL scheme because
$\delta_{stat}$ is quite large. In this case, noise in waveform reconstruction
greatly increases due to the imperfection convergency of experimental data and
the relationship ${\delta_{det}}=O\left({n_{1}^{-1}}\right)$ will be
overwhelmed by noise. However, this type of error is reduced by employing HQL
scheme and the basic scaling laws ${\delta_{det}}=O({n_{1}^{-1}})$ emerges.
When $\delta_{stat}$ deviates from the theoretical prediction under
decoherence, $\delta_{det}$ will also deviate from $O({n_{1}^{-1}})$. So these
two kinds of estimation errors are tangled up in the experiment for the
waveform reconstruction. By making use of the inequality of arithmetic and
geometric means, the trade-off between ${\delta_{stat}}$ and ${\delta_{det}}$
can be optimized numerically SM (SM). Fig. 3(c) shows this key experimental
results of overall waveform estimation errors. For the $1$st-order
differentiable waveform estimation, the fundamental limits bounded by
$\delta{\text{ = }}O\left({{N^{-1/3}}}\right)$ for SQL and $\delta{\text{ =
}}O\left({{N^{-1/2}}}\right)$ for HQL are experimentally demonstrated. The
HQL-scaled waveform estimation based on the PDD-enhanced TDQD protocol clearly
demonstrates the superiority of quantum metrology. For example, we have
demonstrated the use of $\sim 2\times{{\text{10}}^{3}}$ resources to achieve
the waveform estimation error $5$ dB below the SQL. While more than
${1\times{\text{1}}{{\text{0}}^{5}}}$ resources would be required to achieve a
similar error level using standard classical techniques.

We further complete our study by demonstrating the reconstruction of a complex
test waveform, which contains the sum of several frequency components. The
experimentally measured waveform in SQL and HQL scheme together with the input
waveform are shown in Fig. 4(a) and (b), respectively. With the same number of
resources, the superiority of HQL scheme for waveform estimation is observed
clearly in the experiment. By employing quantum correlation with the PDD-
enhanced TDQD protocol, the statistical error is reduced and the measurement
results converge to the ideal value quickly. Clearly, the overall waveform
estimation error $\delta$ is changed with the proportion of resource
distribution. For a given number of resource $N=n{{}_{1}}{n_{2}}$, the optimal
accuracy is determined by the trade-off between ${\delta_{stat}}$ and
${\delta_{det}}$, as shown in Fig. 4(c). When ${n_{1}}\sim{n_{2}}$, the
optimal waveform reconstruction of $q=1$ is realized in HQL scheme with NV
center. A similar relationship also applies to the SQL case SM (SM).

In summary, we present a PDD-enhanced TDQD protocol for Heisenberg-limited
arbitrary reproducible waveform estimation. The current results demonstrate
the potential of the quantum measurement technique, which is readily available
for other quantum probe systems Degen et al. (2017); Mitchell and Palacios
Alvarez (2020); Pirandola et al. (2018). For the $1$st-order differentiable
arbitrary waveform, we demonstrate the SQL of $O\left({{N^{{-}1/3}}}\right)$
and the HQL of $O\left({{N^{{-}1/2}}}\right)$ in the experiment, which is
significantly different with previous parametric estimation. This new
fundamental result sets a new quantum-enhanced metrology scaling law of high
precision detection of continuous physical signals Abadie et al. (2011); Kura
and Ueda (2020); Cooper et al. (2014). And far more than physical scope, the
estimation of continuous signals and images in micro-fluidic chemical analysis
Lim et al. (2015), vital activity Michalet et al. (2005); Barry et al. (2016)
and pattern recognition Wang et al. (2018) in computer science would be
significantly improved by making use of the quantum superiority. Moreover, the
waveform sampling based on the TDQD protocol will benchmark the quantum
version of the Nyquist-Shannon sampling theorem Kura and Ueda (2020), which
would directly knock on the door of the practical quantum oscilloscope.

Beyond the current work, multiple entangled state Pirandola et al. (2018);
Omran et al. (2019); Song et al. (2019); Han et al. (2020) can be applied to
achieve same quantum-enhanced waveform estimation with parallel schemes. And
more general unpredictable signal can be detected at one time. For the TDQD
and parallel schemes, same quantum coherent amplification of unknown phase can
be realized and the detection resolution can reach the Heisenberg-limited
scaling. That is the quantum Cramér-Rao bound can be asymptotically approached
in both cases. Here, the equivalence between entanglement in parallel schemes
and coherence (namely, superposition in the eigenbasis of the generator) can
be understood by observing that both nonclassicality in infinite-dimensional
systems and coherence (superposition) in finite-dimensional systems can be
converted to entanglement within a well-defined resource-theoretic framework
Braun et al. (2018); Nishioka (2018). This precise understanding and control
of genuine quantum effects such as nonclassicality and superposition will
further extend the application of quantum metrology in the future.

This work is supported by the National Key Research and Development Program of
China (Grant No. 2017YFA0304504), the National Natural Science Foundation of
China (Grants No. 91536219, No. 61522508, and No. 91850102), the Anhui
Initiative in Quantum Information Technologies (Grant No. AHY130000), the
Science Challenge Project (Grant No. TZ2018003), and the Fundamental Research
Funds for the Central Universities (No. WK2030000020).

## References

  * Giovannetti et al. (2011) V. Giovannetti, S. Lloyd, and L. Maccone, Nat. Photonics 5, 222 (2011).
  * Pezzè et al. (2018) L. Pezzè, A. Smerzi, M. K. Oberthaler, R. Schmied, and P. Treutlein, Rev. Mod. Phys. 90, 035005 (2018).
  * Braun et al. (2018) D. Braun, G. Adesso, F. Benatti, R. Floreanini, U. Marzolino, M. W. Mitchell, and S. Pirandola, Rev. Mod. Phys. 90, 035006 (2018).
  * Pirandola et al. (2018) S. Pirandola, B. R. Bardhan, T. Gehring, C. Weedbrook, and S. Lloyd, Nat. Photonics 12, 724 (2018).
  * Sun et al. (2008) F. W. Sun, B. H. Liu, Y. X. Gong, Y. F. Huang, Z. Y. Ou, and G. C. Guo, EPL 82, 24001 (2008).
  * Higgins et al. (2007) B. L. Higgins, D. W. Berry, S. D. Bartlett, H. M. Wiseman, and G. J. Pryde, Nature 450, 393 (2007).
  * Dong et al. (2016) Y. Dong, X.-D. Chen, G.-C. Guo, and F.-W. Sun, Phys. Rev. A 94, 052322 (2016).
  * Kessler et al. (2014) E. M. Kessler, I. Lovchinsky, A. O. Sushkov, and M. D. Lukin, Phys. Rev. Lett. 112 (2014).
  * Streltsov et al. (2017) A. Streltsov, G. Adesso, and M. B. Plenio, Rev. Mod. Phys. 89, 041003 (2017).
  * Dong et al. (2018) Y. Dong, Y. Zheng, S. Li, C.-C. Li, X.-D. Chen, G.-C. Guo, and F.-W. Sun, npj Quantum Inf. 4, 3 (2018).
  * Nishioka (2018) T. Nishioka, Rev. Mod. Phys. 90, 035007 (2018).
  * Dong et al. (2019) Y. Dong, X.-D. Chen, G.-C. Guo, and F.-W. Sun, Phys. Rev. B 100, 214103 (2019).
  * Han et al. (2020) B. Han, J. Duan, S. Jin, X. Lu, P. Li, W. Qu, M. Wang, N. Irina, E. E. Mikhailov, Z. Kai-Feng, et al., Nature 581, 159 (2020).
  * Pan et al. (2012) J.-W. Pan, Z.-B. Chen, C.-Y. Lu, H. Weinfurter, A. Zeilinger, and M. Żukowski, Rev. Mod. Phys. 84, 777 (2012).
  * Hou et al. (2019) Z. Hou, R.-J. Wang, J.-F. Tang, H. Yuan, G.-Y. Xiang, C.-F. Li, and G.-C. Guo, Phys. Rev. Lett. 123, 040501 (2019).
  * Motes et al. (2015) K. R. Motes, J. P. Olson, E. J. Rabeaux, J. P. Dowling, S. J. Olson, and P. P. Rohde, Phys. Rev. Lett. 114, 170802 (2015).
  * Monz et al. (2011) T. Monz, P. Schindler, J. T. Barreiro, M. Chwalla, D. Nigg, W. A. Coish, M. Harlander, W. Hänsel, M. Hennrich, and R. Blatt, Phys. Rev. Lett. 106, 130506 (2011).
  * Omran et al. (2019) A. Omran, H. Levine, A. Keesling, G. Semeghini, T. T. Wang, S. Ebadi, H. Bernien, A. S. Zibrov, H. Pichler, S. Choi, et al., Science 365, 570 (2019).
  * Song et al. (2019) C. Song, K. Xu, H. Li, Y.-R. Zhang, X. Zhang, W. Liu, Q. Guo, Z. Wang, W. Ren, J. Hao, et al., Science 365, 574 (2019).
  * Bradley et al. (2019) C. E. Bradley, J. Randall, M. H. Abobeih, R. C. Berrevoets, M. J. Degen, M. A. Bakker, M. Markham, D. J. Twitchen, and T. H. Taminiau, Phys. Rev. X 9, 031045 (2019).
  * Liu et al. (2015) G.-Q. Liu, Y.-R. Zhang, Y.-C. Chang, J.-D. Yue, H. Fan, and X.-Y. Pan, Nat. Commun. 6, 6726 (2015).
  * Abadie et al. (2011) J. Abadie, B. Abbott, R. Abbott, T. D. Abbott, M. R. Abernathy, C. Adams, R. X. Adhikari, C. Affeldt, B. Allen, G. S. Allen, et al., Nat. Phys. 7, 962 (2011).
  * Kura and Ueda (2020) N. Kura and M. Ueda, Phys. Rev. Lett. 124, 010507 (2020).
  * Cooper et al. (2014) A. Cooper, E. Magesan, H. N. Yum, and P. Cappellaro, Nat. Commun. 5, 3141 (2014).
  * Zopes and Degen (2019) J. Zopes and C. Degen, Phys. Rev. Appl. 12, 054028 (2019).
  * Michalet et al. (2005) X. Michalet, F. Pinaud, L. Bentolila, J. Tsay, S. Doose, J. Li, G. Sundaresan, A. Wu, S. Gambhir, and S. Weiss, Science 307, 538 (2005).
  * Budker and Romalis (2007) D. Budker and M. Romalis, Nat. Phys. 3, 227 (2007).
  * Brida et al. (2010) G. Brida, M. Genovese, and I. R. Berchera, Nat. Photonics 4, 227 (2010).
  * Samantaray et al. (2017) N. Samantaray, I. Ruo-Berchera, A. Meda, and M. Genovese, Light Sci. Appl. 6, e17005 (2017).
  * Akiyama et al. (2019a) K. Akiyama, A. Alberdi, W. Alef, K. Asada, R. Azulay, A.-K. Baczko, D. Ball, M. Balokovic, J. Barrett, D. Bintley, et al., Astrophys. J. 875, L3 (2019a).
  * Akiyama et al. (2019b) K. Akiyama, A. Alberdi, W. Alef, K. Asada, R. Azulay, A.-K. Baczko, D. Ball, M. Balokovic, J. Barrett, D. Bintley, et al., Astrophys. J. 875, L4 (2019b).
  * Chen et al. (2018) G. Chen, L. Zhang, W.-H. Zhang, X.-X. Peng, L. Xu, Z.-D. Liu, X.-Y. Xu, J.-S. Tang, Y.-N. Sun, D.-Y. He, et al., Phys. Rev. Lett. 121, 060506 (2018).
  * Kitagawa and Ueda (1993) M. Kitagawa and M. Ueda, Phys. Rev. A 47, 5138 (1993).
  * Zhang et al. (2017) Y.-C. Zhang, X.-F. Zhou, X. Zhou, G.-C. Guo, and Z.-W. Zhou, Phys. Rev. Lett. 118, 083604 (2017).
  * Kuzyk and Wang (2018) M. C. Kuzyk and H. Wang, Phys. Rev. X 8, 041027 (2018).
  * Oppenheim et al. (1997) A. V. Oppenheim, A. S. Willsky, and S. Hamid, _Signals and Systems_ (Prentice Hall, Englewood Cliffs, NJ, 1997).
  * SM (SM) See Supplemental Material at [] for the calculation of estimation error of waveform, intruduction to the NV center, experimental setup, TDQD and PDD-enhanced TDQD protocol, and the optimal resource allocation of SQL and HQL schemes, which includes Refs. [38-40].
  * Souza et al. (2011) A. M. Souza, G. A. Álvarez, and D. Suter, Phys. Rev. Lett. 106, 240501 (2011).
  * Herbschleb et al. (2019) E. D. Herbschleb, H. Kato, Y. Maruyama, T. Danjo, T. Makino, S. Yamasaki, I. Ohki, K. Hayashi, H. Morishita, M. Fujiwara, et al., Nat. Commun. 10, 3766 (2019).
  * Nocedal and Wright (2006) J. Nocedal and S. J. Wright, _Numerical Optimization_ (Springer, New York, 2006).
  * Barry et al. (2020) J. F. Barry, J. M. Schloss, E. Bauch, M. J. Turner, C. A. Hart, L. M. Pham, and R. L. Walsworth, Rev. Mod. Phys. 92, 015004 (2020).
  * Clevenson et al. (2015) H. Clevenson, M. E. Trusheim, C. Teale, T. Schroeder, D. Braje, and D. Englund, Nat. Phys. 11, 393 (2015).
  * Chen et al. (2019) X.-D. Chen, D.-F. Li, Y. Zheng, S. Li, B. Du, Y. Dong, C.-H. Dong, G.-C. Guo, and F.-W. Sun, Phys. Rev. Appl. 12, 044039 (2019).
  * Dong et al. (2018) Y. Dong, S.-C. Zhang, H.-B. Lin, X.-D. Chen, W. Zhu, G.-Z. Wang, G.-C. Guo, and F.-W. Sun, arXiv:2003.02472 (2020).
  * Zhao et al. (2012) N. Zhao, S.-W. Ho, and R.-B. Liu, Phys. Rev. B 85, 115303 (2012).
  * Arai et al. (2018) K. Arai, J. Lee, C. Belthangady, D. R. Glenn, H. Zhang, and R. L. Walsworth, Nat. Commun. 9, 4996 (2018).
  * Genov et al. (2017) G. T. Genov, D. Schraft, N. V. Vitanov, and T. Halfmann, Phys. Rev. Lett. 118, 133202 (2017).
  * Wang et al. (2019) Z.-Y. Wang, J. E. Lang, S. Schmitt, J. Lang, J. Casanova, L. McGuinness, T. S. Monteiro, F. Jelezko, and M. B. Plenio, Phys. Rev. Lett. 122 (2019).
  * Degen et al. (2017) C. L. Degen, F. Reinhard, and P. Cappellaro, Rev. Mod. Phys. 89, 035002 (2017).
  * Mitchell and Palacios Alvarez (2020) M. W. Mitchell and S. Palacios Alvarez, Rev. Mod. Phys. 92, 021001 (2020).
  * Lim et al. (2015) K. Lim, C. Ropp, B. Shapiro, J. M. Taylor, and E. Waks, Nano Lett. 15, 1481 (2015).
  * Barry et al. (2016) J. F. Barry, M. J. Turner, J. M. Schloss, D. R. Glenn, Y. Song, M. D. Lukin, H. Park, and R. L. Walsworth, Proc. Natl. Acad. Sci. 113, 14133 (2016).
  * Wang et al. (2018) Z. Wang, S. Joshi, S. Savel’ev, W. Song, R. Midya, Y. Li, M. Rao, P. Yan, S. Asapu, Y. Zhuo, et al., Nat. Electron. 1, 137 (2018).


# Multi-task Learning with 3D-Aware
Regularization

Wei-Hong Li1, Steven McDonagh1, Ales Leonardis2, Hakan Bilen1
1University of Edinburgh, 2University of Birmingham
github.com/VICO-UoE/MTPSL

###### Abstract

Deep neural networks have become a standard building block for designing
models that can perform multiple dense computer vision tasks such as depth
estimation and semantic segmentation thanks to their ability to capture
complex correlations in high dimensional feature space across tasks. However,
the cross-task correlations that are learned in the unstructured feature space
can be extremely noisy and susceptible to overfitting, consequently hurting
performance. We propose to address this problem by introducing a structured
3D-aware regularizer which interfaces multiple tasks through the projection of
features extracted from an image encoder to a shared 3D feature space and
decodes them into their task output space through differentiable rendering. We
show that the proposed method is architecture agnostic and can be plugged into
various prior multi-task backbones to improve their performance; as we
evidence using standard benchmarks NYUv2 and PASCAL-Context.

##  1 Introduction

Learning models that can perform multiple tasks coherently while efficiently
sharing computation across tasks are the central focus of multi-task learning
(MTL) (Caruana, 1997). Deep neural networks (DNNs), which have become the
standard solution for various computer vision problems, provide at least two
key advantages for MTL. First, they allow for sharing a significant portion of
features and computation across multiple tasks, hence they are computationally
efficient for MTL. Second, thanks to their hierarchical structure and high-
dimensional representations, they can capture complex cross-task correlations
at several abstraction levels (or layers).

Yet designing multi-task DNNs that perform well in all tasks is extremely
challenging. This often requires careful engineering of mechanisms that allow
for the sharing of relevant features between tasks, while also maintaining
task-specific features. Many multi-task methods (Vandenhende et al., 2021) can
be decomposed into shared feature encoder across all tasks and following task-
specific decoders to generate predictions. The technical challenge here is to
strike a balance between the portion of the shared and task-specific features
to achieve good performance-computation trade-off. To enable more flexible
feature sharing and task-specific adaptation, Liu et al. (2019) propose to use
‘soft’ task-specific attention modules appended to the shared encoder that
effectively shares most features and parameters across the tasks while
adapting them to each task through light-weight attention modules. However,
these attention modules are limited to share features across tasks only within
each layer (or scale). Hence, recent works (Vandenhende et al., 2020b;
Bruggemann et al., 2021) propose to aggregate features from different layers
and to capture cross-task relations from the multi-scale features. More
recently, Ye & Xu (2022a) demonstrates that capturing long-range spatial
correlations across multiple tasks achieves better MTL performance through use
of vision transformer modules (Dosovitskiy et al., 2020).

In this paper we propose an approach orthogonal to existing MTL methods and
hypothesize that high-dimensional and unstructured features, shared across
tasks, are prone to capturing noisy cross-task correlations and hence hurt
performance. To this end, we propose regulating the feature space of shared
representations by introducing a structure that is valid for all considered
tasks. In particular, we look at dense prediction computer vision problems
such as monocular depth estimation, semantic segmentation where each input
pixel is associated with a target value, and represent their shared
intermediate features in a 3D-aware feature space by leveraging recent
advances in 3D modeling and differentiable rendering (Niemeyer et al., 2020;
Mildenhall et al., 2020; Chan et al., 2022; 2023; Anciukevičius et al., 2023).
_Our key intuition is that the physical 3D world affords us inherent and
implicit consistency between various computer vision tasks._ Hence, by
projecting high-dimensional features to a structured 3D-aware space, our
method eliminates multiple geometrically-inconsistent cross-task correlations.

To this end, we propose a novel regularization method that can be plugged into
diverse prior MTL architectures for dense vision problems including both
convolutional (Vandenhende et al., 2020b) and transformer (Ye & Xu, 2022a)
networks. Prior MTL architectures are typically composed of a shared feature
extractor (encoder) and multiple task-specific decoders. Our regularizer,
instantiated as a deep network, connects to the output of the shared feature
encoder, maps the encodings to three groups of feature maps and further uses
these to construct a tri-plane representing planes $x{-}y$, $x{-}z$, $y{-}z$,
in similar fashion to Chan et al. (2022). We are able to query any 3D position
by projecting it onto the tri-plane and retrieve a corresponding feature
vector through bi-linear interpolation across the planes, passing them through
light-weight, task-specific decoders and then rendering the outputs as
predictions for each task by raycasting, as in Mildenhall et al. (2020). Once
the model has been optimized by minimizing each task loss for both the base
model and regularizer, the regularizer is removed. Hence our method does not
bring any additional inference cost. Importantly, the regularizer does not
require multiple views for each scene and learns 3D-aware representations from
a single view. Additionally, the model generalizes to unseen scenes, as the
feature encoder is shared across different scenes.

Our method relates to both MTL and 3D modelling work. It is orthogonal to
recent MTL contributions that focus rather on designing various cross-task
interfaces (Vandenhende et al., 2020a; Liu et al., 2019), or optimization
strategies that may obtain more balanced performance across tasks (Kendall et
al., 2018; Chen et al., 2018). Alternatively, our main focus is to learn
better MTL representations by enforcing 3D structure upon them, through our
3D-aware regularizer. We show that our method can be incorporated with several
recent MTL methods and improve their performance. Most related to ours, Zhi et
al. (2021) and Kundu et al. (2022) extend the well-known neural radiance field
(NeRF) (Mildenhall et al., 2020) to semantic segmentation and panoptic 3D
scene reconstruction, respectively. First, unlike them, our main focus is to
jointly perform multiple tasks that include depth estimation, boundary
detection, surface normal estimation, in addition to semantic segmentation.
Second, uniquely, our method does not require multiple views. Finally, our
method is not scene-specific, can learn multiple scenes in a single model and
generalizes to unseen scenes.

To summarize, our main contribution is a novel 3D-aware regularization method
for the MTL of computer vision problems. Our method is architecture agnostic,
does not bring any additional computational cost for inference, and yet can
significantly improve the performance of state-of-the-art MTL models as
evidenced under two standard benchmarks; NYUv2 and PASCAL-Context.

##  2 Related Work

#### Multi-task Learning

MTL (Caruana, 1997) commonly aims to learn a single model that can accurately
generate predictions for multiple desired tasks, given an input (see Figure 2
(a)). We refer to Ruder (2017); Zhang & Yang (2017); Vandenhende et al. (2021)
for comprehensive literature review. The prior works in computer vision
problems can be broadly divided into two groups. The first group focuses on
improving network architecture via more effective information sharing across
tasks (Kokkinos, 2017; Ruder et al., 2019; Vandenhende et al., 2020a; Liang et
al., 2018; Bragman et al., 2019; Strezoski et al., 2019; Xu et al., 2018;
Zhang et al., 2019; Bruggemann et al., 2021; Bilen & Vedaldi, 2016; Zhang et
al., 2018; Xu et al., 2018), by designing cross-task attention mechanisms
Misra et al. (2016), task-specific attention modules (Liu et al., 2019;
Bhattacharjee et al., 2023), cross-tasks feature interaction (Ye & Xu, 2022a;
Vandenhende et al., 2020b), gating strategies or mixture of experts modules
(Bruggemann et al., 2020; Guo et al., 2020; Chen et al., 2023; Fan et al.,
2022), visual prompting (Ye & Xu, 2022b) _etc_. The second group aims to
address the unbalanced optimization for joint minimization of multiple task-
specific loss functions, where each may exhibit varying characteristics. This
is achieved through either actively changing loss term weights (Kendall et
al., 2018; Liu et al., 2019; Guo et al., 2018; Chen et al., 2018; Lin et al.,
2019; Sener & Koltun, 2018; Liu et al., 2021b) and / or modifying the
gradients of loss functions, w.r.t. shared network weights to alleviate task
conflicts (Yu et al., 2020; Liu et al., 2021a; Chen et al., 2020; Chennupati
et al., 2019; Suteu & Guo, 2019) and / or knowledge distillation (Li & Bilen,
2020; Li et al., 2022b). Unlike these methods, our work aims to improve MTL
performance by regularizing deep networks through the introduction of 3D-aware
representations (see Figure 2 (c)).

#### Neural Rendering

Our approach also relates to the line of work that learns a 3D scene from
multiple views and then performs novel view synthesis (Lombardi et al., 2019;
Meshry et al., 2019; Sitzmann et al., 2019; Thies et al., 2019; Mildenhall et
al., 2020). Prior methods with few exceptions can represent only a single
scene per model, require many calibrated views, or are not able to perform
other tasks than novel view synthesis such as semantic segmentation, depth
estimation (see Figure 2 (b)). PixelNeRF (Yu et al., 2021) conditions a neural
radiance field (NeRF) (Mildenhall et al., 2020) on image inputs through an
encoder, allows for the modeling of multiple scenes jointly and generalizes to
unseen scenes, however, the work focuses only on synthesizing novel views. Zhi
et al. (2021) extend the standard NeRF pipeline through a parallel semantic
segmentation branch to jointly encode semantic information of the 3D scene,
and obtain 2D segmentations by rendering the scene for a given view using
raycasting. However, their model is scene-specific and does not generalize to
unseen scenes. Panoptic Neural Fields (Kundu et al., 2022) predict a radiance
field that represents the color, density, instance and category label of any
3D point in a scene through the combination of multiple encoders for both
background and each object instance. The work was designed for predicting
those tasks only on novel views of previously seen scenes, hence it cannot be
applied to new scenes without further training on them and is also limited to
handle only rigid objects (_c.f_. non-rigid, deformable). In contrast, our
method can be used to efficiently predict multiple tasks in novel scenes,
without any such restrictions on object type, can be trained from a single
view and is further not limited to a fixed architecture or specific set of
tasks. Finally, our work harnesses efficient triplane 3D representations from
(Chan et al., 2022) that is originally designed to generate high-quality,
3D-aware representations from a collection of single-view images. Our method
alternatively focuses on the joint learning of dense vision problems and
leverages 3D understanding to bring a beneficial structure to the learned
representations.

##  3 Method

We next briefly review the problem settings for MTL and neural rendering to
provide required background and then proceed to describe our proposed method.

###  3.1 Multi-Task Learning

Figure 1: Illustration of (a) vanilla multi-task learning, (b) standard neural
radiance fields (NeRFs) and (c) our 3D-aware multi-task learning method.

Our goal is to learn a model $\hat{y}$ that takes in an RGB image $\bm{I}$ as
input and jointly predicts ground-truth labels
$Y=\\{\bm{y}_{1},\ldots,\bm{y}_{T}\\}$ for $T$ tasks . In this paper, we focus
on dense prediction problems such as semantic segmentation, depth estimation
where input image and labels have the same dimensionality. While it is
possible to learn an independent model for each task, a more efficient design
involves sharing a large portion of the computation across the tasks, via a
common feature encoder $f$. Encoder $f$ then takes in an image as input and
outputs a high-dimensional feature map which has smaller width and height than
the input. In this setting, the encoder is followed by multiple task-specific
decoders $h_{t}$ that each ingests $f(\bm{I})$ to predict corresponding task
labels _i.e_., $h_{t}(f(\bm{I}))$, as depicted in Fig. 1 (a). Given a labeled
training set $\mathcal{D}$ with $N$ image-label pairs, the model weights can
be optimized as:

$\min_{f,\\{h_{t}\\}_{t=1}^{T}}\frac{1}{N}\sum_{(\bm{I},Y)\in\mathcal{D}}\sum_{\bm{y}_{t}\in{Y}}\mathcal{L}_{t}(h_{t}\circ
f(\bm{I})),\bm{y}_{t}),$ (1)

where $\mathcal{L}_{t}$ is the loss function for task $t$, _i.e_., cross
entropy loss for semantic segmentation, $L_{1}$ loss for depth estimation. We
provide more details in Sec. 4,

###  3.2 3D-Aware Multi-task Learning

An ideal feature extractor $f$ is expected to extract both task-agnostic and
task-specific information, towards enabling the following task-specific
decoders to solve their respective target tasks accurately. However, in
practice, the combination of high-dimensional feature space and highly non-
linear mappings from input to output is prone to overfitting to data and
learning of noisy correlations. To mitigate these issues, we propose a new
3D-aware regularization technique that first maps extracted features to 3D
neural codes, projects them to task-specific fields and finally renders them
to obtain predictions for each target task through differentiable rendering.
In the regularization, outputs for all tasks are conditioned on observations
that lie on a low-D manifold (the density (Mildenhall et al., 2020)),
enforcing 3D consistency between tasks.

Figure 2: Diagram of 3D-aware regularizer $g$. The regularizer $g$ takes as
input the features from the shared encoder and transforms it to a tri-plane
using a tri-plane encoder $e$. Given a 3D point $(x,y,z)$ on a given ray, we
project the coordinates onto three planes and aggregate features from three
planes using summation to obtain the 3D representations, which are then fed
into a light-weight MLP $n_{t}$ to estimate predictions of each task or the
density of the 3D point. Finally, in volume rendering $r$, we integrate the
predictions over the ray to render the predictions of each task.

#### 3D representations.

Training the state-of-the-art MTL models (_e.g_. Vandenhende et al. (2020b);
Ye & Xu (2022a)) on high resolution input images for multiple dense prediction
tasks simultaneously is computation and memory intensive. Hence, naively
mapping their multi-scale high-resolution features to 3D is not feasible due
to memory limitations in many standard GPUs. Hence, we adopt the hybrid
explicit-implicit tri-plane representations of Chan et al. (2022). In
particular, we first feed $\bm{I}$ into a shared encoder and obtain a
$W{\times}H{\times}C$-dimensional feature map where $H$ and $W$ are the height
and width. Then, through a tri-plane encoder $e$, we project the feature map
to three explicit $W{\times}H{\times}C^{\prime}$ dimensional feature maps,
$e_{xy},e_{yz},e_{xz}$, that represent axis aligned orthogonal feature planes.
We can query any 3D coordinate $(x,y,z)$ by projecting it onto each plane,
then retrieve the respective features from three planes via bi-linear
interpolation and finally aggregate features using summation to obtain the 3D
representation ($e(x,y,z)=e_{xy}(x,y){+}e_{yz}(y,z){+}e_{xz}(x,z)$) as in Chan
et al. (2022).

#### Neural task fields.

For each task, we use an additional light-weight network $n_{t}$, implemented
as a small MLP, to estimate both a density value and task-specific vector,
where this element pair can be denoted as a neural task field for the
aggregated 3D representation. We are then able to render these quantities via
neural volume rendering Max (1995); Mildenhall et al. (2020) through a
differentiable renderer $r$ to obtain predictions for each task.

In particular, for the tasks including semantic segmentation, part
segmentation, surface normal estimation, boundary detection, saliency
prediction, we estimate prediction for each point of a given ray (_e.g_.
logits for segmentation) and integrate them over the ray. We normalize the
predictions after rendering for surface normal and apply softmax after
rendering for segmentation tasks. For depth estimation task, we use the raw
prediction as depth maps.

In summary the sequence of mappings can be summarized as; firstly mapping the
shared feature encoding $f(\bm{I})$ to tri-plane features through $e$, further
mapping it to neural task fields through $n_{t}$, finally rendering these to
obtain predictions for task $t$, _i.e_. $g_{t}\circ f(\bm{I})$ where
$g_{t}=r\circ n_{t}\circ e$ is the regularizer for task $t$.

Discussion. While novel view synthesis methods such as NeRF require the
presence of multiple views and knowledge of the camera matrices, here we
assume a single view to extract the corresponding 3D representations and to
render them as task predictions. For rendering, we assume that the camera is
orthogonal to image center here, and depict $r$ as a function that takes only
the output of $n_{t}$ but not the viewpoint as input. In the experiments, we
show that our model consistently improves the MTL performance, even when
learned from a single view per scene, thanks to the 3D structure of
representations imposed by our regularizer.

#### Optimization.

We measure the mismatch between ground-truth labels and the predictions
obtained from our 3D-aware model branch and use this signal to jointly
optimize the model along with the original common task losses found in Eq. 1:

$\min_{f,\\{h_{t},g_{t}\\}_{t=1}^{T}}\frac{1}{N}\sum_{(\bm{I},Y)\in\mathcal{D}}\sum_{\bm{y}_{t}\in{Y}}\mathcal{L}_{t}(h_{t}\circ
f(\bm{I}),\bm{y}_{t})+\alpha_{t}\underbrace{\mathcal{L}_{t}(g_{t}\circ
f(\bm{I})),\bm{y}_{t})}_{\text{3D-aware regularizer}},$ (2)

where $\alpha_{t}$ is a hyperparameter balancing loss terms.

#### Cross-view consistency.

Though our 3D-aware regularizer does not require multiple views of the same
scene to be presented, it can be easily extended to penalize the cross-view
inconsistency on the predictions when multiple views of the same scene are
available, _e.g_. video frames. Let $\bm{I}$ and $\bm{I}^{\prime}$ be two
views of a scene with their camera viewpoints $V$ and $V^{\prime}$,
respectively. In addition to the regularization term in Eq. 2, here we also
compute predictions for $I^{\prime}$ but by using $\bm{I}$ as the input and
render it by using the relative camera transformation $\Delta V$ from $V$ to
$V^{\prime}$. We then penalize the inconsistency between this prediction and
ground-truth labels of $\bm{I}^{\prime}$:

$\min_{{f,\\{h_{t},g_{t}\\}_{t=1}^{T}}}\frac{1}{N}\sum_{\begin{subarray}{c}\\{(\bm{I},Y),\\\
(\bm{I}^{\prime},Y^{\prime})\\}\in\mathcal{D}\end{subarray}}\sum_{\begin{subarray}{c}\bm{y}_{t}\in{Y},\\\
\bm{y}^{\prime}_{t}\in{Y^{\prime}}\end{subarray}}\mathcal{L}_{t}(h_{t}\circ
f(\bm{I}),\bm{y}_{t})+\alpha_{t}\underbrace{\mathcal{L}_{t}(g_{t}\circ
f(\bm{I})),\bm{y}_{t})}_{\text{3D-aware
regularizer}}+\alpha^{\prime}_{t}\underbrace{\mathcal{L}_{t}(g_{t}^{\Delta
V}\circ f(\bm{I})),\bm{y}^{\prime}_{t})}_{\text{cross-view regularizer}},$ (3)

where $\alpha_{t}$ and $\alpha^{\prime}_{t}$ are hyperparameters balancing
loss terms and we set $\alpha_{t}=\alpha^{\prime}_{t}$. Note that in this case
$g_{t}$ is a function of $\Delta V$, as the relative viewpoint $\Delta V$ is
used by the renderer $r$.

##  4 Experiments

Here we first describe the benchmarks used and our implementation details,
then present a quantitative and qualitative analysis of our method.

###  4.1 Dataset

#### NYUv2 (Silberman et al., 2012):

It contains 1449 RGB-D images, sampled from video sequences from a variety of
indoor scenes, which we use to perform four tasks; namely 40-class semantic
segmentation, depth estimation, surface normal estimation and boundary
detection in common with prior work (Ye & Xu, 2022a; Bruggemann et al., 2021).
Following the previous studies, we use the true depth data recorded by the
Microsoft Kinect and surface normals provided in the prior work (Eigen &
Fergus, 2015) for depth estimation and surface normal estimation tasks.

#### NYUv2 video frames:

In addition to the standard data split, NYUv2 (Silberman et al., 2012) also
provides additional video
frames111https://www.kaggle.com/datasets/soumikrakshit/nyu-depth-v2 which are
labeled only for depth estimation. Only for the cross-view consistent
regularization experiments, we merge the original split with video frames, and
train multi-task learning models by minimizing loss on available labeled
tasks, _i.e_. all four tasks on the original data and only the depth on video
frames. To estimate the relative camera pose $\Delta V$ between the frames, we
use colmap (Schönberger & Frahm, 2016; Schönberger et al., 2016).

#### PASCAL-Context (Chen et al., 2014):

PASCAL (Everingham et al., 2010) is a commonly used image benchmark for dense
prediction tasks. We use the data splits from PASCAL-Context (Chen et al.,
2014) which has annotations for semantic segmentation, human part segmentation
and semantic edge detection. Additionaly, following (Vandenhende et al., 2021;
Ye & Xu, 2022a), we also consider surface normal prediction and saliency
detection using the annotations provided by Vandenhende et al. (2021).

###  4.2 Implementation Details

Our regularizer is architecture agnostic and can be applied to different
architectures. In our experiments, it is incorporated into two state-of-the-
art (SotA) MTL methods; MTI-Net (Vandenhende et al., 2020b) and InvPT (Ye &
Xu, 2022a) which builds on the convolutional neural network (CNN), HRNet-48
(Wang et al., 2020) and transformer based ViT-L (Dosovitskiy et al., 2020)
respectively. In all experiments, we follow identical training, evaluation
protocols (Ye & Xu, 2022a). We append our 3D-aware regularizer to these two
models using two convolutional layers, followed by BatchNorm and ReLU, to
project feature maps to the tri-plane space resulting in a common size and
channel width ($64$). A 2-layer MLP is used to render each task as in Chan et
al. (2022). We use identical hyper-parameters; learning rate, batch size, loss
weights, loss functions, pre-trained weights, optimizer, evaluation metrics as
MTI-Net and InvPT, respectively. We jointly optimize task-specific losses and
losses arising from our 3D regularization. During inference, the regularizer
is discarded. We refer to the supplementary material for further details.

###  4.3 Results

#### Comparison with SotA methods.

We compare our method with the SotA MTL methods on NYUv2 and PASCAL-Context
datasets and report results in Tab. 1 and Tab. 2, respectively. Following
Bruggemann et al. (2021), we use HRNet-48 (Wang et al., 2020) as backbone when
comparing to CNN based methods; Cross-Stitch (Misra et al., 2016), PAP (Zhang
et al., 2019), PSD (Zhou et al., 2020), PAD-Net (Xu et al., 2018), ATRC
(Bruggemann et al., 2021), MTI-Net (Vandenhende et al., 2020b). We use ViT-L
(Dosovitskiy et al., 2020) as backbone when comparing to InvPT (Ye & Xu,
2022a).

In NYUv2 (see Table 1), when using HRNet-48 as backbone, we observe that ATRC
(Bruggemann et al., 2021) and MTI-Net (Vandenhende et al., 2020b) obtain the
best performance. By incorporating our method to MTI-Net (Vandenhende et al.,
2020b), we improve its performance on all tasks and outperform all CNN based
MTL methods. In comparison, the InvPT approach (Ye & Xu, 2022a) achieves
superior MTL performance by leveraging both the ViT-L (Dosovitskiy et al.,
2020) backbone and multi-scale cross-task interaction modules. Our method is
also able to quantitatively improve upon the base InvPT by integrating our
proposed 3D-aware regularizer, _e.g_. $+1.31$ mIoU on Seg. The results
evidence that both geometric information is beneficial for jointly learning
multiple dense prediction tasks.

Method | Seg. (mIoU) $\uparrow$ | Depth (RMSE) $\downarrow$ | Normal (mErr) $\downarrow$ | Boundary (odsF) $\uparrow$
---|---|---|---|---
Cross-Stitch (Misra et al., 2016) | 36.34 | 0.6290 | 20.88 | 76.38
PAP (Zhang et al., 2019) | 36.72 | 0.6178 | 20.82 | 76.42
PSD (Zhou et al., 2020) | 36.69 | 0.6246 | 20.87 | 76.42
PAD-Net (Xu et al., 2018) | 36.61 | 0.6270 | 20.85 | 76.38
ATRC (Bruggemann et al., 2021) | 46.33 | 0.5363 | 20.18 | 77.94
MTI-Net (Vandenhende et al., 2020b) | 45.97 | 0.5365 | 20.27 | 77.86
Ours | 46.67 | 0.5210 | 19.93 | 78.10
InvPT (Ye & Xu, 2022a) | 53.56 | 0.5183 | 19.04 | 78.10
Ours | 54.87 | 0.5006 | 18.55 | 78.30

Table 1: Quantitative comparison of our method to the SotA methods; NYUv2
dataset.

Table 2 depicts experimental results on the PASCAL-Context dataset where
previous method results are reproduced from Ye & Xu (2022a). We also report
results from our local implementation of the MTI-Net, denoted by ‘MTI-Net*’,
where we found that our implementation obtains better performance. We observe
that the performance of existing methods is better than in the previous NYUv2
experiment (Tab. 1), as PASCAL-Context has significantly more images available
for training. From Tab. 2 we observe that our method, incorporating our
proposed regularizer to MTI-Net (Vandenhende et al., 2020b), can improve the
performance on all tasks with respect to our base MTI-Net implementation,
_e.g_. +2.29 mIoU on Seg, and obtains the best performance on most tasks
compared with MTL methods that use the HRNet-48 backbone. As in NYUv2, the
InvPT model (Ye & Xu, 2022a) achieves better performance on a majority of
tasks over existing methods. Our method with InvPT again obtains improvements
on all tasks over InvPT, _e.g_. $+1.51$ mIoU on PartSeg and $+1.00$ odsF on
Boundary. This result further suggests that our method is effective for
enabling the MTL network to learn beneficial geometric cues and that the
technique can be incorporated with various MTL methods for comprehensive task
performance improvements.

Method | Seg. (mIoU) $\uparrow$ | PartSeg (mIoU) $\uparrow$ | Sal (maxF) $\uparrow$ | Normal (mErr) $\downarrow$ | Boundary (odsF) $\uparrow$
---|---|---|---|---|---
ASTMT (Maninis et al., 2019) | 68.00 | 61.10 | 65.70 | 14.70 | 72.40
PAD-Net (Xu et al., 2018) | 53.60 | 59.60 | 65.80 | 15.30 | 72.50
MTI-Net (Vandenhende et al., 2020b) | 61.70 | 60.18 | 84.78 | 14.23 | 70.80
ATRC (Bruggemann et al., 2021) | 62.69 | 59.42 | 84.70 | 14.20 | 70.96
MTI-Net* (Vandenhende et al., 2020b) | 64.42 | 64.97 | 84.56 | 13.82 | 74.30
Ours | 66.71 | 65.20 | 84.59 | 13.71 | 74.50
InvPT (Ye & Xu, 2022a) | 79.03 | 67.61 | 84.81 | 14.15 | 73.00
Ours | 79.53 | 69.12 | 84.94 | 13.53 | 74.00

Table 2: Quantitative comparison of our method to the SotA methods; PASCAL-
Context dataset.

#### 3D regularizer with multiple views.

Here we investigate whether learning stronger 3D consistency across multiple
views with our regularizer further improves the performance in multiple tasks.
To this end, we merge the NYUv2 dataset with the additional video frames
possessing only depth annotation and train the base InvPT, our method and our
method with cross-view consistency on the merged data. For InvPT, we train the
model by minimizing losses over the labeled tasks. We train our method by
minimizing both the supervised losses and the 3D-aware regularization loss. We
further include the cross-view consistency loss. To regularize multi-view
consistency, we sample two views of the same scene and we feed the first view
and transform the 3D representations to the second view, rendering the depth,
which is aligned with the ground-truth of the second view.

Method | Seg. (mIoU) $\uparrow$ | Depth (RMSE) $\downarrow$ | Normal (mErr) $\downarrow$ | Boundary (odsF) $\uparrow$
---|---|---|---|---
InvPT (Ye & Xu, 2022a). | 53.44 | 0.4927 | 18.78 | 77.90
Ours | 54.93 | 0.4879 | 18.47 | 77.90
Ours with cross-view consistency | 54.99 | 0.4850 | 18.52 | 78.00

Table 3: Quantitative comparison of our method on NYUv2 dataset + extra video
frames with multiple views.

Results of the three approaches are reported in Tab. 3. Compared with the
results in Tab. 1, we can see that including video frames for training
improves the performance of InvPT on depth and surface normal tasks while
yielding comparable performance on remaining tasks. We also see that our
method obtains consistent improvement over the InvPT on four tasks with
applying 3D-aware regularization using only a single view. Adding the cross-
view consistency loss term to our method, we can observe further performance
improvement beyond using only single view samples. This suggests that better
3D geometry learning through multi-view consistency is beneficial, however,
the improvements are modest. We argue that coarse 3D scene information
obtained from single views can be sufficient to learn more structured and
regulate inter-task relations.

We also note that this experimental setting is also related to the recent MTL
work (Li et al., 2022a) that can learn from partially annotated data by
exploiting cross-task relations. However we here focus on an orthogonal
direction and believe our complementary works have scope to be integrated
together. We leave this as a promising direction for future work.

#### Comparison with auxiliary network heads.

Prior work suggests that the addition of auxiliary heads performing the same
task with identical head architectures yet with different weight
initializations can be further helpful to performance (Meyerson &
Miikkulainen, 2018). To verify whether the improvements obtained by our
regularizer is not due to the additional heads solely but introduced 3D
structure, we conduct a comparison with our baseline and report results in
Tab. 4. The results show that adding auxiliary heads (‘InvPT + Aux. Heads’)
does not necessarily lead to better performance on all tasks; _e.g_. Seg,
whereas our method can be seen to outperform this baseline on all tasks
suggesting the benefit of introducing 3D-aware structure across tasks.

Method | Seg. (mIoU) $\uparrow$ | Depth (RMSE) $\downarrow$ | Normal (mErr) $\downarrow$ | Boundary (odsF) $\uparrow$
---|---|---|---|---
InvPT (Ye & Xu, 2022a) | 53.56 | 0.5183 | 19.04 | 78.10
InvPT + Aux. Heads | 52.45 | 0.5131 | 18.90 | 77.60
Ours | 54.87 | 0.5006 | 18.55 | 78.30

Table 4: Quantitative comparison of our method to the baseline of adding
auxiliary heads to InvPT; NYUv2 dataset.

#### 3D-aware regularizer predictions.

Though we discard the regularizer during inference, the regularizer can also
be used to produce predictions for the tasks. To investigate their estimation
utility, we report task performance using the default task specific heads
$h_{t}$, the regularizer output (_regularizer_) and finally using the averaged
predictions over two in Tab. 5. We observe that the regularizer alone
estimations are worse than the task-specific heads, however, the performance
of their averaged output yields marginal improvements to the boundary
detection task. The lower performance of using the regularizer alone may be
explained by the fact that the rendering image size is typically small (_e.g_.
we render 56${\times}$72 images for NYUv2). The addition of a super-resolution
module, similar to previous work (Chan et al., 2022), can further improve the
quality of the related predictions. We leave this to future work.

outputs | Seg. (mIoU) $\uparrow$ | Depth (RMSE) $\downarrow$ | Normal (mErr) $\downarrow$ | Boundary (odsF) $\uparrow$
---|---|---|---|---
task-specific heads | 54.87 | 0.5006 | 18.55 | 78.30
regularizer | 51.79 | 0.5282 | 18.90 | 74.80
avg | 54.68 | 0.5062 | 18.70 | 78.50

Table 5: Quantitative results of the predictions from the task-specific heads,
regularizer or the average of both the task-specific heads and regularizer in
our method; NYUv2 dataset.

#### Tasks for 3D-aware regularizer.

Our regularizer renders predictions for all learning tasks by default. We
further study the effect of isolating different tasks for rendering with the
regularizer in Tab. 6. Specifically; we jointly optimize the MTL network with
a regularizer that renders only one individual task predictions. From Tab. 6
we observe that rendering different individual tasks in the regularizer leads
to only marginally differing results and yet using all tasks for rendering can
help to better learn the geometric information for multi-task learning, _i.e_.
‘All tasks’ obtains the best performance on the majority of tasks.

render tasks | Seg. (mIoU) $\uparrow$ | Depth (RMSE) $\downarrow$ | Normal (mErr) $\downarrow$ | Boundary (odsF) $\uparrow$
---|---|---|---|---
Seg. | 55.04 | 0.5038 | 18.78 | 77.90
Depth | 54.62 | 0.5041 | 18.93 | 77.50
Normal | 53.43 | 0.5117 | 18.59 | 77.50
Edge | 53.97 | 0.5022 | 18.97 | 77.50
All tasks | 54.87 | 0.5006 | 18.55 | 78.30

Table 6: Quantitative results of our method isolating different tasks for
rendering with the regularizer; NYUv2 dataset.

#### Using less data.

We further investigate the performance gain obtained by our method when
trained with fewer training samples. To this end, we train the baseline InvPT
(Ye & Xu, 2022a) and our method on 25% and 50% of the NYUv2 data after
randomly subsampling the original training set. The results are reported in
Tab. 7. As expected, more training samples result in better performance in all
cases. Our method consistently outperforms the baseline on all tasks in all
label regimes with higher margins when more data is available. As the full
NYUv2 training set is relatively small, contains only 795 images, our
regularizer learns better 3D consistency across tasks from more data too,
hence resulting enhanced task performance.

# images | Method | Seg. (mIoU) $\uparrow$ | Depth (RMSE) $\downarrow$ | Normal (mErr) $\downarrow$ | Boundary (odsF) $\uparrow$
---|---|---|---|---|---
795 (100%) | InvPT (Ye & Xu, 2022a) | 53.56 | 0.5183 | 19.04 | 78.10
Ours | 54.87 | 0.5006 | 18.55 | 78.30
397 (50%) | InvPT (Ye & Xu, 2022a) | 49.24 | 0.5741 | 20.60 | 74.90
Ours | 49.30 | 0.5656 | 20.30 | 76.50
198 (25%) | InvPT (Ye & Xu, 2022a) | 43.83 | 0.6060 | 21.76 | 74.80
Ours | 44.79 | 0.5972 | 21.57 | 74.80

Table 7: Quantitative comparison of the baseline InvPT and our method in the
NYUv2 dataset for varying training set sizes.

###  4.4 Qualitative Results

We visualize the task predictions for both our method and the base InvPT
method on an NYUv2 sample in Fig. 3. Our method can be observed to estimate
better predictions consistently for four tasks. For example, our method
estimates more accurate predictions around the boundary of the refrigerator,
stove and less noisy predictions within objects like curtain and stove. The
geometric information learned in our method helps distinguish different
adjacent objects, avoids noisy predictions within object boundaries and also
improves the consistency across tasks as in the regularizer, all tasks
predictions are rendered based on the same density.

Figure 3: Qualitative results on NYUv2. Each column shows the image or
predictions and performance for each task. The last row shows the ground-truth
of four tasks. The first to the third row shows the predictions of InvPT, the
regularizer in our method and task-specific decoders of our method,
respectively.

We then visualize the predictions of our method’s regularizer and the task-
specific decoder NYUv2 in Fig. 3. As shown in the figure, our regularizer can
also render high quality predictions for different tasks yet it was observed
to obtain worse quantitative performance than the task-specific decoders. As
discussed, this is due to the rendering image size being usually small (_e.g_.
we render 56${\times}$72 images for NYUv2).

##  5 Conclusion and Limitations

We demonstrate that encouraging 3D-aware interfaces between different related
tasks including depth estimation, semantic segmentation and surface normal
estimation consistently improves the multi-task performance when incorporated
to the recent MTL techniques in two standard dense prediction benchmarks. Our
model can be successfully used with different backbone architectures and does
not bring any additional inference costs. Our method has limitations too.
Despite the efficient 3D modeling through the triplane encodings, representing
3D representations for higher resolution 3D volumes is still expensive in
terms of memory or computational cost. Though our proposed method obtains
performance gains consistently over multiple tasks, we balance loss functions
with fixed cross-validated hyperparameters, while it would be more beneficial
to use adaptive loss balancing strategies (Kendall et al., 2018) or discarding
conflicting gradients (Liu et al., 2021a). Finally, in the cross-view
consistency experiments where only some of the images are labeled for all the
tasks, our method does not make use of semi-supervised learning or view-
consistency for the tasks with missing labels which can be further improve the
performance of our model.

#### Acknowledgement.

We thank Octave Mariotti, Changjian Li, and Titas Anciukevicius for their
valuable feedback.

## References

  * Anciukevičius et al. (2023) Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero.  Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation.  In _CVPR_ , pp. 12608–12618, 2023.
  * Bhattacharjee et al. (2023) Deblina Bhattacharjee, Sabine Süsstrunk, and Mathieu Salzmann.  Vision transformer adapters for generalizable multitask learning.  _arXiv preprint arXiv:2308.12372_ , 2023.
  * Bilen & Vedaldi (2016) Hakan Bilen and Andrea Vedaldi.  Integrated perception with recurrent multi-task neural networks.  In _Advances in Neural Information Processing Systems_ , pp. 235–243, 2016.
  * Bragman et al. (2019) Felix JS Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel C Alexander, and Jorge Cardoso.  Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels.  In _ICCV_ , pp. 1385–1394, 2019.
  * Bruggemann et al. (2020) David Bruggemann, Menelaos Kanakis, Stamatios Georgoulis, and Luc Van Gool.  Automated search for resource-efficient branched multi-task networks.  _arXiv preprint arXiv:2008.10292_ , 2020.
  * Bruggemann et al. (2021) David Bruggemann, Menelaos Kanakis, Anton Obukhov, Stamatios Georgoulis, and Luc Van Gool.  Exploring relational context for multi-task dense prediction.  In _ICCV_ , 2021.
  * Caruana (1997) Rich Caruana.  Multitask learning.  _Machine learning_ , 28(1):41–75, 1997.
  * Chan et al. (2022) Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _CVPR_ , 2022.
  * Chan et al. (2023) Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein.  Generative novel view synthesis with 3d-aware diffusion models.  In _CVPR_ , 2023.
  * Chen et al. (2014) Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  Detect what you can: Detecting and representing objects using holistic models and body parts.  In _CVPR_ , pp. 1971–1978, 2014.
  * Chen et al. (2018) Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich.  Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.  In _ICML_ , pp. 794–803. PMLR, 2018.
  * Chen et al. (2020) Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov.  Just pick a sign: Optimizing deep multitask models with gradient sign dropout.  _NeurIPS_ , 2020.
  * Chen et al. (2023) Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik G Learned-Miller, and Chuang Gan.  Mod-squad: Designing mixtures of experts as modular multi-task learners.  In _CVPR_ , pp. 11828–11837, 2023.
  * Chennupati et al. (2019) Sumanth Chennupati, Ganesh Sistu, Senthil Yogamani, and Samir A Rawashdeh.  Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning.  In _CVPR Workshop_ , 2019.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Eigen & Fergus (2015) David Eigen and Rob Fergus.  Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.  In _IEEE International Conference on Computer Vision_ , pp. 2650–2658, 2015.
  * Everingham et al. (2010) Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  _IJCV_ , 88(2):303–338, 2010.
  * Fan et al. (2022) Zhiwen Fan, Rishov Sarkar, Ziyu Jiang, Tianlong Chen, Kai Zou, Yu Cheng, Cong Hao, Zhangyang Wang, et al.  M3vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design.  _Neurips_ , 35:28441–28457, 2022.
  * Guo et al. (2018) Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei.  Dynamic task prioritization for multitask learning.  In _ECCV_ , pp. 270–287, 2018.
  * Guo et al. (2020) Pengsheng Guo, Chen-Yu Lee, and Daniel Ulbricht.  Learning to branch for multi-task learning.  In _ICML_ , pp. 3854–3863. PMLR, 2020.
  * Kendall et al. (2018) Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In _CVPR_ , pp. 7482–7491, 2018.
  * Kokkinos (2017) Iasonas Kokkinos.  Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory.  In _CVPR_ , pp. 6129–6138, 2017.
  * Kundu et al. (2022) Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser.  Panoptic neural fields: A semantic object-aware neural scene representation.  In _CVPR_ , pp. 12871–12881, 2022.
  * Li & Bilen (2020) Wei-Hong Li and Hakan Bilen.  Knowledge distillation for multi-task learning.  In _ECCV Workshop on Imbalance Problems in Computer Vision_ , pp. 163–176. Springer, 2020.
  * Li et al. (2022a) Wei-Hong Li, Xialei Liu, and Hakan Bilen.  Learning multiple dense prediction tasks from partially annotated data.  In _CVPR_ , 2022a.
  * Li et al. (2022b) Wei-Hong Li, Xialei Liu, and Hakan Bilen.  Universal representations: A unified look at multiple task and domain learning.  _arXiv preprint arXiv:2204.02744_ , 2022b.
  * Liang et al. (2018) Jason Liang, Elliot Meyerson, and Risto Miikkulainen.  Evolutionary architecture search for deep multitask networks.  In _Proceedings of the Genetic and Evolutionary Computation Conference_ , pp. 466–473, 2018.
  * Lin et al. (2019) Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong.  Pareto multi-task learning.  _NeurIPS_ , 32:12060–12070, 2019.
  * Liu et al. (2021a) Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu.  Conflict-averse gradient descent for multi-task learning.  _NeurIPS_ , 2021a.
  * Liu et al. (2021b) Liyang Liu, Yi Li, Zhanghui Kuang, Jing-Hao Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang.  Towards impartial multi-task learning.  In _ICLR_ , 2021b.
  * Liu et al. (2019) Shikun Liu, Edward Johns, and Andrew J Davison.  End-to-end multi-task learning with attention.  In _CVPR_ , pp. 1871–1880, 2019.
  * Lombardi et al. (2019) Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  _ACM Trans. Graph._ , 2019.
  * Maninis et al. (2019) Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos.  Attentive single-tasking of multiple tasks.  In _CVPR_ , pp. 1851–1860, 2019.
  * Max (1995) Nelson Max.  Optical models for direct volume rendering.  _IEEE Transactions on Visualization and Computer Graphics_ , 1(2):99–108, 1995.
  * Meshry et al. (2019) Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla.  Neural rerendering in the wild.  In _CVPR_ , pp. 6878–6887, 2019.
  * Meyerson & Miikkulainen (2018) Elliot Meyerson and Risto Miikkulainen.  Pseudo-task augmentation: From deep multitask learning to intratask sharing—and back.  In _ICML_ , pp. 3511–3520. PMLR, 2018.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Misra et al. (2016) Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert.  Cross-stitch networks for multi-task learning.  In _CVPR_ , pp. 3994–4003, 2016.
  * Niemeyer et al. (2020) Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In _CVPR_ , pp. 3504–3515, 2020.
  * Ruder (2017) Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  _arXiv preprint arXiv:1706.05098_ , 2017.
  * Ruder et al. (2019) Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard.  Latent multi-task architecture learning.  In _AAAI_ , volume 33, pp. 4822–4829, 2019.
  * Schönberger & Frahm (2016) Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Schönberger et al. (2016) Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In _European Conference on Computer Vision (ECCV)_ , 2016.
  * Sener & Koltun (2018) Ozan Sener and Vladlen Koltun.  Multi-task learning as multi-objective optimization.  _NeurIPS_ , 2018.
  * Silberman et al. (2012) Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _European conference on computer vision_ , pp. 746–760. Springer, 2012.
  * Sitzmann et al. (2019) Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer.  Deepvoxels: Learning persistent 3d feature embeddings.  In _CVPR_ , pp. 2437–2446, 2019.
  * Skorokhodov et al. (2022) Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, and Peter Wonka.  Epigraf: Rethinking training of 3d gans.  _Neurips_ , 35:24487–24501, 2022.
  * Strezoski et al. (2019) Gjorgji Strezoski, Nanne van Noord, and Marcel Worring.  Many task learning with task routing.  In _ICCV_ , pp. 1375–1384, 2019.
  * Suteu & Guo (2019) Mihai Suteu and Yike Guo.  Regularizing deep multi-task networks using orthogonal gradients.  _arXiv preprint arXiv:1912.06844_ , 2019.
  * Thies et al. (2019) Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures.  _TOG_ , 38(4):1–12, 2019.
  * Vandenhende et al. (2020a) Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool.  Branched multi-task networks: deciding what layers to share.  In _bmvc_ , 2020a.
  * Vandenhende et al. (2020b) Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool.  Mti-net: Multi-scale task interaction networks for multi-task learning.  In _ECCV_ , pp. 527–543. Springer, 2020b.
  * Vandenhende et al. (2021) Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool.  Multi-task learning for dense prediction tasks: A survey.  _PAMI_ , 2021.
  * Wang et al. (2020) Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al.  Deep high-resolution representation learning for visual recognition.  _TPAMI_ , 43(10):3349–3364, 2020.
  * Xu et al. (2018) Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe.  Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing.  In _CVPR_ , pp. 675–684, 2018.
  * Ye & Xu (2022a) Hanrong Ye and Dan Xu.  Inverted pyramid multi-task transformer for dense scene understanding.  In _ECCV_ , pp. 514–530. Springer, 2022a.
  * Ye & Xu (2022b) Hanrong Ye and Dan Xu.  Taskprompter: Spatial-channel multi-task prompting for dense scene understanding.  In _ICLR_ , 2022b.
  * Yu et al. (2021) Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In _CVPR_ , pp. 4578–4587, 2021.
  * Yu et al. (2020) Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.  Gradient surgery for multi-task learning.  _NeurIPS_ , 2020.
  * Zhang & Yang (2017) Yu Zhang and Qiang Yang.  A survey on multi-task learning.  _arXiv preprint arXiv:1707.08114_ , 2017.
  * Zhang et al. (2018) Zhenyu Zhang, Zhen Cui, Chunyan Xu, Zequn Jie, Xiang Li, and Jian Yang.  Joint task-recursive learning for semantic segmentation and depth estimation.  In _ECCV_ , pp. 235–251, 2018.
  * Zhang et al. (2019) Zhenyu Zhang, Zhen Cui, Chunyan Xu, Yan Yan, Nicu Sebe, and Jian Yang.  Pattern-affinitive propagation across depth, surface normal and semantic segmentation.  In _CVPR_ , pp. 4106–4115, 2019.
  * Zhi et al. (2021) Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison.  In-place scene labelling and understanding with implicit scene representation.  In _ICCV_ , pp. 15838–15847, 2021.
  * Zhou et al. (2020) Ling Zhou, Zhen Cui, Chunyan Xu, Zhenyu Zhang, Chaoqun Wang, Tong Zhang, and Jian Yang.  Pattern-structure diffusion for multi-task learning.  In _CVPR_ , pp. 4514–4523, 2020.

##  Appendix A Implementation Details

We implement our approach in conjunction with state-of-the-art multi-task
learning methods; MTI-Net (Vandenhende et al., 2020b) and InvPT (Ye & Xu,
2022a) while following identical training, evaluation protocols (Ye & Xu,
2022a). We use HRNet-48 (Wang et al., 2020) and ViT-L (Dosovitskiy et al.,
2020) to serve as shared encoders and append our 3D-aware regularizer to MTI-
Net and InvPT using two convolutional layers, followed by BatchNorm, ReLU, and
dropout layer with a dropout rate of 0.15 to transform feature maps to the
tri-plane dimensionality, resulting in a common size and channel width ($64$).
A 2-layer MLP with 64 hidden units as in Chan et al. (2022) and a LeakyReLU
non-linearity with the negative slope of -0.2 as in Skorokhodov et al. (2022),
is used to render each task as in Chan et al. (2022). We use identical hyper-
parameters; learning rate, batch size, loss weights, loss functions, pre-
trained weights, optimizer, evaluation metrics as MTI-Net and InvPT,
respectively. We jointly optimize task-specific losses and losses arising from
our 3D regularization. During inference, the regularizer is discarded. We use
the same task-specific loss weights as in Ye & Xu (2022a). We train all models
for 40K iterations with a batch size of 6 for experiments of using InvPT as in
Ye & Xu (2022a) and a batch size of 8 for experiments of using MTI-Net as in
(Vandenhende et al., 2020b). We ramp up the $\alpha_{t}$ from 0 to 4 linearly
in 20K iterations and keep $\alpha_{t}=4$ for the rest 20K iterations. In the
regularizer, we render 56$\times$72 predictions for NYUv2 images (Silberman et
al., 2012) and 64$\times$64 predictions for PASCAL-Context images (Chen et
al., 2014).