$\displaystyle=\frac{\lambda(r_{m}-r_{m}^{\prime})}{(\theta+\lambda)(\theta+2\lambda)}\left[1-e^{-(\theta+2\lambda)t}\right]+\frac{1}{\theta+\lambda}\left(1-e^{-\lambda
t}\right)\left[r_{m}-e^{-(\theta+\lambda)t}r_{m}^{\prime}\right]+\frac{r_{f}}{\theta}e^{-\theta
t},$

noting for example that the probability of the first shock not hitting until
time $s<T$ is $1-G(s)=e^{-\lambda s}$ and thus the expected return of $r_{m}$
at time $s<T$ is $e^{-\theta s}r_{m}[1-G(s)]=e^{-(\theta+\lambda)s}r_{m}$.

Under a sufficiently small discount rate $\theta\simeq 0$, we derive the FOC
with respect to $t$ as

$\displaystyle\frac{d\mathrm{E}[v(t)]}{dt}=(r_{m}-r_{m}^{\prime})\left(e^{-2\lambda
t}+e^{-\lambda t}\right)-r_{f}=0,$

where the SOC trivially holds. Solving this equation gives the optimal timing
of leaving the host country:

$\displaystyle t=\begin{cases}0&\text{if}\ \ r_{f}\geq
2(r_{m}-r_{m}^{\prime})\\\
-\dfrac{1}{\lambda}\log\left(-1+\dfrac{1}{2}\sqrt{1+\dfrac{r_{f}}{r_{m}-r_{m}^{\prime}}}\right)\equiv\widehat{t}&\text{if}\
\ r_{f}<2(r_{m}-r_{m}^{\prime})\end{cases},$

where the term inside the logarithm is smaller than unity if
$r_{f}<2(r_{m}-r_{m}^{\prime})$. Especially if the flow return of foreign
domestic firms is large enough such that $r_{f}\geq 2(r_{m}-r_{m}^{\prime})$,
multinationals leave the host country at time $s=0$. This is in a sharp
contrast to the case without uncertainty. Even though the damage of shock is
so small that multinationals make a higher flow return than foreign domestic
firms after the shock, uncertainty itself may lead foreign capital to move out
of the host country before the actual shock hits, because (we assume) it
expects huge subsequent shocks to come. If the flow return of foreign domestic
firms is small enough such that $r_{f}<2(r_{m}-r_{m}^{\prime})$,
multinationals stay for a certain period. The duration becomes shorter when
they perceive a higher frequency of disaster occurrence:
$d\widehat{t}/d\lambda<0$.

The findings are summarized as follows.

Proposition A3 (Disaster risk and the timing of leaving). Consider the
situation where the host economy is initially at equilibrium $S_{1}$ and is
then hit by a natural disaster at time $T>0$. The disaster raises the fixed
labor input from $F$ to $F^{\prime}(<F_{b})$, where
$r_{m}>r_{m}^{\prime}>r_{f}$ holds. On the optimal timing of multinational
leaving the host country, the following holds:

  * (i)

If multinationals know the exact timing $T$, they stay in the host country
after the disaster and never leave there.

  * (ii)

If multinationals do not know the exact timing $T$ and only know it follows
the cumulative distribution $G(T)=1-e^{-\lambda T}$ and expect huge subsequent
disasters to come, they leave the host country at time $s=0$ if $r_{f}\geq
2(r_{m}-r_{m}^{\prime})$ and do so at time $s=\widehat{t}>0$ otherwise.

## Data Appendix

We here provide supplementary tables for empirical examples in Introduction.

Table A1. Description of variables  Variable Description Source No. of natural
disasters Number of natural disasters that record positive financial damages.
EM-DAT FDI inflows (billion USD) FDI net inflows deflated by the price level
of real GDP (“pl$\\_$gdpo” in PWT, equal to the PPP divided by the nominal
exchange rate). WDI log (L. GDP) Log of real GDP, lagged one year. Real GDP is
output-side real GDP at chained PPPs in 2017USD (“gdppo”). PWT L. Tariff
($\%$) Average effective tariff rate of manufacturing products weighted by the
product import shares corresponding to each partner country, lagged one year.
Missing values are linearly interpolated. WDI L. Unit labor cost Nominal labor
income divided by real GDP, relative to US value, lagged one year. It is given
by the product of the labor share (“labsh”) and the price level
(“pl$\\_$gdpo”). PWT

Notes: EM-DAT: Emergency Events Database by the Centre for Research on the
Epidemiology of Disasters, https://www.emdat.be/ WDI: World Development
Indicators by the World Bank, https://datatopics.worldbank.org/world-
development-indicators/ PWT: Penn World Table 10.0 (Feenstra et al.,, 2015),
https://www.rug.nl/ggdc/productivity/pwt/?lang=en

Table A2. List of countries

Notes: Developing countries are those belonging to low-income, lower-middle-
income, or upper-middle-income groups at some point in 1991 to 2015, according
to the World Bank. See: https://datatopics.worldbank.org/world-development-
indicators/the-world-by-income-and-region.html Developed countries are those
belonging to high-income group at some point in 1991 to 2015. The list does
not include countries identified as tax havens (“big seven” and “dots”) by
Hines and Rice, (1994).

Table A3. Summary statistics

Table A4. Replication of Table 1 without developing-country dummies

Notes: Robust standard errors clustered at region$\times$year ($5\times 25$)
are in parentheses. The sample period is from 1991 to 2015. FDI inflows
(billion USD) are deflated by the price level of real GDP. A severe disaster
in country $i$ in year $t$ is defined as one that records financial damages as
a share of GDP exceeding its median for all countries that have ever
experienced natural disasters in 1981–2015. A non-severe disaster in country
$i$ in year $t$ is defined as one that is not severe. The three control
variables, the log of real GDP, the average tariff rate, and the unit labor
cost, are lagged one year. We exclude tax haven countries identified by Hines
and Rice, (1994).
∗Significant at 10$\%$ level; ∗∗Significant at 5$\%$ level; ∗∗∗Significant at
1$\%$ level.

Table A5. Replication of Table 2 without developing-country dummies

Notes: Robust standard errors clustered at region$\times$year ($5\times 25$)
are in parentheses. The sample period is from 1991 to 2015. FDI inflows
(billion USD) are deflated by the price level of real GDP. A severe disaster
in country $i$ in year $t$ is defined as one that records financial damages as
a share of GDP exceeding its median for all countries that have ever
experienced natural disasters in 1981–2015. A non-severe disaster in country
$i$ in year $t$ is defined as one that is not severe. The three control
variables, the log of real GDP, the average tariff rate, and the unit labor
cost, are lagged one year. We exclude tax haven countries identified by Hines
and Rice, (1994).
∗Significant at 10$\%$ level; ∗∗Significant at 5$\%$ level; ∗∗∗Significant at
1$\%$ level.

## References

  * Abe and Ye, (2013) Abe, M. and Ye, L. (2013).  Building resilient supply chains against natural disasters: The cases of Japan and Thailand.  Global Business Review, 14(4):567–586.
  * Adachi and Saito, (2020) Adachi, D. and Saito, Y. U. (2020).  Multinational production and labor share.  Research Institute of Economy, Trade and Industry (RIETI) 20-E-012.
  * ADB, (2013) ADB (2013).  The Rise of Natural Disasters in Asia and the Pacific: Learning from ADB’s Experience.  Asian Development Bank, Mandaluyong City.
  * Aitken and Harrison, (1999) Aitken, B. J. and Harrison, A. E. (1999).  Do domestic firms benefit from direct foreign investment? Evidence from Venezuela.  American Economic Review, 89(3):605–618.
  * Aizenman, (2003) Aizenman, J. (2003).  Volatility, employment and the patterns of FDI in emerging markets.  Journal of Development Economics, 72(2):585–601.
  * Aizenman and Marion, (2004) Aizenman, J. and Marion, N. (2004).  The merits of horizontal versus vertical FDI in the presence of uncertainty.  Journal of International Economics, 62(1):125–148.
  * Akamatsu et al., (2021) Akamatsu, T., Mori, T., Osawa, M., and Takayama, Y. (2021).  Multimodal agglomeration in economic geography.  arXiv preprint arXiv:1912.05113.
  * Akao and Sakamoto, (2018) Akao, K.-I. and Sakamoto, H. (2018).  A theory of disasters and long-run growth.  Journal of Economic Dynamics and Control, 95:89–109.
  * Alfaro, (2015) Alfaro, L. (2015).  Foreign direct investment: Effects, complementarities, and promotion.  In Auguste, S., Cuevas, M., and Manzano, O., editors, Partners or Creditors? Attracting Foreign Investment and Productive Development to Central America and Dominican Republic, chapter 2, pages 21–76. Inter-American Development Bank, Washington, D.C.
  * Alfaro et al., (2010) Alfaro, L., Chanda, A., Kalemli-Ozcan, S., and Sayek, S. (2010).  Does foreign direct investment promote growth? Exploring the role of financial markets on linkages.  Journal of Development Economics, 91(2):242–256.
  * Alfaro-Ureña et al., (2022) Alfaro-Ureña, A., Manelici, I., and Vasquez, J. P. (2022).  The effects of joining multinational supply chains: New evidence from firm-to-firm linkages.  Quarterly Journal of Economics.  forthcoming.
  * Ando and Kimura, (2012) Ando, M. and Kimura, F. (2012).  How did the Japanese exports respond to two crises in the international production networks? The global financial crisis and the Great East Japan Earthquake.  Asian Economic Journal, 26(3):261–287.
  * Antràs and Yeaple, (2014) Antràs, P. and Yeaple, S. R. (2014).  Multinational firms and the structure of international trade.  In Handbook of International Economics, volume 4, pages 55–130. Elsevier, Amsterdam.
  * Anuchitworawong and Thampanishvong, (2015) Anuchitworawong, C. and Thampanishvong, K. (2015).  Determinants of foreign direct investment in Thailand: Does natural disaster matter?  International Journal of Disaster Risk Reduction, 14:312–321.
  * Arkolakis et al., (2018) Arkolakis, C., Ramondo, N., Rodríguez-Clare, A., and Yeaple, S. (2018).  Innovation and production in the global economy.  American Economic Review, 108(8):2128–2173.
  * Baldwin, (2001) Baldwin, R. E. (2001).  Core-periphery model with forward-looking expectations.  Regional Science and Urban Economics, 31(1):21–49.
  * Baldwin and Okubo, (2006) Baldwin, R. E. and Okubo, T. (2006).  Heterogeneous firms, agglomeration and economic geography: Spatial selection and sorting.  Journal of Economic Geography, 6(3):323–346.
  * Barrot and Sauvagnat, (2016) Barrot, J.-N. and Sauvagnat, J. (2016).  Input specificity and the propagation of idiosyncratic shocks in production networks.  Quarterly Journal of Economics, 131(3):1543–1592.
  * Batala et al., (2021) Batala, L. K., Yu, W., Khan, A., Regmi, K., and Wang, X. (2021).  Natural disasters’ influence on industrial growth, foreign direct investment, and export performance in the South Asian region of Belt and Road initiative.  Natural Hazards, 108(2):1853–1876.
  * Bernard and Jensen, (2007) Bernard, A. B. and Jensen, J. B. (2007).  Firm structure, multinationals, and manufacturing plant deaths.  Review of Economics and Statistics, 89(2):193–204.
  * Blonigen and Piger, (2014) Blonigen, B. A. and Piger, J. (2014).  Determinants of foreign direct investment.  Canadian Journal of Economics, 47(3):775–812.
  * Boehm et al., (2019) Boehm, C. E., Flaaen, A., and Pandalai-Nayar, N. (2019).  Input linkages and the transmission of shocks: Firm-level evidence from the 2011 Tōhoku earthquake.  Review of Economics and Statistics, 101(1):60–75.
  * BOT, (2012) BOT (2012).  Thailand Floods 2011: Impact and Recovery from Business Survey.  The Bank of Thailand, Bangkok.
  * Boucekkine et al., (2013) Boucekkine, R., Camacho, C., and Fabbri, G. (2013).  Spatial dynamics and convergence: The spatial AK model.  Journal of Economic Theory, 148(6):2719–2736.
  * Cainelli et al., (2018) Cainelli, G., Fracasso, A., and Marzetti, G. V. (2018).  Natural disasters and firm resilience in Italian industrial districts.  In Belussi, F. and Hervas-Oliver, J.-L., editors, Agglomeration and Firm Performance, pages 223–243. Springer, Cham.
  * Caliendo and Parro, (2015) Caliendo, L. and Parro, F. (2015).  Estimates of the trade and welfare effects of NAFTA.  Review of Economic Studies, 82(1):1–44.
  * Carluccio and Fally, (2013) Carluccio, J. and Fally, T. (2013).  Foreign entry and spillovers with technological incompatibilities in the supply chain.  Journal of International Economics, 90(1):123–135.
  * Carvalho et al., (2021) Carvalho, V. M., Nirei, M., Saito, Y. U., and Tahbaz-Salehi, A. (2021).  Supply chain disruptions: Evidence from the Great East Japan Earthquake.  Quarterly Journal of Economics, 136(2):1255–1321.
  * Cavallo and Noy, (2011) Cavallo, E. and Noy, I. (2011).  Natural disasters and the economy—A survey.  International Review of Environmental and Resource Economics, 5(1):63–102.
  * Cole et al., (2019) Cole, M. A., Elliott, R., Okubo, T., and Strobl, E. (2019).  Natural disasters and spatial heterogeneity in damages: The birth, life and death of manufacturing plants.  Journal of Economic Geography, 19(2):373–408.
  * Dhyne et al., (2021) Dhyne, E., Kikkawa, A. K., Mogstad, M., and Tintelnot, F. (2021).  Trade and domestic production networks.  Review of Economic Studies, 88(2):643–668.
  * Doytch, (2020) Doytch, N. (2020).  Upgrading destruction? How do climate-related and geophysical natural disasters impact sectoral FDI.  International Journal of Climate Change Strategies and Management, 12(2):182–200.
  * Eaton and Kortum, (2002) Eaton, J. and Kortum, S. (2002).  Technology, geography, and trade.  Econometrica, 70(5):1741–1779.
  * Elliott et al., (2019) Elliott, R. J., Liu, Y., Strobl, E., and Tong, M. (2019).  Estimating the direct and indirect impact of typhoons on plant performance: Evidence from Chinese manufacturers.  Journal of Environmental Economics and Management, 98:102252.
  * Escaleras and Register, (2011) Escaleras, M. and Register, C. A. (2011).  Natural disasters and foreign direct investment.  Land Economics, 87(2):346–363.
  * Feenstra et al., (2015) Feenstra, R. C., Inklaar, R., and Timmer, M. P. (2015).  The next generation of the Penn World Table.  American Economic Review, 105(10):3150–82.
  * Felbermayr and Gröschl, (2013) Felbermayr, G. and Gröschl, J. (2013).  Natural disasters and the effect of trade on income: A new panel IV approach.  European Economic Review, 58:18–30.
  * Felbermayr and Gröschl, (2014) Felbermayr, G. and Gröschl, J. (2014).  Naturally negative: The growth effects of natural disasters.  Journal of Development Economics, 111:92–106.
  * Feliciano and Doytch, (2020) Feliciano, Z. M. and Doytch, N. (2020).  Trade liberalization and firm performance: The case of Thailand.  Contemporary Economic Policy, 38(4):607–621.
  * Fillat and Garetto, (2015) Fillat, J. L. and Garetto, S. (2015).  Risk, returns, and multinational production.  Quarterly Journal of Economics, 130(4):2027–2073.
  * Forslid and Okubo, (2015) Forslid, R. and Okubo, T. (2015).  Which firms are left in the periphery? Spatial sorting of heterogeneous firms with scale economies in transportation.  Journal of Regional Science, 55(1):51–65.
  * Forslid and Okubo, (2016) Forslid, R. and Okubo, T. (2016).  Big is beautiful when exporting.  Review of International Economics, 24(2):330–343.
  * Fujishima and Oyama, (2021) Fujishima, S. and Oyama, D. (2021).  Equilibrium dynamics in a model of growth and spatial agglomeration.  Macroeconomic Dynamics, pages 1–26.
  * Fujita and Thisse, (2006) Fujita, M. and Thisse, J.-F. (2006).  Globalization and the evolution of the supply chain: Who gains and who loses?  International Economic Review, 47(3):811–836.
  * Galbusera and Giannopoulos, (2018) Galbusera, L. and Giannopoulos, G. (2018).  On input-output economic models in disaster impact assessment.  International Journal of Disaster Risk Reduction, 30:186–198.
  * Garetto, (2013) Garetto, S. (2013).  Input sourcing and multinational production.  American Economic Journal: Macroeconomics, 5(2):118–51.
  * Gaspar, (2021) Gaspar, J. M. (2021).  New economic geography: History and debate.  The European Journal of the History of Economic Thought, 28(1):46–82.
  * Gigout and London, (2021) Gigout, T. and London, M. (2021).  Trade networks and natural disasters: Diversion, not destruction.  mimeo.
  * Görg and Strobl, (2002) Görg, H. and Strobl, E. (2002).  Multinational companies and indigenous development: An empirical analysis.  European Economic Review, 46(7):1305–1322.
  * Görg and Strobl, (2003) Görg, H. and Strobl, E. (2003).  ‘Footloose’ multinationals?  The Manchester School, 71(1):1–19.
  * Gumpert et al., (2020) Gumpert, A., Li, H., Moxnes, A., Ramondo, N., and Tintelnot, F. (2020).  The life-cycle dynamics of exporters and multinational firms.  Journal of International Economics, 126:103343.
  * Hallegatte, (2015) Hallegatte, S. (2015).  The indirect cost of natural disasters and an economic definition of macroeconomic resilience.  World Bank Policy Research Working Paper 7357.
  * Hallegatte and Dumas, (2009) Hallegatte, S. and Dumas, P. (2009).  Can natural disasters have positive consequences? Investigating the role of embodied technical change.  Ecological Economics, 68(3):777–786.
  * Haraguchi and Lall, (2015) Haraguchi, M. and Lall, U. (2015).  Flood risks and impacts: A case study of Thailand’s floods in 2011 and research questions for supply chain decision making.  International Journal of Disaster Risk Reduction, 14:256–272.
  * Hayakawa et al., (2015) Hayakawa, K., Matsuura, T., and Okubo, F. (2015).  Firm-level impacts of natural disasters on production networks: Evidence from a flood in Thailand.  Journal of the Japanese and International Economies, 38:244–259.
  * Henriet et al., (2012) Henriet, F., Hallegatte, S., and Tabourier, L. (2012).  Firm-network characteristics and economic robustness to natural disasters.  Journal of Economic Dynamics and Control, 36(1):150–167.
  * Hines and Rice, (1994) Hines, J. R. and Rice, E. M. (1994).  Fiscal paradise: Foreign tax havens and American business.  Quarterly Journal of Economics, 109(1):149–182.
  * Hsu et al., (2020) Hsu, W.-T., Lu, Y., Luo, X., and Zhu, L. (2020).  Foreign direct investment and industrial agglomeration: Evidence from China.  SMU Economics and Statistics Working Paper Series, Paper No. 11-2020.
  * Ikefuji and Horii, (2012) Ikefuji, M. and Horii, R. (2012).  Natural disasters in a two-sector model of endogenous growth.  Journal of Public Economics, 96(9-10):784–796.
  * Inoue and Todo, (2019) Inoue, H. and Todo, Y. (2019).  Firm-level propagation of shocks through supply-chain networks.  Nature Sustainability, 2(9):841–847.
  * Javorcik, (2004) Javorcik, B. S. (2004).  Does foreign direct investment increase the productivity of domestic firms? In search of spillovers through backward linkages.  American Economic Review, 94(3):605–627.
  * Kashiwagi et al., (2021) Kashiwagi, Y., Todo, Y., and Matous, P. (2021).  Propagation of economic shocks through global supply chaines—Evidence from Hurricane Sandy.  Review of International Economics, 29:1186–1220.
  * Kato and Okoshi, (2022) Kato, H. and Okoshi, H. (2022).  Economic integration and agglomeration of multinational production with transfer pricing.  International Economic Review.  forthcoming.
  * Kato and Okubo, (2017) Kato, H. and Okubo, T. (2017).  The impact of a natural disaster on foreign direct investment and vertical linkages.  Keio-IES Discussion Paper Series 2017-018, Institute for Economics Studies, Keio University.
  * Kikkawa and Sasahara, (2020) Kikkawa, A. K. and Sasahara, A. (2020).  Gains from trade and the sovereign bond market.  European Economic Review, 124:103413.
  * Krugman and Venables, (1995) Krugman, P. and Venables, A. J. (1995).  Globalization and the inequality of nations.  Quarterly Journal of Economics, 110(4):857–880.
  * Krugman, (1991) Krugman, P. R. (1991).  History versus expectations.  Quarterly Journal of Economics, 106(2):651–667.
  * Leiter et al., (2009) Leiter, A. M., Oberhofer, H., and Raschky, P. A. (2009).  Creative disasters? Flooding effects on capital, labour and productivity within European firms.  Environmental and Resource Economics, 43(3):333–350.
  * Lu et al., (2017) Lu, Y., Tao, Z., and Zhu, L. (2017).  Identifying FDI spillovers.  Journal of International Economics, 107:75–90.
  * Markusen, (2004) Markusen, J. R. (2004).  Multinational Firms and the Theory of International Trade.  MIT Press, Cambridge, MA.
  * Markusen and Venables, (1999) Markusen, J. R. and Venables, A. J. (1999).  Foreign direct investment as a catalyst for industrial development.  European Economic Review, 43(2):335–356.
  * Matsuyama, (1991) Matsuyama, K. (1991).  Increasing returns, industrialization, and indeterminacy of equilibrium.  Quarterly Journal of Economics, 106(2):617–650.
  * Melitz, (2003) Melitz, M. J. (2003).  The impact of trade on intra-industry reallocations and aggregate industry productivity.  Econometrica, 71(6):1695–1725.
  * Meltzer et al., (2021) Meltzer, R., Ellen, I. G., and Li, X. (2021).  Localized commercial effects from natural disasters: The case of Hurricane Sandy and New York City.  Regional Science and Urban Economics, 86:103608.
  * Milner et al., (2006) Milner, C., Reed, G., and Talerngsri, P. (2006).  Vertical linkages and agglomeration effects in Japanese FDI in Thailand.  Journal of the Japanese and International Economies, 20(2):193–208.
  * Neise et al., (2021) Neise, T., Sohns, F., Breul, M., and Revilla Diez, J. (2021).  The effect of natural disasters on FDI attraction: A sector-based analysis over time and space.  Natural Hazards, pages 1–25.
  * Oh and Oetzel, (2011) Oh, C. H. and Oetzel, J. (2011).  Multinationals’ response to major disasters: How does subsidiary investment vary in response to the type of disaster and the quality of country governance?  Strategic Management Journal, 32(6):658–681.
  * Okazaki et al., (2019) Okazaki, T., Okubo, T., and Strobl, E. (2019).  Creative destruction of industries: Yokohama City in the Great Kanto Earthquake 1923.  Journal of Economic History, 79(1):1–31.
  * Okubo and Strobl, (2021) Okubo, T. and Strobl, E. (2021).  Natural disasters, firm survival, and growth: Evidence from the Ise Bay Typhoon, Japan.  Journal of Regional Science, 61(5):944–970.
  * Okuyama and Chang, (2004) Okuyama, Y. and Chang, S. E. (2004).  Modeling Spatial and Economic Impacts of Disasters.  Springer, Berlin Heidelberg.
  * Ottaviano, (2001) Ottaviano, G. I. (2001).  Monopolistic competition, trade, and endogenous spatial fluctuations.  Regional Science and Urban Economics, 31(1):51–77.
  * Ramondo and Rodríguez-Clare, (2013) Ramondo, N. and Rodríguez-Clare, A. (2013).  Trade, multinational production, and the gains from openness.  Journal of Political Economy, 121(2):273–322.
  * Redding and Turner, (2015) Redding, S. J. and Turner, M. A. (2015).  Transportation costs and the spatial organization of economic activity.  In Handbook of Regional and Urban Economics, volume 5, pages 1339–1398. Elsevier, Amsterdam.
  * Rodríguez-Clare, (1996) Rodríguez-Clare, A. (1996).  Multinationals, linkages, and economic development.  American Economic Review, 86(4):852–873.
  * Russ, (2007) Russ, K. N. (2007).  The endogeneity of the exchange rate as a determinant of FDI: A model of entry and multinational firms.  Journal of International Economics, 71(2):344–372.
  * Schubert and Smulders, (2019) Schubert, K. and Smulders, S. (2019).  How important are uncertainty and dynamics for environmental and climate policy? Some analytics.  Journal of Environmental Economics and Management, 97:1–22.
  * Sivapuram and Shaw, (2020) Sivapuram, P. and Shaw, R. (2020).  International investments and businesses as enablers of globalization of local risks: A case for risk communication and climate fragility reduction.  Progress in Disaster Science, 8:100125.
  * Tanaka, (2015) Tanaka, A. (2015).  The impacts of natural disasters on plants’ growth: Evidence from the Great Hanshin-Awaji (Kobe) earthquake.  Regional Science and Urban Economics, 50:31–41.
  * Todo et al., (2015) Todo, Y., Nakajima, K., and Matous, P. (2015).  How do supply chain networks affect the resilience of firms to natural disasters? Evidence from the Great East Japan Earthquake.  Journal of Regional Science, 55(2):209–229.
  * Toner-Rodgers and Friedt, (2020) Toner-Rodgers, A. and Friedt, F. (2020).  Natural disasters, FDI and intra-national spillovers: Evidence from India.  mimeo.
  * Vu and Noy, (2018) Vu, T. B. and Noy, I. (2018).  Natural disasters and firms in Vietnam.  Pacific Economic Review, 23(3):426–452.


# An Efficient 3D Gaussian Representation
for Monocular/Multi-view Dynamic Scenes

Kai Katsumata Duc Minh Vo Hideki Nakayama
The University of Tokyo, Japan
<EMAIL_ADDRESS>
###### Abstract

In novel view synthesis of scenes from multiple input views, 3D Gaussian
splatting emerges as a viable alternative to existing radiance field
approaches, delivering great visual quality and real-time rendering. While
successful in static scenes, the present advancement of 3D Gaussian
representation, however, faces challenges in dynamic scenes in terms of memory
consumption and the need for numerous observations per time step, due to the
onus of storing 3D Gaussian parameters per time step. In this study, we
present an efficient 3D Gaussian representation tailored for dynamic scenes in
which we define positions and rotations as functions of time while leaving
other time-invariant properties of the static 3D Gaussian unchanged. Notably,
our representation reduces memory usage, which is consistent regardless of the
input sequence length. Additionally, it mitigates the risk of overfitting
observed frames by accounting for temporal changes. The optimization of our
Gaussian representation based on image and flow reconstruction results in a
powerful framework for dynamic scene view synthesis in both monocular and
multi-view cases. We obtain the highest rendering speed of $118$ frames per
second (FPS) at a resolution of $1352\times 1014$ with a single GPU, showing
the practical usability and effectiveness of our proposed method in dynamic
scene rendering scenarios.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
PSNR: 33.56 | PSNR: 36.27 | PSNR: 33.61 | PSNR: 21.52 | PSNR: 37.45 |
FPS: 0.54 | FPS: 1.23 | FPS: 0.25 | FPS: 169 | FPS: 146 |
Mem: 471MB | Mem: 1.2GB | Mem: 48MB | Mem: 21MB | Mem: 89MB |
K-Planes [10] | V4D [11] | TiNeuVox-B [8] | 3DGS [15] | Ours | Ground Truth

Figure 1: Our method achieves real-time dynamic scene rendering of radiance
fields with rendering quality that equals the previous methods with the best
quality. Keys to this performance are a memory-efficient dynamic 3D Gaussian
representation, which approximates 3D Gaussian parameters changing over time
with a few parameters, and the optimization with image and flow
reconstruction. We achieve high-quality novel view synthesis of dynamic scenes
at a rendering speed slightly slower than 3D Gaussian Splatting [15]. It also
suffices with only a minor additional memory consumption. We show the
rendering examples on the Mutant scene in the D-NeRF dataset, visual quality
(PSNR), rendering speed (FPS), and memory used to store optimized parameters.
Non-obvious differences in quality are highlighted.

##  1 Introduction

The landscape of novel view synthesis of scenes captured through multiple
images/videos has undergone a revolutionary transformation, owing principally
to significant advances in neural radiance field (NeRF) approaches. While
achieving remarkable visual quality, particularly in dynamic scenes [26, 19,
12], they inevitably confront hurdles in terms of high-speed training and
rendering [22, 27, 25, 24]. This limitation originates from the multi-layer
perceptron (MLP) optimization process, leading to lengthy inference times and
associated computational costs. The recently proposed 3D Gaussian splatting
[15], introduced differentiable 3D Gaussian representation and point-based
rasterization, signaling a departure from neural network reliance. Notably, 3D
Gaussian splatting emerges as a promising solution that not only accelerates
both training and rendering processes but also delivers rendered scenes with
high quality, rivaling the levels set by NeRF [22]. However, it encounters a
challenge in the realm of memory usage and the need for many observations in
dynamic scene synthesis [21]. It stores a plethora of 3D Gaussian parameters
per each time step. Storing parameters per each time step involves numerous
observations per time step, implying the failure in the monocular or few view
setups. Our study aims to tackle this issue, paving the way for more memory-
efficient while keeping the high-quality rendering characteristic of 3D
Gaussian splitting in dynamic scene synthesis.

We introduce an efficient dynamic 3D Gaussian representation, containing time-
invariant and time-varying parameters to capture dynamic motion effectively.
Like [15, 21], we use scaling factors in the covariance matrix, opacity, and
color as time-invariant parameters. Since modeling the change of positions
over time is important to represent dynamic scenes [26, 24, 25], as time-
varying parameters, we express each 3D Gaussian’s position as a function of
time to model the temporal change of the position. We fit the position by
using the Fourier approximation, inspired by the fact that motion is sometimes
periodic, as commonly observed in humans and articulated objects. The time-
varying parameters make our representation dynamic, meaning that 3D Gaussian
moves and rotates over time. Moreover, as we use a function with a few
parameters to represent the position, the small degree of freedom contributes
to the smoothness of reconstructed scenes, enhancing the robustness to unseen
views. Importantly, our representation’s memory consumption is solely
determined by the number of 3D Gaussians and the number of the approximation
function’s parameters, independent of the input length. Beyond optimizing
Gaussian representations through image-level reconstruction, we further
enhance temporal consistency by supervising the Gaussian with optical flow
obtained from input videos. This not only ensures high-quality reconstruction
but also facilitates the generalization of the representation.

Our experiments on dynamic datasets, including D-NeRF [26], DyNeRF [19], and
HyperNeRF [25], demonstrate the effectiveness of optimizing our dynamic 3D
Gaussian from both monocular and multi-view videos and show that our proposed
method achieves rendering quality that rivals previous neural radiance field
approaches [10, 8, 25, 11]. We also can achieve rendering speeds similar to a
fast radiance field method [15] and reasonable memory usage, and importantly,
our approach offers real-time rendering with high quality for novel view
synthesis of dynamic scenes (see Fig. 1). Finally, we show an editing
application enabled by the explicit property of 3D Gaussian representations.
In a nutshell, our contributions are:

  * •

We present an efficient dynamic 3D Gaussian representation, which enables
real-time novel view synthesis of dynamic scenes from monocular/multi-view
video inputs.

  * •

The time-varying parameters introduce a compact representation for modeling
dynamic scenes.

  * •

Our dynamic 3D Gaussian representation enables the high-quality real-time
dynamic scene rendering of high-resolution images of $1352\times 1014$ at a
frame per second (FPS) of 118 with a single GPU.

##  2 Related Work

We first briefly overview the radiance field for dynamic scenes, then discuss
the recent efforts for efficient radiance fields with explicit representation,
such as grid-, plane-, hash-, and point-based methods, placing our work in
real-time dynamic view synthesis.

###  2.1 Dynamic view synthesis

Applications in virtual reality and computer vision often require the
reconstruction of 4D scenes. Several works extend NeRF [22] to handle dynamic
scenes in multi-view or monocular setups by time-varying NeRF [26, 19, 12].
The regularization techniques for temporal smoothness enable suitable scene
representations from monocular videos [20]. Additional sensory information is
also useful for spatio-temporal regularization. Some attempts [35, 20, 12]
employ depth or flow, which are observed or predicted with external networks
to reconstruct the scene from sparse observations. Deformation-based
approaches [36, 24, 25, 33], another direction of the efforts for dynamic
reconstruction, combine static NeRF and deformation fields. Tremendous efforts
show high visual quality for dynamic view synthesis. However, they are still
afflicted with slow rendering speed. Our study aims to enable real-time
dynamic view synthesis with high visual quality. We aim to extend 3D Gaussian
splatting to dynamic scene reconstruction to achieve high-speed rendering
while maintaining the rendering quality from sparse training views.

###  2.2 Explicit Radiance Fields

NeRF is an implicit model with a large neural network queried many times
during training, making a drawback that it is slow optimization and rendering
[22, 38]. A recent line of work [39, 9] addresses the issue in implicit models
by exploring explicit models, reducing optimization and rendering time.
Plenoxels [9] directly optimizes 3D grid representation instead of neural
networks. Generally, explicit models sacrifice visual quality for fast
training [9]. Hybrid approaches [6, 23, 10, 32] aim to achieve better trade-
offs between training time and visual quality. Instant-NGP allows a compact
MLP by exploiting a multi-level hash grid to encode positions to feature
vectors [23]. Plane-based approaches are principally designed for representing
bounded scenes [5, 1, 13, 7, 10, 4]. MERF [28] employs a multiresolution
representation and a fast contraction function to reconstruct unbounded
scenes. For dynamic scenes, K-planes [10] decomposes 4D dynamic volumes into
multiple feature planes and employs an MLP-based feature decoder for
determining color and density.

###  2.3 Point-based rendering

Points, which naturally come from depth sensors, Structure from Motion (SfM)
[29], or common Multi-View Stereo (MVS) algorithms [30, 31], offer a good
representation for representing fine-grained scenes and complex objects, as
well as facilitating computationally efficient rendering. Due to these aspects
of the representation, they have been well-studied in graphics. The
differentiable pipeline for point-based rendering results in that points can
be used for reconstructing 3D scenes [16, 17, 15, 18]. 3D Gaussian splatting
[15], a NeRF alternative, achieves real-time rendering with high visual
quality for unbounded static scenes at the expense of generalization
performance derived from NeRF’s continuous neural field representation.
Recently, Dynamic 3D Gaussians [21] employs 3D Gaussian splatting for dynamic
scenes, which models dynamic scenes by the Gaussian position and variance at
each timestamp. The position and variance of Gaussians at every timestamp are
effective in modeling scenes from dense multi-view dynamic scenes. However,
this approach presents difficulties in reconstructing monocular dynamic
scenes, resulting in excessive memory consumption, particularly for extended
input sequences. We aim to build a memory-efficient Gaussian representation
for dynamic scenes, even for monocular scenes, while keeping pure 3D Gaussian
representation in order to sacrifice the gift of 3D Gaussians, such as
outstanding rendering speed and ease of direct editing of the scene.

##  3 Overview

The input for our task is a set of images with a timestep and camera
parameters obtained from videos. The original 3D Gaussian representation [15]
is defined by a position (mean), a covariance matrix (decomposed into a
rotation matrix and a scaling vector), an opacity, and a color (determined by
spherical harmonics (SH) [3] coefficient). We extend the 3D Gaussian
representation for dynamic scenes. Each 3D Gaussian in our method regards the
position and rotation in the covariance matrix as time-varying parameters and
others as time-invariant parameters over time (Sec. 4). For rendering images,
we use the 3D Gaussian splatting technique [15], which renders an image by
employing Gaussians within the camera plane out of a set of Gaussians. The
technique decides the color for each pixel with $\alpha$-blending of Gaussians
considering their order with respect to the camera view (Sec. 5). For dynamic
scene reconstruction, we optimize the Gaussian parameters with the training
frames. Moreover, flow reconstruction enhances the temporal consistency of the
learned representation. We render the images from Gaussians, intrinsic and
extrinsic camera parameters, and a timestep using a differentiable renderer,
and we update the Gaussian parameters to decrease the distance between
renderings and training images in the image and flow spaces (Sec. 6). The
small degrees of freedom of our representation essentially facilitate the
reconstruction of dynamic scenes from a few observations.

##  4 Dynamic 3D Gaussian representation

To design a compact dynamic 3D Gaussian representation, we aim to express 3D
Gaussian parameters using only a few parameters. Our dynamic scene
representation comprises of a set of dynamic 3D Gaussians, extending the
static 3D Gaussian introduced in [15]. Our dynamic representation lets the 3D
Gaussians move through the scene over time, using both time-varying parameters
(center position and rotation factors) and time-invariant parameters (scale,
color, and opacity). Each dynamic Gaussian encapsulates:

1) a 3D position at time $t$: $[x(t),y(t),z(t)]^{\sf T}\in^{3}$, 2) a 3D
rotation at time $t$ represented by a quaternion:
$[q_{x}(t),q_{y}(t),q_{z}(t),q_{w}(t)]^{\sf T}\in^{4}$ 3) a scaling factor:
$[s_{x},s_{y},s_{z}]^{\sf T}\in^{3}$ 4) spherical harmonics coefficients
representing color with the degrees of freedom $k$: $h\in^{3\times(k+1)^{2}}$
5) an opacity: $o\in$

We approximate the 3D position $x(t),y(t),z(t)$ using Fourier approximation.
At time $t$, it is represented by

$\displaystyle x(t)$ $\displaystyle=w_{x,0}+\sum_{i=1}^{L}w_{x,2i-1}\sin(\pi
t)+w_{x,2i}\cos(\pi t),$ $\displaystyle y(t)$
$\displaystyle=w_{y,0}+\sum_{i=1}^{L}w_{y,2i-1}\sin(\pi t)+w_{y,2i}\cos(\pi
t),$ $\displaystyle z(t)$
$\displaystyle=w_{z,0}+\sum_{i=1}^{L}w_{z,2i-1}\sin(\pi t)+w_{z,2i}\cos(\pi
t),$

where, $w_{\cdot,0},\ldots,w_{\cdot,2L}$ are intercept and coefficients of the
position, and $L$ is the number of terms (harmonics).

We approximate the 3D rotation (quaternion) over time using a linear
approximation. At time $t$, it is defined as

$\displaystyle q_{x}(t)$ $\displaystyle=w_{qx,0}+w_{qx,1}t,$ $\displaystyle
q_{y}(t)$ $\displaystyle=w_{qy,0}+w_{qy,1}t,$ $\displaystyle q_{z}(t)$
$\displaystyle=w_{qz,0}+w_{qz,1}t,$ $\displaystyle q_{w}(t)$
$\displaystyle=w_{qw,0}+w_{qw,1}t,$

where $w_{\cdot,0}$ and $w_{\cdot,1}$ are intercepts and coefficients of the
rotation of Gaussian, respectively.

Each Gaussian at time $t$ is characterized by a 3D covariance matrix
$\mathbf{\Sigma}(t)$ and a 3D center
$\bm{\mu}(t)=\left[x(t),y(t),z(t)\right]^{\sf T}$. The density of 3D Gaussian
on a ray $\vec{\bm{x}}$ is obtained as follows:

$\displaystyle
G_{t}(\vec{\bm{x}})=e^{-\frac{1}{2}(\vec{\bm{x}}-\bm{\mu}(t))^{\sf
T}\mathbf{\Sigma}(t)^{-1}(\vec{\bm{x}}-\bm{\mu}(t))}.$

To constrain the covariance matrix $\mathbf{\Sigma}(t)$ to be a positive semi-
definite matrix during optimization, the covariance matrix
$\mathbf{\Sigma}(t)$ is decomposed by using a scaling matrix
$\mathbf{S}=\operatorname{diag}(s_{x},s_{y},s_{z})$ and a rotation matrix
$\mathbf{R}(t)$ as

$\displaystyle\mathbf{\Sigma}(t)=\mathbf{R}(t)\mathbf{S}\mathbf{S}^{\sf
T}\mathbf{R}(t)^{\sf T}.$

Here, a rotation matrix $\mathbf{R}(t)$ is represented by quaternion
$(q_{x}(t),q_{y}(t),q_{z}(t),q_{w}(t))$.

For each Gaussian, the preceding definition yields $3L+8+3+3(k+1)^{2}+1$
parameters with respect to 3D center, 3D rotation, scale, color, and opacity.
Notably, the parameter count for each Gaussian is defined merely by the number
of approximation terms and spherical harmonic degrees of freedom, with no
regard to time length. When compared to approaches that require parameters for
each time step, this approach saves on memory usage. Memory consumption in our
dynamic scene representation is determined by two hyperparameters (_i.e_., $L$
and $k$) and the number of Gaussians used. Furthermore, the representation,
which is defined as a function of time over continuous time, inhibits
discontinuous movement through time. This characteristic improves robustness
in novel view synthesis settings.

##  5 Rendering via 3D Gaussian Splatting

Rendering with 3D Gaussian applies splatting techniques [15] to the Gaussian
within the camera planes. Zwicker et al. [40] give the projection of the 3D
covariance matrix to 2D covariance matrix. The 3D covariance matrix
$\mathbf{\Sigma}$ is projected into 2D covariance matrix
$\mathbf{\Sigma}^{\prime}$ given a a viewing transformation $\mathbf{W}$ as
follow:

$\displaystyle\mathbf{\Sigma}^{\prime}(t)=\mathbf{J}\mathbf{W}\mathbf{\Sigma}(t)\mathbf{W}^{\sf
T}\mathbf{J}^{\sf T},$

where $\mathbf{J}$ is the Jacobian of the affine approximation of the
projective transformation at Gaussian center $\bm{\mu}(t)$:

$\displaystyle\mathbf{J}=\begin{bmatrix}\frac{1}{v_{z}}&0&-\frac{v_{x}}{v_{z}^{2}}\\\
0&\frac{1}{v_{z}}&-\frac{v_{y}}{v_{z}^{2}}\\\ 0&0&0\\\ \end{bmatrix},$ (1)

where $[v_{x},v_{y},v_{z}]^{\sf T}=\mathbf{W}\bm{\mu}(t)$ is the camera
coordinate of the Gaussian center $\bm{\mu}(t)$ obtained by the viewing
transformation, which projects the points from the world space to the camera
space.

Similar to NeRF style volumetric rendering, the point-based rendering computes
the color $C$ of a pixel by evaluating the blending of $N$ ordered points that
overlap the pixel:

$\displaystyle
C=\sum_{i=1}^{N}c_{i}\alpha_{i}\prod_{j=1}^{i-1}(1-\alpha_{j}),$

where, $c_{i}$ represents the color of a Gaussian evaluated by SH
coefficients, and $\alpha_{i}$ represents the density that is calculated from
a 2D Gaussian with covariance $\mathbf{\Sigma}^{\prime}$ at time $t$ and a 2D
center $\bm{\mu}^{\prime}$ at time $t$ and an optimized opacity $o$.

##  6 Optimization of the dynamic 3D Gaussian representation

We optimize the Gaussian parameters: intercepts and coefficients of position
and rotation $w$, a scaling factor $s_{x},s_{y},s_{z}$, SH coefficients $h$
and an opacity $o$, based on the iterations of rendering and comparing the
rendered images with training frames in the captured videos. To compare the
rendered and training views, the loss function contains L1 and D-SSIM:

$\displaystyle\operatorname{\mathcal{L}}_{\rm
recon}=(1-\lambda)|\hat{I}-I|+\lambda\operatorname{\mathcal{L}}_{D-SSIM},$

where $I$ and $\hat{I}$ are target and rendered images, respectively. The loss
function moves and rotates the anisotropic Gaussians and changes their color
and opacity so that each Gaussian covers a homogeneous area. Since it just
fixes incorrectly positioned Gaussians, the over- or under-representation of
the set of Gaussians for the scene needs a mechanism for creating Gaussians
that reconstruct the scene or destroy extra Gaussians. We also follow the
divide and prune techniques in 3D Gaussian splatting for producing a compact
and precise representation of the scene. We surveil the gradients of each
Gaussian and densify Gaussians by splitting a Gaussian with a large gradient
and a large scale into two small Gaussians and cloning a Gaussian with a large
gradient and a small scale to two Gaussians. Moreover, we remove transparent
Gaussians with an opacity less than a threshold value of 0.005.

Earning the prior of the point sets for a dynamic scene to gain in the
reconstruction of the scene is challenging. Instead of the initialization of a
set of Gaussians with a set of sparse points from SfM [29], we initialize a
set of Gaussians randomly using a uniform distribution. It avoids time-
intensive SfM and challenges in SfM from dynamic scenes, in particular a
monocular setting. Deeming the frames in the captured datasets as static
scene, and we optimize static representation in the former static stage to
learn the prior of Gaussians. In other words, we optimize the parameters
consistent all over time (_i.e_., scale, SH coefficients, and opacity) and
intercepts for a center and a rotation
($w_{x,0},w_{y,0},w_{z,0},w_{qx,0},w_{qy,0},w_{qz,0},w_{qw,0}$) out of the
Gaussian parameters, in the static stage. After the static stage, we optimize
the whole parameters of the set of Gaussians to reconstruct a dynamic region
as a dynamic stage.

Another challenge in the reconstruction of the dynamic scene is ambiguity come
from the limited number of captured views at a timestep. Since a dynamic scene
contains temporal changes, such as moving objects and changing shape, it is
hard to share the scene information over frames with different timesteps. To
overcome the ambiguity, we employ flow information. We supervises the flows of
the optimizable Gaussians with the ground truth optical flows of the input
frames. We use a RAFT algorithm [34] to obtain ground truth flow for training
views: forward flow $f_{\rm fwd}$ and backward flow $f_{\rm bwd}$ between two
adjacent frames. We combine the flow loss $\operatorname{\mathcal{L}}_{\rm
flow}$ with the reconstruction loss that compare the renderings and training
views:

$\displaystyle\operatorname{\mathcal{L}}=\operatorname{\mathcal{L}}_{\rm
recon}+\lambda_{\rm flow}\operatorname{\mathcal{L}}_{\rm flow}(\hat{F},F)$

where, $F=\\{f_{\rm fwd},f_{\rm bwd}\\}$ and $\hat{F}$ are the ground truth
flow and the flow of the Gaussians, respectively, and $\lambda_{\rm flow}$ is
a balancing hyperparameter for the flow term. Instead of applying an optical
flow algorithm for renderings, we create pseudo optical flow from Gaussian
representation. The motion of the scene is represented by only the
coefficients of the 3D Gaussian mean $w_{x,1\leq i},w_{y,1\leq i},w_{z,1\leq
i}$. We can calculate the scene flow in the 3D space by

$\displaystyle\hat{f}^{x}_{\rm fwd}=x(t+\Delta t)-x(t),\quad\hat{f}^{x}_{\rm
fwd}=x(t)-x(t-\Delta t),$ $\displaystyle\hat{f}^{y}_{\rm fwd}=y(t+\Delta
t)-y(t),\quad\hat{f}^{y}_{\rm fwd}=y(t)-y(t-\Delta t),$
$\displaystyle\hat{f}^{z}_{\rm fwd}=z(t+\Delta t)-z(t),\quad\hat{f}^{z}_{\rm
fwd}=z(t)-z(t-\Delta t),$

where $\Delta t$ is the difference between the timesteps of the two image
frames. The scene flow is projected into 2D camera plane using

$\displaystyle\hat{f}^{\rm xyz}_{\\{\rm
fwd,bwd\\}}=\mathbf{J}[\hat{f}^{x}_{\\{\rm fwd,bwd\\}},\hat{f}^{y}_{\\{\rm
fwd,bwd\\}},\hat{f}^{z}_{\\{\rm fwd,bwd\\}}]^{\sf T},$

where $\mathbf{J}$ is the Jacobian of the affine approximation of the
projective transformation at the Gaussian center $\bm{\mu}$ (Eq. 1). Regarding
the scene flows on camera plane as RGB colors, we can apply the point-based
rendering for calculate an optical flow of a pixel as the $\alpha$-blending:

$\displaystyle\hat{f}_{\rm fwd}=\sum_{i=1}^{N}\hat{f}^{\rm xyz}_{{\rm
fwd},i}\alpha_{i}\prod_{j=1}^{i-1}(1-\alpha_{j}).$

The backward flow is calculated in the same way. The optical flow $\hat{F}$
consists of the forward flows $\hat{f}_{\rm fwd}$ and backward flows
$\hat{f}_{\rm bwd}$ for the all pixels. The flow loss
$\operatorname{\mathcal{L}}_{\rm flow}$ takes L1 loss between the ground truth
flows and the optical flow of the Gaussian for both directions of the flows.
The flow loss provides the spatial-temporal consistency to our method without
the additional computation cost in rendering since it does not need additional
components. We exclude the flow loss for the D-NeRF dataset because the
teleport of the cameras between neighbouring frames causes difficulties in
calculating ground truth flows.

##  7 Experiment

###  7.1 Evaluation data

We evaluate our dynamic Gaussian presentation using three dynamic scene
datasets: a synthetic one D-NeRF [26], and two real ones DyNeRF [19] and
HyperNeRF [25].

| Ground Truth | K-Planes [10] | V4D [11] | TiNeuVox-B [8] | 3DGS [15] | Ours
---|---|---|---|---|---|---
T-Rex |  |  |  |  |  |
|  |  |  |  |  |
Stand Up |  |  |  |  |  |
|  |  |  |  |  |
Hook |  |  |  |  |  |
|  |  |  |  |  |
Bouncing Balls |  |  |  |  |  |
|  |  |  |  |  |
Jumping Jacks |  |  |  |  |  |
|  |  |  |  |  |
Figure 2: Qualitative comparison on D-NeRF [26]. We highlight the difference
by zoom view. Our method achieves competitive visual quality with strong
baselines. While our method successfully reconstructs intricate details like
hands, it causes the blurred sphere shape.

D-NeRF dataset [26]. This dataset comprises eight videos of varying lengths,
ranging from 50 to 200 frames per video. The camera setup is designed to mimic
a monocular camera setting by teleporting between adjacent time steps. The
test views are from novel camera positions. We train and render at the
resolution of $800\times 800$.

DyNeRF dataset [19]. The multi-camera dataset includes six 10-second videos
captured at 30 FPS using 15 – 20 synchronized fixed cameras. For evaluation, a
central camera is used, while training utilizes frames from the other cameras.
The training and rendering resolution is set at $1352\times 1014$.

HyperNeRF dataset [25]. This dataset encompasses videos ranging from 8 to 15
seconds, captured at 15 FPS using two Pixel 3 phones. The training and
rendering processes are conducted at a resolution of $540\times 960$.

Table 1: Quantitative results on the D-NeRF dataset [26]. Our method performs
competitively against neural radiance field approaches in terms of visual
quality and achieves fastest rendering speed in the previous methods with the
high quality. Results excepting FPS of [8, 10, 11] are adopted from the
original papers.

| PSNR$\uparrow$ | MS-SSIM $\uparrow$ | FPS $\uparrow$ | Train Time $\downarrow$ | Mem$\downarrow$
---|---|---|---|---|---
TiNeuVox-S [8] | 30.75 | 0.96 | 0.32 | 8 mins | 8MB
TiNeuVox-B [8] | 32.67 | 0.97 | 0.13 | 28 mins | 48MB
K-Planes [10] | 31.61 | 0.97 | 0.54 | 52 mins |  $\sim$497MB
V4D [11] | 33.72 | 0.98 | 1.47 | 6.9 hrs | 1.2GB
3DGS [15] | 20.51 | 0.89 | 170 | 6 mins |  $\sim$50MB
D-3DGS | 17.22 | 0.81 | 173 | 15 mins |  $\sim$913MB
Ours | 32.07 | 0.96 | 150 | 8 mins |  $\sim$159MB

Table 2: Quantitative results on the DyNeRF scenes [19].

| PSNR$\uparrow$ | MS-SSIM $\uparrow$ | FPS $\uparrow$ | Train Time $\downarrow$ | Mem$\downarrow$
---|---|---|---|---|---
K-Planes [10] | 31.63 | 0.964 | 0.31 | 1.8 hrs |  $\sim$309MB
3DGS [15] | 20.94 | 0.800 | 109 | 20 mins |  $\sim$198MB
D-3DGS | 24.36 | 0.834 | 119 | 51 mins |  $\sim$2.3GB
Ours | 28.89 | 0.945 | 118 | 1 hrs |  $\sim$338MB

###  7.2 Implementation details

We adhere to the experimental setup in the 3D Gaussian splatting paper [15].
The number of approximation terms of Gaussian centers $L$ is set to 2 for the
D-NeRF dataset and 5 for both the DyNeRF and HyperNeRF datasets. Our two-stage
optimization process begins with an initial fitting of parameters, excluding
the coefficients for Gaussian position and variance. This initial stage spans
3000 iterations and utilizes all training views in a static setting.
Subsequently, we engage in a dynamic stage, adjusting all Gaussian parameters
in 27000 iterations. The entire optimization process encompasses 30000
iterations in total. We set the flow loss weight $\lambda_{\rm flow}$ at 1000
and acquire ground truth flow through RAFT pretrained on the Sintel dataset
[2]. All experiments are conducted on a single RTX A6000 GPU.

Table 3: Quantitative results on the HyperNeRF dataset [25].

| FPS$\uparrow$ | Train Time$\downarrow$ | Mem$\downarrow$ | Broom | 3D Printer | Chicken | Peel Banana | Mean
---|---|---|---|---|---|---|---|---
| PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
HyperNeRF [25] | 0.36 | 48 hrs† | 15MB | 19.3 | 0.591 | 20.0 | 0.821 | 26.9 | 0.948 | 23.3 | 0.896 | 22.2 | 0.811
TiNeuVox-B [8] | 0.14 | 30 mins | 48MB | 21.5 | 0.686 | 22.8 | 0.841 | 28.3 | 0.947 | 24.4 | 0.873 | 24.3 | 0.837
V4D [11] | 0.15 | 7 hrs | 1.2GB | 22.1 | 0.669 | 23.2 | 0.835 | 28.4 | 0.929 | 25.2 | 0.873 | 24.7 | 0.827
Ours | 188 | 1 hrs | $\sim$720MB | 20.4 | 0.660 | 21.1 | 0.754 | 25.8 | 0.859 | 26.6 | 0.920 | 23.5 | 0.798

†Train time of HyperNeRF [25] is estimated from descriptions in their paper.
Originally reported as 8 hours on 4 TPU v4s [14], the TPU v4 is slightly
faster than the A100 GPU, and the A100 GPU is at least 1.5 times faster than
th A6000 GPU.

Ground Truth | K-Planes [10] | Ours
---|---|---
|  |
|  |
Figure 3: Qualitative comparison on DyNeRF [19]. We highlight the difference by zoom view. Ground Truth | V4D [11] | Ours
---|---|---
|  |
Figure 4: Qualitative comparison on HyperNeRF [25]. Our method offers sharp
results.

###  7.3 Evaluation setup

Compared methods. We benchmark our method against several SoTA methods,
including TiNeuVox [8], K-Planes [10], V4D [11], HyperNeRF [25], 3D Gaussian
Splatting (3DGS) [15], and a D-3DGS baseline. Notably, D-3DGS is a dynamic
extension of the 3DGS method, which stores both position and rotation for each
time step.

Evaluation metrics. We assess the methods using various metrics, including
PSNR, SSIM [37], FPS, Training time, and memory used to store optimized
parameters. Memory consumption includes 3D Gaussian parameters, neural network
parameters, voxel/plane representation, and so on.

Table 4: Per-scene quantitative comparison on D-NeRF scenes of different $L$.
The highest mean score is achieved with $L=2$, but increasing the complexity L
improves visual quality in some scenes (Jumping Jacks and T-Rex).

PSNR
---
| Stand Up | Jacks | Balls | Lego | Warrior | Hook | T-Rex | Mutant | Mean
K-Planes [10] | 33.72 | 32.64 | 41.22 | 25.47 | 25.70 | 28.50 | 31.79 | 33.56 | 31.61
$L=1$ | 40.21 | 27.22 | 30.27 | 24.00 | 32.42 | 32.84 | 25.15 | 38.04 | 31.26
$L=2$ | 39.10 | 30.95 | 33.29 | 22.21 | 34.15 | 33.19 | 26.22 | 37.45 | 32.07
$L=3$ | 38.09 | 32.78 | 32.54 | 19.58 | 35.36 | 30.23 | 27.73 | 35.06 | 31.42
$L=4$ | 35.83 | 32.93 | 30.39 | 18.17 | 34.38 | 27.74 | 28.17 | 32.58 | 30.02
$L=5$ | 32.89 | 30.71 | 27.68 | 17.63 | 32.64 | 25.43 | 26.11 | 29.09 | 27.77
MS-SSIM
K-Planes [10] | 0.983 | 0.977 | 0.992 | 0.948 | 0.952 | 0.954 | 0.981 | 0.983 | 0.971
$L=1$ | 0.989 | 0.954 | 0.979 | 0.899 | 0.937 | 0.967 | 0.960 | 0.986 | 0.958
$L=2$ | 0.987 | 0.970 | 0.983 | 0.883 | 0.948 | 0.967 | 0.964 | 0.982 | 0.960
$L=3$ | 0.984 | 0.975 | 0.982 | 0.837 | 0.955 | 0.951 | 0.967 | 0.971 | 0.952
$L=4$ | 0.977 | 0.973 | 0.975 | 0.810 | 0.947 | 0.929 | 0.966 | 0.957 | 0.941
$L=5$ | 0.965 | 0.964 | 0.966 | 0.795 | 0.933 | 0.904 | 0.950 | 0.931 | 0.926

Ours without $\operatorname{\mathcal{L}}_{\rm flow}$ |  | Ours
---|---|---
|  |
Figure 5: Qualitative comparison of disabled and enabled flow loss on DyNeRF. We highlight the difference by zoom view. |
---|---
Figure 6: Composition of two scenes. Our method allows adding 3D objects
represented 3D Gaussian into a 3D Gaussian scene. We highlight the added
object.

###  7.4 Results

Quantitative results. The quantitative results on the D-NeRF dataset are
detailed in Tab. 1. Our method demonstrates performance comparable to TiNeuVox
and K-Planes in terms of visual quality indicated by PSNR and SSIM. Notably,
it excels in training time, FPS, and memory size, boasting a rendering speed
that is $300\times$ faster than K-Planes. Furthermore, our method surpasses
both 3DGS and D-3DGS in terms of visual quality without compromising rendering
speed. In the DyNeRF scenes experiment, detailed in Tab. 2, while our method
does not exceed the baseline in reconstruction quality, it shows a substantial
improvement in FPS. Since the DyNeRF scenes contain multi-view data, the
D-3DGS baseline naturally improves static 3DGS, unlike monocular scenes.
Impressively, our method even attains rendering speeds that exceed real-time
performance, at a high resolution of $1354\times 1014$. In the challenging
HyperNeRF dataset, which is captured by only two moving cameras, referenced in
Tab. 3, our method not only demonstrates rapid rendering speeds but also
attains an average PSNR that exceeds that of HyperNeRF. It excels particularly
in the Peel Banana scene, achieving the highest PSNR and MS-SSIM scores.
However, our method’s robustness is compromised in scenarios with inaccurate
camera poses (3D Printer). Our explicit representation and the straightforward
rasterization do not yield plausible results for contradictory views.

Qualitative results. Figs. 2, 3 and 4 show that our method yields sharper
rendering results and it frequently reconstructs fine details of the scenes
accurately. Nonetheless, some cases lacking the approximation complexity lead
to blurred results, such as T-Rex scene in D-NeRF (Fig. 2), and “steak” in the
DyNeRF scene (Fig. 3).

Effect of the number of parameters $L$. Tab. 4 shows per-scene PSNR and SSIM
scores of K-Planes and our method with the different $L$. It is observed that
the optimal $L$ for novel view synthesis varies from scene to scene,
highlighting the necessity for complex approximations to capture intricate
motions effectively. Additionally, visual comparisons drawn from our method
with and without the flow loss (Fig. 5) reveal that incorporating the flow
loss mitigates ghostly artifacts and significantly enhances the accuracy of
color reconstruction.

Scene composition. Since our dynamic 3D Gaussian representation still uses
pure 3D Gaussian representation, the learned representation is easy to edit
Gaussians. We demonstrate the composition of two scenes with our
representation. Fig. 6 illustrates this by combining the Mutant scene from the
D-NeRF dataset with the Seared Steak scene from the DyNeRF dataset. This
demonstrates our method’s capability in editing dynamic 3D scenes.

##  8 Discussion and Conclusion

Limitations and future directions. Our dynamic Gaussians is defined over all
time of the dynamic scene. This representation implicitly assumes Gaussians
exist over all time of the scene. It enables us to model naturally the rigid
and non-rigid deformation in the scene. On the other hand, it is hard to model
the change of topology, the occurrence of Gaussians, and extinction of
Gaussians (_e.g_., fluid). To overcome the limitation, considering the
lifetime of Gaussians, such as adding start and end time parameters, will
model changes in scene topology.

Our Gaussian representation sacrifices the continuity and smoothness of neural
field-based volume rendering, causing performance drops in inaccurate camera
poses and degradation of generalization performance. A potential extension of
our method is the distillation of the learned NeRF model, which improves the
rendering quality without losing the advantage of fast rendering of our
method.

Conclusion. We present a dynamic 3D Gaussian representation that allows the
accurate reconstruction of dynamic scenes and real-time rendering. We propose
a representation for position and variance of 3D Gaussians as a function of
time for modeling the motion of the scene. Our model reconstructs dynamic
scenes by fitting the Gaussian parameters using the views of all timesteps.
Furthermore, we introduce the flow loss that constrains the scene flow of
learned Gaussian representation with the ground truth flow. Our experiments on
synthetic and real datasets show that the proposed method achieves real-time
dynamic scene rendering even for a high resolution. We will explore the
rendering with higher rendering quality with the aid of recent efforts on
dynamic scene reconstruction.

## References

  * An et al. [2023] Sizhe An, Hongyi Xu, Yichun Shi, Guoxian Song, Umit Y Ogras, and Linjie Luo.  PanoHead: Geometry-aware 3D full-head synthesis in 360deg.  In _CVPR_ , pages 20950–20959, 2023.
  * Butler et al. [2012] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black.  A naturalistic open source movie for optical flow evaluation.  In _ECCV_ , pages 611–625, 2012.
  * Cabral et al. [1987] Brian Cabral, Nelson Max, and Rebecca Springmeyer.  Bidirectional reflection functions from surface bump maps.  _SIGGRAPH_ , 21(4):273–281, 1987.
  * Cao and Johnson [2023] Ang Cao and Justin Johnson.  Hexplane: A fast representation for dynamic scenes.  In _CVPR_ , pages 130–141, 2023.
  * Chan et al. [2022] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3D generative adversarial networks.  In _CVPR_ , pages 16123–16133, 2022.
  * Chen et al. [2022] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  TensoRF: Tensorial radiance fields.  In _ECCV_ , pages 333–350, 2022.
  * Dong et al. [2023] Zijian Dong, Xu Chen, Jinlong Yang, Michael J Black, Otmar Hilliges, and Andreas Geiger.  AG3D: Learning to generate 3D avatars from 2D image collections.  In _ICCV_ , pages 14916–14927, 2023.
  * Fang et al. [2022] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian.  Fast dynamic radiance fields with time-aware neural voxels.  In _SIGGRAPH Asia_ , 2022.
  * Fridovich-Keil et al. [2022] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In _CVPR_ , pages 5501–5510, 2022.
  * Fridovich-Keil et al. [2023] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa.  K-Planes: Explicit radiance fields in space, time, and appearance.  In _CVPR_ , pages 12479–12488, 2023.
  * Gan et al. [2023] Wanshui Gan, Hongbin Xu, Yi Huang, Shifeng Chen, and Naoto Yokoya.  V4d: Voxel for 4d novel view synthesis.  _IEEE TVCG_ , 2023.
  * Gao et al. [2021] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.  Dynamic view synthesis from dynamic monocular video.  In _ICCV_ , pages 5712–5721, 2021.
  * He et al. [2023] Honglin He, Zhuoqian Yang, Shikai Li, Bo Dai, and Wayne Wu.  OrthoPlanes: A novel representation for better 3D-awareness of GANs.  In _ICCV_ , pages 22996–23007, 2023.
  * Jouppi et al. [2023] Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Clifford Young, Xiang Zhou, Zongwei Zhou, and David A Patterson.  TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings.  In _Proceedings of the 50th Annual International Symposium on Computer Architecture_ , pages 1–14, 2023.
  * Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3D Gaussian splatting for real-time radiance field rendering.  _ACM TOG_ , 42(4):1–14, 2023.
  * Keselman and Hebert [2022] Leonid Keselman and Martial Hebert.  Approximate differentiable rendering with algebraic surfaces.  In _ECCV_ , pages 596–614, 2022.
  * Kopanas et al. [2021] Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis.  Point-based neural rendering with per-view optimization.  In _Comput. Graph. Forum_ , pages 29–43, 2021.
  * Kopanas et al. [2022] Georgios Kopanas, Thomas Leimkühler, Gilles Rainer, Clément Jambon, and George Drettakis.  Neural point catacaustics for novel-view synthesis of reflections.  _ACM TOG_ , 41(6):1–15, 2022.
  * Li et al. [2022] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al.  Neural 3D video synthesis from multi-view video.  In _CVPR_ , pages 5521–5531, 2022.
  * Li et al. [2021] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In _CVPR_ , pages 6498–6508, 2021.
  * Luiten et al. [2024] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan.  Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis.  In _3DV_ , 2024.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM TOG_ , 41(4):1–15, 2022.
  * Park et al. [2021a] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In _ICCV_ , pages 5865–5874, 2021a.
  * Park et al. [2021b] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz.  HyperNeRF: a higher-dimensional representation for topologically varying neural radiance fields.  _ACM TOG_ , 40(6):1–12, 2021b.
  * Pumarola et al. [2021] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-NeRF: Neural radiance fields for dynamic scenes.  In _CVPR_ , pages 10318–10327, 2021.
  * Reiser et al. [2021] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.  In _ICCV_ , pages 14335–14345, 2021.
  * Reiser et al. [2023] Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman.  MERF: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes.  _ACM TOG_ , 42(4):1–12, 2023.
  * Schönberger and Frahm [2016] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _CVPR_ , 2016.
  * Schönberger et al. [2016] Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys.  Pixelwise view selection for unstructured multi-view stereo.  In _ECCV_ , pages 501–518, 2016.
  * Seitz et al. [2006] Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski.  A comparison and evaluation of multi-view stereo reconstruction algorithms.  In _CVPR_ , pages 519–528, 2006.
  * Shao et al. [2023] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu.  Tensor4D: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering.  In _CVPR_ , pages 16632–16642, 2023.
  * Song et al. [2023] Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger.  NeRFPlayer: A streamable dynamic scene representation with decomposed neural radiance fields.  _IEEE TVCG_ , 29(5):2732–2742, 2023.
  * Teed and Deng [2020] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In _ECCV_ , pages 402–419, 2020.
  * Tian et al. [2023] Fengrui Tian, Shaoyi Du, and Yueqi Duan.  Mononerf: Learning a generalizable dynamic radiance field from monocular videos.  In _ICCV_ , pages 17903–17913, 2023.
  * Tretschk et al. [2021] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhofer, Christoph Lassner, and Christian Theobalt.  Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video.  In _ICCV_ , 2021.
  * Wang et al. [2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE TIP_ , 13(4):600–612, 2004.
  * Xie et al. [2022] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar.  Neural fields in visual computing and beyond.  _Comput. Graph. Forum_ , 2022.
  * Yu et al. [2021] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  PlenOctrees for real-time rendering of neural radiance fields.  In _ICCV_ , pages 5752–5761, 2021.
  * Zwicker et al. [2002] Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross.  EWA splatting.  _IEEE TVCG_ , 8(3):223–238, 2002.