* Mitchell et al. (2021) Mitchell, J. G., De Nolfo, G. A., Hill, M. E., Christian, E. R., Richardson, I. G., McComas, D. J., McNutt, R. L., Mitchell, D. G., Schwadron, N. A., Bale, S. D., Giacalone, J., Joyce, C. J., Niehof, J. T., and Szalay, J. R. (2021), Energetic Electron Observations by Parker Solar Probe/IS$\odot$IS during the First Widespread SEP Event of Solar Cycle 25 on 2020 November 29, ApJ, 919(2), 119, 10.3847/1538-4357/ac110e.
  * Möstl et al. (2012) Möstl, C., Farrugia, C. J., Kilpua, E. K. J., Jian, L. K., Liu, Y., Eastwood, J. P., Harrison, R. A., Webb, D. F., Temmer, M., Odstrcil, D., Davies, J. A., Rollett, T., Luhmann, J. G., Nitta, N., Mulligan, T., Jensen, E. A., Forsyth, R., Lavraud, B., de Koning, C. A., Veronig, A. M., Galvin, A. B., Zhang, T. L., and Anderson, B. J. (2012), Multi-point Shock and Flux Rope Analysis of Multiple Interplanetary Coronal Mass Ejections around 2010 August 1 in the Inner Heliosphere, ApJ, 758, 10, 10.1088/0004-637X/758/1/10.
  * Möstl et al. (2018) Möstl, C., Amerstorfer, T., Palmerio, E., Isavnin, A., Farrugia, C. J., Lowder, C., Winslow, R. M., Donnerer, J. M., Kilpua, E. K. J., and Boakes, P. D. (2018), Forward Modeling of Coronal Mass Ejection Flux Ropes in the Inner Heliosphere with 3DCORE,  Space Weather, 16, 216–229, 10.1002/2017SW001735.
  * Möstl et al. (2022) Möstl, C., Weiss, A. J., Reiss, M. A., Amerstorfer, T., Bailey, R. L., Hinterreiter, J., Bauer, M., Barnes, D., Davies, J. A., Harrison, R. A., Freiherr von Forstner, J. L., Davies, E. E., Heyner, D., Horbury, T., and Bale, S. D. (2022), Multipoint Interplanetary Coronal Mass Ejections Observed with Solar Orbiter, BepiColombo, Parker Solar Probe, Wind, and STEREO-A, ApJ, 924(1), L6, 10.3847/2041-8213/ac42d0.
  * Müller et al. (2017) Müller, D., Nicula, B., Felix, S., Verstringe, F., Bourgoignie, B., Csillaghy, A., Berghmans, D., Jiggens, P., García-Ortiz, J. P., Ireland, J., Zahniy, S., and Fleck, B. (2017), JHelioviewer. Time-dependent 3D visualisation of solar and heliospheric data, A&A, 606, A10, 10.1051/0004-6361/201730893.
  * Müller et al. (2020) Müller, D., St. Cyr, O. C., Zouganelis, I., Gilbert, H. R., Marsden, R., Nieves-Chinchilla, T., Antonucci, E., Auchère, F., Berghmans, D., Horbury, T. S., Howard, R. A., Krucker, S., Maksimovic, M., Owen, C. J., Rochus, P., Rodriguez-Pacheco, J., Romoli, M., Solanki, S. K., Bruno, R., Carlsson, M., Fludra, A., Harra, L., Hassler, D. M., Livi, S., Louarn, P., Peter, H., Schühle, U., Teriaca, L., del Toro Iniesta, J. C., Wimmer-Schweingruber, R. F., Marsch, E., Velli, M., De Groof, A., Walsh, A., and Williams, D. (2020), The Solar Orbiter mission. Science overview, A&A, 642, A1, 10.1051/0004-6361/202038467.
  * Mulligan et al. (2013) Mulligan, T., Reinard, A. A., and Lynch, B. J. (2013), Advancing in situ modeling of ICMEs: New techniques for new observations,  Journal of Geophysical Research: Space Physics, 118(4), 1410–1427, 10.1002/jgra.50101.
  * Murray (2018) Murray, S. A. (2018), The Importance of Ensemble Techniques for Operational Space Weather Forecasting,  Space Weather, 16(7), 777–783, 10.1029/2018SW001861.
  * Nieves-Chinchilla et al. (2022) Nieves-Chinchilla, T., Alzate, N., Cremades, H., Rodriguez-Garcia, L., Dos Santos, L. F. G., Narock, A., Xie, H., Szabo Vratislav Krupar, A., Pulupa, M., Lario, D., Stevens, M. L., Palmerio, E., Wilson III, L. B., Reeves Ryun-Young Kwon, K. K., Mays, M. L., St. Cyr, O. C., Hess, P., Seaton, D. B., Niembro, T., Bale, S. D., and Kasper, J. C. (2022), Direct First Parker Solar Probe Observation of the Interaction of Two Successive Interplanetary Coronal Mass Ejections in 2020 November, ApJ, 930(1), 88, 10.3847/1538-4357/ac590b.
  * Nitta et al. (2012) Nitta, N. V., Liu, Y., DeRosa, M. L., and Nightingale, R. W. (2012), What Are Special About Ground-Level Events?. Flares, CMEs, Active Regions and Magnetic Field Connection, Space Sci. Rev., 171(1-4), 61–83, 10.1007/s11214-012-9877-1.
  * Odstrcil (2003) Odstrcil, D. (2003), Modeling 3-D solar wind structure,  Advances in Space Research, 32(4), 497–506, 10.1016/S0273-1177(03)00332-6.
  * Odstrcil and Pizzo (1999) Odstrcil, D., and Pizzo, V. J. (1999), Distortion of the interplanetary magnetic field by three-dimensional propagation of coronal mass ejections in a structured solar wind, J. Geophys. Res., 104(A12), 28,225–28,240, 10.1029/1999JA900319.
  * Odstrcil et al. (2004) Odstrcil, D., Riley, P., and Zhao, X. P. (2004), Numerical simulation of the 12 May 1997 interplanetary CME event, J. Geophys. Res., 109(A2), A02116, 10.1029/2003JA010135.
  * Ogilvie and Desch (1997) Ogilvie, K. W., and Desch, M. D. (1997), The wind spacecraft and its early scientific results,  Advances in Space Research, 20, 559–568, 10.1016/S0273-1177(97)00439-0.
  * Ogilvie et al. (1995) Ogilvie, K. W., Chornay, D. J., Fritzenreiter, R. J., Hunsaker, F., Keller, J., Lobell, J., Miller, G., Scudder, J. D., Sittler, E. C., Jr., Torbert, R. B., Bodet, D., Needell, G., Lazarus, A. J., Steinberg, J. T., Tappan, J. H., Mavretic, A., and Gergin, E. (1995), SWE, A Comprehensive Plasma Instrument for the Wind Spacecraft, Space Sci. Rev., 71, 55–77, 10.1007/BF00751326.
  * Palmerio et al. (2019) Palmerio, E., Scolini, C., Barnes, D., Magdalenić, J., West, M. J., Zhukov, A. N., Rodriguez, L., Mierla, M., Good, S. W., Morosan, D. E., Kilpua, E. K. J., Pomoell, J., and Poedts, S. (2019), Multipoint Study of Successive Coronal Mass Ejections Driving Moderate Disturbances at 1 au, ApJ, 878(1), 37, 10.3847/1538-4357/ab1850.
  * Palmerio et al. (2021a) Palmerio, E., Nieves-Chinchilla, T., Kilpua, E. K. J., Barnes, D., Zhukov, A. N., Jian, L. K., Witasse, O., Provan, G., Tao, C., Lamy, L., Bradley, T. J., Mays, M. L., Möstl, C., Roussos, E., Futaana, Y., Masters, A., and Sánchez-Cano, B. (2021a), Magnetic Structure and Propagation of Two Interacting CMEs From the Sun to Saturn, Journal of Geophysical Research (Space Physics), 126(11), e2021JA029770, 10.1029/2021JA029770.
  * Palmerio et al. (2021b) Palmerio, E., Kilpua, E. K. J., Witasse, O., Barnes, D., Sánchez-Cano, B., Weiss, A. J., Nieves-Chinchilla, T., Möstl, C., Jian, L. K., Mierla, M., Zhukov, A. N., Guo, J., Rodriguez, L., Lowrance, P. J., Isavnin, A., Turc, L., Futaana, Y., and Holmström, M. (2021b), CME Magnetic Structure and IMF Preconditioning Affecting SEP Transport,  Space Weather, 19(4), e2020SW002654, 10.1029/2020SW002654.
  * Paouris and Mavromichalaki (2017) Paouris, E., and Mavromichalaki, H. (2017), Effective Acceleration Model for the Arrival Time of Interplanetary Shocks driven by Coronal Mass Ejections, Sol. Phys., 292(12), 180, 10.1007/s11207-017-1212-2.
  * Paouris et al. (2021) Paouris, E., Čalogović, J., Dumbović, M., Mays, M. L., Vourlidas, A., Papaioannou, A., Anastasiadis, A., and Balasis, G. (2021), Propagating Conditions and the Time of ICME Arrival: A Comparison of the Effective Acceleration Model with ENLIL and DBEM Models, Sol. Phys., 296(1), 12, 10.1007/s11207-020-01747-4.
  * Pesnell et al. (2012) Pesnell, W. D., Thompson, B. J., and Chamberlin, P. C. (2012), The Solar Dynamics Observatory (SDO), Sol. Phys., 275, 3–15, 10.1007/s11207-011-9841-3.
  * Pinto et al. (2021) Pinto, M., Gonçalves, P., Cardoso, C., Sanchez-Cano, B., Moissl, R., Vainio, R., Oleynik, P., Huovelin, J., Korpela, S., Lehtolainen, A., Grande, M., and Marques, A. (2021), The BepiColombo Radiation Monitor, in European Planetary Science Congress, pp. EPSC2021–204, 10.5194/epsc2021-204.
  * Pizzo et al. (2015) Pizzo, V. J., de Koning, C., Cash, M., Millward, G., Biesecker, D. A., Puga, L., Codrescu, M., and Odstrcil, D. (2015), Theoretical basis for operational ensemble forecasting of coronal mass ejections,  Space Weather, 13(10), 676–697, 10.1002/2015SW001221.
  * Pomoell and Poedts (2018) Pomoell, J., and Poedts, S. (2018), EUHFORIA: European heliospheric forecasting information asset,  Journal of Space Weather and Space Climate, 8, A35, 10.1051/swsc/2018020.
  * Posner (2007) Posner, A. (2007), Up to 1-hour forecasting of radiation hazards from solar energetic ion events with relativistic electrons,  Space Weather, 5(5), 05001, 10.1029/2006SW000268.
  * Posner et al. (2021) Posner, A., Arge, C. N., Staub, J., StCyr, O. C., Folta, D., Solanki, S. K., Strauss, R. D. T., Effenberger, F., Gandorfer, A., Heber, B., Henney, C. J., Hirzberger, J., Jones, S. I., Kühl, P., Malandraki, O., and Sterken, V. J. (2021), A Multi-Purpose Heliophysics L4 Mission,  Space Weather, 19(9), e2021SW002777, 10.1029/2021SW002777.
  * Pulkkinen et al. (2019) Pulkkinen, A. A., Bisi, M. M., Luntama, J. P., Kraft, S., Glover, A., and Heil, M. (2019), ESA Lagrange Space Weather Monitoring Mission to L5 Point, in AGU Fall Meeting Abstracts, vol. 2019, pp. SH24B–01.
  * Qin et al. (2004) Qin, G., Zhang, M., Dwyer, J. R., and Rassoul, H. K. (2004), Interplanetary Transport Mechanisms of Solar Energetic Particles, ApJ, 609(2), 1076–1081, 10.1086/421101.
  * Reames (2021) Reames, D. V. (2021), Solar Energetic Particles. A Modern Primer on Understanding Sources, Acceleration and Propagation, vol. 978, Cham: Springer International Publishing, 10.1007/978-3-030-66402-2.
  * Reiss et al. (2021) Reiss, M. A., Möstl, C., Bailey, R. L., Rüdisser, H. T., Amerstorfer, U. V., Amerstorfer, T., Weiss, A. J., Hinterreiter, J., and Windisch, A. (2021), Machine Learning for Predicting the Bz Magnetic Field Component From Upstream in Situ Observations of Solar Coronal Mass Ejections,  Space Weather, 19(12), e2021SW002859, 10.1029/2021SW002859.
  * Richardson (2018) Richardson, I. G. (2018), Solar wind stream interaction regions throughout the heliosphere,  Living Reviews in Solar Physics, 15(1), 1, 10.1007/s41116-017-0011-z.
  * Richardson and Cane (1996) Richardson, I. G., and Cane, H. V. (1996), Particle flows observed in ejecta during solar event onsets and their implication for the magnetic field topology, J. Geophys. Res., 101(A12), 27,521–27,532, 10.1029/96JA02643.
  * Richardson et al. (2014) Richardson, I. G., von Rosenvinge, T. T., Cane, H. V., Christian, E. R., Cohen, C. M. S., Labrador, A. W., Leske, R. A., Mewaldt, R. A., Wiedenbeck, M. E., and Stone, E. C. (2014), $>$ 25 MeV Proton Events Observed by the High Energy Telescopes on the STEREO A and B Spacecraft and/or at Earth During the First $\sim$ Seven Years of the STEREO Mission, Sol. Phys., 289(8), 3059–3107, 10.1007/s11207-014-0524-8.
  * Richardson et al. (2018) Richardson, I. G., Mays, M. L., and Thompson, B. J. (2018), Prediction of Solar Energetic Particle Event Peak Proton Intensity Using a Simple Algorithm Based on CME Speed and Direction and Observations of Associated Solar Phenomena,  Space Weather, 16(11), 1862–1881, 10.1029/2018SW002032.
  * Riley et al. (2018) Riley, P., Mays, M. L., Andries, J., Amerstorfer, T., Biesecker, D., Delouille, V., Dumbović, M., Feng, X., Henley, E., Linker, J. A., Möstl, C., Nuñez, M., Pizzo, V., Temmer, M., Tobiska, W. K., Verbeke, C., West, M. J., and Zhao, X. (2018), Forecasting the Arrival Time of Coronal Mass Ejections: Analysis of the CCMC CME Scoreboard,  Space Weather, 16, 1245–1260, 10.1029/2018SW001962.
  * Rodríguez-García et al. (2021) Rodríguez-García, L., Gómez-Herrero, R., Zouganelis, I., Balmaceda, L., Nieves-Chinchilla, T., Dresing, N., Dumbović, M., Nitta, N. V., Carcaboso, F., dos Santos, L. F. G., Jian, L. K., Mays, L., Williams, D., and Rodríguez-Pacheco, J. (2021), The unusual widespread solar energetic particle event on 2013 August 19. Solar origin and particle longitudinal distribution, A&A, 653, A137, 10.1051/0004-6361/202039960.
  * Rodríguez-Pacheco et al. (2020) Rodríguez-Pacheco, J., Wimmer-Schweingruber, R. F., Mason, G. M., Ho, G. C., Sánchez-Prieto, S., Prieto, M., Martín, C., Seifert, H., Andrews, G. B., Kulkarni, S. R., Panitzsch, L., Boden, S., Böttcher, S. I., Cernuda, I., Elftmann, R., Espinosa Lara, F., Gómez-Herrero, R., Terasa, C., Almena, J., Begley, S., Böhm, E., Blanco, J. J., Boogaerts, W., Carrasco, A., Castillo, R., da Silva Fariña, A., de Manuel González, V., Drews, C., Dupont, A. R., Eldrum, S., Gordillo, C., Gutiérrez, O., Haggerty, D. K., Hayes, J. R., Heber, B., Hill, M. E., Jüngling, M., Kerem, S., Knierim, V., Köhler, J., Kolbe, S., Kulemzin, A., Lario, D., Lees, W. J., Liang, S., Martínez Hellín, A., Meziat, D., Montalvo, A., Nelson, K. S., Parra, P., Paspirgilis, R., Ravanbakhsh, A., Richards, M., Rodríguez-Polo, O., Russu, A., Sánchez, I., Schlemm, C. E., Schuster, B., Seimetz, L., Steinhagen, J., Tammen, J., Tyagi, K., Varela, T., Yedla, M., Yu, J., Agueda, N., Aran, A., Horbury, T. S., Klecker, B., Klein, K. L., Kontar, E., Krucker, S., Maksimovic, M., Malandraki, O., Owen, C. J., Pacheco, D., Sanahuja, B., Vainio, R., Connell, J. J., Dalla, S., Dröge, W., Gevin, O., Gopalswamy, N., Kartavykh, Y. Y., Kudela, K., Limousin, O., Makela, P., Mann, G., Önel, H., Posner, A., Ryan, J. M., Soucek, J., Hofmeister, S., Vilmer, N., Walsh, A. P., Wang, L., Wiedenbeck, M. E., Wirth, K., and Zong, Q. (2020), The Energetic Particle Detector. Energetic particle instrument suite for the Solar Orbiter mission, A&A, 642, A7, 10.1051/0004-6361/201935287.
  * Rouillard et al. (2012) Rouillard, A. P., Sheeley, N. R., Tylka, A., Vourlidas, A., Ng, C. K., Rakowski, C., Cohen, C. M. S., Mewaldt, R. A., Mason, G. M., Reames, D., Savani, N. P., StCyr, O. C., and Szabo, A. (2012), The Longitudinal Properties of a Solar Energetic Particle Event Investigated Using Modern Solar Imaging, ApJ, 752(1), 44, 10.1088/0004-637X/752/1/44.
  * Salman et al. (2020) Salman, T. M., Winslow, R. M., and Lugaz, N. (2020), Radial Evolution of Coronal Mass Ejections Between MESSENGER, Venus Express, STEREO, and L1: Catalog and Analysis,  Journal of Geophysical Research: Space Physics, 125(1), eJA027084, 10.1029/2019JA027084.
  * Sánchez-Cano et al. (2018) Sánchez-Cano, B., Witasse, O., Lester, M., Rahmati, A., Ambrosi, R., Lillis, R., Leblanc, F., Blelly, P.-L., Costa, M., Cowley, S. W. H., Espley, J. R., Milan, S. E., Plaut, J. J., Lee, C., and Larson, D. (2018), Energetic Particle Showers Over Mars from Comet C/2013 A1 Siding Spring,  Journal of Geophysical Research: Space Physics, 123(10), 8778–8796, 10.1029/2018JA025454.
  * Saunders et al. (2004) Saunders, R. S., Arvidson, R. E., Badhwar, G. D., Boynton, W. V., Christensen, P. R., Cucinotta, F. A., Feldman, W. C., Gibbs, R. G., Kloss, J., C., Landano, M. R., Mase, R. A., McSmith, G. W., Meyer, M. A., Mitrofanov, I. G., Pace, G. D., Plaut, J. J., Sidney, W. P., Spencer, D. A., Thompson, T. W., and Zeitlin, C. J. (2004), 2001 Mars Odyssey Mission Summary, Space Sci. Rev., 110(1), 1–36, 10.1023/B:SPAC.0000021006.84299.18.
  * Schwadron et al. (2010) Schwadron, N. A., Townsend, L., Kozarev, K., Dayeh, M. A., Cucinotta, F., Desai, M., Golightly, M., Hassler, D., Hatcher, R., Kim, M. Y., Posner, A., PourArsalan, M., Spence, H. E., and Squier, R. K. (2010), Earth-Moon-Mars Radiation Environment Module framework,  Space Weather, 8(10), S00E02, 10.1029/2009SW000523.
  * Scolini et al. (2020) Scolini, C., Chané, E., Temmer, M., Kilpua, E. K. J., Dissauer, K., Veronig, A. M., Palmerio, E., Pomoell, J., Dumbović, M., Guo, J., Rodriguez, L., and Poedts, S. (2020), CME–CME Interactions as Sources of CME Geoeffectiveness: The Formation of the Complex Ejecta and Intense Geomagnetic Storm in 2017 Early September, ApJS, 247(1), 21, 10.3847/1538-4365/ab6216.
  * Shea and Smart (2012) Shea, M. A., and Smart, D. F. (2012), Space Weather and the Ground-Level Solar Proton Events of the 23rd Solar Cycle, Space Sci. Rev., 171(1-4), 161–188, 10.1007/s11214-012-9923-z.
  * Shiota and Kataoka (2016) Shiota, D., and Kataoka, R. (2016), Magnetohydrodynamic simulation of interplanetary propagation of multiple coronal mass ejections with internal magnetic flux rope (SUSANOO-CME),  Space Weather, 14(2), 56–75, 10.1002/2015SW001308.
  * Steinvall et al. (2021) Steinvall, K., Khotyaintsev, Y. V., Cozzani, G., Vaivads, A., Yordanova, E., Eriksson, A. I., Edberg, N. J. T., Maksimovic, M., Bale, S. D., Chust, T., Krasnoselskikh, V., Kretzschmar, M., Lorfèvre, E., Plettemeier, D., Souček, J., Steller, M., Štverák, Š., Vecchio, A., Horbury, T. S., O’Brien, H., Evans, V., Fedorov, A., Louarn, P., Génot, V., André, N., Lavraud, B., Rouillard, A. P., and Owen, C. J. (2021), Solar wind current sheets and deHoffmann-Teller analysis. First results from Solar Orbiter’s DC electric field measurements, A&A, 656, A9, 10.1051/0004-6361/202140855.
  * Stone et al. (1998) Stone, E. C., Frandsen, A. M., Mewaldt, R. A., Christian, E. R., Margolies, D., Ormes, J. F., and Snow, F. (1998), The Advanced Composition Explorer, Space Sci. Rev., 86, 1–22, 10.1023/A:1005082526237.
  * Stumpo et al. (2021) Stumpo, M., Benella, S., Laurenza, M., Alberti, T., Consolini, G., and Marcucci, M. F. (2021), Open Issues in Statistical Forecasting of Solar Proton Events: A Machine Learning Perspective,  Space Weather, 19(10), e2021SW002794, 10.1029/2021SW002794.
  * SunPy Community et al. (2020) SunPy Community, Barnes, W. T., Bobra, M. G., Christe, S. D., Freij, N., Hayes, L. A., Ireland , J., Mumford, S., Perez-Suarez, D., Ryan, D. F., Shih, A. Y., Chanda, P., Glogowski, K., Hewett, R., Hughitt, V. K., Hill, A., Hiware, K., Inglis, A., Kirk, M. S. F., Konge, S., Mason, J. P., Maloney, S. A., Murray, S. A., Panda, A., Park, J., Pereira, T. M. D., Reardon, K., Savage, S., Sipőcz, B. M., Stansby, D., Jain, Y., Taylor, G., Yadav, T., Rajul, and Dang, T. K. (2020), The SunPy Project: Open Source Development and Status of the Version 1.0 Core Package, ApJ, 890(1), 68, 10.3847/1538-4357/ab4f7a.
  * Temmer (2021) Temmer, M. (2021), Space weather: the solar perspective,  Living Reviews in Solar Physics, 18(1), 4, 10.1007/s41116-021-00030-3.
  * Thernisien (2011) Thernisien, A. (2011), Implementation of the Graduated Cylindrical Shell Model for the Three-dimensional Reconstruction of Coronal Mass Ejections, ApJS, 194, 33, 10.1088/0067-0049/194/2/33.
  * Thernisien et al. (2009) Thernisien, A., Vourlidas, A., and Howard, R. A. (2009), Forward Modeling of Coronal Mass Ejections Using STEREO/SECCHI Data, Sol. Phys., 256, 111–130, 10.1007/s11207-009-9346-5.
  * Thernisien et al. (2006) Thernisien, A. F. R., Howard, R. A., and Vourlidas, A. (2006), Modeling of Flux Rope Coronal Mass Ejections, ApJ, 652, 763–773, 10.1086/508254.
  * Torsti et al. (1995) Torsti, J., Valtonen, E., Lumme, M., Peltonen, P., Eronen, T., Louhola, M., Riihonen, E., Schultz, G., Teittinen, M., Ahola, K., Holmlund, C., Kelhä, V., Leppälä, K., Ruuska, P., and Strömmer, E. (1995), Energetic Particle Experiment ERNE, Sol. Phys., 162(1-2), 505–531, 10.1007/BF00733438.
  * Čalogović et al. (2021) Čalogović, J., Dumbović, M., Sudar, D., Vršnak, B., Martinić, K., Temmer, M., and Veronig, A. M. (2021), Probabilistic Drag-Based Ensemble Model (DBEM) Evaluation for Heliospheric Propagation of CMEs, Sol. Phys., 296(7), 114, 10.1007/s11207-021-01859-5.
  * Velli et al. (2020) Velli, M., Harra, L. K., Vourlidas, A., Schwadron, N., Panasenco, O., Liewer, P. C., Müller, D., Zouganelis, I., St Cyr, O. C., Gilbert, H., Nieves-Chinchilla, T., Auchère, F., Berghmans, D., Fludra, A., Horbury, T. S., Howard, R. A., Krucker, S., Maksimovic, M., Owen, C. J., Rodríguez-Pacheco, J., Romoli, M., Solanki, S. K., Wimmer-Schweingruber, R. F., Bale, S., Kasper, J., McComas, D. J., Raouafi, N., Martinez-Pillet, V., Walsh, A. P., De Groof, A., and Williams, D. (2020), Understanding the origins of the heliosphere: integrating observations and measurements from Parker Solar Probe, Solar Orbiter, and other space- and ground-based observatories, A&A, 642, A4, 10.1051/0004-6361/202038245.
  * Verbeke et al. (2019a) Verbeke, C., Mays, M. L., Temmer, M., Bingham, S., Steenburgh, R., Dumbović, M., Núñez, M., Jian, L. K., Hess, P., Wiegand, C., Taktakishvili, A., and Andries, J. (2019a), Benchmarking CME Arrival Time and Impact: Progress on Metadata, Metrics, and Events,  Space Weather, 17(1), 6–26, 10.1029/2018SW002046.
  * Verbeke et al. (2019b) Verbeke, C., Pomoell, J., and Poedts, S. (2019b), The evolution of coronal mass ejections in the inner heliosphere: Implementing the spheromak model with EUHFORIA, A&A, 627, A111, 10.1051/0004-6361/201834702.
  * von Rosenvinge et al. (2008) von Rosenvinge, T. T., Reames, D. V., Baker, R., Hawk, J., Nolan, J. T., Ryan, L., Shuman, S., Wortman, K. A., Mewaldt, R. A., Cummings, A. C., Cook, W. R., Labrador, A. W., Leske, R. A., and Wiedenbeck, M. E. (2008), The High Energy Telescope for STEREO, Space Sci. Rev., 136(1-4), 391–435, 10.1007/s11214-007-9300-5.
  * Vourlidas (2015) Vourlidas, A. (2015), Mission to the Sun-Earth L5 Lagrangian Point: An Optimal Platform for Space Weather Research,  Space Weather, 13, 197–201, 10.1002/2015SW001173.
  * Vourlidas et al. (2013) Vourlidas, A., Lynch, B. J., Howard, R. A., and Li, Y. (2013), How Many CMEs Have Flux Ropes? Deciphering the Signatures of Shocks, Flux Ropes, and Prominences in Coronagraph Observations of CMEs, Sol. Phys., 284, 179–201, 10.1007/s11207-012-0084-8.
  * Vourlidas et al. (2017) Vourlidas, A., Balmaceda, L. A., Stenborg, G., and Dal Lago, A. (2017), Multi-viewpoint Coronal Mass Ejection Catalog Based on STEREO COR2 Observations, ApJ, 838, 141, 10.3847/1538-4357/aa67f0.
  * Vourlidas et al. (2019) Vourlidas, A., Patsourakos, S., and Savani, N. P. (2019), Predicting the geoeffective properties of coronal mass ejections: current status, open issues and path forward,  Philosophical Transactions of the Royal Society of London Series A, 377(2148), 20180,096, 10.1098/rsta.2018.0096.
  * Vršnak and Žic (2007) Vršnak, B., and Žic, T. (2007), Transit times of interplanetary coronal mass ejections and the solar wind speed, A&A, 472(3), 937–943, 10.1051/0004-6361:20077499.
  * Vršnak et al. (2013) Vršnak, B., Žic, T., Vrbanec, D., Temmer, M., Rollett, T., Möstl, C., Veronig, A., Čalogović, J., Dumbović, M., Lulić, S., Moon, Y. J., and Shanmugaraju, A. (2013), Propagation of Interplanetary Coronal Mass Ejections: The Drag-Based Model, Sol. Phys., 285(1-2), 295–315, 10.1007/s11207-012-0035-4.
  * Vršnak et al. (2019) Vršnak, B., Amerstorfer, T., Dumbović, M., Leitner, M., Veronig, A. M., Temmer, M., Möstl, C., Amerstorfer, U. V., Farrugia, C. J., and Galvin, A. B. (2019), Heliospheric Evolution of Magnetic Clouds, ApJ, 877(2), 77, 10.3847/1538-4357/ab190a.
  * Webb and Howard (2012) Webb, D. F., and Howard, T. A. (2012), Coronal Mass Ejections: Observations,  Living Reviews in Solar Physics, 9(1), 3, 10.12942/lrsp-2012-3.
  * Wiedenbeck et al. (2013) Wiedenbeck, M. E., Mason, G. M., Cohen, C. M. S., Nitta, N. V., Gómez-Herrero, R., and Haggerty, D. K. (2013), Observations of Solar Energetic Particles from 3He-rich Events over a Wide Range of Heliographic Longitude, ApJ, 762(1), 54, 10.1088/0004-637X/762/1/54.
  * Wiedenbeck et al. (2017) Wiedenbeck, M. E., Angold, N. G., Birdwell, B., Burnham, J. A., Christian, E. R., Cohen, C. M. S., Cook, W. R., Cummings, A. C., Davis, A. D., Dirks, G., Do, D. H., Everett, D. T., Goodwin, P. A., Hanley, J. J., Hernandez, L., Kecman, B., Klemic, J., Labrador, A. W., Leske, R. A., Lopez, S., Link, J. T., McComas, D. J., Mewaldt, R. A., Miyasaka, H., Nahory, B. W., Rankin, J. S., Riggans, G., Rodriguez, B., Rusert, M. D., Shuman, S. A., Simms, K. M., Stone, E. C., von Rosenvinge, T. T., Weidner, S. E., and White, M. L. (2017), Capabilities and Performance of the High-Energy Energetic-Particles Instrument for the Parker Solar Probe Mission, in 35th International Cosmic Ray Conference (ICRC2017), International Cosmic Ray Conference, vol. 301, p. 16.
  * Wijsen et al. (2021) Wijsen, N., Samara, E., Aran, À., Lario, D., Pomoell, J., and Poedts, S. (2021), A Self-consistent Simulation of Proton Acceleration and Transport Near a High-speed Solar Wind Stream, ApJ, 908(2), L26, 10.3847/2041-8213/abe1cb.
  * Witasse et al. (2017) Witasse, O., Sánchez-Cano, B., Mays, M. L., Kajdič, P., Opgenoorth, H., Elliott, H. A., Richardson, I. G., Zouganelis, I., Zender, J., Wimmer-Schweingruber, R. F., Turc, L., Taylor, M. G. G. T., Roussos, E., Rouillard, A., Richter, I., Richardson, J. D., Ramstad, R., Provan, G., Posner, A., Plaut, J. J., Odstrcil, D., Nilsson, H., Niemenen, P., Milan, S. E., Mandt, K., Lohf, H., Lester, M., Lebreton, J.-P., Kuulkers, E., Krupp, N., Koenders, C., James, M. K., Intzekara, D., Holmstrom, M., Hassler, D. M., Hall, B. E. S., Guo, J., Goldstein, R., Goetz, C., Glassmeier, K. H., Génot, V., Evans, H., Espley, J., Edberg, N. J. T., Dougherty, M., Cowley, S. W. H., Burch, J., Behar, E., Barabash, S., Andrews, D. J., and Altobelli, N. (2017), Interplanetary coronal mass ejection observed at STEREO-A, Mars, comet 67P/Churyumov-Gerasimenko, Saturn, and New Horizons en route to Pluto: Comparison of its Forbush decreases at 1.4, 3.1, and 9.9 AU,  Journal of Geophysical Research: Space Physics, 122, 7865–7890, 10.1002/2017JA023884.
  * Wold et al. (2018) Wold, A. M., Mays, M. L., Taktakishvili, A. r., Jian, L. K., Odstrcil, D., and MacNeice, P. (2018), Verification of real-time WSA-ENLIL+Cone simulations of CME arrival-time at the CCMC from 2010 to 2016,  Journal of Space Weather and Space Climate, 8, A17, 10.1051/swsc/2018005.
  * Wu et al. (2007) Wu, C.-C., Fry, C. D., Wu, S. T., Dryer, M., and Liou, K. (2007), Three-dimensional global simulation of interplanetary coronal mass ejection propagation from the Sun to the heliosphere: Solar event of 12 May 1997, J. Geophys. Res., 112(A9), A09104, 10.1029/2006JA012211.
  * Young et al. (2021) Young, M. A., Schwadron, N. A., Gorby, M., Linker, J., Caplan, R. M., Downs, C., Török, T., Riley, P., Lionello, R., Titov, V., Mewaldt, R. A., and Cohen, C. M. S. (2021), Energetic Proton Propagation and Acceleration Simulated for the Bastille Day Event of 2000 July 14, ApJ, 909(2), 160, 10.3847/1538-4357/abdf5f.
  * Zank et al. (2015) Zank, G. P., Hunana, P., Mostafavi, P., Le Roux, J. A., Li, G., Webb, G. M., Khabarova, O., Cummings, A., Stone, E., and Decker, R. (2015), Diffusive Shock Acceleration and Reconnection Acceleration Processes, ApJ, 814(2), 137, 10.1088/0004-637X/814/2/137.
  * Zhang et al. (2007) Zhang, J., Richardson, I. G., Webb, D. F., Gopalswamy, N., Huttunen, E., Kasper, J. C., Nitta, N. V., Poomvises, W., Thompson, B. J., Wu, C. C., Yashiro, S., and Zhukov, A. N. (2007), Solar and interplanetary sources of major geomagnetic storms (Dst $\leq$ -100 nT) during 1996-2005, J. Geophys. Res., 112, A10102, 10.1029/2007JA012321.
  * Zhang and Zhao (2017) Zhang, M., and Zhao, L. (2017), Precipitation and Release of Solar Energetic Particles from the Solar Coronal Magnetic Field, ApJ, 846(2), 107, 10.3847/1538-4357/aa86a8.
  * Zhang et al. (2003) Zhang, M., Jokipii, J. R., and McKibben, R. B. (2003), Perpendicular Transport of Solar Energetic Particles in Heliospheric Magnetic Fields, ApJ, 595(1), 493–499, 10.1086/377301.
  * Zhao and Dryer (2014) Zhao, X., and Dryer, M. (2014), Current status of CME/shock arrival time prediction,  Space Weather, 12(7), 448–469, 10.1002/2014SW001060.
  * Zhao et al. (2002) Zhao, X. P., Plunkett, S. P., and Liu, W. (2002), Determination of geometrical and kinematical properties of halo coronal mass ejections using the cone model, J. Geophys. Res., 107(A8), 1223, 10.1029/2001JA009143.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

STRL’22: First International Workshop on Spatio-Temporal Reasoning and
Learning, July 24, 2022, Vienna, Austria

[orcid=0000-0002-9732-2414<EMAIL_ADDRESS>] [1] [1]

[orcid=0000-0003-2613-7569<EMAIL_ADDRESS>] [1]

[orcid=0000-0002-1185-2722<EMAIL_ADDRESS>] [1]

[1]Corresponding author. [1]These authors contributed equally.

# Scene Separation & Data Selection: Temporal Segmentation Algorithm for Real-
Time Video Stream Analysis

Yuelin Xin  Southwest Jiaotong University, Chengdu, China  University of
Leeds, Leeds, UK  Zihan Zhou  Yuxuan Xia

(2022)

###### Abstract

We present 2SDS (Scene Separation and Data Selection algorithm), a temporal
segmentation algorithm used in real-time video stream interpretation. It
complements CNN-based models to make use of temporal information in videos.
2SDS can detect the change between scenes in a video stream by com-paring the
image difference between two frames. It separates a video into segments
(scenes), and by combining itself with a CNN model, 2SDS can select the
optimal result for each scene. In this paper, we will be discussing some basic
methods and concepts behind 2SDS, as well as presenting some preliminary
experiment results regarding 2SDS. During these experiments, 2SDS has achieved
an overall accuracy of over 90

###### keywords:

scene separation temporal segmentation real-time video analysis dHash

##  1 Introduction

Image recognition models have gone increasingly accurate in the past few
years, yet video semantics tasks are still challenging. A detailed
comprehension on video stream could play a significant part in video
accessibility [1], surveillance footage auto-interpretation [2, 3], and so on.
These technologies have already been proven useful on large video platforms
like YouTube, used for real-time video interpretation and video topic
analysis.

###  1.1 The Problem

In the processing of video stream, a 2D CNN can be extended into 3D CNN by
adding a temporal dimension [4], but this approach can be hazardous if the
video is too long, or it is of indefinite length. However, a 2D CNN is still
very usable in a traditional image recognition or image segmentation task.

The problem is that 2D CNNs only recognise a video as discrete images, rather
than a continuous stream of images. This poses some issues. For example, a CNN
model could not resolve the motion of a person (e.g., walking, dancing) be-
cause the person is stationary in every frame, and this will cause the loss of
significant information in video analysis. So, we need to devise an
implementation that complements the CNN model to solve the continuity issue.
This implementation should group the discrete frames (adjacent on the temporal
axis) that look similar to each other into _scenes_ , this procedure is what
we call _temporal segmentation_ (also referred as _scene separation_ in 2SDS,
see Fig. 1 for example).

Figure 1: Overall effect of the scene separation procedure. The whole video
stream will be separated into scenes, in each of which the images in the video
remain relatively stationary.

###  1.2 Related Work

SlowFast Networks. The SlowFast Networks use a two-pathway architecture for
video recognition, the slow pathway (low frame rate) is used to capture
spatial semantics, and the fast pathway (high frame rate) is used to capture
temporal semantics like motions in a relatively fine temporal resolution [5].

###  1.3 Our Work

What we have achieved is to devise the temporal segmentation algorithm, 2SDS,
which stands for “Scene Separation and Data Selection algorithm”. It can slice
the video stream into segments on the temporal axis, so it can be interpreted
using 2D CNN models while preserving critical information on the temporal
dimension. By combining 2SDS with a CNN model (Fig. 2), this implementation is
similar to the SlowFast Networks on splitting the input into two pathways, in
which the 2SDS is similar to the fast pathway of the SlowFast Networks, except
we do not introduce another neural network, but we replace the network with
the faster 2SDS, which guarantees even better temporal resolution.

Figure 2: 2SDS used together with a CNN model. The 2SDS algorithm can separate
the scenes in a continuous video stream and select the result produced by the
CNN model, the two together, can produce a scene-separated recognition result.

##  2 Motivation: Why Not Neural Networks

Traditionally, RNN-based models have been quite successful in processing
sequential information like time. However, the usage of RNN or even neural
networks is not practical in time sensitive tasks like real-time object
recognition and live video stream analysis, which requires fast responding
algorithms, and RNNs usually cannot meet those requirements.

RNN-based models like LSTM [6] generally have a longer respond time even
compared to CNN-based models (although the difference between them vary with
different settings of hyperparameters). A CNN + RNN architecture model would
mean the doubling of processing time, which is something we would rather avoid
when dealing with video stream analysis tasks.

However, RNNs do have the advantage of acting upon temporal information,
especially for models like LSTM. So, we need to help the CNN-based models to
preserve temporal information, and that is where we introduce our temporal
segmentation algorithm, 2SDS.

By adding the 2SDS algorithm, alongside a CNN model, we were able to achieve
RNN-like results. In the meantime, by avoiding the introduction of a neural
network, this implementation is also faster than the CNN + RNN architecture or
the CNN + CNN architecture.

##  3 Method: 2SDS

2SDS stands for “Scene Separation and Data Selection algorithm”. It works as a
temporal segmentation and result selection algorithm to complement CNN-based
models. It contains a two-part procedure of separating the video stream into
segments and selecting a representative recognition result from the CNN model
for output.

2SDS utilises the difference hash (dHash) method [7] to obtain the rough image
difference between two frames, if two frames have a very little difference,
they will be grouped into the same scene. This method involves a few simple
steps to calculate, and it is the most important method 2SDS uses to achieve
scene separation. As the calculation is relatively simple and straight
forward, this makes 2SDS extremely fast on scene separation.

Also, 2SDS uses a pooling-layer-like data smoothing and data selection method
to pick out the representative recognition result for a particular scene. This
method can generally improve the accuracy of the output because it can smooth
out the data on undesired frame moving (e.g., camera shaking, broken frames).
Alongside the data smoothing mechanism, another data selection mechanism is
implemented to select the representative recognition result (referred as
_representative_ in the following sections) from the whole data segment of a
scene (referred as _candidate_ in the following sections).

###  3.1 Scene Separation: based on dHash

The scene separation procedure of 2SDS (Fig. 3) is based on an improved
version of the dHash algorithm, which is originally used to judge the
similarity of two images. By applying the scene separation procedure, the
temporal information can be preserved by the sequencing of the separated
scenes. The exact workflow of the scene separation process is discussed
extensively below.

Figure 3: Image processing in 2SDS based on an improved dHash algorithm. The
two image processing parts in 2SDS, the first step is down sample, and the
second step is gray scale conversion.

Down sampling. To make a rough comparison between two frames in a video, the
frames need to be down sampled from their original size to an 8 by 9 (row by
column) sub-image. This approach can both simplify the remaining calculation
and make the algorithm less sensitive to subtle changes between frames.

Gray scale. We apply gray scale manipulation on the previous sub-image using
the Luminosity algorithm, this step is purely for reducing the complexity of
calculating the difference on 3 channels. By converting the RGB channels into
one gray scale channel, this approach dramatically lessens the complexity of
the algorithm.

Calculate Hash value. The derived gray scale image is converted into a single
16-bit hexadecimal hash value. The algorithm looks at all the 8 rows
separately, each row has 9 gray scale values from 0 to 255. These 9 values are
converted to 8 binary numbers under the following rules:

  1. (a)

One binary value stands for the gray scale difference between two adjacent
pixels.

  2. (b)

If the gray scale value of the pixel on the left is greater than the pixel on
the right, the binary value should be 1, otherwise, it should be 0.

  3. (c)

Every row should end up with an 8-bit long binary sequence.

An example is given in Fig. 4.

Figure 4: Example on binary sequence conversion. The derived binary sequence
of row 1 in this case is 00111010.

Using this method, we can derive eight 8-bit long binary sequences, each of
them can be represented by a 2-bit long hexadecimal value. And by
concatenating all the 2-bit hexadecimal values, we can obtain a 16-bit long
hexadecimal hash value, and this value will represent the whole image (this is
also the reason why the original image is down sampled into an 8 by 9 sub-
image rather than an 8 by 8 sub-image, because the 8 by 8 image will face some
inconvenience when converting into a hexadecimal hash value).

Calculating the Hamming distance. By calculating the Hamming distance between
the hash values of two adjacent frames, we can judge whether the two frames
are in the same scene or not. If the Hamming distance is greater than a
threshold (usually 5), we consider the two frames to be in two different
scenes, and we can separate them accordingly. For the calculation of Hamming
distance, we can simply use an Exclusive Or operator on the two hash values,
here is an example below (the Hamming distance is 7 in this case):

$\text{c4e0d8988c989898}\ \oplus\ \text{eee6989c8c989898}\ =\ \ 7$ (1)

###  3.2 Data Selection and Data Smoothing

When the scene separation process detected a new scene, the data collected on
the previous scene is packed into an array. This array contains all the
recognition results produced by the CNN model in the previous scene, and the
CNN model would have a recognition output on every frame in this scene.

To have a solid output for 2SDS, we need to perform 2 extra steps: data
smoothing and data selection. The method that is implemented here is inspired
by the pooling layer in a convolutional neural network.

Data smoothing procedure. This step is also called LWAP (Length Weighted
Average Pooling). We start by segmenting the array containing all the
recognition data into small groups of a defined size. Then, we apply the
following formulas:

$WAL=\frac{\sum_{i=1}^{i\leq\varphi}{{(L}_{i}\times\omega_{i})}}{\sum_{i=1}^{i\leq\varphi}{\
\omega}_{i}}$ (2)
$\left\\{\begin{array}[]{l}f_{i}(D)=\min_{i=0}{|\operatorname{card}(D_{i})-\operatorname{WAL}|}\\\
C_{I}=[c\in D\ |\ \operatorname{card}(c)=f_{i}(D)]\end{array}\right.$ (3)

Here, $L_{i}$ stands for the length of each recognition data, for example, in
“object 1, object 2, object 1”, $L_{i}=3$. $\omega_{i}$ stands for the weight
of each recognition data which is $0.1\times L_{i}$.
$\operatorname{card}(D_{i})$ stands for the cardinality of set $D_{i}$, where
$D_{i}$ is the segments previously obtained by segmenting the original array.

This approach is inspired by the pooling layer in CNN, but instead of a Max
Pooling operation, the data smoothing procedure uses a Weighted Average
Pooling operation. By using this data smoothing procedure, we can avoid
unwanted recognition results like broken frames or camera flashes.

Data selection procedure. We apply a data selection procedure that uses the
similar approach that we previously used in the data smoothing procedure,
which is also a Weighted Average Pooling operation. This procedure will select
the recognition result from a frame whose feature intensity is the closest to
the weighted average value of all the candidates (feature intensity refers to
the number of different classes of objects in a particular frame).

Finally, we can output the result that we obtained in the previous steps as
the representative of the whole scene. This particular recognition result will
be used to represent the whole scene it is located in, and through the help of
NLP and other models, this can even be used to output the natural language
interpretation of this video scene.

##  4 Experiments

Due to the lack of similar algorithms and datasets, we could only provide some
preliminary and experimental usage of the 2SDS algorithm111We only did some
preliminary experiments on the accuracy of 2SDS on scene separation (temporal
segmentation) tasks, more detailed experiments are still needed to be
conducted..

We choose YOLOv5s as our image recognition CNN for this experiment, and we
have built an experimental dataset on video object detection using selected
YouTube videos in the YouTube-VOS dataset [8]. Although the YOLOv5s algorithm
is trained on the COCO dataset, this CNN model is still sufficiently usable in
this experiment for it is not the key focus of this experiment.

We are most interested in how 2SDS will perform in scene separation (temporal
segmentation) tasks. We classified the testing videos into 3 classes:
interviews, vibrant, and hybrid.

The interviews are usually straight forward and easier to undergo scene
separation tasks. Vibrant videos are the more difficult ones due to their
fast-moving images and transition effects that might seem deceptive to 2SDS.
The hybrid video sits in between the first two types, they have some features
of the interview videos, as well as features from the vibrant videos, their
difficulty should sit in the middle.

###  4.1 Interview Video Tests

We conducted 3 separate experiments using interview videos (Table 1). The
total amount of scenes in these 3 experiments is $82$. The overall accuracy of
2SDS during these experiments is $90.10\%$. There are 2 cases where we find
the 2SDS algorithm actually over-judged the transition between two scenes.
This is potentially a sensitivity issue posed by the hard coded threshold
during scene separation.

Table 1: Interview video tests results. Experiment No. | Output - Truth | Accuracy
---|---|---
Interview 1 | 25 - 25 | $100.00\%$
Interview 2 | 35 - 29 | $82.86\%$
Interview 3 | 31 - 28 | $90.32\%$

###  4.2 Vibrant Video Tests

We conducted 2 separate experiments using vibrant videos (Table 2). The total
amount of scenes in these 2 experiments is $51$. The overall accuracy of 2SDS
during the two experiments is $54.90\%$. The accuracy in vibrant videos is
substantially lower than interview videos for the 2SDS is unable to separate
two fast-moving scenes effectively. It is important to notice that we used
harsh videos like sport videos and dynamic advertisement videos in this
experiment, so the performance of the 2SDS is expected to be much lower
comparing to the previous experiment. This is the biggest limitation of 2SDS,
but this issue is addressable with future improvements of the algorithm.

Table 2: Vibrant video tests results. Experiment No. | Output - Truth | Accuracy
---|---|---
Vibrant 1 | 9 - 13 | $69.23\%$
Vibrant 2 | 19 - 38 | $50.00\%$

###  4.3 Hybrid Video Tests

We conducted one experiment using a long hybrid video (Table 3). The total
amount of scenes in this experiment is 106. The overall accuracy of 2SDS is
99.06%. Theoretically, the result of this experiment should sit between the
previous two tests, however, an anomaly has arisen most likely due to the lack
of samples. A more detailed experiment should be conducted to further
determine the accuracy of 2SDS on hybrid videos.

Table 3: Hybrid video test result. Experiment No. | Output - Truth | Accuracy
---|---|---
Hybrid 1 | 105 - 106 | $99.06\%$

##  5 Bringing in Spatial Information

Bringing in spatial information and modeling techniques can potentially play a
huge role in video interpretation. Previously difficult and untouchable
problems like continuous gesture recognition and scene recognition are being
cracked using the CNN-based spatio-temporal reasoning model [9] and the 2SDS
algorithm as well.

Our work has only utilised the temporal information in video stream, our
future work can make use of graphs, and spatially model a frame into a graph,
with the objects as the vertices and the spatial relations between the objects
as the edges, like the MST-GNN [10] and the VRD-GCN [11]. Doing this, we can
extract even more information out of a video. For example, a person’s gesture
in a scene can be identified, and the scenes with more significant camera or
object movements (e.g., the vibrant and hybrid video tests) will not cause
significant problem for the algorithm because the spatial relation of the
objects stays the same.

This future work would bring immense potential with the use of spatial
information, which will add a whole other dimension of usable information that
can benefit video analysis with richer semantics and the ability of grouping
fast-moving frames, bringing video interpretation models yet another step
closer to how human perceive visual information.

##  6 Conclusion

Under the context of real-time video stream analysis using temporal
segmentation methods, we devised 2SDS, a temporal segmentation algorithm that
can be used alongside CNNs to complement for the lack of temporal information
handling ability of the CNN-based models. We gave, yet another powerful tool
that CNN models can utilise, the ability to take advantage of the inherent
temporal aspect of videos. Video stream analysis is a completely different
task compared to image recognition, and we are finally seeing some evidence
that we can still use 2D CNNs to interpret video information.

The 2SDS algorithm utilise a refined difference hash value method and a novel
data smoothing and data selection technique to crack the temporal segmentation
problem. Although there are still drawbacks with fast-moving frames in vibrant
videos, the 2SDS algorithm has already done a great job at separating
relatively simple and stationary scenes in videos, and it gets the job done at
a respectful speed, which will allow the 2SDS to get a finer temporal
resolution compared with neural networks.

For future work, some improvements on 2SDS (e.g., adding graphs to model
spatial relations) can potentially boost the algorithm’s performance on fast-
moving scenes and smooth transitions.

###### Acknowledgements.

We would like to dedicate our thank to Dr Zhiguo Long, Dr John Stell, and Dr
Liu Yang, who have been extremely generous and helpful throughout the course
of our work.

## References

  * Stappen et al. [2021] L. Stappen, A. Baird, E. Cambria, B. W. Schuller,  Sentiment analysis and topic recognition in video transcriptions,  IEEE Intell. Syst. 36 (2021) 88–95.
  * Patel et al. [2022] A. S. Patel, R. Vyas, O. P. Vyas, M. Ojha,  A study on video semantics; overview, challenges, and applications,  Multim. Tools Appl. 81 (2022) 6849–6897.
  * Pal et al. [2021] R. Pal, A. A. Sekh, D. P. Dogra, S. Kar, P. P. Roy, D. K. Prasad,  Topic-based video analysis: A survey,  ACM Comput. Surv. 54 (2021) 118:1–118:34.
  * Diba et al. [2017] A. Diba, M. Fayyaz, V. Sharma, A. H. Karami, M. M. Arzani, R. Yousefzadeh, L. V. Gool,  Temporal 3d convnets: New architecture and transfer learning for video classification,  CoRR abs/1711.08200 (2017). arXiv:1711.08200.
  * Feichtenhofer et al. [2019] C. Feichtenhofer, H. Fan, J. Malik, K. He,  Slowfast networks for video recognition,  in: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, IEEE, 2019, pp. 6201–6210.
  * Hochreiter and Schmidhuber [1997] S. Hochreiter, J. Schmidhuber,  Long short-term memory,  Neural Comput. 9 (1997) 1735–1780.
  * Krawetz [2013] N. Krawetz, Kind of like that, 2013\. URL: http://www.hackerfactor.com/blog/?/archives/529-Kind-of-Like-That.html.
  * Xu et al. [2018] N. Xu, L. Yang, Y. Fan, D. Yue, Y. Liang, J. Yang, T. S. Huang,  YouTube-VOS: A large-scale video object segmentation benchmark,  CoRR abs/1809.03327 (2018). arXiv:1809.03327.
  * Köpüklü et al. [2019] O. Köpüklü, F. Herzog, G. Rigoll,  Comparative analysis of CNN-based spatiotemporal reasoning in videos,  CoRR abs/1909.05165 (2019). arXiv:1909.05165.
  * Li et al. [2021] M. Li, S. Chen, Y. Zhao, Y. Zhang, Y. Wang, Q. Tian,  Multiscale spatio-temporal graph neural networks for 3D skeleton-based motion prediction,  IEEE Trans. Image Process. 30 (2021) 7760–7775.
  * Qian et al. [2019] X. Qian, Y. Zhuang, Y. Li, S. Xiao, S. Pu, J. Xiao,  Video relation detection with spatio-temporal graph,  in: L. Amsaleg, B. Huet, M. A. Larson, G. Gravier, H. Hung, C. Ngo, W. T. Ooi (Eds.), Proceedings of the 27th ACM International Conference on Multimedia, MM 2019, Nice, France, October 21-25, 2019, ACM, 2019, pp. 84–93.