# Physics-informed learning fully quantifies uncertainty in seismic structure
and source estimate

Ryoichiro Agata,1∗ Kazuya Shiraishi,1 Gou Fujie1

1Japan Agency for Marine-Earth Science and Technology,
3173-25, Showa-machi, Kanazawa-ku, Yokohama, Kanagawa 2360001, Japan

∗To whom correspondence should be addressed; E-mail<EMAIL_ADDRESS>
> The source estimation of earthquakes and other fault activities in
> seismogenic subduction zones is essential for hazard assessment. Accurate
> and reliable earthquake source estimation (ESE) requires full uncertainty
> quantification (UQ), targeting both the ESE and subsurface seismic velocity
> structure models incorporated in the ESE, which is technically challenging.
> Here, we address this problem by adopting a physics-informed deep learning
> approach that enables such full UQ-based ESE by introducing neural network
> ensembles trained based on both earthquake observation data and the physical
> equation of wavefront propagation. The proposed full UQ-based approach
> enabled the investigation of rupture characteristics of an earthquake in
> southwest Japan from a statistical viewpoint, demonstrating innovations
> toward stronger scientific objectivity in the studies of seismogenic zones
> empowered by physics-informed learning.

## Introduction

Seismogenic subduction zones host large earthquakes, which can cause seismic
and tsunami disasters. A wide range of fault activities occur in subduction
zones, including small- and medium-sized earthquakes, slow earthquakes,
aseismic slips, and inter-seismic inter-plate locking. Quantitatively
analyzing the occurrence of these phenomena through source estimations using
seismic, crustal deformation, and tsunami data (hereafter referred to as
earthquake source estimation; ESE), and understanding the process of
accumulation and release of inter-plate strain are essential for hazard
assessments of earthquakes and tsunamis.

Accurate ESE requires appropriate model assumptions for the subsurface seismic
velocity structure, through which the seismic signals from the source are
transmitted. The estimation of the seismic velocity structure is a highly
nonlinear inverse problem and is conducted using data limited to the Earth’s
surface, leading to large uncertainty in the results. This uncertainty leads
to proposal of numerous velocity structure models that vary significantly
(Fig. 1AB); for example, several different three-dimensional (3D) seismic
velocity structure models [1, 2, 3] have been proposed for the Nankai Trough
region in southwest Japan. ESE is typically conducted by selecting a single-
velocity structure model such as the average one. This is a bold assumption of
zero uncertainty [4]. Moreover, ESE is often conducted by using a model
further simplified by individual scholars and institutions.

Furthermore, ESE has issues related to uncertainty owing to the accuracy and
sparseness of earthquake observation data. For instance, the results of
rupture process estimations from different groups for the same earthquake
often vary extensively; thus, the reliability of scientific findings is
unclear [5]. The ESE of ordinary and slow earthquakes sometimes introduces
additional constraints, such as fixed-fault geometries or depths, to the
inversion problem [6, 7]. However, the reliability of the results reflecting
the effects of these constraints has not been adequately investigated. Efforts
have been made to quantify the uncertainties in ESE [8, 9, 10]. However, the
aforementioned neglect of uncertainties or the simplification of seismic
velocity structural models may lead to bias and the underestimation of
uncertainty in ESE [11, 12, 4] (Fig. 1C). This is because large uncertainty in
a seismic velocity structure model has a non-negligible effect on the
uncertainty in subsequent ESEs, which is mathematically known as uncertainty
propagation [13]. For accurate and reliable ESE, uncertainty quantification
(UQ) of the velocity structure estimation and uncertainty propagation in the
ESE must be properly considered. However, as both the UQ and evaluation of
uncertainty propagation are technically challenging, full UQ-based ESE by
comprehensively considering the above has not yet been realized.

In existing UQ methods for seismic velocity structures, ensemble-based
approaches are adopted based on Bayesian estimation [14, 15, 16, 17, 18] to
cope with inherently strong nonlinearities. In these methods, ordinary mesh-
or grid-based parameterization and partial differential equation (PDE)
solutions are used. The recent remarkable progress in deep learning has led to
the emergence of alternative approaches for solving PDE and PDE-based inverse
problems, represented by physics-informed neural networks (PINNs) [19]. When
analyzing observational and experimental data, PINN incorporates information
from the physical laws described by the PDE. PINN is particularly advantageous
in solving ill-posed inverse problems owing to its inherent regularity [20],
and has already been applied to deterministic seismic tomography [21], which
has recently been extended to ensemble-based estimation [22]. This method has
the advantage of yielding samples that more closely resemble real velocity
structures represented by an infinite-dimensional function, in contrast to
conventional grid-based approaches, which often represent a sample of velocity
structures using low-dimensional grids [14, 15, 16, 17]. The uncertainty
propagation can accurately be accounted for by using such an ensemble of
velocity structures as inputs to the ESE [13, 4, 23] (Fig. 1DE). The PINN-
based UQ of seismic velocity structure models and uncertainty propagation to
the ESE are expected to enable next-generation full UQ-based ESEs by
eliminating the negative effects of unreasonable model selection and
simplification.

Here, we conducted full UQ of hypocenter and seismic velocity structure
estimation for the first time based on new PINN techniques by targeting the Mw
5.9 earthquake that occurred in 2016 in the Nankai Trough region of southwest
Japan. The experts were initially unsure of whether the earthquake was a
plate-boundary earthquake [24, 25]. First, we used the first-arrival travel
time data obtained from refraction surveys conducted near the hypocenter to
conduct UQ for the 2D P-wave velocity structure on the survey line to obtain
an ensemble velocity-structure model. We then conducted the UQ of hypocenter
determination, which is a basic type of ESE, considering the uncertainty
propagation from the estimation of velocity structure using this ensemble.
Moreover, we applied the ensemble to the UQ of the estimation of fault
location using seismic reflection survey data collected in the same area.
Subsequently, we investigated the fault plane wherein the 2016 earthquake
occurred from a statistical viewpoint based on the full-UQ results. Such
analyses demonstrate the potential of full UQ-based ESEs in ensuring stronger
scientific objectivity in studies of seismogenic zones, enabled by the
promising continued development of PINN.

## Results

### Ensemble-based estimation of the P-wave velocity structure

The 2016 Mw 5.9 earthquake occurred off the southeastern coast of Mie
Prefecture [24, 26, 25] in the central part of the Nankai Trough region in
southwest Japan, which is known for historical megathrust earthquakes [27].
This was the largest earthquake in the region since the 1944 Tonankai
earthquake (Mw = 8.0) (Fig. 2A). In the vicinity of the hypocenter, the
seismic refraction data acquired using a tuned airgun array and ocean bottom
seismometers (OBS) are available along the line KI03. We estimated the P-wave
velocity structure and its uncertainty using 14,146 first-arrival travel times
that were manually picked from the refraction data using a PINN-based ensemble
estimation method [22]. This method represents the seismic velocity structure
and travel time function using neural networks (NNs) instead of grids or
meshes. The NN of travel time can be trained for the velocity structure
represented by the NN of velocity through the PINN framework by minimizing the
residual of the Eikonal equation, which can simulate wavefront propagation and
determine travel time; the evaluation was conducted in a straightforward
manner with the help of automatic differentiation of the NN outputs [28, 29].
An ensemble of velocity NNs representing the posterior probability for the
travel time data formulated by Bayes’ theorem, namely, the stochastic property
of the estimation uncertainty, was generated through the combined use of PINN-
based travel time calculations and function-space particle-based variational
inference (ParVI) [30, 31] (see Fig. 2B and Materials and Methods).

From the 256 NNs of velocity obtained through ensemble estimation (Fig. 3A),
we obtained the mean model (Fig. 3B) and standard deviation (Fig. 3C) of the
seismic velocity structure. The obtained mean seismic velocity models clearly
show the north-dipping surface of the subducting oceanic plate and low-
velocity areas corresponding to an accretionary prism and forearc basin,
without the introduction of any prior information. These features are
generally in good agreement with existing seismic structures modeled using
deterministic tomographic methods [24, 26]. The standard deviation of the
ensemble, namely, the uncertainty of the seismic velocities, was generally low
in the area covered by the ray path of the first arrivals. This indicated
spatial variations even within the ray coverage area, suggesting an uneven
distribution of ray paths.

### Full UQ-based hypocenter determination based on the ensemble seismic
velocity structure model

We re-determined the hypocenter of the 2016 earthquake using a full UQ-based
approach, considering the uncertainty propagation from the estimation of the
velocity structure using the ensemble estimation results. We used the P-wave
first-arrival time data from nine nearby seismometers installed in the Dense
Oceanfloor Network system for Earthquakes and Tsunamis (DONET) [32, 33] to
estimate the source location [26]. We used the 2D velocity structure models
obtained in the previous process to create 3D volumes, assuming that the
spatial variation of the seismic velocity structure in the direction
perpendicular to subduction is negligible in the analysis domain [26] (Fig.
S1). Prior to the determination of the hypocenter, we trained the PINN using
3D travel time data for each seismic velocity structure for the source and
receiver point samples in the targeted regions. Subsequently, ensemble
hypocenter determination was conducted with fast travel time calculations
using pre-trained PINNs combined with a ParVI-based Bayesian inversion method
[34]. The uncertainty propagation from velocity structure estimation was
considered by integrating 256 velocity models into this hypocenter
determination method using the Bayesian multi-model approach, namely, a
framework to introduce multiple candidate models in Bayesian inversion [35],
which has later been applied to ESEs [13, 4, 23]. A total of 128 hypocenter
parameter instances (particles) were used to represent the stochastic property
of the uncertainty (see Materials and Methods and Fig. S2).

The mean locations of the hypocenter obtained using the full UQ-based method
were at 136.41∘E and 33.36∘N, and at a depth of 10.81 km. The standard
deviations in the horizontal direction along the cross section, vertical
direction, and the other horizontal direction were 0.50, 0.68, and 0.38 km,
respectively. These values were larger than those estimated using a non-full
UQ-based approach, wherein the uncertainty of the seismic velocity structure
model is not considered (see Materials and Methods) (Fig. 3D). The depth
obtained without the full UQ-based approach is reasonably close to that
obtained by a previous model with similar analysis conditions [26] (Fig. S3).

### Comparison of the locations of the hypocenter and fault under uncertainty

In the target region, two structural interfaces exist near the depth of the
plate boundary: a megasplay fault and the top of the oceanic crust. At depth,
a plate-boundary earthquake is believed to have ruptured the top surface of
the oceanic crust. However, at shallow depths, a plate-boundary earthquake
possibly does not rupture the top of the oceanic crust, but ruptures the
megasplay fault that branches off from it [36, 37, 38]. Here, we consider both
structural interfaces as candidates for the “coseismic plate boundary [39],”
namely, the fault plane ruptured by the plate-boundary earthquake in this
region. We compared our hypocenter with these structural interfaces by
considering the uncertainty in their depth related to the seismic velocity
structure. Thus, we re-examined whether this earthquake ruptured a coseismic
plate boundary and further investigated whether it was located at the
megasplay fault, at the top of the oceanic crust, or elsewhere. A seismic
section in the two-way travel times (TWTs) is available for survey line TK5,
which mostly overlaps with line KI03 [26]. We manually selected reflectors
that likely corresponded to a megasplay fault and the top of the oceanic crust
(Fig. S4). We obtained seismic sections at depth with uncertainty by
converting them to TWT using 256 velocity structure models (see Materials and
Methods). The standard deviations of the depths calculated based on the
ensemble were 0.3-0.4 km for both reflectors.

Combining all our obtained results enabled the comparative analysis of the
locations of the hypocenter and fault under UQ for the first time. The mean
location of the megasplay fault agreed well with that of hypocenter depth, and
the 2-$\sigma$ intervals largely overlapped between the hypocenter depth and
two fault locations (Fig. 4A). When an earthquake hypocenter is located at
either of the two candidate faults or between them, we assumed that the event
occurred at the coseismic plate boundary. By leveraging the estimated
stochastic properties of the uncertainties, we calculated the percentage for
this case using a one-dimensional normal distribution fitted to the ensemble
(see Fig. S5 and Materials and Methods). The probability was calculated as
35%. We conducted the same analysis for the case without a full UQ-based
approach (i.e., without considering the uncertainty of the seismic velocity
structure model). In this case, the candidate fault locations were represented
by single lines (Fig. 4B). As previously indicated, the hypocenter estimate is
biased toward being shallower with the underestimation of uncertainty. The
line of the megasplay fault barely overlaps the 2-$\sigma$ interval of the
hypocenter, whereas that of the top of the oceanic crust is distant from the
interval. In this case, the corresponding probability is only 8%.

## Discussion

In the full UQ-based approach, which accounted for the uncertainty in the
seismic velocity structure model, the probability of the occurrence of an
event at the coseismic plate boundary was 35%. This is significantly higher
than 8%, which is the probability for the case without the use of a full UQ-
based approach. Previously, the determination of whether the earthquake was at
a coseismic plate boundary was based on the latter type of results. This
requires an expert’s evaluation to consider implicit uncertainties that were
not quantitatively considered in the analysis; for example, uncertainty of the
velocity structure estimation, effect of the simplification of the velocity
structure, and uncertainty of the fault location. In contrast, the proposed
full UQ-based method allows us to draw insights solely based on the quantified
uncertainty in the analysis results, indicating the potential to achieve a
more quantitative and objective characterization of various fault activities
in seismogenic zones.

In the results of the full UQ-based approach, the mean depths of the
hypocenters and the two fault locations were similar; thus, the shapes of the
probability distributions were similar. Therefore, we expect that improvements
in the accuracy and precision of hypocenter determination, fault location
estimation, and velocity structure estimation will further increase the
probability. For this purpose, the use of the first-arrival time data from
other DONET stations distributed over a wider region will provide additional
constraints. This requires the estimation of an ensemble 3D seismic velocity
structure model as the effect of material heterogeneity in the trench-parallel
direction cannot be neglected as the target region becomes wider [40]. Since
2019, the Japan Meteorological Agency is expected to issue “Nankai Trough
Earthquake Extra Information” to alert the public when the possibility of the
occurrence of a megaquake in the Nankai Trough is assessed to be higher than
usual [41]. When an earthquake of a certain magnitude occurred in the vicinity
of the assumed focal region, whether or not the earthquake occurred at the
coseismic plate boundary was the basis for additional information. DONET
provides real-time seismic observations at the ocean bottom of the Nankai
Trough region as dense arrays. Additionally, rich survey data on the
subsurface structures in this region have been compiled in the last two
decades [42]. Combined with such abundant data, the proposed full UQ-based
method is expected to provide quantitative and objective information that is
highly useful for both the scientific community and public.

Additionally, we calculated the probabilities of the occurrence of an
earthquake at the megasplay fault and the top of the oceanic crust, whose
values were 5 and 4%, respectively. These low probabilities were largely
attributed to the large uncertainty in the estimated hypocenter depth for the
thin fault plane. These probabilities also suggest the difficultly in
indicating the part of the coseimic plate boundary at which the event occurred
based on depth estimates for the dataset we used. Whether or not a large
earthquake ruptures a megasplay fault is an important question as seafloor
deformation owing to the rupture of megasplay faults may cause larger tsunamis
[43]. Although the observation data of the 1944 Tonankai earthquake did not
possess sufficient resolution to determine which fault was ruptured, the
tsunami inversion result [37] and some interpretations of nearby subsurface
structural information [36, 38] provide supporting evidence of the slip along
the megasplay fault. However, no conclusions have been reached. Further
improvements in the precision and accuracy of the involved estimations may
provide key information to conclude these arguments.

NNs can represent seismic velocity structures in an infinite-dimensional
space. This representation agrees better with the nature of the real
subsurface structure than with the unnatural low-dimensional discretization
introduced in conventional ensemble-based velocity structure estimation
methods with grid- or mesh-based parameterization. Such an infinite-
dimensional representation of the seismic velocity structure enables the
accurate and stable subsequent calculations of wavefront propagation and
conversions of fault depth, leading to an accurate evaluation of uncertainty
propagation. Introducing NN representation of the physical properties into UQ
may have great potential for resolving inverse problems involving significant
and complex uncertainty propagation. Moreover, the determination of the
hypocenter based on the Bayesian multi-model framework required only
approximately 11 min of computation, even though the travel time for 128
hypocenter ensembles was evaluated for 256 seismic velocity structures in 500
steps. Such fast computation was achieved by travel time calculations that
leveraged the pre-trained PINNs, which require only forward propagation
calculations of NNs. The required computation time for the same analysis based
on conventional numerical methods [44] is at least two orders of magnitude
higher than that in our approach as a new numerical calculation is required
for each instance of the source points and seismic velocity structures. Thus,
PINN enabled the fast calculation of the full UQ-based determination of the
hypocenter; this is particularly beneficial when quick dissemination of
information regarding earthquakes to the public is essential, as in the case
of Nankai Trough. The benefits of deep learning approaches are striking
examples of how scientific machine learning [45] methods, such as PINN, can
lead to new scientific breakthroughs.

The concept of a full UQ-based approach can be extended in a straightforward
manner from the determination of hypocenter to ESE in general, including the
rupture process and fault slip inversions for coseismic, aseismic, and
interseismic faulting activities using seismic waveforms, crustal deformation,
and tsunami data. Further development of PINN-based approaches to solve PDEs
for elastic waves [46] and elastic deformation owing to dislocation sources
[47] will help the expansion of full UQ-based strategies to other classes of
ESE problems. Such improvements are expected to be achieved with the continued
development of PINN and relevant scientific machine learning methods as key
actors in the “AI for Science” paradigm [48].

## References

  * 1. A. Nakanishi, et al., Geology and Tectonics of Subduction Zones: A Tribute to Gaku Kimura 534, 69 (2018).
  * 2. D. Bassett, et al., Journal of Geophysical Research: Solid Earth 127, e2022JB024992 (2022).
  * 3. Y. Yamamoto, S. Yada, K. Ariyoshi, T. Hori, N. Takahashi, Progress in Earth and Planetary Science 9, 1 (2022).
  * 4. R. Agata, A. Kasahara, Y. Yagi, Geophysical Journal International 225, 1392 (2021).
  * 5. P. M. Mai, et al., Seismological Research Letters 87, 690 (2016).
  * 6. T. Ragon, A. Sladen, M. Simons, Geophysical Journal International 214, 1174 (2018).
  * 7. K. Sagae, H. Nakahara, T. Nishimura, K. Imanishi, Geophysical Journal International 225, 775 (2021).
  * 8. A. Tarantola, B. Valette, et al., Journal of geophysics 50, 159 (1982).
  * 9. J. Fukuda, K. M. Johnson, Bulletin of the Seismological Society of America 98, 1128 (2008).
  * 10. S. Minson, et al., Geophysical Journal International 198, 922 (2014).
  * 11. Y. Yagi, Y. Fukahata, Geophysical Journal International 186, 711 (2011).
  * 12. Z. Duputel, P. S. Agram, M. Simons, S. E. Minson, J. L. Beck, Geophysical Journal International 197, 464 (2014).
  * 13. A. Gesret, N. Desassis, M. Noble, T. Romary, C. Maisons, Geophysical Journal International 200, 52 (2015).
  * 14. T. Bodin, M. Sambridge, Geophysical Journal International 178, 1411 (2009).
  * 15. N. Piana Agostinetti, G. Giacomuzzi, A. Malinverno, Geophysical Journal International 201, 1598 (2015).
  * 16. R. Hawkins, M. Sambridge, Geophysical Journal International 203, 972 (2015).
  * 17. T. Ryberg, C. Haberland, Geophysical Journal International 212, 1645 (2018).
  * 18. X. Zhang, A. Curtis, Journal of Geophysical Research: Solid Earth 125, e2019JB018589 (2020).
  * 19. M. Raissi, P. Perdikaris, G. E. Karniadakis, Journal of Computational physics 378, 686 (2019).
  * 20. G. E. Karniadakis, et al., Nature Reviews Physics 3, 422 (2021).
  * 21. U. B. Waheed, T. Alkhalifah, E. Haghighat, C. Song, J. Virieux, arXiv preprint arXiv:2104.01588 (2021).
  * 22. R. Agata, K. Shiraishi, G. Fujie, IEEE Transactions on Geoscience and Remote Sensing (2023).
  * 23. R. Agata, et al., Journal of Geophysical Research: Solid Earth 127, e2021JB023712 (2022).
  * 24. L. Wallace, et al., Journal of Geophysical Research: Solid Earth 121, 8338 (2016).
  * 25. S. Takemura, T. Kimura, T. Saito, H. Kubo, K. Shiomi, Earth, Planets and Space 70, 1 (2018).
  * 26. M. Nakano, et al., Progress in Earth and Planetary Science 5, 1 (2018).
  * 27. M. Ando, Tectonophysics 27, 119 (1975).
  * 28. J. D. Smith, K. Azizzadenesheli, Z. E. Ross, IEEE Transactions on Geoscience and Remote Sensing 59, 10685 (2021).
  * 29. U. B. Waheed, E. Haghighat, T. Alkhalifah, C. Song, Q. Hao, Computers & Geosciences 155, 104833 (2021).
  * 30. Q. Liu, D. Wang, Advances in neural information processing systems 29 (2016).
  * 31. Z. Wang, T. Ren, J. Zhu, B. Zhang, International Conference on Learning Representations (2019).
  * 32. Y. Kaneda, et al., Seafloor observatories (Springer, 2015), pp. 643–662.
  * 33. S. Aoi, et al., Earth, Planets and Space 72, 1 (2020).
  * 34. J. D. Smith, Z. E. Ross, K. Azizzadenesheli, J. B. Muir, Geophysical Journal International 228, 698 (2022).
  * 35. A. E. Raftery, D. Madigan, J. A. Hoeting, Journal of the American Statistical Association 92, 179 (1997).
  * 36. J.-O. Park, T. Tsuru, S. Kodaira, P. R. Cummins, Y. Kaneda, Science 297, 1157 (2002).
  * 37. T. Baba, P. R. Cummins, T. Hori, Y. Kaneda, Tectonophysics 426, 119 (2006).
  * 38. G. Moore, et al., Science 318, 1128 (2007).
  * 39. N. Bangs, et al., Earth and Planetary Science Letters 284, 44 (2009).
  * 40. K. Shiraishi, et al., Geochemistry, Geophysics, Geosystems 20, 2252 (2019).
  * 41. Cabinet Office, White Paper Disaster Management in Japan (2019). http://www.bousai.go.jp/kaigirep/hakusho/pdf/R1_hakusho_english.pdf, Accessed XX XXXX, 2023.
  * 42. Y. Nakamura, et al., Geophysical Research Letters 49, e2022GL098180 (2022).
  * 43. Y. Fukao, Journal of Geophysical Research: Solid Earth 84, 2303 (1979).
  * 44. J. A. Sethian, proceedings of the National Academy of Sciences 93, 1591 (1996).
  * 45. N. Baker, et al., Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence (2019).
  * 46. P. Ren, et al., Computer Physics Communications 295, 109010 (2024).
  * 47. T. Okazaki, T. Ito, K. Hirahara, N. Ueda, Nature Communications 13 (2022).
  * 48. H. Wang, et al., Nature 620, 47 (2023).
  * 49. Japan Agency for Marine-Earth Science and Technology (JAMSTEC), JAMSTEC Seismic Survey Database (2004). https://doi.org/doi:10.17596/0002069, Accessed 20 September, 2023.
  * 50. Japan Agency for Marine-Earth Science and Technology (JAMSTEC), Data and Sample Research System for Whole Cruise Information in JAMSTEC (2016). http://www.godac.jamstec.go.jp/darwin/, Accessed 1 May, 2023.

## Acknowledgments

This research was supported by JSPS KAKENHI Grant 21K14024 in Grant-in-Aid for
Early-Career Scientists. Seismic data from JAMSTEC (Japan Agency for Marine-
Earth Science and Technology) Seismic Survey Database [49] were used. The
seismic survey in the line KI03 performed in 2011 is part of a project
entitled “Research on Evaluation of Linkage between Tokai/Tonankai/Nankai
Earthquakes,” funded by the Japanese Ministry of Education, Culture, Sports,
Science, and Technology (MEXT). The bathymetric data in the Data and Sample
Research System for Whole Cruise Information (DARWIN) in JAMSTEC [50] were
used. Computational resources of the Earth Simulator 4 provided by JAMSTEC was
used. The picked P-wave arrival records of seismic survey data were provided
by Dr. Ayako Nakanishi. The picked P-wave arrival records of the 2016 Mw 5.9
earthquake off the southeastern coast of Mie Prefecture from DONET
seismometers were provided by Dr. Masaru Nakano. Comments from Dr. Daisuke
Sato and Dr. Hori Takane were valuable for improving the analysis settings and
the discussion in the manuscript.

## Supplementary materials

Materials and Methods
Figs. S1 to S7
References (51-86)

Figure 1:  A schematic illustration of the comparison between full UQ-based
ESE and conventional (without full UQ) approaches. (A) Example of seismic data
acquisition for the estimation of the velocity structure using a marine
airgun-OBS seismic refraction survey. The dependence of the seismic ray path
on the velocity structure leads to strong nonlinearity in the estimation
problem. Other types of data, such as passive seismic records and ambient
noise, can also be used to estimate the velocity structure. (B) In
conventional non-full UQ-based methods, the seismic velocity structure models
estimated by different individual analyzers using the same or similar seismic
datasets may significantly differ owing to the inherent uncertainty in
tomographic analyses. Moreover, the models may be further simplified by the
analyzers. (C) Schematic illustration of hypocenter determination using the
arrival time data from ocean-bottom seismic monitoring networks (white
triangles) as an example of ESE using the obtained seismic velocity structure
models. The orange cross marks schematically correspond to the confidence
intervals of hypocenter determination. Different seismic velocity structure
models may provide different hypocenter estimates that are biased by the
underestimation of uncertainty. In such cases, the extent to which the
scientific findings can be trusted is unclear. (D) In the full UQ-based
method, the uncertainty in velocity structure estimation was quantified by
estimating the model as an ensemble. (E) By incorporating such a velocity
structure ensemble as an input for hypocenter determination, we can accurately
account for uncertainty propagation and eliminate the negative effects of
unreasonable model selection and simplification.  Figure 2:  (A) A map of the
study area. The yellow star represents the epicenter of the 2016 Mw 5.9
earthquake off the southeastern coast of Mie Prefecture estimated in this
study. The circles and triangles represent the locations of OBSs, which were
installed in the survey line KI03 and DONET observation nodes, respectively.
The gray rectangle represents the approximate focal region of the 1944
Tonankai earthquake. (B) A schematic illustration for the NNs of velocity and
travel time trained by both the Eikonal equation based on the PINN framework
using automatic differentiation and Bayesian posterior probability based on
travel time data. See Materials and Methods for the definitions of the
mathematical expressions.  Figure 3:  (A) 256 velocity models along the line
KI03 represented by NN ensemble members trained by a combination of PINN and
ParVI. In (A) and (B), the gray shaded area represents the region with
standard deviations larger than 0.6. In (A), (B), and (D), the white dashed
line represents the plate boundary model proposed by Nakanishi et al. [1]. (B)
The mean velocity model calculated based on the estimated ensemble. (C)
Standard deviation calculated based on the estimated ensemble. The gray
dotted-dashed line denotes the bottom of the ray coverage for the mean model.
(D) Comparison between the determined hypocenter locations with and without
full UQ plotted in the region with the dashed rectangle in (B). The cross
marks correspond to 2-$\sigma$ intervals.  Figure 4: A comparison of the
estimated hypocenter and fault locations at depth in the region indicated by
the dashed rectangle in Fig. 3B. (A) Those obtained using the full UQ-based
method, considering the uncertainty in the seismic velocity structure model.
The dashed lines and crosses correspond to the 2-$\sigma$ intervals of the
fault locations and hypocenter, respectively. The background image is the
seismic section at depth converted from TWT using the mean velocity model. (B)
The results obtained using an ordinary non-full UQ-based method without
considering the uncertainty in the seismic velocity structure model. In this
case, the two structural interfaces are denoted by single lines.


# Tutorial Recommendation for Livestream Videos
using Discourse-Level Consistency and Ontology-Based Filtering

Amir Pouran Ben Veyseh, 1
Franck Dernoncourt2, and Thien Huu Nguyen1

###### Abstract

Streaming videos is one of the methods for creators to share their creative
works with their audience. In these videos, the streamer share how they
achieve their final objective by using various tools in one or several
programs for creative projects. To this end, the steps required to achieve the
final goal can be discussed. As such, these videos could provide substantial
educational content that can be used to learn how to employ the tools used by
the streamer. However, one of the drawbacks is that the streamer might not
provide enough details for every step. Therefore, for the learners, it might
be difficult to catch up with all the steps. In order to alleviate this issue,
one solution is to link the streaming videos with the relevant tutorial
available for the tools used in the streaming video. More specifically, a
system can analyze the content of the live streaming video and recommend the
most relevant tutorials. Since the existing document recommendation models
cannot handle this situation, in this work, we present a novel dataset and
model for the task of tutorial recommendation for live-streamed videos. We
conduct extensive analyses on the proposed dataset and models, revealing the
challenging nature of this task.

## Introduction

Streaming platforms, such as Twitch, Behance, and YouTube, are effective tools
for creators that equip them with the facility of directly reaching out to
their audience to share their creative content. For instance, on
Behance111http://www.behance.net, creators can share their works on visual
projects, such as illustrations and designs, while employing visual content
editing tools, such as Photoshop and Illustrator. In these videos, the
streamer discusses the details of actions required to fulfill the objective of
the creative task (e.g., designing a logo). Depending on the tools and the
format that the streamer chooses to present their work, the streaming video
can serve as educational content to learn how to use the tools used in the
video. For instance, the streamer might review how to draw the sketches for
designing a fantasy character or they might discuss the various methods for
selecting an object in an image. Thus, these videos can help the audience to
learn the nuances of the tools. However, the edit actions might be discussed
in different details. For instance, to add some shapes to an image, the
streamer might briefly mention the name of the brush employed to perform this
action or he/she might explain the various methods available for this action.
As such, a streaming video on itself might lack all details necessary to learn
an edit action. One way to fill this gap is to accompany the streaming videos
with tutorials in which the details of the actions are presented (see Figure
1). Linking a streaming video with the relevant tutorials helps the audience
to learn all aspects of the tools employed in the video.

Given a live streaming video, we aim to find the relevant tutorial to it. One
solution to this question is to employ the existing document recommendation
tools (Guan et al. 2010; Kim et al. 2016; Xu et al. 2020). However, one
limitation to this approach is that the existing recommendation tools are
trained on formal documents, e.g., books or news articles. As such, directly
employing these models for the recommendation for the transcripts of the live-
streamed videos is not optimal. In particular, unlike formal texts, in the
transcripts of a video, there might be incomplete sentences, incorrect words
due to the ASR (automatic speech recognition) errors, or repeated sentences.
These differences require domain-specific models that are designed to handle
the challenges of the domain of video transcript. Moreover, another limitation
for employing existing document recommendation resources is that there is no
evaluation benchmark for this domain, making it more difficult to compare the
performance of different models.

To address these shortcomings, in this work, we present the first large-scale
tutorial linking dataset for the videos streamed on the Behance platform. More
specifically, 47,403 sentences from the transcripts of 24 live-stream videos
are annotated. In total, 4,126 sentences are annotated with 3 different
tutorials for Photoshop. In addition, we also conduct extensive experiments on
the proposed dataset. In particular, we first present the performance of an
unsupervised model in which the similarity of the video transcript and the
tutorial content is employed for the recommendation. Next, we employ the
annotated data, to provide recommendations for the sentences in the video
transcripts. Our analysis shows the challenging nature of this domain.

Figure 1: Part of the transcript of the live-streamed video ”Adobe Photoshop
Compositing for Beginners”

## Related work

This task could be modeled either as text classification (Zhang, Zhao, and
LeCun 2015) or text similarity (Shahmirzadi, Lugowski, and Younge 2019). For
text classification, the goal is to classify the input text into one of the
pre-defined categories. Here, the categories might be defined as the available
tutorials. The textual content of the tutorial describes the label of the
category. For text similarity, the degree to which the tutorial content is
similar to the video transcript is employed to find the most related tutorial
for a given transcript. However, these solutions suffer from critical
weaknesses which renders them inapplicable or inefficient for our task. First,
most of the existing systems for text classification require manually labeled
data. However, for our task, there is no human supervision available for
training. As such, these methods might not be employed for this task. Second,
both text classification and text similarity methods are evaluated on short
documents (a few sentences) with formal language (i.e., a news article).
However, in our task, the documents might be very large (e.g., transcripts of
several hours of videos) and noisy (due to the automatic transcripts). This
difference in a domain makes the majority of the solutions inapplicable to our
task. Last but not least, the existing similarity-based methods cannot
incorporate background knowledge (e.g., an ontology of concepts or keywords in
the domain of the videos). Moreover, they ignore the discourse-level
consistency between the two texts to compute the similarity score.

Figure 2: Annotation tool for phrase-level tutorial recommendation

## Data Annotation

### Data Collection

To train and evaluate the model we annotate data from the transcripts of the
videos streamed on the Behance platform. The recordings are spilled by
specialists and creators to share/discuss their inventive projects. As such,
verbal substance from the speakers (in English) is imperative for video
understanding. Whereas the recordings have introductory subjects, their
substance is impromptu, thus the streamer might cut sentences, examine
numerous themes, and utilize casual expressions. The recordings have an
average length of 48 minutes. To get the verbal substance of the streamed
recordings, we utilize the Microsoft ASR tool. In addition up to, 24
recordings (whose main editing tool is Photoshop) are transcribed. In total,
47,403 sentences are present in the transcribed videos to be annotated by
human annotators.

### Data Annotation

To annotate data, we hire expert annotators in Upwork who have experience in
using Photoshop and also have experience in data annotation tasks. In total,
three annotators are hired for this task. Every video transcript is assigned
to the three annotators to link them to relevant tutorials. More specifically,
for the tutorial pool, we employ two types of tutorials available for
Photoshop:

  * •

Using: In this type of tutorial the usage of the specific tools is discussed.
These tutorials are helpful to discuss the details of the tools that are used
by the streamer. An example of this type of tutorial is presented in Figure
LABEL:fig:using_example. In this work, we employ 290 Using tutorials.

  * •

How-To: In this type of tutorial, the process to achieve a final edit action
is discussed. For instance, how to design a portrait could be an edit action
discussed in a tutorial. For this type of tutorial, multiple edit tools might
be employed. An example of this tutorial is presented in Figure
LABEL:fig:howto_example. In this work, we employ 126 How-To tutorials.

Figure 3: A Using tutorial. The tutorial discuss the application of layers.
Figure 4: A How-To tutorial. The tutorial discuss the process of creating a
composite image.

For every transcript, the annotator selects the sentences that might refer to
a tutorial and provide at least three tutorial related to the selected
sentence. In total, 4,126 sentences with 3 tutorials are annotated.

## Model

We employ two types of models: (1) An unsupervised model: In this model, the
content of the transcript and the tutorial are employed for linking. This
method provides a tutorial for the entire transcript; (2) A supervised model:
In this model, we employ the annotated data to train a sentence classification
model in which the model selects one of the available tutorials for a given
sentence. If the sentence is not referring to a tutorial, $None$ is selected
as the sentence label. The rest of this section provides details of these two
models.

### Unsupervised Model Overview

Our proposed model has the following novelties:

  * •

A novel unsupervised deep learning model for the task of tutorial
recommendation using video transcripts

  * •

A novel approach to employing domain-specific knowledge for filtering the
target tutorials

  * •

A novel method for summarizing the input transcript into a smaller version
without using any human-curated labels

  * •

A novel method for computing the similarity between the transcript and the
tutorial text based on discourse-level consistency.

### Details

Formally, the input to the system is the transcript
$D=[w^{D}_{1},w^{D}_{2},\ldots,w^{D}_{n}]$, consisting of $n$ words, and a
pool of tutorial textual content, i.e., $P=[T_{1},T_{2},\ldots,T_{|P|}]$ where
$T_{i}=[w^{T}_{1},w^{T}_{2},\ldots,w^{T}_{m}]$ is the textual content of
$i$-th tutorial consisting of $m$ words. The goal is to return the most
relevant tutorial from the pool $P$, i.e., $T_{gold}$. To create a system for
this task, we propose multiple components. Specifically, the proposed system
consists of three major components:

  * •

Filtering Tutorials: In this component, the goal is to remove the tutorials
that are very unlikely to be relevant to a given transcript. In this
component, domain-specific knowledge is employed to assess the relevancy
between the given transcript and the tutorial textual content.

  * •

Transcript summarizing: In this component the objective is to summarize the
given transcript such that only the most important information that could be
helpful for finding the tutorials are preserved. This component employs an
unsupervised deep learning method.

  * •

Tutorial Ranking: Finally, using the summary of the given transcript and the
filtered list of relevant tutorials, this component employs various metrics to
sort the tutorials based on their relevance and similarity to the transcript.

The rest of this section elaborates more on the details of each of these
components.

#### Filtering Tutorials

The pool of tutorials might contain several irrelevant candidates which could
hinder an efficient ranking method. As such, it is necessary to first filter
the pool $P$ such that the unlikely candidates are removed. Formally, the
objective is to define the filter function $\Phi$ such that:

$\begin{split}|\Phi(P)|&<|P|\\\ \forall\text{ }T_{i}\in\Phi(P)\text{,
}T_{j}\in P-\Phi(P)&\quad R(T_{i})<R(T_{j})\end{split}$ (1)

where $|\cdot|$ is the size of the pool and $R(x)$ is the rank of $x$ in the
sorted pool based on the relevance of the candidates to the given transcript
$D$. To define the function $\Phi$, in a novel method, we propose to employ
two types of criteria:

Domain-specific Knowledge: The ontology of names in the domains of interest
(i.e., in our work we use Image Editing Tools such as Adobe Photoshop as the
domain of interest) could be employed as the domain-specific knowledge.
Specifically, in this work, we propose to employ the available list of tool
names for Adobe Photoshop as domain-specific information. To employ this
knowledge in function $\Phi$, we suggest first find the tool names mentioned
in the transcript $D$, i.e., $TN=\\{tn_{1},tn_{2},\ldots,tn_{|TN|}\\}$ where
$tn_{i}$ is a tool name in transcript $D$. Next, we define the filter function
$\Phi_{DK}$ to filter out those tutorials in the pool $P$ which don’t mention
one of the tool names of $TN$:

$\Phi_{DK}(T_{i})=\left\\{\begin{array}[]{ll}True&\text{if }\exists\text{
}tn_{j}\in TN\text{ s.t. }tn_{j}\in T_{i}\\\
False&\text{otherwise}\end{array}\right.$ (2)

String Similarity: In addition to the domain-specific knowledge, we seek to
incorporate string similarity of the tutorial textual content with the
transcript $D$ in the filtering process. More specifically, we first compute
the string similarity of the transcript $D$ with tutorial $T_{i}\in P$ using
normalized point-wise mutual information (PMI):

$\begin{split}Sim(D,T_{i})&=\sum_{w^{D}_{i}\in D}\sum_{w^{T}_{j}\in
T_{i}}\frac{PMI(W^{D}_{i},W^{T}_{j})}{n*m}\\\
PMI(w_{i},w_{j})&=\log\frac{COUNT(w_{i},w_{j})}{COUNT(w_{i})*COUNT(w_{j})}\end{split}$
(3)

where $n$ and $m$ are the number of words in the transcript $D$ and tutorial
$T_{i}$, $COUNT(W_{i})$ is the number of occurrences of the word $w_{i}$ in
all transcripts in the training data and $COUNT(w_{i},w_{j})$ is the number of
occurrences of both words $w_{i}$ and $w_{j}$ in a transcript in the training
data. Next, we define the following filter function based on similarity:

$\Phi_{sim}(T_{i})=\left\\{\begin{array}[]{ll}True&\text{if
}Sim(D,T_{i})>\delta\\\ False&\text{otherwise}\end{array}\right.$ (4)

where $\delta$ is a hyper-parameter to be tuned using development data.
Finally, using the two aforementioned filter functions $\Phi_{DK}$ and
$\Phi_{sim}$, we define the final function $\Phi$:

$\Phi(T_{i})=\left\\{\begin{array}[]{ll}True&\text{if }\Phi_{DK}(T_{i})\text{
and }\Phi_{sim}(T_{i})\\\ False&\text{otherwise}\end{array}\right.$ (5)

#### Transcript Summarizer

The next component in our system is the transcript summarizer whose goal is to
shorten the transcript $D$ such that only the distinctive information is
preserved and the redundant portions, which might not be helpful to identify
the most related tutorial, are excluded. To this end, we proposed to train a
deep learning model to consume the input document $D$ and generate the shorter
document $D^{\prime}$, such that $|D^{\prime}|<|D|$. Specifically, a
transformer-based language model, i.e., BERT, is trained to encode the words
of the document $D$, i.e., $e_{i}=BERT(w_{i})$ for all $w_{i}n\in D$222Note
that for words consisting of multiple word-pieces, we represent them using the
average of their word-piece embeddings obtained from BERT model. In our
experiments, we use the last hidden states of the BERT model to obtain the
embedding vector $e_{i}$. Next, a feed-forward network is utilized to estimate
the likelihood of the word $w_{i}$ to be included in the shorter document
$D^{\prime}$:

$P(w_{i}|D)=FF(w_{i})=\sigma(W_{1}*(W_{2}*e_{i}+b_{2})+b_{1})$ (6)

where $\sigma$ is the sigmoid activation function, $W_{1}$ and $W_{2}$ are the
weight matrices and $b_{1}$ and $b_{2}$ are bias. To train the BERT model and
the feed-forward layer, since the available resources for this task do not
have any labeled data, we resort to unsupervised learning. Concretely, two
criteria are employed to train the model:

Distinctiveness: The shorter documents $D^{\prime}_{i}$ and $D^{\prime}_{j}$
obtained from the original documents $D_{i}$ and $D_{j}$ should be as
different as possible. Moreover, those portions of the documents $D_{i}$ and
$D_{j}$ that do not appear in the summary should have the least differences
(in other words, as these portions are not informative (e.g., they are
chitchat), there will be more similarity between them). To fulfill this
requirement, we employ the following loss function:

$\mathcal{L}_{dist}=\alpha*(\sigma(H^{\prime}_{i})\odot\sigma(H^{\prime}_{j}))-\beta(\sigma(H^{\prime\prime}_{i})\odot\sigma(H^{\prime\prime}_{j}))$
(7)

where $\sigma$ is the softmax function, $\alpha$ and $\beta$ are the trade-off
parameters and $\odot$ is the hadamard product. The vectors $H^{\prime}_{i}$,
$H^{\prime}_{j}$, $H^{\prime\prime}_{i}$ and $H^{\prime\prime}_{j}$ are the
vector representation for the summaries $D^{\prime}_{i}$, $D^{\prime}_{j}$,
and the portions of the documents $D_{i}$ and $D_{j}$ not included in the
summaries, i.e., $D^{\prime\prime}_{i}$ and $D^{\prime\prime}_{j}$,
respectively. To obtain these vector representations, we use max-pooling on
the multiplication of the embeeding vectors and the feed-forward network. For
instance, $H^{\prime}_{i}$ is obtained as follows:

$\begin{split}H^{\prime}_{i}&=MAX\\_POOL(h_{1},h_{2},\ldots,h_{n})\\\
h_{k}&=e_{k}*FF(e_{k})\end{split}$ (8)

where $n$ is the number of words in the document $D$. Note that for the
representations of $H^{\prime\prime}_{i}$ and $H^{\prime\prime}_{j}$, we
replace $FF(e_{k})$ with $(1-FF(e_{k}))$ in equation 8.

Information Retaining: The process to summarize document $D$ into the smaller
version $D^{\prime}$ is supposed to keep the most important information in $D$
intact. As such, it is expected that the information available in both
documents $D$ and $D^{\prime}$ to have considerable overlap. This criterion
can be achieved via increasing the mutual information (MI) between the
representations of the $D$ and $D^{\prime}$. To fulfill this goal, we exploit
contrastive learning. In particular, a discriminator is trained to distinguish
positive samples from negative ones where a positive sample is the
concatenation of the representation of the original document $D_{i}$ and its
summary $D^{\prime}_{i}$, i.e., $pos=[H_{i}:H^{\prime}_{i}]$, and the negative
sample is the concatenation of the representation of the document $D_{i}$ with
the summary of the randomly selected document $D_{j}$, i.e.,
$neg=[H_{i}:H^{\prime}_{j}]$. Formally, the following loss function is
employed to increase the mutual information:

$\mathcal{L}_{IR}=-(log(\Psi[H_{i}:H^{\prime}_{i}])+log(1-\Psi([H_{i}:H^{\prime}_{j}])))$
(9)

where $\Psi$ is the discriminator. The sum of two losses, i.e.,
$\mathcal{L}_{dist}$ and $\mathcal{L}_{IR}$ is employed as the final loss
function to train the transcript summarizer component:

$\mathcal{L}=\alpha\mathcal{L}_{IR}+\beta\mathcal{L}_{dist}$ (10)

where $\alpha$ and $\beta$ are trade-off parameters. At inference time, to
summarize the input transcript, we employ the output of the feed-forward
network $FF$ and every word $w_{i}$ whose corresponding value from $FF$ is
higher than a threshold is selected in $D^{\prime}$.

#### Ranking

Finally, given the short document $D^{\prime}$ and the filtered list of
tutorials, i.e., $P^{\prime}=\Phi(P)$, the final component aims to sort the
tutorials based on their relevance to the given transcript. To this end, we
employ two types of scores:

  * •

String Similarity: The string similarity between every tutorial $T_{i}\in
P^{\prime}$ with the summary $D^{\prime}$ is evaluated using fasttext
(Bojanowski et al. 2016) to obtain its string similarity score:
$Score_{str}=FastText(T_{i},D^{\prime})$

  * •

Discourse Similarity: The objective of this score is to measure how likely is
the tutorial $T_{i}\in P^{\prime}$ to complement the summary $D^{\prime}$ of
the input transcript. To this end, first, we train the text classification
model $C$ that takes the concatenation of the first and second half of the
transcripts $D_{i}$, i.e., $D_{i,1}$ and $D_{i,2}$, as a positive sample and
the concatenation of $D_{i,1}$ and $D_{j,2}$, where $j$ is selected randomly,
as the negative samples333words of the documents are encoded using GloVe
embedding and the max-pooled embeddings of them are fed into classifier $C$.
The model is trained using a similar loss function as equation 9. Next, the
trained classifier $C$ is employed in the ranking component by feeding the
concatenation of the tutorial $T_{i}$ and summary $D^{\prime}$ to the model
and its output (i.e., the likelihood of the input to complement each other),
is employed as the discourse-level score: $Score_{disc}=C([T_{i}:D^{\prime}])$

Finally, to sort the documents, we compute the sum of the two aforementioned
scores:

$Score=Score_{sim}+Score_{disc}$ (11)

The sorted list of the tutorials is returned as the final output of the
system.

### Supervised Model Overview

To train the model for supervised tutorial linking, we model the task as a
sentence classification problem. More specifically, given the words of the
sentence $S$ and the title of the tutorial $T$, the input sequence
$[CLS],w^{S}_{1},w^{S}_{2},\ldots,w^{S}_{n},[SEP],w^{T}_{1},w^{T}_{2},\ldots,w^{T}_{m},[SEP]$
is fed into the BERTbase model. Note that for every tutorial in the tutorial
pool a separate input is prepared. Next, the $[CLS]$ vector representation is
fed into a two-layer feed-forward layer to predict a binary label, e.g., 1/0.
The label is 1 if the tutorial is linked with the given sentence in the
annotated dataset. Note that we add a special tutorial title $None$ for
sentences without any linked tutorial.

### Experiments

Model | Hit@3 | Hit@5
---|---|---
String Similarity Sorting | 40% | 50%
Keyword Sorting | 35% | 45%
Information-based Sorting | 40% | 50%
Ours | 55% | 65%
Table 1: Performance of the unsupervised models Model | F1
---|---
BERT | 35%
GloVe | 28%
Table 2: Performance of the supervised models

To evaluate the proposed system, we manually annotated transcripts from the
Behance corpus. These are the transcripts of videos streamed on Behance.net
and the streamers are all using Adobe Photoshop in their streaming video. We
use Adobe Photoshop tutorials (more than 200 tutorials for using or how-to) as
the initial pool of tutorials. To provide more insight into the performance of
the proposed system, we compare it with the following systems:

  * •

String Similarity Sorting: In this system, the string similarity of the input
transcript and the tutorials are measured and it is employed to sort all
tutorials in the pool.

  * •

Keyword Sorting: In this system, the tutorials are sorted based on the number
of tool names that they have in common with the input transcript.

  * •

Information-based Sorting: In this system, the same PMI-based scoring that is
employed in our filtering component is employed to sort all tutorials.

For the supervised model, in addition to the proposed BERT model, we also
compare the performance of the model when the words of the sentence $S$ and
the title of the tutorial $T$ are presented by GloVe embedding. The max-pooled
representation of the input is fed into the feed-forward network.

To evaluate the models, we use Hit@3 and Hit@5 evaluation metrics. The results
are shown in Table 1. This table clearly shows that the proposed model
significantly outperforms the systems, indicating its effectiveness for this
task.

The results for the supervised model are presented in Table 2. This table
shows that the contextualized representation of the video transcript and the
title are more effective than the GloVe embedding. Nonetheless, both models
suffer from low performance which indicates more research is required.

## References

  * Bojanowski et al. (2016) Bojanowski, P.; Grave, E.; Joulin, A.; and Mikolov, T. 2016.  Enriching Word Vectors with Subword Information.  _arXiv preprint arXiv:1607.04606_.
  * Guan et al. (2010) Guan, Z.; Wang, C.; Bu, J.; Chen, C.; Yang, K.; Cai, D.; and He, X. 2010.  Document recommendation in social tagging services.  In _Proceedings of the 19th international conference on World wide web_ , 391–400.
  * Kim et al. (2016) Kim, D.; Park, C.; Oh, J.; Lee, S.; and Yu, H. 2016.  Convolutional matrix factorization for document context-aware recommendation.  In _Proceedings of the 10th ACM conference on recommender systems_ , 233–240.
  * Shahmirzadi, Lugowski, and Younge (2019) Shahmirzadi, O.; Lugowski, A.; and Younge, K. 2019.  Text similarity in vector space models: a comparative study.  In _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_ , 659–666. IEEE.
  * Xu et al. (2020) Xu, X.; Hassan Awadallah, A.; T. Dumais, S.; Omar, F.; Popp, B.; Rounthwaite, R.; and Jahanbakhsh, F. 2020.  Understanding user behavior for document recommendation.  In _Proceedings of The Web Conference 2020_ , 3012–3018.
  * Zhang, Zhao, and LeCun (2015) Zhang, X.; Zhao, J.; and LeCun, Y. 2015.  Character-level convolutional networks for text classification.  _arXiv preprint arXiv:1509.01626_.

## Appendices

## Tutorials

A list of Using and How-To tutorials employed during the annotation are shown
in Figures 5 and 6. In our work, in total, 290 and 126 Using and How-To
tutorials are employed, respectively.

## Case Study

In order to provide more insight into the performance of the proposed
supervised model, we show the predicted tutorials for the sentence “I’m gonna
work the composition of this and then work on cleaning up the drawing a little
bit” in the paragraph “Right there. I get it. OK, I get it. I will do this
real quick as the last thing. Change this brush. Might get a little bit right
there. Something like that. Anyway. I don’t know. So this is what we did for
warmups. This guy in this lady. Will leave it at that. But now we will jump
back into ours. A party planning here. They actually took a little hit longer
than I want to be out about an hour and a half hour and 1520 minutes left. I’m
gonna work on the composition of this and then work on cleaning up the drawing
a little bit. Will pull this up. I think I still have it. No, it’s not in
here. Just to kind of recap. Z. Well, let’s see. Let’s see if I can find this
cleaning up the drawing ”. The results are shown in Figure 7, 8 and 9.

Figure 5: A sample list of Using tutorials Figure 6: A sample list of HowTo
tutorials Figure 7: A screenshot of the tutorial page for the sentence shown
in Case Study section. Figure 8: A screenshot of the tutorial page for the
sentence shown in Case Study section. Figure 9: A screenshot of the tutorial
page for the sentence shown in Case Study section.