# Physically-based Path Tracer using WebGPU and OpenPBR

Simon Stucki  0009-0006-1172-9165 ZHAW Zurich University of Applied
SciencesWinterthurSwitzerland<EMAIL_ADDRESS>and  Philipp Ackermann
0000-0003-4607-7620 ZHAW Zurich University of Applied
SciencesWinterthurSwitzerland<EMAIL_ADDRESS>
###### Abstract.

This work presents a web-based, open-source path tracer for rendering
physically-based 3D scenes using WebGPU and the OpenPBR surface shading model.
While rasterization has been the dominant real-time rendering technique on the
web since WebGL’s introduction in 2011, it struggles with global illumination.
This necessitates more complex techniques, often relying on pregenerated
artifacts to attain the desired level of visual fidelity. Path tracing
inherently addresses these limitations but at the cost of increased rendering
time. Our work focuses on industrial applications where highly customizable
products are common and real-time performance is not critical. We leverage
WebGPU to implement path tracing on the web, integrating the OpenPBR standard
for physically-based material representation. The result is a near real-time
path tracer capable of rendering high-fidelity 3D scenes directly in web
browsers, eliminating the need for pregenerated assets. Our implementation
demonstrates the potential of WebGPU for advanced rendering techniques and
opens new possibilities for web-based 3D visualization in industrial
applications.

web path tracer, physically-based rendering, WebGPU, OpenPBR

††ccs: Information systems Web applications††ccs: Computing methodologies Ray
tracing††ccs: Software and its engineering Software creation and
management††ccs: Computing methodologies Reflectance modeling

Figure 1. Path traced renderings of pushbutton CAD models. From left to right:
side view with metallic reflection, rear view exhibiting red color bleeding,
rear view with ambient occlusion, and pushbutton mounted on a metallic
surface, including the reflection of the Stanford bunny model (Turk and Levoy,
1994) positioned in the scene.

Four path traced renderings of CAD models showing metallic reflection, color
bleeding, ambient occlusion, and shadows.

##  1\. Introduction

Rasterization is a rendering technique that projects 3D scene geometry onto a
2D plane. Historically, the technique has been widely adopted in real-time
rendering due to its efficiency. However, rasterization has limitations in
achieving photorealism. One of the main limitations is the lack of support for
global illumination. These phenomena can be observed in objects like mirrors
or metallic surfaces. Effects such as refraction, reflection, and ambient
occlusion require additional techniques.

Various methods have been developed to address these limitations, including
pre-baked shadow, environment (Greene, 1986) and light maps; screen space
reflections (SSR) (Stachowiak, 2015); screen space ambient occlusion (SSAO)
(Bavoil and Sainz, 2008); and screen space directional occlusion (SSDO)
(Ritschel et al., 2009).

These approaches induce complexity, may need to be computed at the assembly
level, and can be computationally expensive. An alternative rendering
technique resembles reality more closely and inherently alleviates these
limitations: ray tracing.

###  1.1. Ray Tracing

Ray tracing is a powerful technique for rendering photorealistic scenes
including global illumination. Historically, ray tracing has been mainly used
in offline rendering due to its computational complexity. The theory behind
ray tracing techniques is well established (Whitted, 2020).

Kajiya defined the rendering equation in 1986, coining the term path tracing.
Generally, path tracing can be seen as an integration of the rendering
equation, which is defined as:

$I(x,x^{\prime})=g(x,x^{\prime})[\epsilon(x,x^{\prime})+\int_{S}p(x,x^{\prime},x^{\prime\prime})I(x^{\prime},x^{\prime\prime})dx^{\prime\prime}]$

where $I(x,x^{\prime})$ is the radiance from point $x$ to point $x^{\prime}$.
$x$ for example being the camera position and $x^{\prime}$ the intersected
object. $g(x,x^{\prime})$ is a geometry term determining how much light is
transmitted. This depends on the distance and possibly occlusions such as
transparent surfaces. $\epsilon(x,x^{\prime})$ is the emitted radiance,
generally used for light sources. The integral term is taken over $S$ = $\cup
S_{i}$ which is the union of all surfaces. $p(x,x^{\prime},x^{\prime\prime})$
is the bidirectional reflection function, which describes how light from all
possible directions $x^{\prime\prime}$ is reflected at the surface. (Kajiya,
1986)

Ray tracing techniques focus on optimizing the rendering equation. Research in
the 1990s focused on efficient light transport techniques such as
bidirectional light transport and Metropolis light transport (Veach, 1998).
Concurrently, production offline renderers like Blue Moon Rendering Tools
(BMRT) were developed (Gritz and Hahn, 1996).

Recent hardware advancements, including dedicated ray tracing acceleration
(Dally et al., 2021), have made real-time ray tracing feasible, sparking
increased research. Notable developments include reservoir-based spatio-
temporal importance resampling (ReSTIR) (Bitterli et al., 2020) and subsequent
improvements (Ouyang et al., 2021; Lin et al., 2022).

##  2\. Use Cases

E-commerce represents a key use case for product renderings. Traditional
catalogues struggle with highly configurable products. Computer graphics
addresses this challenge through product configurators for virtual assembly
and visualization.

Leveraging existing CAD models, prevalent in mechanical engineering and
product design, for end-user applications offers a significant advantage.
These models contain geometric and material information, which eliminates the
need for redundant 3D models for marketing purposes. One example is EAO, which
manufactures highly customizable industrial pushbuttons and operator panels as
visualized in Figure 1. Due to the nature of the product, the number of
possible assemblies grows almost exponentially with the number of components.
To facilitate the configuration process, a web-based configurator is optimal
because it can be used on a large variety of devices without having to install
additional software. Given the company’s priority for photorealistic
renderings of the assemblies, using ray tracing techniques is a compelling
choice.

Pre-rendering all product configurations is theoretically possible for a
finite number of configurations, but computationally expensive. If this is not
feasible, real-time rendering is an alternative. For real-time rendering, one
option is remote rendering (Shi and Hsu, 2015), which employs a server to
render the scene and stream the visualization to the browser. The main
drawbacks of this approach are network latency, reliance on network stability,
as well as operational cost for the server infrastructure, which frequently
requires dedicated GPUs for rendering. Another option is client-side
rendering. Most frequently, rasterization approaches are used for web-based
renderings. However, due to the limitations in global illumination effects,
the need for a real-time, client-side ray tracing solution for the web becomes
apparent when considering the use case.

CAD models require pre-processing as described in Figure 2. This step involves
surface triangulation, potential further refinement using algorithms used to
generate level of detail (LOD) artifacts (Luebke, 2003), and measures to
protect intellectual property (IP) rights by removing proprietary data. While
numerous CAD formats include material information, it is often unsuitable for
rendering. To address this, a material mapping can be defined, translating CAD
materials to a suitable representation for the rendering pipeline.

Figure 2. A two-step pre-processing stage is employed for offline as well as
real-time rendering pipelines.

Diagram showing the steps in data pre-processing. The process begins with
original CAD Data, which is then converted into a STEP file. This file is then
further processed and converted into an OBJ file.

An offline rendering pipeline generates static images of the assembly, which
are then displayed in the browser. This means that all possible assemblies
need to be rendered and stored upfront. As the number of components increases,
the number of possible combinations grows exponentially, which can lead to
large amounts of storage and processing power being required, as shown in
Figure 3.

Figure 3. In an offline rendering setup, the number of images grows
exponentially. The number of images increases further when offering 360°
viewing.

Diagram showing offline rendering pipeline. It starts with OBJ files, which
are then assembled into a meta object. This information is then used to
generate images for each viewing angle.

This is the main benefit of using a real-time rendering pipeline, as
illustrated in Figure 4. The rendering is done in the browser, which means
that the server only needs to provide the geometry and material information in
an exchange format such as glTF. This approach is more flexible and can be
used for a larger number of configurations. The amount of data to be stored
and processed offline grows linearly with the number of components,
independent of the number of possible configurations.

Figure 4. A real-time rendering pipeline solely relies on having an adequate
model for each component of the assembly. It does not require pre-rendering
every possible configuration.

Diagram showing real-time rendering pipeline. It starts with OBJ files, which
are then optimized into glTF files for transport. The browser then uses these
files to assemble meta information and render the scene using a path tracer.

##  3\. Prior Work

There are a variety of path tracers available for the web, most of which are
based on WebGL. The first experiments using WebGL for path tracing were
implemented as early as 2010. One such example is the Google experiment
demonstrating a Cornell Box (Goral et al., 1984) with basic primitive shapes
such as spheres and planes (Wallace, 2011). Since then, multiple open-source
implementations for the web have been created. Some of the most widely known
open-source web-based path tracers include:

  * •

three-gpu-pathtracer (Johnson, 2024).

  * •

Three.js PathTracer (Loftis, 2024).

  * •

dspbr-pt (Sdorra and Häußler, 2022).

WebGPU is a new web standard which, unlike WebGL, is no longer based on
OpenGL. It is designed to be more efficient and flexible than its predecessor.
Flexibility is mainly exhibited through its support for general-purpose GPU
(GPGPU) computations by design. Common 3D engines based on rasterization,
including Babylon.js (Babylon.js, 2022), PlayCanvas (Valigursky, 2023),
Three.js (Three.js, 2024), and Unity (Craven et al., 2023), have officially
announced support for or started working on WebGPU.

Various applications of WebGPU have been investigated in recent years.
Examples include Dynamical.JS, a framework for visualizing graphs (Dotson,
2022); RenderCore, a research-oriented rendering engine (Bohak et al., 2024);
and demonstrations of how to use WebGPU for client-side data aggregation
(Kimmersdorfer et al., 2023).

Research on the performance comparison between existing 3D engines and WebGPU
engines has been conducted as part of the development of FusionRender, which
concluded that measurable performance gains can be achieved by using WebGPU,
but only when effectively leveraging its novel design principles (Bi et al.,
2024). Similar findings have emerged from other independent studies,
demonstrating that WebGPU can be faster than WebGL (Usta, 2024; Fransson and
Hermansson, 2023; Chickerur et al., 2024).

In light of these findings, WebGPU presents a transformative opportunity. It
is particularly well-suited for the development of a new real-time path
tracing library for the web.

##  4\. Physically Based Rendering

Defining the geometry is one part of the equation. Another major part is
defining materials. Materials determine how a surface interacts with light.
The core idea of physically based rendering (PBR) is to simulate the
interaction of light using physical models. Instead of fine-tuning parameters
for a specific look and feel and having to adjust based on desired lighting
situations, PBR aims to define a material that behaves consistently in
different lighting situations. One prominent example of a PBR system is pbrt,
an open-source renderer with an associated literate programming book (Pharr et
al., 2023).

Based on these principles, different material shading models have been
developed. MaterialX is an open standard for representing materials using a
node-based system. Originally developed by Lucasfilm in 2012, it has since
been adopted by the Academy Software Foundation as an open standard. While not
strictly limited to PBR, the standard provides a wide range of features for
describing physically based materials (Harrysson et al., 2019).

OpenPBR, also hosted by the Academy Software Foundation as an open standard,
differs from MaterialX by providing an uber shader approach rather than a
node-based system. The uber shader approach is defined by a fixed set of
inputs which can be adjusted but does not support custom node graphs like
those in MaterialX. OpenPBR combines aspects of Autodesk Standard Surface and
Adobe Standard Material. Its parameters allow for configuring metalness,
glossy-diffuse, subsurface, coat, fuzz, emission, and more. To date, the
standard has a reference implementation in MaterialX. (Andersson et al., 2024)

The emphasis on high-quality PBR materials is advantageous for the industrial
use cases described in this paper. Furthermore, the goal of interoperability
within the broader ecosystem strengthens the case for using OpenPBR in our
implementation.

##  5\. Results

The work presented in this paper introduces a web-based path tracer that
utilizes WebGPU for near real-time rendering. The renderer supports loading
glTF models and allows for the configuration of materials using OpenPBR
parameters.

The renderer is available as an open-source project with extensive
documentation at https://www.github.com/StuckiSimon/strahl. The library is
published on the npm registry as strahl.

###  5.1. Implementation

Figure 5 illustrates the procedure of the path tracer. Scene preparation is
performed on the CPU. This setup needs to be done once per visualization.
Subsequent sampling of the scene is carried out repeatedly on the GPU, which
constitutes the most computationally intensive part of the process.

Figure 5. Path Tracer Pipeline, distinguishing GPU and CPU tasks. The stages
of the compute pipeline and render pipeline are executed sequentially.

Diagram showing the path tracer pipeline. The pipeline starts with the CPU
setup, which consists of loading the glTF model, preparing the BVH and setting
up the pipeline. Then the GPU starts the compute pipeline by generating rays,
tracing the ray and finally rendering the output in the render pipeline.

####  5.1.1. CPU Setup

The CPU processing begins by loading the glTF model. As an acceleration
structure for ray intersection tests, the renderer employs a bounding volume
hierarchy (BVH) based on axis-aligned bounding boxes (AABB). The core concept
is to group adjacent objects in a bounding volume and structure the hierarchy
so that all child elements of a node are contained in the bounding volume of
the parent node. This hierarchical structure allows for early rejection of
branches not intersected by the ray, reducing the number of intersection tests
required per ray cast from $O(n)$ to $O(log(n))$. Once the BVH is constructed,
the scene is prepared for rendering and the pipeline is set up.

####  5.1.2. Ray Generator

The ray generator is responsible for casting rays into the scene according to
the view projection. Based on the use case, the generator uses perspective
projection. The path tracer employs a backward ray tracing approach, tracing
rays from the camera into the scene. To mitigate aliasing effects, rays for
the same pixel are slightly varied across multiple samples. Parallelization is
achieved by computing adjacent pixels concurrently across multiple GPU cores.

To sample distributions, the path tracer selects random values using a random
number generator (RNG). Several suitable pseudorandom number generators have
been considered, including Mersenne Twister (Matsumoto and Nishimura, 1998)
and Xorshifts such as the method described by Marsaglia (Marsaglia, 2003). Our
renderer uses a permuted congruential generator (PCG) as the RNG due to its
good performance and quality (O’Neill, 2014).

####  5.1.3. Path Tracer

Afterwards, the core component of the path tracer recursively traces the rays.
The procedure checks for intersections using the BVH and calculates the
radiance contribution of the intersected objects based on the OpenPBR material
parameters. The OpenPBR standard employs an uber shader approach, which
provides a balance between flexibility and performance. The OpenPBR support is
focused on the most relevant features for industrial use cases, permitting
realistic rendering of metallic, glossy, and diffuse surfaces.

Importance sampling is used to reduce the number of samples required for
accurate results by prioritizing sample directions that contribute more
significantly to the final image. For highly metallic surfaces, sampling is
steered towards the reflection direction, while for diffuse surfaces, sampling
is steered towards the surface normal.

The path tracer utilizes Russian roulette for probabilistic path termination,
selectively discarding paths with low expected radiance contribution. The
samples are then averaged to compute the final pixel color.

####  5.1.4. Render Pipeline

The output of the path tracing compute shader is a texture, which is then
passed to a rasterizer. The rasterizer renders the texture to the canvas using
a fullscreen quad consisting of two triangles. Additionally, it applies tone
mapping based on the Khronos PBR neutral tone mapper (Lalish, 2024).

After each sampling iteration, the current result is visualized, resulting in
a progressive rendering approach.

###  5.2. Performance

Real-time performance is a key aspect of web-based applications. The renderer
can visualize complex models in near real-time, requiring approximately 10 ms
per sample on suitable hardware, taking approximately two seconds to render an
image with 200 samples. The number of samples needed for a high-quality image
depends on the scene complexity; effects such as reflections require more
samples to converge.

As indicated by Table 1, CPU performance for BVH construction varies
significantly based on the model’s complexity. Table 2 shows that GPU
performance for path tracing is more stable across different model
complexities and constitutes the most computationally expensive part of the
process.

##  6\. Discussion

The defined approach provides a viable solution to address challenges
encountered in industrial applications. The ability to render complex
assemblies in real-time without the need for pregenerated artifacts
facilitates the use of CAD models with manifold assembly configurations and
customer-specific materials. Product configurators using WebGPU can present
product variety in high visual fidelity while enabling an interactive end-user
experience with arbitrary viewing angles.

Additionally, operating costs can be reduced by eliminating the need to host a
rendering pipeline or remote rendering service.

###  6.1. Real-Time Rendering

Path tracing is computationally expensive and requires multiple samples per
pixel to achieve accurate results. Consequently, the sampling process is
noticeable during interactions with the scene. One technique to improve
perceived interaction quality is to overlay the rendering with a rasterization
preview during interactions.

To reduce the number of samples required for higher quality renderings,
techniques like neural radiance caching (NRC) (Müller et al., 2021) or ReSTIR
(Bitterli et al., 2020) could be employed in future work. Additionally,
denoising algorithms applied as post-processing steps could enhance the
quality of the results. Options include blockwise multi-order feature
regression (BMFR) (Koskela et al., 2019) and Open Image Denoise (OIDN) (Áfra,
2024).

Further improvements can be achieved by aligning glTF PBR with OpenPBR,
focusing on real-time rendering. This alignment could reduce the computational
resources required for surface shading.

###  6.2. Production Readiness of WebGPU

WebGPU’s production readiness remains limited due to incomplete support in
Safari and Firefox. Both browsers have announced plans for implementation
(Wyrzykowski, 2023; Mozilla, 2023). Thanks to the extensive conformance test
suite (W3C group for GPU web standards, 2024), it is more likely that the
different implementations will be compatible with each other.

The main browser supporting WebGPU to date is Chrome, which shipped WebGPU for
general use on desktops in May 2023 (Google, 2023). Since January 2024, WebGPU
has also been supported on modern Android devices (Google, 2024). This makes
it straightforward to use WebGPU on most modern devices, with the notable
exception of Apple iOS and iPadOS devices.

###  6.3. Future Work

Future work may include supporting additional OpenPBR features, implementing
denoising techniques, integrating rendering techniques such as depth of field
and volumetric path tracing, enhancing performance, and comparing the renderer
with alternative web-based path tracers. The open-source nature of the project
facilitates extension and serves as inspiration for other initiatives.

###### Acknowledgements.

We thank the reviewers for their valuable feedback and suggestions. We also
thank EAO and Intelliact for permitting the use of production CAD models,
enabling the verification of a real-world use case.

## References

  * (1)
  * Áfra (2024) Attila T. Áfra. 2024.  Intel® Open Image Denoise.  https://www.openimagedenoise.org.
  * Andersson et al. (2024) Zap Andersson, Paul Edmondson, Julien Guertault, Adrien Herubel, Alan King, Peter Kutz, Andréa Machizaud, Jamie Portsmouth, Frédéric Servant, and Jonathan Stone. 2024.  _OpenPBR Surface Specification_.  Technical Report. Academy Software Foundation (ASWF).  https://academysoftwarefoundation.github.io/OpenPBR/
  * Babylon.js (2022) Babylon.js. 2022.  _WebGPU Support_.  Retrieved July 18, 2024 from https://github.com/BabylonJS/Babylon.js/issues/6443
  * Bavoil and Sainz (2008) Louis Bavoil and Miguel Sainz. 2008.  Screen space ambient occlusion.  _NVIDIA developer information: http://developer.nvidia.com_ 6, 2 (2008).
  * Bi et al. (2024) Weichen Bi, Yun Ma, Yudong Han, Yifan Chen, Deyu Tian, and Jiaqi Du. 2024.  FusionRender: Harnessing WebGPU’s Power for Enhanced Graphics Performance on Web Browsers. In _Proceedings of the ACM on Web Conference 2024_ (Singapore, Singapore) _(WWW ’24)_. Association for Computing Machinery, New York, NY, USA, 2890–2901.  https://doi.org/10.1145/3589334.3645395
  * Bitterli et al. (2020) Benedikt Bitterli, Chris Wyman, Matt Pharr, Peter Shirley, Aaron Lefohn, and Wojciech Jarosz. 2020.  Spatiotemporal reservoir resampling for real-time ray tracing with dynamic direct lighting.  _ACM Trans. Graph._ 39, 4, Article 148 (aug 2020), 17 pages.  https://doi.org/10.1145/3386569.3392481
  * Bohak et al. (2024) Ciril Bohak, Dmytro Kovalskyi, Sergey Linev, Alja Mrak Tadel, Sebastien Strban, Matevž Tadel, and Avi Yagil. 2024.  RenderCore – a new WebGPU-based rendering engine for Root-eve.  _EPJ Web of Conferences_ 295 (2024), 03035.  https://doi.org/10.1051/epjconf/202429503035
  * Chickerur et al. (2024) Satyadhyan Chickerur, Sankalp Balannavar, Pranali Hongekar, Aditi Prerna, and Soumya Jituri. 2024.  WebGL vs. WebGPU: A Performance Analysis for Web 3.0.  _Procedia Computer Science_ 233 (2024), 919–928.  https://doi.org/10.1016/j.procs.2024.03.281 5th International Conference on Innovative Data Communication Technologies and Application (ICIDCA 2024).
  * Craven et al. (2023) Ben Craven, Matthew Buscemi, and Anthony Bowker. 2023.  _Web runtime updates are here: Take your browser to the next level_.  Retrieved July 18, 2024 from https://blog.unity.com/engine-platform/web-runtime-updates-enhance-browser-experience
  * Dally et al. (2021) William J. Dally, Stephen W. Keckler, and David B. Kirk. 2021.  Evolution of the Graphics Processing Unit (GPU).  _IEEE Micro_ 41, 6 (2021), 42–51.  https://doi.org/10.1109/MM.2021.3113475
  * Dotson (2022) Robert L. Dotson. 2022.  _Dynamical.JS: A composable framework for online exploratory visualization of arbitrarily-complex multivariate networks_.  Master’s thesis. Harvard University Division of Continuing Education.
  * Fransson and Hermansson (2023) Emil Fransson and Jonatan Hermansson. 2023.  Performance comparison of WebGPU and WebGL in the Godot game engine.  (2023).  https://urn.kb.se/resolve?urn=urn:nbn:se:bth-24706
  * Google (2023) Google. 2023.  _Chrome Platform Status: WebGPU_.  Retrieved July 18, 2024 from https://chromestatus.com/feature/6213121689518080
  * Google (2024) Google. 2024.  _Chrome Platform Status: WebGPU on Android_.  Retrieved July 18, 2024 from https://chromestatus.com/feature/5119617865613312
  * Goral et al. (1984) Cindy M. Goral, Kenneth E. Torrance, Donald P. Greenberg, and Bennett Battaile. 1984.  Modeling the interaction of light between diffuse surfaces.  _ACM SIGGRAPH computer graphics_ 18, 3 (1984), 213–222.
  * Greene (1986) Ned Greene. 1986.  Environment mapping and other applications of world projections.  _IEEE computer graphics and Applications_ 6, 11 (1986), 21–29.
  * Gritz and Hahn (1996) Larry Gritz and James K. Hahn. 1996.  BMRT: A Global Illumination Implementation of the RenderMan Standard.  _Journal of Graphics Tools_ 1, 3 (1996), 29–47.  https://doi.org/10.1080/10867651.1996.10487462
  * Harrysson et al. (2019) Niklas Harrysson, Doug Smythe, and Jonathan Stone. 2019.  _MaterialX Physically-Based Shading Nodes Introduction_.  Retrieved July 18, 2024 from https://materialx.org/assets/MaterialX.v1.37REV2.PBRSpec.pdf
  * Johnson (2024) Garrett Johnson. 2024.  _three-gpu-pathtracer_.  Retrieved July 18, 2024 from https://github.com/gkjohnson/three-gpu-pathtracer
  * Kajiya (1986) James T. Kajiya. 1986.  The rendering equation. In _Proceedings of the 13th annual conference on Computer graphics and interactive techniques_. 143–150.
  * Kimmersdorfer et al. (2023) Gerald Kimmersdorfer, Dominik Wolf, and Manuela Waldner. 2023.  WebGPU for Scalable Client-Side Aggregate Visualization.  _Proceedings of Eurographics - The European Association for Computer Graphics_ (2023), 1–3.
  * Koskela et al. (2019) Matias Koskela, Kalle Immonen, Markku Mäkitalo, Alessandro Foi, Timo Viitanen, Pekka Jääskeläinen, Heikki Kultala, and Jarmo Takala. 2019.  Blockwise Multi-Order Feature Regression for Real-Time Path-Tracing Reconstruction.  _ACM Trans. Graph._ 38, 5, Article 138 (jun 2019), 14 pages.  https://doi.org/10.1145/3269978
  * Lalish (2024) Emmett Lalish. 2024.  _Khronos PBR Neutral Tone Mapper_.  Retrieved July 18, 2024 from https://github.com/KhronosGroup/ToneMapping
  * Lin et al. (2022) Daqi Lin, Markus Kettunen, Benedikt Bitterli, Jacopo Pantaleoni, Cem Yuksel, and Chris Wyman. 2022.  Generalized resampled importance sampling: Foundations of restir.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–23.
  * Loftis (2024) Erich Loftis. 2024.  _THREE.js-PathTracing-Renderer_.  Retrieved July 18, 2024 from https://github.com/erichlof/THREE.js-PathTracing-Renderer
  * Luebke (2003) David Luebke. 2003.  _Level of detail for 3D graphics_.  Morgan Kaufmann.
  * Marsaglia (2003) George Marsaglia. 2003.  Xorshift rngs.  _Journal of Statistical software_ 8 (2003), 1–6.
  * Matsumoto and Nishimura (1998) Makoto Matsumoto and Takuji Nishimura. 1998.  Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator.  _ACM Trans. Model. Comput. Simul._ 8, 1 (jan 1998), 3–30.  https://doi.org/10.1145/272991.272995
  * Mozilla (2023) Mozilla. 2023.  _Mozilla Platform GFX WebGPU_.  Retrieved July 18, 2024 from https://wiki.mozilla.org/Platform/GFX/WebGPU
  * Müller et al. (2021) Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller. 2021.  Real-time neural radiance caching for path tracing.  _ACM Trans. Graph._ 40, 4, Article 36 (jul 2021), 16 pages.  https://doi.org/10.1145/3450626.3459812
  * O’Neill (2014) Melissa E. O’Neill. 2014.  PCG: A family of simple fast space-efficient statistically good algorithms for random number generation.  _ACM Trans. Math. Software_ (2014).
  * Ouyang et al. (2021) Y. Ouyang, S. Liu, M. Kettunen, M. Pharr, and J. Pantaleoni. 2021.  ReSTIR GI: Path Resampling for Real-Time Path Tracing.  _Computer Graphics Forum_ 40, 8 (2021), 17–29.  https://doi.org/10.1111/cgf.14378
  * Pharr et al. (2023) Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2023.  _Physically Based Rendering: From Theory to Implementation_.  The MIT Press.
  * Ritschel et al. (2009) Tobias Ritschel, Thorsten Grosch, and Hans-Peter Seidel. 2009.  Approximating dynamic global illumination in image space. In _Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games_ (Boston, Massachusetts) _(I3D ’09)_. Association for Computing Machinery, New York, NY, USA, 75–82.  https://doi.org/10.1145/1507149.1507161
  * Sdorra and Häußler (2022) Bastian Sdorra and Tobias Häußler. 2022.  _dspbr-pt_.  Retrieved July 18, 2024 from https://github.com/DassaultSystemes-Technology/dspbr-pt
  * Shi and Hsu (2015) Shu Shi and Cheng-Hsin Hsu. 2015.  A Survey of Interactive Remote Rendering Systems.  _ACM Comput. Surv._ 47, 4, Article 57 (may 2015), 29 pages.  https://doi.org/10.1145/2719921
  * Stachowiak (2015) Tomasz Stachowiak. 2015.  Advances in real time rendering, part I, Stochastic Screen-Space Reflections. In _ACM SIGGRAPH 2015 Courses_ (Los Angeles, California) _(SIGGRAPH ’15)_. Association for Computing Machinery, New York, NY, USA, Article 1.  https://doi.org/10.1145/2776880.2787701
  * Three.js (2024) Three.js. 2024.  _Three.js WebGPURenderer.js_.  Retrieved July 18, 2024 from https://github.com/mrdoob/three.js/blob/ffef51075d125234079aaa494f9da1066f3d3e77/src/renderers/webgpu/WebGPURenderer.js
  * Turk and Levoy (1994) Greg Turk and Marc Levoy. 1994.  Zippered polygon meshes from range images. In _Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’94)_. Association for Computing Machinery, New York, NY, USA, 311–318.  https://doi.org/10.1145/192161.192241
  * Usta (2024) Ziya Usta. 2024.  WebGPU: A new Graphic API for 3D WebGIS Applications.  _The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ XLVIII-4/W9-2024 (2024), 377–382.  https://doi.org/10.5194/isprs-archives-XLVIII-4-W9-2024-377-2024
  * Valigursky (2023) Martin Valigursky. 2023.  _Initial WebGPU support lands in PlayCanvas Engine 1.62_.  Retrieved July 18, 2024 from https://blog.playcanvas.com/initial-webgpu-support-lands-in-playcanvas-engine-1-62/
  * Veach (1998) Eric Veach. 1998.  _Robust Monte Carlo Methods for Light Transport Simulation_.  Stanford University, Department of Computer Science.
  * W3C group for GPU web standards (2024) W3C group for GPU web standards. 2024.  _WebGPU Conformance Test Suite_.  Retrieved July 18, 2024 from https://github.com/gpuweb/cts
  * Wallace (2011) Evan Wallace. 2011.  _WebGL Path Tracing_.  Retrieved July 18, 2024 from https://experiments.withgoogle.com/webgl-path-tracing
  * Whitted (2020) Turner Whitted. 2020.  Origins of Global Illumination.  _IEEE Computer Graphics and Applications_ 40, 1 (2020), 20–27.  https://doi.org/10.1109/MCG.2019.2957688
  * Wyrzykowski (2023) Mike Wyrzykowski. 2023.  _WebGPU now available for testing in Safari Technology Preview_.  Retrieved July 18, 2024 from https://webkit.org/blog/14879/webgpu-now-available-for-testing-in-safari-technology-preview/

##  Appendix A Performance Measurements

The measurements are split into two parts: CPU performance and compute
pipeline GPU performance, as described in Figure 5. CPU performance excludes
loading the glTF, which is heavily dependent on bandwidth and focuses on the
BVH construction. For the GPU phase, a total of 100 samples per pixel with a
ray depth of five was used. The image was rendered in Chrome 126 at a
resolution of 512$\times$512 pixels. Experiments were conducted with different
model complexities. The simplified versions are decimated meshes of the
original, which consists of roughly one million triangles. The LOD artifacts
are shown in Figure 6. The first two levels are intended to be visually
similar, while the third level is a simplified version intended to demonstrate
the effect of non-manifold geometry for ray tracing.

The machine specifications used for the measurements are:

  * •

Apple M1 Max: MacBook Pro with Apple Silicon M1 Max

  * •

AMD/NVIDIA: AMD Ryzen 5 5600X and NVIDIA GeForce RTX 3080 (10 GB)

Figure 6. The three LOD artifacts, from left to right: 1,068,735 triangles,
106,873 triangles, 10,687 triangles. The left and middle figures share similar
visual fidelity characteristics.

Three images showing a pushbutton model with different levels of detail from
most detailed to least detailed. The difference between the most detailed and
middle detail is minimal, while the least detailed version is significantly
simplified.

The results for CPU performance were recorded using the Performance API and
are presented in Table 1. The results for GPU performance were recorded using
WebGPU timestamp queries and are presented in Table 2. All measurements were
calculated using the mean time of 30 benchmark samples, with a 95% confidence
interval given as $\pm$ standard deviation from the mean time in milliseconds.

Table 1. BVH setup time based on model complexity Triangles | Apple M1 Max | AMD/NVIDIA
---|---|---
1,068,735 | 327.57 ms $\pm$ 0.56 ms | 383.10 ms $\pm$ 4.06 ms
106,873 | 45.49 ms $\pm$ 0.31 ms | 41.77 ms $\pm$ 0.43 ms
10,687 | 10.53 ms $\pm$ 0.24 ms | 8.43 ms $\pm$ 0.22 ms
Table 2. GPU path tracer time based on model complexity Triangles | Apple M1 Max | AMD/NVIDIA
---|---|---
1,068,735 | 2,319.45 ms $\pm$ 11.14 ms | 1,058.73 ms $\pm$ 59.47 ms
106,873 | 1,992.11 ms $\pm$ 8.49 ms | 790.20 ms $\pm$ 3.79 ms
10,687 | 2,031.38 ms $\pm$ 9.87 ms | 790.83 ms $\pm$ 4.18 ms

The path tracer has not been optimized for performance. These measurements are
intended to provide a baseline for future optimizations.


# Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition

Pete Warden
Google Brain
Mountain View, California
<EMAIL_ADDRESS>
(April 2018)

##  1 Abstract

Describes an audio dataset[1] of spoken words designed to help train and
evaluate keyword spotting systems. Discusses why this task is an interesting
challenge, and why it requires a specialized dataset that’s different from
conventional datasets used for automatic speech recognition of full sentences.
Suggests a methodology for reproducible and comparable accuracy metrics for
this task. Describes how the data was collected and verified, what it
contains, previous versions[2] and properties. Concludes by reporting baseline
results of models trained on this dataset.

##  2 Introduction

Speech recognition research has traditionally required the resources of large
organizations such as universities or corporations to pursue. People working
in those organizations usually have free access to either academic datasets
through agreements with groups like the Linguistic Data Consortium[3], or to
proprietary commercial data.

As speech technology has matured, the number of people who want to train and
evaluate recognition models has grown beyond these traditional groups, but the
availability of datasets hasn’t widened. As the example of ImageNet[4] and
similar collections in computer vision has shown, broadening access to
datasets encourages collaborations across groups and enables apples-for-apples
comparisons between different approaches, helping the whole field move
forward.

The Speech Commands dataset is an attempt to build a standard training and
evaluation dataset for a class of simple speech recognition tasks. Its primary
goal is to provide a way to build and test small models that detect when a
single word is spoken, from a set of ten or fewer target words, with as few
false positives as possible from background noise or unrelated speech. This
task is often known as keyword spotting.

To reach a wider audience of researchers and developers, this dataset has been
released under the Creative Commons BY 4.0 license[5]. This enables it to
easily be incorporated in tutorials and other scripts where it can be
downloaded and used without any user intervention required (for example to
register on a website or email an administrator for permission). This license
is also well known in commercial settings, and so can usually be dealt with
quickly by legal teams where approval is required.

##  3 Related Work

Mozilla’s Common Voice dataset[6] has over 500 hours from 20,000 different
people, and is available under the Creative Commons Zero license (similar to
public domain). This licensing makes it very easy to build on top of. It is
aligned by sentence, and was created by volunteers reading requested phrases
through a web application.

LibriSpeech[7] is a collection of 1,000 hours of read English speech, released
under a Creative Commons BY 4.0 license, and stored using the open source FLAC
encoder, which is widely supported. Its labels are aligned at the sentence
level only, thus lacking word-level alignment information. This makes it more
suitable for full automatic speech recognition than keyword spotting.

TIDIGITS[8] contains 25,000 digit sequences spoken by 300 different speakers,
recorded in a quiet room by paid contributors. The dataset is only available
under a commercial license from the Language Data Consortium, and is stored in
the NIST SPHERE file format, which proved hard to decode using modern
software. Our initial experiments on keyword spotting were performed using
this dataset.

CHiME-5[9] has 50 hours of speech recorded in people’s homes, stored as 16 KHz
WAV files, and available under a restricted license. It’s aligned at the
sentence level.

##  4 Motivations

Many voice interfaces rely on keyword spotting to start interactions. For
example you might say "Hey Google" or "Hey Siri"[10] to begin a query or
command for your phone. Once the device knows that you want to interact, it’s
possible to send the audio to a web service to run a model that’s only limited
by commercial considerations, since it can run on a server whose resources are
controlled by the cloud provider. The initial detection of the start of an
interaction is impractical to run as a cloud-based service though, since it
would require sending audio data over the web from all devices all the time.
This would be very costly to maintain, and would increase the privacy risks of
the technology.

Instead, most voice interfaces run a recognition module locally on the phone
or other device. This listens continuously to audio input from microphones,
and rather than sending the data over the internet to a server, they run
models that listen for the desired trigger phrases. Once a likely trigger is
heard, the transfer of the audio to a web service begins. Because the local
model is running on hardware that’s not under the web service provider’s
control, there are hard resource constraints that the on-device model has to
respect. The most obvious of these is that the mobile processors typically
present have total compute capabilities that are much lower than most servers,
so to run in near real-time for an interactive response, on-device models must
require fewer calculations than their cloud equivalents. More subtly, mobile
devices have limited battery lives and anything that is running continuously
needs to be very energy efficient or users will find their device is drained
too quickly. This consideration doesn’t apply to plugged-in home devices, but
those do have thermal constraints on how much heat they can dissipate that
restrict the amount of energy available to local models, and are encouraged by
programs like EnergyStar to reduce their overall power usage as much as
possible. A final consideration is that users expect a fast response from
their devices, and network latency can be highly variable depending on the
environment, so some initial acknowledgement that a command was received is
important for a good experience, even if the full server response is delayed.

These constraints mean that the task of keyword spotting is quite different to
the kind of speech recognition that’s performed on a server once an
interaction has been spotted:

  * •

Keyword spotting models must be smaller and involved less compute.

  * •

They need to run in a very energy-efficient way.

  * •

Most of their input will be silence or background noise, not speech, so false
positives on those must be minimized.

  * •

Most of the input that is speech will be unrelated to the voice interface, so
the model should be unlikely to trigger on arbitrary speech.

  * •

The important unit of recognition is a single word or short phrase, not an
entire sentence.

These differences mean that the training and evaluation process between on-
device keyword spotting and general speech recognition models is quite
different. There are some promising datasets to support general speech tasks,
such as Mozilla’s Common Voice, but they aren’t easily adaptable to keyword
spotting.

This Speech Commands dataset aims to meet the special needs around building
and testing on-device models, to enable model authors to demonstrate the
accuracy of their architectures using metrics that are comparable to other
models, and give a simple way for teams to reproduce baseline models by
training on identical data. The hope is that this will speed up progress and
collaboration, and improve the overall quality of models that are available.

A second important audience is hardware manufacturers. By using a publicly-
available task that closely reflects product requirements, chip vendors can
demonstrate the accuracy and energy usage of their offerings in a way that’s
easily comparable for potential purchasers. This increased transparency should
result in hardware that better meets product requirements over time. The
models should also provide clear specifications that hardware engineers can
use to optimize their chips, and potentially suggest model changes that make
it easier to provide efficient implementations. This kind of co-design between
machine learning and hardware can be a virtuous circle, increasing the flow of
useful information between the domains in a way that helps both sides.

##  5 Collection

###  5.1 Requirements

I made the decision to focus on capturing audio that reflected the on-device
trigger phrase task described above. This meant that the use of studio-
captured samples seemed unrealistic, since that audio would lack background
noise, would be captured with high-quality microphones, and in a formal
setting. Successful models would need to cope with noisy environments, poor
quality recording equipment, and people talking in a natural, chatty way. To
reflect this, all utterances were captured through phone or laptop
microphones, wherever users happened to be. The one exception was that I asked
them to avoid recording themselves whenever there were background
conversations happening for privacy reasons, so I asked them to be in a room
alone with the door closed.

I also decided to focus on English. This was for pragmatic reasons, to limit
the scope of the gathering process and make it easier for native speakers to
perform quality control on the gathered data. I hope that transfer learning
and other techniques will still make this dataset useful for other languages
though, and I open-sourced the collection application to allow others to
easily gather similar data in other languages. I did want to gather as wide a
variety of accents as possible however, since we’re familiar from experience
with the bias towards American English in many voice interfaces.

Another goal was to record as many different people as I could. Keyword-
spotting models are much more useful if they’re speaker-independent, since the
process of personalizing a model to an individual requires an intrusive user
interface experience. With this in mind, the recording process had to be quick
and easy to use, to reduce the number of people who would fail to complete it.

I also wanted to avoid recording any personally-identifiable information from
contributors, since any such data requires handling with extreme care for
privacy reasons. This meant that I wouldn’t ask for any attributes like gender
or ethnicity, wouldn’t require a sign-in through a user ID that could link to
personal data, and would need users to agree to a data-usage agreement before
contributing.

To simplify the training and evaluation process, I decided to restrict all
utterances to a standard duration of one second. This excludes longer words,
but the usual targets for keyword recognition are short so this didn’t seem to
be too restrictive. I also decided to record only single words spoken in
isolation, rather than as part of a sentence, since this more closely
resembles the trigger word task we’re targeting. It also makes labeling much
easier, since alignment is not as crucial.

###  5.2 Word Choice

I wanted to have a limited vocabulary to make sure the capture process was
lightweight, but still have enough variety for models trained on the data to
potentially be useful for some applications. I also wanted the dataset to be
usable in comparable ways to common proprietary collections like TIDIGITS.
This led me to pick twenty common words as the core of our vocabulary. These
included the digits zero to nine, and in version one, ten words that would be
useful as commands in IoT or robotics applications; "Yes", "No", "Up", "Down",
"Left", "Right", "On", "Off", "Stop", and "Go". In version 2 of the dataset, I
added four more command words; “Backward”, “Forward”, “Follow”, and “Learn”.
One of the most challenging problems for keyword recognition is ignoring
speech that doesn’t contain triggers, so I also needed a set of words that
could act as tests of that ability in the dataset. Some of these, such as
“Tree”, were picked because they sound similar to target words and would be
good tests of a model’s discernment. Others were chosen arbitrarily as short
words that covered a lot of different phonemes. The final list was "Bed",
"Bird", "Cat", "Dog", "Happy", "House", "Marvin", "Sheila", "Tree", and "Wow".

###  5.3 Implementation

To meet all these requirements, I created an open-source web-based application
that recorded utterances using the WebAudioAPI[11]. This API is supported on
desktop browsers like Firefox and Chrome, and on Android mobile devices. It’s
not available on iOS, which was considered to be unfortunate but there were no
alternatives that were more attractive. I also looked into building native
mobile applications for iOS and Android, but I found that users were reluctant
to install them, for privacy and security reasons. The web experience requires
users to grant permission to the website to access the microphone, but that
seemed a lot more acceptable, based on the increased response rate. The
initial test of the application was hosted at an appspot.com subdomain, but it
was pointed out that teaching users to give microphone permissions to domains
that were easy for malicious actors to create was a bad idea. To address this,
the final home of the application was moved to:

⬇

https://aiyprojects.withgoogle.com/open_speech_recording

This is a known domain that’s controlled by Google, and so it should be much
harder to create confusing spoofs of.

The initial page that a new user sees when navigating to the application
explains what the project is doing, and asks them to explicitly and formally
agree to participating in the study. This process was designed to ensure that
the resulting utterances could be freely redistributed as part of an open
dataset, and that users had a clear understanding of what the application was
doing. When a user clicks on “I Agree”, a session cookie is added to record
their agreement. The recording portion of the application will only be shown
if this session cookie is found, and all upload accesses are guarded by cross-
site request forgery tokens, to ensure that only audio recorded from the
application can be uploaded, and that utterances are from users who have
agreed to the terms.

The recording page asks users to press a “Record” button when they’re ready,
and then displays a random word from the list described above. The word is
displayed for 1.5 seconds while audio is recorded, and then another randomly-
chosen word is shown after a one-second pause. Each audio clip is added to a
list that’s stored locally on the client’s machine, and they remain there
until the user has finished recording all words and has a chance to review
them. The random ordering of words was chosen to avoid pronunciation changes
that might be caused by repetition of the same word multiple times. Core words
are shown five times each in total, whereas auxiliary words only appear once.
There are 135 utterances collected overall, which takes around six minutes in
total to run through completely. The user can pause and restart at any point.

Once the recording process is complete, the user is asked to review all of the
clips, and if they’re happy with them, upload them. This then invokes a web
API which uploads the audio to the server application, which saves them into a
cloud storage bucket. The WebAudioAPI returns the audio data in OGG-compressed
format, and this is what gets stored in the resulting files. The session ID is
used as the prefix of each file name, and then the requested word is followed
by a unique instance ID for the recording. This session ID has been randomly
generated, and is not tied to an account or any other demographic information,
since none has been generated. It does serve as a speaker identifier for
utterances however. To ensure there’s a good distribution of different
speakers, once a user has gone through this process once a cookie is added to
the application that ensures they can’t access the recording page again.

To gather volunteers for this process, I used appeals on social media to share
the link and the aims of the project. I also experimented with using paid
crowdsourcing for some of the utterances, though the majority of the dataset
comes from the open site.

###  5.4 Quality Control

The gathered audio utterances were of variable quality, and so I needed
criteria to accept or reject submissions. The informal guideline I used was
that if a human listener couldn’t tell what word was being spoken, or it
sounded like an incorrect word, then the clip should be rejected. To
accomplish this, I used several layers of review.

To remove clips that were extremely short or quiet, I took advantage of the
nature of the OGG compression format. Compressed clips that contained very
little audio would be very small in size, so a good heuristic was that any
files that were smaller than 5 KB were unlikely to be correct. To implement
this rule, I used the following Linux shell command:

⬇

find ${BASEDIR}/oggs -iname "*.ogg" -size -5k -delete

With that complete, I then converted the OGG files into uncompressed WAV files
containing PCM sample data at 16KHz, since this is any easier format for
further processing:

⬇

find ${BASEDIR}/oggs -iname "*.ogg" -print0 | xargs -0 basename -s .ogg | xargs -I {} ffmpeg -i ${BASEDIR}/oggs/{}.ogg -ar 16000 ${BASEDIR}/wavs/{}.wav

Samples from other sources came as varying sample-rate WAV files, so they were
also resampled to 16 KHz WAV files using a similar ffmpeg command.

###  5.5 Extract Loudest Section

From manual inspection of the results, there were still large numbers of
utterances that were too quiet or completely silent. The alignment of the
spoken words within the 1.5 second file was quite arbitrary too, depending on
the speed of the user’s response to the word displayed. To solve both these
problems, I created a simple audio processing tool called Extract Loudest
Section to examine the overall volume of the clips. As a first stage, I summed
the absolute differences of all the samples from zero (using a scale where
-32768 in the 16-bit sample data was -1.0 as a floating-point number, and
+32767 was 1.0), and looked at the mean average of that value to estimate the
overall volume of the utterance. From experimentation, anything below 0.004 on
this metric was likely to be to quiet to be intelligible, and so all of those
clips were removed.

To approximate the correct alignment, the tool then extracted the one-second
clip that contained the highest overall volume. This tended to center the
spoken word in the middle of the trimmed clip, assuming that the utterance was
the loudest part of the recording. To run these processes, the following
commands were called:

⬇

git clone https://github.com/petewarden/extract_loudest_section
tmp/extract_loudest_section

cd tmp/extract_loudest_section

make

cd ../..

mkdir -p ${BASEDIR}/trimmed_wavs

/tmp/extract_loudest_section/gen/bin/extract_loudest_section
${BASEDIR}’/wavs/*.wav’ ${BASEDIR}/trimmed_wavs/

###  5.6 Manual Review

These automatic processes caught technical problems with quiet or silent
recordings, but there were still some utterances that were of incorrect words
or were unintelligible for other reasons. To filter these out I turned to
commercial crowdsourcing. The task asked workers to type in the word they
heard from each clip, and gave a list of the expected words as examples. Each
clip was only evaluated by a single worker, and any clips that had responses
that didn’t match their expected labels were removed from the dataset.

###  5.7 Release Process

The recorded utterances were moved into folders, with one for each word. The
original 16-digit hexadecimal speaker ID numbers from the web application’s
file names were hashed into 8-digit hexadecimal IDs. Speaker IDs from other
sources (like the paid crowdsourcing sites) were also hashed into the same
format. This was to ensure that any connection to worker IDs or other
personally-identifiable information was removed. The hash function used is
stable though, so in future releases the IDs for existing files should remain
the same, even as more speakers are added.

###  5.8 Background Noise

A key requirement for keyword spotting in real products is distinguishing
between audio that contains speech, and clips that contain none. To help train
and test this capability, I added several minute-long 16 KHz WAV files of
various kinds of background noise. Several of these were recorded directly
from noisy environments, for example near running water or machinery. Others
were generated mathematically using these commands in Python:

⬇

scipy.io.wavfile.write(’/tmp/white_noise.wav’, 16000,
np.array(((acoustics.generator.noise(16000*60, color=’white’))/3) *
32767).astype(np.int16))

scipy.io.wavfile.write(’/tmp/pink_noise.wav’, 16000,
np.array(((acoustics.generator.noise(16000*60, color=’pink’))/3) *
32767).astype(np.int16))

To distinguish these files from word utterances, they were placed in a
specially-named `"_background_noise_"` folder, in the root of the archive.

##  6 Properties

The final dataset consisted of 105,829 utterances of 35 words, broken into the
categories and frequencies shown in Table 1.

Word | Number of Utterances
---|---
Backward | 1,664
Bed | 2,014
Bird | 2,064
Cat | 2,031
Dog | 2,128
Down | 3,917
Eight | 3,787
Five | 4,052
Follow | 1,579
Forward | 1,557
Four | 3,728
Go | 3,880
Happy | 2,054
House | 2,113
Learn | 1,575
Left | 3,801
Marvin | 2,100
Nine | 3,934
No | 3,941
Off | 3,745
On | 3,845
One | 3,890
Right | 3,778
Seven | 3,998
Sheila | 2,022
Six | 3,860
Stop | 3,872
Three | 3,727
Tree | 1,759
Two | 3,880
Up | 3,723
Visual | 1,592
Wow | 2,123
Yes | 4,044
Zero | 4,052
Figure 1: How many recordings of each word are present in the dataset

Each utterance is stored as a one-second (or less) WAVE format file, with the
sample data encoded as linear 16-bit single-channel PCM values, at a 16 KHz
rate. There are 2,618 speakers recorded, each with a unique eight-digit
hexadecimal identifier assigned as described above. The uncompressed files
take up approximately 3.8 GB on disk, and can be stored as a 2.7GB gzip-
compressed tar archive.

##  7 Evaluation

One of this dataset’s primary goals is to enable meaningful comparisons
between different models’ results, so it’s important to suggest some precise
testing protocols. As a starting point, it’s useful to specify exactly which
utterances can be used for training, and which must be reserved for testing,
to avoid overfitting. The dataset download includes a text file called
`validation_list.txt`, which contains a list of files that are expected to be
used for validating results during training, and so can be used frequently to
help adjust hyperparameters and make other model changes. The
`testing_list.txt` file contains the names of audio clips that should only be
used for measuring the results of trained models, not for training or
validation. The set that a file belongs to is chosen using a hash function on
its name. This is to ensure that files remain in the same set across releases,
even as the total number changes, so avoid set cross-contamination when trying
old models on the more recent test data. The Python implementation of the set
assignment algorithm is given in the TensorFlow tutorial code[12] that is a
companion to the dataset.

###  7.1 Top-One Error

The simplest metric to judge a trained model against is how many utterances it
can correctly identify. In principle this can be calculated by running the
model against all the files in the testing set, and comparing the reported
against the expected label for each. Unlike image classification tasks like
ImageNet, it’s not obvious how to weight all of the different categories. For
example, I want a model to indicate when no speech is present, and separately
to indicate when it thinks a word has been spoken that’s not one it
recognizes. These “open world” categories need to be weighted according to
their expected occurrence in a real application to produce a realistic metric
that reflects the perceived quality of the results in a product.

The standard chosen for the TensorFlow speech commands example code is to look
for the ten words "Yes", "No", "Up", "Down", "Left", "Right", "On", "Off",
"Stop", and "Go", and have one additional special label for “Unknown Word”,
and another for “Silence” (no speech detected). The testing is then done by
providing equal numbers of examples for each of the twelve categories, which
means each class accounts for approximately 8.3% of the total. The "Unknown
Word" category contains words randomly sampled from classes that are part of
the target set. The "Silence" category has one-second clips extracted randomly
from the background noise audio files.

I’ve uploaded a standard set of test files[13] to make it easier to reproduce
this metric. If you want to calculate the canonical Top-One error for a model,
run inference on each audio clip, and compare the top predicted class against
the ground truth label encoded in its containing subfolder name. The
proportion of correct predictions will give you the Top-One error. There’s
also a similar collection of test files[14] available for version one of the
dataset.

The example training code that accompanies the dataset[15] provides results of
88.2% on this metric for the highest-quality model when fully trained. This
translates into a model that qualitatively gives a reasonable, but far from
perfect response, so it’s expected that this will serve as a baseline to be
exceeded by more sophisticated architectures.

###  7.2 Streaming Error Metrics

Top-One captures a single dimension of the perceived quality of the results,
but doesn’t reveal much about other aspects of its performance in a real
application. For example, models in products receive a continuous stream of
audio data and don’t know when words start and end, whereas the inputs to Top
One evaluations are aligned to the beginning of utterances. The equal
weighting of each category in the overall score also doesn’t reflect the
distribution of trigger words and silence in typical environments.

To measure some of these more complex properties of models, I test them
against continuous streams of audio and score them on multiple metrics. Here’s
what the baseline model trained with V2 data produces:

⬇

49.0% matched, 46.0% correctly, 3.0% wrongly, 0.0% false positives

To produce this result, I ran the following bash script against the 10 minute
streaming test audio clip and ground truth labels:

⬇

bazel run tensorflow/examples/speech_commands:freeze \--
\--start_checkpoint=/tmp/speech_commands_train/conv.ckpt-18000
\--output_file=/tmp/v2_frozen_graph.pb

bazel run tensorflow/examples/speech_commands:test_streaming_accuracy \--
\--graph=/tmp/v2_frozen_graph.pb
\--wav=/tmp/speech_commands_train/streaming_test.wav
\--labels=/tmp/speech_commands_train/conv_labels.txt
\--ground_truth=/tmp/speech_commands_train/streaming_test_labels.txt

  * •

Matched-percentage represents how many words were correctly identified, within
a given time tolerance.

  * •

Wrong-percentage shows how many words were correctly distinguished as speech
rather than background noise, but were given the wrong class label.

  * •

False-positive percentage is the number of words detected that were in parts
of the audio where no speech was actually present.

An algorithm for calculating these values given an audio file and a text file
listing ground truth labels is implemented in TensorFlow as
`test_streaming_accuracy.cc`[16].

Performing successfully on these metrics requires more than basic template
recognition of audio clips. There has to be at least a very crude set of rules
to suppress repeated recognitions of the same word in short time frames, so
default logic for this is implemented in `recognize_commands.cc`[17].

This allows a simple template-style recognition model to be used directly to
generate these statistics. One of the other configurable features of the
accuracy test is the time tolerance for how close to the ground truth’s time a
recognition result must be to count as a match. The default for this is set to
750ms, since that seems to match with requirements for some of the
applications that are supported.

To make reproducing and comparing results easier, I’ve made available a one-
hour audio file[18] containing a mix of utterances at random times and noise,
together with a text file marking the times and ground truth labels of each
utterance. This was generated using the script included in the TensorFlow
tutorial, and can be used to compare different models performance on streaming
applications.

###  7.3 Historical Evaluations

Version 1 of the dataset[2] was released August 3rd 2017, and contained 64,727
utterances from 1,881 speakers. Training the default convolution model from
the TensorFlow tutorial (based on Convolutional Neural Networks for Small-
footprint Keyword Spotting[19]) using the V1 training data gave a Top-One
score of 85.4%, when evaluated against the test set from V1. Training the same
model against version 2 of the dataset[1], documented in this paper, produces
a model that scores 88.2% Top-One on the training set extracted from the V2
data. A model trained on V2 data, but evaluated against the V1 test set gives
89.7% Top-One, which indicates that the V2 training data is responsible for a
substantial improvement in accuracy over V1. The full set of results are shown
in Table 2.

Data | V1 Training | V2 Training
---|---|---
V1 Test | 85.4% | 89.7%
V2 Test | 82.7% | 88.2%
Figure 2: Top-One accuracy evaluations using different training data

These figures were produced using the checkpoints produced by the following
training commands:

⬇

python tensorflow/examples/speech_commands/train.py
\--data_url=http://download.tensorflow.org/data/speech_commands_v0.01.tar.gz

python tensorflow/examples/speech_commands/train.py
\--data_url=http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz

The results of these commands are available as pretrained checkpoints[20]. The
evaluations were performed by running variations on the following command line
(with the v1/v2’s substituted as appropriate):

⬇

python tensorflow/examples/speech_commands/train.py
\--data_url=http://download.tensorflow.org/data/speech_commands_v0.0{1,1}.tar.gz
\--start_checkpoint=${HOME}/speech_commands_checkpoints/conv-v{1,2}.ckpt-18000

###  7.4 Applications

The TensorFlow tutorial gives a variety of baseline models, but one of the
goals of the dataset is to enable the creation and comparison of a wide range
of models on a lot of different platforms, and version one of has enabled some
interesting applications. CMSIS-NN[21] covers a new optimized implementation
of neural network operations for ARM microcontrollers, and uses Speech
Commands to train and evaluate the results. Listening to the World[22]
demonstrates how combining the dataset and UrbanSounds[23] can improve the
noise tolerance of recognition models. Did you Hear That[24] uses the dataset
to test adversarial attacks on voice interfaces. Deep Residual Learning for
Small Footprint Keyword Spotting[25] shows how approaches learned from ResNet
can produce more efficient and accurate models. Raw Waveform-based Audio
Classification[26] investigates alternatives to traditional feature extraction
for speech and music models. Keyword Spotting Through Image Recognition[27]
looks at the effect virtual adversarial training on the keyword task.

##  8 Conclusion

The Speech Commands dataset has shown to be useful for training and evaluating
a variety of models, and the second version shows improved results on
equivalent test data, compared to the original.

##  9 Acknowledgements

Massive thanks are due to everyone who donated recordings to this data set,
I’m very grateful. I also couldn’t have put this together without the help and
support of Billy Rutledge, Rajat Monga, Raziel Alvarez, Brad Krueger, Barbara
Petit, Gursheesh Kour, Robert Munro, Kirsten Gokay, David Klein, Lukas
Biewald, and all the AIY and TensorFlow teams.

## References

  * [1] (2018) Speech commands dataset version 2. [Online]. Available: http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz
  * [2] (2017) Speech commands dataset version 1. [Online]. Available: http://download.tensorflow.org/data/speech_commands_v0.01.tar.gz
  * [3] (2018) Linguistic data consortium. [Online]. Available: https://www.ldc.upenn.edu/
  * [4] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in _CVPR09_ , 2009.
  * [5] (2018) Creative commons international attribution international 4.0 license. [Online]. Available: https://creativecommons.org/licenses/by/4.0/
  * [6] (2017) Mozilla common voice. [Online]. Available: https://voice.mozilla.org/en
  * [7] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an ASR corpus based on public domain audio books,” in _Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2015.
  * [8] R. G. Leonard and G. R. Doddington. (1992) A speaker-independent connected-digit database. [Online]. Available: https://catalog.ldc.upenn.edu/docs/LDC93S10/tidigits.readme.html
  * [9] (2018) The 5th chime speech separation and recognition challenge. [Online]. Available: http://spandh.dcs.shef.ac.uk/chime_challenge/data.html
  * [10] (2017) Hey siri: An on-device dnn-powered voice trigger for apple’s personal assistant. [Online]. Available: https://machinelearning.apple.com/2017/10/01/hey-siri.html
  * [11] (2015) Web audio api. [Online]. Available: https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API
  * [12] (2018) Implementation of set assignment algorithm. [Online]. Available: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/speech_commands/input_data.py#L61
  * [13] (2018) Speech commands dataset test set version 2. [Online]. Available: http://download.tensorflow.org/data/speech_commands_test_set_v0.02.tar.gz
  * [14] (2017) Speech commands dataset test set version 1. [Online]. Available: http://download.tensorflow.org/data/speech_commands_test_set_v0.01.tar.gz
  * [15] (2017) Tensorflow audio recognition tutorial. [Online]. Available: https://www.tensorflow.org/tutorials/audio_recognition
  * [16] (2018) test_streaming_accuracy.cc source file. [Online]. Available: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/speech_commands/test_streaming_accuracy.cc
  * [17] (2018) recognize_commands.cc source file. [Online]. Available: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/speech_commands/recognize_commands.cc
  * [18] (2018) Speech commands dataset streaming test version 2. [Online]. Available: http://download.tensorflow.org/data/speech_commands_streaming_test_v0.02.tar.gz
  * [19] T. N. Sainath and C. Parada, “Convolutional Neural Networks for Small-Footprint Keyword Spotting,” in _Sixteenth Annual Conference of the International Speech Communication Association_ , 2015. [Online]. Available: https://www.isca-speech.org/archive/interspeech_2015/papers/i15_1478.pdf
  * [20] (2018) Speech commands tutorial checkpoints. [Online]. Available: https://storage.googleapis.com/download.tensorflow.org/models/speech_commands_checkpoints.tar.gz
  * [21] L. Lai, N. Suda, and V. Chandra, “CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs,” _ArXiv e-prints_ , Jan. 2018.
  * [22] B. McMahan and D. Rao, “Listening to the World Improves Speech Command Recognition,” _ArXiv e-prints_ , Oct. 2017.
  * [23] J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in _Proceedings of the 22Nd ACM International Conference on Multimedia_ , ser. MM ’14. New York, NY, USA: ACM, 2014, pp. 1041–1044. [Online]. Available: http://doi.acm.org/10.1145/2647868.2655045
  * [24] M. Alzantot, B. Balaji, and M. Srivastava, “Did you hear that? Adversarial Examples Against Automatic Speech Recognition,” _ArXiv e-prints_ , Jan. 2018.
  * [25] R. Tang and J. Lin, “Deep Residual Learning for Small-Footprint Keyword Spotting,” _ArXiv e-prints_ , Oct. 2017.
  * [26] J. Lee, T. Kim, J. Park, and J. Nam, “Raw Waveform-based Audio Classification Using Sample-level CNN Architectures,” _ArXiv e-prints_ , Dec. 2017.
  * [27] S. Krishna Gouda, S. Kanetkar, D. Harrison, and M. K. Warmuth, “Speech Recognition: Keyword Spotting Through Image Recognition,” _ArXiv e-prints_ , Mar. 2018.