# Real-time quantitative imaging of RTV silicone pyrolysis

Collin W. Foster , Sreevishnu Oruganti, Francesco Panerai
Center for Hypersonics and Entry Systems Studies, Department of Aerospace
Engineering
University of Illinois at Urbana-Champaign
Urbana, 61801, Illinois, USA
{collinf3, so24<EMAIL_ADDRESS>
Corresponding author<EMAIL_ADDRESS>
###### Abstract

Quantitative microstructural analysis of Room Temperature Vulcanized (RTV)
silicone pyrolysis at high temperatures is presented. RTV is used as a bonding
agent in multiple industries, particularly filling gaps in ablative tiles for
hypersonic (re-)entry vehicles and fire prevention. Decomposition of RTV is
resolved in real time using in situ high-temperature X-ray computed micro-
tomography. Full tomographies are acquired every 90 seconds for four different
linear heating rates ranging from 7 to 54∘C/min. The microstructure is
resolved below 5 $\mu$m/pixel, allowing for a full quantitative analysis of
the micro-structural evolution and porous network development. Results are
highly heating rate dependent, and are evaluated for bulk sample volume
change, porosity, pore network size, and observed densification from X-ray
attenuation. The outcome of this work is critical to develop multi-physics
models for thermal response.

_K_ eywords RTV $\cdot$ Pyrolysis $\cdot$ Microstructure $\cdot$ X-ray
tomography $\cdot$ Synchrotron radiation

##  1 Introduction

High temperature materials are advancing in the fields of nuclear reactors,
fire-prevention, and thermal protection systems (TPS) for hypersonic flight.
With the increasing applications of such materials, their characterization as
a function of temperature is critical to application and modeling efforts.
Room Temperature Vulcanizing Silicone (RTV) is a material system noted for its
excellent thermal and mechanical properties and resistance to chemical
degradation at high temperatures [1, 2, 3]. RTV is also adaptable as a gap-
filler for ablative heat shields, seeing widespread applications in
exploration missions such as Mars2020 [4], Mars Science Laboratory (MSL)
heatshield [5], and other sample-return missions [6]. RTV’s chemical
composition contains iron compounds, which are well-known for being flame
retardant, smoke suppressant, and can delay thermal decomposition, all of
which benefit the fire-safety industries [7, 8]. Data recovered from the Space
Shuttle missions and MSL describe the extreme aerothermal (re-)entry
environments with temperatures over 2500∘C where RTV was used as a bonding
agent [9, 10, 11]. RTV and silicone-based coatings are unique in that they are
also intumescent, meaning that they will swell when heated to create an
insulating barrier from the source of the heat[12]. This property of
intumescence is helpful in coating applications (such as on wood, plastic, or
metal) [13] but must be investigated further when swelling could cause
unforeseen mechanical stress on a highly compact TPS [14, 15, 16]. RTV as a
gap-filler on tiled TPS is observed to create protuberances in hypersonic flow
as it ablates differently from the heat shield tiles, transitioning the flow
from laminar to turbulent, resulting in areas of higher heating on the
heatshield [17]. This behavior can be non-destructively examined with X-ray
computed micro-tomography ($\mu$-CT), as a method to examine the inner-
morphology of the material system as it pyrolyzes at high temperatures.
Performing $\mu$-CT is a proven technique for evaluating complicated porous
networks that are chemically decomposing, and the results can then be used to
model multi-scale material behavior [18, 19, 20, 21, 22]. Recent work has
thoroughly characterized the microstructure of RTV, emphasizing the utility of
$\mu$-CT as a quantitative tool for examining the formation of voids at high
temperatures [23, 24]. However, further investigation must be made into its
heating rate dependence on micro-scale
(1-$100\text{\,}\mathrm{\SIUnitSymbolMicro m}$) porosity evolution.
Experiments to resolve the microstructural evolution of RTV at various heating
ramp rates will be used to inform multi-physics codes, to then incorporated
into larger TPS and fire modeling efforts [25, 26, 27, 28, 29, 30]. Thus, this
study utilizes synchrotron radiation to resolve the microstructural effects of
pyrolysis at a range of heating rates _in situ_. This is quantified by
measurements of porosity, total volume change, X-ray attenuation, followed by
further discussion with scanning electron microscopy (SEM) images.

##  2 Materials and Methods

The experiments are conducted at the Advanced Light Source (ALS) synchrotron
facility at Lawrence Berkeley National Lab. Data is collected at the beamline
8.3.2, where _in situ_ $\mu$-CT can be performed using the high flux of X-rays
directly from the synchrotron source, combined with environmental chambers to
test materials under thermal, mechanical and chemical loads [31, 32]. The
inner morphology of the material is evaluated during heating through
continuous tomographies being generated every 90 seconds. This temporal
resolution enables microstructural analysis beyond that of a typical lab-based
computed tomography device. The RTV samples are contained in a sealed quartz-
chamber as shown in Fig. 1a, flushed with inert Argon gas constantly being
cycled at 0.091 g/s to prevent oxidation from occurring at elevated
temperatures. The sample enclosure is aligned in the beam path while also
being surrounded by six confocally arranged infrared lamps aimed on the RTV
cylinder (Fig. 1b) to increase temperature to 1000∘C, similar to previous _in
situ_ work [33, 34, 35, 36, 37]. The power supplied to the lamps is controlled
such that a range of heating rates could be evaluated, this included:
54∘C/min, 28∘C/min, 14∘C/min, and 7∘C/min. Experiments are performed at
temperatures ranging from 23∘C (virgin material) to 1000∘C (charred). A lens
with a resolution of 3.25 $\mu$m/pixel is used for the high heating rates (54
and 28∘C/min), and a higher resolution 1.303 $\mu$m/pixel for the low heating
rates (14 and 7∘C/min) to further evaluate micro-pore evolution at smaller
length scales. The samples are shaped to fit completely within the field of
view (FOV) of the lens, $3.3\text{\,}\mathrm{mm}$ diameter for the high
heating rates and $2\text{\,}\mathrm{mm}$ diameter for the low heating rates.
Lastly, the smaller pores ($<$$1\text{\,}\mathrm{\SIUnitSymbolMicro m}$) below
the resolution of the $\mu$-CT are qualitatively imaged with Scanning Electron
Microscopy (SEM) using a FEI ESEM Quanta 450 FEG (FELMI ZFE, Gras, Austria).

Figure 1: _In situ_ $\mu$-CT at the ALS. a) Controlled environment enclosure
housing the sample. b) Sample enclosure installed in the focal spot of 6 IR
heating lamps in the ALS $\mu$-CT hutch.

##  3 Results

(a) 54∘C/min

(b) 28∘C/min

(c) 14∘C/min

(d) 7∘C/min

Figure 2: Grayscale output from $\mu$-CT.

Results are presented in Fig. 2 for the four heating rates investigated.
Sample cross sections are shown from reconstructed $\mu$-CT datasets at
increasing temperatures. Beginning with 54∘C/min, Fig. 2(a) shows the virgin
sample with small voids formed that grow and interconnect into large
porosities shown in its mid pyrolysis region (500∘C). Along with the clearly
visible pore network, small shear porosities (thin with high eccentricity)
develop in the surrounding virgin region, further connecting in the late
charring regime ($>$800∘C). Next in Fig. 2(b) at 28∘C/min, the sample
similarly begins with pores from curing that contribute in joining adjacent
voids forming during peak pyrolysis. Seen in both Fig. 2(a) and Fig. 2(b),
while the sample generally experiences a reduction in height, swelling is
observed from the outer circumference where the incident thermal radiation
from the lamp heating occurs. This swelling behavior is clearly observed in
both high heating rates (28 and 54∘C/min) due to the rapid build up of
internal pressure that causes a anisotropic distribution of new pores to form
until the pyrolysis gases are liberated through open pores.

For the low heating rates shown in Fig. 2(c) and Fig. 2(d), smaller sample
sizes are used to achieve higher spatial resolution while keeping the sample
entirely in the FOV. Less initial defects (voids) are seen, but the same
swelling and shrinking behavior is observed as the virgin material pyrolyzes
to its charred state. Qualitatively, much less micro-pore development is
observed with the lower heating rates (7 and 14∘C/min), indicating a large
dependence on heating rate. As further detailed in the subsequent quantitative
analysis of the $\mu$-CT data, the tomographies are seen to get brighter in
the solid phase, due to an increase in the X-ray attenuation of the sample.
The increased X-ray absorption relates to the chemical structure transition
from virgin RTV to char, and can be used as an estimate in material density
shift. Additional video files of the RTV pyrolysis for all heating rates is
provided in the supplementary material section.

Figure 3: Quantitative analysis of RTV morphological evolution during
pyrolysis.

The quantitative behavior of RTV decomposition for the four heating rates is
summarized in Fig. 3. First we examine the overall change in sample height,
and we see that there is a percentage loss of 20-30%. The higher heating rates
tend to create a more anisotropically distributed pore network, contributing
to a less-predictable change in the bulk volume as the pyrolysis progresses.

The incident heat flux is applied radially to the sample, encapsulating a
$5\text{\,}\mathrm{mm}$ diameter sphere. As a result, the change in mean
sample diameter is shown here to be similar for all samples. The initial solid
swelling caused by internal gas pressure build-up is seen in the region
leading to peak pyrolysis at 200-450∘C. This swelling is followed by an
outgassing of pyrolysis byproducts and a shrinkage of the solid phase giving
the reduction in diameter seen for the remainder of the pyrolysis process.

The result of the structural shifts in height and diameter are then fully
realized by computing the change in solid volume during pyrolysis. Across all
heating rates the sample is seen to swell at low temperatures
($\lesssim$400∘C) during the initial phase of pyrolysis, as a result of gas
build-up inside the material. Intumescence is found to be heating rate
dependant with a volume increase up to nearly 15% at the highest heating rate.
An increase in heating rate is also observed to delay peak intumescence
towards higher temperatures. At higher pyrolysis temperatures ($\gtrsim$400∘C)
there is a prominent material shrinkage that leads to up to 60% volume
reduction for the fully charred material, compared to the virgin sample.
Shrinkage rate is higher in the 400-600∘C range, where the bulk of the
pyrolysis occurs [24] and decreases in the 600-1000∘C range.

While the higher heating rates create larger, more interconnected porosities
shown in Fig. 2, the lower heating rates exhibit dissimilar behavior shown in
the porosity measurements. We observed an increase in porosity up to nearly
20% in the 400-600∘C pyrolysis region, with both 28∘C/min and 54∘C/min
producing numerous pores that join neighboring voids until eventually reaching
the surface. As the reaction progresses beyond peak pyrolysis gas release
($>$600∘C), volumetric shrinkage begins to dominate in an overall reduction
and collapse of pores within the samples. Similar behavior can be observed for
the 14∘C/min sample, as it experiences an increase in porosity through
pyrolysis followed by a subsequent decrease thereafter. To be noted is the
behavior of the slowest heating rate, 7∘C/min, as porosity appears to
monotonically increase during pyrolysis. This likely is the case as porosity
is increasing on the nano-scale but is not resolved by the 1.303 $\mu$m/pixel
resolution of the lens.

Porosity variation is further examined by the mean pore diameter. Shown to be
largely unchanging for the two lower heating rates, they fluctuate slightly
with the opening and closing of nano-porosities nearly undetectable by the
resolution of the $\mu$-CT. Because the two higher heating rates begin with
larger samples, they also have larger pre-existing pores that account for
larger initial mean pore diameters. Where the 28∘C/min generates more numerous
disconnected porosities, the pores of the 54∘C/min case coalescence with
neighboring pores that do not shift dramatically in size nor shape.

Lastly, the change from initial material attenuation is plotted for all
heating rates to show the change in relative density of the RTV. The change in
attenuation was also qualitatively observed in Fig. 2. All the RTV samples
experience a small decrease in material attenuation leading up to peak
pyrolysis (200-400∘C) during volumetric swelling due to pressure build-up of
the air trapped in the initial voids, followed by pyrolysis gas release, and
subsequent cross-linking reactions that continue to increase the attenuation
monotonically to char. This observed trend in attenuation is subject to
further analysis by incorporating mass measurements to get predictions on
density transition.

Figure 4: SEM of the 1000∘C charred RTV samples.

SEM images for the tested materials are shown in Fig. 4. The RTV examined is a
two-phase material with a matrix engulfing smaller solid particles deemed
whiskers. Previous energy dispersive electron spectroscopy described the
whiskers and remaining matrix phase being a slurry of C, O, Si, and Fe [24,
23]. The iron in the RTV matrix transitions from a red Hematite (Fe2O3) to the
black Magnetite (Fe3O4), with charred Si, C, and SiO2 matrix when fully-
charred; decomposition reactions and the release of volatile compounds of
O-Si-C chains account for the mass loss in the RTV [24]. This is first
visualized in Fig. 4a-a’ for the 7∘C/min case. At low heating rates the
resulting microstructure resembles the observations of past TGA investigations
at 10∘C/min [23, 24]. More of the matrix material is pyrolyzed and coalesces
with the whiskers, exposing porosities at the sub-micron scale not resolved by
the $\mu$-CT. The same is observed to a lesser degree for the 14∘C/min sample,
as there is evidence of the whiskers material still exposed. After the final
tomography is taken at 1000∘C, the heating is immediately shut-off and the
sample is cooled to room-temperature, quenching any further pyrolysis
reaction. Therefore, at the highest heating rates pyrolysis remains
incomplete. Moving to 28∘C/min, it is seen that the matrix phase has not fully
pyrolyzed, with exposed whisker material visible in 4c’. This trend is further
emphasized by the 54∘C/min case, with uniform outgassing leaving behind a more
opaque surface with few nano-porosities that would have likely formed given
more time at high temperature. Evidence of incomplete pyrolysis of the
material is also seen in the plot of X-ray attenuation in Fig. 2, with the
lower attenuation values for the high heating rates, indicating pyrolysis not
being complete.

Figure 5: Decomposition model of RTV.

A generalized graphic summarizing the key observations of the pyrolysis of RTV
between high (28 and 54∘C/min) and low (7 and 14∘C/min) heating rates is shown
in Fig. 5. Both non-pristine samples begin similarly with slight volumetric
expansion, although the higher heating rate experiences a larger degree of new
porosity development, where the low heating rates generally just expand the
pre-existing pores. The pyrolysis continues and the high heating rate
experiences large internal pressure gradients causing the development of
anisotropically distributed pores that interconnect with the pre-existing
pores. The low heating develops nano-pores as the pyrolysis process nears
closer to completion. Lastly, both samples experience large volumetric
shrinkage as mass is lost from pyrolysis gas escaping, and the overall
porosity decreases as the solid volume collapses.

##  4 Conclusion

In conclusion, this study is a critical contribution to informing micro- and
meso-scale behavior of material systems that utilize RTV in high temperature
environments. Examined is a quantitative analysis of the micro-structural
development of RTV under a range of relevant pyrolysis heating rates. The
results show stark differences from a higher heating rate ($>$28∘C/min) to a
low rate (7-14∘C/min) in porosity formation and morphology. Bulk material
swelling and shrinking is also observed and is relatively similar across all
heating rates, critical when evaluating mechanical performance of the material
system. The continuation of this work will investigate the densification of
this material utilizing the attenuation measurements, and conduct a similar
set of experiments on RTV samples that are defect-free from manufacturing.
This information will be fed directly to multi-physics reentry and fire codes
to evaluate for relevant effective transport properties.

## Acknowledgements

This work is supported by a NASA Space Technology Graduate Research
Opportunities Award Grant No: 80NSSCC22K1192 and 80NSSC21K1117. This research
used resources of the Advanced Light Source, which is a DOE Office of Science
User Facility under contract no. DE-AC02-05CH11231. This material is based
upon work supported by the U.S. Department of Energy, Office of Science,
Office of Workforce Development for Teachers and Scientists, Office of Science
Graduate Student Research (SCGSR) program. The SCGSR program is administered
by the Oak Ridge Institute for Science and Education for the DOE under
contract number DE-SC0014664. Work was also done in part at the Beckman
Institute Microscopy Suite at the University of Illinois at Urbana-Champaign.
We would also like to thank the support of Dula Parkinson and Harold Barnard
at the ALS for their guidance in experiments.

## References

  * [1] C.L. Lee, R.G. Niemi, and K.M. Kelly.  New Silicone RTV Foam.  Journal of Cellular Plastics, 13(1):62–67, Jan 1977.
  * [2] E.A. Cherney and R.S. Gorur.  RTV silicone rubber coatings for outdoor insulators.  IEEE Transactions on Dielectrics and Electrical Insulation, 6(5):605–611, Oct 1999.
  * [3] C.P. Fenimore and F.J. Martin.  Flammability of polymers.  Combustion and Flame, 10(2):135–139, Jun 1966.
  * [4] Robert W Maddock, Alicia M Dwyer Cianciolo, Ashley M Korzun, Daniel K Litton, Carlie H Zumwalt, and Chris D Karlgaard.  InSight Entry, Descent, and Landing Postflight Performance Assessment.  Journal of Spacecraft and Rockets, 58(5):1530–1537, 2021.
  * [5] Robin AS Beck, David M Driver, Michael J Wright, Helen H Hwang, Karl T Edquist, and Steven A Sepka.  Development of the mars science laboratory heatshield thermal protection system.  Journal of Spacecraft and Rockets, 51(4):1139–1150, 2014.
  * [6] E. Venkatapathy, B. Laub, G.J. Hartman, J.O. Arnold, M.J. Wright, and G.A. Allen.  Thermal protection system development, testing, and qualification for atmospheric probes and sample return missions.  Advances in Space Research, 44(1):138–150, Jul 2009.
  * [7] Edward D. Weil and Navin G. Patel.  Iron compounds in non-halogen flame-retardant polyamide systems.  Polymer Degradation and Stability, 82(2):291–296, Jan 2003.
  * [8] Xilei Chen, Mei Li, Jinlong Zhuo, Cuiyong Ma, and Chuanmei Jiao.  Influence of Fe2O3 on smoke suppression and thermal degradation properties in intumescent flame-retardant silicone rubber.  Journal of Thermal Analysis and Calorimetry, 123(1):439–448, Jan 2016.
  * [9] D. Curry.  Space Shuttle Orbiter thermal protection system design and flight experience, 1993.
  * [10] Todd R White, Milad Mahzari, Deepak Bose, and Jose A Santos.  Post-flight Analysis of Mars Science Laboratory’s Entry Aerothermal Environment and Thermal Protection System Response.  In 44th AIAA Thermophysics Conference, page 2779, 2013.
  * [11] Helen Hwang, Deepak Bose, Henry Wright, Todd R. White, Mark Schoenenberger, Jose Santos, Chris D. Karlgaard, Chris Kuhl, Tomo Oishi, and Dominic Trombetta.  Mars 2020 Entry, Descent, and Landing Instrumentation (MEDLI2).  In 46th AIAA Thermophysics Conference, Washington, D.C., Jun 2016\. American Institute of Aeronautics and Astronautics.
  * [12] B. Gardelle, S. Duquesne, V. Rerat, and S. Bourbigot.  Thermal degradation and fire performance of intumescent silicone-based coatings: THERMAL DEGRADATION AND FIRE PERFORMANCE OF SILICONE-BASED COATINGS.  Polymers for Advanced Technologies, 24(1):62–69, Jan 2013.
  * [13] Yuval Halpern, Donna M. Mott, and Ronald H. Niswander.  Fire retardancy of thermoplastic materials by intumescence.  Industrial & Engineering Chemistry Product Research and Development, 23(2):233–238, Jun 1984.
  * [14] Thomas McKeon.  Ablative Degradation of a Silicone Foam.  Journal of Macromolecular Science: Part A - Chemistry, 3(4):585–612, Jul 1969.
  * [15] Rui Fu, Haoyue Weng, Jonathan F. Wenk, and Alexandre Martin.  Thermomechanical coupling for charring ablators.  Journal of Thermophysics and Heat Transfer, 32(2):369–379, Apr 2018.
  * [16] Rui Fu, Haoyue Weng, Jonathan F. Wenk, and Alexandre Martin.  Thermal expansion for charring ablative materials.  Journal of Thermophysics and Heat Transfer, 34(1):57–65, Jan 2020\.
  * [17] David M. Driver, J. Enrique Carballo, Robin Beck, Dinesh Prabhu, Jose A. Santos, Alan Cassell, Kristina Skokova, Chun Tang, Helen H. Hwang, Eric Slimko, William Willcockson, and Jarvis Songer.  Arcjet Testing in Shear Environment for Mars Science Laboratory Thermal Protection System.  Journal of Spacecraft and Rockets, 51(4):1151–1166, Jul 2014.
  * [18] Francesco Panerai, Joseph C. Ferguson, Jean Lachaud, Alexandre Martin, Matthew J. Gasch, and Nagi N. Mansour.  Micro-tomography based analysis of thermal conductivity, diffusivity and oxidation behavior of rigid and flexible fibrous insulators.  International Journal of Heat and Mass Transfer, 108:801–811, May 2017.
  * [19] Joseph C. Ferguson, Francesco Panerai, Arnaud Borner, and Nagi N. Mansour.  PuMA: the Porous Microstructure Analysis software.  SoftwareX, 7:81–87, January 2018.
  * [20] Federico Semeraro, Joseph C. Ferguson, Francesco Panerai, Robert J. King, and Nagi N. Mansour.  Anisotropic analysis of fibrous and woven materials part 1: Estimation of local orientation.  Computational Materials Science, 178:109631, Jun 2020.
  * [21] Federico Semeraro, Joseph C. Ferguson, Marcos Acin, Francesco Panerai, and Nagi N. Mansour.  Anisotropic analysis of fibrous and woven materials part 2: Computation of effective conductivity.  Comput. Mater. Sci., 186:109956, January 2021.
  * [22] Scott A. Roberts, Victor E. Brunini, Kevin N. Long, and Anne M. Grillet.  A Framework for Three-Dimensional Mesoscale Modeling of Anisotropic Swelling and Mechanical Deformation in Lithium-Ion Electrodes.  Journal of The Electrochemical Society, 161(11):F3052, September 2014.
  * [23] Sreevishnu Oruganti, Nagi N. Mansour, Marco Panesi, and Francesco Panerai.  Thermal Decomposition of Ablating Gap Filler Material for Atmospheric Entry Heatshields.  In AIAA AVIATION 2022 Forum, Chicago, IL & Virtual, Jun 2022. American Institute of Aeronautics and Astronautics.
  * [24] Sreevishnu Oruganti, Nagi N. Mansour, Marco Panesi, and Francesco Panerai.  Microstructure of pyrolyzing RTV silicone.  Polymer Degradation and Stability, 207:110237, 2023.
  * [25] Jeremie B.E. Meurisse, Jean Lachaud, Francesco Panerai, Chun Tang, and Nagi N. Mansour.  Multidimensional material response simulations of a full-scale tiled ablative heatshield.  Aerospace Science and Technology, 76:497–511, May 2018.
  * [26] Gizem Okyay, Anil D. Naik, Fabienne Samyn, Maude Jimenez, and Serge Bourbigot.  Fractal conceptualization of intumescent fire barriers, toward simulations of virtual morphologies.  Scientific Reports, 9(1):1872, Dec 2019.
  * [27] Francisco Torres-Herrador, Joffrey Coheur, Francesco Panerai, Thierry E. Magin, Maarten Arnst, Nagi N. Mansour, and Julien Blondeau.  Competitive kinetic model for the pyrolysis of the Phenolic Impregnated Carbon Ablator.  Aerospace Science and Technology, 100:105784, May 2020.
  * [28] Francisco Torres-Herrador, Jeremie B.E. Meurisse, Francesco Panerai, Julien Blondeau, Jean Lachaud, Brody K. Bessire, Thierry E. Magin, and Nagi N. Mansour.  A high heating rate pyrolysis model for the Phenolic Impregnated Carbon Ablator (PICA) based on mass spectroscopy experiments.  Journal of Analytical and Applied Pyrolysis, 141:104625, Aug 2019\.
  * [29] J. Lachaud, J.B. Scoggins, T.E. Magin, M.G. Meyer, and N.N. Mansour.  A generic local thermal equilibrium model for porous reactive materials submitted to high temperatures.  International Journal of Heat and Mass Transfer, 108:1406–1417, May 2017.
  * [30] Jean Lachaud and Nagi N. Mansour.  Porous-Material Analysis Toolbox Based on OpenFOAM and Applications.  Journal of Thermophysics and Heat Transfer, 28(2):191–202, Apr 2014.
  * [31] Hrishikesh A Bale, Abdel Haboub, Alastair A MacDowell, James R Nasiatka, Dilworth Y Parkinson, Brian N Cox, David B Marshall, and Robert O Ritchie.  Real-time quantitative imaging of failure events in materials under load at temperatures above 1,600 ∘C.  Nature materials, 12(1):40–46, 2013.
  * [32] Alastair A MacDowell, Harold Barnard, Dilworth Y Parkinson, Abdel Haboub, Natalie Larson, Frank Zok, Francesco Panerai, Nagi N Mansour, Hrishikesh Bale, Bernd Gludovatz, et al.  High temperature x-ray micro-tomography.  In AIP conference proceedings, volume 1741, page 050005. AIP Publishing LLC, 2016.
  * [33] E. Maire and P. J. Withers.  Quantitative X-ray tomography.  International Materials Reviews, 59(1):1–43, Jan 2014.
  * [34] Harold S. Barnard, A.A Macdowell, D.Y. Parkinson, N.M. Larson, J.C. Peterson, F. Panerai, N.N. Mansour, and Y. Gao.  Synchrotron X-ray Micro Tomography at the Advanced Light Source: In-Situ Sample Environments for Advanced Aerospace Materials.  Microscopy and Microanalysis, 24(S2):444–445, August 2018.
  * [35] Hrishikesh Bale, Matthew Blacklock, Matthew R. Begley, David B. Marshall, Brian N. Cox, and Robert O. Ritchie.  Characterizing Three-Dimensional Textile Ceramic Composites Using Synchrotron X-Ray Micro-Computed-Tomography.  Journal of the American Ceramic Society, 95(1):392–402, 2012.
  * [36] Abdel Haboub, Hrishikesh A. Bale, James R. Nasiatka, Brian N. Cox, David B. Marshall, Robert O. Ritchie, and Alastair A. MacDowell.  Tensile testing of materials at high temperatures above 1700 ∘C with in situ synchrotron X-ray micro-tomography.  Review of Scientific Instruments, 85(8):083702, August 2014.
  * [37] Alastair A. MacDowell, Harold Barnard, Dilworth Y. Parkinson, Abdel Haboub, Natalie Larson, Frank Zok, Francesco Panerai, Nagi N. Mansour, Hrishikesh Bale, Bernd Gludovatz, Claire Acevedo, Dong Liu, and Robert O. Ritchie.  High temperature x-ray micro-tomography.  page 050005, New York, NY USA, 2016.


$\varepsilon\in[0,\alpha/2)$. Then there exists $\xi\in\\{+1,-1\\}$ such that
for any $t\in\\{1,2,\dots,\ell\\}$, $\lVert u^{\otimes t}-\xi^{t}v^{\otimes
t}\rVert_{F}\leq\sqrt{2}\varepsilon/\alpha$. Also $|\alpha-\beta|\leq
3\varepsilon$.

We remark that if $\ell$ is odd, we can additionally conclude that $\xi=+1$,
but this is not used in the arguments, so we skip its proof.

###### Proof.

Suppose $A_{1}=u^{\otimes t},B_{1}=v^{\otimes t}$ and
$A_{2}=u^{\otimes\ell-t},B_{2}=v^{\otimes\ell-t}$. Note that they all have
unit norm. Let $\eta=\min\\{\lVert A_{1}-B_{1}\rVert_{F},\lVert
A_{1}+B_{1}\rVert_{F}\\}$. Then
$A_{1}=\sqrt{1-\eta^{2}/2}B_{1}+\tfrac{\eta}{\sqrt{2}}B_{1}^{\perp}$ for some
$B_{1}^{\perp}$ with unit norm orthogonal to $B_{1}$. We have

$\displaystyle\alpha u^{\otimes\ell}-\beta v^{\otimes\ell}$
$\displaystyle=\alpha A_{1}\otimes A_{2}-\beta B_{1}\otimes
B_{2}=\alpha\sqrt{1-\tfrac{\eta^{2}}{2}}B_{1}\otimes
A_{2}+\alpha\cdot\tfrac{\eta}{\sqrt{2}}B_{1}^{\perp}\otimes A_{2}-\beta
B_{1}\otimes B_{2}$ $\displaystyle\text{Hence }\varepsilon^{2}$
$\displaystyle=\lVert\alpha u^{\otimes\ell}-\beta
v^{\otimes\ell}\rVert_{F}^{2}\geq\frac{\alpha^{2}\eta^{2}}{2}\lVert
B_{1}^{\perp}\otimes
A_{2}\rVert_{F}^{2}\geq\frac{\alpha^{2}\eta^{2}}{2}~{}~{}(\text{ since
}B_{1}\perp B_{1}^{\perp}).$

Hence $\eta^{2}\leq 2\varepsilon^{2}/\alpha^{2}$. For even $t$, it is easy to
see that $\lVert u^{\otimes t}-v^{\otimes t}\rVert_{F}\leq\lVert u^{\otimes
t}+v^{\otimes t}\rVert_{F}$; hence $\lVert u^{\otimes t}-v^{\otimes
t}\rVert_{F}\leq\sqrt{2}\varepsilon/\alpha$. For odd $t$, it could be either
$\lVert u^{\otimes t}-v^{\otimes t}\rVert_{F}=\eta$ or $\lVert u^{\otimes
t}-v^{\otimes t}\rVert_{F}=\eta$; moreover the sign (in front of $v^{\otimes
t}$) is coordinated across the different $t$ since $\lVert u^{\otimes
t}-v^{\otimes t}\rVert_{F}^{2}+\lVert u^{\otimes t}-v^{\otimes
t}\rVert_{F}^{2}=2$. Hence for an appropriate sign $\xi\in\\{+1,1\\}$ we have
$\lVert u^{\otimes t}-\xi^{t}v^{\otimes t}\rVert_{F}=\eta$.

Finally, to give an upper bound on $|\alpha-\beta|$, we use the conclusion
from the above bound with $t=1$, to argue that for some
$v^{\perp}\in\mathbb{S}^{d-1}$ that is orthogonal to $v$

$\displaystyle\varepsilon^{2}$
$\displaystyle\geq\Big{\lVert}\alpha\big{(}\sqrt{1-\tfrac{\eta^{2}}{2}}\cdot
v+\tfrac{\eta}{\sqrt{2}}v^{\perp}\big{)}^{\otimes\ell}-\beta
v^{\otimes\ell}\Big{\rVert}_{F}^{2}$
$\displaystyle=\Big{|}\alpha\big{(}1-\frac{\eta^{2}}{2}\big{)}^{\ell/2}-\beta\Big{|}^{2}+\alpha^{2}\Big{(}1-\big{(}1-\frac{\eta^{2}}{2}\big{)}^{\ell}\Big{)}.$

Since $\eta^{2}\in(0,1)$, we can use a simple linear approximation to claim
that $t\eta^{2}/4\leq|1-(1-\eta^{2}/2)^{t}|\leq t\eta^{2}$ for any $t>0$.
Hence, we get that

$\displaystyle\varepsilon^{2}$
$\displaystyle\geq(|\alpha-\beta|-|\alpha|\ell\eta^{2})^{2}+\alpha^{2}\big{(}\tfrac{1}{4}\ell\eta^{2}\big{)}.$
$\displaystyle\text{Hence }|\alpha-\beta|$
$\displaystyle\leq\varepsilon+|\alpha|\sqrt{\ell}\eta,\text{ and
}|\alpha|\sqrt{\ell}\eta\leq 2\varepsilon.$

Hence the claim follows.

∎

We now proceed to the proof of Lemma 5.3.

###### Proof of Lemma 5.3.

The proof proceeds by first identifying a tensor $\widetilde{T}$ which we show
satisfies all the conditions for Jennrich’s robust algorithmic guarantee
(Theorem 1.5) with rank $k^{\prime}$, which corresponds to the $k^{\prime}$-th
largest $|\lambda_{i}|$. Note that the recovery error in the rank-$1$ terms
may be larger than some of the $k^{\prime}$ terms of $\widetilde{T}$ (for
example if there is not much separation between the $k^{\prime}$ largest and
$(k^{\prime}+1)$th largest of the $\\{|\lambda_{i}|\\}$). Therefore, we argue
that if $|\lambda_{i}|$ is sufficiently large, it will be recovered up to
small error.

We start with some notation. Suppose $s_{\min}\coloneqq s_{m}(W^{\odot\ell})$.
Let $\widetilde{M}=\text{flatten}(\widetilde{T},\ell_{1},\ell_{2}+\ell_{3},0)$
and $M=\text{flatten}(T,\ell_{1},\ell_{2}+\ell_{3},0)$. Set
$U=W^{\odot\ell_{1}},V=W^{\odot\ell_{2}},Y=\text{diag}(\lambda)Z=\text{diag}(\lambda)W^{\odot\ell_{3}}$.
Recall that $k^{\prime}\coloneqq\operatorname*{argmax}_{r\leq
m}s_{r}(\widetilde{M})>\eta_{1}$. Note that $M=U\text{diag}(\lambda)(V\odot
Y)^{\top}$, where $\lambda=(\lambda_{1},\dots,\lambda_{m})$. We remark that by
Claim 1.4

$s_{m}(U)\geq\frac{s_{m}(W^{\odot\ell})}{(2m)^{\ell_{1}-\ell}}\geq\frac{s_{\min}}{(2m)^{\ell_{1}}},~{}~{}\text{
and similarly
}~{}s_{m}(V)\geq\frac{s_{\min}}{(2m)^{\ell_{2}}},~{}~{}s_{m}(V\odot
Z)\geq\frac{s_{\min}}{(2m)^{\ell_{2}+\ell_{3}}}.$ (72)

We first argue that there are at least $k^{\prime}$ values of $|\lambda_{i}|$
that are non-negligible. Since $\lVert
T-\widetilde{T}\rVert_{F}\leq\eta^{\prime}_{1}<\eta_{1}/2$, we have from
Weyl’s inequality that $s_{k^{\prime}}(M)>\eta_{1}/2$. Let $S\subset[m]$
denote the indices corresponding to the largest $k^{\prime}$ values of
$|\lambda_{i}|$ (this is for analysis). The rank-1 terms restricted to $S$
will constitute the “ground-truth” decomposition $\widetilde{T}$. We first
observe that

$\displaystyle\min_{i\in S}|\lambda_{i}|$
$\displaystyle>\frac{\eta_{1}}{2m},~{}~{}~{}\text{ and }$ (73)
$\displaystyle\forall i\in[m]~{}\text{ s.t. }~{}|\lambda_{i}|$
$\displaystyle\geq\frac{\eta_{1}(2m)^{\ell_{1}+\ell_{2}+\ell_{3}}}{s_{\min}^{2}},~{}~{}\text{
we have }~{}~{}i\in S.$ (74)

To see why (73) holds, note that

$\frac{\eta_{1}}{2}<s_{k^{\prime}}(M)=s_{k^{\prime}}\Big{(}U\text{diag}(\lambda)(V\odot
Y)^{\top}\Big{)}\leq s_{k^{\prime}}(\text{diag}(\lambda))\cdot s_{1}(U)\cdot
s_{1}(V\odot Y)\leq m\cdot\min_{i\in S}|\lambda_{i}|,$

where we used the fact that all the columns of $U$ and $V\odot Z$ are unit
vectors. To show (74), suppose we assume for contradiction that
$|\lambda_{i}|>2(2m)^{\ell_{1}+\ell_{2}+\ell_{3}}\eta_{1}/s_{\min}^{2}$, but
$i\notin S$. Let $S^{\prime}=S\cup\\{i\\}$. Then we can see that at least
$k^{\prime}+1$ singular values of $\widetilde{M}$ are greater than $\eta_{1}$
since by Weyl’s inequality,

$\displaystyle s_{k^{\prime}+1}(\widetilde{M})$ $\displaystyle\geq
s_{k^{\prime}+1}(M)-\eta_{1}^{\prime}=s_{k^{\prime}+1}\Big{(}U\text{diag}(\lambda)(V\odot
W)^{\top}\Big{)}\eta^{\prime}_{1}$ $\displaystyle\geq s_{m}(U)\cdot
s_{k^{\prime}+1}(\text{diag}(\lambda))\cdot s_{m}(V\odot
W)\geq\frac{s_{\min}}{(2m)^{\ell_{1}}}\cdot|\lambda_{i}|\cdot\frac{s_{\min}}{(2m)^{\ell_{2}}}$
$\displaystyle\geq 2\eta_{1}-\eta_{1}^{\prime}>\eta_{1}.$

Hence (73) and (74) are both true.

We now argue that we satisfy the requirements of Theorem 1.5 (the robust
guarantee). Let $U_{S},V_{S}$ and $Y_{S}$ denote the restriction of the factor
matrices $U,V,Y$ to the columns corresponding to $S$. Then by Claim 1.4

$s_{k^{\prime}}(U)\geq
s_{m}(U)\geq\frac{s_{m}(W^{\odot\ell})}{(2m)^{\ell_{1}-\ell}}\geq\frac{s_{\min}}{(2m)^{\ell_{1}}},\text{
and }s_{k^{\prime}}(U)\geq
s_{m}(U)\geq\frac{s_{m}(W^{\odot\ell})}{(2m)^{\ell_{2}-\ell}}\geq\frac{s_{\min}}{(2m)^{\ell_{2}}}.$

Moreover for any two columns $i,j\in S$, we have that the restriction of $Z$
to these two columns $Y_{\\{i,j\\}}$ satisfies

$s_{2}(Y_{\\{i,j\\}})\geq\min\\{|\lambda_{i}|,|\lambda_{j}|\\}\cdot
s_{2}(W_{\\{i,j\\}})\geq\frac{\eta_{1}\cdot s_{\min}}{\sqrt{2\ell}\cdot m}.$

Moreover the maximum singular values of the factor matrices $U,V$ are all
upper bounded by $\sqrt{m}$.

Finally, suppose $T_{S}=\sum_{i\in S}\lambda_{i}u_{i}\otimes v_{i}\otimes
z_{i}$, then the error between the input tensor and $T_{S}$

$\displaystyle\lVert\widetilde{T}-T_{S}\rVert_{F}$
$\displaystyle\leq\lVert\widetilde{T}-T\rVert_{F}+\lVert
T-T_{S}\rVert_{F}\leq\eta^{\prime}_{1}+\Big{\lVert}\sum_{i\notin
S}\lambda_{i}u_{i}\otimes v_{i}\otimes z_{i}\Big{\rVert}_{F}$
$\displaystyle=\eta^{\prime}_{1}+\Big{\lVert}\text{flatten}\Big{(}\sum_{i\notin
S}\lambda_{i}u_{i}\otimes v_{i}\otimes
z_{i},\ell_{1},\ell_{2}+\ell_{3},0\Big{)}\Big{\rVert}_{F}$
$\displaystyle\leq\eta^{\prime}_{1}+\sqrt{m}s_{1}\Big{(}\text{flatten}\big{(}\sum_{i\notin
S}\lambda_{i}u_{i}\otimes v_{i}\otimes
z_{i},\ell_{1},\ell_{2}+\ell_{3},0\big{)}\Big{)}$
$\displaystyle\leq\eta^{\prime}_{1}+\sqrt{m}\eta_{1}\leq(\sqrt{m}+1)\eta_{1}.$

Now applying Theorem 1.5 with
$\varepsilon_{\ref{thm:jennrich}}=\varepsilon_{2}^{2}$, and setting
$\eta^{\prime}_{1}$ such that
$\eta^{\prime}_{1}=\min\Big{\\{}\eta_{\ref{thm:jennrich}}\big{(}\varepsilon_{\ref{thm:jennrich}}=\varepsilon_{2}^{2},\kappa=\tfrac{\sqrt{m}(2m)^{\ell_{1}+\ell_{2}}}{s_{\min}},d^{\ell_{1}+\ell_{2}+\ell_{3}},\delta=\tfrac{\eta_{1}\cdot
s_{\min}}{\sqrt{2\ell}\cdot m}\big{)}/(\sqrt{m}+1),\eta_{1}/2\Big{\\}}$, we
have that the rank-1 terms can be recovered up to accuracy
$\varepsilon_{2}^{2}$ (up to renaming the indices $i\in[m]$):

$\displaystyle\forall i\in S,~{}~{}\lVert\lambda_{i}u_{i}\otimes v_{i}\otimes
z_{i}-\widetilde{\lambda}_{i}\widetilde{u}_{i}\otimes\widetilde{v}_{i}\otimes\widetilde{z}_{i}\rVert_{F}$
$\displaystyle\leq\varepsilon_{2}^{2}.$ (75) $\displaystyle\text{From
Claim~{}\ref{claim:rank1vectors} },\forall i\in S,~{}~{}\lVert
w_{i}-\xi_{i}\widetilde{w}_{i}\rVert_{2}$
$\displaystyle\leq\frac{\varepsilon_{2}^{2}}{|\lambda_{i}|},$ (76)

for appropriate signs $\xi_{i}\in\\{+1,-1\\}$. We remark that the choice of
$\eta^{\prime}_{1}$ is consistent with both Theorem 1.5 and this lemma, since
in our case $\kappa\leq\mathrm{poly}_{\ell}(m)/s_{\min}$ and
$1/\delta\leq\mathrm{poly}_{\ell}(m,1/\eta_{1})/s_{\min}$. If $s_{\min}>0$
becomes too small, we will directly set $\eta^{\prime}_{1}$ as $\eta_{1}/2$
instead.

Note that from (75) and triangle inequality, we already have for $i\in S$

$|\lambda_{i}-\widetilde{\lambda}_{i}|=\Big{|}\lVert\lambda_{i}u_{i}\otimes
v_{i}\otimes
z_{i}\rVert_{F}-\lVert\widetilde{\lambda}_{i}\widetilde{u}_{i}\otimes\widetilde{v}_{i}\otimes\widetilde{z}_{i}\rVert_{F}\Big{|}\leq\lVert\lambda_{i}u_{i}\otimes
v_{i}\otimes
z_{i}-\widetilde{\lambda}_{i}\widetilde{u}_{i}\otimes\widetilde{v}_{i}\otimes\widetilde{z}_{i}\rVert_{F}\leq\varepsilon_{2}^{2}.$

(For terms not in $S$, the output $\widetilde{\lambda_{i}}=0$ and
$|\lambda_{i}|<\eta_{1}/(2m)$, hence it is still satisfied). For any $i\in[m]$
s.t., $|\lambda_{i}|>\varepsilon_{1}$, we have from (74) that $i\in S$; hence

$\displaystyle\text{For all }i~{}\text{ s.t.
}|\lambda_{i}|>\varepsilon_{1},~{}~{}~{}\lVert
w_{i}-\xi_{i}\widetilde{w}_{i}\rVert_{2}$
$\displaystyle\leq\frac{\varepsilon_{2}^{2}}{\varepsilon_{1}}\leq\varepsilon_{2},$

as long as $|\varepsilon_{1}|\geq\sqrt{\varepsilon_{2}}$. A similar proof also
holds for $\lVert w_{i}^{\otimes t}-\xi_{i}^{t}\widetilde{w}_{i}^{\otimes
t}\rVert_{F}$ by using Claim 1.4 with general $t\geq 1$ in (76). This
completes the proof.

∎

##### Lemma 5.4

For any $\varepsilon_{2}>0$, there exists an
$\eta_{2}^{\prime}=\frac{\mathrm{poly}(\varepsilon_{2},s_{m}(W^{\odot\ell}))}{\mathrm{poly}_{\ell}(m,d,B)}>0$
such that if the estimates $\lVert
T_{k}-\hat{f}_{k}\rVert_{F}\leq\eta_{2}^{\prime}$ for all
$k\in\\{0,1,\dots,2\ell+2\\}$, then steps 1-4 of Algorithm 4 finds a set
$\\{\widetilde{w}_{i}:i\in[m^{\prime}]\\}$ such that there exists a one-to-one
map $\pi:[m^{\prime}]\to[m]$ satisfying (i) every $i\in G$ has a pre-image in
$\pi$ (i.e., every term in $G$ is recovered), and for appropriate signs
$\\{\xi_{i}\in\\{1,-1\\}:i\in[m^{\prime}]\\}$,

$\forall i\in[m^{\prime}],\forall
t\in[2\ell],~{}~{}\lVert\xi_{i}^{t}\widetilde{w}_{i}^{\otimes
t}-w_{\pi(i)}^{\otimes t}\rVert_{F}\leq\varepsilon_{2}.$ (77)

In particular $\forall i\in[m^{\prime}]$, we have
$\lVert\xi_{i}\widetilde{w}_{i}-w_{\pi(i)}\rVert_{2}\leq\varepsilon_{2}$.

The stronger guarantee for all $t\in[2\ell]$ will be useful in bounding the
recovery error of the $a_{i},b_{i}$ in later steps.

###### Proof of Lemma 5.4.

The proof uses the robust guarantees of Jennrich’s algorithm in Lemma 5.3
along with the crucial property of separation of roots in Lemma 4.3.

Set $\varepsilon^{\prime}_{1}\coloneqq\varepsilon_{1}/\sqrt{k!/2}$ and
$\varepsilon_{1}=\mathrm{poly}(\varepsilon_{2},m,d,s_{m}(W^{\odot\ell}))$ be
given by Lemma 5.3. Similarly $\eta^{\prime}_{2}$ is specified by the
requirement of Lemma 5.3. Set also $\eta_{2}=4\varepsilon_{1}$.

Consider a $\widetilde{w}_{i}$ output by the algorithm in step 4; and suppose
w.l.o.g. it was output in step 2. Then we have that
$|\widetilde{\lambda}_{i}|\geq\eta_{2}$. Further,
$|\widetilde{\lambda}_{i}-\lambda_{i}|\leq\varepsilon_{2}^{2}<\eta_{2}/4$.
Hence, for every term $i\in[m^{\prime}]$ that is output after step 4, we have
$|\lambda_{i}|>\eta_{2}/2\geq\varepsilon_{1}$.

We first argue that every term in $G$ is one of the $m^{\prime}$ terms output
by the algorithm in step 4. Consider the decompositions of the two tensors
obtained from the Hermite coefficients of $f$ i.e.,

$\displaystyle\hat{f}_{2\ell+1}$ $\displaystyle=\sum_{i=1}^{m}(-a_{i})\cdot
He_{2\ell-1}(b_{i})\cdot\frac{1}{\sqrt{2\pi}}\exp(-b_{i}^{2}/2)\cdot
w_{i}^{\otimes
2\ell+1}=\sum_{i=1}^{m}\lambda_{i}(w_{i}^{\otimes\ell})\otimes(w_{i}^{\otimes\ell})\otimes
w_{i}$ (78) $\displaystyle\hat{f}_{2\ell+2}$
$\displaystyle=\sum_{i=1}^{m}a_{i}\cdot
He_{2\ell}(b_{i})\cdot\frac{1}{\sqrt{2\pi}}\exp(-b_{i}^{2}/2)\cdot
w_{i}^{\otimes
2\ell+2}=\sum_{i=1}^{m}\lambda^{\prime}_{i}(w_{i}^{\otimes\ell})\otimes(w_{i}^{\otimes\ell})\otimes
w_{i}^{\otimes 2}.$ (79)

Note that from Lemma 4.3 we have that for every $x\in\mathbb{R}$, at least one
of $|He_{2\ell+2}(x)|,|He_{2\ell+1}(x)|$ is at least $\sqrt{k!/2}$. Moveover
since $i\in G$ for our choice of $c_{k}$ in (16), we have that
$e^{-b_{i}^{2}/2}/\sqrt{2\pi}>\varepsilon^{\prime}_{1}$. Hence for each $i\in
G$, we have that
$\max\\{|\lambda_{i}|,|\lambda^{\prime}_{i}|\\}>\varepsilon_{1}$.

Finally, we now prove that when $|\lambda_{i}|\geq\varepsilon_{1}$, the
corresponding $w_{i}$ is recovered up to error $\varepsilon_{2}$. From Lemma
5.3, if $\widetilde{w}_{i}$ is the vector output by one of the decompositions
for $w_{i}$, we have for all $t\in[2\ell]$ that $\lVert w_{i}^{\otimes
t}-\xi_{i}^{t}\widetilde{w}_{i}^{\otimes t}\rVert_{F}\leq\varepsilon_{2}$ for
some sign $\xi_{i}\in\\{1,-1\\}$ as required. Moreover since
$\eta_{2}\coloneqq\varepsilon_{1}/2$ and the error in each rank-1 term is at
most $\varepsilon_{\ref{thm:jennrich}}<\eta_{1}/2$, we have that none of these
terms are removed as duplicates of other terms. On the other hand, since
$\eta_{3}\coloneqq 2\varepsilon_{2}$, we have that duplicates are correctly
removed. Hence we have that for every $i\in[m^{\prime}]$, $\lVert
w_{i}^{\otimes t}-\xi_{i}^{t}\widetilde{w}_{i}^{\otimes
t}\rVert_{F}\leq\varepsilon$ for appropriate signs $\xi_{i}\in\\{\pm 1\\}$.

∎

####  B.2.2 Recovering error for the parameters $a_{i},b_{i}$ for terms $i\in
G$.

In this section we prove the following claim that shows the recovery of all
the $\\{a_{i},b_{i}:i\in G\\}$ (and in fact, all the terms output in steps 1-5
of Algorithm 4).

Before we start the proof of the main lemmas, we first show a key property of
Hermite polynomials we will utilize later.

######  Claim B.4.

$\forall x\in\mathbb{R}$, $|He_{k}(x)|\exp(-x^{2}/2)\leq\sqrt{k!}$

###### Proof.

We utilize Cramér’s inequality for Hermite functions that for all
$x\in\mathbb{R}$, $|\psi_{k}(x)|\leq\pi^{-1/4}$, where $\psi_{k}(x)$ is the
$k$’th Hermite function given by

$|\psi_{k}(x)|=(2^{k}k!\sqrt{\pi})^{-1/2}\exp(-x^{2}/2)|H_{k}(x)|$

with $H_{k}(x)$ denoting the $k$’th physicist’s Hermite polynomial999The
physicist’s Hermite polynomials are defined as
$H_{k}(x)=\frac{(-1)^{k}}{\exp(-x^{2})}\cdot\frac{d^{k}}{dx^{k}}\exp(-x^{2})$.
Now, substituting $x$ with $x/\sqrt{2}$ yields

$|\psi_{k}(\frac{x}{\sqrt{2}})|=(k!\sqrt{\pi})^{-1/2}\exp(-x^{2}/4)|He_{k}(x)|\leq\pi^{-1/4}$
$\Longrightarrow~{}~{}\exp(-x^{2}/2)|He_{k}(x)|\leq\sqrt{k!}\exp(-x^{2}/4)\leq\sqrt{k!}$

∎

With this claim, we are now ready to proceed.

##### Lemma 5.5

For $\varepsilon>0$ in the definition of $G$ in (16), there exists
$\eta_{3}^{\prime}=\frac{\mathrm{poly}(\varepsilon,s_{m}(W^{\otimes\ell}))}{\mathrm{poly}_{\ell}(m,d,B)}>0$,
and
$\varepsilon_{3}^{\prime}=\frac{\mathrm{poly}(\varepsilon,s_{m}(W^{\otimes\ell}))}{\mathrm{poly}_{\ell}(m,d,B)}>0$
such that for some $\xi_{i}\in\\{\pm 1\\}~{}\forall i\in[m^{\prime}]$

$\text{if}~{}~{}\lVert
T_{k}-\hat{f}_{k}\rVert_{F}\leq\eta_{3}^{\prime}~{}~{}\forall k\leq
2\ell+2,~{}~{}~{}~{}\text{ and }~{}~{}~{}\lVert\widetilde{w}_{i}^{\otimes
t}-\xi^{t}w_{i}^{\otimes t}\rVert_{F}\leq\varepsilon_{3}^{\prime},\forall
i\in[m^{\prime}],\forall\ell\leq t\leq\ell+3.$

then steps 5-6 of the algorithm finds
$(\widetilde{a}_{i},\widetilde{b}_{i}:i\in[m^{\prime}])$ such that

$|\widetilde{a}_{i}-a_{i}|\leq\varepsilon,\text{ and
}|\widetilde{b}_{i}-\xi_{i}b_{i}|\leq\varepsilon.$ (80)

The above uses Lemma 5.6, a robust version of Lemma 4.5 when there are errors
in the estimates. Also $\beta=\alpha e^{-z^{2}/2}$ in the notation of Lemma
4.5.

##### Lemma 5.6

(Robust version of Lemma 4.5 for $k\geq 2$) Suppose $k\in\mathbb{N},k\geq
2,B\geq 1$, and $\alpha,z\in\mathbb{R}$ be unknown parameters. There exists a
constant $c_{k}=c(k)\geq 1$ such that for any $\varepsilon\in(0,\tfrac{1}{4})$
satisfying (i) $|\alpha|\in[\tfrac{1}{B},B]$ and $|z|\leq B$, and (ii)
$|z|<2\sqrt{\log(c_{k}/\varepsilon^{1/4}(1+B)^{3}))}$ , if we are given values
$\gamma_{k},\gamma_{k+1},\gamma_{k+2},\gamma_{k+3}$ s.t. for some
$\xi\in\\{\pm 1\\}$,

$\Big{|}\gamma_{j}-\alpha\cdot\frac{(-1)^{j}}{\sqrt{2\pi}}e^{-z^{2}/2}He_{j}(\xi
z)\Big{|}\leq\varepsilon^{\prime}=\frac{\varepsilon^{4}}{2(1+B)^{2}}~{}~{}~{}\forall
j\in\\{k,k+1,k+2,k+3\\},$

then the estimates $\widetilde{z},\widetilde{\alpha}$ obtained as:

$\displaystyle\widetilde{z}=-\frac{\gamma_{q+1}+q\cdot\gamma_{q-1}}{\gamma_{q}}~{}~{}\text{where
}q:=\operatorname*{argmax}_{\begin{subarray}{c}j\in\\{k+1,k+2\\}\end{subarray}}|\gamma_{j}|,~{}~{}\text{and}~{}~{}\widetilde{\alpha}$
$\displaystyle=(-1)^{q}\frac{\sqrt{2\pi}\gamma_{q}}{e^{-\frac{\widetilde{z}^{2}}{2}}He_{q}(\widetilde{z})}$
$\displaystyle\text{satisfy}~{}~{}~{}\big{|}\widetilde{z}-\xi
z\big{|}\leq\frac{\varepsilon|z|}{B+1}\leq\varepsilon,~{}~{}\text{ and
}~{}~{}\big{|}\widetilde{\alpha}-\alpha\big{|}$
$\displaystyle\leq\varepsilon.$ (81)

###### Proof.

Set $\varepsilon^{\prime}\coloneqq\varepsilon^{4}/(2(1+B)^{2})$, and let
$\beta:=\alpha e^{-z^{2}/2}/\sqrt{2\pi}$. Under our assumptions
$|\beta|>c_{k}(\varepsilon^{\prime})^{1/4}(1+B)^{2}$. We use the following
fact about Hermite polynomials:

$He_{r+1}(z)=zHe_{r}(z)-r\cdot He_{r-1}(z).$ (82)

For convenience, for a scalar quantity $v$ we denote by $v=a\pm\varepsilon$
iff $|v-a|\leq\varepsilon$. Recall that
$q=\operatorname*{argmax}_{\begin{subarray}{c}j\in\\{k+1,k+2\\}\end{subarray}}|\gamma_{j}|$.
Since $\beta\sqrt{k!/2}>4\varepsilon^{\prime}$ by the conditions, we have that
$|\gamma_{q}|>\beta\sqrt{k!}/2$.

Setting $r=q$ in (82) and dividing by $He_{q}(z)$ we get using its odd or even
function property depending on parity of $q$,

$\displaystyle\xi z$
$\displaystyle=\xi\cdot\frac{He_{q+1}(z)+qHe_{q-1}(z)}{He_{q}(z)}=\frac{\beta
He_{q+1}(\xi z)+q\cdot\beta He_{q-1}(\xi z)}{\beta He_{q}(\xi z)}$
$\displaystyle=-\frac{(\gamma_{q+1}\pm\varepsilon^{\prime})+q(\gamma_{q-1}\pm\varepsilon^{\prime})}{\gamma_{q}\pm\varepsilon^{\prime}}=-\frac{\gamma_{q+1}+q\gamma_{q-1}}{\gamma_{q}(1\pm\tfrac{\varepsilon^{\prime}}{\gamma_{q}})}\pm\frac{(k+3)\varepsilon^{\prime}}{\gamma_{q}(1\pm\tfrac{\varepsilon^{\prime}}{\gamma_{q}})}$
$\displaystyle=-\frac{\gamma_{q+1}+q\cdot\gamma_{q-1}}{\gamma_{q}}\cdot\Big{(}1\pm\frac{2\varepsilon^{\prime}}{|\gamma_{q}|}\Big{)}\pm\frac{(k+3)\varepsilon^{\prime}}{\gamma_{q}(1\pm\tfrac{\varepsilon^{\prime}}{|\gamma_{q}|})}$
$\displaystyle=\widetilde{z}(1\pm\sqrt{\varepsilon^{\prime}})\pm\sqrt{\varepsilon^{\prime}},~{}~{}~{}~{}~{}~{}\text{
since }\beta\sqrt{k!}/2\geq 2(k+3)\sqrt{\varepsilon^{\prime}}.$

Let $g(z)=e^{-z^{2}/2}He_{q}(z)/\sqrt{2\pi}$. Hermite polynomials satisfy
$He^{\prime}_{q}(x)=qHe_{q-1}(x)$. Hence

$g^{\prime}(z)=e^{-z^{2}/2}\big{(}qHe_{q-1}(z)-zHe_{q}(z)\big{)}/\sqrt{2\pi}=-\frac{e^{-z^{2}/2}He_{q+1}(z)}{\sqrt{2\pi}},$

by applying (82). Also, by Claim B.4,
$\max_{z^{\prime}}|g^{\prime}(z^{\prime})|\leq\max_{z^{\prime}}e^{-z^{\prime
2}/2}|He_{q+1}(z^{\prime})|/\sqrt{2\pi}\leq(q+1)!$. Hence,

$|g(\widetilde{z})-g(\xi z)|\leq\max_{z^{\prime}\in[z,\widetilde{z}]\cup
z^{\prime}\in[\widetilde{z},z]}|g^{\prime}(z)||\widetilde{z}-\xi z|\leq
4(q+1)!(1+B)\sqrt{\varepsilon^{\prime}}$

Plugging these error bounds into $\alpha$, and using $He_{q}(\xi
z)=\xi^{q}He_{q}(x)$ we have

$\displaystyle\widetilde{\alpha}$
$\displaystyle=\frac{\gamma_{q}}{g(\widetilde{z})}=\frac{\alpha\xi^{q}\cdot\tfrac{e^{-z^{2}/2}}{\sqrt{2\pi}}He_{q}(z)\pm\varepsilon^{\prime}}{\tfrac{e^{-z^{2}/2}He_{q}(\xi
z)}{\sqrt{2\pi}}\pm|g(\widetilde{z})-g(\xi z)|}$ $\displaystyle=\frac{\alpha
g(z)\pm\varepsilon^{\prime}}{g(z)\pm
4(q+1)!(1+B)\sqrt{\varepsilon^{\prime}}}=\alpha\Big{(}1\pm\frac{8(q+1)!(1+B)\sqrt{\varepsilon^{\prime}}}{|g(z)|}\Big{)}\pm\frac{2\varepsilon^{\prime}}{|g(z)|}$
$\displaystyle|g(z)|$ $\displaystyle=\frac{|\beta
He_{q}(z)|}{|\alpha|}\geq\frac{|\beta|\sqrt{k!}}{2B}>c_{k}(1+B)^{2}(\varepsilon^{\prime})^{1/4}$
$\displaystyle\text{Hence }\big{|}\widetilde{\alpha}-\alpha\big{|}$
$\displaystyle\leq
8|\alpha|\times\frac{(q+1)!(1+B)\sqrt{\varepsilon^{\prime}}}{c_{k}(1+B)^{2}(\varepsilon^{\prime})^{1/4}}+\frac{2\varepsilon^{\prime}}{c_{k}(1+B)^{2}\varepsilon^{\prime
1/4}}\leq(\varepsilon^{\prime})^{1/4}\leq\varepsilon,$

because of our choice of $c_{k}=16(k+3)!$.

∎

The simpler variant of the above lemma (Lemma 5.6) for $k=1$ follows a very
similar analysis.

##### Lemma 5.7

(Robust version of Lemma 4.5 for $k=1$) Suppose $B\geq 1$, and
$\alpha,z\in\mathbb{R}$ be unknowns. There exists a constant $c\geq 1$ such
that for any $\varepsilon\in(0,\tfrac{1}{4})$ satisfying (i)
$|\alpha|\in[\tfrac{1}{B},B]$ and $|z|\leq B$, and (ii)
$|z|<2\sqrt{\log(c/\varepsilon^{1/4}(1+B)^{3}))}$, if we are given values
$\gamma_{0},\gamma_{1}$ s.t. for some $\xi\in\\{\pm 1\\}$,

$\Big{|}\gamma_{j}-\alpha\cdot\frac{(-1)^{j}}{\sqrt{2\pi}}e^{-z^{2}/2}He_{j}(\xi
z)\Big{|}\leq\varepsilon^{\prime}=\frac{\varepsilon^{4}}{2(1+B)^{2}}~{}~{}~{}\forall
j\in\\{0,1\\},$

then the estimates $\widetilde{z},\widetilde{\alpha}$ obtained as:

$\displaystyle\widetilde{z}$
$\displaystyle=-\frac{\gamma_{1}}{\gamma_{0}},~{}~{}\text{ and
}~{}~{}\widetilde{\alpha}=\frac{\sqrt{2\pi}\gamma_{0}}{e^{-\frac{\widetilde{z}^{2}}{2}}}$
$\displaystyle\text{satisfy}~{}~{}~{}\big{|}\widetilde{z}-\xi
z\big{|}\leq\frac{\varepsilon|z|}{B+1}$
$\displaystyle\leq\varepsilon,~{}~{}\text{ and
}~{}~{}\big{|}\widetilde{\alpha}-\alpha\big{|}\leq\varepsilon.$ (83)

Note that $He_{0}(z)=1$ and $He_{1}(z)=z$ to see the similarities between
Lemma 5.6 and Lemma 5.7

###### Proof.

Set $\varepsilon^{\prime}\coloneqq\varepsilon^{4}/(2(1+B)^{2})$, and let
$\beta:=\alpha e^{-z^{2}/2}/\sqrt{2\pi}$. Under our assumptions
$|\beta|>c(\varepsilon^{\prime})^{1/4}(1+B)^{2}$. For convenience we denote
for a scalar $v$, $v=a\pm\varepsilon$ iff $|v-a|\leq\varepsilon$.

Since $\beta>4\varepsilon^{\prime}$ by the conditions, we have that
$|\gamma_{0}|>\beta$. Recall that $He_{0}(z)=1$ and $He_{1}(z)=z$. Hence,

$\displaystyle\xi z$
$\displaystyle=\xi\cdot\frac{He_{1}(z)}{He_{0}(z)}=\frac{\beta He_{1}(\xi
z)}{\beta He_{0}(z)}$
$\displaystyle=-\frac{(\gamma_{1}\pm\varepsilon^{\prime})}{\gamma_{0}\pm\varepsilon^{\prime}}=-\frac{\gamma_{1}\pm\varepsilon^{\prime}}{\gamma_{0}(1\pm\tfrac{\varepsilon^{\prime}}{\gamma_{0}})}=-\frac{\gamma_{1}}{\gamma_{0}}\cdot\Big{(}1\pm\frac{2\varepsilon^{\prime}}{|\gamma_{0}|}\Big{)}\pm\frac{\varepsilon^{\prime}}{\gamma_{0}(1\pm\tfrac{\varepsilon^{\prime}}{|\gamma_{0}|})}$
$\displaystyle=\widetilde{z}(1\pm\sqrt{\varepsilon^{\prime}})\pm\sqrt{\varepsilon^{\prime}},~{}~{}~{}~{}~{}~{}\text{
since }\beta\geq 8\sqrt{\varepsilon^{\prime}}.$

To argue about $|\widetilde{\alpha}-\alpha|$, let
$g(z)=e^{-z^{2}/2}/\sqrt{2\pi}$. Its derivative $g^{\prime}(z)$ satisfies by
Claim B.4,
$\max_{z^{\prime}}|g^{\prime}(z^{\prime})|\leq\max_{z^{\prime}}|z^{\prime}|e^{-z^{\prime
2}/2}/\sqrt{2\pi}\leq 1$. Hence,

$|g(\widetilde{z})-g(\xi z)|\leq\max_{z^{\prime}\in[z,\widetilde{z}]\cup
z^{\prime}\in[\widetilde{z},z]}|g^{\prime}(z)||\widetilde{z}-\xi z|\leq
4(1+B)\sqrt{\varepsilon^{\prime}}$

Plugging these error bounds into $\alpha$ we have

$\displaystyle\widetilde{\alpha}$
$\displaystyle=\frac{\gamma_{0}}{g(\widetilde{z})}=\frac{\alpha\cdot\tfrac{e^{-z^{2}/2}}{\sqrt{2\pi}}\pm\varepsilon^{\prime}}{\tfrac{e^{-z^{2}/2}}{\sqrt{2\pi}}\pm|g(\widetilde{z})-g(\xi
z)|}$ $\displaystyle=\frac{\alpha g(z)\pm\varepsilon^{\prime}}{g(z)\pm
4(1+B)\sqrt{\varepsilon^{\prime}}}=\alpha\Big{(}1\pm\frac{8(1+B)\sqrt{\varepsilon^{\prime}}}{g(z)}\Big{)}\pm\frac{2\varepsilon^{\prime}}{g(z)}$
$\displaystyle g(z)$
$\displaystyle=\frac{\beta}{\alpha}\geq\frac{|\beta|}{B}>c(1+B)^{2}(\varepsilon^{\prime})^{1/4}$
$\displaystyle\text{Hence }\big{|}\widetilde{\alpha}-\alpha\big{|}$
$\displaystyle\leq
8|\alpha|\times\frac{(1+B)\sqrt{\varepsilon^{\prime}}}{c(1+B)^{2}(\varepsilon^{\prime})^{1/4}}+\frac{2\varepsilon^{\prime}}{c(1+B)^{2}\varepsilon^{\prime
1/4}}\leq(\varepsilon^{\prime})^{1/4}\leq\varepsilon,$

because of our choice of $c=16$.

∎

We now prove Lemma 5.5.

###### Proof of Lemma 5.5.

Set
$\varepsilon_{3}=\varepsilon^{4}s_{m}(W^{\otimes\ell}/(16m^{3/2}(1+B)^{2})$.
For each of the $j\in\\{\ell,\ell+1,\ell+2,\ell+2\\}$, we have from Lemma 3.5
that

$\hat{f}_{j}=\sum_{i=1}^{m}(-1)^{j}\cdot a_{i}\cdot
He_{j-2}(b_{i})\cdot\frac{\exp(\frac{-b_{i}^{2}}{2})}{\sqrt{2\pi}}\cdot
w_{i}^{\otimes j}.$

Moreover $\lVert T_{j}-\hat{f}_{j}\rVert_{F}\leq\eta^{\prime}_{3}$. Also for
the terms $i\notin G$, we have for each $\ell\leq j\leq\ell+3$, the
corresponding term

$\displaystyle\Big{|}a_{i}\cdot\frac{\exp(-b_{i}^{2}/2)}{2\pi}\cdot
He_{j-2}(b_{i})\Big{|}\leq B\cdot(\varepsilon mdB)^{c_{\ell}^{2}/2}$
$\displaystyle<\frac{\varepsilon_{3}}{2m}.$ $\displaystyle\text{ Hence
}\Big{\lVert}T_{j}-\sum_{i\in G}(-1)^{j}\cdot a_{i}\cdot
He_{j-2}(b_{i})\cdot\frac{\exp(\frac{-b_{i}^{2}}{2})}{\sqrt{2\pi}}\cdot
w_{i}^{\otimes j}\Big{\rVert}_{F}$
$\displaystyle\leq\varepsilon^{\prime}_{3}+m\cdot\frac{\varepsilon_{3}}{2m}\leq\varepsilon_{3},$

where the first line follows from our choice of $\varepsilon_{3}$ and our
choice of $c_{\ell}$.

Let $m^{\prime}\coloneqq|G|$. Next we establish that the linear system is
well-conditioned for each $\ell\leq j\leq\ell+3$. For any signs
$\xi_{i}\in\\{\pm 1\\}~{}\forall i\in[m]$, the matrix formed by the vectors
$\\{\xi_{i}^{\ell}w_{i}^{\otimes\ell}:i\in G\\}$ has non-negligible least
singular value. Moreover from Claim 1.4 (applied three times), we have for
$\ell\leq j\leq\ell+3$, we have the matrix formed by columns
$\\{\xi_{i}^{j}w_{i}^{\otimes j}:i\in G\\}$ has least singular value
$s_{m^{\prime}}\big{(}(w_{i}^{\otimes j}:i\in G)\big{)}\geq
s_{m}(W^{\otimes\ell})/(2m)^{3/2}$. Suppose
$M_{j},\widetilde{M}_{j}\in\mathbb{R}^{d^{j}\times m^{\prime}}$ be the
matrices with the $i$th columns $(\xi_{i}w_{i})^{\otimes j}$ and
$\widetilde{w}_{i}^{\otimes j}$ respectively for $i\in G$. Then by Weyl’s
inequality, we have

$\displaystyle s_{m^{\prime}}(\widetilde{M}_{j})$ $\displaystyle\geq
s_{m^{\prime}}(M_{j})-\lVert\widetilde{M}_{j}-M_{j}\rVert_{F}\geq\frac{s_{m}(W^{\otimes\ell})}{(2m)^{3/2}}-\sum_{i\in
G}\lVert\widetilde{w}_{i}^{\otimes j}-\xi_{i}w_{i}^{\otimes j}\rVert_{F}$
$\displaystyle\geq\frac{s_{m}(W^{\otimes\ell})}{(2m)^{3/2}}-m\cdot\varepsilon_{3}^{\prime}\geq\frac{s_{m}(W^{\otimes\ell})}{4m^{3/2}},~{}~{}~{}\text{
since }\varepsilon_{3}<\frac{s_{m}(W^{\otimes\ell})}{4m^{2}}.$

The target solution to the linear system for each $\ell\leq j\leq\ell+3$

$\forall i\in G,~{}~{}\zeta^{*}_{j}(i)\coloneqq
a_{i}\cdot(-1)^{j}\xi_{i}^{j}\cdot\frac{\exp(\frac{-b_{i}^{2}}{2})}{\sqrt{2\pi}}~{}He_{j-2}(b_{i}).$

Note that since for any $j$, $\sup_{z\in\mathbb{R}}e^{-z^{2}/2}He_{j}(z)\leq
c^{\prime}_{j}$ for some bounded constant $c^{\prime}_{j}<\infty$. Now a
standard error analysis of the linear system yields (see e.g., [Bha97]) we
have for all $\ell\leq j\leq\ell+3$,

$\displaystyle\lVert\zeta_{j}-\zeta^{*}_{j}\rVert_{2}$
$\displaystyle\leq\Big{(}s_{m^{\prime}}(\widetilde{M}_{j})\Big{)}^{-1}\Big{(}\varepsilon_{3}+\lVert
M_{j}-\widetilde{M}_{j}\rVert_{F}\lVert\zeta^{*}_{j}\rVert_{2}\Big{)}$
$\displaystyle\leq\frac{4m^{3/2}}{s_{m}(W^{\otimes\ell})}\big{(}\varepsilon_{3}+c^{\prime}_{j}\cdot
B\cdot\sqrt{m^{\prime}}\varepsilon^{\prime}_{3}\big{)}.$
$\displaystyle\text{Hence, }\forall i\in
G,~{}~{}\big{|}\zeta_{j}(i)-\zeta^{*}_{j}(i)\big{|}$
$\displaystyle\leq\frac{8m^{3/2}}{s_{m}(W^{\otimes\ell})}\cdot\varepsilon_{3}\leq\frac{\varepsilon^{4}}{2(1+B)^{2}}.$

since
$\varepsilon^{\prime}_{3}\leq\tfrac{1}{2}(c^{\prime}_{\ell+3}B\sqrt{m})\varepsilon_{3}$,
and for our choice of $\varepsilon$.

Finally we can now apply Lemma 5.6 for $\ell\geq 2$ or Lemma 5.7 for $\ell=1$
for each of the $i\in G$ separately with $\gamma_{j}=\zeta_{j}(i)$ (note that
the error is at most $\varepsilon^{\prime}$ as in Lemmas 5.6 and 5.7). The
output is $\widetilde{a}_{i}=\alpha,\widetilde{b}_{i}=z$ and conclude that
$|\widetilde{a}_{i}-a_{i}|\leq\varepsilon$, and
$|\widetilde{b}_{i}-\xi_{i}b_{i}|\leq\varepsilon$.

∎

###  B.3 Other claims for the robust analysis

The following claim shows how one can combine some of the activation units
output by the regression step to get a ReLU network with at most $|G|+2$
units.

######  Claim B.5.

Given a function $g(x)$ of the form

$g(x)=v^{\top}x+c+\sum_{i\in
m^{\prime}}\alpha_{i}\sigma(w_{i}^{\top}x+b_{i})+\alpha^{\prime}_{i}\sigma(-w_{i}^{\top}x-b_{i}),$
(84)

then $g(x)$ can be expressed as a ReLU network with at most $m^{\prime}+2$
activation units as

$g(x)=\beta_{0}\sigma({w_{0}}^{\top}x+b_{0})-\beta_{0}\sigma(-w_{0}^{\top}x-b_{0})+\sum_{i=1}^{m^{\prime}}\beta_{i}\sigma(w_{i}^{\top}x+b_{i}),$
(85)

where for each $i\in[m^{\prime}]$, $\beta_{i}=\alpha_{i}+\alpha^{\prime}_{i}$
and $w_{0}\in\mathbb{S}^{d-1},b_{0}\in\mathbb{R},\beta_{0}\in\mathbb{R}$
chosen to satisfy
$\beta_{0}w_{0}=v-\sum_{i=1}^{m^{\prime}}\alpha^{\prime}_{i}w_{i}$ and $\beta
b_{0}=c-\sum_{i-1}^{m^{\prime}}\alpha^{\prime}_{i}b_{i}$.

###### Proof.

First we note that for any $z\in\mathbb{R}$, $\sigma(z)=\tfrac{1}{2}(|z|+z)$
and $\sigma(-z)=\tfrac{1}{2}(|z|-z)$. Hence we have

$\gamma\sigma(z)+\gamma^{\prime}\sigma(-z)=\tfrac{1}{2}(\gamma+\gamma^{\prime})|z|+\tfrac{1}{2}(\gamma-\gamma^{\prime})z,\text{
and }z=\sigma(z)-\sigma(-z).$ (86)

Hence the terms are consolidated by replacing terms of the form
$\sigma(w_{i}^{\top}x+b_{i})$ and $\sigma(-w_{i}^{\top}x-b_{i})$ by one ReLU
unit so that the coefficient of $|w_{i}^{\top}x+b_{i}|$ match, along with a
linear term. All the linear terms are themselves consolidated together, and
replaced by a sum of two ReLU units. Now, substituing the setting of
$\beta_{i}$ $w_{0},b_{0}$ in (85) and simplifying, we have

$\displaystyle\beta_{0}\sigma({w_{0}}^{\top}x+b_{0})-\beta_{0}\sigma(-w_{0}^{\top}x-b_{0})+\sum_{i=1}^{m^{\prime}}\beta_{i}\sigma(w_{i}^{\top}x+b_{i})$
$\displaystyle=$
$\displaystyle\beta_{0}(w_{0}^{\top}x+b_{0})+\sum_{i=1}^{m^{\prime}}(\alpha_{i}+\alpha^{\prime}_{i})\cdot\frac{1}{2}\big{(}|w_{i}^{\top}x+b_{i}|+(w_{i}^{\top}x+b_{i})\big{)}$
$\displaystyle=$ $\displaystyle
v^{\top}x+c+\sum_{i=1}^{m^{\prime}}\frac{1}{2}(\alpha_{i}+\alpha^{\prime}_{i})|w_{i}^{\top}x+b_{i}|+\frac{1}{2}(\alpha_{i}-\alpha^{\prime}_{i})(w_{i}^{\top}x+b_{i})$
$\displaystyle=$ $\displaystyle
v^{\top}x+c+\sum_{i=1}^{m^{\prime}}\alpha_{i}\sigma(w_{i}^{\top}x+b_{i})+\alpha^{\prime}_{i}\sigma(-w_{i}^{\top}x-b_{i})=g(x),$

where the last line follows from (86) and (84). ∎