# Combining AFM imaging and elementally resolved spectro-electrochemistry for
understanding stability and quality of passive films formed on Alloy 600

Dominik Dworschak  Vienna University of Technology, Institute for Applied
Physics, Wiedner Hauptstrasse 8-10, A-1040 Vienna, Austria  Marina Bishara
Vienna University of Technology, Institute for Applied Physics, Wiedner
Hauptstrasse 8-10, A-1040 Vienna, Austria  Hsiu-Wei Cheng  hsiu-
<EMAIL_ADDRESS>Vienna University of Technology, Institute for Applied
Physics, Wiedner Hauptstrasse 8-10, A-1040 Vienna, Austria  Markus Valtiner
<EMAIL_ADDRESS>Vienna University of Technology, Institute for
Applied Physics, Wiedner Hauptstrasse 8-10, A-1040 Vienna, Austria

###### Abstract

Understanding elemental corrosion currents and visualizing corroding
topographies provide a detailed insight into corrosion mechanisms at the nano-
scale. Here, we develop a strategy to understand the elemental composition,
corrosion resistivity and local stability of passive materials. Specifically,
we utilize a pulse voltammetry approach in a novel electrochemical AFM cell
and complement this data by real-time dissolution currents based on spectro-
electrochemical online analysis in an ICP-MS flow cell. We study the oxide
properties and their protective behaviour, when formed under different applied
potentials using alloy 600 as model sample. Both AFM and ICP-MS data show that
passive films formed on alloy 600 at around +0.3 to +0.4 V in neutral 1 mM
NaCl solution are most stable during anodic corrosion at +1.0 V, while AFM
further demonstrates that local dissolution occurs, indicating locally varying
defect levels in the passive film. In combination of both techniques, our
approach provide real-time elementally resolved and localized information of
passive film quality under corrosive conditions, and it may prove useful for
other corroding materials.

nickel base alloys, Electrochemical ICP-MS, Electrochemical AFM, pulse
voltammetry, passivation

##  I Introduction

Nickel-based alloys (NBAs) show outstanding corrosion resistance under many
extreme environments enabling them to be used in broad applications such as
steam reactors, aerospace and chemical processing.[1, 2, 3, 4, 5]. It is well-
established that the unique corrosion resistance of NBAs such as alloy 600 can
only be achieved when the chromium content is more than 10 %[6, 7, 8, 9],
which is necessary for the formation of a stable passive/chemically inert
chromium oxide enriched thin film (typically 1-2 nm[10]) on the alloy surface.
With its outstanding corrosion inhibition property, such thin films have
attracted great attention from researchers [1, 11] to study their passivation
mechanism, as well as to characterize its physical structure, chemical
composition and electronic properties.[12, 13, 14] While alloy 600 is prone to
pitting corrosion [15], under mild conditions passive film breakdown is
considered to be the critical step [16] leading to corrosive failure. Hence,
understanding the properties and anti-corrosion mechanism of the passive film
are central for predicting corrosion tendency of NBA in mild environments.

Up to now, the formation of passive films have been described using models
such as the High Field Mechanism (HFM)[17], or the Point Defect Model
(PDM)[18], which can be used to explain growth kinetics as well as their
chemical composition. These models suggest that changing the conditions under
which the passive layer is formed can influence the formation kinetics that
leads to grow distinct passive films, and therefore, result in varying
corrosion resistance.

The passivity of the chromium oxide thin film is often understood as formation
of a dense physical barrier that is limiting the permeability of ions across
the film [19] or a change of the electronic structure (band gap) that creates
a higher energy barrier for charge exchange at the interface[20]. It is easy
to picture when growing passive films under different potentiostatic
polarization, the created potential drop across the oxide film can result in a
different degree of ion migration/permeability, which directly influences the
passive film structure.

However, due to the nature of its extremely small film thickness,
characterizing its physicochemical properties during a corrosive process
remains challenging. Most of the surface sensitive analytical methods are ex
situ techniques and very often requiring to operate in specific environment
such as under ultra high vacuum. In this aspect, obtained results from these
techniques do not directly reflect on the in situ passive film composition and
behaviour.

Consequently, techniques capable of performing real-time study such as
scanning probe techniques e.g. Atomic Force Microscopy (AFM) and Scanning
Tunneling Microscopy (STM) are more suitable in studying actively corroding
surfaces, which has increasingly gained more impact in recent years. [21, 22,
23, 24, 10]

Most of the research conducted observed the growth behaviour of passive films
e.g. during linear sweep voltammetry (LSV). In this perspective, the created
oxide films are not well-defined and the composition may vary with the varying
potential. I.e. properties of a passive oxide from a LSV may be influenced by
the scan rate and applied solution conditions.

It is therefore crucial to develop a strategy to understand the elemental
composition, corrosion resistivity and stability of oxide films that formed
under in situ conditions. We considered passive films grown under
potentiostatic conditions as most controlled conditions for systematically
studying their behaviour. In this work, we utilize a pulse voltammetry
approach in a novel EC-AFM cell (Fig. 1b, c) as well as in a flow cell
connected to an inductively coupled plasma mass spectroscopy (ICP-MS) online
analysis (inset in Fig. 1a) to study the oxide properties and their protective
behaviour, when formed under different applied potentials. Both techniques
provide time-resolved information about the surface condition with the ICP-MS
having the focus on elemental composition and the AFM providing
topographic/structural information of a materials during corrosive breakdown
of alloy 600 as model system. Both AFM and ICP-MS data show that passive films
formed at around +0.3 - +0.4 V in neutral NaCl solution are most stable during
during anodic corrosion at +1.0 V, while AFM further demonstrates that local
dissolution occurs, indicating locally varying defect levels in the passive
film.

Figure 1:  a) Tafel characteristic curve of the examined alloy: NBA 600
($Ni_{72}Fe_{8}Cr_{15.5}$) and Cr5 ($Ni_{86}Fe_{9}Cr_{5}$) for comparison. The
blue circles mark the regions of interest in this study. Inset shows a
schematic of the home-designed ICP-MS coupled electrochemical flow cell. A
stamp is used for establishing laminar flow across the surface of working
electrode (WE, studied alloy). The counter electrode (CE, Pt) and reference
electrode (RE, Ag|AgCl) are arranged to avoid WE being contaminated by the
reaction products from CE (hydronium or hydroxide that may vary the pH near WE
surface). b) Design principle of the home-designed three-electrode system
equipped AFM liquid cell, bottom view. A platinum foil forms the CE directly
at the optical window of the used cantilever holder. Only a small section is
left out for the AFM laser. Reference electrode (Ag|AgCl) is located outside
the cell with a tubing (green) connected. c) Side view of the assembly with
the sample (WE) at the bottom in parallel with the CE, resulting in a even
electric field distribution.

##  II Results and Discussion

Corrosion resistance of NBA increases with increasing chromium content.
Exemplary, a Tafel analysis shown in Fig. 1a) for an NBA with 5 % and 15.5 %
chromium demonstrates the effect of chromium on the window of passivity. When
comparing the polarization curves, alloy 600 exhibits a well defined region of
passivity from -0.1 to +0.5 V shown as the flat plateau in the Tafel plot
(indicated by grey arrow), whereas the Cr5 is showing an earlier break-down
(light grey) around +0.2 V.

### Results of ICP-MS experiments.

The linear polarization shown in Fig. 1a) changes the passive film properties
continuously, our interest is however to study potential dependent passive
film properties and their protective characteristics. Hence, in order to
approach the passive film formation under thermodynamic equilibrium, we
introduced a multi-step chronoamperometry sequence (see inset figure of Fig.
2c)) to perform step-wise surface oxide growth, corrosion resistance test and
reconditioning as shown in Fig. 2. Specifically, freshly polished alloy 600
was first polarised at a passivation potential of $E_{(pre)cond}$ over 5
minutes for growing a passive layer. Subsequently, the polarization jumped to
$E_{diss}$ = +1.0 V for another 5 minutes for testing the corrosion resistance
of the material after passive film formation at the given $E_{(pre)cond}$. At
the end of the corrosion test, polarization switched back to reconditioning at
$E_{recond}$ = -0.3 V for 2 minutes to remove the remaining surface oxides,
which resets the alloys’ surface chemical state for the coming repetition
sequences.

This approach can therefore provide a clear insight into protective
characteristics of passive films prepared under different electrochemical
potentials. Based on the Tafel analysis shown in Fig. 1a) we selected five
potentials of interest: $E_{(pre)cond}$ = -0.3, +0.2, +0.3, +0.4 and +0.5 V,
where -0.3 V was chosen as reference region. There we expect no or only minor
oxide film formation as it is located at the cathodic branch of the Tafel plot
and as is estimated from Pourbaix diagrams.[25] The other potentials are well
in the passive region.

We used ICP-MS and EC-AFM (see setup in Fig. 1b) and c)) to identify the
alloy’s potential-dependent elemental dissolution tendency and surface
morphology change, respectively. The advantage of such flow-cell coupled ICP-
MS arrangement is that it provides online corrosion product analysis at trace
levels, which can directly be correlated to the applied electrochemical
polarization during a real-time process. In combination with EC-AFM, we may
further understand local and micro structure effects.

Figure 2: Relative dissolution during anodic polarization. Electrochemical
flow cell coupled ICP-MS dissolution characteristics in designed passivation-
dissolution sequences. a) b) Integration of collected mass during corrosion at
1.0 V over the course of single sequences (4 pulse periods), resolved per
elemental contribution. Error whiskers show standard deviation of value over
three repetitions of the experiment with >4 pulse sequences. c) Overlap of 4
consecutive pulse periods of recorded ICP-MS mass-time profiles for all
elements during one full sequence. Please note that the scale for nickel is
about 1 order of magnitude higher. Inset shows the schematic of the designed
applied electrochemical pulse profile in multi-step amperometry.

Fig. 2a) shows integrated dissolved mass of nickel, chromium and iron marked
in blue, purple and red, respectively, from ICP-MS during polarization of +1.0
V averaging over four repetitions (see Fig. 2c)). Among all the examined
potentials, we surprisingly found that alloy 600 treated with passivation
potentials around +0.3 V to +0.4 V shows a statistical relevant decrease in
material loss during active corrosion at +1.0 V. Comparing to the amount of
measured dissolved mass at $E_{(pre)cond}$ = -0.3 V, where no passivation is
expected, the passivity at $E_{(pre)cond}$ = +0.3 V shows an expected
reduction of the mass-loss by around 50 %. In the case of anodic dissolution
of the metal surface after $E_{(pre)cond}$ = +0.5 V, the measured dissolved
mass at +1.0 V increased again compared to passivation at +0.3 V. Hence, the
corrosion resistance of the passive film formed at this potential is again
reduced, which agrees well with the observed passivity range and the
increasing passive current in the Tafel plot.

Further examination of the ratios of dissolved mass surprisingly revealed that
nickel, chromium and iron were demonstrating entirely different dissolution
trends depending on the passivation potential. Specifically, Fig. 2b) shows
the relative elemental ratio of dissolved mass of individual alloy elements
normalized by their bulk ratio, where zero, positive and negative values
indicate equivalent, more and less dissolved mass compared to the bulk
composition, respectively.

Comparing the trend of dissolution for the elements, nickel exhibited a higher
dissolution rate compared to its bulk ratio at all passivation potentials,
with a clear reduction by about 50% in the passive region above +0.3 V. This
is also associated to the main reduction of the total mass loss observed in
Fig. 2a).

In contrast to nickel, chromium shows a weakly passivation potential-
independent dissolution below bulk concentration. However, iron is showing a
very clear passivation-dependent dissolution trend. Iron dissolved less
compared to bulk ratio at lower potentials but gradually shifted into over-
stoichiometric dissolution at higher passivation potential of +0.5 V.

The dissolution profile of iron shown in Fig. 2c) reveals another key feature,
which may be relevant for understanding the underlying mechanism: As indicated
in the figure, the detected iron release exhibited a two-step mechanism -
anodic dissolution at +1.0 V and cathodic dissolution when jumping from +1.0 V
to -0.3 V, indicated by red dotted and shaded areas, respectively.

This observation suggests that the iron in the passive film undergoes at least
two types of dissolution mechanisms, occurring under different polarization
potentials. Based on Pourbaix diagrams the detected iron species at anodic
polarization potential of +1.0 V may originate from the release of ferrate
anion (FeO${}_{4}^{2-}$). The additional peak upon jumping cathodically back
from +1.0 V to -0.3 V, indicates dissolution of a surface bound excess iron
species formed during anodic corrosion.

In contrast to the two-step dissolution pattern of iron, nickel shows a well
reproducible profile over several repetitions, which is characterised by a
$\sim$1 order of magnitude dissolution rate increase, above bulk ratio, at
+1.0 V. Similar reproducibility of dissolution profiles is observed for Cr.
For this element the first sequence in a repetitive set shows a delayed
dissolution kinetic, which may relate to an effect of the initial native
oxide, adapting to flow cell environment after the first cycle.

### Results of in situ EC-AFM surface morphology probing

Figure 3: AFM topography during different stages ($E_{(pre)cond}$, $E_{diss}$, $E_{prob}$) of multiple step voltammetry with different (pre-)conditioning potentials $E_{(pre)cond}$ = -0.3, +0.3 and +0.5 V. Red dashed lines indicate the position of the line sections shown in b). RMS of each picture and relative change compared to the initial OCP-topography can be found in Tab. 1. Potential | RMS [nm] (relative to OCP)
---|---
$E_{precon}$ = |  -0.3 V |  +0.3 V |  +0.5 V
OCP |  0.83 (1.0) |  2.32 (1.0) |  3.05 (1.0)
precon |  0.84 (1.0) |  2.36 (1.0) |  3.27 (1.1)
+1.0 V |  2.37 (2.8) |  2.57 (1.1) |  4.12 (1.4)
-0.3 V |  2.09 (2.5) |  2.86 (1.2) |  4.98 (1.6)
Table 1: Root mean square (RMS) values and their relative change of images
shown in Fig. 3

To complement integral ICP-MS data, we further visualised the (local)
evolution of the surface morphology upon passivisation using electrochemically
modulated AFM measurements.

Fig. 3a) shows representative real-time changes in surface topography of alloy
600 during corrosion after different conditioning potentials $E_{(pre)cond}$
where applied (similar to ICP-MS). The red dashed lines indicated in the OCP
(open circuit potential) topograms show line profiles that are compared in
Fig. 3b), for $E_{(pre)cond}$ (blue), +1.0 V (green) and -0.3 V (red),
respectively.

As a side note, for better visualisation height changes were overlaid on the
recorded images. Increase in height is indicated by green shading, while
dissolution from the surface is shown in magenta. Further, AFM topography does
not provide information of absolute but only relative heights. Hence,
consecutively measured topographies were shifted during post-processing, using
a referencing protocol described in the methods section, in order to
approximate the absolute change in topography during corrosion. The selected
images summarise recurring patterns and characteristics seen in repeat
experiments.

The potential-dependent EC-AFM images reveal a number of interesting details
as described in the following.

First, during the passive film formation (preconditioning) the surface
morphology of all three examined precondition potentials show no significant
changes compared to OCP. Only very minor height variations are observed.
Jumping from precondition potential to dissolution potential, the alloy
treated at $E_{(pre)cond}$ = -0.3 V shows a very vigorous dissolution. The RMS
roughness increases significantly (see Table 1), and most of the pronounced
features (polishing scratches, etc.) become indistinguishable, while only few
areas remained stable for referencing the corrosive height loss.

Second, and in a sharp contrast, the surface morphology during anodic
corrosion has no significant change when the passive film was grown under
polarization of +0.3 V. Obviously, the surface oxide film does protect the
surface very effectively, and dissolution occurs homogeneously along
preexisting features such as scratches.

Third, surfaces passivated at +0.5 V again show more pronounced corrosive
degradation at +1.0 V. Here, the surface morphology did not undergo a
similarly massive dissolution seen for preconditioning at -0.3 V. However,
significant dissolution preferentially occurred along the edges of the nano-
granular features. Additionally, in some areas measurable material growth was
observed, indicating possible local re-precipitation reactions.

Fourth, upon switching the polarization from +1.0 V to -0.3 V, the dissolution
reactions stopped and only minor topographic changes were observed for
preconditioning at +0.3 V and +0.5 V. However, for samples pretreated at -0.3
V dissolution was still observed at the top of particles. This correlates well
with the observed significant cathodic iron dissolution peak (see Fig. 2c)),
which further supports that the surface enriches in iron oxide during active
corrosion, if corrosion starts without a significant passive film presence.

### Overall discussion of results

In summary, the AFM topography analysis agrees very well with the ICP-MS
online analysis results, which reveals that the passivation potential, i.e.
the conditions during passive film formation are directly influencing the
behaviour during active corrosion. Together with the transition of the ICP-MS
dissolution pattern our data suggests that the quantitative elemental
dissolution during active corrosion is highly dependent on the passive film
properties. In the following, we want to discuss several interesting aspects
of our data.

First, if the material is directly corroded after polarization at -0.3 V, ICP-
MS and AFM data suggested a massive material loss. Given the Pourbaix diagram,
one may expect a pure chromium oxide layer, as the most stable chromium phase
is chromium oxide at -0.3 V at the experimental pH. Further, the passive film
might be much thinner, or even defect rich, or not complete. As a consequence,
any protective effect is limited or not even present. Second, in line with
this thought we find clear and pronounced cathodic iron dissolution only after
corrosion starting directly from -0.3 V. This suggests that a steady state
surface film evolves during anodic breakdown, which is significantly enriched
in iron, and presumably iron oxide. This is consistent with the Pourbaix
diagram suggesting that Chromium oxide is not stable at corrosive conditions
at 1.0 V. Our data may hence indicate that iron oxide forms as a transient
oxide during corrosion, which is specifically growing at defects or areas that
are not covered by a stable chromium oxide. Further, during corrosion, this
oxide is not sufficient to suppress the material loss as effectively as a
proper passive film grown at passivation potentials within the passive region.
This iron oxide enrichment is evident form the significant iron dissolution
peak during repassivation, after corrosion at 1.0 V.

Third, and along the same line, passivation at $\geq$ +0.3 V establishes a
passive film, which can significantly decrease the material loss during active
corrosion. Still, we find a small but detectable cathodic iron peak after
active corrosion. This may suggest that the passive film breaks down locally,
presumably at defect rich sites, and/or iron oxide accumulates at the outside
of the passivating chromium oxide.

Considering the AFM data at +0.3 V, which shows a rather uniform material loss
at defect rich areas (i.e. preexisting polishing defects, etc.), our findings
appear more consistent with an increased iron dissolution across the defect
rich areas of the passive film. Further, as seen in Fig. 2b) relative iron
dissolution also increases significantly if a passive film is grown between
+0.3-0.5 V. Hence, iron may accumulate at the oxide|water interface and/or
gets transported through the passive film at a higher ratio during corrosive
dissolution. This suggests a change in the passive film chemistry.

Fourth, after passivation at +0.5 V the again increasing iron content during
cathodic polarization after corrosion, suggests a decisive role of defect
levels on the transport across the passive film during corrosion. As such, the
growth conditions apparently render the passive film with different defect
levels. Together with the increase in iron ratio during corrosion (see again
Fig. 2b)) this suggest a structural change in the oxide that accelerated iron
diffusion across the existing passive film in a high-field condition applied
during corrosion at +1.0 V. This is further supported by AFM, where we see
increased dissolution at areas that are defect rich. The high field model
(HFM)[17, 20] describing the built up of the passive film during
$E_{(pre)cond}$ provides an explanation for the differences in density of
defects, and hence, corrosion resistance.

Also, during the cathodic jump AFM data at +0.5 V shows further material loss
at the preexisting grain boundaries. The passive film grown at +0.5 V may
hence trigger faster oxide growth, due to an increase of defect rich areas, as
indicated by the localised loss seen in AFM topography.

We refrain from interpreting too much into the apparent material growth areas
at this point in time, as AFM is not an absolute technique, and post-
processing shifts may not truly reflect any growth areas, while material loss
is clearly distinguishable. Along this line, one important aspect to consider
when comparing the processes observed in both the AFM and the ICP-MS flow cell
is their general differences: While the ICP-MS requires a constant flow of
electrolyte the AFM measurements are conducted in a stagnant solution.
Quantitative comparision also indicates that corrosion is more severe in AFM,
where for instance after preconditioning at +0.3 V a few nm of material are
lost during corrosion at 1.0 V, while ICP-MS indicates only a monolayer
material loss per cycle (300 ng/cm² $\cong$ 0.35 nm). This discrepancy is
considerably lower when the material corrodes less.

Hence, the dissolution in AFM is also accelerated due to typical surface
confinement in an AFM (typically the AFM cantilever holder is separated by
less than 200 $\mu$m from the probed surfaces), which renders this experiment
more effectively "crevice like", where electrolyte of the dissolving species
may render the interface at a different pH. This needs to be considered for
AFM and scanning probe techniques in general when studying corrosion, and may
become a feature, if properly controlled. How scanning probe conditions can
even compare to flow-cell experiments, and other electrochemical cells, is in
our view an aspect that will need further attention, e.g. by directly coupling
AFM and ICP-MS, and by further comparative studies with model systems.

##  III Conclusions

In this work, we successfully used two complementary analytical techniques to
study the influence of passivation potential on the formed metal oxide film
quality in real-time. We designed an AFM cell that performs very well during
corrosion. The - for scanning probe techniques - novel plate-plate geometry
provides an idealized field distribution. Using an ICP-MS flow cell and EC-
AFM, passivation and anodic dissolution was tracked with both time resolved
elemental dissolution rates during chronoamperometric pulse experiments and
complementary changes in morphology, respectively. Regarding the characterised
passivation potentials on alloy 600 we could find an optimum at around +0.3 -
+0.4 V in neutral NaCl solution exhibiting lowest dissolution and therefore
forming the most stable passive film during anodic corrosion at +1.0 V.

Our results suggest the following specific conclusions:

  * •

Quality of the passive film depends on the potential at which it is grown.

  * •

The passivation potential controls the defect density and ion-conducting
properties of the passive film during active corrosion.

  * •

Combination of in situ techniques provide information about stability,
indirectly composition and local stability of the passive film.

With the ICP-MS flow cell and the EC-AFM it is hence possible to relate two in
situ techniques to consistently describe the quality and stability of a
passive film. In combination both techniques provide real-time and localized
information under operating conditions, and are much closer to realistic
corrosion conditions, compared to other surface sensitive analytical methods.

## Acknowledgements

The authors acknowledge support by the European Research Council (Grant:
CSI.interface, ERC-StG 677663, characterization of surfaces).

## Competing Interests

The Authors declare no Competing Financial or Non-Financial Interests.

## Data availability

The raw and processed data required to reproduce these findings are available
from the corresponding author via www.repositum.tuwien.ac.at upon reasonable
request.

## Methods and Materials

### Chemicals and Materials.

Sodium chloride (Carl Roth, p.a.) and Milli-Q water (resistivity >18
M$\Omega\cdot$ cm, total organic carbon < 4 ppb) was used for making
electrolyte solutions. Alloy 600 was obtained from VDM-Metals, Cr5 was
provided by Hauke Springer (MPI f. Eisenforschung, Düsseldorf). The metals
were ground with sand paper of decreasing grain size (from P80 to P2500), then
polished with diamond paste down to 0.05 $\mu$m. Prior to electrochemical
experiments the metals were maximum 5 min in situ in the ICP-MS flow cell by 5
minutes potentiostatic polarization at $\rm-0.2V$ vs OCP.

### Atomic Force Microscopy (AFM)

AFM topographies were taken on with AM-mode Cypher ES (Asylum Research, Oxford
Instruments, Santa Barbara, CA) using ArrowTM UHFAuD (NanoWorld, CH) and SCOUT
350 RAu (NuNano, UK) probes. blueDrive was used to oscillate the probes.
Embedded NBA samples were contacted with silver glue. For embedded samples a
rectangular piece of metal of about 5 mm side length was used. The block
material was embedded in methylmethacrylate based VariDur 200 (Buehler). A
PalmSens 4 potentiostat (PalmSens, Netherlands) was used.

#### Image processing

Prior drift correction and further analysis images were only processed by
applying flattening of first order. For alignment in z-direction a minimum of
10 line profiles were chosen per time-series and adapted in height to match
significant - and non-changing - features of the images, e.g. deep scratches
from polishing or distinct peak features that remain over time.

### Cell design of the electrochemical AFM (EC-AFM)

Our home-built electrochemical cell design (see schematics in Fig. 1 b) & c)
is inspired by concepts of Valtiner et al.[26] and derives from a standard
Perfusion Cantilever Holder (Asylum Research). We modified it by placing a
platinum foil as a counter electrode (CE) at the bottom side of the cantilever
holder. When assembled the working electrode (WE) and the platinum CE form a
parallel plate geometry resulting in a very evenly distributed electrical
field. A majority of the upper surface is covered with the platinum foil to
provide a high area, hence avoiding any rate limitation from reactions at the
CE. Only the optical path for the laser of the AFM and a feed trough to
connect the reference electrode remain cleared. A commercial silver/silver
chloride reference electrode is connected with the cell via a capillary. For
compensation of the expected high electrical current especially during anodic
dissolution and given that the CE is in close vicinity to the sample a
reduction of the WE size is limiting the possibilty of disturbance through
reactions happening at the CE (e.g. hydrogen evolution and bubble formation).
The size ration of WE to CE is at least 5:1.

### Inductively Coupled Plasma Mass Spectrometry (ICP-MS)

Measurements were performed using an Agilent 7900 ICP-MS from Agilent
Technologies. Calibration was performed with multi-element standard solutions
provided by Agilent and Inorganic Ventures. The ICP-MS uses a collision cell
with 5 mL/min flow of helium as cell gas. Downstream of the electrochemical
cell the analyte was mixed with standard solution. Cobalt was chosen as
internal standard due to its similar mass with most of the alloy components.
Electrochemical experiments were conducted in a home-built flow cell out of
PEEK and PTFE insipired by the Ogle- and Mayrhofer designs. [27, 28, 29] The
exposed electrode area is circular and sealed with a 3 mm inner diameter
O-ring. Inset in Figure 1a) shows schematically the flow-cell used in this
work. Pumping with pressurized nitrogen establishes a stable and pulsation
free laminar flow of electrolyte. Flow was monitored both by an in-flow
pressure sensor and by weighing the collected waste electrolyte after ICP-MS.
The flow is set to $6\pm 0.2$ mg of solution per second. Before each
experiment the electrolyte was purged with compressed and filtered air for at
least 30 minutes to guarantee the same concentration of dissolved oxygen. A
Biologic VSP-300 potentiostat (Biologic, France) was used. All electrochemical
experiments were done using a Ag|AgCl-Electrode as reference and presented
data is referenced to that potential.

## References

## References

  * [1] B. T. Lu, J. L. Luo, and Y. C. Lu, “A Mechanistic Study on Lead-Induced Passivity-Degradation of Nickel-Based Alloy,” Journal of The Electrochemical Society, vol. 154, no. 8, p. C379, 2007.
  * [2] S. J. Rosenberg, Nickel and its alloys.  National Bureau of Standards, 1968.
  * [3] A. Rowcliffe, L. Mansur, D. Hoelzer, and R. Nanstad, “Perspectives on radiation effects in nickel-base alloys for applications in advanced reactors,” Journal of Nuclear Materials, vol. 392, pp. 341–352, July 2009\.
  * [4] M. B. Henderson, D. Arrell, R. Larsson, M. Heobel, and G. Marchant, “Nickel based superalloy welding practices for industrial gas turbine applications,” Science and Technology of Welding and Joining, vol. 9, pp. 13–21, Feb. 2004\.
  * [5] S. Persaud, B. Langelier, A. Korinek, S. Ramamurthy, G. Botton, and R. Newman, “Characterization of initial intergranular oxidation processes in alloy 600 at a sub-nanometer scale,” Corrosion Science, vol. 133, pp. 36–47, Apr. 2018.
  * [6] J. R. Hayes, J. J. Gray, A. W. Szmodis, and C. A. Orme, “Influence of Chromium and Molybdenum on the Corrosion of Nickel-Based Alloys,” CORROSION, vol. 62, no. 6, pp. 491–500, 2006.
  * [7] A. C. Lloyd, J. J. Noël, S. McIntyre, and D. W. Shoesmith, “Cr, Mo and W alloying additions in Ni and their effect on passivity,” Electrochimica Acta, vol. 49, no. 17-18, pp. 3015–3027, 2004.
  * [8] N. Ebrahimi, P. Jakupi, J. Noël, and D. Shoesmith, “The Role of Alloying Elements on the Crevice Corrosion Behavior of Ni-Cr-Mo Alloys,” CORROSION, vol. 71, pp. 1441–1451, Sept. 2015\.
  * [9] S. Persaud and R. Newman, “A Review of Oxidation Phenomena in Ni Alloys Exposed to Hydrogenated Steam Below 500∘C,” Corrosion, vol. 72, pp. 881–896, July 2016.
  * [10] V. Maurice and P. Marcus, “Current developments of nanoscale insight into corrosion protection by passive oxide films,” Current Opinion in Solid State and Materials Science, vol. 22, no. 4, pp. 156–167, 2018.
  * [11] A. Galtayries, A. Machet, P. Jolivet, P. Scott, M. Foucault, P. Combrade, and P. Marcus, “Kinetics of passivation of nickel-base alloys (Alloy 600 and Alloy 690) in high temperature water,” in Passivation of metals and semiconductors, and properties of thin oxide layers (P. Marcus and V. Maurice, eds.), pp. 403–409, Amsterdam and Oxford: Elsevier, 2006.
  * [12] S. Marcelin, B. Ter-Ovanessian, and B. Normand, “Electronic properties of passive films from the multi-frequency Mott–Schottky and power-law coupled approach,” Electrochemistry Communications, vol. 66, pp. 62–65, 2016\.
  * [13] Y. Mito, M. Ueda, and T. Ohtsuka, “Photo-luminescence from passive oxide films on nickel and chromium by photo-excitation of UV light,” Corrosion Science, vol. 51, no. 7, pp. 1540–1544, 2009.
  * [14] H. Tsuchiya, S. Fujimoto, O. Chihara, and T. Shibata, “Semiconductive behavior of passive films formed on pure Cr and Fe–Cr alloys in sulfuric acid solution,” Electrochimica Acta, vol. 47, no. 27, pp. 4357–4366, 2002.
  * [15] C. Merola, H.-W. Cheng, D. Dworschak, C.-S. Ku, C.-Y. Chiang, F. U. Renner, and M. Valtiner, “Nanometer Resolved Real Time Visualization of Acidification and Material Breakdown in Confinement,” Advanced Materials Interfaces, vol. 6, no. 10, p. 1802069, 2019.
  * [16] G. S. Frankel, T. Li, and J. R. Scully, “Perspective—Localized Corrosion: Passive Film Breakdown vs Pit Growth Stability,” Journal of The Electrochemical Society, vol. 164, no. 4, pp. C180–C181, 2017.
  * [17] N. Cabrera and N. F. Mott, “Theory of the oxidation of metals,” Reports on Progress in Physics, vol. 12, no. 1, pp. 163–184, 1949.
  * [18] C. Y. Chao, L. F. Lin, and D. D. Macdonald, “A Point Defect Model for Anodic Passive Films: I . Film Growth Kinetics,” Journal of The Electrochemical Society, vol. 128, no. 6, pp. 1187–1194, 1981.
  * [19] P. Marcus, Corrosion mechanisms in theory and practice, vol. 26 of Corrosion technology.  Boca Raton: CRC Press, 3rd ed. ed., 2012.
  * [20] G. Tranchida, M. Clesi, F. Di Franco, F. Di Quarto, and M. Santamaria, “Electronic properties and corrosion resistance of passive films on austenitic and duplex stainless steels,” Electrochimica Acta, vol. 273, pp. 412–423, 2018.
  * [21] H. Chen, Z. Qin, M. He, Y. Liu, and Z. Wu, “Application of Electrochemical Atomic Force Microscopy (EC-AFM) in the Corrosion Study of Metallic Materials,” Materials, vol. 13, no. 3, 2020.
  * [22] Y. Shi, L. Collins, N. Balke, P. Liaw, and B. Yang, “In-situ electrochemical-afm study of localized corrosion of alxcocrfeni high-entropy alloys in chloride solution,” Applied Surface Science, vol. 439, 05 2018.
  * [23] X. Deng, F. Galli, and M. Koper, “In situ electrochemical afm imaging of a pt electrode in sulfuric acid under potential cycling conditions,” Journal of the American Chemical Society, vol. 140, 09 2018.
  * [24] V. Maurice, T. Nakamura, L. Klein, and P. Marcus, “7 - Initial stages of localised corrosion by pitting of passivated nickel surfaces studied by scanning tunnelling microscopy (STM) and atomic force microscopy (AFM),” in Local Probe Techniques for Corrosion Research (R. Oltra, V. Maurice, R. Akid, and P. Marcus, eds.), European Federation of Corrosion (EFC) Series, pp. 71–83, Woodhead Publishing, 2007.
  * [25] B. Beverskog and I. Puigdomenech, “Revised pourbaix diagrams for iron at 25-300 ∘C,” Corrosion Science, vol. 38, pp. 2121–2135, Dec. 1996.
  * [26] M. Valtiner, G. N. Ankah, A. Bashir, and F. U. Renner, “Atomic force microscope imaging and force measurements at electrified and actively corroding interfaces: challenges and novel cell design,” The Review of scientific instruments, vol. 82, no. 2, p. 023703, 2011.
  * [27] K. Ogle and S. Weber, “Anodic dissolution of 304 stainless steel using atomic emission spectroelectrochemistry,” Journal of The Electrochemical Society, vol. 147, no. 5, pp. 1770–1780, 2000.
  * [28] A. K. Schuppert, A. A. Topalov, I. Katsounaros, S. O. Klemm, and K. J. J. Mayrhofer, “A Scanning Flow Cell System for Fully Automated Screening of Electrocatalyst Materials,” Journal of The Electrochemical Society, vol. 159, no. 11, pp. F670–F675, 2012.
  * [29] S. O. Klemm, A. A. Topalov, C. A. Laska, and K. J. Mayrhofer, “Coupling of a high throughput microelectrochemical cell with online multielemental trace analysis by ICP-MS,” Electrochemistry Communications, vol. 13, pp. 1533–1535, Dec. 2011.


invertible transformations of the representations that preserve information
generate equivalent representations even if the individual neurons encode
entirely different input features. Therefore, we quantify the representations
by two numbers, or order parameters for each layer - the MI of $T$ with the
input $X$ (the decoder’s information) and with the desired output $Y$ (the
information of the encoder). These quantities are invariant to any invertible
re-parameterization of $T$.

For any fixed realization of the weights, the network is, in principle, a
deterministic map. This does not imply that information is not lost between
the layers; the layers’ inherent finite precision, with possible saturation of
the nonlinear activation function $\sigma_{k}$, can result in non-invertible
mapping between the layers. Moreover, we argue below that for large networks,
this mapping becomes effectively stochastic due to the diffusion phase of the
SGD.

On the other hand, the paths of the layers in the information plane are
invariant to invertible transformations of the representation $T_{k}$. Thus,
the same paths are shared by very different weights and architectures and
possibly different encoder-decoder pairs. This freedom is drastically reduced
when the target information, $I(T_{k},Y)$, increases, and the layers approach
the IB limit. Minimizing the training error, together with standard uniform
convergence arguments, clearly increases $I(T;Y)$. This raised the question
what in the SGD dynamics can lead to the observed representation compression,
which further improves the generalization? Moreover, can the SGD dynamics push
the layer representations to the IB limit, as claimed in shwartz2017opening?

We provide affirmative answers to both questions, using the properties of the
drift and diffusion phases of the SGD dynamics.

Figure 1.20: The DNN layers form a Markov chain of successive representations
of the input layer $X$. Every representation of the input, $T$, is defined by
an encoder, $p(t|x)$, and a decoder $p(y|t)$, and may be characterized on the
information plane as a pair of coordinates: $I(X;T)$ and $I(T;Y)$.

####  1.7.1 Successive refinement approximation

A DNN forms a Markov chain of representations. Therefore, it is natural to
approximate the original IB bound as a sequence of smaller IB problems, where
each problem corresponds to a single layer. We refer to such an approximation
as a successive refinement (as in (cover2012elements)). Specifically, assume
that the layer $T_{k-1}$ is fixed. Then, the successive refinement
approximation seeks a pair of optimal encoder-decoder with respect to
$T_{k-1}$ and $Y$ (as opposed to $X$ and $Y$). The quality of the
approximation depends on the quality of the previous layer. In other words,
the optimal Information Curve induced by sequentially solving the IB problem
for each layer separately is a lower bound to the original IB curve. When this
bound equal to the IB curve, we say that the problem is successively refinable
(cover2012elements).

In a standard training procedure, we would like to minimize the loss function
of the network. However, our goal is to show the converges of all the layers
to the information curve.

To do so, we prove a new lower bound on the standard cross-entropy loss
function, which bounds the information between consecutive layers. Using this
bound, the minimization of the loss function will minimize the $D$ of the
individual layers. We also know that the minimization of the cross-entropy is
equivalent to increasing the MI. Therefore, each layer’s information will
increase during the learning and converge to the optimal information curve
with its constraint on the information with $X$.

Consider the cross-entropy loss function,
$D\left(p(y|x)||p\left(y|t_{K}\right)\right)$ and denote the $Q_{k}$ as the
set of distributions with linear constrain on the information with X, namely
$Q_{k}=\\{p\left(y|\bar{t}_{k}\right)|I\left(X;\bar{T_{k}}\right)=I\left(X;T_{k}\right)\\},1\leq
k\leq K$ Denote the reverse I projections of a distribution $p(y|x)$ onto a
set of distributions $Q_{k}$ , $1\leq k\leq K$ in $p(y|t_{k})^{\ast}$, namely

$p(y|t_{k})^{\ast}=\operatorname*{arg\,min}_{q\in
Q_{k}}D\left[p(y|x)||q\right]$

Using Pythagorean theorem (csiszar2003information), if $Q_{k}$ is log-convex
set, the following inequality holds

$\displaystyle D\left(p\left(y|x\right)||p\left(y|t_{K}\right)\right)\geq
D\left(p\left(y|x\right)||p\left(y|t_{k}\right)^{\ast}\right)+D\left(p\left(y|t_{k}\right)^{\ast}||p\left(y|t_{K}\right)\right)$
(1.27)

Due to the minimization of the right side in the optimization process, we get
that the right side above will also decrease, and that the cross entropy of
the k-th layer -
$D\left(p\left(y|x\right)||p\left(y|t_{k}\right)^{\ast}\right)$ can only
decrease during the learning as requested.

#### SGD Dynamics in the Information Plane

shwartz2017opening studied the Information Paths of DNNs on the information
plane; they evaluated and visualized the pairs of $I(X;T_{k})$ and
$I(T_{k};Y)$, along the learning process and compared different network
architectures. Specifically, they focused on the Information Curve dynamics
(the change of its location on the plane) during the SGD process.
Interestingly, they showed that the SGD consists of two distinct phases on the
information plane. First, they observe a short phase in which $I(T_{k};Y)$
increases quite rapidly. They then observed a significantly longer phase
(which we called representation compression), in which $I(X;T_{k})$ drops. In
other words, they claimed that the SGD first learns the target and then
compresses the representation. This behavior seems to be related to the drift
and diffusion phases, discussed in Section The Different Phases of SGD
Optimization. It is not surprising that $I(T_{k};Y)$ increases during the
drift phase; the drift phase is characterized by a rapid decrease in the
cross-entropy loss function. This is equivalent to an increase in the
(empirical) MI. On the other hand, the correspondence between diffusion and
compression appears to be quite vague. We study this phenomenon in the
following section.

Figure 1.22: Snapshots of layers (different colors) of $50$ randomized
networks during the SGD optimization process on the information plane (in
bits). Left - initial weights. Center - at $400$ epochs. Right - after $9000$
epochs

#### Representation Compression by Diffusion

This section quantifies the roles of the drift and diffusion SGD phases and
their influence on the MI between consecutive layers. Specifically, we show
that the drift phase corresponds to an increase in information with the target
label $I(T_{k};Y)$, whereas the diffusion phase corresponds to representation
compression or reduction of the $I(X;T_{k})$. The representation compression
is accompanied by further improvement in the generalization.

The general idea is as follows: the drift phase increases $I(T_{k};Y)$ as it
reduces the cross-entropy empirical error. On the other hand, the diffusion
phase in high-dimensional weight space effectively adds an independent
nonuniform random component to the weights, mainly in the directions that do
not influence the loss - i.e., _irrelevant directions_. This results in a
reduction of the SNR of the patterns’ irrelevant features, which leads to a
reduction in $I(X;T_{k})$, or representation compression. We further argue
that different layers filter out different irrelevant features, resulting in
their convergence to different information plane locations.

#### The SGD Compression Mechanism

In this section, we introduce a novel asymptotic result which shows that under
several assumptions, a DNN compresses the original representation $X$ during
the diffusion phase, in the sense that $I(X;T_{k})$ reduces with the
iterations of the SGD along this phase.

First, DPI implies that $I(X;T_{k})\leq I(T_{k-1};T_{k})$. We focus on the
second term during the diffusion phase and prove an asymptotic upper bound for
$I(T_{k-1};T_{k})$, which reduces sub-linearly with the number of SGD updates.

For clarity, we describe the case where $T_{k}\in\mathbb{R}^{d_{k}}$ is a
vector and $T_{k+1}\in\mathbb{R}$ is a scalar. The generalization to higher
$d_{k+1}$ is straightforward. We examine the network during the diffusion
phase, after $\tau$ iterations of the SGD beyond the drift-diffusion
transition. For each layer,$k$, the weights matrix, $W^{k}(\tau)$ can be
decomposed as follows,

$W^{k}(\tau)={W^{k}}^{\star}+\delta W^{k}(\tau).$ (1.28)

The first term, ${W^{k}}^{\star}$, denotes the weights at the end of the drift
phase ($\tau_{0}=0$) and remains constant with increasing $\tau$. As we assume
that the weights converge to a (local, flat) optimum during the drift phase,
${W^{k}}^{\star}$ is close to the weights at this local optimum. The second
term, $\delta W^{k}(\tau)$, is the accumulated Brownian motion in $\tau$ steps
due to the batch-to-batch fluctuations of the gradients near the optimum. For
large $\tau$ we know that $\delta W^{k}(\tau)\sim\mathcal{N}(0,\tau
C(W^{k}(\tau_{0})))$ where $\tau_{0}$ is the time of the beginning of the
diffusion phase. Note that at any given $\tau$, we can treat the weights as a
fixed (quenched) realization, $w^{k}(\tau)$, of the random Brownian process
$W^{k}(\tau)$. We can now model the mapping between the layers $T_{k}$ and
$T_{k+1}$ at that time as

$T_{k+1}=\sigma_{k}\left({w^{*}}^{T}T_{k}+{\delta
w^{k}(\tau)}^{T}T_{k}+Z\right)$ (1.29)

where $w^{*}\in\mathbb{R}^{d_{k}}$ is the SGD’s empirical minimizer, and
$\delta w\in\mathbb{R}^{d_{k}}$ is a realization from a Gaussian vector
$\delta w\sim\mathcal{N}(0,C_{\delta w})$, of the Brownian process. In
addition, we consider $Z\sim\mathcal{N}(0,\sigma^{2}_{z})$ to be the small
Gaussian measurement noise, or quantization, independent of $\delta w^{k}$ and
$T_{k}$. This standard additive noise allows us to treat all the random
variables as continuous.

For simplicity, we assume that the $d_{k}$ components of $T_{k}$ have zero
mean and are asymptotically independent for $d_{k}\rightarrow\infty$, and that
$\lim_{d_{k}\rightarrow\infty}{w^{*}}^{T}\delta w=0$ almost surely.

######  Proposition 2.

Assume that the moments of $T_{k}$ are finite. Further assume that the
components of $w^{*}$ and $\delta w(\tau)$ are in-general-positions,
satisfying
$\lim_{d_{k}\rightarrow\infty}{\sum_{i=1}^{d_{k}}{w^{*}_{i}}^{4}}\big{/}{\left(\sum_{i=1}^{d_{k}}{w^{*}_{i}}^{2}\right)^{2}}=0$
and $\lim_{d_{k}\rightarrow\infty}{\sum_{i=1}^{d_{k}}{\delta
w_{i}}^{4}}\big{/}{\left(\sum_{i=1}^{d_{k}}{\delta w_{i}}^{2}\right)^{2}}=0$
almost surely. Then,

$\frac{1}{\sqrt{\sigma_{T_{k}}^{2}}}\left[\frac{{w^{*}}^{T}T_{k}}{||w^{*}||_{2}}\quad\frac{{\delta
w}^{T}T_{k}}{||\delta
w||_{2}}\right]^{T}\xrightarrow[d_{k}\rightarrow\infty]{\mathcal{D}}\mathcal{N}(0,I)$
(1.30)

almost surely, where $\sigma_{T_{k}}^{2}$ is the variance of the components of
$T_{k}$.

A proof for this CLT proposition is given in Appendix B.

Proposition 2 shows that under the standard conditions above,
${w^{*}}^{T}T_{k}$ and ${\delta w}^{T}T_{k}$ are asymptotically jointly
Gaussian and independent, almost surely. We stress that the components of
$T_{k}$ do not have to be identically distributed to satisfy this property;
Proposition 2 may be adjusted for this case with different normalization
factors. Similarily, the i.i.d. assumption on $T_{k}$ can be relaxed to
Markovian ergodic. It is easy to verify that Proposition 2 can be extended to
the general case where $w^{*},\delta w\in\mathbb{R}^{d_{k}\times d_{k+1}}$,
under similar general position conditions, with almost sure orthogonality of
$w^{*}$ and $\delta w$. we have that

$\frac{1}{\sqrt{\sigma^{2}_{T_{k}}}}\left[\left({w^{*}}^{T}w^{*}\right)^{-1/2}{w^{*}}^{T}T_{k}\quad\left(\delta
w^{T}\delta w\right)^{-1/2}\delta
w^{T}T_{k}\right]\xrightarrow[d_{k}\rightarrow\infty]{\mathcal{D}}\mathcal{N}(0,I)$
(1.31)

almost surely. This means that under the conditions stated above, $w^{*}T_{k}$
and $\delta wT_{k}$ are asymptotically jointly Gaussian and statistically
independent.

We can now bound the mutual information between $T_{k+1}$ and the linear
projection of the previous layer $W^{*}T_{k}$, during the diffusion phase, for
sufficiently high dimensions $d_{k},d_{k+1}$, under the above conditions. Note
that in this case, Equation 1.29 behaves like an additive Gaussian channel
where ${w^{*}}^{T}T_{k}$ is the signal and $\delta w^{T}T_{k}+Z$ is an
independent additive Gaussian noise (i.e., independent of signal and normally
distributed). Hence, for sufficiently large $d_{k}$ and $d_{k+1}$, we can
write

$\displaystyle I(T_{k+1};T_{k}|w^{*})\leq$ $\displaystyle
I(T_{k+1};{w^{*}}^{T}T_{k}|w^{*})$ (1.32) $\displaystyle\leq$ $\displaystyle
I\left({w^{*}}^{T}T_{k}+{\delta w}^{T}T_{k}+Z;{w^{*}}^{T}T_{k}|{w^{*}}\right)$
$\displaystyle=$
$\displaystyle\frac{1}{2}\log\left(\frac{{\left|\sigma_{T_{k}}^{2}{w^{*}}^{T}{w^{*}}+\sigma_{T_{k}}^{2}{\delta
w}^{T}{\delta w}+\sigma^{2}_{z}I\right|}}{{\left|\sigma_{T_{k}}^{2}{\delta
w}^{T}{\delta w}+\sigma^{2}_{z}I\right|}}\right)$

almost surely, where the first inequality is due to DPI for the Markov chain
$T_{k}-{w^{*}}^{T}T_{k}-T_{k+1}$. Finally, we apply an orthogonal eigenvalue
decomposition to the multivariate Gaussian channel in Equation 1.32. Let
${\delta w}^{T}{\delta w}=Q\Lambda Q^{T}$ where $QQ^{T}=I$ and $\Lambda$ is a
diagonal matrix whose diagonal elements are the corresponding eigenvalues,
$\lambda_{i}$, of ${\delta w}^{T}{\delta w}$. Then, we have that

$\displaystyle{\left|\sigma_{T_{k}}^{2}{w^{*}}^{T}{w^{*}}+\sigma_{T_{k}}^{2}{\delta
w}^{T}{\delta w}+Z\right|}=$
$\displaystyle\sigma^{2}_{T_{k}}|Q|\cdot|Q^{T}{w^{*}}^{T}{w^{*}}Q+\Lambda+\frac{\sigma^{2}_{z}}{\sigma^{2}_{T_{k}}}Q^{T}Q|\cdot|Q^{T}|$
(1.33) $\displaystyle=$
$\displaystyle\sigma^{2}_{T_{k}}|Q^{T}{w^{*}}^{T}{w^{*}}Q+\Lambda+\frac{\sigma^{2}_{z}}{\sigma^{2}_{T_{k}}}I|$
$\displaystyle\leq$
$\displaystyle\sigma^{2}_{T_{k}}\prod_{i=1}^{d_{k+1}}\left(A_{ii}+\lambda_{i}+\frac{\sigma^{2}_{z}}{\sigma^{2}_{T_{k}}}\right)$

where $A\triangleq Q^{T}{W^{*}}^{T}{W^{*}}Q$. The last inequality is due to
the Hadamard inequality. Plugging Equation 1.33 into Equation 1.32 yields that
for sufficiently large $d_{k}$ and $d_{k+1}$,

$\displaystyle I(T_{k+1};T_{k}|w^{*})\leq$
$\displaystyle\frac{1}{2}\log\left(\frac{\prod_{i=1}^{d_{k+1}}\left(A_{ii}+\lambda_{i}+\frac{\sigma^{2}_{z}}{\sigma^{2}_{T_{k}}}\right)}{\prod_{i=1}^{d_{k+1}}\left(\lambda_{i}+\frac{\sigma^{2}_{z}}{\sigma^{2}_{T_{k}}}\right)}\right)$
(1.34) $\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{i=1}^{d_{k+1}}\log\left(1+\frac{A_{ii}}{\lambda_{i}+\frac{\sigma^{2}_{z}}{\sigma^{2}_{T_{k}}}}\right)\xrightarrow[\sigma^{2}_{z}\rightarrow
0]{}\frac{1}{2}\sum_{i=1}^{d_{k+1}}\log\left(1+\frac{A_{ii}}{\lambda_{i}}\right).$

As previously established, $\delta w$ is a Brownian motion along the SGD
iterations during the diffusion phase. This process is characterized by a low
(and fixed) variance of the informative gradients (relevant dimensions),
whereas the remaining irrelevant directions suffer from increasing variances
as the diffusion proceeds (see, e.g. sagun2017empirical; zho2018;
jastrzkebski2017three). In other words, we expect the “informative”
$\lambda_{i}$ to remain fixed, while the irrelevant consistently grow as sub-
linearly with time. Denote the set of “informativ” directions as $\Lambda^{*}$
and the set of “non-informative” as $\Lambda_{NI}$. Then our final limit, as
the number of SGD steps grows, is

$I(T_{k+1};T_{k}|w^{*})\leq\frac{1}{2}\sum_{\lambda^{*}_{i}\in\Lambda^{*}}\log\left(1+\frac{A_{ii}}{\lambda^{*}_{i}}\right).$

Note that for real problems the distinction between informative and non-
informative directions may not be that sharp and we can expect a gradual
(exponential asymptotically) decrease of $A_{ii}$ with $i\rightarrow\infty$.
Which directions are compressed and which are preserved depend on the required
compression level. This is why different layers converge to different values
of $I(T_{k};X)$.

#### Relation to Other Works

The analysis above suggests that the SGD compresses during the diffusion phase
in many directions of the gradients. We argue that these directions are the
ones in which the gradients’ variance is increasing (non-informative), whereas
the information is preserved in the directions where the variance of the
gradients remains small.

This statement is consistent with recent work on the statistical properties of
gradients and generalization (sagun2017empirical; zho2018; zhang2018energy).
These works showed that the gradients’ covariance matrix is typically highly
non-isotropic and that this is crucial for generalization by SGD. They
suggested that the reason lies in the gradients’ covariance matrix’s proximity
to the Hessian of the loss approximation. Furthermore, zhang2018energy;
keskar2016large; jastrzkebski2017three argued that SGD tends to converge to
flat minima, which often results in a better generalization. zhang2018energy
emphasized that SGD converges to flat minima values characterized by high
entropy due to the non-isotropic nature of the gradients’ covariance and its
alignment with the error Hessian at the minima. In other words, the findings
above suggest that non-isotropic gradients and Hessian typically characterize
good generalization performance in orthogonal directions to the flat minimum
of the training error objective.

### The Computational Benefit of the Hidden Layers

Our Gaussian bound on the representation compression (Equation 1.34) allows us
to relate the convergence time of the layer representation information,
$I(T_{k};X)$, to the diffusion exponent $\alpha$, defined above. Denote the
representation information at the diffusion time $\tau$ as $I(X;T_{k})(\tau)$.
It follows from Eqaution 1.34 that

$\displaystyle I(X;T_{k})(\tau)\leq
C+\frac{1}{2}\sum_{\lambda_{i}\in\Lambda^{NI}}\log\left(1+\frac{A_{ii}}{\lambda_{i}(\tau)}\right)\leq
C+\frac{1}{2}\sum_{\lambda_{i}\in\Lambda^{NI}}\left(\frac{A_{ii}}{\lambda_{i}(\tau)}\right)$
(1.35)

where $C$ depends on the informative for this layer, but not on $\tau$.

Notice that $\lambda_{i}(\tau)$ are the singular values of the weights of a
diffusion process, which grow as $\tau^{\alpha}$, where $\alpha$ is the
diffusion exponent. Hence,
$\lambda_{i}(\tau)=\lambda^{0}_{i}\cdot\tau^{\alpha}$. Therefore,

$I(X;T_{k})(\tau)\leq
C+\frac{1}{\tau^{\alpha}}{\sum_{\lambda_{i}\in\Lambda^{NI}}\left(\frac{A_{ii}}{\lambda_{i}^{0}}\right)}$

Inverting this relation, the time to compress the representation $T_{k}$ by
$\Delta I(X;T_{k})=\Delta I_{k}$ scales as: $\tau(\Delta
I_{k})\propto\left(\frac{-R}{\Delta I(X;T)}\right)^{\frac{1}{\alpha}}$, where
$R=\frac{1}{2}\sum_{\lambda_{i}\in\Lambda^{NI}}\left(\frac{A_{ii}}{\lambda_{i}^{0}}\right)$.
Note that $R$ depends solely on the problem, $f(x)$ or $p(y,x)$, and not on
the architecture. The idea behind this argument is as follows - one can expand
the function in any orthogonal basis (e.g. Fourier transform). The expansion
coefficients determine both the dimensionality of the relevant/informative
dimensions and the total trace of the irrelevant directions. Since these
traces are invariant to the specific function basis, these traces remain the
same when expanding the function in the network functions using the weights.

With $K$ hidden layers, each layer only needs to compress from the previous
(compressed) layer, by $\Delta I_{k}$, and the total compression is $\Delta
I_{X}=\sum_{k}{\Delta I_{k}}$. Under these assumptions, even if the layers
compress one after the other, the total compression time can be broken down
into $k$ smaller steps, as at

$\left(\frac{R}{\sum_{k}{\Delta
I_{k}}}\right)^{\frac{1}{\alpha}}\ll\sum_{k}{\left(\frac{R}{\Delta
I_{k}}\right)^{\frac{1}{\alpha}}}$

If the $\Delta I_{k}$ are similar, we obtain a super-linear boost in the
computational time by a factor $K^{\frac{1}{\alpha}}$. Since $\alpha\leq 0.5$
this is at least a quadratic boost in $K$. For ultra-slow diffusion, we obtain
an exponential boost (in $K$) in the convergence time to a good
generalization.

### Experiments

The two main predictions that emerge from our theory are:

  * •

The diffusion phase corresponds to a reduction of MI with the input.

  * •

There is a direct connection between the distinct phases in the gradients, the
transfer to the diffusion, and the information’s compression.

We now illustrate our results in a series of experiments. We examine several
different setups. In this group, we have a setup that closely follows the
experiment described in shwartz2017opening. Here, we train a fully-connected
neural network with six hidden layers of width $12-10-7-7-5-4-3$ and a
hyperbolic tangent (tanh) activation functions to produce a binary
classification from a $12$-dimensional input (for more information about the
dataset, see appendix Add it to the appendix)

(b)

(d)

Figure 1.24: MNIST data-set

MNIST dataset – In the first experiment, we evaluate the MNIST handwritten
digit recognition task (lecun1990handwritten). For this data set, we use a
fully connected network with $5$ hidden layers of width $500-250-100-50-20$,
with an hyperbolic tangent (tanh) activation function. The relatively low
dimension of the network and the bounded activation function allow us to
empirically measure the MI in the network. The MI is estimate by binning the
neurons’ output into the interval $[-1,1]$. The discretized values are then
used to estimate the joint distributions and the corresponding MI, as
described by shwartz2017opening.

Figure LABEL:fig:mnist_grad depicts the norms of the weights, the signal-to-
noise ratio (the ratio between the means of the gradients and their standard
deviations), the compression rate $I(X;T)$ and the Gaussian upper bound on
$I(X;T)$, as defined in Equation 1.34. As expected, the two distinct phases
correspond to the drift and diffusion phases. Furthermore, these two phases
are evident by independently observing the SNR, the change of the weights
$||W(l)-W(0)||$, the MI, and the upper bound. In the first phase, the weights
grow almost linearly with the iterations, the SNR of the gradients is high,
and there is almost no change in the MI. Then, after the transition point
(that accrues almost at the same iteration for all the measures above), the
weights behave as a diffusion process. In this phase, the SNR and MI decrease
remarkably. In this phase, there is also a clear-cut reduction of the bound.

CIFAR-10 and CIFAR-100 – Next, we validate our theory on large-scale modern
networks. In the second experiment, we consider two data sets, CIFAR-10 and
CIFAR-100. Here, we train a ResNet-32 network, using a standard architecture
(including ReLU activation functions as described in (he2016deep). In this
experiment, we do not estimate the MI directly due to the problem’s large
scale. Figure 1.25 shows the SNR of the gradients and the Gaussian bound for
one layer in CIFAR-10 and CIFAR-100 on the ResNet-32 network averaged over 50
runs. Here, we observed similar behavior, as reported in the MNIST experiment.
Specifically, there is a clear distinction between the two phases and a
reduction of the MI bound along with the diffusion phase. Note that the same
behavior was observed in most of the $32$ layers in the network.

Recently, several attempts characterize the correspondence between the
diffusion rate of the SGD and the size of the mini-batch (hu2017diffusion;
hoffer2017train). In these articles, the authors claimed that a larger mini-
batch size corresponds to a lower diffusion rate. Here, we examine the effect
of the mini-batch size on the transition phase in the information plane. For
each mini-batch size, we find both the starting point of the information
compression and the gradient phase transition (the iteration where the
derivative of the SNR is maximal). Figure 1.32 illustrates the results. The
$X$-axis is the iteration where the compression started, and the $Y$-axis is
the iteration where the phase transition in the gradients accrued for
different minibatch sizes. There is a clear linear trend between the two. This
further justifies our suggested model since the two measures are strongly
related.

(b)

(d)

Figure 1.26: Change in the SNR of the gradients and the Gaussian bound on the
MI during the training of the network for one layer on ResNet-32, in log-log
scale.

NNext, we validate our results on the computational benefit of the layers. We
train networks with different number of layers (1-5 layers) and examine the
number of iterations a network takes to converge. Then, we find the $\alpha$
which fits the best trend $K^{\frac{1}{\alpha}}$, where $K$ is the number of
layers. Figure 1.27 shows the results for two data-sets - MNIST and the
symmetric dataset from shwartz2017opening. As our theory suggest, as we
increase the number of layers, the convergence time decreases with a factor of
$k^{\frac{1}{\alpha}}$ for different values of $\alpha$.

(b)

(d)

Figure 1.28: The computational benefit of the layers - The converged iteration
as function of the number of layers in the network

Figure 1.30 demonstrates the Gaussian upper bound on $I(X;T)$, as defined in
(1.34), for all the layers in the network, starting from the diffusion phase.
As we can see, there is an evident reduction in all the layers.

Figure 1.30: The Gaussian bound in each layer during the training of the
network. The iterations start from the beginning of the diffusion phase

Figure 1.32: The transition point of the SNR of gradient versus the starting
point of the compression in the information plane, for different mini-batch
sizes. Each point is a different batch size. The $X$-axis is the iteration
with the maximal $I(X;T_{k})$, while the $Y$-axis describes the iteration with
the minimum value of the gradient’s derivative (the phase transition)

### Discussion and Conclusions

In this work, we study DNNs using information-theoretic principles. We
describe the network’s training process as two separate phases, as has been
previously done by others. In the first phase (drift), we show that
$I(T_{k};Y)$ increases, corresponding to an improved generalization with ERM.
In the second phase (diffusion), the representation information, $I(X;T_{k})$
slowly decreases, while $I(T_{K};Y)$ continues to increase. We rigorously
prove that the representation compression is a direct consequence of the
diffusion phase, independent of the nonlinearity of the activation function.
We provide a new Gaussian bound on the representation compression and then
relate the diffusion exponent to the compression time. One key outcome of this
analysis is a new proof of the computational benefit of the hidden layers,
where we show that they boost the overall convergence time of the network by
at least a factor of $K^{2}$, where $K$ is the number of non-degenerate hidden
layers. This boost can be exponential in the number of hidden layers if the
diffusion is “ultra-slow”, as recently reported. Interestingly, an equivalent
distinction to two separate phases was also observed by others (beginning with
(murata1998statistical)) in independent studies of the SGD process. There, it
is claimed that the SGD first seeks a minimizer of the objective function and
then lingers around it. We claim that this is not by chance; the SGD algorithm
is the key to understanding these phenomena. In other words, we show that
minimizing the objective corresponds to maximizing the MI with the target,
while the random walk around the minimum results in a reduction of MI with the
input patterns, which can further improve the generalization. This
understanding allows us to characterize the behavior of the DNN on the
information plane and claim that there is a fundamental limit for its
performance – the Information Bottleneck curve.

### Appendix

### Appendix A \- proof of Theorem 1

We first revisit the well-known Probably Approximately Correct (PAC) bound.
Let $\mathcal{H}$ be a finite set of hypotheses. Let
$\ell_{h}\left(x_{i},y_{i}\right)$ be a bounded loss function, for every
$h\in\mathcal{H}$. For example,
$\ell_{h}\left(x_{i},y_{i}\right)=(y_{i}-h(x_{i}))^{2}$ is the squared loss
while $\ell_{h}\left(x_{i},y_{i}\right)=-y_{i}\log h(x_{i})$ is the
logarithmic loss (which may be treated as bounded, assuming that the
underlying distribution is bounded away from zero and one). Let
$\mathcal{L}_{h}\left(S_{m}\right)=\frac{1}{m}\sum_{i=1}^{m}\ell_{h}\left(x_{i},y_{i}\right)$
be the empirical error. Hoeffding’s inequality hoeffding1963probability shows
that for every $h\in\mathcal{H}$

$\mathbb{P}\left[\big{|}\mathcal{L}_{h}\left(S_{m}\right)-\mathbb{E}_{S_{m}}\left[\mathcal{L}_{h}\left(S_{m}\right)\right]\big{|}\geq\epsilon\right]\leq
2\exp{\left(-2\epsilon^{2}m\right)}.$ (1.36)

Then, we can apply the union bound and conclude that

$\mathbb{P}\left[\exists
h\in\mathcal{H}\bigg{|}\big{|}\mathcal{L}_{h}\left(S_{m}\right)-\mathbb{E}_{S_{m}}\left[\mathcal{L}_{h}\left(S_{m}\right)\right]\big{|}\geq\epsilon\right]\leq
2\big{|}\mathcal{H}\big{|}\exp{\left(-2\epsilon^{2}m\right)}.$

We want to control the above probability with a confidence level of $\delta$.
Therefore, we ask that
$2\big{|}\mathcal{H}\big{|}\exp{\left(-2\epsilon^{2}m\right)}\leq\delta$. This
leads to a PAC bound, which states that for a fixed $m$ and for every
$h\in\mathcal{H}$, we have with probability $1-\delta$ that

$\big{|}\mathcal{L}_{h}\left(S_{n}\right)-\mathbb{E}_{Sm}\left[\mathcal{L}_{h}\left(S_{m}\right)\right]\big{|}^{2}\leq\frac{\log\big{|}\mathcal{H}\big{|}+\log{\frac{2}{\delta}}}{2m}.$
(1.37)

Note that under the definitions stated above, we have that $|\mathcal{H}|\leq
2^{\mathcal{X}}$. However, the PAC bound above also holds for a infinite
hypotheses class, where $\log|\mathcal{H}|$ is replaced with the VC dimension
of the problem, with several additional constants (vapnik1968uniform;
shelah1972combinatorial; sauer1972density).

Let us now assume that $X$ is a $d$-dimensional random vector that follows a
Markov random field structure. As stated above, this means that
$p(x_{i})=\prod_{i}p(x_{i}|Pa(x_{i}))$ where $Pa(X_{i})$ is a set of
components in the vector $X$ that are adjacent to $X_{i}$. Assuming that the
Markov random field is ergodic, we can define a typical set of realizations
from $X$ as a set that satisfies the Asymptotic Equipartition Property (AEP)
(cover2012elements). Therefore, for every $\epsilon>0$, the probability of a
sequence drawn from $X$ to be in the typical set $A_{\epsilon}$ is greater
than $1-\epsilon$ and $|A_{\epsilon}|\leq 2^{H(X)+\epsilon}$. Hence, if we
only consider a typical realization of $X$ (as opposed to every possible
realization), we have that asymptotically $\big{|}\mathcal{H}\big{|}\leq
2^{H(X)}$. Finally, let $T$ be a mapping of $X$. Then, $2^{H(X|T)}$ is the
number of typical realizations of $X$ that are mapped to $T$. This means that
the size of the typical set of $T$ is bounded from above by
${2^{H(X)}}\big{/}{2^{H(X|T)}}=2^{I(X;T)}$. Plugging this into the PAC bound
above yields that with probability $1-\delta$, the typical squared
generalization error of $T$ ,$\epsilon_{T}^{2}$ satisfies

$\epsilon_{T}^{2}\leq\frac{2^{I(X;T)}+\log{\frac{2}{\delta}}}{2m}.$ (1.38)

### Appendix B - Proof of Proposition 2

We make the following technical assumptions:

  1. 1.

${w^{*}}$ and $\delta w$ satisfy
$\lim_{d_{k}\rightarrow\infty}{w^{*}}^{T}\delta w=0$ almost surely.

  2. 2.

The moments of $T_{k}$ are finite.

  3. 3.

The components of $w^{*}$ and $\delta w$ are in-general-positions, satisfying

$\lim_{d_{k}\rightarrow\infty}{\sum_{i=1}^{d_{k}}{w^{*}_{i}}^{4}}\big{/}{\left(\sum_{i=1}^{d_{k}}{w^{*}_{i}}^{2}\right)^{2}}=0$

and

$\lim_{d_{k}\rightarrow\infty}{\sum_{i=1}^{d_{k}}{\delta
w_{i}}^{4}}\big{/}{\left(\sum_{i=1}^{d_{k}}{\delta w_{i}}^{2}\right)^{2}}=0$

almost surely.

Consider a sequence of i.i.d. random variables,
$\left\\{X_{i}\right\\}_{i=1}^{d}$ with zero mean and finite moments,
$\mathbb{E}\left[X_{i}^{r}\right]<\infty$ for every $r\geq 1$.

Let $\\{a_{i}\\}_{i=1}^{d}$ be a sequence of constants. Denote
$Y_{i}=a_{i}X_{i}$, so that $\left\\{Y_{i}\right\\}_{i=1}^{d}$ are independent
with zero mean and $\text{Var}(Y_{i})=a^{2}_{i}\mathbb{E}\left[X^{2}\right]$.
Let $S=\sum_{i=1}^{d}a_{i}X_{i}=\sum_{i=1}^{d}Y_{i}$ and denote
$U_{d}^{2}=\sum_{i=1}^{d}\text{Var}(Y_{i})=\mathbb{E}[X^{2}]\sum_{i=1}^{d}a_{i}^{2}$.

The Lyapunov Central Limit Theorem (CLT) billingsley2008probability states
that if there exists some $\delta>0$ for which

$\lim_{d\rightarrow\infty}\frac{1}{U_{d}^{2+\delta}}\sum_{i=1}^{d}\mathbb{E}\left[|Y_{i}|^{2+\delta}\right]=0$
(1.39)

then

$\frac{1}{U_{d}}\sum_{i=1}^{d}Y_{i}\xrightarrow[d\rightarrow\infty]{\mathcal{D}}\mathcal{N}(0,1).$
(1.40)

Plugging $\delta=2$ yields the following sufficient condition,

$\lim_{d\rightarrow\infty}\frac{1}{U_{d}^{4}}\sum_{i=1}^{d}\mathbb{E}\left[Y_{i}^{4}\right]=\frac{\sum_{i=1}^{d}a_{i}^{4}}{\left(\sum_{i=1}^{d}a_{i}^{2}\right)^{2}}\frac{\mathbb{E}[X^{4}]}{\mathbb{E}^{2}[X^{2}]}=0$
(1.41)

Let us apply the Lyapunov CLT to our problem. Here, the components of $T_{k}$
are i.i.d. for sufficiently large $d_{k}$, with zero mean and finite $r^{th}$
moments for every $r\geq 1$. Furthermore, we assume that the components of
$w^{*}$ and $\delta w$ are in-general-positions. This means that Lyapunov
condition (1.41) is satisfied for both ${w^{*}}^{T}T_{k}$ and ${\delta
w}^{T}T_{k}$ almost surely, which means that

$\frac{1}{\sqrt{\sigma_{T_{k}}^{2}}||w^{*}||_{2}}{w^{*}}^{T}T_{k}\xrightarrow[d_{k}\rightarrow\infty]{\mathcal{D}}\mathcal{N}(0,1)$
(1.42)

and

$\frac{1}{\sqrt{\sigma_{T_{k}}^{2}}||\delta w||_{2}}{\delta
w}^{T}T_{k}\xrightarrow[d_{k}\rightarrow\infty]{\mathcal{D}}\mathcal{N}(0,1).$
(1.43)

almost surely, where $\sigma_{T_{k}}^{2}$ is the variance of the components of
$T_{k}$.

Furthermore, for every pair of constants $a$ and $b$, the linear combination
$\left(a{w^{*}}+b\delta w\right)^{T}T_{k}$ also satisfies Lyapunov’s condition
almost surely, which means that ${w^{*}}^{T}T_{k}$ and ${\delta w}^{T}T_{k}$
are asymptotically jointly Gaussian, with

$\mathbb{E}\left[{w^{*}}^{T}T_{k}\left({\delta
w}^{T}T_{k}\right)^{T}\right]=\sigma_{T_{k}}^{2}{w^{*}}^{T}\delta
w\xrightarrow[d_{k}\rightarrow\infty]{}0$

almost surely. $\square$

### Appendix C - The benefit of the hidden layers To illustrate the layers’
computational benefit, we trained six different architectures with $1-6$
hidden layers, with the same settings as described in shwartz2017opening. We
train a fully connected network, where the activation function was hyperbolic
tangent function. We train it on 80% of the patterns and repeated each
training 50 times with randomized initial weights and training samples. Figure
1.34 demonstrates the Information Paths for these six architectures during the
training epochs, averaged over different networks. Three main conclusions
arise from our experiments. 1\. _Adding hidden layers dramatically reduces the
number of training epochs to attain good generalization._ Compare the color of
the paths in the left panels of Figure 1.34 (which correspond to $1$ and $2$
hidden layers) with the colors of the right panels (which correspond to $5$
and $6$ hidden layers). While a single-layer network cannot achieve a high
value of $I(T;Y)$, even after $10^{4}$ epochs, a $6$ layers network reaches
the full relevant information at the output layer after only $400$ epochs. 2\.
_The diffusion phase of each layer is shorter when it starts from a previously
compressed layer._ This can be seen by comparing the elapsed time required to
attain good generalization with $4$ and $5$ hidden layers. The yellow color at
the left indicates a much slower convergence with $4$ layers than networks
with $5$ or $6$ layers. 3\. _The compression is faster for the deeper (closer
to the output) layers._ During the drift phase, the lower layers move first
due to the data processing inequality. Then, during the diffusion phase, the
top layers compress first and” pull” the lower layers with them. Adding more
layers seems to add intermediate representations, which accelerates the
compression.

Figure 1.34: The network Information Paths during the SGD optimization when
adding hidden layers. Each panel is the information plane for a network with a
different number of hidden layers. The width of the hidden layers start with
$12$, and each additional layer has $2$ fewer neurons. The last layer with $2$
neurons is shown in all panels. The line colors correspond to the number of
training epochs.