# Polarity imbalance of the photospheric magnetic field

ElenaVernova  Marta Tyasto  Dmitrii Baranov  Olga Danilova  IZMIRAN, SPb.
Filial, Laboratory of Magnetospheric Disturbances, St. Petersburg, Russian
Federation  Ioffe Physical-Technical Institute, St. Petersburg, Russian
Federation

###### keywords:

Magnetic fields, Photosphere; Polarity imbalance, Sunspot zone, Polar field

##  1 Introduction

intro Magnetic field of the Sun varies with a 22-year periodicity that
manifests itself both in the change of signs of leading and following sunspots
during solar minimum (the Hale’s law), and in the change of sign of the polar
field (polar field reversal) during period of maximum solar activity (see,
e.g., Charbonneau, 2010). The distributions of the local and the global
magnetic fields exhibit antisymmetry of polarities with respect to the solar
equator. At the same time, there exists some asymmetry of the magnetic fields
which is reflected in the asymmetric distribution of different forms of solar
activity. The north-south asymmetry was discovered in various manifestations
of solar activity, such as sunspots, flares, or sudden disappearances of solar
prominences (see, e.g., Carbonell et al., 2007; Ballester, Oliver, and
Carbonell, 2005; Swinson, Shea, and Humble, 1986; Deng et al., 2016, and
references therein).

The magnetic fluxes of the Sun and their imbalance were studied on the basis
of different data that characterize the magnetic activity. Asymmetry of the
leading and following sunspot polarities in an active region was studied by
many authors (see, e.g., van Driel-Gesztelyi and Petrovay, 1990; Fan, 2009).

During the polar field reversals the two hemispheres develop to some extent
independently (Svalgaard and Kamide, 2013); as a result the polar fields
complete their reversals not synchronously. There exist time intervals when
the global solar field loses its dipole structure and behaves like a monopole
(Wilcox, 1972; Kotov, 2009). For example, during the solar cycle 24 the south
polar field completed its reversal from positive to negative in 2014. In
contrast, the north polar field, after a series of reversals changed its sign
from negative in 2012 to positive at the beginning of 2015 (Wang, 2017). As
emphasized by Kotov (2009), there is no theoretic explanation of the fact that
the positive or negative field can dominate practically on the whole Sun for
one year or more.

Reversal of the polar magnetic field affects not only the features of solar
activity, but also considerably changes the structure of interplanetary space.
Significant difference was found in the intensity-time profiles of the
galactic cosmic rays around the solar activity minima in the alternate solar
magnetic field polarities (Lockwood, Webber, and Debrunner, 2001). One of
manifestations of the Sun’s magnetic field asymmetry is the displacement of
heliospheric current sheet in the south direction (Mursula and Hiltula, 2003;
Erdös and Balogh, 2010). Observations of the interplanetary magnetic field
(IMF) have suggested a statistical tendency for the heliospheric current sheet
to be shifted a few degrees southward of the heliographic equator during the
period 1965 – 2010, particularly in the years near sunspot minimum (Wang and
Robbrecht, 2011).

The important information on the asymmetry of the Sun’s magnetic field can be
obtained by consideration of the magnetic field net flux and its changes in
time. Net flux or flux imbalance can be defined as

$\ilabel{NetFl}F_{net}=|F^{pos}|-|F^{neg}|.$ (1)

i.e., as a difference of positive and negative polarity fluxes. The similar
parameter was used by Tian, Liu, and Wang (2003) while considering magnetic
fluxes for twelve solar active regions in the Cycle 23. It was found that the
fluxes of these regions were appreciably imbalanced. For construction of the
butterfly diagram (Choudhary, Venkatakrishnan, and Gosain, 2002; Petrie, 2012)
signed net flux was used also.

Choudhary, Venkatakrishnan, and Gosain (2002) found, using the magnetograms
obtained from the National Solar Observatory at Kitt Peak, that the maximum
and the median values of the flux imbalance for 137 active regions were
respectively 62% and 9.5%. The $10^{\circ}-40^{\circ}$ active latitudinal zone
in the individual hemispheres during the solar maximum showed a flux imbalance
of more than 20%. This is reduced to below 10% when the entire Sun is
considered.

When studying hemispheric asymmetry of magnetic fields Petrie and Ettinger
(2015) explored correlations between three classes of photospheric magnetic
field: active region fields, which are compact, intense, flux-balanced and
usually bipolar in structure, located equatorward of about $\pm 30^{\circ}$;
polar fields, which are located poleward of $\pm 60^{\circ}$ and are
relatively weak, almost unipolar and have large spatial scale, so that over
most of the solar cycle they appear dipole-like; and high-latitude surges of
field forming from decaying active regions, observed to stream poleward
between about $\pm 30^{\circ}$ and $\pm 60^{\circ}$ from active latitudes in
plume-like formations.

The present work continues the studies of Vernova, Tyasto, and Baranov (2014)
where the problem of polarity imbalance of magnetic flux for the sunspot zone
(latitudes form $-40^{\circ}$ to $40^{\circ}$ was considered for the period
1976 – 2003 and for fields greater than $100$ G. Here we used the data of
SOLIS which enabled us to continue the studied period till year 2016. Also,
somewhat different threshold values were used (fields greater than $50$ G and
latitudes $5^{\circ}-40^{\circ}$). However, the main novelty of the present
work is that the polarity imbalance of the sunspot zone is compared with that
of the high-latitude fields and their connection with the quadrupole moment is
established.

In papers Vernova, Tyasto, and Baranov (2016) and Vernova, Tyasto, and Baranov
(2017) the connection of magnetic field intensity with location at certain
heliospheric latitudes was studied. It was shown that in the latitudinal
distribution of the magnetic field averaged for three cycles strong magnetic
fields with strength of 15 G and higher are located in sunspot zone from
$5^{\circ}$ to $40^{\circ}$. Strong fields from both sides are surrounded by
the weakest fields (less than 5 G: latitudes from $0^{\circ}$ to $5^{\circ}$
and from $40^{\circ}$ to $60^{\circ}$). Above $40^{\circ}$ field strength does
not exceed 15 G, except for a narrow strip of latitudes around $70^{\circ}$
where polar faculae with fields from 15 to 50 G are observed.

In the present paper time-dependencies of the magnetic flux for high-latitude
and low-latitude magnetic fields are studied as well as the imbalance of
positive and negative fluxes. Following criteria for distinction between low
and high latitude regions were used with this aim: low latitudes (sunspot
zone) from $5^{\circ}$ to $40^{\circ}$ in each hemisphere; high latitudes from
$40^{\circ}$ to $90^{\circ}$ in each hemisphere. The choice of boundary
latitude of $40^{\circ}$ allows to consider latitudinal regions with quite
different properties of magnetic fields. Only strong fields ($B>50$ G) were
considered in the low latitude region, whereas in the high latitude region the
weaker fields ($B<50$ G) were taken into account. The lower/upper limit of
magnetic field intensity ($B=50$ G) was used to underline a role of
strong/weak fields in fluxes of low/high latitudes, accordingly. The choice of
this limit makes no major difference and does not influence the main
conclusions.

In Section DataMeth we describe the data and discuss the method applied in the
article. Section Time is devoted to the time changes of absolute values of
magnetic fluxes. In Section Imbal positive and negative magnetic fluxes and
their imbalance for a) high latitudes and b) sunspot zone are considered
separately. In Section Build we discuss and interpret the obtained results and
consider four building blocks of the 22-year magnetic cycle. In Section Concl
the main conclusions are drawn.

##  2 Data and Method

DataMeth For our study we used synoptic maps of the photospheric magnetic
field produced at the NSO Kitt Peak (from 1976 to 2016). Data for 1976 – 1977
had many gaps and were not included in our analysis. Combining data sets of
two devices, KPVT for 1978 – 2003 (available at
ftp://nispdata.nso.edu/kpvt/synoptic/mag/) and SOLIS for 2003 – 2016
(available at https://magmap.nso.edu/solis/archive.html) allowed to study
evolution of magnetic fields for nearly four solar cycles. Synoptic maps have
the following spatial resolution: $1^{\circ}$ in longitude (360 steps), 180
equal steps in the sine of the latitude from $-1$ (south pole) to $+1$ (north
pole). Thus, every map consists of $360\times 180$ pixels of magnetic field
strength values in Gauss. On the assumption that the fields are radial to the
solar surface, the average line-of-sight component from the original
observations was divided by the cosine of the heliocentric angle (the angle
between the line-of-sight and the vertical direction).

Noisy values near the poles were detected and replaced by a cubic spline fit
to valid values in the polar regions. Although the instrumental noise per
resolution element is reduced by the map construction process, the data are
less reliable at the poles (in the region with a viewing angle larger than
$80^{\circ}$) because of noisy measurements associated with the limb and the
relatively small number of original measurements that cover the poles.

However, in the sine of the latitude representation of the Kitt Peak data,
only one latitude zone corresponds to the range $80^{\circ}-90^{\circ}$, and
so this part of the data becomes negligible after the averaging and does not
affect the results of the paper. The NSO data are complete in the sense that
where observational data are missing, such as when the tilt of the Sun’s axis
to the ecliptic causes one of the polar regions to be invisible, interpolated
values are used (Durrant and Wilson, 2003). Using as initial data magnetic
field synoptic maps, we have calculated fluxes for different intensities of
magnetic fields and for different latitudinal ranges.

##  3 Time changes of magnetic fluxes

Time The magnetic field strength for both hemispheres (1978 – 2016) shows a
nearly symmetric distribution of positive and negative fields with 65.2% of
the pixels in the strength range $0-5$ G, whereas pixels with the strength
above 50 G occupy only 2.6% of the solar surface. Magnetic fields in the
$5-50$ G strength range occupy 32.2%. Strong magnetic fields of both
polarities occupy a relatively small part of the Sun’s surface. However just
the strong magnetic fields of active regions give the main contribution to the
total magnetic flux (see Figure compval).

Let us consider time changes of the magnetic flux for the whole range of
heliolatitudes, but for different values of magnetic field strength. In Figure
compvala the flux of the strongest magnetic fields ($B>50$ G) is presented.
Only absolute values of magnetic field strength were considered. For
comparison 13-month smoothed monthly sunspot number is shown
(http://sidc.oma.be/silso/DATA/SN˙ms˙tot˙V2.0.txt) by blue line in Figure
compvala. The magnetic flux tracks the 11-year cycle based on sunspot number,
reaching a maximum around the time of the second (Gnevyshev) maximum of solar
activity (SA). In the minimum of SA the flux falls practically to zero. In
three consecutive maxima (Cycles 22, 23 and 24) the flux maximum decreased
monotonously more than two times. For the sunspot number this decrease is
observed during four solar cycles.

Figure 1.: Time changes of the magnetic flux for different ranges of strength
(absolute values) for the period 1978 – 2016: (a) fields with $|B|>50$ G and
sunspot number (SSN); (b) $50$ G $>|B|>5$ G; (c) $|B|<5$ G. Black line –
magnetic flux for each Carrington rotation; red line – smoothed curves, which
were obtained using 20-rotation running mean.compval

For magnetic fields from 5 to 50 G (Figure compvalb) time changes closely
agree with time changes of the strongest fields (correlation coefficient
$R=0.91$). Changes with the 11-year cycle and monotonous decrease of the flux
maxima can be seen for three last solar cycles (noted in Figure compvalb by a
straight line). The difference between Figures compvala and compvalb is that
the flux of $5-50$ G fields is lower in comparison with the flux of the
strongest fields. Other difference is in that the $5-50$ G flux does not fall
anywhere below a threshold $~{}1\times 10^{23}$ Mx.

Very special time change display fields with strength $B<5$ G which develop in
antiphase with the solar cycle (Figure compvalc). In years when the flux of
the strong fields reaches its maximum, fields of $B<5$ G have a minimum. The
minimum flux increases from Cycle 22 to 24 while for the fields of higher
strength the flux maximum falls (compare Figures compvalb and compvalc). The
maximum values of fluxes of weak magnetic fields ($B<5$ G) are about ten times
lower, than the total flux of stronger fields ($B>5$ G). Correlation
coefficient between fields of $0-5$ G and fields $B>50$ G is $R=-0.91$, and
between fields of $0-5$ G and fields of $5-50$ G it is equal to $R=-0.98$.
Features of the weakest-field change are in accordance with the results
obtained by analysis of SOHO/MDI data for 1996 – 2011 which show that magnetic
structures with low fluxes change in antiphase with the solar cycle (Jin and
Wang, 2014).

Figure 2.: Change of magnetic flux (absolute values) in different latitude
ranges for Solar Cycles 21–24: (a) low latitudes $5^{\circ}-40^{\circ}$,
$B>50$ G; (b) high latitudes $40^{\circ}-90^{\circ}$, $B<50$ G. Black line –
magnetic flux for each Carrington rotation; red line – 20-point moving
average.complat

In Figure complat change of magnetic flux for Solar Cycles 21–24 is shown
separately for low latitudes $5^{\circ}-40^{\circ}$, $B>50$ G (a) and for high
latitudes $40^{\circ}-90^{\circ}$, $B<50$ G (b). Only absolute values of
magnetic field strength were considered. For low latitudes the main flux is
connected with active regions; thus, the magnetic flux at low latitudes
changes with a 11-year cycle (Figure complata), and the flux maximum coincides
with the time of the second maximum of Gnevyshev (in the same way as in Figure
compvala). The time course of magnetic flux for high latitudes (Figure
complatb) does not show appreciable recurrence. The basic feature of magnetic
flux at high latitudes is monotonous decrease (approximately two times) from a
maximum around year 1991, to the minimum values around 2014, that is within 25
years. According to (2009), after 1980 the magnetic moment of the solar dipole
showed a tendency to gradually decrease and, in 2007, it has already reached
values lower than those at the beginning of the 20th century. The polar field
during the Cycle 23 minimum was about 40% weaker than during two previous
cycles (Wang, Robbrecht, and Sheeley, 2009). One can see the general
tendencies of magnetic flux decrease both at high, and at low latitudes:
falling of the polar field during three cycles coincides with decline of the
maxima of these cycles for low latitude fields.

##  4 Imbalance of positive and negative magnetic fluxes

Imbal

###  4.1 High latitudes

Highlat The results obtained previously (Vernova, Tyasto, and Baranov, 2016;
Vernova, Tyasto, and Baranov, 2017) show that the latitude $40^{\circ}$ is the
boundary, above which the main contribution is made by the magnetic fields
lower than 50 G.

Figure 3.: Magnetic fluxes of high-latitude fields from $+40^{\circ}$ to
$+90^{\circ}$ and from $-40^{\circ}$ to $-90^{\circ}$ ($B<50$ G): (a),(b) –
positive fluxes of high latitudes of the northern ($F_{N}^{pos}$) and the
southern ($F_{S}^{pos}$) hemispheres; (c), (d) – absolute values of negative
fluxes in the northern ($|F_{N}^{neg}|$) and the southern ($|F_{S}^{neg}|$)
hemispheres. Black lines – flux values for each solar rotation, red lines – 20
points smoothing.highflux

At the same time, latitudes from $40^{\circ}$ to $90^{\circ}$ break into three
latitudinal strips, each of which corresponds to different intensities of
magnetic fields and different manifestations of solar activity: 1) fields of
$15-50$ G from $65^{\circ}$ to $75^{\circ}$ (polar faculae); 2) fields of
$5-15$ G from $60^{\circ}$ to $90^{\circ}$ (coronal holes); 3) fields lower
than 5 G from $40^{\circ}$ to $60^{\circ}$. Time changes and flux imbalance
proved to be similar for these field groups which allows to combine the three
groups and to study magnetic fluxes of high latitude fields from $+40^{\circ}$
to $+90^{\circ}$ and from $-40^{\circ}$ to $-90^{\circ}$ ($B<50$ G). In
Figures highfluxa,b positive fluxes of high latitudes of the northern
($F_{N}^{pos}$) and the southern ($F_{S}^{pos}$) hemispheres are presented.
For negative fluxes in Figures highfluxc,d their absolute values
($|F_{N}^{neg}|,|F_{S}^{neg}|$) are presented.

Positive and negative fluxes for each hemisphere change with the period of 22
years opposite in phases to each other. Transitions from low values to high
ones and from high to low values occur quickly, in comparison with wide flat
maxima that is most pronounced in the northern hemisphere. It is worth noting
the sharp increases of the flux (black line, without smoothing) in the
northern hemisphere lasting for several solar rotations in 1981.2 and in
1991.2. The peak in 1981.2 occurred in the positive flux, whereas sharp
increase of negative flux produced the peak in 1991.2. Approximately at the
same periods of time (in 1982 and 1991) Wang (2004)) found the highest peaks
in the IMF strength during last 40 years, and these peaks coincided with peaks
in the equatorial dipole strength.

Figure 4.: (a) Imbalance of magnetic fluxes for the northern ($\Delta
F_{N}=F_{N}^{pos}-|F_{N}^{neg}|$, blue curve) and the southern ($\Delta
F_{S}=F_{S}^{pos}-|F_{S}^{neg}|$, red curve) hemispheres. The average
imbalance ($\Delta F_{avg}=(\Delta F_{N}-\Delta F_{S})/2$, black curve). (b)
The dipole moment $g_{10}$ of the multipole expansion of the photospheric
magnetic field (data of the WSO observatory).imbg10

In Figure imbg10a the imbalance of magnetic fluxes for the northern ($\Delta
F_{N}=F_{N}^{pos}-|F_{N}^{neg}|$, blue curve) and the southern ($\Delta
F_{S}=F_{S}^{pos}-|F_{S}^{neg}|$, red curve) hemispheres are shown. The
average imbalance ($\Delta F_{avg}=(\Delta F_{N}-\Delta F_{S})/2$, black
curve) changes in phase with the northern hemisphere imbalance. For latitudes
above $40^{\circ}$ dominating fields in each hemisphere are those whose sign
coincides with the sign of the polar field in this hemisphere. This sign
changes once in 11 years after polar field reversal. Thus, the imbalance of
positive and negative fluxes in a separate hemisphere changes with the 22-year
period. The dipole moment $g_{10}$ of the multipole expansion of the
photospheric magnetic field according to potential-field source-surface model
(PFSS) (Hoeksema and Scherrer, 1986) is presented in Figure imbg10b (data of
the WSO observatory, available at http://wso.stanford.edu/). Comparison of the
average imbalance with the dipole moment $g_{10}$ shows their good coincidence
(Figure imbg10b). These results provide evidence that the fields in the
latitude range of $40^{\circ}-90^{\circ}$ are directly connected with dipole
component of the Sun’s magnetic field.

Figure 5.: (a) Imbalance $\Delta F_{high}$ of positive and negative fields of
both polar caps for the fields of $B<50$ G ($\Delta
F_{high}=|F_{N}^{pos}+F_{S}^{pos}|-|F_{N}^{neg}+F_{S}^{neg}|$). (b) Axial
quadrupole moment $g_{20}$ of the photospheric magnetic field.imbhigh

Imbalance $\Delta F_{high}$ of positive and negative fields of both polar caps
for the fields of $B<50$ G was calculated (Figure imbhigha) as the sum of the
signed high latitude fluxes for the northern hemisphere
$F_{N}^{pos},F_{N}^{neg}$ (from $+40^{\circ}$ to $+90^{\circ}$ and for the
southern one $F_{S}^{pos},F_{S}^{neg}$ (from $-40^{\circ}$ to $-90^{\circ}$):

$\ilabel{Imbal1}\Delta
F_{high}=|F_{N}^{pos}+F_{S}^{pos}|-|F_{N}^{neg}+F_{S}^{neg}|$ (2)

Throughout four solar cycles the strict regularity of the imbalance change is
observed (change of the imbalance sign is marked in Figure imbhigha by green
circles). From one maximum of SA to another the imbalance looks like two peaks
of one sign divided by the period of the SA minimum. The sign of the imbalance
remains constant during 11 years from one polar field reversal to the other.
Thus the full period of change of the imbalance sign makes 22 years. In the
lower part of Figure imbhigha, the polarity of the polar magnetic field in the
southern hemisphere is displayed. Shading marks time intervals when the
imbalance sign coincides with the sign of polar field in the southern
hemisphere: positive sign – dark blue shading, and negative sign – green
shading. Most of the time the sign of the imbalance coincides with the sign of
the polar magnetic field in the southern hemisphere.

In Figure imbhighb the axial quadrupole moment $g_{20}$ of the photospheric
magnetic field is shown (data of the WSO observatory). Good coincidence of the
imbalance sign with the sign of the quadrupole moment can be seen. Not only
the sign, but also the basic features of the time course of the imbalance and
the quadrupole moment are very close. The study of the north-south asymmetry
of the solar and heliospheric magnetic field (Bravo and González-Esparza,
2000) showed that during activity minima the magnitude of the northern
magnetic field is smaller than the magnitude of the southern magnetic field.
According to the PFSS model the dominant multipoles are the dipole, the
hexapole, and the quadrupole. The sign of the quadrupole component is opposite
to that of the dipole and hexapole in the north, but the three components have
the same sign in the south which leads to domination of the southern
hemisphere fields. This agrees with the fact that the imbalance sign coincides
always with the sign of the polar field of the southern hemisphere (Figure
imbhigh ).

The obtained results allow us to draw the following conclusions for high-
latitude fields (from $40^{\circ}$ to $90^{\circ}$ in each hemisphere), $B<50$
G. The imbalance of positive and negative fields in a separate hemisphere and
the total imbalance of high-latitude fields for two hemispheres change with
the 22-year period. Change of the sign of the imbalance occurs near reversal
of the polar field. However, while in a separate hemisphere the imbalance sign
changes similarly to the polar field in this hemisphere and to the dipole
moment $g_{10}$, the sign of the total imbalance for two hemispheres always
coincides with the sign of the polar field in the southern hemisphere. Change
of the sign of the total imbalance coincides with the sign change of the
quadrupole moment $g_{20}$.

###  4.2 Active zone

Actzone In our papers (Vernova, Tyasto, and Baranov, 2016; Vernova, Tyasto,
and Baranov, 2017) it was shown that strong magnetic fields which occupy
latitude range from $5^{\circ}$ to $40^{\circ}$ are surrounded from both sides
by the weakest fields ($B<5$ G, latitudes from $0^{\circ}$ to $5^{\circ}$ and
from $40^{\circ}$ to $60^{\circ}$).

In the present paper for the study of strong fields in the sunspot zone we set
the following boundaries: field strength $B>50$ G and latitudinal regions from
$5^{\circ}$ to $40^{\circ}$. Positive and negative magnetic fluxes of the
near-equatorial region $\pm 5^{\circ}$ display special features which we do
not consider in this paper.

In Figure fourflux fluxes of strong magnetic fields ($B>50$ G) are presented
for the sunspot zone ($5^{\circ}-40^{\circ}$). For each synoptic map four
different characteristics of magnetic flux were obtained: absolute values of
positive and negative fluxes for the northern and the southern hemispheres –
$F_{N}^{pos}$, $F_{S}^{pos}$, $F_{N}^{neg}$, $F_{S}^{neg}$. All these fluxes
follow the 11-year cycle of solar activity. Some difference between fluxes can
be seen at periods of high solar activity. For each of solar cycles these four
fluxes depending on its polarity can be interpreted as the magnetic flux of
the leading or the following sunspots of one of the solar hemispheres. The
signs of the leading/following sunspots remain constant during solar cycle
from one minimum to the next one when sunspots of a new cycle appear with
polarities opposite to the previous ones.

Figure 6.: Fluxes of strong magnetic fields ($B>50$ G) for the sunspot zone
($5^{\circ}-40^{\circ}$): positive ($F_{N}^{pos}$, $F_{S}^{pos}$) and negative
($F_{N}^{neg}$, $F_{S}^{neg}$) fluxes (absolute values) for the northern and
the southern hemispheres.fourflux Figure 7.: (a) Flux imbalance between
positive and negative fluxes for the northern hemisphere $\Delta
F_{N}=F_{N}^{pos}-|F_{N}^{neg}|$. (b) Flux imbalance between positive and
negative fluxes for the southern hemisphere $\Delta
F_{S}=F_{S}^{pos}-|F_{S}^{neg}|$. Blue line denotes sunspot number. The sign
of the leading sunspots is shown for each hemisphere.imbhemi

Flux imbalance between positive and negative fluxes for the northern
hemisphere $\Delta F_{N}=F_{N}^{pos}-|F_{N}^{neg}|$ (Figure imbhemia) varies
with the 22-year cycle and reaches extrema during maxima of solar activity.
The flux imbalance passes through zero around the minima of solar activity.
Thus, from one minimum to another, the sign of the difference between positive
and negative fluxes ($\Delta F_{N}$) does not change. In Figure imbhemia the
sign of leading sunspots in bipolar sunspot groups is shown. The sign of the
flux imbalance coincides with the sign of leading sunspots in the northern
hemisphere. Similar results are obtained for the southern hemisphere: the sign
of the imbalance $\Delta F_{S}=F_{S}^{pos}-|F_{S}^{neg}|$ always coincides
with the sign of leading sunspots (Figure imbhemib). The imbalances of
positive and negative fluxes in each of the solar hemispheres show a 22-year
recurrence that is directly connected with the Hale cycle. Evidently, for each
of solar hemispheres magnetic flux of the leading sunspots exceeds that of the
following sunspots.

Figure 8.: Flux imbalance of leading sunspots of the northern and the southern
hemispheres $\Delta F_{lead}$ (red line) and flux imbalance of following
sunspots $\Delta F_{foll}$ (blue line). Times of polar field reversal are
marked by green circles. The signs of the polar magnetic fields in the
northern and southern hemispheres are shown, respectively, in the upper part
and in the lower part of the figure.leadfol

There arises a question: if the fluxes of leading sunspots of two hemispheres
are compared, which of these fluxes will be higher? It is similarly possible
to consider the flux imbalance of following sunspots of two hemispheres. For
example, during Solar Cycle 21, leading sunspots of the northern hemisphere
had positive polarity (corresponding flux $F_{N}^{pos}$), while the leading
sunspots of the southern hemisphere had negative polarity (corresponding flux
$F_{S}^{neg}$). Then the imbalance of leading sunspots can be defined as
$\Delta F_{lead}=F_{N}^{pos}-|F_{S}^{neg}|$. During the next solar cycle the
leading sunspots in both hemispheres change their polarity. In this case the
imbalance of leading sunspots will be $\Delta
F_{lead}=F_{S}^{pos}-|F_{N}^{neg}|$. Flux imbalance of leading sunspots of the
northern and the southern hemispheres $\Delta F_{lead}$ is presented in Figure
leadfol (red line). Flux imbalance of following sunspots $\Delta F_{foll}$
(blue line) is in antiphase with the imbalance of the leading sunspot fluxes
with correlation coefficient $R=-0.98$. At the time of polar field reversal
both parameters change their sign (marked by green circles). The sign of the
imbalance between fluxes of leading sunspots of two hemispheres changes with a
22-year magnetic cycle in the same way as the sign of the polar magnetic field
in the northern hemisphere (shown in the upper part of Figure leadfol). The
imbalance between fluxes of following sunspots repeats the sign of the polar
magnetic field in the southern hemisphere (shown in the lower part of Figure
leadfol).

It is possible to show that the imbalance of fluxes of leading sunspots and
the imbalance of fluxes of following sunspots define the sign of north-south
asymmetry of the magnetic field in the activity zone. North-south asymmetry
can be defined as:

$\ilabel{Asym}\Delta_{NS}=(F_{N}^{pos}+|F_{N}^{neg}|)-(F_{S}^{pos}+|F_{S}^{neg}|)$
(3)

The sign of the imbalance of leading sunspots is opposite to the sign of the
imbalance of following sunspots, hence, dominating leading sunspots and
dominating following sunspots will have different signs. As fields of leading
and following sunspots in the same hemisphere have opposite signs, it follows
from this that both fluxes of leading sunspots of a hemisphere, and fluxes of
following sunspots of the same hemisphere will dominate simultaneously over
corresponding fluxes of the other hemisphere (N-S asymmetry). The similar
conclusion was made by Petrie (2012) for magnetic fields during decrease of
Cycle 23 and ascent of Cycle 24.

Considering two phases of a 11-year cycle, from a minimum before reversal and
from reversal to a minimum, we will show that change of domination of
hemispheres (change of N-S asymmetry sign) occurs during polar field reversal
and during minimum of SA. Figure leadfol shows that the sign of the imbalance
of leading sunspots always coincides with the sign of the polar field in the
northern hemisphere. It follows that those sunspots which have the same sign,
as the polar field of the northern hemisphere will dominate. Thus, from the
minimum to the reversal the flux of leading sunspots of the northern
hemisphere which has the sign of the polar field in this hemisphere, will
always exceed the flux of leading sunspots of the southern hemisphere (the
northern hemisphere dominates). After the reversal the sign of the polar field
in the northern hemisphere changes and coincides with the sign of leading
sunspots in the southern hemisphere (the southern hemisphere dominates from
the reversal to the minimum).

During SA minimum the domination returns to the northern hemisphere because
leading and following sunspots in each of hemispheres change their signs,
whereas the sign of the polar field remains unchanged. As a result north-south
asymmetry changes its sign both during polar field reversal and during minimum
of SA, so that the northern hemisphere will always dominate from the minimum
to the reversal, but the southern hemisphere will dominate from the reversal
to the minimum (See Figure 2 in Vernova, Tyasto, and Baranov (2014)).

Figure 9.: (a) Total imbalance $\Delta
F_{low}=|F_{N}^{pos}+F_{S}^{pos}|-|F_{N}^{neg}+F_{S}^{neg}|$ of two sunspot
zones (latitudes from $+5^{\circ}$ to $+40^{\circ}$ and from $-5^{\circ}$ to
$-40^{\circ}$). Blue line – sunspot number. The sign of the polar field in the
northern hemisphere is shown in the upper part of the figure. (b) Reversed
quadrupole moment $-g_{20}$. (c) Sun’s mean magnetic field (SMMF) $\Delta B$.
(d) The difference between numbers of days with positive and negative polarity
of the interplanetary magnetic field (according to L. Svalgaard).imblow

In the same way as the imbalance of magnetic field for two polar caps $\Delta
F_{high}$ (Formula (Imbal1)), one can define the total imbalance of positive
and negative fields for both sunspot zones $\Delta F_{low}$ (latitudes from
$+5^{\circ}$ to $+40^{\circ}$ and from $-5^{\circ}$ to $-40^{\circ}$):

$\ilabel{Imbal2}\Delta
F_{low}=|F_{N}^{pos}+F_{S}^{pos}|-|F_{N}^{neg}+F_{S}^{neg}|$ (4)

The total imbalance of two sunspot zones is presented in Figure imblowa for
magnetic fields $B>50$ G (red line). The total imbalance changes in time
similarly to the imbalance of leading sunspots, but the imbalance of leading
sunspots is about five times higher. Change of the imbalance sign (marked in
Figure imblow by green circles) occurs during the period close to the polar
field reversal. The sign of the polar field in the northern hemisphere is
shown in the upper part of Figure imblowa. The imbalance sign does not change
within 11 years, thus the full period of the sign change makes 22 years. The
sign of the imbalance coincides with the sign of the polar field in the
northern hemisphere. Time intervals of coincidence are marked by shading for
two polarities: positive (blue shading) or negative (green shading). As can be
seen from comparison of the flux imbalance with SA cycle (Figure imblowa, the
dark blue line) the imbalance changes with a strict regularity: each solar
cycle contains two parts – one with a positive imbalance, another with
negative. In Figure imblowb the change of the quadrupole moment taken with a
reversed sign ($-g_{20}$) is shown (the shading as in Figure imblowa). It is
evident that the sign of the magnetic flux imbalance and the sign of the
quadrupole moment ($-g_{20}$) change with the same periodicity.

To clarify the physical meaning of the imbalance one should compare it with
the north-south asymmetry $\Delta_{NS}$. In Figure leadfol the change of the
imbalances $\Delta F_{lead}$ and $\Delta F_{foll}$ is shown. The north-south
asymmetry $\Delta_{NS}$ defined by Formula (Asym) can be rewritten as the
difference between $\Delta F_{lead}$ and $\Delta F_{foll}$, whereas the total
imbalance for sunspot zone (see Figure imblowa) is given by the sum of $\Delta
F_{lead}$ and $\Delta F_{foll}$. Since $\Delta F_{lead}$ and $\Delta F_{foll}$
have opposite signs (see Figure leadfol), the value of imbalance is
significantly lower than that of the north-south asymmetry. Thus, the total
imbalance provides a more subtle characteristic of magnetic fields. All the
more surprising is that this characteristic has such regular structure and is
so closely related with 22-year and 11-year cycles.

It is of interest to compare the flux imbalance with the Sun’s mean magnetic
field (SMMF). The SMMF represents an imbalance of the magnetic flux ($\Delta
B$) integrated over the entire visible solar disk (Garcia et al., 1999). It is
small, less than $\pm 1$ G, reaching a typical value of about 0.15 G during
solar minimum. SMMF is important in many problems of solar physics. Its
influence on the interplanetary magnetic field is strong, which can be deduced
from a high correlation between the two. Daily values of the mean magnetic
field of the Sun (Sun-as-a-Star) in microTesla (WSO data:
http://wso.stanford.edu/meanfld/MF˙timeseries.txt) were averaged over period
of one month. For comparison with the imbalance of the sunspot-zone fluxes
(Figure imblowa) the monthly values of $\Delta B$ were smoothed over 20 points
(Figure imblowc). The values of SMMF differ from the threshold of $B>50$ G
chosen by us; however, the basic features of the imbalance change can be
observed in the mean field of the Sun. The sign of the SMMF also displays the
22-year periodicity and changes around the time of the polar field reversal.

The question arises, whether the imbalance of positive and negative fields is
a purely local effect or it is reflected in the structure of the heliosphere?
To clarify this point, data on the polarity of the IMF (data of L. Svalgaard,
see: http://www.leif.org/research/spolar.txt) were used. The difference
between numbers of days with positive and negative polarity of the IMF was
calculated (Figure imblowd). For convenience of comparison with the flux
imbalance (Figure imblowa) coincidence of polarities is marked by shading.
Good agreement between the imbalance and the IMF polarity can be seen, except
for years 2005 – 2008 of the prolonged minimum of solar activity. The sign of
the difference between positive and negative days of the IMF changes with the
22-year period and coincides mainly with the sign of the polar field in the
northern hemisphere. Thus, the imbalance of the magnetic flux of the Sun can
be the cause of the asymmetry observed in the IMF.

Results obtained for fields of the sunspot zone (from $5^{\circ}$ to
$40^{\circ}$ in each hemisphere), $B>50$ G testify:

  1. 1)

The imbalance in a separate hemisphere changes with the 22-year cycle and the
imbalance sign changes near the solar activity minimum. The imbalance sign
coincides with the sign of leading sunspots.

  2. 2)

The sign of the total imbalance for two hemispheres also changes with the
22-year cycle, but the sign change occurs near the reversal of the polar
magnetic field. The imbalance sign always coincides with the sign of the polar
field in the northern hemisphere. Total imbalance of positive and negative
fluxes shows similar evolution as the quadrupole component $(-g_{20})$ of the
photospheric magnetic field.

##  5 Building blocks of a 22-year cycle

Build Different manifestations of the magnetic-flux imbalance considered
above, have one common feature: they appear regularly during the period of
four solar cycles. We will show that this regularity can be expressed by
several simple formulas. Solar dynamo models establish the connection of the
11-year cycle of solar activity to the 22-year cycle of magnetic polarity. The
scheme of the polarity change of local and global fields is presented in
Figure scheme. Two solar cycles are shown in Figure schemea: the even cycle
and the following odd cycle. Such choice is connected with the results of Ohl
who found a good correlation between SA of two consecutive cycles: an even and
the following odd cycle. On the other hand correlation between an odd cycle
and the following even one is very weak. According to the Ohl scheme, an even
cycle with the succeeding odd cycle form a pair, a single whole (Nagovitsyn,
Nagovitsyna, and Makarova, 2009; Ponyavin, 2009). We use number $n$ to denote
parity of the cycle. Thus, parity of the solar cycle $n=1$ corresponds to an
odd solar cycle, $n=2$ to an even one.

In the course of a 22-year magnetic cycle there are moments when the relation
between the polarities of the global and local magnetic fields changes. These
moments are related either to the change of the Sun’s global magnetic-field
polarity during high solar activity (polar magnetic-field reversal, marked in
Figure schemea by green circles), or to the alternation of the polarities of
leading and following sunspots at the solar activity minimum (minima of SA are
marked by red circles). In the minimum of SA there is a change of the leading-
sunspot sign in each of hemispheres. During the transition from an even to an
odd cycle leading sunspots of a new cycle in the northern hemisphere will have
the positive sign which coincides with the sign of the polar field in this
hemisphere. During the transition from an odd cycle to an even the sign of
leading sunspots in the northern hemisphere also will coincide with the sign
of the polar field, however it will be a negative sign.

Another situation will take place at the reversal of the polar magnetic field
(around maximum of SA). After reversal and up to SA minimum the sign of the
polar field in the northern hemisphere will coincide with the sign of leading
sunspots in the southern hemisphere. The following is remarkable: in both
cases, i.e., throughout all magnetic cycle, leading sunspots of that
hemisphere whose sign coincides with the polar-field sign of the northern
hemisphere (Figure schemeb) will dominate in the magnetic flux. Thus, a
22-year cycle is composed of four intervals (we will call them “building
blocks”): during each of these intervals the relation between polarities of
the polar magnetic field and of the leading sunspots does not change. These
four intervals are from the solar-activity minimum to the magnetic field
reversal of the Sun and from the reversal to the next minimum in an even
11-year solar cycle, and analogously for the next odd cycle. The relations
between the polarities of the global and local magnetic fields will repeat
themselves in the next 22-year cycle. Connection of global and local magnetic
field polarities in a 22-year magnetic cycle is illustrated by Figure schemeb.
Each solar cycle can be divided in two parts according to the phase of the
cycle: $k=1$ corresponds to the interval of the 11-year cycle from the minimum
up to the reversal; $k=2$ to the interval from the reversal up to the minimum.
The sign of the polar magnetic field in the northern hemisphere is shown at
the top of the Figure schemea and in the scheme of Figure schemeb for each of
the four quarters of the 22-year magnetic cycle.

Figure 10.: Polar magnetic field sign and leading sunspot polarity for each of
the building blocks in Cycles 22 and 23.scheme

The leading sunspots in each of hemispheres are displayed as circles with
corresponding polarity. Large circles show dominating leading sunspots. It can
be seen that in every building block the sign of the dominating leading
sunspots coincides with the sign of the polar magnetic field in the northern
hemisphere. The scheme of Figure schemeb shows that at each boundary of two
characteristic intervals domination of leading sunspots passes from one
hemisphere to the other one. In the first quarter of the 22-cycle the northern
hemisphere dominates, in the next quarter the southern hemisphere dominates
and so on. As is shown in Figure leadfol, fluxes of leading and following
sunspots in the same hemisphere will dominate simultaneously over
corresponding fluxes of leading and following sunspots of the other
hemisphere. Thus, from the minimum to the reversal the northern hemisphere
always dominates, whereas from the reversal to the minimum – the southern
hemisphere dominates (N-S asymmetry).

Following simple formulas describe the magnetic field polarities for different
phases ($k=1,2$) of the odd ($n=1$) and even ($n=2$) solar cycles:

1\. The sign of the polar magnetic field of the northern hemisphere $F_{N}$;
the sign of leading-sunspot flux imbalance $\Delta F_{lead}$; the sign of
positive and negative flux imbalance for sunspot zones $\Delta F_{low}$
(latitudes from $+5^{\circ}$ to $+40^{\circ}$ and from $-5^{\circ}$ to
$-40^{\circ}$) are determined by two factors: the parity of the solar cycle
$n$, and the phase of the 11-year cycle $k$:

$\ilabel{Form1}\left.\begin{array}[]{c}{\rm sign}\,F_{N}\\\ {\rm sign}\,\Delta
F_{lead}\\\ {\rm sign}\,\Delta F_{low}\end{array}\right\\}=(-1)^{n+k}.$ (5)

2\. The sign of the polar magnetic field of the southern hemisphere $F_{S}$;
the sign of following-sunspot flux imbalance $\Delta F_{foll}$; the sign of
positive and negative flux imbalance for high latitudes from $+40^{\circ}$ to
$+90^{\circ}$ and from $-40^{\circ}$ to $-90^{\circ}$ $\Delta F_{high}$ are
determined by the same two factors $n$, $k$:

$\ilabel{Form2}\left.\begin{array}[]{c}{\rm sign}\,F_{S}\\\ {\rm sign}\,\Delta
F_{foll}\\\ {\rm sign}\,\Delta F_{high}\end{array}\right\\}=(-1)^{n+k+1}.$ (6)

3\. The sign of leading sunspots in the northern hemisphere; the sign of
imbalance between positive and negative fluxes in the northern hemisphere
$\Delta F_{N}$ are determined by the parity of the solar cycle n:

$\ilabel{Form3}\left.\begin{array}[]{c}{\rm sign}\,(lead.\,sunspot)\\\ {\rm
sign}\,\Delta F_{N}\end{array}\right\\}=(-1)^{n+1}.$ (7)

4\. The sign of leading sunspots in the southern hemisphere; the sign of
imbalance between positive and negative fluxes in the southern hemisphere
$\Delta F_{S}$ are determined by the parity of the solar cycle n:

$\ilabel{Form4}\left.\begin{array}[]{c}{\rm sign}\,(lead.\,sunspot)\\\ {\rm
sign}\,\Delta F_{S}\end{array}\right\\}=(-1)^{n}.$ (8)

5\. The sign of the north-south asymmetry depends on the phase of the 11-year
cycle k (before or after the reversal):

$\ilabel{Form5}{\rm sign}\,\Delta_{NS}=(-1)^{k+1}$ (9)

6\. Previously we considered the longitudinal distribution of sunspots
(Vernova et al., 2004) and of photospheric magnetic fields (Vernova, Tyasto,
and Baranov, 2007). The longitudinal distribution for the ascending phase and
the maximum sharply differ from the longitudinal distribution for the
descending phase and the minimum of the solar cycle. Active longitudes change
by $180^{\circ}$ when SA evolves from the ascending phase to the descending
one. The maximum of the longitudinal distribution is attained at $180^{\circ}$
for the ascending phase and the maximum of the solar cycle when the polarities
of the leading sunspot and of the global field coincide in each hemisphere,
and at $0^{\circ}/360^{\circ}$ for the descending phase and the minimum when
these polarities are opposite. The active longitude is determined by the phase
of solar cycle $k$:

$\ilabel{Form6}Active\,\,Longitude=\pi k$ (10)

The above formulas evidence that four building blocks of the 22-year magnetic
cycle manifest itself in periodic changes of magnetic field polarities. We
believe that representation of the 22-year magnetic cycle as consisting of
four characteristic intervals (building blocks) can be useful for studying the
processes underlying the observed changes of magnetic field polarities. This
representation does not merely establish the fact of alternation of global and
local field polarities, but also states the change of domination of certain
magnetic field group that is reflected in changes of solar activity and solar-
terrestrial relations. In fact, many observations show that behavior of the
solar activity manifestations changes significantly both for different phases
of the solar cycle and for the odd and even cycles.

Some examples of this connection are presented below. Difference between
phases of ascent and descent leads to occurrence of a hysteresis for many
indexes of SA (Bachmann et al., 2004). The hysteresis effect that shows up as
a nonunique relationship among the emissions from the photosphere,
chromosphere, and corona during the rising and declining phases of solar and
stellar activity was analyzed by Bruevich and Yakunina (2016). Solar-
terrestrial relations also display the dependence on the phase of the cycle.
It was shown that ionospheric indices fo2 and Ap weakly depend on the level of
solar activity, but the effect of hysteresis is clearly seen (Bruevich et al.,
2016). The number of M-class and X-class flares tends to follow the sunspot
number, but there is a tendency to have more flares on the declining phase of
a sunspot cycle (Hathaway, 2015).

The 22-year periodicity of SA displays itself as the difference between two
successive maxima of the 11-year cycle, which follow the Gnevyshev–Ohl rule:
the maximum of an even cycle is lower than the maximum of the following odd
cycle. Studying the Fraunhofer lines in the solar spectrum variations
Livingston et al. (2007) found that solar minimum around 1985 was clearly seen
in the high photosphere lines, but the following minimum in 1996 was missing,
perhaps indicating a role for the 22-year Hale cycle.

Cosmic-ray modulation shows strong dependence both on the 11-year cycle of SA,
and on the 22-year magnetic cycle. When the solar magnetic field in the
northern hemisphere is positive, the galactic cosmic rays drift downward from
the solar poles toward the Earth and then out along the warped current sheet
to the boundary of the heliosphere (Lockwood, Webber, and Debrunner, 2001).
When it is negative, the drift pattern is the opposite. The difference in the
appearance of the two halves of the 22-year solar magnetic cycle, peaked and
flat-topped, supports the role of drifts near the cosmic ray intensity maxima.
The intensity of cosmic rays at the solar activity minima also depends on the
solar magnetic field polarity. Singh, Badruddin, and Ananth (2005) observed
differences between time lags of the solar activity and cosmic ray intensity
in odd and even cycles as well as differences in the shape, size etc. of
hysteresis loops during odd and even cycles. The time lag between cosmic ray
intensity and the solar index is different in odd ($10-14$ months) and even
($1-4$ months) cycles. Differences in time lag between periods of $A<0$
polarity ($9-10$ months) and $A>0$ polarity ($3-5$ months) was also found
($A>0$ corresponds to the positive polarity of the polar field in the north
hemisphere). These examples show the influence of different phases of the
22-year magnetic cycle on solar activity and heliosphere.

##  6 Conclusions

Concl

In this paper we studied polarity imbalance of photospheric magnetic field for
high latitudes (from $40^{\circ}$ to $90^{\circ}$ in each hemisphere) and for
sunspot zone (from $5^{\circ}$ to $40^{\circ}$ in each hemisphere) during four
solar cycles (Solar Cycles 21–24). We used the threshold $B<50$ G for high
latitudes and $B>50$ G for the sunspot zone. For these two latitude zones we
considered imbalance of positive and negative magnetic fields for each
hemisphere and for both hemispheres together. For sunspot zone we calculated
polarity imbalance for leading sunspots of two hemispheres as well as for
following sunspots. All these polarity imbalances display regular structure
and change with the 22-year period.

The threshold $40^{\circ}$ separates two latitude intervals whose imbalances
($\Delta F_{high}$ and $\Delta F_{low}$) develop in antiphase. The sign of the
total imbalance for high latitudes (from $40^{\circ}$ to $90^{\circ}$ and from
$-90^{\circ}$ to $-40^{\circ}$) coincides with the sign of the polar field in
the southern hemisphere and also with the imbalance of following sunspots
according to Formula (Form2). On the other hand, the sign of the imbalance for
sunspot zone (from $5^{\circ}$ to $40^{\circ}$ and from $-40^{\circ}$ to
$-5^{\circ}$) coincides with the sign of the polar field in the northern
hemisphere as well as with the imbalance of leading sunspots (see Formula
(Form1)). We obtained a good agreement of the imbalance of high-latitude
fields with the quadrupole moment $g_{20}$, while for low-latitude fields
there is an agreement with $-g_{20}$, the quadrupole moment taken with the
opposite sign.

It was noted in Virtanen and Mursula (2016) and Wang (2014) that the
quadrupole component plays in important role in the shift of the heliospheric
current sheet southward. Our paper reveals a very close relation of the
magnetic-field imbalance with the quadrupole moment. It looks reasonable to
suggest that just the imbalance of sunspot-zone fields, which always shows
domination of the fields with the same sign as the polar field in the northern
hemisphere, results in the southward shift of the heliospheric current sheet.

Imbalances for high latitudes and for sunspot zone have opposite signs and, as
a result, the imbalance for all latitudes has lower values. However, while
magnetic flux for sunspot zone is approximately three times greater than
magnetic flux for high latitudes (Figure complat), the total imbalance for
sunspot zone is about two times less than the total imbalance for high
latitudes (see Figure imbhigha and Figure imblowa). This difference between
the imbalances for sunspot zone and for high latitudes (which is especially
clearly seen in Cycle 22) shows that total imbalance for the whole range of
latitudes cannot be zero.

In the course of a 22-year magnetic cycle there are four intervals (from the
solar-activity minimum to the magnetic-field reversal of the Sun and from the
reversal to the next minimum in an even 11-year solar cycle, and analogously
for the next odd cycle) when the relation between polarities of the polar
magnetic field and of the leading sunspots does not change. We believe that
this representation of the 22-year magnetic cycle as consisting of the four
characteristic intervals (“building blocks”) can be useful in the study of the
processes on the Sun and of solar-terrestrial relations.

The obtained results show that in addition to well-known 22-year periodic
patterns in the change of global and local fields there also exists a 22-year
periodicity in the change of imbalance of both high-latitude and low-latitude
fields.

#### Acknowledgments

NSO/Kitt Peak data used here are produced cooperatively by NSF/NOAO,
NASA/GSFC, and NOAA/SEL. This work utilizes SOLIS data obtained by the NSO
Integrated Synoptic Program (NISP), managed by the National Solar Observatory,
which is operated by the Association of Universities for Research in Astronomy
(AURA), Inc. under a cooperative agreement with the National Science
Foundation. Wilcox Solar Observatory data used in this study was obtained via
the web site http://wso.stanford.edu courtesy of J.T. Hoeksema. We are
thankful to Prof. L. Svalgard for providing data on the IMF sector structure
polarity.

## References

  * Bachmann et al. (2004) Bachmann, K.T., Maymani, H., Nautiyal, K., te Velde, V.: 2004, Adv. Space Res. 34, 274. ADS, DOI.
  * Ballester, Oliver, and Carbonell (2005) Ballester, J.L., Oliver, R., Carbonell, M.: 2005, Astron. Astrophys. 431, L5-L8. ADS, DOI.
  * Bravo and González-Esparza (2000) Bravo, S., González-Esparza, J.A.: 2000, Geophys. Res. Lett. 27, 847. ADS, DOI.
  * Bruevich and Yakunina (2016) Bruevich, E.A., Yakunina, G.V.: 2016, Astrophysics 59, 369. ADS, DOI.
  * Bruevich et al. (2016) Bruevich, E.A., Kazachevskaya, T.V., Katyushina, V.V., Nusinov, A.A., Yakunina, G.V.: 2016, Geomagnetism Aeronomy 56, 1075\. ADS, DOI.
  * Carbonell et al. (2007) Carbonell, M., Terradas, J., Oliver, R., Ballester, J.L.: 2007, Astron. Astrophys. 476, 951. ADS, DOI.
  * Charbonneau (2010) Charbonneau, P.: 2010, Living Rev. Solar Phys. 7, 3. ADS, DOI.
  * Choudhary, Venkatakrishnan, and Gosain (2002) Choudhary, D.P., Venkatakrishnan, P., Gosain, S.: 2002, Astrophys. J. 573, 851. ADS, DOI.
  * Deng et al. (2016) Deng, L.H., Xiang, Y.Y., Qu, Z.N., An, J.M.: 2016, Astron. J. 151, 70. ADS, DOI.
  * Durrant and Wilson (2003) Durrant, C.J., Wilson, P.R.: 2003, Solar Phys. 214, 23. ADS, DOI.
  * Erdös and Balogh (2010) Erdös, G., Balogh,A.: 2010, J. Geophys. Res. 115, A01105. ADS, DOI.
  * Fan (2009) Fan, Y.: 2009, Living Rev. Solar Phys. 6, 4. ADS, DOI.
  * Garcia et al. (1999) Garcia, R.A., Boumier, P., Charra, J., Foglizzo, T., Gabriel, A.H., Grec et al.: 1999, Astron. Astrophys. 346, 626. ADS,
  * Hathaway (2015) Hathaway, D.H.: 2015, Living Rev. Solar Phys., 12, 4. ADS, DOI.
  * Hoeksema and Scherrer (1986) Hoeksema, J.T., Scherrer, P.H.: 1986, Solar Magnetic Field – 1976 through 1985. Report UAG-94, WDCA, Boulder, USA. ADS.
  * Jin and Wang (2014) Jin, C.L., Wang, J.X.: 2014, J. Geophys. Res.: Sp. Phys. 119, 11. ADS, DOI.
  * Kotov (2009) Kotov, V.A.: 2009, Bull. Crimean Astrophys. Observatory 105, 45. ADS, DOI.
  * Livingston et al. (2007) Livingston, W.; Wallace, L.; White, O. R.; Giampapa, M.S.: 2007, Astrophys. J. 657, 1137. ADS, DOI.
  * Lockwood, Webber, and Debrunner (2001) Lockwood, J.A., Webber, W.R., Debrunner, H.: 2001, J. Geophys. Res. 106, Iss. A6, 10635. ADS, DOI.
  * Mursula and Hiltula (2003) Mursula, K., Hiltula, T.: 2003, Geophys. Res. Lett. 30(22), SSC 2-1. ADS, DOI.
  * Nagovitsyn, Nagovitsyna, and Makarova (2009) Nagovitsyn, Yu.A., Nagovitsyna, E.Yu., Makarova, V.V.: 2009, Astronomy Letters 35, 564. ADS, DOI.
  * (22) Obridko, V.N., Shelting, B.D.: 2009, Astronomy Letters 35, Iss. 4, 247. ADS, DOI.
  * Petrie (2012) Petrie, G.J.D.: 2012, Solar Phys. 289, 577. ADS, DOI.
  * Petrie and Ettinger (2015) Petrie, G., Ettinger, S.: 2015, Space Sci. Rev. 210, 77. ADS, DOI.
  * Ponyavin (2009) Ponyavin, D.I.: 2009, Adv. Space Res. 29, 421. ADS, DOI.
  * Singh, Badruddin, and Ananth (2005) Singh, M.; Badruddin; Ananth, A.G.: 2005, Proceedings of the 29th International Cosmic Ray Conference. Pune, India. Volume 2, 139. ADS.
  * Svalgaard and Kamide (2013) Svalgaard, L., Kamide, Y.: 2013, Astrophys. J. 763, 23. ADS, DOI.
  * Swinson, Shea, and Humble (1986) Swinson, D.B., Shea, M.A., Humble, J.E.: 1986, J. Geophys. Res. 91, 2943. ADS, DOI.
  * Tian, Liu, and Wang (2003) Tian, L., Liu, Y., Wang, H. : 2003, Solar Phys. 215, Iss. 2, 281. ADS, DOI.
  * van Driel-Gesztelyi and Petrovay (1990) van Driel-Gesztelyi, L., Petrovay, K.: 1990, Solar Phys. 126, 285. ADS, DOI.
  * Vernova et al. (2004) Vernova, E.S., Mursula, K., Tyasto, M.I., Baranov, D.G.: 2004, Solar Phys. 221, 151. ADS, DOI.
  * Vernova, Tyasto, and Baranov (2007) Vernova, E.S., Tyasto, M.I., Baranov, D.G.: 2007, Solar Phys. 245, 177. ADS, DOI.
  * Vernova, Tyasto, and Baranov (2014) Vernova, E.S., Tyasto, M.I., Baranov, D.G.: 2014, Solar Phys. 289, 2845. ADS, DOI.
  * Vernova, Tyasto, and Baranov (2016) Vernova, E.S., Tyasto, M.I., Baranov, D.G.: 2016, Solar Phys. 291, 741. ADS, DOI.
  * Vernova, Tyasto, and Baranov (2017) Vernova, E.S., Tyasto, M.I., Baranov, D.G.: 2017, arXiv:1704.02586.
  * Virtanen and Mursula (2016) Virtanen, I., Mursula, K.: 2016, Astron. Astrophys. 591, id.A78. ADS, DOI.
  * Wang (2004) Wang, Y.-M.: 2004, Solar Phys. 224, 21. ADS, DOI.
  * Wang (2014) Wang, Y.-M.: 2014, Space Sci Rev. 186, 387. ADS, DOI.
  * Wang (2017) Wang, Y.-M.: 2017, Space Sci. Rev. 210, 351. ADS, DOI.
  * Wang and Robbrecht (2011) Wang, Y.M., Robbrecht, E.: 2011, Astrophys. J. 736, 136. ADS, DOI.
  * Wang, Robbrecht, and Sheeley (2009) Wang, Y.-M., Robbrecht, E., Sheeley, N.R., Jr.: 2009, Astrophys. J. 707, 1372. ADS, DOI.
  * Wilcox (1972) Wilcox, J.M.: 1972, Comments on Astrophysics and Space Physics 4, 141. ADS.


# Generating gender-ambiguous voices for
privacy-preserving speech recognition

###### Abstract

Our voice encodes a uniquely identifiable pattern which can be used to infer
private attributes, such as gender or identity, that an individual might wish
not to reveal when using a speech recognition service. To prevent attribute
inference attacks alongside speech recognition tasks, we present a generative
adversarial network, GenGAN, that synthesises voices that conceal the gender
or identity of a speaker. The proposed network includes a generator with a
U-Net architecture that learns to fool a discriminator. We condition the
generator only on gender information and use an adversarial loss between
signal distortion and privacy preservation. We show that GenGAN improves the
trade-off between privacy and utility compared to privacy-preserving
representation learning methods that consider gender information as a
sensitive attribute to protect.

Index Terms: audio privacy, speech recognition, speaker verification, gender
recognition, generative adversarial networks

##  1 Introduction

The human voice is shaped by the physical characteristics of the speaker and
the spoken language. The voiceprint, which uniquely defines each individual
[1], contains attributes of the speaker that can be inferred by voice-based
services [2, 3]. To mitigate this risk and protect the identity of a speaker
in Automatic Speech Recognition (ASR) tasks, adversarial training on privacy
and utility objectives can be used [4, 5]. Adversarial representation learning
determines the network weights that minimise the likelihood of finding the
correct labels corresponding to identity (or gender) and removes information
related to these attributes from the encoded representation [4]. Training with
a speaker-adversarial branch acting as a gradient reversal layer has been used
to remove speaker identity from the learned representation [6]. Adversarial
feature extraction is used to improve the privacy-utility trade-off, when
gender classification is considered as the utility objective and speaker
identification accuracy as the privacy objective [5].

Gender information is typically used to condition models preserving the
identity of a speaker. However, only a handful of methods explicitly consider
gender as a sensitive attribute to protect [7, 8, 9, 10, 11]. A hybrid model
combining Variational Autoencoders and Generative Adversarial Networks (GANs)
can be used to protect gender information through voice conversion with a
disentanglement approach targeted for the speech recognition task [7]. Two
encoders are trained to independently encode content and speaker identity
information that is then used to hide (or mask) gender information. Privacy
methods that operate at feature-level have been used to disentangle gender
information from x-vectors [12] with adversarial learning and an encoder-
decoder based architecture [8]. Because this adversarial method removes the
unwanted information at the level of the feature representation instead of the
speech waveform, it is not useful for tasks such as speech recognition. Vector
Quantized Variational Autoencoders (VQ-VAE) [13] are used to create
disentangled representations of linguistic content and speaker identity [9],
and both identity and gender [11]. Linguistic content is quantised into a
discrete latent space using a learned codebook. The decoding stage
reconstructs the speech waveform by combining learned embedding spaces
encoding the selected attributes. These methods have limited reconstruction
capabilities and induce distortion at the decoding stage, when quantised
content information is reconstructed as speech. PCMelGAN [10] synthesises
speech using a generative adversarial approach that considers gender as an
attribute to protect and reconstructs mel-scale spectrograms using the MelGAN
vocoder [14] to maintain the utility (accuracy) in a digit recognition task.
However, the dataset used by this work is composed of utterances of spoken
digits, which is limited in vocabulary and size. Finally, PCMelGAN is based on
PCGAN [15], which uses a filtering module to replace the sensitive information
in speech with generated synthetic information. However, we will show that
results can be improved _without_ this additional process.

We aim to produce gender-ambiguous [16] voices. To this end we first determine
the extent to which modifying gender information impacts the identity
information of a speaker from a privacy-preservation perspective. Next, we
produce a light-weight generative method that protects against the inference
of gender and identity of synthesised speech signals. We achieve this without
considering any identity information and maintain the utility of speech
without explicitly optimising for the ASR task.

We propose GenGAN, a generative privacy transformation method that conceals
gender and much of the identity information of speakers by synthesising voices
with gender-ambiguous characteristics. To the best of our knowledge GenGAN is
the first attempt to create gender-ambiguous voices in a privacy-preserving
setting. The generator samples from a designed distribution that models a
gender-ambiguous voice and learns to smooth spectral differences between
genders. GenGAN is only conditioned on gender information, without considering
any information on speaker identity during training. Furthermore, content
information is preserved in the transformed signal independently of the ASR
used for the speech transcription task.

##  2 Proposed approach

###  2.1 Attack scenario

Let the identity and gender of a speaker be the personal information to
protect in an ASR scenario. An attacker attempts to infer, from the privacy-
transformed audio, the gender of the speaker by classification and their
identity by verification.

We assume that the attacker has access to the data (anonymised utterances)
produced by the privacy-preserving transformation and shared with the speech
recognition service (see Fig. 1). We also assume the attacker has no knowledge
of the applied privacy-transformation.

Figure 1: Evaluation of our privacy transformation method on utility  and
privacy  objectives. The transformed audio is used to perform privacy-
preserving speech recognition. The privacy objective assesses the ability of
an attacker to recognise the gender and verify the identity of a speaker given
the transformed gender-ambiguous speech signal. KEY – $\mathcal{X}$: original
waveform, $\mathcal{X^{\prime}}$: transformed waveform, $\mathcal{Y}$: ground-
truth gender, $\mathcal{Y^{\prime}}$: transformed gender, ASR: automatic
speech recognition.

###  2.2 GenGAN

By assuming an attack scenario where voice signals are accessed prior to the
downstream task (speech recognition), we are interested in reproducing voice
and hence operate on the input waveforms, which are converted into 80-band
mel-spectrograms. We consider a Generator, $\mathcal{G}$, and a Discriminator,
$\mathcal{D}$, to be trained in an adversarial game between privacy and
utility objectives. $\mathcal{G}$ has a U-Net architecture [17] with a
contracting and an expanding path, while $\mathcal{D}$ consists of a simple
AlexNet [18] architecture. $\mathcal{G}$ produces audio data with the aim to
fool $\mathcal{D}$, whose task is to discriminate between original and
synthetic audio data. We maximise utility by minimising the distortion in the
generated audio and minimise the risk of privacy inference by training the
model to learn gender-ambiguous information. $\mathcal{D}$ learns to
discriminate between true and generated gender data, conditioned only on
gender information. By conditioning only on gender information, we aim to
distort the generated voice to protect both gender and identity.

The Generator, $\mathcal{G}$, takes as input mel-spectrograms $\mathcal{M}$, a
noise vector $\mathcal{Z}$ and the labels of the sensitive attribute to
protect $\mathcal{Y}$, and synthesises the transformed spectrograms
$\mathcal{M}^{\prime}$ (see Fig. 2). During training, a batch of $n$ audio
signals, $\mathcal{X}$, and their corresponding labels, $\mathcal{Y}$,
representing the sensitive attribute (gender) are sampled uniformly at random
from the dataset:

$(x_{1},y_{1}),...,(x_{n},y_{n})\sim\mathcal{X}_{train}.$ (1)

The audio signals $\mathcal{X}$ are converted to mel-spectrograms
$m_{i}\in\mathcal{M}$ $\forall i=1,...,n$, and normalized such that amplitudes
are bounded in $[0,1]$ with

$m_{i}=\mathcal{FFT}(x_{i}),\quad$ (2)

where $\mathcal{FFT}$ is the Short-Time Fourier Transform.

Figure 2: Privacy transformation and training procedure with the losses we use
for GenGAN. The time-signal waveform $\mathcal{X}$ is first converted into
mel-spectrogram and then fed as input to the Generator along with the noise
vector $\mathcal{Z}\sim\mathcal{N}(0,1)$ and the synthetic non-binary gender
label $\mathcal{Y}_{N}$. The generated mel-spectrogram $\mathcal{M^{\prime}}$
is fed to the Discriminator to predict if the sample corresponds to the
ground-truth binary gender or non-binary gender. We use the MelGAN pre-trained
vocoder to perform the mel-spectrogram inversion. KEY – $\mathcal{X}$:
original waveform, $\mathcal{X^{\prime}}$: transformed waveform,
$\mathcal{M}$: original mel-spectrogram, $\mathcal{M^{\prime}}$: transformed
mel-spectrogram, $\mathcal{Y}$(⚤): ground-truth gender label,
$\mathcal{Y}_{N}$: non-binary gender vector, $\mathcal{Y^{\prime}}$(⚬):
synthesised gender, $\mathcal{Y}_{F/R}$: predicted gender with Discriminator
from real (original) and fake (generated) spectrograms, $\mathcal{L}_{d}$:
distortion loss, $\mathcal{L}_{a}$: adversarial loss, $\mathcal{FFT}$: Fast
Fourier Transform.

As we consider a binary encoding for gender, we propose to sample from a
synthetic distribution

$\\{\hat{y}_{1},...,\hat{y}_{n}\\}\in\mathcal{Y}_{N}\sim\mathcal{N}(0.5,0.05).$
(3)

As $\mathcal{N}(0.5,0.05)$ is centred around 0.5 (i.e. equidistant from the
ground-truth labels), $\mathcal{G}$ learns to smooth feature-level differences
between the spectral representations of the two gender labels and synthesises
a new voice that is gender-ambiguous. We select a small distribution variance
($\sigma^{2}=0.05$) for the synthetic distribution to minimise the overlap
with the gender distributions. The noise vector
$\mathcal{Z}\sim\mathcal{N}(0,1)$ is inserted at the bottleneck of
$\mathcal{G}$, at the transition of the contracting to the expansive path to
ensure synthesised voices are different from the original ones, increasing
reconstruction variability. $\mathcal{Z}$ is reshaped and concatenated with
the last convolutional layer ending the contracting path in UNet before being
expanded back to a mel-spectrogram $\mathcal{M}^{\prime}$.

The generator loss $\mathcal{L}_{\mathcal{G}}$ is computed as an adversarial
game between reducing distortion on utility and maximising privacy by learning
a non-binary gender. We take the Mean Squared Error between $\mathcal{M}$ and
$\mathcal{M}^{\prime}$ as distortion loss $\mathcal{L}_{d}$ over the $L_{1}$
distance used in PCMelGAN to produce a smoother output. The adversarial loss
$\mathcal{L}_{a}$ is a cross-entropy loss between ground-truth gender
$\mathcal{Y}$ and predicted gender $\mathcal{Y}_{F}$, maximising the log
probability of generating samples drawn from the synthetic distribution. The
generator loss is the sum

$\mathcal{L}_{\mathcal{G}}=\mathcal{L}_{d}(\mathcal{M},\mathcal{M^{\prime})}+\epsilon\mathcal{L}_{a}(\mathcal{Y},\mathcal{Y}_{F}),$
(4)

where $\epsilon\in[0,1]$ represents the dis-utility budget [19] in the
privacy-utility trade-off.

The discriminator loss function is composed of two losses with respect to real
$\mathcal{M}$ or generated spectrogram $\mathcal{M}^{\prime}$. The real loss
is taken between the prediction of $\mathcal{D}$ on real data
$\mathcal{Y}_{R}$, and ground-truth gender $\mathcal{Y}$. The fake loss is
taken between the prediction of $\mathcal{D}$ on generated data
$\mathcal{Y}_{F}$, and non-binary gender $\mathcal{Y}_{N}$. Both losses are
computed as cross-entropies:

$\mathcal{L}_{\mathcal{D}}=\mathcal{L}_{a}(\mathcal{Y},\mathcal{Y}_{R})+\mathcal{L}_{a}(\mathcal{Y}_{N},\mathcal{Y}_{F}).$
(5)

Finally, to transform the generated spectrogram $\mathcal{M}^{\prime}$ back to
raw waveform $\mathcal{X}^{\prime}$, we use the MelGAN vocoder [14], a non-
autoregressive conditional waveform synthesis model.

Figure 3 shows sample spectrograms of different utterances spoken by a male
and by a female speaker. The bottom row shows the privacy-transformed
spectrograms produced by GenGAN. GenGAN synthesises higher resolution
spectrograms than PCMelGAN [10], while both methods affect high formant
frequencies associated with gender identification [20], smoothing out
differences between male and female voices.

Male

(a) Original

Female

(b) Original

(c) MelGAN

(d) MelGAN

(e) PCMelGAN

(f) PCMelGAN

(g) GenGAN

(h) GenGAN

Figure 3: Spectrograms of two different utterances spoken by a male (left
column) and a female speaker (right column). The privacy-preserving
transformation affects the high formant frequencies associated with gender
(see (e), (f), (g), (h)). (c) and (d) show the distortion induced on high
frequencies by the MelGAN vocoder with respect to the original utterances.

##  3 Validation

In this section we perform the validation of GenGAN in comparison with
existing attribute-protecting privacy-preserving methods. We use the
LibriSpeech dataset [21], which contains over 100 hours of transcribed clean
speech and over 250 different speakers with balanced gender distribution.

###  3.1 Performance measures

We now describe the measures we use to compare the performance of different
models in the scenario presented in Section 2.1.

We view speaker identity in the context of a speaker verification task, which
consists in determining whether a trial utterance corresponds to the utterance
of a certain, enrolled speaker. We chose speaker verification instead of
speaker identification as in our scenario the attacker uses a speaker
verification system to recover the identity of a speaker from anonymised
speech. We consider the equal error rate (EER) to assess the speaker
verification performance. In biometrics verification systems, a high accuracy
corresponds to a low EER, as a higher rejection over false acceptance rate is
desired. To assess privacy, the target randomness in verifying the speaker
identity [22] corresponds to an EER around 50%.

We measure success in gender inference as the standard binary classification
accuracy, where the sum of correct predictions is taken over the total number
of predictions. As in speaker verification, the target accuracy is 50%, which
represents randomness in gender prediction. We compute the discrepancy from
the 50% randomness value both on gender recognition and speaker verification
and introduce a new measure. The measure normalises the EER and the gender
recognition accuracy (GR) such that the absolute difference from the 50%
accuracy is retained. The normalised Gender Recognition ($gr$) and normalised
Equal Error Rate ($eer$) perform conversions given by $gr=100-2\times\lvert
GR-50\rvert$, and $eer=100-2\times\lvert EER-50\rvert$. A value of 100 for
$gr$ (or $eer$) denotes the highest level of privacy.

We use the transcription accuracy of the speech recognition results as the
utility measure. The Word Error Rate (WER) represents the utility of the
spoken signal and is computed by taking the Levenshtein [23] distance between
the words of the correct, expected transcription and the transcription
provided by the speech recognition service. To facilitate the comparison
between privacy and utility, rather than the error rate we consider the
correct recognition rate with the use of the Word Accuracy metric ($A_{w}$)
[11, 24], where ${A_{w}}=100-WER$. $A_{w}=100$ denotes no transcription
errors.

###  3.2 Classifiers

For speech recognition, we use Quartznet [25], an end-to-end neural acoustic
model trained with Connectionist Temporal Classification (CTC) [26] loss,
based on the Jasper architecture [27]. Our testing reported an initial
performance on the LibriSpeech test-clean set of $A_{w}$ of 95.64% (or 4.36%
WER).

For speaker verification, we extract speaker-identifying features with a
modified version of ResNet-34 [28] with reduced computational cost, which we
refer to as SpeakerNet [29], pre-trained on VoxCeleb2 [30] dataset and
reported an EER of 5.73% in our experiments when tested on the LibriSpeech
test-clean set.

For gender classification, we use a deep convolutional binary classifier
trained on spectrograms with 5 stacked one-dimensional convolutional layers
followed by batch normalisation and max pooling [11], which we refer to as
GenderNet. The output layer is a fully connected layer that outputs a pair of
predictions for each binary class, passing through a sigmoid function. We
tested the gender classifier on the LibriSpeech clean test set and reported an
accuracy of 91.37%.

###  3.3 Methods under comparison

We compare methods that consider gender as an attribute to protect in an
attribute inference scenario. VQ-VAE-Gen [11] considers gender information as
private and assesses the impact of the privacy-transformation on speaker
identity on LibriSpeech. The Client-Hybrid [7] model protects gender
information from inference by using speaker identity information and provides
results on the same dataset. We also compare our method with PCMelGAN [10]
anonymisation method which uses a filtering process to remove gender
information and conditions the model with identity information during
training. We did not modify the architecture of PCMelGAN and trained it from
scratch on the LibriSpeech clean train set. We improve and address the
limitations in PCMelGAN’s pipeline with GenGAN’s implementation. We train
PCMelGAN with the additional Filter network prior to the generation and keep
the original loss function used in [10] for the same number of epochs and
identical hyper-parameter setting. We use the same pre-trained MelGAN vocoder
model in our experiments with PCMelGAN and GenGAN. We also use the same pre-
trained models to evaluate the privacy tasks for GenGAN, PCMelGAN and VQ-VAE-
Gen. For utility, we use the same ASR for GenGAN and PCMelGAN models for a
fair comparison. We run all our experiments on a single Tesla V100 GPU with
32GB memory. The models were trained for 100 epochs by shuffling batches, a
learning rate of 0.001 and Adam optimiser [31].

###  3.4 Discussion

Table 1: Comparison of privacy and utility results for various models on the Librispeech [21] test clean set. KEY – $A_{w}$: Word Accuracy computed with QuartzNet [25], EER: Equal Error Rate computed with SpeakerNet [29], $eer$: normalised EER, GR: Gender Recognition computed with GenderNet [11], $gr$: normalised Gender Recognition, $\dagger$: RG (Random Gender) setting [11] is reported. $\star$: a random accuracy close to 50% is desired for high privacy. Results for GenGAN are obtained with $\epsilon=0.001$ and $\mathcal{Y}_{N}\sim(0.5,0.05)$. The MelGAN model is reported along with the Original signal to assess the impact of the spectrogram inversion without privacy-preserving transformation. Model | Utility | Privacy
---|---|---
|  $A_{w}$ $\uparrow$ |  EER$\star$ |  $eer$ $\uparrow$ |  GR$\star$ |  $gr$ $\uparrow$
Original | 95.64 | 5.73 | 11.46 | 91.37 | 17.26
MelGAN [14] | 93.22 | 17.42 | 34.84 | 89.04 | 21.92
Client-Hybrid [7] | 71.54 | – | – | 53.90 | 92.20
VQ-VAE-Gen [11]$\dagger$ | 26.84 | 51.88 | 96.24 | 50.01 | 99.98
PCMelGAN [10] | 66.81 | 41.95 | 83.90 | 48.39 | 96.78
GenGAN (ours) | 76.64 | 38.37 | 76.74 | 53.63 | 92.74

In Tab. 1 “Original” refers to the original audio prior to any transformation.
We also include the MelGAN [14] vocoder model to assess the distortion induced
by the mel-spectrogram inversion and compare the effect of this operation on
utility and privacy of the generated signal.

The values of privacy and utility for Client-Hybrid [7] and VQ-VAE-Gen [11]
are taken from the literature. The values corresponding to the ‘Hybrid’ model
with fine-tuning on two speakers were chosen for Client-Hybrid, and the
‘Random Gender’ (RG) setting, which considers only gender information during
training, for VQ-VAE-Gen. Although VQ-VAE-Gen provides the best privacy
guarantees both on gender and identity inference, it is also the worst
performing in terms of utility with an $A_{w}$ value of 26.84%. The Client-
Hybrid model performs equally well in terms of utility and privacy, however
the selected model for comparison was fine-tuned on two speakers from the
train set while no results on the speaker verification task are provided.
Results for PCMelGAN show that the privacy objectives are achieved but utility
remains low ($A_{w}=66.81$). GenGAN improves upon the utility with respect to
existing related models without being trained explicitly to optimise for the
speech recognition task. GenGAN is a simplified version of PCMelGAN with
modified cost functions that improve the privacy-utility trade-off with
respect to methods considering gender information. In terms of privacy, the
76.74 $eer$ value suggests that some residual identity information is still
present in the generated speech, since identity information was not explicitly
manipulated in the GenGAN pipeline.

Figure 4: Comparisons of the privacy-utility trade-off for the methods
compared in Table 1. Values close to the top-right corner denote higher
utility and privacy. GenGAN provides the best utility and comparable privacy
performance. KEY – $A_{w}$: Word Accuracy, $gr$: normalised Gender
Recognition, $eer$: normalised Equal Error Rate.

Privacy-utility results are reported in Fig. 4. The larger $eer$, $gr$ and
$A_{w}$, the higher the privacy and the higher the utility. GenGAN reaches the
best trade-off with improved reconstruction capabilities and comparable
privacy performance.

##  4 Conclusion

We proposed GenGAN, a generative adversarial network that synthesises gender-
ambiguous voices that can be used for privacy-preserving speech recognition
scenarios. The generator and discriminator are adversarially trained to limit
the distortion in the resulting gender-ambiguous signal. Our model improves
the privacy-utility trade-off with respect to existing methods.

Future work includes improving the signal reconstruction capabilities of the
network without compromising privacy and assessing the naturalness of the
generated voices with subjective evaluations.

## References

  * [1] L. G. Kersta, “Voiceprint Identification,” in _Nature_ , vol. 196, Dec 1962, pp. 1253–1257.
  * [2] D. Toney, D. Feinberg, and K. Richmond, “Acoustic Features for Profiling Mobile Users of Conversational Interfaces,” in _International Conference on Mobile Human-Computer Interaction (MobileHCI_. Berlin, Heidelberg: Springer, 2004, pp. 394–398.
  * [3] C. Y. Huang, Y. Y. Lin, H. Y. Lee, and L. S. Lee, “Defending Your Voice: Adversarial Attack on Voice Conversion,” in _IEEE Spoken Language Technology Workshop (SLT)_ , 2021, pp. 552–559.
  * [4] B. M. L. Srivastava, A. Bellet, M. Tommasi, and E. Vincent, “Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion?” in _Proceedings INTERSPEECH 2019-20th Annual Conference of the International Speech Communication Association_ , 2019.
  * [5] A. Nelus and R. Martin, “Gender Discrimination Versus Speaker Identification Through Privacy-Aware Adversarial Feature Extraction,” in _Speech Communication; 13th ITG-Symposium_ , 2018, pp. 1–5.
  * [6] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial Training of Neural Networks,” in _The Journal of Machine Learning Research (JMLR)_ , vol. 17, no. 1, 2016, pp. 2096–2030.
  * [7] P. Wu, P. Liang, R. Salakhutdinov, and L.-P. Morency, “Understanding the Tradeoffs in Client-Side Privacy for Speech Recognition,” _arXiv preprint arXiv:2101.08919_ , Jan. 2021.
  * [8] P.-G. Noé, M. Mohammadamini, D. Matrouf, T. Parcollet, A. Nautsch, and J.-F. Bonastre, “Adversarial Disentanglement of Speaker Representation for Attribute-Driven Privacy Preservation,” in _Proceedings INTERSPEECH 2021_ , Brno, Czech Republic, Aug 2021. [Online]. Available: https://hal.archives-ouvertes.fr/hal-03046920
  * [9] R. Aloufi, H. Haddadi, and D. Boyle, “Privacy-preserving Voice Analysis via Disentangled Representations,” _Proceedings of the 2020 ACM SIGSAC Conference on Cloud Computing Security Workshop_ , Nov 2020. [Online]. Available: http://dx.doi.org/10.1145/3411495.3421355
  * [10] D. Ericsson, A. Östberg, E. L. Zec, J. Martinsson, and O. Mogren, “Adversarial Representation Learning for Private Speech Generation,” in _International Conference on Machine Learning, Workshop on Self-supervision in Audio and Speech (SAS)_ , 2020.
  * [11] D. Stoidis and A. Cavallaro, “Protecting Gender and Identity with Disentangled Speech Representations,” in _Proceedings INTERSPEECH 21-22nd Annual Conference of the International Speech Communication Association_ , 2021, pp. 1699–1703.
  * [12] D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-Vectors: Robust DNN Embeddings for Speaker Recognition,” in _Proceedings International Conference on Acoustics, Speech and Signal Processing, IEEE_ , 2018, pp. 5329–5333.
  * [13] A. Van Den Oord, O. Vinyals, and K. Kavukcuoglu, “Neural Discrete Representation Learning,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 6306–6315.
  * [14] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brébisson, Y. Bengio, and A. C. Courville, “MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis,” in _Advances in Neural Information Processing Systems_ , vol. 32, 2019, p. 14910–14921.
  * [15] J. Martinsson, E. L. Zec, D. Gillblad, and O. Mogren, “Adversarial Representation Learning for Synthetic Replacement of Private Attributes,” in _IEEE International Conference on Big Data_ , Dec. 2021.
  * [16] S. J. Sutton, “Gender Ambiguous, Not Genderless: Designing Gender in Voice User Interfaces (VUIs) with Sensitivity,” in _Proceedings of the 2nd Conference on Conversational User Interfaces_ , ser. CUI ’20. New York, NY, USA: Association for Computing Machinery, 2020. [Online]. Available: https://doi.org/10.1145/3405755.3406123
  * [17] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in _Proceedings Medical Image Computing and Computer Assisted Intervention_ , 2015.
  * [18] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in _Advances in Neural Information Processing Systems_ , vol. 25, Dec. 2012, p. 1097–1105.
  * [19] A. Tripathy, Y. Wang, and P. Ishwar, “Privacy-preserving Adversarial Networks,” in _57th Annual Allerton Conference on Communication, Control, and Computing (Allerton)_. IEEE, 2019, pp. 495–505.
  * [20] M. S. F. Poon and M. L. Ng, “The Role of Fundamental Frequency and Formants in Voice Gender identification,” _Speech, Language and Hearing_ , vol. 18, no. 3, pp. 161–165, 2015.
  * [21] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an ASR Corpus Based on Public Domain Audio Books,” in _International Conference on Acoustics, Speech and Signal Processing, IEEE_ , South Brisbane, Queensland, Australia, April 2015, pp. 5206–5210.
  * [22] N. Tomashenko, B. M. L. Srivastava, X. Wang, E. Vincent, A. Nautsch, J. Yamagishi, N. Evans, J. Patino, J.-F. Bonastre, P.-G. Noé, and M. Todisco, “Introducing the VoicePrivacy Initiative,” in _Proceedings INTERSPEECH 2020-21st Annual Conference of the International Speech Communication Association, IEEE_ , 2020, pp. 1693–1697.
  * [23] V. I. Levenshtein, “Binary Codes Capable of Correcting Deletions, Insertions, and Reversals,” in _Soviet physics doklady_ , vol. 10, 1966, pp. 707–710.
  * [24] J. Qian, H. Du, J. Hou, L. Chen, T. Jung, and X.-Y. Li, “Hidebehind: Enjoy Voice Input with Voiceprint Unclonability and Anonymity,” in _Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems_ , 2018, pp. 82–94.
  * [25] S. Kriman, S. Beliaev, B. Ginsburg, J. Huang, O. Kuchaiev, V. Lavrukhin, R. Leary, J. Li, and Y. Zhang, “Quartznet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions,” in _Proceedings International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE_ , May 2020, pp. 6124–6128.
  * [26] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks,” in _Proceedings of the 23rd International Conference on Machine Learning_ , 2006, pp. 369–376.
  * [27] J. Li, V. Lavrukhin, B. Ginsburg, R. Leary, O. Kuchaiev, J. M. Cohen, H. Nguyen, and R. Gadde, “Jasper: An End-to-End Convolutional Neural Acoustic Model,” in _Proceedings INTERSPEECH 2019-20th Annual Conference of the International Speech Communication Association, IEEE_ , 2019, pp. 71–75.
  * [28] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Dec. 2016, pp. 770–778.
  * [29] J. S. Chung, J. Huh, S. Mun, M. Lee, H.-S. Heo, S. Choe, C. Ham, S. Jung, B.-J. Lee, and I. Han, “In Defence of Metric Learning for Speaker Recognition,” in _Proceedings INTERSPEECH 2020-21st Annual Conference of the International Speech Communication Association_ , Sep. 2020, pp. 2977–2981.
  * [30] J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” in _Proceedings INTERSPEECH 2018-19th Annual Conference of the International Speech Communication Association_ , 2018, pp. 1086–1090.
  * [31] D. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in _3rd International Conference on Learning Representations (ICLR)_ , San Diego, CA, USA, Dec. 2015.