# Expanding Cybersecurity Knowledge Through an Indigenous Lens: A First Look

Farrah Huntinghawk Department of Mathematics and Computer Science, Brandon
University, Manitoba, Canada
E-mail<EMAIL_ADDRESS>
Candace Richard Department of Mathematics and Computer Science, Brandon
University, Manitoba, Canada
E-mail<EMAIL_ADDRESS>
Sarah Plosker Department of Mathematics and Computer Science, Brandon
University, Manitoba, Canada
E-mail<EMAIL_ADDRESS>
Gautam Srivastava Department of Mathematics and Computer Science, Brandon
University, Manitoba, Canada
E-mail<EMAIL_ADDRESS>

###### Keywords:

Cybersecurity, Indigenous, Education, Public Interest, Internet.

## Abstract

Decolonization and Indigenous education are at the forefront of Canadian
content currently in Academia. Over the last few decades, we have seen some
major changes in the way in which we share information. In particular, we have
moved into an age of electronically-shared content, and there is an increasing
expectation in Canada that this content is both culturally significant and
relevant. In this paper, we discuss an ongoing community engagement initiative
with First Nations communities in the Western Manitoba region. The initiative
involves knowledge-sharing activities that focus on the topic of
cybersecurity, and are aimed at a public audience. This initial look into our
educational project focuses on the conceptual analysis and planning stage. We
are developing a ``Cybersecurity 101'' mini-curriculum, to be implemented over
several one-hour long workshops aimed at diverse groups (these public
workshops may include a wide range of participants, from tech-adverse to tech-
savvy). Learning assessment tools have been built in to the workshop program.
We have created informational and promotional pamphlets, posters, lesson
plans, and feedback questionnaires which we believe instill relevance and
personal connection to this topic, helping to bridge gaps in accessibility for
Indigenous communities while striving to build positive, reciprocal
relationships. Our methodology is to approach the subject from a community
needs and priorities perspective. Activities are therefore being tailored to
fit each community. We hope this will lead to increased awareness and
engagement by community members. Two Indigenous student research assistants
were hired to assist in this project, which has developed into a blend of
community outreach on the topic of security and data protection (most notably
with respect to social media and online banking) and a computing education
student-led educational research project.

##  1 Introduction

In 2019, Drs. Plosker and Srivastava received a Community Investment Grant
from the Canadian Internet Registration Authority (CIRA) for a knowledge-
sharing community engagement project aimed at broadening public understanding
of cybersecurity. The project, which is now underway, is aimed specifically at
facilitating broad-interest workshops on cybersecurity in rural First Nations
communities in Western Manitoba. Participants of the workshops will gain broad
understanding of cybersecurity, as well as practical information on security
and data protection. The main goal of the project is to include Indigenous
peoples in the Canadian cybersecurity dialogue, couched in the Four R's of
First Nations and Higher Education: Respect, Relevance, Reciprocity, and
Responsibility [9].

There are 63 First Nations in the province of Manitoba. In 2016, census data
shows that Manitoba had a population of 1,240,695 people, with 223,310 or
18.0% identifying as Aboriginal [17]. The term ``Western Manitoba" is somewhat
loosely defined, but comprises Treaties 1, 2, 4, and 5. Brandon University is
located on Treaty 2 Lands, traditional homelands of the Anishinabek, Cree,
Dakota, Dene, Oji-Cree, and Métis peoples.

Cybersecurity in Manitoba has made headlines recently: At the time of writing,
the province of Manitoba has announced a commitment of over half a million
dollars to help create a Cyber Security Technical Centre of Excellence [4]. In
the news release, Economic Development and Training Minister Ralph Eichler
states ``Cyberattacks are on the rise, and experts predict that cybercrime
damages will cost the world more than $6 trillion annually by 2021''. Other
recent notable cybersecurity news in Manitoba include a ransomware attack on
an insurance and financial brokerage based in Manitoba, that did not report
the attack to the Office of the Privacy Commissioner [7], and a cyberattack on
the overseeing body of foster children in ten agencies in southern Manitoba,
made possible because specific information on their IT system and software was
published publicly on their website [16].

Our project is incredibly timely: with a planned $83.9 million infrastructure
investment to provide a fibre-optic cable network for internet service into
112 rural Manitoba communities, including 48 First Nations [8, 11], the
communities we visit will soon have faster, more reliable internet. It is
therefore beneficial to discuss email phishing, password security, and other
important and relevant aspects of cybersecurity preemptively.

This project is in its initial stages. Although the project is still underway,
we describe here in detail our work done thus far, as we feel that knowledge
transfer of this project will have a positive impact on the larger Computer
Science Education sphere within Canada. We note that it is important to
understand how much time and effort is put into the organizing, planning, and
preparing stage, so that educational activities are ultimately successful.
Aligning educational resources in cybersecurity with the Truth and
Reconciliation Commission's Calls to Action, and disseminating these resources
and initial findings, will benefit those involved in the broader Canadian
Education system.

##  2 Project Methodology and Description

Drs. Plosker and Srivastava are both Associate Professors in the Department of
Mathematics and Computer Science at Brandon University in Brandon, Manitoba,
Canada. Dr. Plosker has a research background in applied mathematics,
including quantum cryptography. Dr. Srivastava has a research background in
cybersecurity. Both are interested in Indigenous community outreach.

The initial grant application for this project involved consultations with
Chris Lagimodiere, Director of the Indigenous People' Center at Brandon
University, to help ensure Indigenous cultures and communities are treated
respectfully and to ensure a focus on community needs. We decided to limit our
project to First Nations communities that are within a two hour drive from
Brandon, Manitoba, so that we could reasonably drive to the communities to
facilitate on-site workshops. We feel that it is important to go to the
communities rather than invite community members to come to Brandon to attend
workshops, in part from a logistics standpoint (to increase participation) and
from a standpoint of respect and consideration.

Two Brandon University students were hired to assist with the project as
Indigenous outreach workshop facilitators. One of the major candidate
qualifications was knowledge of Indigenous communities in the Western Manitoba
area. Huntinghawk (of Rolling River First Nations) and Richard (of Sandy Bay
First Nation) were hired. Huntinghawk had previously worked under Plosker in
facilitating a series of Fun Math Workshops at the Indigenous Peoples' Center
on campus in 2016-17 (e.g. Sudoku tournament, $\pi$-related activities for
March 14, etc). Though it was not mandatory in terms of the job description
and candidate qualifications, both of the student assistants employed for this
project are Indigenous.

The student assistants have worked closely on the creation and Indigenization
of the handouts and other educational materials related to the project in a
respectful manner. These educational materials used for the workshops can be
found in the supplemental material at [13] and contain basic information on
cybersecurity for a public audience. The educational materials have gone
through a series of editorial iterations; as we facilitate the cybersecurity
workshops, we notice what works well and what does not, and have made changes
based on our observations. We have, to the best of our abilities, aligned the
educational resource materials with the Truth and Reconciliation Commission's
Calls to Action [18], specifically with regard to Education: to develop
culturally appropriate curricula. The pamphlet was written to be both
entertaining and informative. Rather than being a sterile list of do's and
don't's, the pamphlet reads more like a friendly chat: ``Canadians love social
media, eh?'' starts off the section on social media security. The pamphlet
mentions the usefulness of social media in terms of following the goings-on of
comedians Don Burnstick, Wabigut and other Native icons, before cautioning the
reader to avoid posting too much personal information on social media, and
recommending having a strong password. The pamphlet also includes the logo of
the reserve that we visit, and can be further customized as needed.

At the time of writing, we have completed the initial stages of the project.
We are working closely with the Brandon University Research in Ethics
Committee (BUREC) and have received ethics certification for the project.
Plosker and Srivastava received certificates of completion of the Tri-Council
Policy Statement: Ethical Conduct for Research Involving Humans (TCPS) Course
on Research Ethics (CORE) tutorial. We have had positive initial discussions
and interactions with band Chiefs, Council members, Directors of Education,
school principals, and other designates. We have created survey questionnaires
(with feedback from BUREC), educational pamphlets, promotional posters, and
lesson plans. The actual ``Cybersecurity 101'' content can be customized based
on whether we provide a workshop in a school, for youth learning about
electrical and computer engineering, or a community-wide workshop, aimed more
at adults and seniors. All educational resource materials created have some
flexibility built in, so they can be edited based on comments from the band
Chief, Council, school principal, instructor, Elders, or others, and can be
tweaked on an as-needed basis.

We have, at the time of submitting this article, facilitated workshops within
the community of Waywayseecappo. A workshop within Sandy Bay First Nation had
to be rescheduled due to circumstances outside of our control. We are in the
process of finalizing dates for the community of Sandy Bay as well as the
remaining workshops.

##  3 Related Work

There is very little literature on the subject of Indigenous education
initiatives in the area of electrical and computer engineering, let alone
cybersecurity specifically. Much of the computer science literature
surrounding Indigenous cultures relates to preserving their languages through
digital technologies, or Indigenous health. Alternatively there are examples
of integrating Indigenous culture with digital technology. A community in
Alaska is respectfully integrating a part of Alaskian Indigenous culture by
combining digital story-telling with cancer education [6].

Numerous articles exist on the topic of cybersecurity and social media as it
relates to Indigenous youth in Australia: a recent literature review of 22
articles related to `social media and/or digital technologies and Indigenous
Australians'' [15] found several major themes surrounding ``how and why
Indigenous young people use social media: identity, power and control,
cultural compatibility and community and family connections''. Positive
applications of social media and digital technology included health promotion
and marketing, whereas negative applications included cyberbullying, cyber
racism, and the exchange of sexually explicit content between minors.

With regard to Canadian Indigenous peoples, research includes interviews of
Iqaluit students showing how their use of social media—in particular,
Facebook—was closely linked to their identity [5]. Similarly, Inuit young
adults are using Youtube as a digital storytelling medium [19]. An in-depth
literature review supplemented by personal communications on the use of
digital technologies in remote and northern Indigenous communities in Canada
[12] analyses uses and challenges of technology in indigenous communities. The
paper develops a community technology adoption assessment and identifies the
different levels of technology adoption throughout indigenous communities.

Using five First Nations communities as case studies, four in Ontario and one
in British Columbia, Polyzois asked ``How are First Nations communities using
online tools to enable residents to more actively participate in the planning
of their community?" [14]. The work includes a thorough literature review of
over 300 documents, through which two central themes emerged: public
engagement and Indigenous planning. In relation to these two themes, a number
of the documents discuss the _digital divide_ between First Nations and other
communities in Canada, defined in [14] as social and economic inequality with
respect to access to and use of information and computer technologies (ICT).
The study found a number of challenges: digital connectivity, internet
accessibility, affordability, and quality of ICT service. It also identified a
number of opportunities: ownership and control of ICT, building capacity and
developing technical skills, cultivating social capital, and fostering
community resilience. As noted in [14], addressing some of these challenges
and opportunities would be ``a key step in moving closer to establishing new
best practices in First Nation community development in rural and remote
locations in Canada''.

The work of [10] teases out the link between ``networked digital
infrastructure development and the autonomy and agency of Indigenous
peoples''. It should be noted that the majority of the references mentioned
herein are work done not within Computer Science departments, but within the
areas of Health, Geography, Sociology, and Communications.

###  3.1 Data Collection

We have found it easiest to start the workshops in communities where one of
the two student Indigenous outreach workshop facilitators (Huntinghawk and
Richard) have personal connections: Waywayseecappo and Sandy Bay. For these
two communities, Richard has worked closely with educational representatives
on finalizing dates and other logistics. The first workshops were at
Waywayseecappo Community School, one for both Grade 7 classes, and one for
both Grade 8 classes. The next workshop will be at Isaac Beaulieu Memorial
School in Sandy Bay, aimed at Grades 11 and 12. These grades were chosen by
the representatives that our team was in contact with in the respective
communities.

Waywayseecappo First Nation was established through the signing of Treaty 4 in
1874. Situated near the southwest corner of Riding Mountain National Park, it
is within 8 km of the town of Rossburn, and consists of 24,600 acres. The
total registered (status) population living in Waywayseecappo is 1,365 (2016
data) [3].

The Sandy Bay Ojibway First Nation was established through the signing of
Treaty 1 in 1871, with about 180 people of mixed-blood ancestry. Sandy Bay is
situated on Reserve No. 5, which comprises 16,456 acres on the western shore
of Lake Manitoba. The total registered population of Sandy Bay is 6,174 (July,
2013 data) and is increasing. See [2] for further details.

We plan to provide workshops in several other First Nations communities
surrounding Brandon and potentially at local institutions. As a first step, we
have contacted and have had positive response from several communities that
have pre-established relationships with Brandon University (through the
Student Recruitment office). Our team is working to finalize dates of the
remaining workshops.

We have constructed pre- and post-workshop survey questionnaires for workshop
participants; the former to gather information so as to learn how much the
community members already know about cybersecurity and how important they
think it is, and the latter to test how effective our workshops are, so that
we can make changes as necessary as we go forward with the project. Questions
range from basic administrative data (age, sex, name of community) to
questions about smartphone, social media, and internet use, whether they sign
out of online accounts or just close the web browser, how important they feel
internet safety is, and whether or not they've experienced cyberbullying,
among other things (a full list of survey questions is available at [13]). The
draft questionnaires have been given to our community contacts in advance, for
feedback and customization (e.g. if a particular community is having issues
surrounding social media, more questions related to that topic could be added
to the questionnaire in advance of the workshop).

Consent forms to use the data have been developed and are sent to the
instructor or designate prior to the workshop, for signature by the guardians.
To complement the workshops, we are also hoping to Indigenize some of the
``Cybersecurity 101'' materials which we are using at the workshops and that
could potentially be used in introductory computer science courses at Brandon
University as well.

Most research in cybersecurity education uses analogies: sealed envelopes,
boxes locked with padlocks, malicious eavesdroppers/adversaries, colour-
coding, and the use of signatures, seals, or fingerprints. This education
framework does not take Indigenous knowledge-sharing and storytelling into
account. We plan to work closely with several Elders and Knowledge-Keepers in
the area to Indigenize the cybersecurity educational materials through the
development of cybersecurity educational analogies using culturally relevant
stories and ideas.

###  3.2 Strengths & Limitation

Awareness, education, and training can minimize or prevent cybersecurity
risks. Ensuring that the public understands the importance of being vigilant
is key to obtaining ``buy-in'' to attending and participating in educational
activities. For this to happen, the topics and ways of approaching the topics
must be made personally relevant. For our project, the context so far has been
Treaty 1 and Treaty 4 First Nations communities that will be receiving new
fibre optics infrastructure for high-speed internet. Brandon University's main
campus is located on Treaty 2 land, with roughly 13% of our students self-
identifying as Aboriginal (First Nations, Métis, or Inuit)[1], and has a
general positive connection with many Indigenous communities in the Western
Manitoba region (e.g., our recruitment officers regularly recruit students
living on First Nations reserves, there is a weekly community traditional
beadwork event hosted at the Indigenous Peoples' Center on campus, a regular
graduation Pow Wow to honour Indigenous graduates, etc). For more information
on our Indigenous Peoples' Center and its services, see [1].

Individuals or groups looking to implement similar initiatives will need to
reflect on the realities of their geographic location, the relationship their
institution has developed with Indigenous communities in the area, and the
needs of these communities. Researchers and educators will find that one
cannot take our educational materials and ``skip to the finish line''; it is a
process. Time needs to be invested into creating meaningful, reciprocal
relationships with community members. All materials should be tailored to each
community.

##  4 Conclusions and Future Work

For this project, we felt it imperative that our work was guided by an
Indigenous knowledge framework. The creation of new educational materials on
cybersecurity is being done so as to incorporate culturally appropriate and
relevant Indigenous knowledge-sharing in a respectful way. We believe this
will lead to increased community engagement and level of satisfaction with the
cybersecurity activities. The educational materials were first written in
skeletal form, so that they could be edited and tailored to fit each community
and each situation.

We found it important to emphasize relationship building during this research
project. Researchers should not be motivated solely to obtain results; it is
about the entire process. Having pre-established relationships through
students involved in the project, or through various offices at one's
institution, helps but we caution that it does not instantaneously guarantee
success.

At present, we have funding to visit five First Nations communities in Western
Manitoba. We hope to secure more funding for future events, both for visiting
more communities as well as returning to communities to ensure a continued,
lasting relationship.

Upon completion of the workshops, we plan to reflect on not only the results
of the survey questionnaires but also the process as a whole. Workshop lesson
plans and educational materials created through this initiative have been made
freely available at [13]. We encourage any interested groups to tailor the
activities and handouts to their geographic region.

##  5 Acknowledgements

This work was supported by a Community Investment Grant from the Canadian
Internet Registration Authority (CIRA). The authors thank Chris Lagimodiere
for discussions at the initial stage of this project, as well as the Brandon
University Research Ethics Committee (BUREC) for guidance. C. Richard was
supported by a travel grant through the Sandy Bay Band office.

## References

  * [1] About the Indigenous Peoples’ Centre. Brandon University (2014-2019), https://www.brandonu.ca/ipc/about-ipc/
  * [2] About; Early History; History of Sandy Bay. Sandy Bay First Nation (2019), http://www.sandybayfirstnation.com/about-us.html
  * [3] Census profile, 2016 census. Statistics Canada (2019), https://www12.statcan.gc.ca/census-recensement/2016/dp-pd/prof/details/page.cfm?Lang=E&Geo1=CSD&Code1=4616017&Geo2=CSD&Code2=4616017&SearchText=Parklands&SearchType=Begins&SearchPR=01&B1=All&type=0
  * [4] Province promises $550K for new Manitoba centre focused on cybersecurity. CBC News (2019), https://www.cbc.ca/news/canada/manitoba/cybersecurity-technical-centre-funding-1.5400080
  * [5] Castleton, A.: Technology and Inuit identity: Facebook use by Inuit youth. AlterNative: An International Journal of Indigenous Peoples 14(3), 228–236 (2018)
  * [6] Cueva, M., Kuhnley, R., Revels, L.J., Cueva, K., Dignan, M., Lanier, A.P.: Bridging storytelling traditions with digital technology. International Journal of Circumpolar Health 72 (2013), dOI: 10.3402/ijch.v72i0.20717
  * [7] Daigle, T.: Prairie insurance and financial brokerage failed to disclose ransomware attack. CBC News (2019), https://www.cbc.ca/news/technology/andrew-agencies-ransomware-1.5400101
  * [8] Hoye, B.: 'A good start': High-speed internet to reach 112 rural, remote Manitoba communities. CBC News (2018), https://www.cbc.ca/news/canada/manitoba/internet-rural-remote-manitoba-communities-1.4478989
  * [9] Kirkness, V.J., Barnhardt, R.: First Nations and higher education: The four R's—respect, relevance, reciprocity, responsibility. Journal of American Indian Education pp. 1–15 (1991)
  * [10] McMahon, R.: Digital Self-determination: Aboriginal Peoples and the Network Society in Canada. Ph.D. thesis, Simon Fraser University, Simon Fraser University, Burnaby, BC, Canada (2013)
  * [11] Monkman, L.: Groups partner to build fibre optic network to rural Manitoba First Nations. CBC News (2018), https://www.cbc.ca/news/indigenous/groups-partner-to-build-fibre-optic-network-to-rural-manitoba-first-nations-1.4734386
  * [12] O’Donnell, S., Beaton, B., McMahon, R., Hudson, H.E., Williams, D., Whiteduck, T.: Digital technology adoption in remote and northern Indigenous communities in Canada. Canadian Sociological Association 2016 Annual Conference pp. 1–44 (2016)
  * [13] Plosker, S., Srivastava, G.: Expanding knowledge-sharing of cybersecurity through an Indigenous lens. Brandon University https://www.brandonu.ca/indigenous-cybersecurity/
  * [14] Polyzois, K.: Internet-based planning in First Nations communities: Challenges and opportunities (2018)
  * [15] Rice, E.S., Haynes, E., Royce, P., Thompson, S.C.: Social media and digital technology use among Indigenous young people in Australia: a literature review. International journal for equity in health 15(1), 81 (2016)
  * [16] von Stackelberg, M.: 'dysfunctional' child welfare authority left itself open for cyberattack, Sandy Bay CFS says. CBC News (2019), https://www.cbc.ca/news/canada/manitoba/foster-care-authority-cyberattack-sandy-bay-cfs-1.5374789
  * [17] Statistics Canada: Focus on geography series, 2016 census. Statistics Canada Catalogue no. 98-404-X2016001. Ottawa, Ontario (2017), https://www12.statcan.gc.ca
  * [18] Truth, Reconciliation Canada: Final report of the Truth and Reconciliation Commission of Canada: Summary : honouring the truth, reconciling for the future. Winnipeg: Truth and Reconciliation Commission of Canada (2015), http://trc.ca
  * [19] Wachowich, N., Scobie, W.: Uploading selves: Inuit digital storytelling on YouTube. Études/Inuit/Studies 34(2), 81–105 (2010)


# Learning to be adversarially robust and differentially private

Jamie Hayes, Borja Balle, M. Pawan Kumar  DeepMind

###### Abstract.

We study the difficulties in learning that arise from robust and
differentially private optimization. We first study convergence of gradient
descent based adversarial training with differential privacy, taking a simple
binary classification task on linearly separable data as an illustrative
example. We compare the gap between adversarial and nominal risk in both
private and non-private settings, showing that the data dimensionality
dependent term introduced by private optimization compounds the difficulties
of learning a robust model. After this, we discuss what parts of adversarial
training and differential privacy hurt optimization, identifying that the size
of adversarial perturbation and clipping norm in differential privacy both
increase the curvature of the loss landscape, implying poorer generalization
performance.

##  1\. Convergence rates of the logistic loss on a linearly separable problem
with gradient descent

Table 1. Convergence rates of empirical nominal and adversarial risk of logistic loss in non-private and private settings. Optimizer | Loss | Bound
---|---|---
Robust | Private
✗ | ✗ | $L(\theta^{t})$ | $\frac{8-\eta}{8t\eta}\bigg{(}(\frac{\log t}{\gamma})^{2}+1\bigg{)}+(\frac{8-\eta}{4})\log(\frac{t+1}{t})$
✗ | ✓ | $\mathbb{E}[L(\theta^{t})]$ | $\frac{8-\eta}{8t\eta}\bigg{(}(\frac{\log t}{\gamma})^{2}+1+d\sigma^{2}\bigg{)}+(\frac{8-\eta}{4})\log(\frac{t+1}{t})+\eta d\sigma^{2}$
✓ | ✗ | $L_{a}(\theta^{t})$ | $\big{(}\frac{8-\eta(1+c)^{2}}{8t\eta}-\frac{c}{t\gamma}\big{)}\bigg{(}(\frac{\log t}{\gamma-c})^{2}+(1+c)^{2}\bigg{)}+\big{(}\frac{8-\eta(1+c)^{2}}{4}-\frac{2c}{\gamma}\big{)}\log(\frac{t+1}{t})$
✓ | ✓ | $\mathbb{E}[L_{a}(\theta^{t})]$ | $\big{(}\frac{8-\eta(1+c)^{2}}{8t\eta}-\frac{c}{t\gamma}\big{)}\bigg{(}(\frac{\log t}{\gamma-c})^{2}+(1+c)^{2}+d\sigma^{2}\bigg{)}+\big{(}\frac{8-\eta(1+c)^{2}}{4}-\frac{2c}{\gamma}\big{)}\log(\frac{t+1}{t})+\eta d\sigma^{2}$

We begin by analyzing logistic regression as a motivating example.

Let
$\ell(\cdot,(\cdot,\cdot)):\mathcal{W}\times(\mathcal{X},\mathcal{Y})\rightarrow\mathbb{R}$
be the logistic loss $\ell(\theta;x,y)=\log(1+e^{-yx^{T}\theta})$. We assume
$\mathcal{X}\subseteq\mathbb{R}^{d}$ and
$\forall(x,y)\in(\mathcal{X},\mathcal{Y})$, $\left\lVert x\right\rVert\leq 1$
and $y\in\\{\pm 1\\}$. We further assume the data is linearly separable with
margin $\gamma$, and set
$u=\operatorname*{arg\,max}_{\left\lVert\theta\right\rVert=1}\min_{i\in[n]}y_{i}x_{i}^{T}\theta$,
the optimal hyperplane that classifies all $(x,y)$ correctly with margin at
least $\gamma$. We let $t\in\mathbb{N}$ denote the training step, and
$\eta\in\mathbb{R}$ denote the learning rate. We will see that the difference
in rate of convergence between non-robust and robust optimization grows with
the dimensionality $d$ of the input space $\mathcal{X}$.

Following a similar analysis in (Li et al., 2019), we compute convergence
rates for the loss under gradient descent with and without adversarial
training and differential privacy. Without robustness we optimize the loss
$L(\theta)=\frac{1}{n}\sum_{i=1}^{n}\ell(\theta;x_{i},y_{i})$, while in
adversarial training we minimize the adversarial loss
$L_{a}(\theta)=\frac{1}{n}\sum_{i=1}^{n}\ell_{a}(\theta;x_{i},y_{i})$, where
$\ell_{a}(\theta)=\max_{\left\lVert\delta\right\rVert\leq
c}\log(1+e^{-y(x+\delta)^{T}\theta})=\log(1+e^{-yx^{T}\theta+c\left\lVert\theta\right\rVert_{*}})$,
where $c$ is the size of adversarial perturbations. Throughout our analysis we
use the $\ell_{2}$ norm on $L_{a}$, however we also conduct experiments with
the $\ell_{\infty}$ norm. Under differential privacy the gradient at each time
step is perturbed by Gaussian noise with variance $\sigma$ — the privacy
guarantees can be computed based on the amount of noise, number of iterations
and Lipschitz constant of the loss in a standard way (Bassily et al., 2014).

Table 1 gives the converge rates in each of our four possible optimization
settings, with proofs given in appendix A. 111We stress that we do not claim
our bounds are tight. For example, we should be able to improve the bound in
the non-private non-robust setting by a factor of $\log t$ by appealing to
Theorem 3.3 in (Bubeck, 2014). We observe that optimizing the adversarial loss
introduces a dependence on the adversarial budget $c$ in the convergence
bound, while making the optimizer differentially private introduces terms
$d\sigma^{2}$ depending on the dimension and amount of noise. Combining
adversarially robust and private optimization introduces the two modifications
on the bound almost independently – the only cross-contribution is through a
_negative_ term. In general, two factors dominate the rate of convergence in
robust and private optimization: (i) the dimensionality of data $d$; and (ii)
the size of the adversarial budget $c$ with respect to the size of the margin
$\gamma$.

We discuss some interesting observations that arise from considering these
different optimization settings.

### Hierarchy of empirical risk bounds

We inspect the hierarchy of empirical nominal and adversarial risk in non-
private and private settings. Firstly, empirical nominal risk is always
smaller than empirical adversarial risk in either setting. The non-private
empirical adversarial risk can be smaller or larger than private empirical
nominal risk. The position of overlap depends upon respective hyperparameters
$c$ and $\sigma$, and data dependent parameters such as dimensionality. These
effects are shown in fig. 1 for a simple binary classification task.

Figure 1. Converge of logistic regression with $d=10$, $\sigma=0.25$,
$\gamma=1.0$, $c=0.1$, $\eta=0.1$.

Figure 2. Gap between bounds on empirical adversarial and nominal risk as a
function as the number of optimization steps. We measure the gap in the non-
private setting, and in the private setting for different values of $d$. We
use the same parameters as described in fig. 1.

### The gap between empirical adversarial risk and nominal risk is
approximately equal in non-private and private settings for small $d$

Despite the aforementioned hierarchy, it is worth asking: does the gap between
nominal and adversarial risk expand as we introduce privacy into the
optimization process? The answer is yes, however, this gap tends to zero over
training given a sufficient number of steps. We measure the gap between
adversarial and nominal risk throughout training in either the private or non-
private setting. Figure 2 shows that this gap tends to zero as
$t\rightarrow\infty$, and that the gap increases in the private setting as $d$
increases.

### Empirical adversarial risk comparison

It is worth noting that fig. 1 is not an apples-to-apples comparison because
we are comparing standard loss ($L$) under standard training against robust
loss ($L_{a}$) under adversarial training. We can compare with robust accuracy
under standard training by noting that under gradient descent, we can upper
bound on the robust loss by
$L_{a}(\theta^{t})\leq\frac{8-\eta}{8t\eta}\bigg{(}1+(\frac{\log
t}{\gamma})^{2}\bigg{)}+(\frac{8-\eta}{4})\log(\frac{t+1}{t})+c\big{(}1+\eta(t-1)\big{)}$
(a full derivation can be found in appendix A). We can then plot the robust
loss under gradient descent and adversarial training + gradient descent in
fig. 3, where the benefits of adversarial training are clear to see.

Figure 3. Comparison of robust loss under gradient descent with and without
adversarial training, using the same parameters as described in fig. 1.

### Hierarchy of convergence on non-linear models

We now evaluate how well our bounds on logistic regression match the
empirically observed convergence rates of non-linear models. We train a three
layer convolutional neural network on the MNIST dataset, and evaluate
canonical measures as introduced previously. Firstly, fig. 4 shows the
convergence hierarchy as previously introduced is preserved in practice;
nominal training has the lowest loss while private and robust optimization
incurs the largest loss over training. Nearly identical observations were made
for the test loss.

Figure 4. Training loss of a three-layer CNN on MNIST.

The nominal model has a test set accuracy of 99.03%, the robust model a test
set accuracy of 99.20%, the private model a test set accuracy of 96.10%, and
the robust and private model a test accuracy of 94.43%. The private models are
$(\epsilon,\delta)$-DP, with $\epsilon=3.8$ and $\delta=10^{-5}$, where
individual gradients were clipped to norm one (cf. Abadi et al. (2016)). The
robust models were trained with $\ell_{\infty}$ perturbations of size $0.3$
using 40 steps of PGD. All models are trained with standard SGD. Additionally,
we train analogous models on CIFAR-10 using six layers with max-pooling and
tanh activations, and observe similar descent curves. The private models are
$(\epsilon,\delta)$-DP, with $\epsilon=8$ and $\delta=10^{-5}$, where
individual gradients were clipped to norm 0.1. The robust models are trained
to be robust to $\ell_{\infty}$ perturbations of size $\nicefrac{{8}}{{255}}$
using 20 steps of PGD.

### Gap in practice

We now evaluate the gap between empirical adversarial and nominal risk on this
non-linear model. From fig. 2, we would expect that this gap shrinks as
$t\rightarrow\infty$, with the gap being strictly larger in the private
setting. This is precisely what we observe in fig. 5. Of course, we can no
longer be confident the gap will tend to zero in this non-toy setting.

Figure 5. Gap in loss between robust and non-robust optimization on MNIST.

### Robustness gap

In fig. 6(a) and fig. 6(c), we empirically evaluate the robustness of each
model trained on MNIST and CIFAR-10, using 100 PGD steps. For both robust and
non-robust optimized models, the private robustness curve resembles a shifted
version of the non-private robustness curve.

From fig. 6(b), we note the relative _improvement_ in robust accuracy when
going from a non-robust model to a robust model is slightly higher in the non-
private setting in comparison to the private setting. This implies that
learning to be both robust and private is a more difficult learning task, and
as a result robust accuracy suffers. We also note, from fig. 6(c) and fig.
6(d), the relative gap in improvement in robust accuracy between private and
non-private optimization on CIFAR-10 models is larger than on MNIST. This is
to be expected as our previous analysis suggested learning to be robust and
private is more difficult for datasets of larger dimensions.

(a) Robust accuracy on MNIST test set.

(b) Improvement in robust accuracy with robust optimization on MNIST.

(c) Robust accuracy on CIFAR-10 test set.

(d) Improvement in robust accuracy with robust optimization on CIFAR-10.

Figure 6. Robustness comparison of MNIST and CIFAR-10. For fig. 6(b) and fig.
6(d), given an attack budget, $c$, we plot the difference in accuracy between
a robust model and a nominal model, in either a non-private or private
setting. We refer to this as the _improvement_ of the robust model over the
nominal model.

##  2\. The role of clipping, noise and adversarial budget

(a) Maximum eigenvalue for different training attack budgets ($c$) against
clipping thresholds ($k$).

(b) The correspondence between maximum eigenvalue and test accuracy for
different training attack budgets ($c$).

(c) The correspondence between maximum eigenvalue and test accuracy for
different clipping thresholds ($k$).

Figure 7. Robust and clipped logistic regression. Each point in the figures
represents a different model trained to be robust to perturbations of size $c$
with gradients clipped to be smaller than $k$.

(a) Maximum eigenvalue for different training attack budgets ($c$) against
privacy guarantees ($\epsilon_{DP}$).

(b) The correspondence between maximum eigenvalue and test accuracy for
different training attack budgets ($c$).

(c) The correspondence between maximum eigenvalue and test accuracy for
different privacy guarantees ($\epsilon_{DP}$).

Figure 8. Private and robust logistic regression. Each point in the figures
represents a different model trained to be robust to perturbations of size $c$
and differentially private with $\epsilon_{DP}$.

In the previous section we have seen examples of the increased risk brought
about by robust and private optimization, and that theory and practice are in
alignment. We now take a deeper look at why private and robust optimization is
difficult. In particular, we concretize the connection between the smoothness
of the loss landscape, adversarial training and the magnitude of learned
parameters. Once again, we use the example of binary logistic regression on
separable data.

### Adversarial training

We again consider binary logistic regression with loss
$\ell_{a}(\theta)=\max_{\left\lVert\delta\right\rVert\leq
c}\log(1+e^{-y(x+\delta)^{T}\theta})=\log(1+e^{-yx^{T}\theta+c\left\lVert\theta\right\rVert_{*}})$.
We show in appendix B that:

(1)
$\displaystyle\nabla^{2}\ell_{a}(\theta^{*})=\frac{c}{2\left\lVert\theta^{*}\right\rVert}\bigg{(}I-\frac{\theta^{*}\theta^{*^{T}}}{\left\lVert\theta^{*}\right\rVert^{2}}\bigg{)},$

where $\theta^{*}$ is the optimal solution, perfectly separating data from the
two classes. Clearly $\nabla^{2}\ell_{a}(\theta^{*})$ is a positive semi-
definite matrix with eigenvalues at $0$ and
$\frac{c}{2\left\lVert\theta^{*}\right\rVert}$ – we can take the eigenvector
$v$ to be $\theta^{*}$ and then
$(I-\frac{\theta^{*}\theta^{*^{T}}}{\left\lVert\theta^{*}\right\rVert^{2}})\theta^{*}=0$,
and we can also take the eigenvector $v$ to be orthogonal to $\theta^{*}$
resulting in:

(2)
$\displaystyle\frac{c}{2\left\lVert\theta^{*}\right\rVert}\bigg{(}I-\frac{\theta^{*}\theta^{*^{T}}}{\left\lVert\theta^{*}\right\rVert^{2}}\bigg{)}v=\frac{c}{2\left\lVert\theta^{*}\right\rVert}v$

Thus, the curvature of our loss landscape is completely determined by the
ratio of adversarial budget to the magnitude of the optimal parameters, and as
$c$ decreases we achieve smoother solutions.

### Role of clipping

We define the clipping operation on a function, $f$, as
$clip(f,k)=\min(1,\frac{k}{\left\lVert f\right\rVert})f$, for some
$k\in\mathbb{R}_{+}$. Xie et al. (2018) showed that that gradient clipping
implies bounded weight parameters. We can assume the optimal parameter takes
the form
$\theta^{*}_{clip}:=h(k)\frac{\theta}{\left\lVert\theta\right\rVert}$, where
$h(k)$ is a monotonically increasing function of the gradient clipping
parameter, $k$. Note, we also assume we can find the optimum under the
clipping norm $k>0$, and so we can repeat the above analysis. This leaves the
smoothness of the loss landscape determined by $\frac{c}{2h(k)}$. Similar to
the purely adversarial setting, this has the intuitive property that
decreasing the clipping norm decreases the smoothness of the loss landscape.

### Role of noise

The landscape sharpness is governed by the ratio of adversarial noise to
weight norm. If we perform noisy gradient descent:
$\theta^{t+1}=\theta^{t}-\eta(\nabla+b)$, where
$b\sim\mathcal{N}(0,\sigma^{2}I_{d})$, then
$\theta^{t+1}\sim\mathcal{N}(\theta^{t}-\eta\nabla,\eta^{2}\sigma^{2}I_{d})$.
We can use Jensen’s inequality to upper bound the expected norm of
$\theta^{t+1}$:

(3)
$\displaystyle\mathbb{E}[\left\lVert\theta^{t+1}\right\rVert]\leq\mathbb{E}[\left\lVert\theta^{t+1}\right\rVert^{2}]^{\frac{1}{2}}=\sqrt{\left\lVert\theta^{t}-\eta\nabla\right\rVert^{2}+d\sigma^{2}}$

This implies the expected norm is larger with an additional Gaussian noise
term added to gradients.

### Empirical validation

We now demonstrate that these effects are observed in practice using multi-
class logistic regression on MNIST. We show that smaller adversarial budgets,
$c$, and larger clipping norms, $k$, lead to smoother solutions with smaller
generalization error.

Firstly, we train without differentially private noise; only clipping and
adversarial training are activated. This means our models are not
differentially private but allows us to show our analysis holds empirically
(c.f. fig. 7). In total, we train 2500 multi-class logistic regression on
MNIST, sweeping over different training attack budget ($c$) and clipping
threshold ($k$) configurations. Each point in the fig. 7 represents a model
trained with a specific ($c$, $k$) configuration.

After this, we will show that moving to differentially private models has
virtually no difference empirically, implying our analysis is useful in these
settings (c.f. fig. 8). Again, in total, we train 2500 multi-class logistic
regression on MNIST, sweeping over different training attack budget ($c$) and
differentially private $\epsilon_{DP}$ configurations. Each point in the fig.
8 represents a model trained with a specific ($c$, $\epsilon_{DP}$)
configuration.

In fig. 7(a), we plot the maximum eigenvalue of the Hessian for different
training attack budgets, $c$, and maximum gradient norm, $k$. As expected, the
smallest eigenvalues belong to solutions with small $c$ and large $k$. For
values of $k$ close to zero, effective learning becomes increasingly
difficult, resulting in non-smooth solutions regardless of the choice of
attack budget, $c$. Likewise, for attack budgets close to $0.1$, the maximum
Hessian eigenvalue remains high even if we increase the clipping threshold,
$k$, (i.e. the space of learnable solutions). In fig. 8(a), we plot analogous
results but for fully differential private models, and observe similar
results; small attack budgets, $c$, and less privacy (large $\epsilon_{DP}$)
results in smaller eigenvalues.

In fig. 7(b) and fig. 8(b), we fix $k$ and $\epsilon_{DP}$, respectively, and
visualize the relationship between maximum Hessian eigenvalue and test
accuracy as a function of the attack budget $c$. We can immediately note that
smaller eigenvalues (that imply smoother solutions) result in smaller
generalization error, confirming that smooth solutions are preferable.
Unsurprisingly, as $c$ increases, the probability of finding a smooth model
with high test accuracy decreases.

Similarly, in fig. 7(c) and fig. 8(c), we fix the attack budget $c$, and
visualize the relationship between maximum Hessian eigenvalue and test
accuracy as a function of either $k$ or $\epsilon_{DP}$. Again, we observe
that less smooth solutions are more likely to have smaller test set accuracy.
Decreasing $k$ or $\epsilon_{DP}$ (more privacy) increase the chances of
finding a model with a large maximum eigenvalue and (comparatively) small test
set accuracy.

We have a theoretical understanding of how clipping and adversarial training
affect the learned model. Importantly, the trends observed in fig. 7 are
mirrored in fig. 8, implying that our analysis is useful in the fully
differentially private and robust setting.

##  3\. Discussion

Recent work by Song et al. (2019) has empirically shown a tension between
privacy and robust learning. In particular, it is shown that six state-of-the-
art defense methods designed to reduce the success of adversarial examples
_increase_ the risk of membership inference attacks due to overfitting on the
training set (Shokri et al., 2017). The phenomenon of overfitting in
adversarially robust optimization has also been observed by Rice et al.
(2020). If Song et al. (2019) showed that robust models are less private,
Tursynbek et al. (2020) recently identified the contrapositive relation,
empirically showing that differentially private models come at the expense of
robustness. Meanwhile, Ghazi et al. (2021) shows that the sample complexity of
learning both robust and private halfspaces is worse than learning with either
property by itself.

## References

  * Abadi et al. (2016) Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_ , pp. 308–318, 2016.
  * Bassily et al. (2014) Bassily, R., Smith, A., and Thakurta, A.  Private empirical risk minimization: Efficient algorithms and tight error bounds.  In _2014 IEEE 55th Annual Symposium on Foundations of Computer Science_ , pp. 464–473. IEEE, 2014.
  * Bubeck (2014) Bubeck, S.  Convex optimization: Algorithms and complexity.  _arXiv preprint arXiv:1405.4980_ , 2014.
  * Bun & Steinke (2016) Bun, M. and Steinke, T.  Concentrated differential privacy: Simplifications, extensions, and lower bounds.  In _Theory of Cryptography Conference_ , pp. 635–658. Springer, 2016.
  * Ghazi et al. (2021) Ghazi, B., Kumar, R., Manurangsi, P., and Nguyen, T.  Robust and private learning of halfspaces.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 1603–1611. PMLR, 2021.
  * Li et al. (2019) Li, Y., Fang, E. X., Xu, H., and Zhao, T.  Inductive bias of gradient descent based adversarial training on separable data.  _arXiv preprint arXiv:1906.02931_ , 2019.
  * Rice et al. (2020) Rice, L., Wong, E., and Kolter, Z.  Overfitting in adversarially robust deep learning.  In _International Conference on Machine Learning_ , pp. 8093–8104. PMLR, 2020.
  * Shamir & Zhang (2013) Shamir, O. and Zhang, T.  Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes.  In _International conference on machine learning_ , pp. 71–79. PMLR, 2013.
  * Shokri et al. (2017) Shokri, R., Stronati, M., Song, C., and Shmatikov, V.  Membership inference attacks against machine learning models.  In _2017 IEEE Symposium on Security and Privacy (SP)_ , pp. 3–18. IEEE, 2017.
  * Song et al. (2019) Song, L., Shokri, R., and Mittal, P.  Privacy risks of securing machine learning models against adversarial examples.  In _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ , pp. 241–257, 2019.
  * Tursynbek et al. (2020) Tursynbek, N., Petiushko, A., and Oseledets, I.  Robustness threats of differential privacy.  _arXiv preprint arXiv:2012.07828_ , 2020.
  * Xie et al. (2018) Xie, L., Lin, K., Wang, S., Wang, F., and Zhou, J.  Differentially private generative adversarial network.  _arXiv preprint arXiv:1802.06739_ , 2018.

##  Appendix A Convergence rates of logistic loss on a linearly separable
problem with gradient descent

Let
$\ell(\cdot,(\cdot,\cdot)):\mathcal{W}\times(\mathcal{X},\mathcal{Y})\rightarrow\mathbb{R}$
be the logistic loss, we assume $\mathcal{X}\subseteq\mathbb{R}^{d}$ and
$\forall(x,y)\in(\mathcal{X},\mathcal{Y})$, $\left\lVert x\right\rVert\leq 1$
and $y\in\\{\pm 1\\}$.

Following a similar analysis in (Li et al., 2019), we compute convergence
rates for gradient descent with and without adversarial training and
differential privacy. We assume the data is linearly separable with margin
$\gamma$, and set
$u=\operatorname*{arg\,max}_{\left\lVert\theta\right\rVert=1}\min_{i\in[n]}y_{i}x_{i}^{T}\theta$,
the optimal hyperplane that classifies all $(x,y)$ correctly with margin at
least $\gamma$.

No privacy / No robustness. We find convergence rates under the logistic loss
$\ell(\theta;y,x)=\log(1+e^{-yx^{T}\theta})$222We omit the $x$ and $y$ terms
in $\ell$ for brevity hereon in.. The logistic loss has the following first
and second derivatives:

(4)
$\displaystyle\nabla\ell(\theta)=\frac{-ye^{-yx^{T}\theta}x}{1+e^{-yx^{T}\theta}}$
(5)
$\displaystyle\nabla^{2}\ell(\theta)=\frac{e^{-yx^{T}\theta}xx^{T}}{(1+e^{-yx^{T}\theta})^{2}}$

Assuming $\left\lVert x\right\rVert\leq 1$, we have
$\left\lVert\nabla\ell(\theta)\right\rVert\leq 1$ and
$\left\lVert\nabla^{2}\ell(\theta)\right\rVert\leq\frac{1}{4}$. We let
$L(\theta)=\frac{1}{n}\sum_{i=1}^{n}\ell(\theta)_{i}$, and so $\nabla
L(\theta)=\frac{1}{n}\sum_{i=1}^{n}\frac{-y_{i}e^{-y_{i}x_{i}^{T}\theta}x_{i}}{1+e^{-y_{i}x_{i}^{T}\theta}}$.
By Taylor expansion we have:

(7) $\displaystyle L(\theta^{t+1})=L(\theta^{t}-\eta_{t}\nabla
L(\theta^{t}))\leq L(\theta^{t})-\eta_{t}\left\lVert\nabla
L(\theta^{t})\right\rVert^{2}+\frac{\eta_{t}^{2}}{8}\left\lVert\nabla
L(\theta^{t})\right\rVert^{2}$

where $\eta_{t}$ is the learning rate at step $t$. For simplicity, we assume
gradient descent is performed with a constant learning rate $\eta_{t}:=\eta$
for every step $t$. Our following analysis will depend on a sufficiently small
learning rate, which we specify at the relevant places. Because $L$ is smooth
(bounded Hessian), we can apply standard gradient descent convergence
analysis. For any $\theta\in\mathbb{R}^{d}$:

(8) $\displaystyle\left\lVert\theta^{t+1}-\theta\right\rVert^{2}$
$\displaystyle=\left\lVert\theta^{t}-\theta\right\rVert^{2}-2\eta\langle\nabla
L(\theta^{t}),\theta^{t}-\theta\rangle+\eta^{2}\left\lVert\nabla
L(\theta^{t})\right\rVert^{2}$ (9)
$\displaystyle\leq\left\lVert\theta^{t}-\theta\right\rVert^{2}-2\eta(L(\theta^{t})-L(\theta))+\eta^{2}\left\lVert\nabla
L(\theta^{t})\right\rVert^{2}\qquad\text{(by convexity of $\ell$)}$ (10)
$\displaystyle\leq\left\lVert\theta^{t}-\theta\right\rVert^{2}-2\eta(L(\theta^{t})-L(\theta))+\frac{\eta}{1-\frac{\eta}{8}}(L(\theta^{t})-L(\theta^{t+1}))\qquad\text{(by
Taylor expansion)}$ (11)
$\displaystyle=\left\lVert\theta^{t}-\theta\right\rVert^{2}+(\frac{\eta}{1-\frac{\eta}{8}}-2\eta)L(\theta^{t})+2\eta
L(\theta)-\frac{\eta}{1-\frac{\eta}{8}}L(\theta^{t+1})$

If $\eta<4$, the contribution of the $L(\theta^{t})$ term is negative and so:

(12) $\displaystyle\left\lVert\theta^{t+1}-\theta\right\rVert^{2}$
$\displaystyle\leq\left\lVert\theta^{t}-\theta\right\rVert^{2}+2\eta
L(\theta)-\frac{\eta}{1-\frac{\eta}{8}}L(\theta^{t+1})$

If we set $\theta^{0}=0$, then
$\theta^{1}=-\frac{\eta}{n}\sum_{i=1}^{n}\frac{-y_{i}x_{i}}{2}$ and
$\left\lVert\theta^{1}\right\rVert\leq\frac{\eta}{2}\leq 1$ as long as
$\eta<2$. Under the assumption that $L(\theta^{t+1})\leq L(\theta^{t})$,
summing the above between $s=1,...,t$, we get:

(13)
$\displaystyle\left\lVert\theta^{t+1}-\theta\right\rVert^{2}\leq\left\lVert\theta^{1}-\theta\right\rVert^{2}-t\bigg{(}\frac{\eta}{1-\frac{\eta}{8}}L(\theta^{t+1})-2\eta
L(\theta)\bigg{)}$ (14) $\displaystyle\implies$ $\displaystyle
L(\theta^{t+1})\leq\frac{1-\frac{\eta}{8}}{t\eta}(\left\lVert\theta^{1}\right\rVert^{2}+\left\lVert\theta\right\rVert^{2})+2(1-\frac{\eta}{8})L(\theta)$

Since we are free to choose $\theta$, we set it to $\theta:=\frac{\log
t}{\gamma}u$, then:

(15) $\displaystyle L(\theta)$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{n}\log(1+e^{\frac{-y\log
t}{\gamma}x^{T}u})\leq\frac{1}{n}\sum_{i=1}^{n}\log(1+e^{-\log
t})=\log(\frac{t+1}{t})$

Together with the fact that $\left\lVert\theta\right\rVert=\frac{\log
t}{\gamma}$, we get the final upper bound:

(16) $\displaystyle L(\theta^{t+1})$
$\displaystyle\leq\frac{1-\frac{\eta}{8}}{t\eta}\bigg{(}1+(\frac{\log
t}{\gamma})^{2}\bigg{)}+2(1-\frac{\eta}{8})\log(\frac{t+1}{t})$ (17)
$\displaystyle=\frac{8-\eta}{8t\eta}\bigg{(}1+(\frac{\log
t}{\gamma})^{2}\bigg{)}+(\frac{8-\eta}{4})\log(\frac{t+1}{t})$

No privacy / robustness. We next find convergence rates under gradient-based
adversarial training. Let
$\ell_{a}(\theta)=\max_{\left\lVert\delta\right\rVert\leq
c}\log(1+e^{-y(x+\delta)^{T}\theta})=\log(1+e^{-yx^{T}\theta+c\left\lVert\theta\right\rVert})$.
We first find the first and second derivative of this loss.

If we let $h(\theta):=e^{-yx^{T}\theta+c\left\lVert\theta\right\rVert}$, then
$\ell_{a}(\theta)=\log(1+h(\theta))$ and
$\nabla\ell_{a}(\theta)=\frac{h^{\prime}(\theta)}{1+h(\theta)}$, where
$h^{\prime}(\theta)=h(\theta)r(\theta)$ if we take
$r(\theta):=-yx+c\frac{\theta}{\left\lVert\theta\right\rVert}$. Similarly:

(18)
$\displaystyle\nabla^{2}\ell_{a}(\theta)=\frac{h(\theta)r(\theta)r(\theta)^{T}+h(\theta)^{2}r^{\prime}(\theta)+h(\theta)r^{\prime}(\theta)}{(1+h(\theta))^{2}}$

To bound the above we must bound
$r^{\prime}(\theta)=c(\frac{I}{\left\lVert\theta\right\rVert}-\frac{\theta\theta^{T}}{\left\lVert\theta\right\rVert^{3}})$.
We first note that the problem can be reduced to finding a lower bound to
$\left\lVert\theta^{t}\right\rVert$ for any $t>0$ because $\left\lVert
r^{\prime}(\theta)\right\rVert\leq\frac{2c}{\left\lVert\theta\right\rVert}$.
If we let $\theta^{0}=0$, then $\ell_{a}$ is no longer differentiable at zero,
so we take the sub-gradient to be:

(19)
$\displaystyle\frac{e^{-yx^{T}\theta+c\left\lVert\theta\right\rVert}(-yx+cr)}{1+e^{-yx^{T}\theta+c\left\lVert\theta\right\rVert}}\in\partial\ell_{a}(\theta^{1})$

where $r$ is a vector with $\left\lVert r\right\rVert\leq 1$. For example, we
can take $r=0$, and then the sub-derivative becomes
$-\frac{yx}{2}\in\partial\ell_{a}(\theta^{1})$, and is bounded above like
$\theta^{1}=\frac{\eta_{0}}{2n}\sum_{i=1}^{n}y_{i}x_{i}$, and
$\langle\theta^{1},u\rangle\geq\frac{\eta_{0}\gamma}{2}$. We then take
$\eta_{t}:=\eta$ for $t>1$ such that $\frac{\eta_{0}}{2}>\eta$. We then note
the inner product between $-\nabla\ell_{a}(\theta)$ and $u$ is strictly
positive:

(20)
$\displaystyle\langle-\nabla\ell_{a}(\theta),u\rangle\geq\frac{e^{-yx^{T}\theta+c\left\lVert\theta\right\rVert}}{1+e^{-yx^{T}\theta+c\left\lVert\theta\right\rVert}}(\gamma-c)>0$

and so

(21) $\displaystyle\langle\theta^{2},u\rangle=\langle\theta^{1}-\eta\nabla
L_{a}(\theta^{1}),u\rangle=\langle\theta^{1},u\rangle+\eta\langle-\nabla
L_{a}(\theta^{1}),u\rangle\geq\langle\theta^{1},u\rangle\geq\eta\gamma$

By a similar argument for any $t>1$,
$\langle\theta^{t},u\rangle\geq\eta\gamma$. Because $u$ is the global
minimizer it follows that $\left\lVert\theta^{t}\right\rVert\geq\eta\gamma$.
We have a lower bound on $\theta^{t}$ for any $t$ and so $\left\lVert
r^{\prime}(\theta)\right\rVert\leq\frac{2c}{\eta\gamma}$.

The other term we upper bound is
$r(\theta)r(\theta)^{T}=xx^{T}-2cyx^{T}\frac{\theta}{\left\lVert\theta\right\rVert}+c^{2}\frac{\theta\theta^{T}}{\left\lVert\theta\right\rVert^{2}}$,
and since we assume bounded data we can bound this like $\left\lVert
r(\theta)r(\theta)^{T}\right\rVert\leq(1+c)^{2}$. Piecing this altogether
gives:

(22) $\displaystyle\left\lVert\nabla^{2}L_{a}(\theta)\right\rVert$
$\displaystyle\leq\left\lVert\frac{h(\theta)r(\theta)r(\theta)^{T}}{(1+h(\theta))^{2}}\right\rVert+\left\lVert\frac{h(\theta)(1+h(\theta))r^{\prime}(\theta)}{(1+h(\theta))^{2}}\right\rVert$
(23)
$\displaystyle=\left\lVert\frac{h(\theta)r(\theta)r(\theta)^{T}}{(1+h(\theta))^{2}}\right\rVert+\left\lVert\frac{h(\theta)r^{\prime}(\theta)}{(1+h(\theta))}\right\rVert$
(24)
$\displaystyle\leq\left\lVert\frac{h(\theta)}{(1+h(\theta))^{2}}\right\rVert\left\lVert
r(\theta)r(\theta)^{T}\right\rVert+\left\lVert\frac{h(\theta)}{(1+h(\theta))}\right\rVert\left\lVert
r^{\prime}(\theta)\right\rVert$ (25)
$\displaystyle\leq\frac{(1+c)^{2}}{4}+\frac{2c}{\eta\gamma}$

Letting $s:=\frac{(1+c)^{2}}{4}+\frac{2c}{\eta\gamma}$, now that we have a
bound on the second derivatives we can again use the Taylor expansion of
gradient descent:

(26) $\displaystyle L_{a}(\theta^{t+1})$
$\displaystyle=L_{a}(\theta^{t}-\eta\nabla L_{a}(\theta^{t}))$ (27)
$\displaystyle\leq L_{a}(\theta^{t})-\eta\left\lVert\nabla
L_{a}(\theta^{t})\right\rVert^{2}+\frac{s\eta^{2}}{2}\left\lVert\nabla
L_{a}(\theta^{t})\right\rVert^{2}$

Then for any $\theta\in\mathbb{R}^{d}$,

(28) $\displaystyle\left\lVert\theta^{t+1}-\theta\right\rVert^{2}$
$\displaystyle\leq\left\lVert\theta^{t}-\theta\right\rVert^{2}-2\eta(L_{a}(\theta^{t})-L_{a}(\theta))+\frac{\eta}{1-\frac{s\eta}{2}}(L_{a}(\theta^{t})-L_{a}(\theta^{t+1}))$

As long as $\eta$ satisfies $s\eta<1$, the $L_{a}(\theta^{t})$ contribution is
negative and we get:

(30) $\displaystyle\left\lVert\theta^{t+1}-\theta\right\rVert^{2}$
$\displaystyle\leq\left\lVert\theta^{t}-\theta\right\rVert^{2}+2\eta
L_{a}(\theta)-\frac{2\eta}{2-s\eta}L_{a}(\theta^{t+1})$

Isolating $\eta$ in the condition $s\eta<1$ gives
$\eta<\frac{4(\gamma-2c)}{\gamma(1+c)^{2}}$ which also implies that we require
$c<\frac{\gamma}{2}$ for our results to hold. Summing over the iterations as
before gives:

(32)
$\displaystyle\left\lVert\theta^{t+1}-\theta\right\rVert^{2}\leq\left\lVert\theta^{1}-\theta\right\rVert^{2}-t(\frac{2\eta}{2-s\eta}L_{a}(\theta^{t+1})-2\eta
L_{a}(\theta))$ (33) $\displaystyle\implies$
$\displaystyle\frac{2t\eta}{2-s\eta}L_{a}(\theta^{t+1})\leq\left\lVert\theta^{1}-\theta\right\rVert^{2}+2t\eta
L_{a}(\theta)$

By choosing $\theta=\frac{\log t}{\gamma-c}u$ we again get
$\ell_{a}(\theta)=\log(1+\frac{1}{t})$ and
$\left\lVert\theta\right\rVert=\frac{\log t}{\gamma-c}$, giving the final
upper bound:

(34) $\displaystyle L_{a}(\theta^{t+1})$
$\displaystyle\leq\frac{2-s\eta}{2t\eta}(\left\lVert\theta\right\rVert^{2}+\left\lVert\theta^{1}\right\rVert^{2}+2t\eta
L_{a}(\theta))$ (35)
$\displaystyle\leq\frac{2-s\eta}{2t\eta}\bigg{(}(\frac{\log
t}{\gamma-c})^{2}+(1+c)^{2}\bigg{)}+(2-s\eta)\log(1+\frac{1}{t})$

Privacy / No robustness. We next find convergence rates under gradient descent
with differential privacy. At each step we add noise; we do not clip gradients
as they are already $1$-Lipschitz and we assume here the clipping norm is
larger than one. Let $\overline{\nabla L(\theta)}=\nabla L(\theta)+b$, where
$b\sim\mathcal{N}(0,\sigma^{2}I_{d})$. Throughout the following analysis we
find convergence results under expectations taken over the noise added for
differential privacy.

Our analysis is almost identical to the non-private case, and we sometimes
write $\nabla$ or $\overline{\nabla}$ where the context is clear. Firstly, the
Taylor expansion follows:

(36) $\displaystyle L(\theta^{t+1})$
$\displaystyle=L(\theta^{t}-\eta\overline{\nabla})$ (37) $\displaystyle\leq
L(\theta^{t})-\eta\nabla^{T}\overline{\nabla}+\frac{\eta^{2}}{2}\overline{\nabla}^{T}\nabla^{2}\overline{\nabla}$

Note that $\left\lVert\nabla^{2}\right\rVert\leq\frac{1}{4}$ and
$\overline{\nabla}=\nabla+b$. Although we take expectations over the random
noise in following, we omit the notation $\mathbb{E}$ for conciseness:

(38) $\displaystyle L(\theta^{t+1})$ $\displaystyle\leq
L(\theta^{t})-\eta\left\lVert\nabla\right\rVert^{2}+\frac{\eta^{2}}{8}(\left\lVert\nabla\right\rVert^{2}+d\sigma^{2})$

Then for any $\theta\in\mathbb{R}^{d}$, and again taking expectations over the
noise,

(39) $\displaystyle\left\lVert\theta^{t+1}-\theta\right\rVert^{2}$
$\displaystyle=\left\lVert\theta^{t}-\theta\right\rVert^{2}-2\eta\langle\nabla+b,\theta^{t}-\theta\rangle+\eta^{2}\left\lVert\nabla+b\right\rVert^{2}$
(40) $\displaystyle\leq\left\lVert\theta^{t}-\theta\right\rVert^{2}+2\eta
L(\theta)-\frac{8\eta}{8-\eta}L(\theta^{t+1})+\frac{8d\eta^{2}\sigma^{2}}{8-\eta}$

Repeating the same process as in the non-private case we get:

(41) $\displaystyle L(\theta^{t+1})$
$\displaystyle\leq\frac{8-\eta}{8t\eta}\bigg{(}1+d\sigma^{2}+(\frac{\log
t}{\gamma})^{2}\bigg{)}+(\frac{8-\eta}{4})\log(\frac{t+1}{t})+\eta
d\sigma^{2}$

Privacy / robustness. We can repeat the analysis as in non-private and robust
case, while taking expectations over the random noise to give the upper bound:

(42) $\displaystyle L_{a}(\theta^{t+1})$
$\displaystyle\leq\frac{2-s\eta}{2t\eta}\bigg{(}(1+c)^{2}+d\sigma^{2}+(\frac{\log
t}{\gamma-c})^{2}\bigg{)}+(2-s\eta)\log(1+\frac{1}{t})+\eta d\sigma^{2}$

Importantly, if we compare the non-private case to the private case for either
the robust or non-robust setting, we take an equivalent hit in convergence
that depends on the variance of noise and dimensionality of data.

### Empirical adversarial risk comparison

Figure 1 is not an apples-to-apples comparison because we are comparing
standard loss ($L$) under standard training against robust loss ($L_{a}$)
under adversarial training. We can compare with robust accuracy under standard
training by noting that under gradient descent, if
$\left\lVert\theta^{1}\right\rVert\leq 1$, then
$\left\lVert\theta^{t}\right\rVert\leq 1+\eta(t-1)$, and together with:

(43) $\displaystyle\ell_{a}(\theta)$
$\displaystyle=\log(1+e^{-yx^{T}\theta+c\left\lVert\theta\right\rVert_{*}})$
(44)
$\displaystyle=\log(e^{-c\left\lVert\theta\right\rVert_{*}}+e^{-yx^{T}\theta})+c\left\lVert\theta\right\rVert_{*}$
(45)
$\displaystyle\leq\log(1+e^{-yx^{T}\theta})+c\left\lVert\theta\right\rVert_{*}$
(46) $\displaystyle=\ell(\theta)+c\left\lVert\theta\right\rVert_{*},$

we get the following upper bound on the robust loss under gradient descent:

(47)
$\displaystyle\begin{split}L_{a}(\theta^{t})\leq&\frac{8-\eta}{8t\eta}\bigg{(}1+(\frac{\log
t}{\gamma})^{2}\bigg{)}+\\\
&(\frac{8-\eta}{4})\log(\frac{t+1}{t})+c\big{(}1+\eta(t-1)\big{)}\end{split}$

##  Appendix B Binary logistic regression with clipping and adversarial
training.

The following analysis only considers a single input, but can extended to a
batch of inputs through simple averaging. We again consider binary logistic
regression with loss $\ell_{a}(\theta)=\max_{\left\lVert\delta\right\rVert\leq
c}\log(1+e^{-y(x+\delta)^{T}\theta})=\log(1+e^{-yx^{T}\theta+c\left\lVert\theta\right\rVert})$.
As stated previously the first and second derivatives are given by:

(48)
$\displaystyle\nabla\ell_{a}(\theta)=\frac{h(\theta)r(\theta)}{1+h(\theta)}$
(49)
$\displaystyle\nabla^{2}\ell_{a}(\theta)=\frac{h(\theta)r(\theta)r(\theta)^{T}+h(\theta)^{2}r^{\prime}(\theta)+h(\theta)r^{\prime}(\theta)}{(1+h(\theta))^{2}}$

where $h(\theta):=e^{-yx^{T}\theta+c\left\lVert\theta\right\rVert}$,
$r(\theta):=-yx+c\frac{\theta}{\left\lVert\theta\right\rVert}$, and
$r^{\prime}(\theta)=c(\frac{I}{\left\lVert\theta\right\rVert}-\frac{\theta\theta^{T}}{\left\lVert\theta\right\rVert^{3}})$.
At the optimal $\theta^{*}=\operatorname*{arg\,min}_{\theta}\ell_{a}(\theta)$,
we have
$-yx+c\frac{\theta^{*}}{\left\lVert\theta^{*}\right\rVert}=0\implies\theta^{*}=\frac{yx\left\lVert\theta^{*}\right\rVert}{c}$.
The $h(\theta)r(\theta)r(\theta)^{T}$ term in the second derivative vanishes
at the optimum and $h(\theta^{*})=1$ and we are left with:

(50)
$\displaystyle\nabla^{2}\ell_{a}(\theta^{*})=\frac{r^{\prime}(\theta^{*})}{2}=\frac{c}{2\left\lVert\theta^{*}\right\rVert}\bigg{(}I-\frac{\theta^{*}\theta^{*^{T}}}{\left\lVert\theta^{*}\right\rVert^{2}}\bigg{)}$

Clearly $\nabla^{2}\ell_{a}(\theta^{*})$ is a positive semi-definite matrix
with eigenvalues at $0$ and $\frac{c}{2\left\lVert\theta^{*}\right\rVert}$ –
we can take the eigenvector $v$ to be $\theta^{*}$ and then
$(I-\frac{\theta^{*}\theta^{*^{T}}}{\left\lVert\theta^{*}\right\rVert^{2}})\theta^{*}=0$,
and we can also take the eigenvector $v$ to be orthogonal to $\theta^{*}$
resulting in:

(51)
$\displaystyle\frac{c}{2\left\lVert\theta^{*}\right\rVert}\bigg{(}I-\frac{\theta^{*}\theta^{*^{T}}}{\left\lVert\theta^{*}\right\rVert^{2}}\bigg{)}v=\frac{c}{2\left\lVert\theta^{*}\right\rVert}v$

Thus, the curvature of our loss landscape is completely determined by the
ratio of adversarial noise to the magnitude of the optimal parameters, and as
$c$ decreases we achieve smoother solutions.

##  Appendix C Excess risk analysis for robust & private learning

So far we have concentrated our analysis on logistic regression. We now give a
simple risk analysis for general convex losses for robust learning with
differential privacy.

Let
$\ell(\cdot,(\cdot,\cdot)):\mathcal{W}\times(\mathcal{X},\mathcal{Y})\rightarrow\mathbb{R}$
be a loss function with the following properties: $L$-Lipschitz, $G$-smooth,
convex and linear in its parameters, $\ell(w,(x,y))=\ell(yw^{T}x)$, $\forall
w\in\mathcal{W}$, $\left\lVert w\right\rVert\leq\beta$,
$\mathcal{X}\subseteq\mathbb{R}^{d}$,
$\forall(x,y)\in(\mathcal{X},\mathcal{Y})$, $\left\lVert x\right\rVert\leq 1$
and $y\in\\{\pm 1\\}$.

Consider two batch updates that differ in their final element:

(52) $\displaystyle
P=\sum_{i=1}^{n-1}y_{i}\nabla\ell(y_{i}w_{t}^{T}x_{i})x_{i}+y_{n}\nabla\ell(y_{n}w_{t}^{T}x_{n})x_{n}+z$
(53) $\displaystyle
Q=\sum_{i=1}^{n-1}y_{i}\nabla\ell(y_{i}w_{t}^{T}x_{i})x_{i}+y_{n}^{\prime}\nabla\ell(y_{n}^{\prime}w_{t}^{T}x_{n}^{\prime})x_{n}^{\prime}+z$

Setting $A=\sum_{i=1}^{n-1}y_{i}\nabla\ell(y_{i}w_{t}^{T}x_{i})x_{i}$,
$B=y_{n}\nabla\ell(y_{n}w_{t}^{T}x_{n})x_{n}$,
$B^{\prime}=y_{n}^{\prime}\nabla\ell(y_{n}^{\prime}w_{t}^{T}x_{n}^{\prime})x_{n}^{\prime}$,
we have $P=\mathcal{N}(A+B,\sigma^{2}I_{d})$ and
$Q=\mathcal{N}(A+B^{\prime},\sigma^{2}I_{d})$. To achieve
$(\epsilon,\delta)-DP$, we must bound
$\alpha_{M}(\lambda)=\max_{X,X^{\prime}}\alpha_{M}(\lambda;X,X^{\prime})$ as
defined in (Abadi et al., 2016), where $\alpha_{M}(\lambda)=\lambda
D_{\lambda+1}(P||Q)$ is as defined in (Bun & Steinke, 2016). This results in
the following:

(54) $\displaystyle\alpha_{M_{k}}(\lambda)=\lambda D_{\lambda+1}(P\rVert
Q)=\frac{\lambda(\lambda+1)\left\lVert
B-B^{\prime}\right\rVert^{2}}{2\sigma^{2}}$

Under the assumption that $\left\lVert x\right\rVert\leq 1$, $y\in\\{\pm
1\\}$, and $\ell$ is L-Lipschitz, we have the following upper bound:

(55) $\displaystyle\left\lVert B-B^{\prime}\right\rVert\leq
2L\implies\alpha_{M_{k}}(\lambda)\leq\frac{2\lambda(\lambda+1)L^{2}}{\sigma^{2}}$

Summing over all iterations gives:

(56)
$\displaystyle\alpha_{M}(\lambda)\leq\sum_{k=1}^{T}\alpha_{M_{k}}(\lambda)\leq\frac{2\lambda(\lambda+1)L^{2}T}{\sigma^{2}}\leq\frac{c\lambda^{2}L^{2}T}{\sigma^{2}}$

for some constant $c$. Then for some $c^{\prime}$, taking
$\sigma^{2}=\frac{c^{\prime}L^{2}T\log(\frac{1}{\delta})}{\epsilon^{2}}$ will
give $(\epsilon,\delta)-DP$ according to Theorem 2.2 in (Abadi et al., 2016).

Turning to the robust setting, we wish to solve
$\min_{w\in\mathcal{W}}\max_{\left\lVert v\right\rVert_{p}\leq
r}\ell\big{(}yw^{T}(x+v)\big{)}$, and in the convex setting the inner-
maximization can be given in closed-form as:

(57) $\displaystyle\max_{\left\lVert v\right\rVert_{p}\leq
r}\ell\big{(}yw^{T}(x+v)\big{)}=\ell(yw^{T}x-r\left\lVert w\right\rVert_{q})$

where $\frac{1}{p}+\frac{1}{q}=1$. If we set $p=\infty$, then squared
Euclidean distance between $B$ and $B^{\prime}$ becomes

(58) $\displaystyle\left\lVert B-B^{\prime}\right\rVert$
$\displaystyle\leq\left\lVert
y_{n}\nabla\ell_{n}x_{n}-y_{n}^{\prime}\nabla\ell_{n}^{\prime}x_{n}^{\prime}-r\operatorname{sign}(w)\nabla\ell_{n}+r\operatorname{sign}(w)\nabla\ell_{n}^{\prime}\right\rVert$
(59) $\displaystyle\leq 2\sqrt{d}(1+r)L$

This similarly holds for $p=2$ by noting that that the derivative of
$\left\lVert w\right\rVert_{2}$ is of unit length and replacing
$\operatorname{sign}{(w)}$ with this unit length value. Then
$(\epsilon,\delta)-DP$ holds by taking
$\sigma^{2}=\frac{c^{\prime}(1+r)^{2}L^{2}T\log(\frac{1}{\delta})}{\epsilon^{2}}$.

First note that if $\ell$ is $L$-Lipschitz, then
$\mathbb{E}[\left\lVert\hat{g}\right\rVert^{2}]=\left\lVert
g\right\rVert^{2}+\mathbb{E}[\left\lVert\hat{z}\right\rVert^{2}]\leq
L^{2}+d\sigma^{2}$. Assuming $\ell$ is $\lambda$-strongly convex, then by
Theorem 1 of (Shamir & Zhang, 2013) the bound on excess risk is given by the
following:

(60)
$\displaystyle\mathbb{E}[\ell(w_{T})-\ell(w^{*})]\leq\frac{17(L^{2}+d\sigma^{2})(1+\log(T))}{\lambda
T}$

where for $t>0$ the per step learning rate satisfies
$\eta_{t}=\frac{1}{\lambda t}$.

For private learning we can plug in
$\sigma^{2}=\frac{c^{\prime}L^{2}T\log(\frac{1}{\delta})}{\epsilon^{2}}$ and
for robust and private learning we can plug in
$\sigma^{2}=\frac{c^{\prime}(1+r)^{2}L^{2}T\log(\frac{1}{\delta})}{\epsilon^{2}}$.


# Fisher Information of a Family of Generalized Normal Distributions

Precious Ugo Abara and Sandra Hirche Precious Ugo Abara and Sandra Hirche are
with Technical University of Munich, Germany, Department of Electrical and
Computer Engineering, Chair of Information-oriented Control (ITR) e-mail:
{ugoabara<EMAIL_ADDRESS>
###### Abstract

In this brief note we compute the Fisher information of a family of
generalized normal distributions. Fisher information is usually defined for
regular distributions, i.e. continuously differentiable (log) density
functions whose support does not depend on the family parameter $\theta$.
Although the uniform distribution in $[-\theta,+\theta]$ does not satisfy the
regularity requirements, as a special case of our result, we will obtain the
Fisher information for this family.

###### Index Terms:

Fisher information, Generalized Normal Distribution, Uniform distribution

##  I Introduction

A natural question is that arises in the context of parameter estimation is:
how much information can a sample of data provide about the unknown parameter?
This is a widely studied problem in literature [1, 2]. One information metric
is the Fisher information [3]. Given a family of density functions
$f(x,\theta)$ function of a parameter $\theta$, it always hold

$\int f(x,\theta){\rm d}x=1.$ (1)

Under certain regularity conditions, if $\theta$ is the true parameter, it can
be shown that of (1) translates into

$\mathbb{E}\left[\frac{\partial}{\partial\theta}\operatorname{log}f(x,\theta)\Big{|}\theta\right]=0.$

The partial derivate ${\partial}/{\partial\theta}$ of the natural logarithm of
the likelihood $f(x,\theta)$ is referred to as score. The Fischer information
$\mathcal{I}(\theta)$is defined as the variance of the score i.e.

$\mathcal{I}(\theta)=\mathbb{E}\left[\left(\frac{\partial}{\partial\theta}\operatorname{log}f(x,\theta)\right)^{2}\Big{|}\theta\right].$

If the second derivative with respect to $\theta$ of
$\operatorname{log}f(x,\theta)$ exists for all $x$ and $\theta$ and the second
derivative with respect to $\theta$ of the left side of (1) can be obtained by
differentiating twice under the integral sign then the Fisher information may
also be written as [3]

$\mathcal{I}(\theta)=-\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta^{2}}\operatorname{log}f(x,\theta)\Big{|}\theta\right].$
(2)

We note that the condition of interchange of the derivative and integration
excludes the uniform distribution $f(x,\theta)=\frac{1}{\theta}$,
$\theta\in[a,b]$. In order words, the uniform distribution does not meet the
regularity conditions.

##  II Preliminaries

In this section we give results for a random variable $X$ with a generalized
normal distribution with zero mean i.e.

$f(x,\theta)=\frac{\beta}{2\theta\Gamma(1/\beta)}\operatorname{exp}\left(-\frac{|x|^{\beta}}{\theta^{\beta}}\right)$
(3)

where $\Gamma$ denotes the gamma function, $\theta>0$ is called the scale,
$\beta>0$ is the shape. For a complex variable $z$, $\Gamma$ is defined as

$\Gamma(z)=\begin{cases}(z-1)!,&\text{if }z\text{ is positive integer},\\\
\int_{0}^{\infty}x^{z-1}\operatorname{exp}(-x){\rm d}x,&\text{if
}\mathbb{R}(z)>0.\end{cases}$

Equation (3) is a parametric family of symmetric distributions. It includes
the zero-mean Laplace distribution when $\beta=1$, the zero-mean normal
distribution when $\beta=2$. Furthermore, it converges point-wise to the
uniform density for $\beta\to\infty$. See Fig. 1.

Figure 1: Generalized normal distribution for $\theta=1$ and different shape
parameter $\beta$.

######  Lemma 1 (Moments [4])

Let $X$ be zero mean generalized Gaussian distribution of shape $\beta$ and
scaling parameter $\theta$ as in (3). The moments of $X$ exist and are finite
for any $k$ greater than $−1$. For any non-negative integer $k$, the plain
central moments are

$\mathbb{E}\left[X^{k}\right]=\begin{cases}\theta^{k}\frac{\Gamma\left(\frac{k+1}{\beta}\right)}{\Gamma\left(\frac{1}{\beta}\right)},&\text{if
}k\text{ is even},\\\ 0,&\text{otherwise}.\end{cases}$

######  Lemma 2 ($\Gamma$ at some rational arguments)

Given integers $n>0$ and $p>0$ it holds that

$\Gamma\left(n+\frac{1}{p}\right)=\Gamma\left(\frac{1}{p}\right)\frac{(pn-(p-1))!^{(p)}}{p^{n}}$
(4)

where $n!^{(p)}$ denotes the $p$-th multi-factorial of $n$.

###### Proof:

Using the functional equation $\Gamma(\alpha+1)=\alpha\Gamma(\alpha)$ for
$\alpha>0$, and by its successive application, it is easy to obtain

$\Gamma\left(n+\frac{1}{p}\right)=\Gamma\left(\frac{1}{p}\right)\frac{(pn-p+1)\left(p(n-1)-p+1\right)\cdots\left(p(n-n+1)-p+1\right))}{p^{n}}.$

This concludes the proof. ∎

##  III Result

###### Theorem 1

Suppose $X$ is a random variable with a zero-mean generalized normal
distribution given by (3), and an even shape $\beta$, i.e. $\beta=2n$, with
$n$ a positive integer. Then

$\mathcal{I}(\theta)=\frac{\beta}{\theta^{2}}.$

###### Proof:

Since (3) is twice differentiable we can use (2) to compute the Fisher
information. The log-likelihood is given by

$\operatorname{log}f(x,\theta)=\operatorname{log}\frac{\beta}{2}-\operatorname{log}\theta-\operatorname{log}\Gamma\left(\frac{1}{\beta}\right)-\frac{|x|^{\beta}}{\theta^{\beta}}.$

Its first partial derivative with respect to the parameter $\theta$ is

$\frac{\partial}{\partial\theta}\operatorname{log}f(x,\theta)=-\frac{1}{\theta}+\frac{\beta\theta^{\beta-1}}{\theta^{2\beta}}|x|^{\beta}=-\frac{1}{\theta}+\frac{\beta|x|^{\beta}}{\theta^{\beta+1}}.$

The second derivative then follows as

$\frac{\partial^{2}}{\partial\theta^{2}}\operatorname{log}f(x,\theta)=\frac{1}{\theta^{2}}-\frac{\beta(\beta+1)\theta^{\beta}}{\theta^{2\beta+2}}|x|^{\beta}=\frac{1}{\theta^{2}}-\frac{\beta(\beta+1)|x|^{\beta}}{\theta^{\beta+2}}.$
(5)

Using (2) and (5) it then follows

$\mathcal{I}(\theta)=-\mathbb{E}\left[\frac{\partial^{2}}{\partial\theta^{2}}\operatorname{log}f(x,\theta)\Big{|}\theta\right]=-\frac{1}{\theta^{2}}+\frac{\beta(\beta+1)\mathbb{E}\left[|x|^{\beta}|\theta\right]}{\theta^{\beta+2}}.$
(6)

Since $\beta$ is assumed to be a positive even integer it holds that

$\mathbb{E}\left[|x|^{\beta}|\theta\right]=\mathbb{E}\left[x^{\beta}|\theta\right]=\theta^{\beta}\frac{\Gamma\left(\frac{\beta+1}{\beta}\right)}{\Gamma\left(\frac{1}{\beta}\right)}\overset{\eqref{eq:
gamma rational
argument}}{=}\frac{\theta^{\beta}}{\beta}\frac{\Gamma\left(\frac{1}{\beta}\right)}{\Gamma\left(\frac{1}{\beta}\right)}=\frac{\theta^{\beta}}{\beta}.$
(7)

where we used (4) with $p=\beta$, $n=1$ and noticing that $1!^{(\beta)}=1$.
Substituting (7) in (6)

$\mathcal{I}(\theta)=-\frac{1}{\theta^{2}}+\frac{\beta+1}{\theta^{2}}=\frac{\beta}{\theta^{2}}.$

∎

###### Example 1

A zero-mean Gaussian distribution is obtained from (3) with $\beta=2$. It is
easy to see that a family of Gaussian distributions with unknown standard
deviation has Fisher information $\mathcal{I}(\theta)=\frac{2}{\theta^{2}}$.

###### Remark 1

When $\beta\to\infty$ the distribution (3) converges to a uniform density on
$(-\theta,+\theta)$. The Fisher information is thus
$\mathcal{I}(\theta)=\infty$, for $\beta\to\infty$.

## References

  * [1] S. Kullback and R. A. Leibler, “On information and sufficiency,” _The annals of mathematical statistics_ , vol. 22, no. 1, pp. 79–86, 1951.
  * [2] T. M. Cover and J. A. Thomas, _Elements of information theory_. John Wiley & Sons, 2012.
  * [3] E. L. Lehmann and G. Casella, _Theory of point estimation_. Springer Science & Business Media, 2006.
  * [4] S. Nadarajah, “A generalized normal distribution,” _Journal of Applied Statistics_ , vol. 32, no. 7, pp. 685–694, 2005.