# Gender Bias in Remote Pair Programming among Software Engineering Students:
The twincode Exploratory Study

Amador Durán  SCORE Lab, I3US InstituteUniversidad de SevillaSevillaSpain
<EMAIL_ADDRESS>,  Pablo Fernández  SCORE Lab, I3US InstituteUniversidad de
SevillaSevillaSpain<EMAIL_ADDRESS>,  Beatriz Bernárdez  I3US
InstituteUniversidad de SevillaSevillaSpain<EMAIL_ADDRESS>,  Nathaniel Weinman
Computer Science DivisionUniversity of California, BerkeleyBerkeley, CAUSA
<EMAIL_ADDRESS>,  Aslı Akalın  Computer Science DivisionUniversity of
California, BerkeleyBerkeley, CAUSA<EMAIL_ADDRESS>and  Armando Fox
Computer Science DivisionUniversity of California, BerkeleyBerkeley, CAUSA
<EMAIL_ADDRESS>
(2021)

###### Abstract.

_Context_. Pair programming has been found to increase student interest in
Computer Science, particularly so for women, and would therefore appear to be
a way to help remedy the under–representation of women in the field. However,
one reason for this under–representation is the unwelcoming climate created by
gender stereotypes applied to engineers in general, and to software engineers
in particular, assuming that men perform better than their women peers. If
this same bias is present in pair programming, it could work against the goal
of improving gender balance in computing. _Objective._ In a remote setting in
which students cannot directly observe the gender of their peers, we aim to
explore whether Software Engineering students behave differently when the
_perceived_ gender of their remote pair programming partners changes,
searching for differences in (i) the perceived productivity compared to solo
programming; (ii) the partner’s perceived technical competency compared to
their own; (iii) the partner’s perceived skill level; (iv) the interaction
behavior, such as the frequency of source code additions, deletions,
validations, etc.; and (v) the type and relative frequencies of dialog
messages used for collaborative behavior in a chat window. Although there are
some studies on pair programming performance and gender pair combination, to
the best of our knowledge there are no studies on the impact of gender
stereotypes and bias _within_ the pairs themselves. _Method_. We have
developed an online platform (twincode) that randomly classifies students into
gender–balanced groups, arranges them in pairs for remote pair programming
(sharing an editor window and a chat window), and can selectively deceive one
or both partners regarding the gender of the other via the use of a clearly
gendered avatar. Several behaviors are automatically measured during the pair
programming process, together with two questionnaires and a semantic tagging
of the pairs’ conversations. We will perform a series of experiments to
identify the effect, if any, of possible gender bias in remote pair
programming interactions. Students in the control group will have no
information about their partner’s gender; students in the treatment group will
receive such information but will be selectively deceived about their
partner’s true gender. To analyze the data, apart from checking reliability of
questionnaire data using Cronbach’s alpha and Kaiser criterion, for each
response variable we will (i) compare control and experimental groups for the
score distance between two in–pair tasks; then, using the data from the
experimental group only, we will (ii) compare scores using the partner’s
perceived gender as a within–subjects variable; and (iii) analyze the
interaction between the partner’s perceived gender (within–subjects) and the
subject’s gender (between–subjects). For the (i) and (ii) analyses we will use
t–tests, whereas for the (iii) analyses we will use mixed–model ANOVAs.

Gender Bias, Pair Programming, Remote Pair Programming, Distributed Pair
Programming, Software Engineering Education

††copyright: acmcopyright††journalyear: 2021††conference: 15th ACM/IEEE
International Symposium on Empirical Software Engineering and Measurement;
October 11–15, 2021; Bari, Italy††booktitle: ESEM 2021: 15th ACM/IEEE
International Symposium on Empirical Software Engineering and Measurement,
October 11–15, 2021, Bari, Italy††price: 15.00††isbn:
978-1-4503-XXXX-X/21/10††ccs: General and reference Empirical studies††ccs:
Software and its engineering Agile software development

##  1\. Introduction

Pair programming is an increasingly popular collaboration paradigm that has
been shown to be an effective tool in Computer Science education as measured
by positive influence on grades, class performance, confidence, productivity,
and motivation to stay (da Silva Estácio and Prikladnicki, 2015), especially
for women (Rodríguez et al., 2017; Werner et al., 2004). In pair programming,
two partners work closely together to solve a programming task. As such, their
ability to engage with each other is key. However, these interactions are
influenced by implicit gender bias (Hofer, 2015; Martell et al., 1996), such
as assuming women are less technically competent (Martell et al., 1996). This
is a widely observed phenomenon even in highly–structured settings (Jarratt et
al., 2019; da Silva Estácio and Prikladnicki, 2015). Social sciences research
indicates that one’s behavior of an individual is affected by the behavior of
their peers (Eckles et al., 2016). Therefore, implicit gender bias based on
perception of peers may have effects on one’s behavior, potentially
influencing pair programming experience.

In this work, in a non–colocated (i.e. remote) environment in which the gender
of the peers cannot be directly observed, our goal is to explore whether
Software Engineering students change their behavior when the _perceived_
gender of their remote pair programming partners changes from man to woman or
vice versa. Note that, while we recognize that many students may identify as
neither men nor women, our initial exploration focuses primarily on
interactions between students who identify as one of these, so that we can
better align our findings with the existing literature on implicit gender
bias. The potential biases in interactions involving gender-fluid, non-gender-
conforming, or nonbinary students is a rich and complex topic deserving its
own subsequent study.

|
---|---
Figure 1. twincode user interface for control group (left) and experimental
group (right)

To achieve our goal, we plan to search for differences not only in the
perceived productivity of pair programming compared to solo programming, the
partner’s perceived technical competency compared to their own, and the
partners’ perceived skill level, but also in the interaction behavior, i.e.
the frequency of source code additions, deletions, validations, etc., and the
type and relative frequency of dialog messages used for collaborative
behavior.

To get early feedback on the infrastructure supporting our proposal, we ran
two pilot studies, one at each university, with a limited number of students,
where we could check the comprehensibility of the questionnaires used to
gather subjective data, the applicability of the message tagging (described in
Section 2), and the capabilities of the twincode platform, which is briefly
described below.

###  1.1. The twincode platform

To support our study, we have developed the twincode remote pair programming
platform, which manages the registration of students, the random allocation to
gender-balanced groups, the random allocation into pairs, the random
assignment of programming exercises to pairs, and the automatic collection of
interaction metrics and dialog messages.

As shown in Figure 1, twincode offers a source code editor where the students
concurrently develop the solution to a proposed exercise and can validate it
against several test cases. It also offers a chat window, where they can
collaborate to solve the exercise. Note that a gendered avatar is displayed
for the student in the experimental group only (right), but not for the one in
the control group (left).

###  1.2. Related Work

Reference |  Object of study |  Metrics |  Findings
---|---|---|---
Katira et al. (Katira et al., 2005) |  Compatibility of student pair programmers |  Web–based peer evaluation survey that required the students to evaluate the contributions of their partner and the pair compatibility as perceived by the student |  Students are compatible with partners whom they perceive of similar skill. Mixed-gender pairs are less likely to report compatibility.
Choi et al. (Choi, 2015) |  PP gender combinations |  Productivity, quality of source code, compatibility and communication between pairs |  Significant differences in the levels of pair compatibility and communication between the same gender pair type, woman–woman and man–man.
Gómez et al. (Gómez et al., 2017) |  PP gender combinations |  Productivity |  Similar productivity rates for the three gender pair combinations. The programming assignments had a significant impact on the productivity. Greater variability of productivity rates with mixed gender pairs (man–woman) was observed.
Jarrat et al. (Jarratt et al., 2019) |  PP gender combinations |  Weekly attendance, work accomplished during lab and perceptions of productivity |  Students who were randomly assigned a woman as a partner (rather than a man as a partner) attended class more often, were more confident that the solution was correct, and more confident in the finished product that they created. However, being assigned a woman as a partner was also associated with completing a smaller percentage of the assignment.
Table 1. Empirical studies about gender in pair programming

Several literature reviews (Salleh et al., 2010; Hanks et al., 2011; Kaur
Chahal et al., 2021) have compiled the empirical research on using pair
programming in higher education, with (da Silva Estácio and Prikladnicki,
2015) being focused on distributed pair programming from a teaching
perspective. By means of controlled experiments, remote and co–located pair
programming are compared in (Stotts et al., 2003; Al-Jarrah and Pontelli,
2016), showing comparable results. In most of the cases, the analyzed
variables are related to performance in terms of time, quality, or code tests
passed. Students perceptions have been also analyzed in terms of confidence,
satisfaction, motivation, or personality (Salleh et al., 2014).

Table 1 summarizes the empirical studies on the influence of gender in pair
programming, including findings such as (i) same–gender pairs are more
“democratic”; (ii) women working in pairs were more confident than those
working solo; and (iii) in mixed-gender, pairing women particularly do not
benefit (Kaur Chahal et al., 2021). Although such studies reveal that gender
seems to be a key factor in pair programming, none of them study gender bias
in pair programming.

Many factors other than gender may affect the outcomes of remote programming
sessions (Chaparro et al., 2005; Thomas et al., 2003). Previous research on
productive pairing looked at factors such as skill levels, autonomy in
choosing one’s partner (Xinogalos et al., 2017), and different personalities
(Hannay et al., 2010). Nevertheless, the work on gender composition of pairs
found conflicting results about whether same-gender or mixed-gender pairings
are more effective (Choi, 2015, 2013; Hofer, 2015; Kaur Kuttal et al., 2019).
One possible explanation is that gender correlates with other dimensions that
may affect the pairs’ collaboration, but these correlations may vary between
different environments. For example, women in a class may, on average, have
higher skill level than men because they had to face more societal barriers to
enter the class. On the other hand, they may, on average, have lower skill
level if women with no background are more actively recruited.

##  2\. Research Questions

Our study is based on the hypothesis that gender bias will lead to observable
differences based on subjects’ perceptions of the gender of their partners,
i.e., they will score men and women differently for similar tasks and also
behave differently depending on the perceived gender of their partner. To
study our hypothesis, we plan to apply methodological triangulation (Denzin,
2006), using several methods to collect data and approaching a complex
phenomenon like human behavior from more than one standpoint (Cohen et al.,
2018). In our case, three different data sources will be used: questionnaires
completed by the subjects, data collected automatically by the twincode
platform, and data produced by two different experimenters analyzing and
tagging dialog messages and cheking interrater agreement using Cohen’s kappa
coefficient (Landis and Koch, 1977).

With respect to the data collected using questionnaires, our research
questions are:

  1. RQ1

In remote pair programming, does gender bias affect perceived productivity
compared to solo programming?

  2. RQ2

In remote pair programming, does gender bias affect the partner’s perceived
technical competency compared to one’s own technical competency?

  3. RQ3

In remote pair programming, does gender bias affect how partners’ skills are
perceived?

With respect to the data automatically collected by the twincode
platform—which could be increased in the future—our research question is:

  1. RQ4

In remote pair programming, does gender bias affect the frequencies or
relative frequencies with which each partner produces source code additions,
source code deletions, successful validations, failed validations, and dialog
messages?

Tag |  Description |  Examples
---|---|---
I |  Informal |  _LOL! Hahaha!_
F |  Formal |  All messages except informal
S |  Statement of information or explanation |  _We need to create a program for kids to learn math_
U |  Opinion or indication of uncertainty |  _Unsure how to add strings together_
D |  Explicit instruction |  _Wait put the if back_
SU |  Polite or indirect instruction |  _Maybe we can do if user choice = +_
ACK |  Acknowledgement |  _Oh ok gotcha_
M |  Meta–comment or reflection |  _Hmmm_
QYN |  Yes/no question |  _Can the answer be negative?_
QWH |  Wh– question (who, what, where, when, why, and how) |  _How do I take in their input?_
AYN |  Answer to yes/no question |  _Yea_
AWH |  Answer to wh– question |  _The program should be able to generate erroneous questions_
FP |  Positive task feedback |  _Oh nice_
FNON |  Non–positive task feedback |  _Thats weird_
O |  Off–task |  _Wow its sweet in this room_
Table 2. Dialogue tags from (Rodríguez et al., 2017) augmented with orthogonal
informal/formal tags

The manual semantic tagging of the dialog messages classifies each message
into two orthogonal dimensions. The first dimension uses the 13 tags proposed
in (Rodríguez et al., 2017) (tags from S to O in Table 2). The second
dimension classifies each message as formal or informal. With respect to this
data source, our research questions are:

  1. RQ5

In remote pair programming, does gender bias affect the frequency or relative
frequency of the different types of dialog messages?

  2. RQ6

In remote pair programming, does gender bias affect the relative frequency of
formal and informal dialog messages?

##  3\. Variables

In this section, we describe all the variables we will consider in our study.
Note that depending on the development of the twincode platform, more
automatically measured dependent variables could be added in the future.

When used, abbreviations are enclosed in parentheses after variables’ names.

###  3.1. Independent Variables

group:

nominal factor representing the group (experimental or control) subjects are
randomly allocated to.

time:

nominal factor representing the moment (t1 and t2) in which the first and
second in–pair tasks are performed by the subjects.

induced partner’s gender (ipgender):

nominal factor representing the induced partner’s binary gender (man or woman
for the experimental group, and none for the control group, in which gender is
not revealed) during the in–pair tasks. This variable, which is directly
related to the gender bias treatment, is operationalized by means of the
instructions provided at the beginning of the tasks (“…work with your partner.
She is …”) and by the gendered avatar that is visible during the in–pair tasks
for the experimental group and that is swapped between tasks. Subjects in the
control group receive no treatment, i.e., they do not see any information
about the gender of their partners in any way, neither textual nor graphical.

gender:

nominal factor representing subject’s gender, which may be man, woman, or any
other option as freely expressed in the demographic form during registration.

###  3.2. Dependent Variables

The response variables measured using questionnaires containing 0–10 linear
numerical response items are the following:

Perceived productivity compared to solo programming (pp):

interval variable measuring the subject’s perceived productivity compared to
solo programming after each in–pair task (see RQ1). Low values correspond to
better solo programming productivity, i.e., “solo programming would have been
more productive than pair programming”, whereas high values correspond to
better pair programming productivity, i.e. “pair programming has been more
productive than solo programming”.

Perceived partner’s technical competency compared to their own (pptc):

interval variable measuring the subject’s partner’s perceived technical
competency compared to their own after each in–pair task (see RQ2). Low values
correspond to higher subject’s productivity, i.e., “I have been more
productive than my partner”, whereas higher values correspond to high
partner’s productivity, i.e. “My partner has been more productive than me”.

Compared partners’ skills (cps):

interval variable measuring whether the subject perceived better skills in
their first or second partner in the in–pair tasks (see RQ3). Low values
correspond to the first partner, i.e., “My first partner was a better partner
than my second partner”, whereas high values correspond to the second partner,
i.e. “My second partner was a better partner than my first partner”. In the
case of the experimental group only, this variable is transformed after
collection using an R script in such a way that low values correspond to the
partner perceived as a man, and high values to the partner perceived as a
woman, in order to analyze whether there is a gender bias in the scoring.

Apart from the variables described above, the questionnaires will also include
questions about the perceived gender of their parners at each task. The
corresponding variable is described below:

perceived partner’s gender (ppgender):

nominal factor representing the subject’s perception of their partner’s gender
(woman, man, I don’t know, or I don’t remember) at each in–pair task.

The response variables automatically collected by the twincode platform and
related to the interaction during the in–pair programming exercises (see RQ4)
are listed below. Every variable v represents a frequency, i.e., a count, and
its associated relative frequency is computed with respect to the the sum of
the frequencies of the two subjects in a pair. For example, let us suppose
that subjects $i$ and $j$ are the two members of a pair, and vi and vj are the
corresponding values of the v variable. In this case, the relative frequencies
for each subject would be $\textsf{v}_{i}\over{\textsf{v}_{i}+\textsf{v}_{j}}$
and $\textsf{v}_{j}\over{\textsf{v}_{i}+\textsf{v}_{j}}$, respectively.

source code additions (sca):

Ratio scale variable representing the count of characters added by a subject
to the source code window during an in–pair task.

source code deletions (scd):

Ratio scale variable representing the count of characters deleted by a subject
from the source code window during an in–pair task.

successful validations (okv):

Ratio scale variable representing the count of successful validations of the
source code performed by a subject during an in–pair task.

unsuccessful validations (kov):

Ratio scale variable representing the count of unsuccessful validations of the
source code performed by a subject during an in–pair task.

dialog messages (dm):

Ratio scale variable representing the count of dialog messages sent by a
subject during an in–pair task.

The response variables related to the manual tagging of the dialog messages
(see RQ5 and RQ6) correspond to the tags in Table 2 and are listed below.
Every variable represents a frequency, i.e., a count, and its associated
relative frequency is computed with respect to the number of dialog messages
generated by the subject during an in–pair task, which is defined by the dm
variable specified above.

i:

Ratio scale variable representing the count of informal messages generated by
a subject during an in–pair task.

f:

Ratio scale variable representing the count of non–informal, i.e. formal,
messages generated by a subject during an in–pair task.

s:

Ratio scale variable representing the count of statement of information or
explanation messages generated by a subject during an in–pair task.

u:

Ratio scale variable representing the count of opinion or indication of
uncertainty messages generated by a subject during an in–pair task.

d:

Ratio scale variable representing the count of explicit instruction messages
generated by a subject during an in–pair task.

su:

Ratio scale variable representing the count of polite or indirect instruction
messages generated by a subject during an in–pair task.

ack:

Ratio scale variable representing the count of acknowledgment messages
generated by a subject during an in–pair task.

m:

Ratio scale variable representing the count of meta–comment or reflection
messages generated by a subject during an in–pair task.

qyn:

Ratio scale variable representing the count of yes/no question messages
generated by a subject during an in–pair task.

qwh:

Ratio scale variable representing the count of wh- question (who, what, where,
when, why, and how) messages generated by a subject during an in–pair task.

ayn:

Ratio scale variable representing the count of answer to yes/no question
messages generated by a subject during an in–pair task.

awh:

Ratio scale variable representing the count of answer to wh- question messages
generated by a subject during an in–pair task.

fp:

Ratio scale variable representing the count of positive task feedback messages
generated by a subject during an in–pair task.

fnon:

Ratio scale variable representing the count of non–positive task feedback
messages generated by a subject during an in–pair task.

o:

Ratio scale variable representing the count of off–task messages generated by
a subject during an in–pair task.

###  3.3. Confounding Variables

The confounding variables that will be controlled during the experiment are
the following.

Subject’s technical skills:

To control the variability caused by each subject on their partner, pairs are
kept the same during the entire experiment, although the subjects are not
informed about this fact until the end of the experiment. Ideally, this would
make the conditions of the two in–pair tasks the same except for the
programming exercises (see below) and for the perceived gender in the case of
the experimental group.

Programming exercises:

In order to avoid potential differences among the programming exercises used
during in–pair tasks, they are of similar difficulty and are randomly
assigned.

##  4\. Participants

As we have done in the pilot studies, our plan is offering our students the
possibility to participate in the twincode study. At the University of
Seville, the participants will be 3${}^{\text{rd}}$–year students of the
Degree in Software Engineering. We expect the number of participants to be
around 150–200.

At the University of California Berkeley, the participants will be students
enrolled in the 1${}^{\text{st}}$ and 2${}^{\text{nd}}$ semester Computer
Science courses. In this case, we expect the number of participants to be
around 100–300.

##  5\. Execution Plan

We plan to perform a baseline experiment at the University of Seville in the
first semester of the 2021–2022 academic course, and then, replicate the
experiment at the University of California Berkeley at the beginning of the
second semester. At each location, the experimental material provided to the
students will be localized, i.e., in Spanish in Spain and in English at the
USA, and the translation will be carefully checked by bilingual
experimenters.111At the time of writing, discussions are underway for another
replication at the Northern Technical University at Ecuador. If this
replication is eventually carried out, the experimental material will also be
adapted to the local variant of Spanish if deemed necessary.

Independently of being the baseline or the replication, the execution plan of
the twincode study consists of the steps described below.

###  5.1. Recruitment

In this initial step, we plan to motivate the students to voluntarily
participate in the study as an interesting experience in remote pair
programming but without mentioning that the main goal is to study the effect
of gender bias. We also remark that for the purpose of the study, they must
remain anonymous to their partners, so they must neither mention nor ask any
personal information.

The interested students must register in the twincode platform providing some
demographic data and accepting the participation conditions.

###  5.2. Training

One week or so before the experiment execution, we plan to provide a short
training session about the twincode platform, so students become familiar with
it and any concerns about how to use the platform were dispelled, thus
reducing any potential anxiety for using a new platform.

Figure 2. Experimental process (subject allocation to groups)

###  5.3. Experiment Execution

For experiment execution, all registered students must log into the twincode
platform, which will automatically allocate them into the control and
experimental groups. This allocation is random and gender–balanced, i.e., the
proportion of men and women in each group is as close as possible. Once all
students are allocated to groups, they are randomly allocated into
control–experimental pairs, as shown in Figure 2.

After subject allocation, the pairs are presented a programming exercise that
they must solve collaboratively (labeled as Task #1 in Figure 3) in the
twincode platform. They are given 20 minutes to do it, and in case they
finished earlier, another exercise of increasing difficulty is presented. At
the end of the 20–minute period, they are asked to individually fill in a
questionnaire about the perceived productivity compared to solo programming
and about the perceived partner’s technical competency compared to their own.
They are given 10 minutes to fill in the questionnaire.

After filling the first questionnaire, the students are presented another
programming exercise to be solved individually in 10 minutes (Task #2 in
Figure 3). As in the previous task, another exercise of increasing difficulty
is presented if they finish earlier. The main purpose of this individual task
is to make students forget about their first partners, i.e. their way of
writing dialog messages or source code, so they do not recognize them in the
second in–pair task.

After the individual task, pairs are presented again a new collaborative
programming exercise that they must solve in similar conditions to the
exercise in Task #1. In this second in–pair exercise, the avatar gender (which
in this initial study will always indicate binary gender) is swapped with
respect to the first exercise for the subjects in the experimental group. Note
that pairs are kept the same in order to reduce the variability due to the
subjects themselves, which could possibly have had a confounding effect in
case of a new pair allocation for Task #3.

Once Task #3 is finished, students are asked to fill in the same questionnaire
than they filled after Task #1, and another questionnaire comparing the
perceived genders and skills of the first and second partners. They are given
15 minutes for responding both questionnaires.

Finally, they are informed about the actual goal of the study and that the
pairs remained the same during the experiment. At this moment, they are
allowed to withdraw their data if they wish to do so.

###  5.4. Data Analysis

Figure 3. Experimental process (tasks)

During the manual tagging of the dialog messages, all pairs in which the
gender of any of the peers is disclosed in any way are excluded from the data
analysis. Then, before analyzing response variables, the internal consistency
of the questionnaire data is checked using Cronbach’s alpha and Kaiser
criterion. After that, for every dependent variable v, we compute the distance
between the two in–pair tasks as the absolute value of the difference, i.e. —
v(t2) – v(t1) —. Ideally, this distance should be lower for the students in
the control group (no information about partners’ genders) than for those in
the experimental group (with two different perceived partners’ genders at t1
and t2). Therefore, for every variable, we perform a t–test to detect distance
differences between the groups.

Then, using the data from the experimental group only, we perform a t–test to
detect differences in the scores of every dependent variable between perceived
partner’s gender, i.e. to detect differences in the scores when partners are
perceived as men vs. as women. Finally, to detect a potential interaction
between the perceived partner’s gender and the subject’s gender, we perform a
mixed–model ANOVA with the perceived gender as a within–subjects variable and
subject’s gender as a between–subjects variable. As complementary analyses, we
also study (i) the correlation between the induced and the perceived gender
for the subjects in the experimental group, and the distribution of the
perceived gender (if any) in the control group; and (ii) the potential
cultural impact of the different locations at which the experiment is carried
out.

All the data analysis will be performed using R scripts, that will be
available in a public repository together with the datasets in the
corresponding laboratory package.

###### Acknowledgements.

We would like to thank the students who volunteered to participate in the
pilot studies at the University of California, Berkeley and the University of
Seville. We particularly acknowledge Vron Vance’s assistance regarding
inclusive language around gender identity. This work has been partially
supported by FEDER/Ministerio de Ciencia e Innovación – Agencia Estatal de
Investigación under projects OPHELIA (RTI2018–101204–B–C22), HORATIO
(RTI2018-101204–B–C21), and Junta de Andalucía under project EKIPMENT-PLUS
(P18–FR–2895).

## References

  * (1)
  * Al-Jarrah and Pontelli (2016) Ahmad Al-Jarrah and Enrico Pontelli. 2016.  On the effectiveness of a collaborative virtual pair-programming environment. In _International Conference on Learning and Collaboration Technologies_. Springer, 583–595.
  * Chaparro et al. (2005) E. A. Chaparro, Aybala Yuksel, Pablo Romero, and Sallyann Bryant. 2005.  Factors Affecting the Perceived Effectiveness of Pair Programming in Higher Education. In _Proceedings of the 17th Workshop of the Psychology of Programming Interest Group_.
  * Choi (2013) Kyungsub S. Choi. 2013\.  Evaluating Gender Significance within a Pair Programming Context. In _Proceedings of the 2013 46th Hawaii International Conference on System Sciences_ _(HICSS ’13)_. IEEE Computer Society, USA, 4817–4825.  https://doi.org/10.1109/HICSS.2013.209
  * Choi (2015) Kyungsub Stephen Choi. 2015\.  A comparative analysis of different gender pair combinations in pair programming.  _Behaviour & Information Technology_ 34, 8 (2015), 825–837.  https://doi.org/10.1080/0144929X.2014.937460 arXiv:https://doi.org/10.1080/0144929X.2014.937460
  * Cohen et al. (2018) Louis Cohen, Lawrence Manion, and Keith Morrison. 2018\.  _Research Methods in Education_ (8th ed.).  Routledge.
  * da Silva Estácio and Prikladnicki (2015) Bernardo José da Silva Estácio and Rafael Prikladnicki. 2015.  Distributed Pair Programming: A Systematic Literature Review.  _Information and Software Technology_ 63 (2015), 1–10.
  * Denzin (2006) N.K. Denzin. 2006\.  _Sociological Methods: A Sourcebook_ (5th ed.).  Aldine Transaction.
  * Eckles et al. (2016) Dean Eckles, René F. Kizilcec, and Eytan Bakshy. 2016\.  Estimating peer effects in networks with peer encouragement designs.  _Proceedings of the National Academy of Sciences_ 113, 27 (2016), 7316–7322.  https://doi.org/10.1073/pnas.1511201113 arXiv:https://www.pnas.org/content/113/27/7316.full.pdf
  * Gómez et al. (2017) O. Gómez, M. Solari, C. Calvache, and A. Ledezma-Carrizalez. 2017.  A Controlled Experiment on Productivity of Pair Programming Gender Combinations: Preliminary Results. In _Proceedings of the XX Ibero–American Conference on Software Engineering_. 197–210.
  * Hanks et al. (2011) Brian Hanks, Sue Fitzgerald, Renée McCauley, Laurie Murphy, and Carol Zander. 2011\.  Pair programming in education: A literature review.  _Computer Science Education_ 21, 2 (2011), 135–173.
  * Hannay et al. (2010) Jo E. Hannay, Erik Arisholm, Harald Engvik, and Dag I.K. Sjoberg. 2010.  Effects of Personality on Pair Programming.  _IEEE Transactions on Software Engineering_ 36, 1 (2010), 61–80.  https://doi.org/10.1109/TSE.2009.41
  * Hofer (2015) Sarah I. Hofer. 2015\.  Studying Gender Bias in Physics Grading: The role of teaching experience and country.  _International Journal of Science Education_ 37, 17 (2015), 2879–2905.  https://doi.org/10.1080/09500693.2015.1114190 arXiv:https://doi.org/10.1080/09500693.2015.1114190
  * Jarratt et al. (2019) Lindsay Jarratt, Nicholas A. Bowman, K.C. Culver, and Alberto Maria Segre. 2019.  A Large-Scale Experimental Study of Gender and Pair Composition in Pair Programming. In _Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education_ (Aberdeen, Scotland Uk) _(ITiCSE ’19)_. Association for Computing Machinery, New York, NY, USA, 176–181.  https://doi.org/10.1145/3304221.3319782
  * Katira et al. (2005) N. Katira, L. Williams, and J. Osborne. 2005.  Towards increasing the compatibility of student pair programmers. In _Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005._ 625–626.  https://doi.org/10.1109/ICSE.2005.1553618
  * Kaur Chahal et al. (2021) K. Kaur Chahal, A. Kaur, and M. Saini. 2021.  _Research and Evidence in Software Engineering: From Empirical Studies to Open Source Artifacts_.  Taylor & Francis Group, Chapter Empirical Studies on Using Pair Programming as a Pedagogical Tool in Higher Education Courses: A Systematic Literature Review, 251–287.
  * Kaur Kuttal et al. (2019) Sandeep Kaur Kuttal, Kevin Gerstner, and Alexandra Bejarano. 2019\.  Remote Pair Programming in Online CS Education: Investigating through a Gender Lens. In _2019 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)_. 75–85.  https://doi.org/10.1109/VLHCC.2019.8818790
  * Landis and Koch (1977) J.R. Landis and G.G. Koch. 1977.  The Measurement of Observer Agreement for Categorical Data.  _Biometrics_ 33, 1 (1977), 159–174.
  * Martell et al. (1996) Martell, D. M. Lane, and C. Emrich. 1996.  Male-female differences: A computer simulation.  _American Psychologist_ 2, 51 (7 1996), 157–158.  https://doi.org/10.1037/0003-066X.51.2.157
  * Rodríguez et al. (2017) Fernando J. Rodríguez, Kimberly Michelle Price, and Kristy Elizabeth Boyer. 2017.  Exploring the Pair Programming Process: Characteristics of Effective Collaboration. In _Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education_ (Seattle, Washington, USA) _(SIGCSE ’17)_. Association for Computing Machinery, New York, NY, USA, 507–512.  https://doi.org/10.1145/3017680.3017748
  * Salleh et al. (2014) Norsaremah Salleh, Emilia Mendes, and John Grundy. 2014\.  Investigating the effects of personality traits on pair programming in a higher education setting through a family of experiments.  _Empirical Software Engineering_ 19, 3 (2014), 714–752.
  * Salleh et al. (2010) Norsaremah Salleh, Emilia Mendes, John Grundy, and Giles St J Burch. 2010.  The effects of neuroticism on pair programming: an empirical study in the higher education context. In _Proceedings of the 2010 ACM-IEEE international symposium on empirical software engineering and measurement_. 1–10.
  * Stotts et al. (2003) David Stotts, Laurie Williams, Nachiappan Nagappan, Prashant Baheti, Dennis Jen, and Anne Jackson. 2003\.  Virtual teaming: Experiments and experiences with distributed pair programming. In _Conference on Extreme Programming and Agile Methods_. Springer, 129–141.
  * Thomas et al. (2003) Lynda Thomas, Mark Ratcliffe, and Ann Robertson. 2003\.  Code Warriors and Code-a-Phobes: A Study in Attitude and Pair Programming. In _Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education_ (Reno, Navada, USA) _(SIGCSE ’03)_. Association for Computing Machinery, New York, NY, USA, 363–367.  https://doi.org/10.1145/611892.612007
  * Werner et al. (2004) Linda L. Werner, Brian Hanks, and Charlie McDowell. 2004\.  Pair-Programming Helps Female Computer Science Students.  _J. Educ. Resour. Comput._ 4, 1 (March 2004), 4–es.  https://doi.org/10.1145/1060071.1060075
  * Xinogalos et al. (2017) S. Xinogalos, M. Satratzemi, A. Chatzigeorgiou, and D. Tsompanoudi. 2017.  Student perceptions on the benefits and shortcomings of distributed pair programming assignments.  _2017 IEEE Global Engineering Education Conference (EDUCON)_ (2017), 1513–1521.


# Learning Compact Features via In-Training Representation Alignment

Xin Li, Xiangrui Li, Deng Pan, Yao Qiang, Dongxiao Zhu

###### Abstract

Deep neural networks (DNNs) for supervised learning can be viewed as a
pipeline of the feature extractor (i.e., last hidden layer) and a linear
classifier (i.e., output layer) that are trained jointly with stochastic
gradient descent (SGD) on the loss function (e.g., cross-entropy). In each
epoch, the true gradient of the loss function is estimated using a mini-batch
sampled from the training set and model parameters are then updated with the
mini-batch gradients. Although the latter provides an unbiased estimation of
the former, they are subject to substantial variances derived from the size
and number of sampled mini-batches, leading to noisy and jumpy updates. To
stabilize such undesirable variance in estimating the true gradients, we
propose In-Training Representation Alignment (ITRA) that explicitly aligns
feature distributions of two different mini-batches with a matching loss in
the SGD training process. We also provide a rigorous analysis of the desirable
effects of the matching loss on feature representation learning: (1)
extracting compact feature representation; (2) reducing over-adaption on mini-
batches via an adaptive weighting mechanism; and (3) accommodating to multi-
modalities. Finally, we conduct large-scale experiments on both image and text
classifications to demonstrate its superior performance to the strong
baselines.

## Introduction

Recently, deep neural networks (DNNs) have achieved remarkable performance
improvements in a wide range of challenging tasks in computer vision (He et
al. 2016; Huang et al. 2019; Pan, Li, and Zhu 2021; Qiang et al. 2022a),
natural language processing (Sutskever, Vinyals, and Le 2014; Chorowski et al.
2015; Qiang et al. 2022b) and healthcare informatics (Miotto et al. 2018; Li,
Zhu, and Levy 2020). For supervised learning, DNNs can be viewed as a feature
extractor followed by a linear classifier on the latent feature space, which
are jointly trained using stochastic gradient descent (SGD). Specifically, in
each iteration of SGD, a mini-batch of $m$ samples
$\\{(x_{i},y_{i})\\}_{i=1}^{m}$ is sampled from the training data
$\\{(x_{i},y_{i})\\}_{i=1}^{n}(n>m)$. The gradient of loss function
$L(x,\theta)$ is calculated on the mini-batch, and network parameter $\theta$
is updated via one step of gradient descent (learning rate $\alpha$):

$\begin{split}&\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}L(x_{i},\theta)\approx\frac{1}{m}\sum_{i=1}^{m}\nabla_{\theta}L(x_{i},\theta),\\\
&\theta\leftarrow\theta-\alpha\cdot\frac{1}{m}\sum_{i=1}^{m}\nabla_{\theta}L(x_{i},\theta).\end{split}$
(1)

This update in Eq.(1) can be interpreted from two perspectives. First, from
the conventional approximation perspective, the true gradient of the loss
function (i.e., gradient on the entire training data) is approximated by the
mini-batch gradient. As each mini-batch gradients are unbiased estimators of
the true gradient of the loss function and the computation is inexpensive,
large DNNs can be efficiently and effectively trained with modern computing
infrastructures. Second, Eq. (1) can also be interpreted as an exact gradient
descent update on the mini-batch. In other words, SGD updates network
parameters $\theta$ to achieve maximum improvement in fitting the mini-batch.
As each mini-batch is often uniformly sampled from each class of the training
data, such exact update inevitably introduces the undesirable variance in
gradients calculation via backpropagation, resulting in the over-adaption of
model parameters to that mini-batch.

Figure 1: A comparison of ITRA and vanilla SGD training on the CIFAR10 testing
data. Left: normalized distance between samples of the same class from
different mini-batches used in training; middle: testing accuracy; right:
testing cross-entropy loss. The model is Resnet18.

A natural question then to ask is, “can we reduce the over-adaption to mini-
batches?”, to reduce the mini-batch dependence on SGD update in Eq. (1). In
this paper, we propose In-Training Representation Alignment (ITRA) that aims
at reducing the mini-batch over-adaption by aligning feature representation of
different mini-batches that is learned by the feature extractor in SGD. Our
motivation for feature alignment is: if the SGD update using one mini-batch A
is helpful for DNNs learning good feature representations with respect to the
entire data, then for another mini-batch B, their feature representation
should align well with each other. In this way, we can reduce mini-batch over-
adaption by forcing accommodation of SGD update to B and reducing dependence
of the parameter update on A. Ideally, if the distribution $P(h)$ of latent
feature $h$ is known as a prior, we could explicitly match the mini-batch
feature $h_{\text{mb}}$ with $P(h)$ via maximum likelihood. However, in
practice, $P(h)$ is not known or does not even have an analytic form. To
achieve this, we utilize the maximum mean discrepancy (MMD) (Gretton et al.
2012) from statistical hypothesis testing for the two-sample problem. MMD is
differentiable that can be trained via back propagation. Moreover, we show in
an analysis that the gradient of MMD enjoys several good theoretical merits.
Based on the analysis, ITRA reduces SGD update adaption to the mini-batch by
implicitly strengthening the supervision signal of high-density samples via an
adaptive weighting mechanism (see details in Section 4), where high-density
samples are closely clustered to form modalities for each class.

To check effect of gradient update on feature representation learning, an
illustrative example is presented in Figure 1. The model is Resnet18 with BN
layers trained with cross-entropy (CE) loss. We calculate the distance between
a pair of same-class samples from two mini-batches respectively and plot the
normalized distance in the left panel of Figure 1, after model training
stabilizes and achieves relatively good performance. We see that when model is
trained only with CE loss in vanilla SGD, the distance stabilizes while the
training makes progresses. This is due to that after the model capturing the
classification pattern for each class, vanilla SGD adapts to mini-batch
samples to achieve gain for the loss function yet does not further encourage
feature alignment to learn compact feature representations. Hence, vanilla SGD
has little effect on the compactness of feature representations. However, in
ITRA, the distance between a pair of samples keeps decreasing. This implies
ITRA indeed helps DNN to learn more compact feature representations by
aligning different mini-batches and thus achieves higher accuracy and lower
loss (Figure 1 middle and right panels).

We summarize our original contributions as follows. (1) We propose a novel and
general strategy ITRA for training DNNs. ITRA augments conventional SGD with
regularization by forcing feature alignment of different mini-batches to
reduce variance in estimating the true gradients using mini-batches. ITRA can
enhance the existing regularization approaches and is compatible with a broad
range of neural network architectures and loss functions. (2) We provide
theoretical analysis on the desirable effects of ITRA and explains why ITRA
helps reducing the over-adaption of vanilla SGD to the mini-batch. With MMD,
ITRA has an adaptive weighting mechanism that can help neural networks learn
more discriminative feature representations and avoid the assumption of uni-
modality on data distribution. Results on benchmark datasets demonstrate that
training with ITRA can significantly improve DNN performance, compared with
other state-of-the-art methods.

## Related Work

Modern architectures of DNNs usually have an extremely large number of model
parameters, which often outnumbers the available training data. To reduce
overfitting in training DNNs, regularizations are needed. Those regularization
methods include classic ones such as $L_{1}/L_{2}$-norm penalties and early
stopping (Li, Zhu, and Dong 2018; Li and Zhu 2018). For deep learning,
additional useful approaches are proposed motivated by the SGD training
dynamics (Li et al. 2022). For example, dropout (Srivastava et al. 2014) and
its variants (Gao, Pei, and Huang 2019; Ghiasi, Lin, and Le 2018) achieve
regularization by reducing the co-adaption of hidden neurons of DNNs. (Ioffe
and Szegedy 2015) proposes batch normalization (BN) to reduce the internal
covariate shift caused by SGD. For image classification, data-augmentation
types of regularization are also developed (DeVries and Taylor 2017; Gastaldi
2017; Li et al. 2020, 2021). Different from those approaches, our proposed
ITRA is motivated by the perspective of exact gradient update for each mini-
batch in SDG training, and achieves regularization by encouraging the
alignment of feature representations of different mini-batches. Those methods
are compatible with ITRA for training DNNs and hence can be applied in
conjunction with ITRA.

Another line of regularization are loss function based that the supervision
loss is augmented with other penalties under different considerations. One
example is label smoothing (Szegedy et al. 2016), which corrupts the true
label with a uniformly-distributed noise to discourage DNNs’ over-confident
predictions for training data. The work that is closest to ours is Center loss
(Wen et al. 2016), which reduces the intra-class variation by aligning feature
of each class to its “center”. With the assumption of distribution uni-
modality for each class, it explicitly encourages the feature representations
clustering around its center. However, this assumption may be too strict since
true data distribution is generally unknown and can be multi-model. On the
contrary, ITRA reduces variances and encourages intra-class compactness by
aligning a pair of features from two minibatches to each other, which avoids
the distribution assumption and is accommodating to multi-modalities.

To match the distribution of features learned from different mini-batches,
ITRA uses MMD as its learning objective. MMD (Gretton et al. 2007, 2012) is a
probability metric for testing whether two finite sets of samples are
generated from the same distribution. Using a universal kernel (i.e., Gaussian
kernel), minimizing MMD encourages to match all moments of the empirical data
distribution. MMD has been widely applied in many machine learning tasks. For
example, (Li, Swersky, and Zemel 2015) and (Li et al. 2017) use MMD to train
unsupervised generative models by matching the generated distribution with the
data distribution. Another application of MMD is for the domain adaption. To
learn domain-invariant feature representations, (Long et al. 2015) uses MMD to
explicitly match feature representations from different domains. There are
also other probability-based distance metrics applied in domain adaption such
as $\mathcal{A}$-divergence (Ben-David et al. 2007) and Wasserstein distance
(Shen et al. 2018). However, these metrics are non-differentiable while the
differentiability of MMD enables the adaptive weighting mechanism in ITRA.
Moreover, our goal is different from those applications. In ITRA, we do not
seek exact distribution matching. Instead, we use class-conditional MMD as a
regularization to improve SGD training.

## Preliminary: Maximum Mean Discrepancy

Given two finite sets of samples $S_{1}=\\{x_{i}\\}_{i=1}^{n}$ and
$S_{2}=\\{y_{i}\\}_{i=1}^{m}$, MMD (Gretton et al. 2007, 2012) is constructed
to test whether $S_{1}$ and $S_{2}$ are generated from the same distribution.
MMD compares the sample statistics between $S_{1}$ and $S_{2}$, and if the
discrepancy is small, $S_{1}$ and $S_{2}$ are then likely to follow the same
distribution.

Using the kernel trick, the empirical estimate of MMD (Gretton et al. 2007)
w.r.t. $S_{1}$ and $S_{2}$ can be rewritten as:

$\begin{split}\text{MMD}(S_{1},S_{2})=&\big{[}\frac{1}{n^{2}}\sum_{i,j=1}^{n}\mathcal{K}(x_{i},x_{j})+\frac{1}{m^{2}}\sum_{i,j=1}^{m}\mathcal{K}(y_{i},y_{j})\\\
&-\frac{2}{mn}\sum_{i=1}^{n}\sum_{j=1}^{m}\mathcal{K}(x_{i},y_{j})\big{]}^{1/2},\end{split}$

where $\mathcal{K}(\cdot,\cdot)$ is a kernel function. (Gretton et al. 2007)
shows that if $\mathcal{K}$ is a characteristic kernel, then asymptotically
MMD = 0 if and only $S_{1}$ and $S_{2}$ are generated from the same
distribution. A typical choice of $\mathcal{K}$ is the Gaussian kernel with
bandwidth parameter $\sigma$:
$\mathcal{K}(x,y)=\exp(-\frac{||x-y||^{2}}{\sigma})$. With Gaussian kernel,
minimizing MMD is equivalent to matching all orders of moments of the two
datasets (Li, Swersky, and Zemel 2015).

## In-Training Representation Alignment

The Proposed ITRA The idea of ITRA is to reduce the DNN over-adaption to a
mini-batch if we view the SGD iteration as an exact update for that mini-
batch. In terms of feature learning, we attempt to train the feature extractor
to encode less mini-batch dependence into the feature representation. From the
distribution point of view, the latent feature distribution of the mini-batch
should approximately match with, or more loosely, should not deviate much from
that of the entire data. However, aligning a mini-batch with the global
statistics from entire data may not be available, we sample a pair of mini-
batch to match each other to reduce the variance. It is possible to sample
more mini-batches to further reduce variances but is computationally
expensive.

More formally, let $f_{\theta}(x)$ be a convolutional neural network model for
classification that is parameterized by $\theta$. It consists of a feature
extractor $h=E_{\theta_{e}}(x)$ and a linear classifier $C_{\theta_{c}}(h)$
parameterized by $\theta_{e}$ and $\theta_{c}$ respectively. Namely,
$f_{\theta}(x)=C_{\theta_{c}}(E_{\theta_{e}}(x))$ and
$\theta=\\{\theta_{e},\theta_{c}\\}$. Without ambiguity, we drop $\theta$ in
$f,E$ and $C$ for notational simplicity. In each iteration, let
$S_{(1)}=\\{(x_{i}^{(1)},y_{i}^{(1)})\\}_{i=1}^{m_{1}}$ be the mini-batch of
$m_{1}$ samples. Then the loss function using cross-entropy (CE) on $S_{(1)}$
can be written as

$L_{mb}(\theta)=-\frac{1}{m_{1}}\sum_{i=1}^{m_{1}}\log
f_{y_{i}^{(1)}}(x_{i}^{(1)}),$ (2)

where $f_{y_{i}^{(1)}}(x_{i}^{(1)})$ is the predicted probability for
$x_{i}^{(1)}$’s true label $y_{i}^{(1)}$. SGD performs one gradient descent
step on $L_{mb}$ w.r.t. $\theta$ using Eq. (1). To reduce $\theta$’s
dependence on $S_{1}$ in this exact gradient descent update, we sample from
the training data another mini-batch
$S_{(2)}=\\{(x_{i}^{(2)},y_{i}^{(2)})\\}_{i=1}^{m_{2}}$ to match the latent
feature distribution between $S_{(1)}$ and $S_{(2)}$ using MMD:

$\begin{split}&H_{(1)}=\\{h_{i}^{(1)}=E(x_{i}^{(1)}):i=1,\cdots,m_{1}\\},\\\
&H_{(2)}=\\{h_{i}^{(2)}=E(x_{i}^{(2)}):i=1,\cdots,m_{2}\\},\\\
&\text{Match}(\theta_{e};H_{(1)},H_{(2)})=\text{MMD}(H_{(1)},H_{(2)}).\end{split}$
(3)

Our proposed ITRA modifies the conventional gradient descent step in SGD by
augmenting the CE loss (Eq. (2)) with the matching loss, which justifies the
name of ITRA:

$\theta\leftarrow\theta-\alpha\nabla_{\theta}\big{[}L_{mb}(\theta)+\lambda\text{Match}(\theta_{e};H_{(1)},H_{(2)})\big{]},$
(4)

where $\lambda$ is the tuning parameter controlling the contribution of the
matching loss. Note that mini-batch $S_{(2)}$ is not used in the calculation
of cross-entropy loss $L_{mb}(\theta)$.

Class-conditional ITRA For classification tasks, we could also utilize the
label information and further refine the match loss as a sum of class-
conditional matching loss, termed as ITRA-c $(k=1,\cdots,K)$:

$\begin{split}&H_{(1)}^{k}=\\{h_{i}^{(1)}=E(x_{i}^{(1)}):y_{i}=k,i=1,\cdots,m_{1}\\}\\\
&H_{(2)}^{k}=\\{h_{i}^{(2)}=E(x_{i}^{(2)}):y_{i}=k,i=1,\cdots,m_{2}\\}\\\
&\text{Match}_{c}(\theta_{e};H_{(1)},H_{(2)})=\frac{1}{K}\sum_{k=1}^{K}\text{MMD}(H_{(1)}^{k},H_{(2)}^{k}),\end{split}$
(5)

where $K$ is the total number of classes and $y_{i}=k$ the true label of
sample $x_{i}$. The ITRA-c update is

$\theta\leftarrow\theta-\alpha\nabla_{\theta}\big{[}L_{mb}(\theta)+\lambda\text{Match}_{c}(\theta_{e};H_{(1)},H_{(2)})\big{]}.$
(6)

### Analysis on ITRA

On learning compact feature representations To further gain insight on the
desirable effects of ITRA on the SGD training procedure, we analyze the
matching loss at the sample level. With the same notation in Eq. (5), the
matching loss for class $k$ is

$M:=\text{Match}_{k}=\text{MMD}(H_{(1)}^{k},H_{(2)}^{k}).$

Since MMD is symmetric with respect to $H_{(1)}^{k}$ and $H_{(2)}^{k}$,
without loss of generality, we consider sample $x_{i}^{(1)}$ with its feature
representation $h_{i}^{(1)}=E(x_{i}^{(1)})$ from $H_{(1)}^{k}$ (but the CE
loss is not symmetric and only calculated on the first mini-batch $H_{(1)}$).
Then the gradient of matching loss with respect to $h_{i}^{(1)}$ is
(superscript (1) in $x_{i}^{(1)}$ and $h_{i}^{(1)}$ are dropped.)

$\begin{split}\nabla_{h_{i}}M=&\frac{1}{\sqrt{M}}\nabla_{h_{i}}\big{[}\frac{1}{m_{1}^{2}}\sum_{j=1}^{m_{1}}\mathcal{K}(h_{i},h_{j}^{(1)})\\\
&-\frac{2}{m_{1}m_{2}}\sum_{j=1}^{m_{2}}\mathcal{K}(h_{i},h_{j}^{(2)})\big{]}.\\\
\end{split}$

For Gaussian kernel $\mathcal{K}(x,y)$, its gradient with respect to $x$ is
$\nabla_{x}\mathcal{K}(x,y)=-2\exp(-\frac{||x-y||^{2}}{\sigma})\frac{x-y}{\sigma}$.
Note that $\sigma$ is data-dependent and treated as hyperparameter. Hence, it
is not back propagated in the training process and in practice set as the
median of sample pairwise distances (Gretton et al. 2012; Long et al. 2015;
Li, Swersky, and Zemel 2015). By the linearity of gradient operator, we have

$\begin{split}\nabla_{h_{i}}M=&-\frac{2}{\sqrt{M}}\big{[}\frac{1}{m_{1}^{2}}\sum_{j=1}^{m_{1}}\exp(-\frac{||h_{i}-h_{j}^{(1)}||^{2}}{\sigma})\frac{h_{i}-h_{j}^{(1)}}{\sigma}\\\
&-\frac{2}{m_{1}m_{2}}\sum_{j=1}^{m_{2}}\exp(-\frac{||h_{i}-h_{j}^{(2)}||^{2}}{\sigma})\frac{h_{i}-h_{j}^{(2)}}{\sigma}\big{]}.\end{split}$
(7)

We notice that for function $g_{a}(x)=\exp(-x^{2}/a)x/a$ ($a$ is some
constant), $g_{a}(x)\rightarrow 0$ exponentially as $x\rightarrow\infty$.
Hence, for fixed $\sigma$, using the triangle inequality of $L_{2}$ norm,

$\begin{split}||\nabla_{h_{i}}M||\leq&\frac{2}{\sqrt{M}}\big{[}\frac{1}{m_{1}^{2}}\sum_{j=1}^{m_{1}}g_{\sigma}(||h_{i}-h_{j}^{(1)}||)\\\
&+\frac{2}{m_{1}m_{2}}\sum_{j=1}^{m_{2}}g_{\sigma}(||h_{i}-h_{j}^{(2)}||)\big{]}.\end{split}$
(8)

Within the mini-batch, $\sqrt{M}$ remain as constant for all samples. From Eq.
(8), we observe that when $x_{i}$ deviates significantly away from the
majority of samples of the same class, i.e., noisy samples or outliers,
$||h_{i}-h_{j}^{(1)}||$ and $||h_{i}-h_{j}^{(2)}||$ are large, the magnitude
of its gradient in matching loss diminishes. In other words, $x_{i}$ will only
provide signal from the supervision loss (e.g., CE loss) and its impact on
matching loss is negligible. On the other hand, training ITRA with matching
loss promotes the alignment of feature representations of samples that stay
close in the latent feature space. From the data distribution perspective,
samples deviating from the majority are likely of low-density or even
outliers. Then such behavior of ITRA implies that it can help DNNs to better
capture information from high density areas and reduce the distraction of “low
density” samples in learning feature representations on the data manifold.

On reducing over-adaption to mini-batches The analysis above shows that low-
density samples only provide supervision signal in ITRA, we now analyze how
ITRA reduces the over-adaption to mini-batches. It turns out that this effect
is achieved by an adaptively weighted feature alignment mechanism, which
implicitly boosts the supervision signal from high-density samples and
resultantly downweights relatively the contribution of low-density samples.

To understand this, we examine the full gradient of supervision loss $L$ and
matching loss MMD. Note that in ITRA, the gradient of supervision loss is only
calculated on one mini-batch. Without loss of generality, we consider sample
$x_{i}$ from the first mini-batch. The full gradient of $L(x_{i})$ and
$M$=MMD$(x_{i},H_{(2)}^{k})$ with respect to $h_{i}$ is (using the same
notation as above)

$\begin{split}\nabla_{h_{i}}(M+L)=&\frac{4}{\sqrt{M}m_{2}}\sum_{j=1}^{m_{2}}\exp(-\frac{||h_{i}-h_{j}^{(2)}||^{2}}{\sigma})\frac{h_{i}-h_{j}^{(2)}}{\sigma}\\\
&+\nabla_{o_{i}}L\cdot\frac{\partial o_{i}}{\partial h_{i}},\end{split}$

where $o_{i}$ is the output for $x_{i}$. Let
$A=\sum_{j=1}^{m_{2}}\exp(-||h_{i}-h_{j}^{(2)}||^{2}/\sigma)$ and
$w_{j}=\exp(-||h_{i}-h_{j}^{(2)}||^{2}/\sigma)/A$
($\sum_{j=1}^{m_{2}}w_{j}=1$), then equivalently:

$\nabla_{h_{i}}(M+L)=\frac{4A}{\sqrt{M}m_{2}\sigma}(h_{i}-\sum_{j=1}^{m_{2}}w_{j}h_{j}^{(2)})+\nabla_{o_{i}}L\cdot\frac{\partial
o_{i}}{\partial h_{i}}.$ (9)

When ITRA converges and DNNs is well trained with good performance,
$||\nabla_{h_{i}}(M+L)||\approx 0$ and $||\nabla_{o_{i}}L\cdot\partial
o_{i}/\partial h_{i}||$ is close to zero, we have
$||h_{i}-\sum_{j=1}^{m_{2}}w_{j}h_{j}^{(2)}||<\epsilon$ ($\epsilon$ is a small
scalar). In other words, ITRA promotes the feature representation $h_{i}$ of
$x_{i}$ to align with the weighted average
$\sum_{j=1}^{m_{2}}w_{j}h_{j}^{(2)}(\sum_{j=1}^{m_{2}}w_{j}=1)$, where each
$w_{j}$ is adaptively adjusted in the training process based on similarity
between $h_{i}$ and $h_{j}^{(2)}$ in the latent feature space. As mini-batch
samples are uniformly sampled from the training data, it is expected that on
average, the majority of $\\{h_{j}^{(2)}\\}_{j=1}^{m_{2}}$ are from high-
density area of the data distribution. For DNNs with good generalizability,
DNNs must perform well for samples from those areas (as testing samples are
more likely to be generated from high-density areas in the data manifold).
Hence, provided that sample $x_{i}$ is of high-density that already provides
useful supervision signal, ITRA further boosts its contribution by aligning
$h_{i}$ with $\sum_{j=1}^{m_{2}}w_{j}h_{j}^{(2)}$ of other high-density
samples in the 2nd mini-batch. The adaptive weight $w_{j}$ is critical: if
sample $h_{j}^{(2)}$ is of low-density and deviates far from $x_{i}$, its
weight $w_{j}$ is automatically adjusted small, having vanishing contribution
in the gradient. This in turn downweights relatively the contribution of low-
density samples in SGD, resulting in the reduction of over-adaption to mini-
batches.

Accommodating multi-modalities The adaptively weighting mechanism brings
another benefit: if the data distribution (for each class) is multi-modality
in the latent feature space, ITRA automatically aligns $x_{i}$ with its
corresponding modality. Specifically, without loss of generality, assume two
modalities $md_{1}$ and $md_{2}$, $\\{h_{j}^{(2)}\\}$ consists of samples from
$md_{1}$ and $md_{2}$ and $x_{i}$ is generated from $md_{1}$. We can rewrite
$h_{i}-\sum_{j=1}^{m_{2}}w_{j}h_{j}^{(2)}=h_{i}-(\sum_{j\in
md_{1}}w_{j}h_{j}^{(2)}+\sum_{j\in md_{2}}w_{j}h_{j}^{(2)})$. As $x_{i}$ is
generated from $md_{1}$ and deviates from $md_{2}$, implying that $x_{i}$ is
closer to samples from the same modality than those from the other modality.
Hence, with the adaptively weighting mechanism in Eq. (9), $w_{j}\approx 0$
$(j\in md_{2})$ and $h_{i}-\sum_{j=1}^{m_{2}}w_{j}h_{j}^{(2)}\approx
h_{i}-\sum_{j\in md_{1}}w_{j}h_{j}^{(2)}$. That is, align $x_{i}$ only with
samples from the same modality. Therefore, ITRA avoids the uni-modality
assumption on data distribution as in (Wen et al. 2016; Wan et al. 2018) and
justifies the advantage of nonparametric MMD for feature alignment.

## Experiments

In this Section, we extensively evaluate the ITRA performance using benchmark
datasets on both image classification (i.e., KMNIST (Clanuwat et al. 2018),
FMNIST (Xiao, Rasul, and Vollgraf 2017), CIFAR10, CIFAR100 (Krizhevsky, Hinton
et al. 2009), STL10 (Coates, Ng, and Lee 2011) and ImageNet (Deng et al.
2009)) and text classification (i.e., AG’s News, Amazon Reviews, Yahoo Answers
and Yelp Reviews) tasks. In our experiments, class-conditional ITRA-c is
tested as it exploits implicit label information with better supervision in
the training process. In addition to using vanilla SGD training as the
baseline (i.e., w/o ITRA), we also compare ITRA with more widely used loss-
function based regularization methods as the strong baselines for comparison:
label smoothing (LSR) (Szegedy et al. 2016) and center loss (Center) (Wen et
al. 2016). For evaluation metrics, we report the Top-1 accuracy and CE loss
value for all methods. The optimal hyperparameter value $\lambda$ for each
method is also reported. Results on other tuning parameter values as well as
experimental details are provided in supplementary materials.

Image classification Table 1 shows the performance for KMNIST and FMNIST
testing data. From the Table, we see that training with ITRA achieves better
results in terms of higher accuracy and lower CE. In terms of the testing
loss, ITRA has a smaller loss value compared with other methods. The testing
loss with respect to different $\lambda$ values are shown in Supplementary
Materials. As CE is equivalent to negative log-likelihood, smaller CE value
implies that the network makes predictions on testing data with higher
confidence on average. In each iteration of ITRA, there is a trade-off between
the CE and matching loss. This leads to that ITRA has a regularization effect
by alleviating the over-confident predictions on training data. As a result,
the smaller gap between training and testing losses implies that ITRA has
better generalization performance. When trained with vanilla SGD, we observe
that the increasing testing loss exhibits an indication of overfitting, which
is due to that FMNIST has a significant number of hard samples (e.g., those
from pullover, coat and shirt classes). However, ITRA is capable of
regularizing the training process hence prevents overfitting and stabilizes
the testing loss as shown in the Figure 2.

Table 1: Accuracy (in %, larger is better) and CE (smaller is better) on
KMNIST and FMNIST data. The optimal performances are obtained by tuning
multiple $\lambda$s according to existing literature (Szegedy et al. 2016; Wen
et al. 2016).

| KMNIST | FMNIST
---|---|---
| $\lambda$ | Acc $\uparrow$ | CE $\downarrow$ | $\lambda$ | Acc $\uparrow$ | CE $\downarrow$
Baseline | - | 95.57 | 0.183 | - | 92.43 | 0.294
LSR | 0.1 | 95.60 | 0.181 | 0.1 | 92.47 | 0.292
Center | 0.1 | 94.90 | 0.214 | 0.1 | 92.10 | 0.263
ITRA | 0.8 | 95.79 | 0.170 | 0.6 | 92.57 | 0.224

Table 2: Accuracy and CE loss on CIFAR10, STL10 and CIFAR100 datasets.

|  | CIFAR10 | STL10 | CIFAR100
---|---|---|---|---
|  | $\lambda$ | Acc $\uparrow$ | CE $\downarrow$ | $\lambda$ | Acc $\uparrow$ | CE $\downarrow$ | $\lambda$ | Acc $\uparrow$ | CE $\downarrow$
Resnet18 | Baseline | - | 92.99 | 0.40 | - | 70.88 | 1.63 | - | 74.19 | 1.05
LSR | 0.1 | 92.73 | 0.42 | 0.1 | 71.08 | 1.55 | 0.1 | 74.21 | 1.04
Center | 0.1 | 92.30 | 0.35 | 0.1 | 70.97 | 1.10 | 0.05 | 73.98 | 0.98
ITRA | 0.8 | 93.70 | 0.27 | 0.6 | 72.78 | 1.05 | 0.6 | 74.88 | 0.97
VGG13 | Baseline | - | 92.49 | 0.47 | - | 74.40 | 1.55 | - | 71.72 | 1.46
LSR | 0.1 | 92.53 | 0.46 | 0.1 | 74.50 | 1.51 | 0.1 | 71.75 | 1.43
Center | 0.05 | 92.11 | 0.38 | 0.05 | 74.04 | 1.16 | 0.05 | 71.65 | 1.31
ITRA | 0.8 | 92.72 | 0.33 | 0.8 | 75.80 | 0.93 | 0.6 | 72.55 | 1.22
MobileV2 | Baseline | - | 88.55 | 0.62 | - | 59.09 | 2.14 | - | 66.42 | 1.57
LSR | 0.1 | 88.77 | 0.61 | 0.1 | 59.01 | 2.12 | 0.1 | 66.60 | 1.55
Center | 0.1 | 88.81 | 0.53 | 0.1 | 58.24 | 1.46 | 0.05 | 66.39 | 1.51
ITRA | 1.0 | 89.37 | 0.43 | 0.6 | 62.02 | 1.60 | 0.6 | 67.23 | 1.49

Figure 2: Testing loss of Resnet18 w.r.t. different $\lambda$ values on
CIFAR10, STL10 and CIFAR100. Table 3: Accuracy and CE loss of Resnet-101 on
CIFAR-100 and Resnet-18 on ImageNet.

| CIFAR-100 (Resnet-101) | ImageNet (Resnet-18)
---|---|---
| Batch Size | Acc | CE | Batch Size | Acc | CE
Baseline | 200 | 75.85 | 1.04 | 256 | 50.01 | 2.24
Center ($\lambda=0.05$) | 200 | 75.23 | 1.05 | 256 | 51.83 | 2.21
ITRA ($\lambda=0.6$) | 50 | 75.97 | 1.01 | 64 | 51.32 | 2.19
100 | 77.35 | 1.01 | 128 | 51.69 | 2.17
200 | 76.85 | 0.94 | 256 | 53.13 | 2.07

Additionally, in Table 2, we present the performance of Resnet18, VGG13 and
MobilenetV2 on CIFAR10, STL10 and CIFAR100. From the Table, we see that ITRA
achieves the best performance compared among all the four methods. Especially
for the relatively more challenging (lower accuracy) STL10 data set, ITRA
outperforms the baseline with a significant margin, i.e., Resnet $1.9\%$,
VGG13 $1.4\%$ and MobilenetV2 $2.9\%$. In terms of CE loss, all methods have
similar training losses that are close to zero. However, ITRA and Center have
significantly better testing loss than other the baseline and LSR. A closer
gap between training and testing losses indicates a better generalization of
the DNN models enabled by regularization capability of ITRA.

Table 4: Accuracy and CE loss on text classification task.

|  | AG’s News | Amazon Full | Yahoo Answers | Yelp Full
---|---|---|---|---|---
|  | $\lambda$ | Acc $\uparrow$ | CE $\downarrow$ | $\lambda$ | Acc $\uparrow$ | CE $\downarrow$ | $\lambda$ | Acc $\uparrow$ | CE $\downarrow$ | $\lambda$ | Acc $\uparrow$ | CE $\downarrow$
Bert-base | Baseline | - | 92.10 | 0.42 | - | 58.53 | 1.66 | - | 66.60 | 1.36 | - | 61.12 | 1.52
Center | 0.1 | 91.70 | 0.45 | 0.1 | 59.38 | 1.23 | 0.1 | 66.95 | 1.25 | 0.1 | 60.58 | 1.17
ITRA | 0.5 | 92.45 | 0.29 | 0.2 | 60.08 | 1.29 | 0.6 | 67.47 | 1.25 | 0.4 | 61.68 | 1.09
DistillBert | Baseline | - | 92.13 | 0.38 | - | 58.10 | 1.38 | - | 66.50 | 1.28 | - | 60.02 | 1.32
Center | 0.1 | 91.33 | 0.47 | 0.1 | 57.60 | 1.21 | 0.1 | 66.57 | 1.26 | 0.1 | 59.58 | 1.14
ITRA | 0.2 | 92.27 | 0.26 | 0.4 | 58.30 | 1.20 | 0.1 | 66.65 | 1.17 | 0.1 | 60.30 | 1.17
XLNet | Baseline | - | 91.43 | 0.40 | - | 60.65 | 1.34 | - | 66.90 | 1.29 | - | 62.78 | 1.21
Center | 0.1 | 90.95 | 0.45 | 0.1 | 59.88 | 1.18 | 0.1 | 66.93 | 1.28 | 0.1 | 62.82 | 1.10
ITRA | 0.8 | 91.85 | 0.27 | 0.5 | 60.90 | 1.23 | 0.5 | 67.05 | 1.15 | 1.0 | 63.32 | 1.08

Larger-scale experiment on image classification Table 3 shows results on the
large-scale ImageNet dataset and the deeper Resnet-101 network architecture.
Note that compared with performance of Resnet-18 in Table 2, the deeper
Resnet-101 indeed demonstrates a better performance over the CIFAR100 dataset.
For both larger dataset and deeper network, ITRA consistently achieves better
accuracy and lower CE value than other methods. Markedly, for the ImageNet
dataset, ITRA improves the accuracy by $5.0\%(3.12/62.92)$ and CE value by
$9.7\%(0.15/1.55)$ over the standard baseline. When compared with the strong
baseline Center loss, ITRA also improves the accuracy by $2.1\%(1.36/64.68)$
and CE value by $4.8\%(0.07/1.47)$. The training time of ResNet-101
(CIFAR-100) and ResNet-50 (ImageNet) using ITRA are $7.5\%$ and $3.9\%$ more
than baseline on an RTX 3090 GPU. Despite the moderate increase in training
time for large-scale experiments due to the extra computation incurred by
sampling additional mini-batch, it demonstrates a reasonable trade-off between
the increase in computational cost and gaining attractive analytic properties
of ITRA.

Large-scale experiment on text classification ITRA performance is also
evaluated on large-scale text classification experiments. We use different
loss functions for fine-tuning the pre-trained Bert-base, DistillBert and
XLNet models from Huggingface transformers library (Wolf et al. 2019). Table 4
shows that the models fine-tuned with ITRA achieve a better performance in
terms of accuracy and CE value on most datasets. Specifically, for Bert-base,
DistillBert, XLNet models, ITRA achieves an average accuracy improvement of
$1.3\%$, $0.3\%$, $0.5\%$, respectively, and an average CE value improvement
of $22.4\%$, $16.1\%$, and $15.6\%$, respectively. It is worth noting that the
Center loss also reduces CE value occasionally in the experiments, its
accuracy performance is nevertheless compromised. The potential reason behind
this phenomenon could be the multi-modality of natural language. The Center
loss’s uni-modality assumption helps model to minimize the distances within
the class (hence CE), but can therefore lead to sub-optimal feature learning
for hard samples near the boundary between modals if the class conditional
distribution is indeed multi-modal.

Comparing with Center loss As discussed in Section 2, center loss (Wen et al.
2016) is the closest work to ours. It effectively characterizes the intra-
class variations by aligning features of each to its “center” which is
designed to reduce variance in feature learning and results in compact feature
representations. Different from ITRA which aligns a pair of features from two
minibatches to each other, Center loss explicitly assumes uni-modality of data
distribution at feature level for each class, which may be valid in face
recognition task where the Center loss is initially proposed for, but can be
too stringent in classification task as class-conditional density can be
multi-modal. On the contrary, ITRA is capable of accommodating the multi-
modalities supported by a rigorous analysis in Section 4.1. In Figure 3, the
model trained with ITRA effectively captures the “typical pattern” of each
class at feature level and misses some hard samples to improve
generalizability. The results on both image and text classifications concur
with our analysis: as shown in the Tables 1, 2 and 4, although Center loss can
occasionally reduce the CE value, it is still outperformed by our method due
to its strong assumption.

Hyperparameter $\lambda$ and batch size Here we also investigate the influence
of hyperparameter $\lambda$ and batch size of ITRA. As shown in the Tables
1&2, when $\lambda$ is set with a relatively large value of 0.8 or 1, ITRA can
outperform other methods in terms of both accuracy and testing loss, which is
due to that larger $\lambda$s incorporate stronger implicit supervision
information as mini-batches from the same class are matched. We also plot the
CE loss for different $\lambda$s in Figure 2 w.r.t. Resnet. Comparing with
baseline, we see that training with ITRA results in significant gain in CE,
regardless of network architecture. Looking at Figure 2 (b) in more detail,
when trained with the baseline, the testing loss shows an increasing trend as
a sign of overfitting while ITRA can alleviate this trend as $\lambda$
increases. For batch size, Table 3 demonstrates that the increase of batch
size indeed helps the reduction of variation in feature learning (lower CE
loss), however, it usually requires advanced large-scale GPU clusters. Table 3
illustrates that ITRA achieves a better performance even with $1/4$ batch size
compared to the baseline and thus avoid this hardware restriction.

Figure 3: T-SNE plot for CIFAR10 testing data. Networks are trained with
$\lambda$ that achieves best accuracy in Table 2.

Learning compact feature representations From the geometric perspective,
samples from the same class should stay close (i.e., intra-class compactness)
and those from different classes are expected to stay far apart (i.e., inter-
class separability) in the feature space (so that $f_{k}$ output by softmax is
large). We visualize the distribution of CIFAR10 testing samples with T-SNE
(Maaten and Hinton 2008) in Figure 3. From the figure, we have the following
observations: (1) ITRA learns feature representation that is much tighter with
clearer inter-class margin than that learned by vanilla SGD training. (2) The
data distribution in the latent space learned by ITRA exhibits a consistent
pattern that for each class, the majority of testing samples are closely
clustered to form a data manifold, while a small subset of samples deviate
from the majority. This phenomenon concurs with our analysis that the matching
loss provides diminishing gradient signals for “low-density” samples while
encourages the closeness of “high-density” samples. Hence, ITRA can
effectively capture the “typical pattern” of each class but can miss some hard
samples that overlap with other classes. This explains why ITRA achieves
impressive improvement in CE value but not as much in accuracy. Overall, ITRA
still outperforms vanilla SGD training and can be used as a promising training
prototype that enjoys theoretical merits as shown in the analysis for matching
loss.

## Conclusion

In this paper, we propose a new training strategy, ITRA, as a loss function
based regularization approach that can be embedded in the standard SGD
training procedure. ITRA augments vanilla SGD with a matching loss that uses
MMD as the objective function. We show that ITRA enjoys three theoretical
merits that can help DNN learn compact feature representations without
assuming uni-modality on the feature distribution. Experimental results
demonstrate its excellent performance on both image and text classification
tasks, as well as its impressive feature learning capacity. We outline two
possible directions for future studies. The first is to improve ITRA that can
learn hard sample more effectively. The second is the ITRA application in
learning from poisoned datasets as ITRA is able to capture the high density
areas (i.e., modalities) for each class where poisoned samples deviates far
from those areas (e.g., erroneously labeled samples from other classes).

## Acknowledgements

This work is supported by the National Science Foundation under grant
IIS-2211897.

## Ethical Impact

This paper proposes ITRA to improve the performance of feature representation
learning for training DNNs. As a general training strategy for supervised
classification problems, ITRA can be used as a drop-in replacement in wherever
the vanilla SGD is used. Most modern deep learning model training utilizes SGD
as the standard optimization algorithm where researchers have already proposed
various alternatives and/or enhancements to overcome its intrinsic
limitations; the over adaption to mini-batch is more pronounced. The proposed
in-training regularization via aligning representations enables learning more
compact features thus to improve the generalizability of the predictive
models. With our and many other effective feature representation learning
approaches, manual feature engineering requiring profound domain knowledge and
expertise will eventually phase out. As such, our research has positive
impacts to a broad range of machine learning and artificial intelligence
domains where domain adaption and generalization become the primary concern.
For example, in medical imaging based diagnosis, leveraging ITRA on a smaller
labeled and heterogeneous training set is expected to demonstrate a
competitive and consistent performance to other medical imaging data sets.

## References

  * Ben-David et al. (2007) Ben-David, S.; Blitzer, J.; Crammer, K.; and Pereira, F. 2007.  Analysis of representations for domain adaptation.  In _Advances in neural information processing systems_ , 137–144.
  * Chorowski et al. (2015) Chorowski, J. K.; Bahdanau, D.; Serdyuk, D.; Cho, K.; and Bengio, Y. 2015.  Attention-based models for speech recognition.  In _Advances in neural information processing systems_ , 577–585.
  * Clanuwat et al. (2018) Clanuwat, T.; Bober-Irizar, M.; Kitamoto, A.; Lamb, A.; Yamamoto, K.; and Ha, D. 2018.  Deep learning for classical japanese literature.  _arXiv preprint arXiv:1812.01718_.
  * Coates, Ng, and Lee (2011) Coates, A.; Ng, A.; and Lee, H. 2011.  An analysis of single-layer networks in unsupervised feature learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , 215–223.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * DeVries and Taylor (2017) DeVries, T.; and Taylor, G. W. 2017.  Improved regularization of convolutional neural networks with cutout.  _arXiv preprint arXiv:1708.04552_.
  * Gao, Pei, and Huang (2019) Gao, H.; Pei, J.; and Huang, H. 2019.  Demystifying Dropout.  In _The 36th International Conference on Machine Learning (ICML 2019)_.
  * Gastaldi (2017) Gastaldi, X. 2017.  Shake-shake regularization.  _arXiv preprint arXiv:1705.07485_.
  * Ghiasi, Lin, and Le (2018) Ghiasi, G.; Lin, T.-Y.; and Le, Q. V. 2018.  Dropblock: A regularization method for convolutional networks.  In _Advances in Neural Information Processing Systems_ , 10727–10737.
  * Gretton et al. (2007) Gretton, A.; Borgwardt, K.; Rasch, M.; Schölkopf, B.; and Smola, A. J. 2007\.  A kernel method for the two-sample-problem.  In _Advances in neural information processing systems_ , 513–520.
  * Gretton et al. (2012) Gretton, A.; Borgwardt, K. M.; Rasch, M. J.; Schölkopf, B.; and Smola, A. 2012\.  A kernel two-sample test.  _Journal of Machine Learning Research_ , 13(Mar): 723–773.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Howard et al. (2017) Howard, A. G.; Zhu, M.; Chen, B.; Kalenichenko, D.; Wang, W.; Weyand, T.; Andreetto, M.; and Adam, H. 2017.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_.
  * Huang et al. (2019) Huang, G.; Liu, Z.; Pleiss, G.; Van Der Maaten, L.; and Weinberger, K. 2019.  Convolutional Networks with Dense Connectivity.  _IEEE transactions on pattern analysis and machine intelligence_.
  * Ioffe and Szegedy (2015) Ioffe, S.; and Szegedy, C. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  _arXiv preprint arXiv:1502.03167_.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Li et al. (2022) Li, C.; Dong, Z.; Fisher, N.; and Zhu, D. 2022.  Coupling User Preference with External Rewards to Enable Driver-centered and Resource-aware EV Charging Recommendation.  _arXiv preprint arXiv:2210.12693_.
  * Li et al. (2017) Li, C.-L.; Chang, W.-C.; Cheng, Y.; Yang, Y.; and Póczos, B. 2017.  Mmd gan: Towards deeper understanding of moment matching network.  In _Advances in Neural Information Processing Systems_ , 2203–2213.
  * Li et al. (2020) Li, X.; Li, X.; Pan, D.; and Zhu, D. 2020.  On the learning property of logistic and softmax losses for deep neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 4739–4746.
  * Li et al. (2021) Li, X.; Li, X.; Pan, D.; and Zhu, D. 2021.  Improving adversarial robustness via probabilistically compact loss with logit constraints.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 8482–8490.
  * Li and Zhu (2018) Li, X.; and Zhu, D. 2018.  Robust feature selection via l2,-norm in finite mixture of regression.  _Pattern Recognition Letters_ , 108: 15–22.
  * Li, Zhu, and Dong (2018) Li, X.; Zhu, D.; and Dong, M. 2018.  Multinomial classification with class-conditional overlapping sparse feature groups.  _Pattern Recognition Letters_ , 101: 37–43.
  * Li, Zhu, and Levy (2020) Li, X.; Zhu, D.; and Levy, P. 2020.  Predicting Clinical Outcomes with Patient Stratification via Deep Mixture Neural Networks.  _AMIA Summits on Translational Science Proceedings_ , 2020: 367.
  * Li, Swersky, and Zemel (2015) Li, Y.; Swersky, K.; and Zemel, R. 2015.  Generative moment matching networks.  In _International Conference on Machine Learning_ , 1718–1727.
  * Long et al. (2015) Long, M.; Cao, Y.; Wang, J.; and Jordan, M. I. 2015.  Learning transferable features with deep adaptation networks.  _arXiv preprint arXiv:1502.02791_.
  * Maaten and Hinton (2008) Maaten, L. v. d.; and Hinton, G. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ , 9(Nov): 2579–2605.
  * Miotto et al. (2018) Miotto, R.; Wang, F.; Wang, S.; Jiang, X.; and Dudley, J. T. 2018.  Deep learning for healthcare: review, opportunities and challenges.  _Briefings in bioinformatics_ , 19(6): 1236–1246.
  * Pan, Li, and Zhu (2021) Pan, D.; Li, X.; and Zhu, D. 2021.  Explaining Deep Neural Network Models with Adversarial Gradient Integration.  In _IJCAI_ , 2876–2883.
  * Qiang et al. (2022a) Qiang, Y.; Li, C.; Brocanelli, M.; and Zhu, D. 2022a.  Counterfactual interpolation augmentation (cia): A unified approach to enhance fairness and explainability of dnn.  In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI_ , 732–739.
  * Qiang et al. (2022b) Qiang, Y.; Pan, D.; Li, C.; Li, X.; Jang, R.; and Zhu, D. 2022b.  AttCAT: Explaining Transformers via Attentive Class Activation Tokens.  In _Advances in Neural Information Processing Systems_.
  * Shen et al. (2018) Shen, J.; Qu, Y.; Zhang, W.; and Yu, Y. 2018.  Wasserstein distance guided representation learning for domain adaptation.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Simonyan and Zisserman (2014) Simonyan, K.; and Zisserman, A. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_.
  * Srivastava et al. (2014) Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1): 1929–1958.
  * Sutskever, Vinyals, and Le (2014) Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014.  Sequence to sequence learning with neural networks.  In _Advances in neural information processing systems_ , 3104–3112.
  * Szegedy et al. (2016) Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z. 2016.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2818–2826.
  * Wan et al. (2018) Wan, W.; Zhong, Y.; Li, T.; and Chen, J. 2018.  Rethinking feature distribution for loss functions in image classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 9117–9126.
  * Wen et al. (2016) Wen, Y.; Zhang, K.; Li, Z.; and Qiao, Y. 2016.  A discriminative feature learning approach for deep face recognition.  In _European conference on computer vision_ , 499–515. Springer.
  * Wolf et al. (2019) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Xiao, Rasul, and Vollgraf (2017) Xiao, H.; Rasul, K.; and Vollgraf, R. 2017.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_.

## Supplementary Materials

Table 5: Text classification tasks datasets statistics. Dataset | Classes | Train Samples | Test Samples
---|---|---|---
AG’s News | 4 | 120,000 | 76,00
Amazon Full | 5 | 650,000 | 50,000
Amazon Polarity | 2 | 560,000 | 38,000
Yahoo Answers | 10 | 1,400,000 | 60,000
Yelp Full | 5 | 3,600,000 | 650,000
Yelp Polarity | 2 | 3,000,000 | 400,000

Comparison In the experiments, in addition to the vanilla SGD training as the
baseline (i.e., w/o ITRA), we also compare ITRA with other loss-function based
regularization methods. All DNNs in our experiments include BN layers as part
of the model architectures. Label smoothing (Szegedy et al. 2016) (LSR) is a
target-based regularization that the Dirac distribution for ground truth label
is replaced with a mixture of Dirac distribution and uniform distribution;
center loss (Wen et al. 2016) (Center) encourages interclass compactness of
feature representations which augments the cross-entropy loss with the maximum
likelihood (Wan et al. 2018) assuming that each class follows Gaussian
distribution in latent feature space.

Implementation Details Through all experiments, the optimization algorithm is
the standard stochastic gradient descent with momentum and the loss function
is cross-entropy (CE) loss. In ITRA-c, CE loss is further combined with the
matching loss Eq. (6) in each iteration.

For the initial experiments on QMNIST, KMNIST and FMNIST, the CNN architecture
of 2 convolutional layers is
Conv(C20K5S1)-MP(K2S2)-Conv(C50K5S1)-MP(K2S2)-100-10, where CxKySz for a
convolutional layer means x convolutional kernels with kernel size y, stride
z; MP represents max-pooling. To train the network, we use the SGD algorithm
with a momentum of 0.5. The number of epochs is 50. Learning rate is 0.01 and
multiplied by 0.2 every 20 epochs. Batch size is set to 150. We don’t use
$L_{2}$ regularization. In ITRA, the tuning parameter $\lambda$ is set to 1.

Table 6: Test accuracy w.r.t. different $\lambda$s.

| LSR
---|---
$\lambda$ | 0.05 | 0.1 | 0.15 | 0.2
KMNIST | 95.51 | 95.60 | 95.18 | 95.02
FMNIST | 92.39 | 92.47 | 92.07 | 91.83
CIFAR10 | 92.68 | 92.73 | 92.33 | 92.06
STL10 | 71.05 | 71.08 | 70.95 | 70.94
CIFAR100 | 74.19 | 74.21 | 73.91 | 73.72

| Center
---|---
$\lambda$ | 0.05 | 0.1 | 0.15 | 0.2
KMNIST | 94.78 | 94.91 | 94.62 | 94.14
FMNIST | 91.99 | 92.10 | 91.63 | 91.27
CIFAR10 | 92.16 | 92.30 | 91.97 | 91.63
STL10 | 70.82 | 70.97 | 70.23 | 69.94
CIFAR100 | 73.98 | 73.84 | 73.26 | 72.86

| ITRA
---|---
$\lambda$ | 0.2 | 0.4 | 0.6 | 0.8 | 1
KMNIST | 95.68 | 95.59 | 95.75 | 95.79 | 95.7
FMNIST | 92.42 | 92.44 | 92.57 | 92.51 | 92.45
CIFAR10 | 93.52 | 92.91 | 92.93 | 93.71 | 92.98
STL10 | 71.64 | 72.78 | 72.88 | 71.29 | 72.53
CIFAR100 | 74.16 | 74.15 | 74.88 | 74.43 | 74.35

For KMNIST and FMNIST in “Experiments” section, we build a 5-layer
convolutional neural network (CNN) with batch normalization applied. The CNN
architecture is Conv(C32K3S1) - BN - Conv(C64K3S1) - BN - Conv(C128K3S1) -
MP(K2S2) - Conv(C256K3S1) - BN - Conv(C512K3S1) - MP(K8S1) - 512 - 10, where
BN represents batch-normalization. Momentum is set to 0.5, batch size 150,
number of epochs 50, initial learning rate 0.01 and multiplied by 0.2 at 20th
and 40th epoch. No data augmentation is applied. For CIFAR10 and STL10, we use
publicly available implementation of VGG13 (Simonyan and Zisserman 2014),
Resnet18 (He et al. 2016) and MobilenetV2 (Howard et al. 2017). All models are
trained with 150 epochs, SGD momentum is set to 0.5, initial learning rate is
0.5 and multiplied by 0.1 every 50 epochs, batch size 150. We resize STL10 to
32 $\times$ 32\. For colored image datasets, we use random crop and horizontal
flip for data augmentation. For CIFAR100 with more classes, we use weight
decay of $5\times 10^{-4}$ and a mini-batch size of 200. For large ImageNet
dataset, we trained a plain Resnet18 for 45 epochs with initial learning rate
of 0.2 and multiplied by 0.1 every 15 epochs. For text classification tasks,
we fine-tune the pre-trained Bert-base, DistillBert and XLNet models for 4
epochs with learning rate as $5\times 10^{-5}$ and weight decay as $1\times
10^{-5}$ for all comparison methods on all the datasets.

In all experiments, networks are trained with each method under the same
setting (learning rate, batch size et al.). For the bandwidth parameter in
Gaussian kernels, we follow the practice in (Gretton et al. 2007, 2012; Long
et al. 2015) that takes the heuristic of setting $\sigma$ as the median
squared distance $\sigma_{\text{Med}}$ between two samples and use a mixture
of Guassian kernels with bandwidth set as a multiple of $\sigma_{\text{Med}}$.
In our experiments, we use 5 kernels
$k_{\text{mix}}(x,y)=\frac{1}{5}\sum_{i=1}^{5}k_{\sigma_{i}}(x,y)$ with
$\\{\sigma_{i}=2^{i}\sigma_{\text{Med}}:i=0,\cdots,4\\}$. To select the tuning
parameter $\lambda$ in each method, we split out 5,000 images from the
training data as validation dataset. For $\lambda$ in ITRA, we test {0.2, 0.4,
0.6, 0.8, 1} for checking ITRA’s sensitivity to it. Note that when
$\lambda=0$, ITRA is equivalent to vanilla SGD training. In LSR, we select the
tuning parameter from $\\{0.2,0.15,0.1,0.05\\}$; for center loss,
$\\{0.2,0.15,0.10,0.05\\}$ are tested. We utilize the standard train/test
split given in the benchmark datasets and train all models once on the
training data and performances are evaluated on the testing data. Performance
w.r.t. different $\lambda$ values for each method is shown in Table 6.

KMNIST and FMNIST loss curve We plot the testing loss with respect to
different $\lambda$ values for KMNIST and FMNIST in Figure 4 (a) and (b)
respectively. We can observe that for all $\lambda$ values, ITRA always
achieves better testing loss than the baseline (w/o ITRA). Particularly for
FMNIST, the baseline exhibits overfitting with an increasing trend of testing
loss while ITRA provides regularization that stabilizes the testing loss.

Figure 4: Testing loss w.r.t. different $\lambda$ values on KMNIST and FMNIST