# Exploring Context-Aware Conversational Agents in Software Development

Glaucia Melo, Edith Law, Paulo Alencar, Don Cowan  David R. Cheriton School of
Computer Science
University of Waterloo
Waterloo, Canada
<EMAIL_ADDRESS>
###### Abstract

Software development is a complex endeavor that depends on a wide variety of
contextual factors involving a large amount of distributed information. This
knowledge could include: technology-related tasks, software operating
environments and stakeholder requirements. A major roadblock to using this
knowledge in software development is that most of this information is implicit
and captured in the developers’ minds (tacit) or spread through volumes of
documentation. Developers, as they work often have to maintain mental models
of these tasks as they produce the software. As a result, context can be
easily lost or forgotten and developers often use trial-and-error approaches
while finishing the project. This study aims at analyzing whether supporting
software developers with a chatbot during task execution can improve the
overall development experience. The chatbot can assist the developers with
executing different tasks based on implicit contextual information. We propose
an implementation to explore the viability of using textual chatbots to assist
developers automatically and proactively with software development project
activities that recur.

Keywords: Software engineering, context, software development, software
projects, literature review.

##  I Introduction

Software development is a complex activity, with an increasing demand for
professionals. Developers perform various software development tasks, with
different context switches [1, 2]. Most of this context, which can include:
tool preferences, approaches to tasks, documentation, and the project, is
tacit, and brings significant added complexity to software development.
Developers maintain mental models of a project combining items such as various
types of data including system, version control, environment, project details
and tasks [1], and it would be beneficial if at least some of these details
did not have to rely on a developers’ ability to remember the execution
sequence. A simple solution to support developers when executing tasks, they
must remember should be to automate these tasks. However, automation leads to
a set of challenges, as indicated by Bradley et al. [1] because:

  * •

It is hard to determine in advance all the tasks and workﬂows developers will
need to complete;

  * •

These workflows consist of various actions that often span tool boundaries and
require a diverse set of parameters that may depend on the current context and
developer intent;

  * •

Even if there are scripts configured for automating workflows, the developer
needs to remember their existence and how to invoke them in the current
context.

There are also other examples of activities that are hard to automate, such as
searching for support in an implementation, or editing a set of three files
but wanting to commit only two, because the developer thinks the third one
needs further review. Moreover, in software development, having the human-in-
the-loop has proven to be beneficial in terms of searching for support when
developing project tasks [3].

A few examples of project tasks are: the next step in process (open task /
code / test / deploy / close task), some recurrent commands (git push/pull,
deploy, close task/issue), the available testing options, and code review
options, determining the developers’ next assigned task, deploying workflow
based on the environment and is there support available for this task (API
Tutorial, Stack Overflow post, commit commented). In this paper, we conduct an
exploratory study that aims at understanding the variety of tasks in software
development that could be supported by a textual conversational agent
(chatbot). We want to identify through a pilot study how developers interact
with the agent, and developers’ preferences if the agent can support the
execution of current project tasks and not just be a developer’s memory (mind
maps).

We aim to investigate this problem of developer support through the following
research question: How well can a chatbot support software developers while
executing development tasks? We believe that the use of a proactive chatbot
can make these mostly implicit contexts more explicit and make the development
process less prone to errors as the developer does not have to remember all
the tasks to be performed. We hypothesize that capturing context (embedded in
software development tasks) in software development and having a chatbot to
suggest these tasks to developers can leverage software development, as
recurrent and implicit tasks can be forgotten. For this exploratory pilot
study, we performed a Wizard-of-Oz study using Slack as the platform to
support developers. We also interviewed participants (using a questionnaire)
at the beginning and end of the experiment. Preliminary results indicate that
developers are interested in using chatbots as a support tool during software
development, with a focus on using the chatbot for task management purposes.

We plan to contribute through performing what is, to the best of our
knowledge, the first experiment that aims at capturing possible preferences of
software developers when interacting with chatbots. Additionally, we identify
as a contribution, the empirical mixed-method study (Wizard-of-Oz +
questionnaire) to evaluate the questions software developers would ask when
interacting with chatbots in their daily work.

The remainder of this paper is organized as follows: Section II presents
important related literature. Section III presents an example of a situation
where the system could be useful and how it might operate. Sections IV and V
present the study design and results, respectively. Lastly, Section VI
concludes the paper.

##  II Related Work

Researchers have studied software development contexts with a view to
supporting developers in their tasks. Automating development tasks has also
been presented as a means to improve software development, including the use
of bots in Software Engineering (SE). Another emerging solution is the use of
machine learning to manage dialogue between humans and bots. In this context,
we present Rasa, an open-source tool that uses interactive learning and other
features for building conversational software, as an example of a tool that
could support the evolution of this study.

###  II-A Context in Software Development

Research has defined context in computing as the information that is part of
an application’s operating environment and that can be sensed by the
application [4]. The same work mentions that context-aware applications, which
sense context information and modify their behavior accordingly without
explicit user intervention, are of major interest.

A broad corpus of research presents ways in which context in software
development might ease the work of a human trying to perform a task. As stated
by Murphy et al., leveraging context using smart assistants, for example,
could allow developers to remain focused on hard parts of problems while the
assistant keeps tracks of normal tasks [5]. Although latent, the context in
software engineering can also be implicit. Therefore, some studies might have
worked with the context in SE but without explicitly defining context nor
having its usage motivated [6]. Nonetheless, several studies have considered
the context in software development and identified it as ”context” [7].

Gasparic et. al. [6] propose a context model that aims at supporting and
enhancing several essential applications related to developers’ interactions
with IDEs. In this paper, we propose the use of a software development context
that is not restrained to the current IDE, but supports different and IDE-
independent development tasks. These tasks could be happening in different
stages of the development process including coding, testing or deployment.

Other research has used context in the sense of the support of software
producers’ needs while developing software [3]. This approach is interesting
as it concludes that this specific context information such as knowledge from
Stack Overflow, can be reused if documented. With the proposal in this paper,
the context can also be a source of support for developers,
indicated/recommended by the chatbot either proactively or while sensing that
the developer needs help. This information such as the support needed as a
Stack Overflow post, may or may not be available depending on the current
context of the developer.

Meyer et al. [2] have considered context in the sense of how switching between
contexts while developing can interfere in a software developer’s
productivity. This work is really interesting because it contributes to how
developers perceive their productivity, and more importantly, how switching
between tasks gives developers a sense of diminishing their productivity. In
this paper, we suggest developers’ tasks that should be done for a specific
context, guiding them to the conclusion of a more significant goal. An
approach that could guide developers back to what they have to do in case
their context suddenly changes.

In all, we aim to guide and support the human who is in-the-loop of a software
development process to execute tasks that can differ when the context is
different. Bringing humans into the loop might result in a bot that is smarter
than the ones reported in recent work [8]. The integration of bots and humans
brings a sense of partnership, where one can complement the other, while
integrating different contexts in one single interaction channel.

###  II-B Conversational Agents in Software Development

Conversational agents (CAs) have been around since the 1960s when Joseph
Weizenbaum developed ELIZA to reproduce typical human interactions between a
user and a computer [9]. CAs development assumes people interacting with
systems that are able to capture, interpret and respond in natural language,
comparable to engaging in a conversation with another human being [10].

In software development, bots are used to support different activities, from
automating repetitive tasks to bridging knowledge and communication gaps in
software teams [11]. Our proposal sits within the scope of providing means for
developers to interact with a system that is supposed to support them during
development, through a conversation where the chatbot should be aware of what
is there for the developer to do (supported by a context-model). This means
the chatbot should know both the workflow and the contextual information such
as the repository, projects and commands of the developer.

Bradley and colleagues [1] have considered a context-model for supporting
conversational developer assistants that utilize the context elements needed
to support workflow involving a distributed version control system that starts
to support the assistant concept just outlined. This design is very similar to
the proposal in this paper. The difference comes when those authors
investigate voice control agents for version control tasks. Moreover, our
intention is guiding developers in what they have to do more than executing
the task for them. We believe that the nature of capturing the different
contexts and presenting to developers the tasks related to that context is
already a significant contribution, similar to what Bradley and colleagues
have performed.

Other work has investigated two different aspects of the usage of chatbots in
the realm of software engineering: to detect code conflicts when developers
are working in parallel [12] and as an expert recommendation system to help
developers find the right person to contact within open source projects [13].
Both methods are very interesting, but differ from the approach in this paper.
We aim to identify the developers’ current context, guiding them through the
tasks that are expected to be executed in that specific context with the help
of a proactive chatbot assistant.

###  II-C Chatbots and Rasa

CAs can be classified by their mode of communicating (voice-activated, text-
based or embodied) [14] [15], and also whether their setting is general-
purpose or task-oriented. Other kinds of CAs consider usability, efficiency,
effectiveness and user satisfaction [16]. A chatbot is a software application
used to conduct an online chat conversation via text or text-to-speech, in
lieu of providing direct contact with a live human
agent111wikipedia.org/wiki/Chatbot.

Several different chatbot tools have been developed to promote conversational
support including: Botpress222botpress.com, Google
Dialogflow333dialogflow.com, Microsoft Bot Framework444dev.botframework.com,
Wit.ai555wit.ai and Amazon Lex666aws.amazon.com/lex. These tools serve a
similar purpose, which is to support the development of services for
conversational interfaces using voice or text. Modern platforms include
artificial intelligence (AI) approaches to automate communication and create
scalable personalized customer experiences [17]. These solutions, although
serving a similar purpose, might have specific properties. For future work, we
intend to use a tool that can bootstrap its operation from minimal (or no)
initial training data, and that also can manage different contexts in its
conversation. Being open-source and easy to use, as well as having decoupled
features, are also reasons why we chose Rasa777rasa.com [18] for future work.
Rasa is an open-source framework. It has two major components: Rasa NLU and
Rasa Core. Rasa NLU is responsible for natural language understanding. Rasa
Core is a framework for building a conversational chatbot. Rasa core allows
sophisticated dialogue, which is trained using interactive and supervised
machine learning. The key benefit of using the open-source NLU module is that
you don’t need to send your data outside to Google or Amazon to train on
intents and entities. Being open source, you can tweak the models or develop
models to satisfy your needs. The architecture for the Rasa NLU also allows it
to run from anywhere as a service [19].

##  III Illustrative Example and Potential System

Software development is a complex endeavor that usually involves a team of
people with different backgrounds and expertise, a variety of tools and
different domains. Developers have an end-goal, and this goal can be broken
into smaller sub-goals to be managed and achieved by the members of the team.
Of course, these sub-goals also have to comply with existing workflows.

Consider the example illustrated in [7], where Gabi is a software developer
who has been programming in Java for nine years and has recently been working
on project X. When there is a new project, one of Gabi’s goals is to create a
minimal viable product (MVP) to show her clients. She deploys the software
locally, using a container tool such as Docker and manually uploads the
project to a web server. Gabi also reboots the server manually after each
deployment, so changes become effective. This approach is faster, and she does
not have to configure a job or a server to generate a deployment
automatically, which would cause a longer wait for her clients to see the MVP.

When a version of the software in project Y, a huge mature project, has to go
to production, all Gabi does is commit the code from the local to the shared
code repository. Then, the scheduled automated job in a Jenkins server will
take care of the other steps, which are checking out the code, building the
project, uploading the package on the server and rebooting the server.

In theory, the steps for Project X and Y are the same (deploy), but because
the projects are different, Gabi’s work is different. Since the context,
namely the Project differs, the context defines the steps for the deployment
phase. If Gabi, who has been working on project Y for years, forgets she needs
to deploy Project X manually, this can create a problem. A representation of
this situation and different possible contexts are shown in Figure 1. The
information in red represents the context that can vary in this example, and
therefore, influence the workflow.

Figure 1: Example of Workflow with High and Low-Level Tasks.

This figure does not represent several other contextual attributes, including
team expertise, hardware and software technologies, IDE tools, as well as
location and time-zone. These other attributes could also influence Gabi’s
workflow.

The idea is to provide a chatbot that recognizes the developer’s current
context and is able to suggest the steps the developer should follow. The
workflow and context should be embedded in the conversational agent’s
understanding, and tracked by the chatbot. The following conversation, for
example, could occur between Gabi and the chatbot.

Chatbot: Hi!
Gabi: Hi, I need to deploy project x in development environment.
Chatbot: Sure. You should manually copy the files to server 10.1.1.1, into
folder \\\usr\\\system\\\folder1. Use your username gabidev to access the
server.
Gabi: Alright, I copied the files. What is next?
Chatbot: You should reboot the server using this command: pseudo-reboot-n. The
password is 1234567890.
Gabi: What’s next?
Chatbot: You are all done! Please check the application using the following
link localhost:8000/mvp.

Knowing all these different contexts are available and could be communicated
to developers using a chatbot, we then execute a study to understand the
preferences of software developers when using chatbots to support their work.
The next section presents the study, followed by the results.

##  IV Pilot Study Design

The goal of our experiment is to understand how software developers would use
a chatbot to facilitate daily software engineering tasks and processes. For
that, we aim to provide a chatbot interface to developers, and using an
example scenario of what developers might perform in one day and the related
questions. We would then capture these questions and analyze the ones that
were asked. In effect, we want to investigate that if developers are given a
chatbot they are willing to use for their daily work, for what kinds of tasks
would this chatbot be useful. Further, are there unexpected questions/requests
that were not anticipated. In this context, we examine the following research
question:

RQ: How well can a chatbot support software developers while executing
development tasks?

To attempt an answer to this research question, we have conducted an
experiment with graduate students as participants. First, we give participants
a scenario of a day in the life of a software developer. Then, we ask
participants to ask questions regarding any step of this scenario. We intend
to determine if there is one part of the scenario participants ask more or
fewer questions about (interest) and the types of words and questions used
(vocabulary).

###  IV-A Procedure Setup

We use a mixed-method experiment, combining Wizard-of-Oz (WOZ) [20] [21] and a
structured questionnaire. We decided to do the pilot study using WOZ to test
if there would be user engagement and understanding of the study before we
actually invested time in developing a real chatbot. The experiment is
organized in three different steps.

Step 1: a questionnaire to collect information from the participants and
explain the procedures of the study and hand them the scenario, which
participants can read. Participants can keep the scenario for reference and
read it anytime during the study.

Step 2: an experiment with the WOZ using Slack, where participants after being
presented with a scenario, are supposed to type the questions they would ask
on the chatbot interface.

Step 3: a post-study questionnaire to gather participant’s opinions regarding
the usefulness of the prototype, completing the Likert scale and gather
suggestions and impressions from participants.

###  IV-B Participants

For this study, students from the Conversational Agents Winter 2020 course
(David R. Cheriton School of Computer Science, University of Waterloo) were
recruited. For step 1, we have collected the following information from the
participants using Google Forms:

  1. 1.

Current degree: Undergrad, Masters or Ph.D. Student

  2. 2.

Background (Program and area attending)

  3. 3.

Experience with software development (in months)

  4. 4.

Knowledge regarding software repositories (Git/Jira)

After completing the demographic data, the participants were presented with a
scenario, as step 1 of the study. The scenario, which is presented next, was
created after asking seven software developers currently working in the
industry about their workday activities. They were asked to describe a day in
their office, and we have merged the answers to create a feasible, realistic,
coherent scenario.

Scenario: You arrive at your office at 8 am, sit on your chair and open the
task manager to check what tasks are assigned to you. You look for the one
task with the highest priority and read the text of the task. You understand
what you are supposed to do for this task. You have a stand-up meeting and
redefine task priority. You go to the cafeteria and grab a coffee. You’re back
to your desk and open Eclipse. You sync your local code repository to get the
newest version of code for that specific task. You look for the artifacts that
you might have to edit to complete this task. You ask your colleague what he
thinks about the task and the artifacts you decided to edit. You start coding
the test case for that new implementation and then edit the .java class that
is supposed to receive the edits. You get another coffee. You save your edits,
commit your code and create a pull request, inserting a comment in the pull
request. It’s 4 pm. You update the task assigned to you to status Done and
with the time you invested in solving the task. You leave the office.

After reading and understanding the given scenario, participants were supposed
to interact with the chatbot, by asking any question they thought to be
adequate. The chatbot is prepared to answer regarding the possible questions
related to the scenario above and deal with greetings and exceptions. All
cases with prepared answers are listed in Table I.

TABLE I: Pre-configured answers for Study. Greetings
---
Greet | Hey! What do you need to know regarding your scenario today?
Goodbye | Bye!!
Scenario
Artifacts | You need to edit class client.java and class diagram OrderMain.
Commit | OK, I am committing your code to your repository.
Tasks | Tasks 27405, 67933 and 13003 are open and assigned to you.
Priority | The task with the highest priority assigned to you is Task 13003.
Pull Request | Please, create a pull request to a branch identified with your task ID.
Code Push |  | Please, download or clone your repository to your local machine.
---
Keep it synchronized.
Sync |  | You should synchronize your repo. Click in the sync repo
---
button on your version control page.
Tool |  | You have to open Eclipse or VSCode and synchronize your code.
---
After, assign a test task to the test team.
Exceptions
No answer | Did that help you? What else can I help you with?
What can the bot do |  | I can help you discover which tasks are assigned to you,
---
their priorities, how to do a pull request and you can ask me
to commit your code.
Not in the list | Sorry, I can’t help you with that. Is there anything else I can help with?

Then we ask the participants: What questions would you be willing to ask your
assistant bot? Participants are then invited to talk to ”DevBot” on Slack.
Because it was a WOZ study, DevBot was actually one of the authors of this
paper, answering participants according to the preconfigured answers in Table
I888”What can the bot do” answer was presented either when the participant
asked explicitly what the bot could do, or after the third time the bot had to
fallback.. When participants asked for the bot to execute tasks repetitively,
the chatbot responds with ”Done.” and asks the question configured in ”No
answer”, to stimulate the participant to move forward with other questions.

For step 3 of the study, we have used the same Google Forms to ask
participants about their experience with the chatbot with which they have just
interacted. The questions about experience with the Chatbot follow a Likert
scale. The open-ended questions and the rationale behind them are listed next.

  * •

Do you think that chatbot was helpful for the questions you asked? (0 = Helped
zero tasks - 5 = Helped in all tasks): We aim to investigate the perception of
participants regarding the usefulness of the questions asked during the study.

  * •

Do you think the chatbot was useful for the presented scenario? (0 = Not
useful - 5 = Very useful): We aim to investigate the perception of
participants regarding the usefulness of the simple answers planned to be
answered during the study.

  * •

Speed of answer perception? (0= Very slow - 5 = Very fast): We aim to
investigate how the users perceive the speed of response of the bot. Because
it was a WOZ experiment, participants might perceive the bot as being slow.

  * •

What other steps do you think this chatbot could cover? (Open question): We
aim to investigate the participant’s suggestions of topics that can be
covered.

  * •

What did you like about the chatbot? (Open question): We aim to investigate
positive aspects of the chatbot.

  * •

What can be improved in the chatbot? (Open question): We aim to investigate
constructive criticism by participants.

  * •

Do you think this solution adds value? Why / why not? (Open question): We aim
to investigate if participants think the idea to have software development
supported by a smart chatbot is interesting and worthwhile.

  * •

Do you have any general comments? (Open question): We aim to investigate if
participants have other general comments regarding the interaction with the
chatbot.

None of the participants should know which questions the bot is trained to
answer, to guarantee they use their vocabulary when interacting with the bot.
However, when during the study, participants asked what the bot could do, or
when they were asking many questions the bot did not know how to answer, they
were given a list of possibilities based on the current scenario. Every
participant was presented with the same scenario. The next section describes
the study results.

##  V Study Results

###  V-A Step 1: Demographics Collection

Five students participated in the first part of the study, four masters’
students and one undergraduate. Participants are either from the Computer
Science or Software Engineering programs. Regarding their experience with
software development, only one participant reported having less than one year
experience, as three participants have between one and five years, and one
participant had more than five years of experience. Participants have either
some knowledge with software repositories (60%) or good knowledge (40%).

###  V-B Step 2: Interacting with WOZ Chatbot

After being presented with the scenario and completing the first part of the
questionnaire, participants were supposed to interact with the chatbot. The
answers were provided according to keywords in the participants’ questions,
and if no expected keyword was present, a default response (Sorry, I can’t
help you with that. Is there anything else I can help with?) was presented.
Participants interacted with the chatbot, providing an average of 15
interactions/questions. Based on the analysis of the questions, we can
highlight the following expected interactions from each participant in Table
II. The ”x” in the table represents that this participant, in particular, has
asked questions regarding what the experiment was ready to answer. The table
also shows that all participants asked unexpected questions.

TABLE II: Mapping of expected interactions. Expected Intention | P1 | P2 | P3 | P4 | P5
---|---|---|---|---|---
Greet | x | x | x | x | x
Goodbye | x | x | x | x | x
Artifacts | x | x |  | x | x
Commit | x | x |  | x | x
Priority | x | x | x | x | x
Pull Request |  | x |  | x | x
Code Push |  |  |  |  |
Sync |  | x |  |  | x
Tasks | x | x | x | x | x
Tool | x |  | x |  |
Others/fallback | x | x | x | x | x
What can the bot do | x | x |  |  | x

###  V-C Semantic Analysis

We have also attempted to extract entities through semantic analysis (KH Coder
tool) of participants’ questions. We have loaded stopwords from
https://gist.github.com/sebleier/554280. For word frequency, results of the
top 10 words are presented in Table III.

TABLE III: Word Frequency in Participants’ Questions, using KH Coder. Word | Frequency |  | Word | Frequency
---|---|---|---|---
task | 20 |  | need | 7
pull | 9 |  | artifact | 5
request | 9 |  | code | 5
DevBot | 8 |  | commit | 5
help | 8 |  | create | 5

We have also analysed the question in clusters, using Jaccard distance, Ward
method, and selecting five clusters. The clusters created relate to the
questions of: task (18 documents), pull request (9 documents), code (9
documents), greeting (3 documents) and others (40 documents). Nine documents
weren’t included in any cluster.

###  V-D Step 3: Questionnaire

After talking to the bot, participants were asked a few questions regarding
the experience with the bot. The results of the Likert-scale questions are
presented in Figures 2, 3 and 4.

Figure 2: Step 3 - Question 1. Figure 3: Step 3 - Question 2. Figure 4: Step 3
- Question 3.

As of the open questions, the results are presented next.

What other steps not covered you think this chatbot could be useful for?
Respondents have suggested including best practices for coding tips, telling
the developer what each task is, providing steps on how to perform tasks,
rather than doing it for the developer. Also, respondents have suggested that
the chatbot could also help finding out similar tasks to a given particular
task, helping to find the experts in some type of task, comparing versions of
a file across multiple past commits. Participants also mentioned the need for
the chatbot to leverage some context, such as knowing the current repository
and pull request comment templates.

What did you like about the chatbot? Participants suggested that the chatbot
was able to clarify what it can be used for, that it was ”helpful in terms of
finding my schedule and how my day looks like so I can plan ahead”. Also
another participant has mentioned DevBot ”has the potential to be a good
resource for someone who is not very familiar with software development
workflows”.

What can be improved in the chatbot? In terms of what can be improved,
participants have suggested many different aspects:

  * •

Handling more questions

  * •

Clarifying its purpose at first

  * •

Analyze and debug code

  * •

Offer best practices when coding

  * •

Know more details regarding what the developer is working on

  * •

Looking for similar tasks

  * •

Incorporate webpages (crowd knowledge support)

  * •

Being able to clarify steps or suggest alternatives instead of stating its
inability to help

One participant has given the following rich suggestion: ”The leading prompts
were helpful but it would have been better if the DevBot could lead with the
tasks it is able to complete and how a user can interact with it. Perhaps it
could have different modes depending on what git tools, IDEs a user is working
with (ie. Eclipse, VScode, vim XD) that could give specific advice for each
type of software. A user could select the level of help they require, ie. if a
user is brand new, maybe they’d prefer lots of prompts and extra resources for
reading about how version control works etc. for an advanced user, maybe
they’d prefer just the tasks and a reminder of the workflow…”

Do you think this solution adds value? Why / why not? When asked if the
chatbot adds value, most participants answered that it does adds value and
that a solution such as DevBot can be really useful for software developers.

Do you have any general comments? Respondents have raised that it would be
interesting to have the chatbot integrated into a software development
workflow as a personal assistant and adding personality to the chatbot.

###  V-E Discussion

Most of the questions asked by participants were in the subject of tasks and
how to implement development steps. Therefore, there are indications that
developers are interested in being guided in their tasks rather than having a
chatbot that can execute tasks on their behalf. We find this interesting, as
managing tasks is not essential to contributing code per se but is perceived
to be important, hence there might be this gap in expectations between CA
designers and users that must be taken into account.

Most respondents have suggested that adding context to the chatbot would be
interesting. For example, the chatbot should know the repository address,
branch or patterns expected to commit messages. However, the simple study with
a few expected questions and given a simple scenario was also indicated as
being helpful by most of the participants. Based on feedback from
participants, adding context to the chatbot could immensely improve the
chatbot’s capabilities, as it would be personalized. The chatbot could also
incorporate discovery of context or recommendations based on the history of
interactions.

Participant P1, the least experienced developer (less than 1 year according to
demographics), was the participant that indicated a low score when asked if
the chatbot was helpful for the questions asked. This can indicate that the
chatbot has to be prepared to give more details about the process to less
experienced users. All other participants, who have more than one year of
experience, indicated that the chatbot was helpful.

Two participants have asked questions regarding the tool, interestingly, the
ones that have not done questions regarding pull requests. We believe that
this can indicate less experience developing software and interacting with Git
as version control repositories. When analyzing if participants took time to
create the questions, the perception was that the questions were created
quickly, as the study had an average of 15 questions asked in an average
period of 10.8 minutes of chatting with the bot actively.

All participants have greeted the bot at the beginning of the interaction and
when leaving the chat. Only one participant realized it was a WOZ experiment.
This demonstrates that the bot should be able to handle greeting interactions.
None of the participants asked questions regarding pushing code to their local
repository or pulling, merging or cloning. This could indicate developers are
not worried about these tasks, that are done less often during development.
However, most participants asked about pull requests (requesting the master
repository’s owner to pull files from your repository), which can happen quite
often during software development.

##  VI Conclusion

We have presented an exploratory study regarding the preferences of software
developers when interacting with a chatbot to support work. Developers usually
work with various tools in a very dynamic situation, and promoting ways to
support them is vital for the quality of their work. As well, a chatbot could
improve productivity, lessen training times and make implicit preferences
explicit through capture processes. Our results demonstrate developers are
willing to work with such tools and have found this solution to be
interesting, providing ideas for the future and hints on how chatbots could
behave and connect.

As future work, we intend to implement a real chatbot using Rasa and run this
study with more participants, while also developing a context model that is
capable of supporting the interactions with developers, storing their work
variables (context) as indicated by the participants of this pilot study and
the literature. We believe such types of solutions can be useful for teams
that work in shared office space and especially for teams that work remotely.

## Acknowledgements

The authors thank the Natural Sciences and Engineering Research Council of
Canada (NSERC), the MITACS Accelerate Program, and the Centre for Community
Mapping (COMAP).

## References

  * [1] N. Bradley, T. Fritz, and R. Holmes, “Context-aware conversational developer assistants,” in _2018 IEEE ACM 40th International Conference on Software Engineering (ICSE)_. IEEE, 2018, pp. 993–1003.
  * [2] A. N. Meyer, L. E. Barton, G. C. Murphy, T. Zimmermann, and T. Fritz, “The work life of developers: Activities, switches and perceived productivity,” _IEEE Transactions on Software Engineering_ , vol. 43, no. 12, pp. 1178–1193, 2017.
  * [3] G. Melo, T. Oliveira, P. Alencar, and D. Cowan, “Retrieving curated stack overflow posts from project task similarities,” in _International Conference on Software Engineering Knowledge Engineering_ , 2019, pp. 415–418.
  * [4] D. Salber, A. K. Dey, and G. D. Abowd, “The context toolkit: aiding the development of context-enabled applications,” in _Proceedings of the SIGCHI conference on Human Factors in Computing Systems_ , 1999, pp. 434–441.
  * [5] G. C. Murphy, “Beyond integrated development environments: adding context to software development,” in _2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)_. IEEE, 2019, pp. 73–76.
  * [6] M. Gasparic, G. C. Murphy, and F. Ricci, “A context model for ide-based recommendation systems,” _Journal of Systems and Software_ , vol. 128, pp. 200–219, 2017.
  * [7] G. Melo, P. Alencar, and D. Cowan, “Context-augmented software development in traditional and big data projects: Literature review and preliminary framework,” in _2019 IEEE International Conference on Big Data (Big Data)_. IEEE, 2019, pp. 3449–3457.
  * [8] M. Wessel, B. M. de Souza, I. Steinmacher, I. S. Wiese, I. Polato, A. P. Chaves, and M. A. Gerosa, “The power of bots: Characterizing and understanding bots in oss projects,” _Proceedings of the ACM on Human-Computer Interaction_ , vol. 2, no. CSCW, Nov. 2018. [Online]. Available: https://doi.org/10.1145/3274451
  * [9] J. Weizenbaum, “Eliza—a computer program for the study of natural language communication between man and machine,” _Communications of the ACM_ , vol. 9, no. 1, pp. 36–45, 1966.
  * [10] M. F. McTear, “The rise of the conversational interface: A new kid on the block?” in _International Workshop on Future and Emerging Trends in Language Technology_. Springer, 2016, pp. 38–49.
  * [11] M.-A. Storey and A. Zagalsky, “Disrupting developer productivity one bot at a time,” in _Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering_ , 2016, pp. 928–931.
  * [12] E. Paikari, J. Choi, S. Kim, S. Baek, M. Kim, S. Lee, C. Han, Y. Kim, K. Ahn, C. Cheong _et al._ , “A chatbot for conflict detection and resolution,” in _2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE)_. IEEE, 2019, pp. 29–33.
  * [13] J. Cerezo, J. Kubelka, R. Robbes, and A. Bergel, “Building an expert recommender chatbot,” in _2019 IEEE/ACM 1st International Workshop on Bots in Software Engineering (BotSE)_. IEEE, 2019, pp. 59–63.
  * [14] U. Gnewuch, S. Morana, and A. Maedche, “Towards designing cooperative and social conversational agents for customer service.” in _Proceedings of the International Conference on Information Systems (ICIS)_ , 2017.
  * [15] C. Lee, S. Jung, S. Kim, and G. G. Lee, “Example-based dialog modeling for practical multi-domain dialog system,” _Speech Communication_ , vol. 51, no. 5, pp. 466–484, 2009.
  * [16] N. M. Radziwill and M. C. Benton, “Evaluating quality of chatbots and intelligent conversational agents,” _arXiv preprint arXiv:1704.04579_ , 2017\.
  * [17] J. Brenier and J. Prial, “An overview of conversational ai,” May 2019, accessed on = 2020-02-27. [Online]. Available: https://georgianpartners.com/investment-thesis-areas/overview-conversational-ai/
  * [18] T. Bocklisch, J. Faulkner, N. Pawlowski, and A. Nichol, “Rasa: Open source language understanding and dialogue management,” _arXiv preprint arXiv:1712.05181_ , 2017.
  * [19] A. Singh, K. Ramasubramanian, and S. Shivam, “Introduction to microsoft bot, rasa, and google dialogflow,” in _Building an Enterprise Chatbot_. Springer, 2019, pp. 281–302.
  * [20] J. D. Gould, J. Conti, and T. Hovanyecz, “Composing letters with a simulated listening typewriter,” _Commun. ACM_ , vol. 26, no. 4, p. 295–308, Apr. 1983. [Online]. Available: https://doi.org/10.1145/2163.358100
  * [21] J. Kelley, “Where did the usability term wizard of oz come from.” [Online]. Available: http://www.musicman.net/oz.html


# Analysis of Branch Specialization and its Application in Image Decomposition

David S. Hippocampus
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213
<EMAIL_ADDRESS>
Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies. Jonathan
Brokman, Guy Gilboa

###### Abstract

Branched neural networks have been used extensively for a variety of tasks.
Branches are sub-parts of the model that perform independent processing
followed by aggregation. It is known that this setting induces a phenomenon
called Branch Specialization, where different branches become experts in
different sub-tasks. Such observations were qualitative by nature. In this
work, we present a methodological analysis of Branch Specialization. We
explain the role of gradient descent in this phenomenon, both experimentally
and mathematically. We show that branched generative networks naturally
decompose animal images to meaningful channels of fur, whiskers and spots and
face images to channels such as different illumination components and face
parts.

##  1 Introduction

The use of branching is a common practice in many neural network
architectures. From ensemble learning in the 90’s [17], [18], to grouped
convolutional blocks of the classification era [21], [32], [29], to the
sparsely gated mixture of experts of today [28], [34], [11].

A branched model propagates information to several processing units called
branches, usually of identical architecture. The branches do not communicate
with each other and produce outputs which are then combined and aggregated.

In the case of ensembles, where each sub-network is trained separately - [12]
shown a typical disagreement between the sub-networks’ predictions, even
though performace of each sub-network on the task at hand (image
classification) was similar. In their experiments they observed that two
disagreeing sub-networks where attained at two disjoint basins of attraction
of the loss landscape.

In the case of mixture of experts, which are trendy today, the branches are
trained together alongside a routing function - which selects the relevant
branch(es) for the selected input and task. Different normalization and
routing techniques are used to ensure that each branch learns a specific
specializations. Nevertheless it has been observed early on, for example in
AlexNet [21], that without any type of regularization or routing, branched CNN
sub-modules produce specialized filters, where each branch specializes in
distinct image features. In this work we would like to address this
phenomenon, characterize the specialization effect in branching and provide
preliminary mathematical insights.

We restrict ourselves to a setting of cumulative aggregation as follows. Let
$X=\\{x_{1},..\,,x_{N}\\}$, $Y=\\{y_{1},..\,,y_{N}\\}$ be sets of $N$ inputs
and corresponding desired outputs (labels or target images). We have $M$
branches of sub neural-networks, identical in their architecture.

Figure 1: We examine a general branched neural network structure as
illustrated above. Each branch consists of a sub neural network (blue). The
response of all branches is summed to produce the global network output $f$.
All sub neural networks have exactly the same architecture. The only
difference is their different random initialization at the training phase.

For an input $x$ the output of each branch is $v_{k}(x)$ and the combined
network output is

$f_{\theta}(x)=\sum_{k=1}^{M}v_{k}(x),$ (1)

where $\theta$ is the set of model parameters. See Fig. 1 for an illustration.
From associativity and cummutativity of summation the role of each branch is
identical, in principle. The only difference is the different random parameter
initialization of each sub-network, which occurs at the beginning of the
training process. Nevertheless, this seemingly minor difference has a
remarkable effect on the role of the branch at inference. Through the gradient
descent process, the branches spontaneously tend to specialize in certain
tasks of the global network goal, some become silent or “null-networks” which
produce essentially zero output for any input $x$. Moreover, the correlation
between the branches output is low and the tendency is not to share the same
task by several branches. We show that this phenomenon is fundamental to
branched architecture and happens from the smallest single scalar perceptron
element, through standard CNN classifiers, to complex generative networks. In
the generative case, we obtain an unsupervised decomposition of an image into
various meaningful channels like mouth and eyes, lightings of specular and
diffusive components, and different fur textures and color patterns in
animals. We sketch preliminary possible applications that such decompositions
might yield.

##  2 Previous work

Throughout the last decade, Branch Specialization was repeatedly observed in
various deep learning architectures and tasks, for instance [21], [28], [34],
[31].

Splitting networks to independent sub-parts dates back to [17], [18]. It was
popularized for CNNs by AlexNet [21], which introduced grouped convolutions.
This trend continued to re-appear in classification models [29], [32].

Conditional computation, first suggested in [2], [9] splits the model to sub-
parts too: A gating mechanism selects sub-parts to be used, conditioned on the
input. This is efficient, since only a these parts are used for inference.
Today its used both for NLP and for vision tasks, e.g. [19], [34], [28], [11]
[30], [25], [14], [4], [33], [24].

Recently [31] have shown how dimensionality reduction techniques cluster
convolutional kernels of branched classifiers by their branch. They termed
this phenomenon _Branch Specialization_ , a term which we adopt.

In this work we systematically analyze branched models, and the naturally-
occurring interactions between their branches. We analyze how branching
affects gradient distribution, and experiment with several tasks of various
degrees of complexity.

##  3 Notations and Preliminaries

Denote the data domain $X$, a sample $x\in X$, a loss function $L(x)$, and a
parameter-dependent function model $f_{\theta}:X\rightarrow\mathbb{R}^{C}$,
where $\theta$ denotes the parameters and $C$ is the output dimension. We
assume the model is branched with $M$ branches as in Eq. (1). Branch $k$ is
$v_{k}:X\rightarrow\mathbb{R}^{C}$, and depends on parameters
$\theta_{k}\subset\theta$. Each branch is a neural-network, where different
branches have identical architecture and differ by their parameters.

A reminder of common losses and their derivatives: For
$L^{2}(x)=\frac{1}{2}||y-f_{\theta}(x)||_{2}^{2}$, we have

$\frac{dL^{2}}{df_{\theta}(x)}=y-f_{\theta}(x).$ (2)

For $L_{cross\,entropy}(x)=\sum_{c}p(x,c)log(s(f_{\theta}(x,c)))$, where $s$
is the softmax operator we get

$\frac{dL_{cross\,entropy}}{df_{\theta}(x)}=p(x)-s(f_{\theta}(x)).$ (3)

##  4 Main Observations and Toy Examples

We summarize below the main observations on branching that are supported by
our experiments (detailed hereafter).

  1. 1.

Each network tends to specialize in certain aspects of the global problem. We
refer to that as region of specialization (ROS). In classification it can be
certain features belonging to certain classes. In image synthesis and
generation - certain image characteristic (structural or semantic).

  2. 2.

The networks tend not to share ROS’s. That is, in each ROS very few networks
are active (and often only a single one is active) .

  3. 3.

In order not to interfere in solving the global task, each network is silent
(inactive) in most ROS’s.

  4. 4.

There may be completely silent networks, which are null (close to zero
response) for any input data. This typically happens when the global task can
be solved well by less than the total number of branches of the network.

  5. 5.

The response of each branch is lowly correlated to the response of other
branches.

  6. 6.

The full network learning capacity grows with the number of branches $M$, as
can be expected. However, the number of active branches may be almost constant
for different $M$, if $M$ is large enough.

  7. 7.

The specialization process happens naturally through gradient descent
optimization. We

     * •

provide theoretical support for that by analyzing the Hessian matrix.

  8. 8.

The properties above happen at all levels of complexity, from the neural level
to very large, highly parameterized networks, in each branch.

###  4.1 Toy Examples

We present here two very simple one dimensional toy examples. In both cases
each branch $k$ is a neural component of the form:

$v_{k}(x)=\sigma(w_{k}\cdot x+b_{k}),$ (4)

where $\sigma$ is a nonlinear activation function (we use Leaky ReLU), $w_{k}$
is a scalar weight parameter and $b_{k}$ is a scalar bias parameter. Our
combined network consists of $M$ such branches, where the network output
$f_{\theta}$ is defined in (1). The training set is composed of $N=4$ examples
with an input set $X=\\{x_{1},..\,\,x_{N}\\}$ and a corresponding desired
output set $Y=\\{y_{1},..\,\,y_{N}\\}$. The first toy example $Toy1$ consists
of the training set $X=\\{-1,0,0,1\\}$, $Y=\\{1,0.5,0.5,1\\}$. It is somewhat
similar to a one-dimensional XOR problem, often used as toy data that a linear
model cannot solve [3]. To approximate a classification problem we have a
balanced set of both classes $1$ and $0.5$. Note that we have not chosen in
$Y$ a target value of $0$ since this value has a special role of null
contribution in a sum, which we want to examine. The second toy problem $Toy2$
models a regression problem where $X=\\{-1,0,1,2\\}$ and
$Y=\\{1,0.25,0.5,0.75\\}$. A square $L^{2}$ loss is used
$L=\frac{1}{2}\|f-y\|_{2}^{2}$. Training is done using gradient descent. In
Fig. 2 we show two examples of the dynamics along the training for $Toy1$. In
the caption some typical phenomena are explained. For this task we define
success of the training if the loss is close to zero
($\sum_{j=1}^{N}(f(x_{j})-y_{j})^{2}<\delta$, we chose $\delta=0.0001$). Thus
we can measure success rate. This is shown in Fig. 3 (left). For a network
configuration, with fixed $M$ (in the range 2 to 30), we train the network
1000 times with different random initialization. It is clearly seen that as
the number of branches increases - the optimization task is easier (success
rate increases monotonically) and around $100\%$ success rate is reached for
10 branches or more.

The output of the branches is also loosely correlated. We define the response
matrix (of size $M\times N$) for all training set and all branches by $F$,
where a matrix element at row $j$ and column $i$ is $F_{i,j}=v_{i}(x_{j})$.
The covariance matrix of the branches response is $Cov=F\cdot(F)^{T}$, where
$T$ denotes transpose. Examples for $M=10$ are shown in Fig. 3 (right),
showing generally low covariance.

Let us examine the gradient descent process. The gradient for some weight
parameter $w_{i}$ for a certain data element $x_{j}$ is

$\nabla_{w_{i}}L(x_{j})=(f(x_{j})-y_{j})\cdot(\sigma^{\prime}(x_{j}w_{i}+b_{i})x_{j},$
(5)

where $\sigma^{\prime}(q)=d\sigma(q)/dq$. We can write it as,

$\nabla_{w_{i}}L(x_{j})=D_{collab}\cdot D_{dist},$ (6)

where $D_{collab}=(f(x_{j})-y_{j})$ is the collaborative part and
$D_{dist}=(\sigma^{\prime}(x_{j}w_{i}+b_{i})x_{j}$ is the distributive part.
The collaborative part ensures that whenever the task is solved for this
specific data element by the entire network, this element will not affect the
parameters of the network. The distributive part becomes relevant when the
collaborative part is not negligible and attempts to reduce the loss locally
for the specific branch. We will later see that this characteristics can be
generalized and tend to induce specialization. Some examples and statistics of
the two Toy examples are shown and explained in Figs. 2, 3 and 4.

Figure 2: Dynamics of Toy Example 1 with 2 branches ($M=2$). Two cases are
shown, left - a more common case of specialization and inactivity of the other
branch in each region. On the right we see two more rare phenomena: first for
$x=1$ (yellow) both branches are inactive until the second branch
spontaneously “takes responsibility” for this task. Second, sharing the task
for $x=0$, where both branches are active.

Figure 3: Toy 1. Left - success rate as a function of the number of branches
in the network (1000 trials are performed for each branch configuration).
Learning capacity grows with $M$ (although the number of active branches is
similar, see below). Right - 10 examples of covariance matrices for 10
branches. The output of the different branches is loosely correlated.

Figure 4: Toy 2. Left and middle - an example of the response of 20 branches.
Right - average number of active branches for different $M$. We define an
active branch as one where the norm of its response is at least $10\%$ of the
maximal branch response norm. Silent branches are the complement of active. On
the left we have an example of 3 active branches. The number of active
branches is almost constant, such that silent branches grow linearly with $M$.

##  5 Loss Gradient and Hessian of Branched Models

In the setting of Sec. 3, $f_{\theta}(x)\in\mathbb{R}^{C}$ is optimized w.r.t.
a loss $L(x)$ on data $x\in X$. We analyze gradient descent (GD) optimization
as the following nonlinear dynamical system,

$\dot{\theta}(t)=-\nabla_{\theta(t)}L(x),$ (7)

where $t$ is the time variable, and
$\dot{\theta}=\frac{\partial\theta}{\partial t}$. The trained model is
attained at an equilibrium point, $\dot{\theta}=0$. For stochastic GD (SGD)
this translates to $\nabla_{\theta(t)}L(x)\approx 0,\,\forall x\in X$. The
fixed point stability is characterized by the Jacobian of $\dot{\theta}$, in
our case the Hessian matrix of $L(x)$ w.r.t $\theta$. This type of analysis is
common, e.g. [22], [26], [27], [1], [10].

###  5.1 Gradient

Denote
$\nabla_{\theta_{k}}v_{k}(x),\,\nabla_{\theta_{k}}f(x)\in\mathbb{R}^{C\times|\theta_{k}|}$
Jacobian matrices w.r.t. $\theta_{k}$. By the chain rule

$\nabla_{\theta_{k}}L(x)=\left(\nabla_{\theta_{k}}f_{\theta}(x)\right)^{T}\cdot\left(\frac{dL}{df_{\theta}(x)}\right),$
(8)

plugging Eq. (1) we have by linearity

$\nabla_{\theta_{k}}L(x)=\left(\sum_{l=1}^{M}\nabla_{\theta_{k}}v_{l}(x)\right)^{T}\cdot\left(\frac{dL}{df_{\theta}(x)}\right),$
(9)

and since $\theta_{k}\cap\theta_{l}=\emptyset\forall k\neq l$ we have

$\nabla_{\theta_{k}}L(x)=\underbrace{\left(\nabla_{\theta_{k}}v_{k}(x)\right)^{T}}_{\text{distributive}}\cdot\underbrace{\left(\frac{dL}{df_{\theta}(x)}\right)}_{\text{collaborative}},$
(10)

or equivalently, for a single parameter $w_{k}\in\theta_{k}$,

$\frac{\partial L(x)}{\partial w_{k}}=\underbrace{\left(\frac{\partial
v_{k}(x)}{\partial
w_{k}}\right)^{T}}_{\text{distributive}}\cdot\underbrace{\left(\frac{dL}{df_{\theta}(x)}\right)}_{\text{collaborative}}.$
(11)

Gradient is factorized to distributive and collaborative parts, affecting a
branch or the whole model.

###  5.2 Hessian

Let $w_{k}\in\theta_{k},\,w_{l}\in\theta_{l}$. Using Eq. (11) we get

$\frac{\partial^{2}L(x)}{\partial w_{k}\partial
w_{l}}=\frac{\partial}{\partial w_{l}}[\left(\frac{\partial v_{k}(x)}{\partial
w_{k}}\right)^{T}\cdot\left(\frac{dL}{df_{\theta}(x)}\right)].$ (12)

Case 1: If $\theta_{l}=\theta_{k}$ (but $w_{k}$ might be different than
$w_{l}$) we get

$\frac{\partial^{2}L(x)}{\partial w_{k}\partial
w_{l}}=\left(\frac{\partial}{\partial w_{l}}[\frac{\partial v_{k}(x)}{\partial
w_{k}}]\right)^{T}\cdot\left(\frac{dL}{df_{\theta}(x)}\right)+\left(\frac{\partial
v_{k}(x)}{\partial w_{k}}\right)^{T}\cdot\left(\frac{\partial}{\partial
w_{l}}[\frac{dL}{df_{\theta}(x)}]\right).$ (13)

Case 2:

$\theta_{l}\neq\theta_{k}\Rightarrow\frac{\partial^{2}L(x)}{\partial
w_{k}\partial w_{l}}=\underbrace{\left(\frac{\partial v_{k}(x)}{\partial
w_{k}}\right)^{T}}_{\text{distributive}}\cdot\left(\frac{\partial}{\partial
w_{l}}[\underbrace{\frac{dL}{df_{\theta}(x)}}_{\text{collaborative}}]\right).$
(14)

Note again the distributive and collaborative terms. When the distributive
part of the gradient is zero, case 2 is zero, i.e. the Hessian becomes block
diagonal, where a block corresponds to a branch.

##  6 Image classification

Here we train our neural networks on the CIFAR-10 dataset [20]. For this task,
each branch $v_{k}$ is a "slimmed down" version of ResNet18 [16] as
implemented at [15] with $|\theta_{k}|=0.08\cdot 10^{6}$ parameters. We sum
$M=16$ branches, thus $|\theta|=M\times|\theta_{k}|=1.3\cdot 10^{6}$. For
reference - ResNet18 uses about $11.2\times 10^{6}$ parameters. Optimization
is done on the cross-entropy loss, between ground truth and learnt probability
density functions (PDF) $p(x),q_{\theta}(x)$. Summing an ensemble of logit
vectors results with a valid logits vector, in contrast to PDF summation.
Hence we apply softmax on $f_{\theta}(x)$, which makes summation of
$\\{v_{k}(x)\\}_{k}$ valid.

For ease of analysis, we clamp $f_{\theta}(x)$ to $[-1,1]\subset\mathbb{R}$.
This restricts the logits from diverging to uncontrollably large values, which
are hard to predict and analyze. Thus $p(x)$ is not a one-hot vector, but a
softmax on the class-indicating $\pm 1$ values.

Following training, Branch Specialization indeed occurs as can be observed in
Fig. 6. As in the toy examples - some branches become experts in specific
classes, while others are "turned off" and do not contribute to
classification. In Fig. 5 we see that the covariance of branches is also
consistent with the toy example. As can be seen in Fig. 7 \- different
branches are confident about different cases. In Fig. 8 it is shown that
branches specialize in certain characteristics within each class, here Branch
10 specialize in closer animals and Branch 14 in further ones.

Figure 5: Classification. From left to right: 1) Logarithmic No. of Active and
Silent Branches as function of $2^{M}$ where active branches are defined as in
Fig. 4. 2) Inter-branch covariance for $M=16$. 3-4) Hessian w.r.t. parameters
from the first layer of all 16 branches (at initialization and at training
convergence). An almost block-diagonal Hessian is obtained, with 16 blocks
corresponding to the 16 branches. Figure 6: Classification, output by branch:
Before summation, each branch produces its contribution to the classification
inference. Branches differ by the amount of contribution - they range from
highly important contributors to being completely turned off (Silent
branches). The branches also differ by the classes they specialize in (but
have also intra-class specialization). Figure 7: From each class, 20 samples
were randomly selected, and the confidence of each branch in correctly
classifying each sample is shown. Larger positive values (yellow) indicate
greater true confidence, and negative values (blue) - false confidence.
Processing is distributed differently between branches, depending on the input
samples. We obtain both inter-class and intra-class specialization. Figure 8:
Intra-class specialization. We examine the two most informative branches - 10
and 14 (see Fig. 6) on classes Horse, Deer and Bird. These classes have their
classification inference distributed between several branches (see Fig. 7).
For each branch-class pair , the confidence of all test samples was assessed,
then samples with top $2\%$ confidence where selected for this figure. Samples
are sorted by their confidence from left (highest confidence) to right. This
exposes that branch 10 mainly specializes in classifying head close-up shots
of horses, deer and large birds, while branch 14 specializes in whole body
shots within these classes. Specifically - branch 14 seems to be more
confident about horses with their equestrians.

##  7 Class Transfer GAN

So far we saw that different branches solve the task at hand from different
aspects. Here we show how specialization is manifested in a generative model.

We adapt StarGAN-v2 [7], and train our neural network on the same datasets
AFHQ [8], and Celeb-a-HQ [23]. While [7] was designed for diverse image style-
transfer, we restrict our discussion to non-diverse style transfer, or class
transfer. Following Eq. (1), $f_{\theta}$ is now a Generator network, where
each branch $v_{k}$ is a "slim version" of StarGAN-v2’s Generator, using $M=9$
branches.

Figure 9: Celeb-a-HQ and AFHQ re-generated images: High quality and input
fidelity are attained.

Figure 10: Decomposition of faces. Top: the output of the 9 branches (with
increased contrast). The sum of all branches produce the generated images,
Fig. 9. Bottom: each branch output (channel) is added (without amplification)
to the generated image, to visualize its role. We clearly observe each branch
is specializing in different image characteristics.

Figure 11: Decomposition of animal images as outputs of the 9 branches (with
increased contrast). The sum of all branches produce the generated images in
Fig. 9, excluding $R$.

We train our NN as a GAN, but do not use any of the additional loss terms
originally used by StarGAN-v2. Cycle loss is discarded as well, and fidelity
to the input image is attained by slightly modifying Eq. (1) as follows:
$f_{\theta}(x)=R+\sum_{k=1}^{M}v_{k}(x)$, where $R$ is an over-smoothed
version of $x$.

We added an optional learnt nonlinear diffusion step, which produces $R$ and
pre- processes the inputs. This results in higher quality decompositions,
nevertheless - image decomposition occurs also without this step, as shown in
the supplementary material, where full details of the architecture and loss
are provided as well.

As in previous experiments, different branches specialize w.r.t. a different
aspect of the task at hand - in this case it is generating natural images. By
construction - $v_{k}(x)$ are ambient-space representations, summed to
generate the image - hence we say that the branch outputs are "decompositions
of the image". Remarkably, these specializations are mostly interpretable:
Celeb-a-HQ images seem to decompose to specular light reflection, diffusive
light reflection, color hue, and texture. AFHQ images are decomposed to color
patterns, different fur textures, whiskers and contours, color hue, eyes and
nose. See Fig. 9 for re-generated images, and Fig. 10 for their learnt
specialized decompositions.

These decompositions may be useful for a variety of tasks. As an example, we
devised a segmentation procedure for cat’s eyes, see demonstration in Fig. 12,
and Full details in the supplementary. Further applications can include 3D
shape recovery from lighting, random channel perturbations for data
augmentation, image manipulation and filtering and more (see the supplementary
for some examples).

Figure 12: Segmentation of cat’s eyes. Only class labels (dog/ cat/ wild) are
used with no segmentation labels. Right: Decompositions are used for inputs of
Chan-Vese segmentation algorithm [5].

##  8 Discussion of Branch Equilibrium

Here we extend Sec. 5. A branch $v_{k}$ attains equilibrium when
$\dot{\theta}_{k}=0$. In SGD this happens when
$\nabla_{\theta_{k}}L(x)=0,\,\forall x\in X$. Denote
$D_{collab}(x)=\frac{dL}{df_{\theta}(x)}\in\mathbb{R}^{C}$ and
$D_{dist}^{k}(x)=\nabla_{\theta_{k}}v_{k}(x)\in\mathbb{R}^{C\times|\theta_{k}|}$.

Consider $D_{collab}(x)=0$. We make the sensible assumption that this happens
for a small subset of $X$. Explanation - plugging Eq. (3) or (2) to
$D_{collab}(x)=0$ translates to a perfect fit by the model. Usually in machine
learning, a successfully trained model does not fit perfectly most of of the
data.

Consider $\nabla_{\theta_{k}}L(x)=0,\,D_{collab}(x)\neq 0$. Then by Eq. (10),
$D_{dist}^{k}(x)$’s columns lie in a $C-1$ dimensional plane, orthogonal to
$D_{collab}$. Because $|\theta_{k}|\gg C$, we assume orthogonality is attained
for a small subset of the columns, and the rest are zero columns. Otherwise we
get high linear dependency between the columns in the near neighbourhood of
equilibrium, which is usually unlikely.

Re-phrasing the above, when $\nabla_{\theta_{k}}L(x)=0$ is attained, Eq. (11)
has a zero distributive part for most of $(w_{k},x)$ pairs. Under this
mechanism different branches may reach equilibrium independently. There are
many open problems, such as when is linear dependency of the gradients
expected? How much data do we expect to fit perfectly?

To conclude, gradients of a branched model are split to distributive and
collaborative factors (10). Inter-branch Hessian entries have the same
distributive factor (14). We conjecture the distributive part is the main
factor in driving stability. In such a case - the Hessian is block diagonal,
where blocks describe branches, i.e. perturbations in one branch do not cause
loss gradients to change any of the other branches. Thus each branch minimizes
the loss function from an independent aspect.

##  9 Conclusion

Branch Specialization is observed in all experiments as different branches
naturally solve different sub-tasks of the global task at hand. In the context
of classification, we show that different branches specialize at classifying
different types of data samples of the same class (such as close horses in one
branch and far-away horses in another). In the context of generative models,
we found that specialization is manifested as a first of its kind image
decomposition. We envision these decompositions can be harnessed for self-
supervised learning - as we demonstrate for segmentation. Preliminary analysis
of gradient descent on branched architectures has shown that (under sensible
assumptions) there is a strong inclination towards independent local minima of
each branch, which do not interfere with the optimization process of other
branches.

## References

  * Arora et al. [2018] Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu.  A convergence analysis of gradient descent for deep linear neural networks.  _arXiv preprint arXiv:1810.02281_ , 2018.
  * Bengio [2013] Yoshua Bengio.  Deep learning of representations: Looking forward.  In _International conference on statistical language and speech processing_ , pages 1–37. Springer, 2013.
  * Brutzkus and Globerson [2019] Alon Brutzkus and Amir Globerson.  Why do larger models generalize better? a theoretical perspective via the xor problem.  In _International Conference on Machine Learning_ , pages 822–830. PMLR, 2019.
  * Cai et al. [2021] Shaofeng Cai, Yao Shu, and Wei Wang.  Dynamic routing networks.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 3588–3597, 2021.
  * Chan and Vese [2001] Tony F Chan and Luminita A Vese.  Active contours without edges.  _IEEE Transactions on image processing_ , 10(2):266–277, 2001.
  * Chen and Pock [2016] Yunjin Chen and Thomas Pock.  Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration.  _IEEE transactions on pattern analysis and machine intelligence_ , 39(6):1256–1272, 2016.
  * Choi et al. [2020a] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8188–8197, 2020a.
  * Choi et al. [2020b] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  pytorch-stargan-v2-afhq.  https://github.com/clovaai/stargan-v2, 2020b.
  * Davis and Arel [2013] Andrew Davis and Itamar Arel.  Low-rank approximations for conditional feedforward computation in deep neural networks.  _arXiv preprint arXiv:1312.4461_ , 2013.
  * Du et al. [2018] Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh.  Gradient descent provably optimizes over-parameterized neural networks.  _arXiv preprint arXiv:1810.02054_ , 2018.
  * Fedus et al. [2021] William Fedus, Barret Zoph, and Noam Shazeer.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _arXiv preprint arXiv:2101.03961_ , 2021.
  * Fort et al. [2019] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan.  Deep ensembles: A loss landscape perspective.  _arXiv preprint arXiv:1912.02757_ , 2019.
  * Gilboa [2014] Guy Gilboa.  A total variation spectral framework for scale and texture analysis.  _SIAM journal on Imaging Sciences_ , 7(4):1937–1961, 2014.
  * Gregory et al. [2021] Stephen Gregory, Hu Cheng, Sharlene Newman, and Yu Gan.  Hydranet: a multi-branch convolutional neural network architecture for mri denoising.  In _Medical Imaging 2021: Image Processing_ , volume 11596, pages 881–889. SPIE, 2021.
  * Hangzhou et al. [2019] Kuangliu Hangzhou, Wei Yang, Yang Peiwen, and Felipe Ducau.  pytorch-cifar.  https://github.com/kuangliu/pytorch-cifar, 2019.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Jacobs et al. [1991] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton.  Adaptive mixtures of local experts.  _Neural computation_ , 3(1):79–87, 1991.
  * Jordan and Jacobs [1994] Michael I Jordan and Robert A Jacobs.  Hierarchical mixtures of experts and the em algorithm.  _Neural computation_ , 6(2):181–214, 1994.
  * Kirsch et al. [2018] Louis Kirsch, Julius Kunze, and David Barber.  Modular networks: Learning to decompose neural computation.  _Advances in neural information processing systems_ , 31, 2018.
  * [20] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.  Cifar-10 (canadian institute for advanced research).  URL http://www.cs.toronto.edu/~kriz/cifar.html.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Li et al. [2020] Xinyan Li, Qilong Gu, Yingxue Zhou, Tiancong Chen, and Arindam Banerjee.  Hessian based analysis of sgd for deep nets: Dynamics and generalization.  In _Proceedings of the 2020 SIAM International Conference on Data Mining_ , pages 190–198. SIAM, 2020.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , December 2015.
  * Ren et al. [2022] Pengzhen Ren, Changlin Li, Guangrun Wang, Yun Xiao, and Qing Du Xiaodan Liang Xiaojun Chang.  Beyond fixation: Dynamic window visual transformer.  _arXiv preprint arXiv:2203.12856_ , 2022.
  * Rosenbaum et al. [2017] Clemens Rosenbaum, Tim Klinger, and Matthew Riemer.  Routing networks: Adaptive selection of non-linear functions for multi-task learning.  _arXiv preprint arXiv:1711.01239_ , 2017.
  * Sagun et al. [2016] Levent Sagun, Leon Bottou, and Yann LeCun.  Eigenvalues of the hessian in deep learning: Singularity and beyond.  _arXiv preprint arXiv:1611.07476_ , 2016.
  * Sagun et al. [2017] Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou.  Empirical analysis of the hessian of over-parametrized neural networks.  _arXiv preprint arXiv:1706.04454_ , 2017.
  * Shazeer et al. [2017] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  _arXiv preprint arXiv:1701.06538_ , 2017.
  * Szegedy et al. [2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1–9, 2015.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Voss et al. [2021] Chelsea Voss, Gabriel Goh, Nick Cammarata, Michael Petrov, Ludwig Schubert, and Chris Olah.  Branch specialization.  _Distill_ , 6(4):e00024–008, 2021.
  * Xie et al. [2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1492–1500, 2017.
  * Xu et al. [2021] Boqiang Xu, Jian Liang, Lingxiao He, and Zhenan Sun.  Meta: Mimicking embedding via others’ aggregation for generalizable person re-identification.  _arXiv preprint arXiv:2112.08684_ , 2021.
  * Zoph et al. [2022] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus.  Designing effective sparse expert models.  _arXiv preprint arXiv:2202.08906_ , 2022.

##  Appendix A Class Transfer GAN: Ambient Space Image Manipulation

Figure 13: Top left: Images and their re-generation; Right: Two ambient-space
components, of make-up (top) and lighting (bottom); Middle: Sweeping magnitude
of each ambient-space component in the re-generated image. Figure 14: Top
left: Images and their re-generation; Right: Two ambient-space components, of
fur (top) and comic-like-celluloid component (bottom); Middle: Sweeping
magnitude of each ambient-space component in the re-generated image. Figure
15: Classes Cat and Wild: Images, their re-generation, and separation between
two types of textures by summing different subsets of the ambient-space
components.

##  Appendix B Cifar10 Classifiers Intra-Branch Specialization

Figure 16: Branches 10, 14: We examine horse samples of top $7\%$ confidence,
where confidence was assessed for each branch separately - as described in
Fig. 8 in the main article. Confidence values of these branches continue to be
higher for either close-ups or full body shots of horses even at $7\%$.

##  Appendix C Cifar10 Classifiers with various No. of Branches

Figure 17: Left: Branch outputs for models with $M=[2,4,8,16,32]$ branches -
as seen in Fig. 6 in the main paper. The No. of Silent and Active Branches, as
quantified in Fig. 5 in the main paper, is calculated according to the branch
outputs shown here. Right: Covariance matrices, as in Fig. 5 in the main paper

.

##  Appendix D Class Transfer GAN: with and without learnt pre-processing

In the main paper we add a nonlinear diffusion pre-processing of the input
(details in Sec. E). Here we show that branch outputs of the Class Transfer
GAN are ambient-space representations with this addition, and without it as
well (Figs. 18, 19, 20).

Figure 18: Cats: Images, their re-generation, their $R$ and outputs of the 9
branches of the generator network. Top: With learnt non-linear diffusion pre-
processing, each branch outputs an ambient-space representation, where
branches differ by textures, colors, and segments; Bottom: Without non-linear
diffusion pre-processing, i.e. each branch recieves $x$ as a direct input and
$R$ is a Gaussian-smoothed image. Similar ambient-space decompositions are
obtained, but quality is inferior.

Figure 19: Dogs: Similarly to Fig. 18.

Figure 20: Wild: Similarly to Fig. 18.

##  Appendix E Class transfer GAN: Loss and optional Nonlinear Diffusion of
the input

To have a conciser meaning of the training objective, we train our NN as a
GAN, but do not use any of the additional loss terms originally used by
StarGAN-v2. This requires two adaptations: First, the style diversification
and style reconstruction losses are simply discarded, as we do not require
diverse synthesis. Second the cycle-loss is discarded, and fidelity to input
image is attained by slightly modifying Eq. (1) of the main paper as follows:

$f_{\theta}(x)=R+\sum_{k=1}^{M}v_{k}(x),$ (15)

where $R$ is an over-smoothed version of $x$ \- either learnt smoothing or
Gaussian smoothing.

Additionally - we added a learnt pre-processing step, which learns $R$, and
processes inputs: This procedure was devised as a neural-network that suffices
$R$, as well as lowly-correlated inputs for each branch. The nonlinear
diffusion is implemented as a recurrent neural-network with architecture
$p(x;W)$, hence the nonlinear diffusion process is

$u_{t}(t)=-p(t),\,u(0)=x\in X,$ (16)

where time steps are often implemented discretely. This was done before, for
instance [6]. Let $\phi(t)=t\cdot u_{tt}$ then we can compute a reconstruction
formula, for a general stopping time $T$, using integration by parts (and
assuming $u_{t}(0)$ is bounded)

$\int_{0}^{T}\phi(t)\,dt=tu_{t}|_{0}^{T}-\int_{0}^{T}u_{t}\,dt=Tu_{t}(T)-u(T)+x=-Tp(u(T))-u(T)+x.$
(17)

In other words - let $R=Tp(u((T))+u(T)$, then the following reconstruction
identity holds $x=\int_{0}^{T}\phi(t)\,dt+R$. This holds for a discretized
nonlinear diffusion process as well, where integrals are replaced by weighted
sums, and time derivatives by discrete time derivatives. This is a similar
mechanism to Spectral TV [13].

Because $R$ is the residual of a nonlinear diffusion process, driven by a CNN,
it generalizes the regular Gaussian smoothing. Hence we use it as a learnt
over-smoothing of the image. $\phi(t)$ capture different parts of the
nonlinear diffusion process - hence for a smoothing process, it suffices a
multi-scale decomposition of $x$. $\phi$ alongside $R$ enables full
reconstruction of the input $x$ (17). Discretizing the diffusion process to
$M+1$ steps, we a obtain a discrete set
$\\{\phi_{k}\\}_{k=1}^{M},\,\phi_{i}\in X$, and each $\phi_{k}$ is fed to a
different branch.

Thus the inference becomes $f_{\theta}(x)=R+\sum_{k=1}^{M}v_{k}(\phi_{k})$.
The architecture of $p$ is an 8-layer bottleneck CNN with $4.05\cdot 10^{4}$
parameters, where input and output dimensions are the same and equivalent to
dimension of $x$. Evidently (Figs. 18, 19, 20), $p$ is learnt to be a
smoothing operator - as $R$ is indeed a smoothed version of $x$.

Remark: We noticed that throughout the training procedure $R$ may fluctuate
between a slightly smoothed version $x$ to a significantly smoothed $x$ \- at
times, even a constant constant image.

##  Appendix F Unsupervised Segmentation

Figure 21: In the main paper, we demonstrate how our decompositions, which are
ambient-space components, may be useful for segmentation. Because the
components are in ambient-space, they are completely aligned with the
generated image, and segmentation can be done directly upon them. We segment
cats’ eyes. We use the Chan-Vese [5] algorithm to optimize a surface-image so
that image-regions, which correspond to the positive regions of the surface-
image, have low variance. Some of our components are good at locating eyes and
nose - thus they are used for the initial surface. Other components tend to
have constant values in the eye regions - hence we use these as the image we
segment. Finally - this procedure often captures the nose as well - so as a
final criterion, we filter the connected components that have row-wise
multiplicity of 2. Remark: A pre-processing step entails converting the RGB
components to scalar images. We found that using the Value channel of HSV
works nicely.

##  Appendix G Computational Resources

We use VGA compatible controller: NVIDIA Corporation GV100GL [Tesla V100 DGXS
32GB] (rev a1).