# A motion planning algorithm in a figure eight track

C. Jardon, B. Sheppard, V. Zaveri

###### Abstract -

We design a motion planning algorithm to coordinate the movements of two
robots along a figure eight track, in such a way that no collisions occur. We
use a topological approach to robot motion planning that relates instabilities
in motion planning algorithms to topological features of configuration spaces.
The topological complexity of a configuration space is an invariant that
measures the complexity of motion planning algorithms. We show that the
topological complexity of our problem is $3$ and construct an explicit
algorithm with three continuous instructions.

Keywords : topological robotics; motion planning; topological complexity

Mathematics Subject Classification (2020) : 55P99; 05C85; 90C35, 05C90

##  1 Introduction

The motion planning problem in robotics relates to the movement of robots from
one location to another in their physical space. When there are several robots
moving in the same physical space without collisions we need to study the
configuration space.

The configuration space gives a complete specification of the position of
every robot in the physical space and a path in the configuration space is
interpreted as a particular collision-free movement from the initial position
of the robots to their final one.

When $X$ is the configuration space, the motion planning problem consists in
constructing an algorithm that takes as input a pair of initial and final
configurations $(x_{i},x_{f})\in X\times X$, and produces a continuous path
$\alpha:I\to X$ from the initial configuration $x_{i}=\alpha(0)$ to the final
one $x_{f}=\alpha(1)$. Farber initiated a topological approach to this problem
in [1]. Let $PX$ be the space of all paths in $X$. The motion planning
algorithm (MPA) is a map $s:X\times X\to PX$ defined by sending a pair of
points to a path between them. This approach favors maps that depend
continuously on the input since they will translate into more robust
algorithms in real-life problems. However, Farber proved that there exists a
continuous MPA in $X$ if and only if $X$ is contractible. Therefore, most MPAs
in real life may have essential discontinuities, or instabilities, due to the
topology of $X$.

Motivated by this, Farber introduced the topological complexity of a space.
This is a numerical homotopy invariant which measures the minimal number of
instabilities of any MPA on a space. Essentially, the topological complexity
of a space $X$, $TC(X)$, is the smallest $k$ such that $X\times X$ can be
covered by $k$ sets over each of which there is a continuous local section. We
will call instructions each of these local sections. Then, the topological
complexity of a space $X$ is the minimal number of instructions in any MPA in
$X$.

We present our approach to calculating the topological complexity of the
motion planning problem for two robots moving in a figure eight graph, and
construct an explicit algorithm with the minimal number of instructions given
by the topological complexity.

Our constructive strategy is based on finding a spine $Z\subset X$ which is a
deformation retract of $X$ with lesser dimension and design the algorithm in
$Z$. Then extend the algorithm to the whole configuration space $X$ by using
the traces of the homotopy deformation. We also give a translation of the
algorithm for the physical space that shows explicitly how to move each of the
two robots from any initial position to any final position.

This paper will be organized as follows. Section 2 introduces basic
terminology used throughout the paper along with the basic definitions of MPA
and configuration spaces. Section 3 explains the construction of the
configuration space and its flat representation. In section 4 we discuss the
topological complexity and its basic properties. Section 5 gives a detailed
description of the homotopy deformation into the spine. We also show in this
section that the spine is homotopy equivalent to a wedge of seven circles,
which allows us to calculate the topological complexity of the configuration
space. In section 6 we present our main algorithm in the spine, and extend it
to the whole configuration space. We translate the previous algorithm in the
configuration space to the physical space in section 7.

##  2 Definitions and terminology

###  2.1 Motion planning problem

One of the goals of robotics is to create autonomous robots. A set of robots
is a mechanical system that accepts descriptions of tasks and executes them
without further human intervention. The input description specifies what
should be done and the robots decide how to do it and perform the task. The
motion planning problem consists of designing a general algorithm for the
robots to move from an initial position to a final one. We will use a
topological approach to the robot motion planning problem initiated by Farber
in [1].

###  2.2 Basic notions

Since continuity is a key feature of our algorithm, we will start by recalling
its definition for topological spaces. Let $X$ and $Y$ be topological spaces.

We say that $f:X\to Y$ is continuous at $a\in X$ if whenever $V$ is an open
neighborhood of $f(a)$ in $Y$, there is an open neighborhood $U$ of $a$ in $X$
such that $f(U)\subset V$. If $f$ is continuous at every point $a\in X$ then
we say $f$ is continuous on $X$.

A path from a point $A$ to a point $B$ in a topological space $X$ is a
continuous function $\alpha$ from the unit interval $I=\left[0,1\right]$ to
$X$ with $\alpha(0)=A$ and $\alpha(1)=B$. The path space, $PX$, is the space
of all paths $\alpha$ in $X$, i.e. $PX=\left\\{\alpha\ \lvert\
\alpha:\left[0,1\right]\to X\right\\}$. A space $X$ is path-connected if for
all points $x,y\in X$ there exists a path from $x$ to $y$.

The product space of $X$ and $Y$ will be the Cartesian product $X\times
Y=\\{(x,y)\ \lvert\ x\in X,y\in Y\\}$ with the product topology.

The map $e:PX\to X\times X$ which takes a path $\alpha$ as its input and
returns the pair $(\alpha(0),\alpha(1))$ is called the evaluation map. Here
$\alpha(0)$ is the initial point of the path and $\alpha(1)$ is the final one.

If $X$ is a path-connected space, let the function $s:X\times X\to PX$ be a
section of the evaluation such that $e\circ s=id_{X\times X}$. This section
takes as input two points and produces a path $\alpha$ between them as output:

$s:(A,B)\mapsto\alpha\text{ and }\alpha(0)=A,\alpha(1)=B$

where $\alpha(0)=A$ is the initial point and $\alpha(1)=B$ is the final point
of the path.

Let $f:X\to Y$ be a bijection. If both the function $f$ and the inverse
function $f^{-1}:Y\to X$ are continuous, then $f$ is called a homeomorphism.
If such a function exists, we say $X$ is homeomorphic to $Y$ using the
notation $X\approx Y.$

Let $h$ and $h^{\prime}$ be continuous maps from $X$ to $Y.$ We say that $h$
is homotopic to $h^{\prime}$, denoted by $h\simeq h^{\prime}$, if there exists
a continuous map

$F:X\times I\to Y$

such that

$F(x,0)=h(x)\text{ and }F(x,1)=h^{\prime}(x)$

for each $x\in X$.

A map $f:X\to Y$ is a homotopy equivalence if there exists a map $g:Y\to X$
where $f\circ g\simeq id_{Y}$ and $g\circ f\simeq id_{X}.$ If such maps exist,
we say $X$ and $Y$ are homotopy equivalent and write $X\simeq Y$. Note that
homeomorphic spaces are always homotopy equivalent.

A topological space $X$ is contractible if it is homotopy equivalent to a
point.

###  2.3 Configuration space

The variety of all the possible states that any mechanical system can take is
called the configuration space. Each point in the configuration space
represents a state of the system. The space $\Gamma$ where the robots are able
to move will be called the physical space. The configuration space of $n$
robots moving in a space $\Gamma$ without collisions, denoted $C^{n}(\Gamma)$,
is defined as:

$C^{n}(\Gamma)=(\Gamma\times\Gamma\times\cdots\times\Gamma)-D.$

Here $D$ represents the pairwise diagonal:

$D=\left\\{(x_{1},x_{2},\ldots,x_{n})\in\Gamma^{n}\ \lvert\ x_{i}=x_{j}\text{
for some }i\neq j\right\\}$

given by the states in which at least two robots occupy the same place.

##  3 Configuration space of two robots on a figure eight track

Let $\Gamma$ denote the figure eight space, which is a wedge sum of two
circles $S^{1}\vee S^{1}$. We will consider the motion planning problem of two
robots moving in the track $\Gamma=S^{1}\vee S^{1}$. See figure 1.

Figure 1: Physical space $\Gamma$.

In this case, the configuration space is
$C^{2}(\Gamma)=(\Gamma\times\Gamma)-D.$ We will visualize this space
$X=C^{2}(\Gamma)$ first when embedded in $\mathbb{R}^{3}$ and then we will
explain its flat representation that will be useful for calculations.

###  3.1 Visualization of the configuration space $X$

The robots in $\Gamma$ are represented by triangles and squares. The triangle
robot will be also referred to as the first robot and the square robot as the
second one. We shade each circle with a different color to distinguish the two
separate circles in the configuration space. Each concrete position in the
physical space, see for instance figure 2, will have a corresponding state in
the configuration space.

Figure 2: First and second robot in $\Gamma$

The Cartesian product $\Gamma\times\Gamma$ represents all positions of the two
robots in $\Gamma$. The product of a figure eight by itself is given by four
connected tori as in figure 3. For robots in the same circle, their states are
represented by the red and blue tori whereas the purple tori represent states
where the robots are in different circles.

Figure 3: Cartesian product $(S^{1}\vee S^{1})\times(S^{1}\vee S^{1})$.

To obtain the configuration space, we need to remove the points in this
Cartesian product that correspond to positions in which both robots are at the
same place. We observe that the two tori representing positions of robots in
the same circle have a diagonal removed from them, whereas the other two tori
have just one point removed.

The diagonal $D=\\{(x,y)\in(S^{1}\vee S^{1})\times(S^{1}\vee S^{1})|\;x=y\\}$
determines a curve on the red and blue tori. The configuration space $X$ is
the Cartesian product $\Gamma\times\Gamma$ with this curve removed. See figure
4.

Figure 4: $X=(\Gamma\times\Gamma)-D$.

We will represent now the configuration space in the two-dimensional space by
first considering the circle $S^{1}$ as a quotient of the unit interval where
the points $0$ and $1$ are identified, i.e. $S^{1}\approx I/\\{0\sim 1\\}$.
See figure 5.

Figure 5: Flat representation of the circle.

The flat representation of the Cartesian product $\Gamma\times\Gamma$ is given
by four connected squares with their sides identified as in figure 6.

Figure 6: The product $\Gamma\times\Gamma$

The diagonal $D=\\{(x,y)\in(S^{1}\vee S^{1})\times(S^{1}\vee S^{1})|\;x=y\\}$
is given by the diagonals of the red and blue squares in the flat
representation. The configuration space is the Cartesian product
$\Gamma\times\Gamma$ with the diagonal $D$ removed. See figure 7.

Figure 7: Flat representation of $X=(\Gamma\times\Gamma)-D$

Once that the diagonal is removed, we can visualize the flat representation of
the configuration space as the following parallelogram with sides identified
as shown in figures 8 and 9.

Figure 8: Transition from square to parallelogram representation. Figure 9:
Flat representation of $X=(\Gamma\times\Gamma)-D$ as a parallelogram.

##  4 Topological Complexity

In order to program robots to move autonomously from any initial state to any
desired state we will need a motion planning algorithm that takes as input the
pair $(A,B)$ where $A$ is the initial state and $B$ is the final state and
outputs a continuous motion of the system starting at $A$ and ending at $B$.

A Motion Planning Algorithm (MPA) is a section $s:X\times X\rightarrow PX$ of
the evaluation map. It is a function that takes as input a pair of initial and
final states, and outputs a path between them.

A motion planning algorithm $s:X\times X\rightarrow PX$ is a map that is not
necessarily continuous. When it is continuous, if the inputs are slightly
modified then the path between the inputs is only slightly modified too. The
discontinuities of the MPA emerge as instabilities of the robot motion. We
want to minimize such instabilities to produce instructions which are robust
to noise: if there are small errors in the initial and final positions
measurement, we want the paths associated to the exact positions and the one
given by the measurement to be nearby.

Farber proved that the only case where a continuous motion planning algorithm
exists is when the space is contractible.

###### Theorem 4.1

[1] A continuous motion planning $s$: $X\times X\rightarrow PX$ exists if and
only if the configuration space $X$ is contractible.

For spaces that are not contractible, all MPAs will be discontinuous. In real-
life cases, we would like to minimize these discontinuities to produce
optimally stable MPAs.

Farber invented a number known as Topological Complexity, $TC(X)$, that
roughly can be described as the minimal number of continuous instructions
which are needed to describe any motion planning algorithm in $X$.

This number measures how complex it is for the robots to navigate the space.
For example, if the topological complexity of a space is two, then robots need
a minimal number of two continuous instructions to move. The higher the $TC$
is, the harder it is for the robots to navigate the space.

###### Definition 4.2

[1][4] Let $X$ be a path-connected topological space. The topological
complexity of $X$, $TC(X)$, is the smallest number $k$, such that $X\times X$
can be covered by $k$ sets $U_{1},U_{2},U_{3}$, $\ldots$, $U_{k}$ where on
each of them there is a continuous local section $s_{i}:U_{i}\to PX$ for each
$i=1,2,\ldots,k$.

Each of these local sections will be our continuous instructions in the motion
planning algorithm. For instance, if the space $X$ is contractible we need
only one section and $TC(X)=1$. In the case of a circle, we need two sets to
cover $S^{1}\times S^{1}$. The first set is $U_{1}=\\{(x,y)|\;x\text{ is
antipodal to }y\\}$ and the second set is $U_{2}=\\{(x,y)|\;x\text{ is not
antipodal to }y\\}$. The first instruction over the set $U_{1}$ is “move
counterclockwise” while the second instruction over $U_{2}$ is “move following
the shortest path”. Both of these instructions are continuous and
$TC(S^{1})=2$. Note that none of these instructions are continuous over
$S^{1}\times S^{1}$.

Farber also proved that this number is a homotopy invariant. If a space is
homotopy equivalent to another, then their topological complexities are the
same on their domains.

###### Theorem 4.3

[1] If $X$ is homotopy equivalent to $Y$, then $TC(X)=TC(Y)$.

##  5 Homotopy type of $X$

Our aim will be to find a simpler space that is homotopy equivalent to the
space $X$ and then construct our algorithm in the simpler space. Finding an
algorithm and controlling the discontinuities directly in $X$ is not trivial.
Theorem 4.3 is crucial to reduce the complexity of the task by using the
homotopy equivalence not only to calculate the topological complexity of a
simpler space, but also to construct the actual algorithm in the simpler space
and extend it to $X$ afterward using the homotopy traces.

Ghrist proved that the configuration space of robots moving in a graph is
homotopy equivalent to a space of lesser dimension.

###### Theorem 5.1

[2] Given a graph $\Gamma$, the configuration space of $N$ distinct points on
$\Gamma$ can be deformation retracted to a spine whose dimension is bounded
above by the number of vertices of $\Gamma$ of valency greater than two.

Our figure-eight graph has only one vertex with valency greater than two and
therefore the configuration space retracts to a spine of dimension $1$. We
know then that it is possible to shrink the space $X$ to a one-dimensional
space. We will show next the explicit construction of this spine for our
graph.

###  5.1 Construction of the Spine

The spine $Z$ of the configuration space is a one-dimensional space that
carries all of the topology of the configuration space through reducing the
full space by deformation retraction to a simpler space. The center of the
physical space $\Gamma$ is the point at which the circles intersect and the
poles are the antipodal points to the center in each circle.

Figure 10: The center and the poles, respectively.

The spine $Z$ is a set of segments in the flat representation with their
extremes identified that represents the positions of the robots in $\Gamma$
where at least one robot is at a pole and the other at a different circle or
both robots are at antipodal positions in the same circle.

We construct a retraction $r$ of the whole space $X$ into the spine $Z$. The
homotopy deformation $H$ will follow the traces $H$($\textunderscore$,t) shown
in figure 11.

Figure 11: Homotopy traces of the retraction of $X$ into $Z$.

The map $r$ is a deformation retraction since it is a retraction and the
composition with the inclusion is homotopic to the identity map in $X$.We
observe the following identifications of the segment extremes in $Z$ shown in
figure 12.

Figure 12: Spine $Z$.

These identifications make the spine $Z$ homeomorphic to a chain of circles
$C$. We color each segment to describe this homeomorphism.

Figure 13: The spine $Z$ is homeomorphic to the chain $C$.

We can also visualize the chain of circles $C$ directly in the representation
of $X$ embedded in $\mathbb{R}^{3}$. By following the traces of the homotopy,
the two cylinders deform into circles and each torus minus a point deforms
into a figure-eight. See figure 14. Note that in figures 14 and 15, the dashed
lines are the diagonal $D$.

Figure 14: Traces of the homotopy in $X$.

We can visualize in the following figure the chain of circles $C$ embedded in
the three-dimensional space.

Figure 15: The chain $C$.

###  5.2 The wedge of seven circles

We will show in this section that the chain $C$ is homotopy equivalent to a
wedge of seven circles $Y$. We perform a series of contractions in each circle
as shown in figure 16.

Figure 16: Contraction of a segment in a circle.

We repeat this deformation in each of the circles. See figures 17 and 18.

Figure 17: First stages of the deformation. Figure 18: Last stages of the
deformation.

The final space is a wedge of seven circles as shown in figure 19.

Figure 19: Wedge of seven circles $Y$.

###  5.3 Calculation of the topological complexity of the configuration space

The configuration space $X$ is homotopy equivalent to the spine $Z$ and $Z$ is
homeomorphic to the chain of circles $C$, which is homotopy equivalent to the
wedge $Y$, i.e.

$X\simeq Z\approx C\simeq Y.$

Then, by Farber’s theorem 4.3 we have that

$TC(X)=TC(Z)=TC(C)=TC(Y).$ (1)

Farber also studied and calculated the topological complexity of wedges of
spheres:

###### Lemma 5.2

[3] Let $W$ denote the wedge of $n$ spheres $S^{m}$, $W=S^{m}\vee
S^{m}\vee\cdots\vee S^{m}$. Then $TC(W)=\left\\{\begin{array}[]{ll}2&\mbox{if
}n=1\mbox{ and }m\mbox{ is odd,}\\\ 3&\mbox{if either }n>1\mbox{, or m is
even.}\end{array}\right.$

For our space $Y$ we have that $m=1$ and $n=7$. By lemma 5.2, we know that the
topological complexity of a wedge of seven circles is $3$, then $TC(X)=3$
following equation 1. We know now that our algorithm will have to have at
least $3$ continuous instructions.

##  6 Algorithm in the configuration space

We will construct first our algorithm with three instructions in the spine $Z$
that is homotopy equivalent to $X$. Then we will extend the algorithm to the
whole configuration space $X$ following the traces of the homotopy.

Recall that the spine $Z$ in the flat representation is a network consisting
of two crosses and four sub-diagonals.

Figure 20: Space Z

When the state is in a diagonal segment in $Z$, we will say that the state is
a sub-diagonal state. In the physical space $\Gamma$, this state will
correspond to antipodal positions in the same circle.

Figure 21: A sub-diagonal state.

When a state is located in a cross-segment in $Z$, we will say that it is a
cross-state. In the physical space $\Gamma$, cross-states translate as the
position of at least one robot being at a pole and the other anywhere on the
opposite circle.

Figure 22: A cross-state.

A cross center is a cross-state at the intersection of horizontal and vertical
cross-segments. A cross center corresponds to the positions in which both
robots are at poles in different circles in $\Gamma$.

Figure 23: Cross centers in the physical space $\Gamma$. Figure 24: Cross
centers in $Z$ and $C$.

An intersection point is a state in $Z$ at the intersection between a cross-
segment and a sub-diagonal segment. In $\Gamma$, this state corresponds to
positions in which one robot is at the center and the other at a pole. There
are two types of intersection points: $H$-points located between a sub-
diagonal segment and a horizontal cross-segment, and $V$-points located
between a sub-diagonal segment and a vertical cross-segment. An $H$-point
corresponds in the physical space to a position in which the first robot is at
the center and the second robot is at a pole. Conversely for $V$-points.

Figure 25: Intersection V-points(left) and H-points(right) in the physical
space $\Gamma$. Figure 26: Intersection points in $Z$ and $C$.

The cross and sub-diagonal segments in $Z$ will constitute the circles in the
chain $C$ whereas the intersection points and cross centers will be the
vertices in $C$.

###  6.1 Algorithm in the chain $C$

The initial, final, and current states are the states corresponding to the
initial, final, and current positions respectively.

If the current state is at a vertex in $C$, we will say that the current
circle is the next immediate circle counterclockwise. Otherwise, the current
circle is the circle where the current state is.

Let $(I,F)=((x_{i},y_{i}),(x_{f},y_{f}))$ be a pair of initial and final
states in $C\times C$. We define the following subsets of $C\times C$:

  * •

$A=\left\\{(I,F)\in C\times C\ \lvert\ I\text{ is antipodal to }F\text{ in the
same circle}\right\\}$

  * •

$V=\left\\{(I,F)\in C\times C\ \lvert\ I\text{ or }F\text{ is a
vertex}\right\\}$

  * •

$V^{\prime}=\left\\{(I,F)\in C\times C\ \lvert\ I\text{ and }F\text{ are
vertices}\right\\}$

###### Algorithm 6.1

Set current state to initial state $I$

Instruction 1

For $(I,F)\in C\times C-(A\cup V)$:

  1. 1.

Check if final state is in current circle.

     * •

If true, take shortest path to final state.

     * •

If false, move counterclockwise in current circle to next vertex and set
current state to that vertex. Repeat 1.

Instruction 2

For $(I,F)\in(A\cup V)-V^{\prime}$:

  1. 2.

Check if final state is in current circle.

     * •

If true, check if final state is antipodal to current state.

       * –

If true, go counterclockwise to final state.

       * –

If false, take shortest path to final state.

     * •

If false, check if current state is a vertex.

       * –

If true, move counterclockwise in current circle to next vertex and set
current state to that vertex. Repeat 2.

       * –

If false, take shortest path to next counterclockwise vertex and set current
state to that vertex. Repeat 2.

Instruction 3

For $(I,F)\in V^{\prime}$:

  1. 3.

Check if final state is next counterclockwise vertex.

     * •

If true, go counterclockwise in the current circle to final state.

     * •

If false, go counterclockwise in the current circle to next vertex and set
current state to that vertex. Repeat 3.

Each of these instructions is continuous in its domain.

###  6.2 Algorithm in $Z$

By following the homeomorphism between $C$ and $Z$ we will translate now each
of these instructions to the spine $Z$.

We define a distinguished direction of movement in $Z$ that will correspond to
moving from vertices counterclockwise in the current circle in $C$. The
positive direction in $Z$ is given by:

  * •

from a cross center, move to the right in the horizontal segment;

  * •

from a $H$-point, move up in the diagonal segment;

  * •

from a $V$-point, move up in the vertical segment;

  * •

from any other point, move to the right in horizontal or diagonal segments and
up in vertical segments.

Figure 27: Positive direction in $C$ and $Z$.

If the current state is at an intersection point or cross center in $Z$, we
will say that the current segment is the next immediate segment in the
positive direction. Otherwise, the current segment is where the current state
is.

We consider $Z$ to be a metric graph where each edge has length one and the
distance between two points is given by the infimum of the lengths of paths
joining them.

Let $(I,F)=((x_{i},y_{i}),(x_{f},y_{f}))$ be a pair of initial and final
states in $Z\times Z$. We define the following subsets of $Z\times Z$:

  * •

$H=\\{(I,F)\in Z\times Z\ \lvert\ I\text{ and }F\text{ are on the same segment
and }d(I,F)=\frac{1}{2}\\}$

  * •

$J=\\{(I,F)\in Z\times Z\ \lvert\ I\text{ or }F\text{ are at a cross center or
intersection point}\\}$

  * •

$J^{\prime}=\\{(I,F)\in Z\times Z\ \lvert\ I\text{ and }F\text{ are both at
cross centers or intersection points}\\}$

###### Algorithm 6.2

Set current state to initial state $I$

Instruction 1

For $(I,F)\in Z\times Z-(H\cup J)$:

  1. 1.

Check if final state is in current segment.

     * •

If true, take shortest path to final state.

     * •

If false, move in the positive direction in current segment to next point of
intersection and set current state to that point of intersection. Repeat 1.

Instruction 2

For $(I,F)\in(H\cup J)-J^{\prime}$:

  1. 2.

Check if final state is in current segment.

     * •

If true, check if the distance from current state to final state is
$\frac{1}{2}$.

       * –

If true, move in the positive direction to final state.

       * –

If false, take shortest path to final state.

     * •

If false, check if current state is a point of intersection.

       * –

If true, move in the positive direction along current segment to next point of
intersection and set current state to that point of intersection. Repeat 2.

       * –

If false, take shortest path, in the positive direction, to next point of
intersection and set current state to that point of intersection. Repeat 2.

Instruction 3

For $(I,F)\in J^{\prime}$:

  1. 3.

Check if final state is the next point of intersection in the positive
direction.

     * •

If true, move in the positive direction along the current segment to final
state.

     * •

If false, move in the positive direction along the current segment to next
point of intersection and set current state to that point of intersection.
Repeat 3.

###  6.3 Algorithm in $X$

We will extend now the algorithm 6.2 to the whole configuration space.

Given an initial state $I$, let $I_{s}$ be the state corresponding to the
point of intersection between the trace of the homotopy passing through $I$
and the spine $Z$. Analogously, $F_{s}$ is the corresponding state to the
final one in the spine.

###### Algorithm 6.3

Let $I$ and $F$ be the initial and final states.

  1. 1.

Find $(I_{s},F_{s})$.

  2. 2.

Move state $I$ to $I_{s}$.

  3. 3.

Execute algorithm 6.2 for $(I_{s},F_{s})$.

  4. 4.

Move state $F_{s}$ to $F$.

Given the initial and final states shown in figure 28, the following figures
illustrate algorithm 6.3 for this case.

Figure 28: Step 2. Figure 29: Step 3. Figure 30: Step 4.

##  7 Algorithm in the physical space

We will translate now our algorithm 6.3 to the physical space $\Gamma$.

Observe that states in the spine $Z$ correspond to positions in which at least
one robot is at a pole if the robots are in different circles and antipodal
positions if the robots are in the same circle.

###  7.1 States $(I_{s},F_{s})$ in the physical space $\Gamma$

To translate the first step of the algorithm, we need to find the positions of
the robots in $\Gamma$ corresponding to the intersection of the traces with
the spine.

If the robots are in the same circle, move the robots away from each other
until they are in antipodal positions. See figure 31. If they are in different
circles, move them away from the center until at least one of them reaches a
pole position. See figure 32. If one robot is at the center, move the other
robot until it reaches a pole position. See figure 33.

Figure 31: Moving to antipodal positions Figure 32: Moving until one robot
reaches the pole. Figure 33: Robot in center scenario.

The ratio between the speeds at which the robots are moving is determined by
the slope of the traces of the homotopy discussed in section 5.1. If a trace
of the homotopy is given by the curve $(x(t),y(t))$, then the slope of the
trace is given by the ratio between the speeds $y^{\prime}(t)$ and
$x^{\prime}(t)$.

###  7.2 Algorithm in the physical space for the positions in $Z$

We will translate here the movement in the positive direction in $Z$ to the
physical space. Recall that a cross-center state corresponds to both robots
being at poles, and intersection points in $Z$ correspond to one robot being
at a pole and the other at the center. In the physical space, we will call
these positions vertex positions.

The positive direction in $\Gamma$ is given by:

  * •

if both robots are at poles, move robot $1$ counterclockwise;

  * •

if the first robot is at the center and the other at a pole, move both
counterclockwise;

  * •

if the first robot is at a pole and the other at the center, move the second
robot counterclockwise.

Figure 34: Positive direction in $\Gamma$.

If the current position is a vertex position, we will say that its
neighborhood is the set of positions determined by moving the current position
in the positive direction until it circles back to itself. Otherwise, the
neighborhood is given by moving counterclockwise the robot that is not at a
pole all around its circle if the robots are at different circles or moving
counterclockwise both robots together if they are antipodal in the same
circle.

Figure 35: A neighborhood of a non-vertex position. Figure 36: A neighborhood
of a vertex position.

Let $(I,F)=((x_{i},y_{i}),(x_{f},y_{f}))$ be a pair of initial and final
positions in $\Gamma\times\Gamma$. We define the following subsets of
$\Gamma\times\Gamma$:

  * •

$K=\\{((x_{i},y_{i}),(x_{f},y_{f}))\ \lvert\ x_{i}\text{ and }x_{f}\text{ are
antipodal}\text{ or }y_{i}\text{ and }y_{f}\text{ are antipodal}\\}$

  * •

$L=\\{((x_{i},y_{i}),(x_{f},y_{f}))\ \lvert\ (x_{i},y_{i})\text{ or
}(x_{f},y_{f})\text{ is a vertex position}\\}$

  * •

$L^{\prime}=\\{((x_{i},y_{i}),(x_{f},y_{f}))\ \lvert\ (x_{i},y_{i})\text{ and
}(x_{f},y_{f})\text{ are vertex positions}\\}$

###### Algorithm 7.1

Set current position to initial position $I$.

Instruction 1

For $(I,F)\in\Gamma\times\Gamma-(K\cup L)$:

  1. 1.

Check if final position is in the neighborhood of the current position.

     * •

If true, move each robot simultaneously to their final position taking the
shortest path.

     * •

If false, move in the positive direction in current neighborhood until next
vertex position. Set current position to this one. Repeat 1.

Instruction 2

For $(I,F)\in(K\cup L)-L^{\prime}$:

  1. 2.

Check if final position is in the current neighborhood.

     * •

If true, check if at least one of the robots have its initial and final
positions antipodal.

       * –

If true, move in the positive direction to final position.

       * –

If false, move each robot simultaneously to their final position taking the
shortest path.

     * •

If false, check if current position is a vertex position.

       * –

If true, move in the positive direction in current neighborhood to the next
vertex. Set current position to this one. Repeat 2.

       * –

If false, take shortest path to next vertex position in the positive direction
and set current position to this one. Repeat 2.

Instruction 3

For $(I,F)\in L^{\prime}$:

  1. 3.

Check if final position is the next vertex position in the positive direction.

     * •

If true, move in the positive direction in the current neighborhood to final
position.

     * •

If false, move in the positive direction in the current neighborhood to next
vertex position and set current state to that position. Repeat 3.

###  7.3 Complete algorithm in the physical space $\Gamma$

We will write now the complete algorithm for all positions in $\Gamma$.

###### Algorithm 7.2

Let $I$ and $F$ be the initial and final positions.

  1. 1.

If the robots are in the same circle, move the robots away from each other
until they are in antipodal positions.

  2. 2.

If they are in different circles, move them away from the center until at
least one of them reaches a pole position.

  3. 3.

Repeat steps $1$ and $2$ with the final position. Let us call initial
$Z$-position and final $Z$-position the output of this step.

  4. 4.

Execute algorithm 7.1 for the initial $Z$-position and final $Z$-position.

  5. 5.

Move back from the final $Z$-position to the final position reversing the
movement done in the first step.

In the following case scenarios, we describe the movements in the physical
space as well as in the configuration space.

####  7.3.1 Case scenario 1

Let us consider the initial and final positions shown in figure 37.

Figure 37: Initial and final positions and their corresponding spine
positions.

The path to move from initial state to final state in the configuration space
is shown in figure 38.

Figure 38: Path in $X$ for case 1.

The steps of the algorithm to move from initial to final positions in the
physical space are shown in figure 39.

Figure 39: Steps of the algorithm in $\Gamma$ for case 1.

####  7.3.2 Case scenario 2

Let us consider the initial and final positions shown in figure 40.

Figure 40: Initial and final positions and their corresponding skeleton
positions.

The path to move from initial state to final state in the configuration space
is shown in figure 41.

Figure 41: Path in $X$ for case 2.

The steps of the algorithm to move from initial to final positions in the
physical space are shown in figure 42.

Figure 42: Steps of the algorithm in $\Gamma$ for case 2.

####  7.3.3 Case scenario 3

Let us consider the initial and final positions shown in figure 43.

Figure 43: Initial and final positions and their corresponding skeleton
positions.

The path to move from initial state to final state in the configuration space
is shown in figure 44.

Figure 44: Path in $X$ for case 3.

The steps of the algorithm to move from initial to final positions in the
physical space are shown in figure 45.

Figure 45: Steps of the algorithm in $\Gamma$ for case 3.

## References

  * [1] M. Farber, Topological complexity of motion planning, Discrete Comput. Geom., 29 (2003), 211–221.
  * [2] R. Ghrist, D. Koditschek, Safe Cooperative Robot Dynamics on Graphs, SIAM J. Control and Optimization, 40 (2002), 1556–1575.
  * [3] M. Farber, Instabilities of robot motion, Topology and its Applications., 140 (2004), 245–266.
  * [4] J.M. García-Calcines, A note on covers defining relative and sectional categories, Topology and its Applications, 265 (2019), 106810.

Cristian Jardon
Wilbur Wright College
4300 N Narragansett Ave
Chicago, IL 60634
E-mail<EMAIL_ADDRESS>

Brian Sheppard
Harold Washington College
30 E. Lake Street
Chicago, IL 60601
E-mail<EMAIL_ADDRESS>

Veet Zaveri
Wilbur Wright College
4300 N Narragansett Ave
Chicago, IL 60634
E-mail<EMAIL_ADDRESS>


# Collaborative Teacher-Student Learning via Multiple Knowledge Transfer

Liyuan Sun  Jianping Gou<EMAIL_ADDRESS>School of Computer Science
and Telecommunication Engineering, Jiangsu University, Zhenjiang, 212013,
China  Baosheng Yu  Lan Du  Dacheng Tao  Faculty of information technology,
Monash University, Australia  UBTECH Sydney AI Centre, School of Computer
Science, Faculty of Engineering, The University of Sydney, Darlington, NSW
2008, Australia.

###### Abstract

Knowledge distillation (KD), as an efficient and effective model compression
technique, has been receiving considerable attention in deep learning. The key
to its success is to transfer knowledge from a large teacher network to a
small student one. However, most of the existing knowledge distillation
methods consider only one type of knowledge learned from either instance
features or instance relations via a specific distillation strategy in
teacher-student learning. There are few works that explore the idea of
transferring different types of knowledge with different distillation
strategies in a unified framework. Moreover, the frequently used offline
distillation suffers from a limited learning capacity due to the fixed
teacher-student architecture. In this paper we propose a collaborative
teacher-student learning via multiple knowledge transfer (CTSL-MKT) that
prompts both self-learning and collaborative learning. It allows multiple
students learn knowledge from both individual instances and instance relations
in a collaborative way. While learning from themselves with self-distillation,
they can also guide each other via online distillation. The experiments and
ablation studies on four image datasets demonstrate that the proposed CTSL-MKT
significantly outperforms the state-of-the-art KD methods.

###### keywords:

##  1 Introduction

Deep neural networks have achieved state-of-the-art performance on many
applications such as computer vision, natural language processing, and speech
recognition in recent years. The remarkable performance of deep learning
relies on designing deeper or wider network architectures with many layers and
millions of parameters to enhance the learning capacity. However, it is almost
impossible to deploy the large-scale networks on platforms with limited
computation and storage resources, e.g., mobile devices and embedded systems.
Thus, the model compression and acceleration techniques mainly including
network pruning [2, 3], model quantization [34, 35] and knowledge distillation
[20, 21, 39, 40, 36] are proposed for training lightweight deep models. Among
compressing methods, knowledge distillation, which carries out knowledge
transfer from a high-capacity teacher network to a low-capacity student one,
has received increasing interest recently since it was first introduced in
[20].

In knowledge distillation, the type of knowledge, the distillation strategy
and the teacher-student architecture are three crucial factors that determine
the KD performance [1]. As pointed out in [1], there are three kinds of
knowledge, i.e., the response-based, the feature-based and the relation-based
knowledge. Generally, most of KD methods distill the response-based knowledge
(e.g., soft logits of the output layer) from a large teacher network and
transfer it to a small student [20, 32, 22]. To overcome the limitation of
knowledge from the output layer of teacher, the feature-based knowledge from
the middle layers of teacher is also used to train the student [21, 18, 19].
Unlike both the response-based and the feature-based knowledge from individual
instances, the relation-based knowledge from instance relations is modelled
for improving student learning [37, 38, 15, 16, 17]. Each kind of knowledge
can provide student training with an informative teacher guidance, and they
can also compensate each other to enrich learning. However, most existing KD
methods only consider either knowledge from individual instance features to
maintain instance consistency between teacher and student or knowledge from
instance relations to preserve the instance correlation consistency. There are
a few works that consider more than one kind of knowledge in knowledge
distillation [37, 14] at the same time and explore the efficacy of each kind
of knowledge.

Transferring different types of knowledge can be implemented with different
distillation methods, e.g., offline distillation, online distillation and
self-distillation [1]. Most of the KD methods employ offline distillation,
which is one-way knowledge transfer from a pre-trained large teacher to a
small student [20, 22, 13]. In offline distillation, the capacity gap caused
by a fixed teacher-student architecture and the requirement of a large dataset
for pre-training the teacher often result in a degraded performance [22].
Thus, finding a proper teacher-student architecture in offline distillation is
challenging. In contrast, online distillation provides a one-phase end-to-end
training scheme via teacher-student collaborative learning on a peer-network
architecture instead of a fixed one [25, 33, 36, 32, 28, 12]. Self-
distillation performs online distillation within the same network to reduce
model over-fitting [23, 24]. Online distillation and self-distillation are
promising methods for knowledge distillation as they bridge the capacity gap
via avoiding the need of a large teacher network, leading to an improved
performance. However, both KD methods used individually are limited to
knowledge distillation from a single source, i.e., individual instances,
online distillation could further suffer from the poor instance consistency
between peer networks caused by the discrepancy in their network outputs.

Figure 1: The overview diagram of CTSL-MKT.

Consequently, it is desirable to have a unified framework that can integrate
the advantages of different KD methods and make efficient use of different
types of knowledge. Inspired by the idea of knowledge distillation via
multiple distillation strategies to transfer more than one types of knowledge,
we propose a collaborative teacher-student learning via multiple knowledge
transfer (CTSL-MKT), which fuses self-distillation and online distillation in
such a way that the former transfers the response-based knowledge within each
peer network and the latter bidirectionally transfers both the response-based
knowledge and the relation-based knowledge between peer networks. CTSL-MKT can
overcome the aforementioned issued faced by existing KD methods that often use
only one distillation strategy to transfer a single type of knowledge. The
overview framework of CTSL-MKT is illustrated in Figure 1. To our knowledge,
this is the first framework that integrates different distillation strategies
together to transfer more than one type of knowledge simultaneously.

In CTSL-MKT, each pear network conducts self-learning via self-distillation.
Meanwhile, they carry out teacher-student collaborative learning to mutually
teach each other. CTSL-MKT can also adopt a variety of peer network
architectures, where the two peer networks can either share the same network
architecture or have different ones. We believe that multiple knowledge
transfer can provide much more informative knowledge to guide each peer
network so that they can obtain better performance with a better
generalization ability. We conduct a set of image classification experiments
on four commonly-used datasets i.e., CIFAR-10, CIFAR-100, Tiny-ImageNet, and
Market-1501. Experimental results demonstrate the superior performance of the
proposed CTSL-MKT over the state-of-the-art KD methods. The main contributions
in our works can be summarized as follows:

  * •

A new teacher-student mutual learning framework effectively fuses the
knowledge from individual instances and the knowledge from instance
relationships.

  * •

A self-learning enhanced collaborative learning integrates the advantages of
both self-learning and online learning.

  * •

The extensive experiments on a variety of the peer teacher-student networks
that compare CTSL-MKT with the state-of-the-art methods to validate its
effectiveness in image classification tasks.

  * •

A set of ablation studies of different combinations of knowledge and
distillation methods provides insights into how multiple knowledge transfer
contribute to knowledge distillation.

##  2 Related Work

###  2.1 Self-Distillation

Self-distillation is a novel training scheme for knowledge transfer [23, 24,
27, 11, 10]. In self-distillation, the teacher and student networks are
identical and knowledge transfer is carried out within the same network. Yuan
et al. empirically analyzed the performance of normal, reversed and defective
KD methods, and showed that a weak teacher can strengthen the student and
vice-versa [23]. A teacher-free knowledge distillation method (Tf-KD) instead
makes student model conduct self-learning. To enhance the generalization and
overcome over-fitting, class-wise self-knowledge distillation makes use of
soft logits of different intra-class samples within a model [27]. Phuong and
Lampert [11] proposed a distillation-based training method to reduce time
complexity, where the output of later exit layer supervises the early exit
layer via knowledge transfer. Rather than at the layer level, snapshot
distillation [10] transfers knowledge from earlier to later epochs while
training a deep model.

Overall, self-distillation can overcome the issue of over-fitting and the
capacity gap on the teacher-student architectures, improve the generalization
ability and reduce the inference time of a deep model. However, the self-
distillation performance could be limited by the one-sided response-based
knowledge from model itself. To further improve knowledge distillation, we
integrate both online and self-distillation into CTSL-MKT with more
informative relation-based knowledge.

###  2.2 Collaborative Learning

Recently, there are many new online distillation methods that train a teacher
and a student simultaneously during knowledge transfer. Collaborative learning
is the one used most often [25, 29, 32, 8, 9, 7], where the teacher and the
student as peer networks collaboratively teach and learn from each other, and
the peer network architectures can be different. In particular, Zhang et al.
[25] proposed a deep mutual learning method (DML) for online distillation
using the response-based knowledge. DML uses an ensemble of soft logits as
knowledge and transfers it among arbitrary peer networks via collaborative
learning [9]. Yao and Sun [8] further extended DML with dense cross-layer
mutual-distillation, which learns both the teacher and the student
collaboratively from scratch.

Unlike the ensemble of peer networks, the advantage of a mutual knowledge
distillation method is that it can fuse features of peer networks to
collaboratively learn a powerful classifier [32]. However, the knowledge
distilled by those online mutual distillation methods is limited to the
response-based knowledge from individual instance features. In contrast, our
work can further make use of the relation-based knowledge from instance
relationships to further enrich the transferred knowledge.

###  2.3 Structural Knowledge

Most of knowledge distillation approaches adopt the output logits of a deep
model from individual samples as knowledge and make the logits of the teacher
and student match each other. However, such response-based knowledge ignores
the structural knowledge from the mutual relations of data examples, known as
relation-based knowledge. In recent years, there are some newly proposed
knowledge distillation methods based on structural relations of data samples
[26, 30, 31, 6, 5, 4]. Park et al. [26] proposed a relational knowledge
distillation method (RKD), which transfers the instance relation knowledge
from a teacher to a student. Chen et al. [4] borrowed the idea of manifold
learning to design a novel knowledge distillation method, in which the student
preserves the feature embedding similarities of samples from the teacher. Peng
et al. [31] designed a knowledge transfer method that makes sure the student
matches the instance correlation consistently with the teacher.

However, those structural knowledge distillation methods often ignore the
knowledge directly from individual samples. Our proposed CTSL-MKT instead
considers the knowledge from both individual instances and instance
relationships, and the bidirectional knowledge transfer is carried out between
peer networks via collaborative learning.

##  3 The Proposed CTSL-MKT Method

Figure 2: The framework of CTSL-MKT with two peer networks. Note that the
losses $L_{KL}(\emph{{p}}_{1},\emph{{p}}_{2})$ and
$L_{KL}(\emph{{p}}_{2},\emph{{p}}_{1})$ are for mutual learning via the
response-based knowledge transfer, $L_{RD}$ for mutual learning via the
relation-based knowledge transfer,
$L_{SD}^{k}(\emph{{p}}_{k}^{t},\bar{\emph{{p}}}_{k}^{t})$ for self-learning
via the response-based knowledge transfer.

CTLS-MKT unifies student self-learning and teacher-student mutual learning
under one framework in such a way that it can utilise multiple types of
knowledge and more than one distillation methods during the teacher-student
learning. The teacher-student architectures used in CTSL-MKT are peer
networks, such as ResNet [41] and MobilenetV2 [42]. Different from previous
works, CTSL-MKT distills both the response-based knowledge from individual
instance features and the relation-based knowledge from instance
relationships. During teacher-student mutual learning, peer networks trained
collaboratively can teach each other via online distillation with the two
kinds of knowledge. Meanwhile, each peer network can also self-learn via self-
distillation with the response-based knowledge. The two leaning processes
working together can complement each other to explore different knowledge
space and enhance the learning. Our CTSL-MKT can be seen as a new model
compression technique that generalises the existing self-distillation and
online distillation methods and enables fast computations and improves the
generalization ability. The overall framework of CTSL-MKT with two peer
networks is shown in Figure 2 as an example, and notations are summarized in
Table 1.

Notations Descriptions $X=\\{x_{1},x_{2},\cdots,x_{n}\\}$ $n$ input samples
from $m$ classes $\emph{{y}}=\\{y^{1},y^{2},\cdots,y^{m}\\}$ the one-hot label
vector for $x\in X$
$\emph{{z}}_{k}(x)=\\{z_{k}^{1},z_{k}^{2},\ldots,z_{k}^{m}\\}$ logits of a
network $N_{k}$ for $x\in X$ where $z_{k}^{i}$ is the logit for class $i$
$\sigma_{i}(\emph{{z}}_{k}(x),t)$ softmax function with temperature $t$
$p_{k}^{it}=\sigma_{i}(\emph{{z}}_{k}(x),t)$ output of softmax for $z_{k}^{i}$
$\emph{{p}}_{k}^{t}=\\{p_{k}^{1t},p_{k}^{2t},\ldots,p_{k}^{mt}\\}$ predictions
of $N_{k}$ with $t$
$\emph{{p}}_{k}=\\{p_{k}^{1},p_{k}^{2},\ldots,p_{k}^{m}\\}$ predictions of
$N_{k}$ when $t=1$ $f(s_{1}^{k},s_{2}^{k},\cdots,s_{n}^{k})$ similarity loss
of $n$ samples in $N_{k}$

Table 1: Notations used in CTSL-MKT.

###  3.1 Teacher-Student Mutual Learning

Teacher-student mutual learning contains the response-based knowledge transfer
and the relation-based knowledge transfer among peer network architectures.

Response-Based Knowledge Transfer: The response-based knowledge (i.e., the
output of a peer network) is learned from individual instances. Given a peer
network $N_{k}$ and its output $\emph{{p}}_{k}$ with temperature parameter
$t=1$, the collaborative response-based knowledge transfer makes the student
network $N_{k}$ imitate the teacher network $N_{k^{\prime}}$ ($k\neq
k^{\prime}$) with the following Kullback-Leibler (KL) divergence loss,

$\small L_{KL}(\emph{{p}}_{k},\emph{{p}}_{k^{\prime}})=\sum_{x\in
X}\sum_{i=1}^{m}\sigma_{i}(\emph{{z}}_{k^{\prime}}(x),1)log\frac{\sigma_{i}(\emph{{z}}_{k^{\prime}}(x),1)}{\sigma_{i}(\emph{{z}}_{k}(x),1)}.$
(1)

Similarly, the loss that the student network $N_{k^{\prime}}$ uses to learn
from the teacher network $N_{k}$ is
$L_{KL}(\emph{{p}}_{k^{\prime}},\emph{{p}}_{k})$.

During the collaborative learning for a classification task, each peer network
$N_{k}$ will then be trained with both the KL divergence loss (Eq (1)) and the
cross-entropy (CE) loss (Eq (2)).

$L_{CE}(\emph{{y}},\emph{{p}}_{k})=-\sum_{x\in
X}\sum_{i=1}^{m}y^{i}log(\sigma_{i}(\emph{{z}}_{k}(x),1))~{}.$ (2)

Take two peer networks in Figure 2 as example, the losses used to train
$N_{1}$ and $N_{2}$ will be
$L_{CE}(\emph{{y}},\emph{{p}}_{1})+L_{KL}(\emph{{p}}_{1},\emph{{p}}_{2})$ and
$L_{CE}(\emph{{y}},\emph{{p}}_{2})+L_{KL}(\emph{{p}}_{2},\emph{{p}}_{1})$,
respectively.

Relation-Based Knowledge Transfer: CTSL-MKT further integrates the relation-
based knowledge learned from the instance relationships via the teacher-
student mutual leaning in order to enrich the transferred knowledge and
enhance the teacher guidance. Let $s_{j}^{k}=\phi_{k}(x_{j})$ (where
$\phi_{k}(.)$ is a feature mapping function of $N_{k}$) be the output of any
layer of the network $N_{k}$ for $x_{j}$, and $\chi^{\tau}$ denote a set of
$\tau$-tuples of different samples. A set of $2$-tuples and a set of
$3$-tuples thus correspond to $\chi^{2}=\left\\{(x_{u},x_{v})|u\neq
v\right\\}$ and $\chi^{3}=\left\\{x_{u},x_{v},x_{w})|u\neq v\neq w\right\\}$,
respectively. As in [26], the relation-based knowledge learned by the network
$N_{k}$ can be modelled jointly by a distance-wise function and an angle-wise
function.

Given $N_{k}$, the distance-wise function captures the similarities between
two samples in a $2$-tuple, which is defined as

$f(s_{u}^{k},s_{v}^{k})=\frac{1}{\pi}||s_{u}^{k}-s_{v}^{k}||_{2}~{},$ (3)

where
$\pi=\frac{1}{|\chi^{2}|}\sum_{(x_{u},x_{v})\in\chi^{2}}||s_{u}^{k}-s_{v}^{k}||_{2}$
is a normalization constant. Accordingly, the instance relationships between
any two peer networks $N_{k}$ and $N_{k^{\prime}}$ are transferred by the
following distance-wise distillation loss

$L_{DD}(x_{u},x_{v})=\sum_{(x_{u},x_{v})\in\chi^{2}}R\big{(}f(s_{u}^{k},s_{v}^{k}),f(s_{u}^{k^{\prime}},s_{v}^{k^{\prime}})\big{)}~{},$
(4)

where $R(.)$ is Huber loss that reflects instance relationships and is defined
as

$R(a,b)=\left\\{\begin{array}[]{lr}\frac{1}{2}(a-b)^{2},\quad if~{}|a-b|\leq
1&\\\ |a-b|-\frac{1}{2},\quad otherwise&\end{array}\right.~{}.$ (5)

Furthermore, the similarities between samples in a $3$-tuple are measured by
an angle-wise function

$f(s_{u}^{k},s_{v}^{k},s_{w}^{k})=\cos\angle
s_{u}^{k}s_{v}^{k}s_{w}^{k}=<e^{uv},e^{wv}>~{},$ (6)

where $e^{uv}=\frac{s_{u}^{k}-s_{v}^{k}}{||s_{u}^{k}-s_{v}^{k}||_{2}}$ and
$e^{wv}=\frac{s_{w}^{k}-s_{v}^{k}}{||s_{w}^{k}-s_{v}^{k}||_{2}}$. The instance
relationships are transferred between any two peer networks $N_{k}$ and
$N_{k^{\prime}}$ with the angle-wise distillation loss, defined as

$\displaystyle L_{AD}(x_{u},x_{v},x_{w})$ (7) $\displaystyle=$
$\displaystyle\sum_{(x_{u},x_{v},x_{w})\in\chi^{3}}R\big{(}f(s_{u}^{k},s_{v}^{k},s_{w}^{k}),f(s_{u}^{k^{\prime}},s_{v}^{k^{\prime}},s_{w}^{k^{\prime}})\big{)}~{}.$

It has been shown that the relation-based knowledge transfer can be more
effective if the distance-wise function is used jointly with the angle-wise
function [26], as they capture different degrees of similarities between
samples. We formulate the instance relation distillation loss used in the
collaborative learning between peer networks as

$L_{RD}=L_{DD}(x_{u},x_{v})+\beta_{1}L_{AD}(x_{u},x_{v},x_{w})~{},$ (8)

where $\beta_{1}$ is a tuning parameter that controls the balance between loss
terms.

Consequently, the mutual distillation loss with both the response-based and
the relation-based knowledge between two peer networks ($N_{k}$ and
$N_{k^{\prime}}$) is defined as: for network $N_{k}$, we have

$L_{MD}^{k}=L_{RD}+\beta_{2}L_{KL}(\emph{{p}}_{k},\emph{{p}}_{k^{\prime}})~{},$
(9)

where $\beta_{2}$ is a tuning parameter; for network $N_{k^{\prime}}$, we have

$L_{MD}^{k^{\prime}}=L_{RD}+\beta_{2}L_{KL}(\emph{{p}}_{k^{\prime}},\emph{{p}}_{k})~{}.$
(10)

Algorithm 1 The proposed CSL-MKT

0: Input samples $X$ with labels, learning rate $\eta$, hyperparameters
$\alpha$, $\beta$, $\gamma$, $\beta_{1}$ and $\beta_{2}$.

1: Initialize: Initialize peer networks $N_{1}$ and $N_{2}$ to different
conditions.

2: Stage 1: Pre-train $N_{1}$ and $N_{2}$ for use of the process of self-
learning.

3: for k=1 to 2 do

4: Repeat:

5: Compute stochastic gradient of $L_{CE}$ in Eq. (2) and update $N_{k}$:

6: $N_{k}\leftarrow\text{$N_{k}$}$+$\eta$$\frac{\partial L_{CE}}{\partial
N_{k}}$.

7: Until: $L_{CE}$ converges.

8: end for

9: Stage 2: Train $N_{1}$ and $N_{2}$ collaboratively.

10: Repeat:

11: for k=1 to 2 do

12: Compute stochastic gradient of $L_{KD}^{k}$ in Eq. (12) and update
$N_{k}$:

13: $N_{k}\leftarrow\text{$N_{k}$}$+$\eta$$\frac{\partial L_{KD}^{k}}{\partial
N_{k}}$.

14: end for

15: Until: $L_{KD}^{k}$ converges.

16: return _$N_{k}$_.

###  3.2 Student Self-learning

During the collaborative learning between peer networks, if the outputs of
peer networks are very diverse, the mutual knowledge transfer could become
poor. Since the self-learning via self-distillation can improve the power of
knowledge transfer [23], CTSL-MKT further introduces the self-learning of each
peer network into collaborative learning via the response-based knowledge
self-distillation. To conduct self-learning for each peer network $N_{k}$, we
use the outputs $\bar{\emph{{p}}}_{k}^{t}$ of the pre-trained network $N_{k}$
to supervise itself with the following self-distillation loss:

$\small L_{SD}^{k}(\emph{{p}}_{k}^{t},\bar{\emph{{p}}}_{k}^{t})=\sum_{x\in
X}\sum_{i=1}^{m}\sigma_{i}^{t}(\bar{\emph{{z}}}_{k}(x),t)log\frac{\sigma_{i}^{t}(\bar{\emph{{z}}}_{k}(x),t)}{\sigma_{i}^{t}(\emph{{z}}_{k}(x),t)}~{}.$
(11)

###  3.3 The CSL-MKT Algorithm

Finally, CSL-MKT conducts mutual learning and self-learning simultaneously in
a unified framework, as shown in Algorithm 1. Its objective function for each
pear network is defined as

$L_{KD}^{k}=\alpha L_{CE}^{k}+\beta L_{MD}^{k}+\gamma L_{SD}^{k}~{},$ (12)

where $\alpha$, $\beta$ and $\gamma$ are the tuning parameters, which balance
the contribution of each loss in the collaborative learning, $L_{CE}^{k}$ is
defined in Eq. (2), $L_{MD}^{k}$ for two peer networks in Eqs. (9) or (10),
and $L_{SD}^{k}$ in Eq. (11).

##  4 Experiments

Network | Parameter size | Baseline
---|---|---
(CIFAR-100) | B_10 | B_100 | B_Tiny
ResNet14 | 6.50M | 94.94 | 76.12 | -
ResNet18 | 11.22M | 95.13 | 75.77 | 62.90
ResNet34 | 21.33M | 95.39 | 77.66 | -
VGG19 | 139.99M | 92.83 | 69.42 | -
MobileNetV2 | 2.37M | 90.97 | 68.23 | -
ShuffleNetV2 | 1.36M | 91.03 | 70.10 | -
AlexNet | 57.41M | - | - | 50.25
SqueezeNet | 0.77M | - | - | 43.68
Table 2: The parameter size of each peer network on CIFAR-100 and its
classification performance on three datasets. Note that B_10, B_100, and
B_Tiny denote Top-1 accuracy (%) achieved by each peer network on CIFAR-10,
CIFAR-100 and Tiny-ImageNet, respectively.

We conducted extensive experiments to verify the effectiveness of CTSL-MKT on
image classification tasks using datasets including CIFAR-10 [47], CIFAR-100
[47], Tiny-ImageNet [48] and Market-1501 [49]. The peer network architectures
were chosen from ResNet [41], MobileNet [42], ShuffleNet [43], VGG [45],
AlexNet [44] and SqueezeNet [46]. CTSL-MKT was compared to the state-of-the-
art KD methods, which are DML [25], Tf-KD [23] and RKD [26]. For a fair
comparison, RKD uses online distillation with the peer networks. In all the
experiments, the relation-based knowledge was modelled by the final feature
embedding outputs by the peer networks.

###  4.1 Datasets and Settings

CIFAR-10 and CIFAR-100. Both datasets have 60,000 $32\times 32$ images, where
50,000 images are for training and the other 10,000 images are for testing.
The number of classes is 10 for CIFAR-10 and 100 for CIFAR-100, and each class
has the same numbers of samples in both the training and the testing sets. On
each dataset, data augmentation with random crops and horizontal flips was
used to change the zero-padded $40\times 40$ images to $32\times 32$ ones. The
peer networks were trained for 200 epochs with batch size 128 and initial
learning rate 0.1 which is then multiplied by 0.2 at 60, 120, and 160 epochs.
Temperature parameter was set to 10 for CIFAR-10 and 3 for CIFAR-100.

Tiny-ImageNet. Tiny-ImageNet contains 100,000 training and 10,000 testing
$64\times 64$ images from 200 classes, each of which has the same number of
samples. Each image was randomly resized to $224\times 224$. The peer networks
were trained for 90 epochs with batch size 64 and initial learning rate 0.1
which is then multiplied by 0.1 at 30, 60, 80 epochs. Temperature parameter
was set to 2.

Market-1501. Market-1501 includes 32,688 images taken from 1,501 identities
under condition of six camera views. It has 751 identities for training and
750 ones for testing. Each image was zero-padded by 10 pixels on each side,
and data augmentation with random crops and horizontal flips was used to
change the zero-padded images to $256\times 128$ ones. The peer networks were
trained for 60 epochs with batch size 32 and initial learning rate 0.05 which
is then multiplied by 0.2 at 40 epochs. Temperature parameter was set to 6.

We used the SGD optimizer for training the peer networks with momentum 0.9 and
weight decay 5e-4, and all input images were normalized by Mean-Std
normalization. All the hyper-parameters were greedily searched and set as
follows. The hyper-parameters used on CIFAR and Tiny-ImageNet were set as
$\alpha=0.1$, $\beta=0.05$ and $\gamma=0.9$ for MobileNet and ShuffleNet, and
$\alpha=0.4$, $\beta=0.4$ and $\gamma=0.6$ for the other networks. On
Market-1501, the hyper-parameters were set as $\alpha=1$, $\beta=0.9$ and
$\gamma=1$ for all the networks. Besides, both $\beta_{1}$ and $\beta_{2}$
were set to 2. In all the experiments, we considered a pair of peer networks
that have the same architecture or different architectures. Table 2 shows the
parameter size of each network on CIFAR-100 and its top-1 accuracy on the two
CIFAR datasets and the Tiny-ImageNet dataset, which serves as a baseline.

###  4.2 Results on CIFAR-10

Table 3 reports the average Top-1 accuracy of CTSL-MKT and the state-of-the-
art competitors on CIFAR-10. It is not surprising that those knowledge
distillation methods perform better than the corresponding single network due
to the knowledge transfer, except for DML and RKD with ResNet14 and ResNet18.
The possible reason for the slightly poor performance of DML and RKD with
ResNet14 or ResNet18 could be that the discrepancies between outputs of small
peer networks for individual instances hinder mutual learning. Among those
knowledge distillation methods, CTSL-MKT performs the best with a significant
improvement, which indicates that our idea of collaborative learning with
multiple knowledge transfer is effective. Meanwhile, CTSL-MKT outperforms all
the corresponding baselines shown in Table 2 with a noticeable margin. For
example, CTSL-MKT with ShuffleNetV2-MobileNetV2 has increased the Top-1
accuracy by 1.68% and 1.18%, compared with the corresponding signal network
baselines, i.e., ShuffleNetV2 and MobileNetV2 respectively. Moreover, although
CTSL-MKT, DML, and RKD collaboratively learn the two peer networks, which can
have the same network structures or different ones, each peer network in CTSL-
MKT (i.e., CTSL-MKT_$N_{1}$ or CTSL-MKT_$N_{2}$) performs much better than its
counterpart in DML and RKD, due to the multiple knowledge transfer.

Network $N_{1}$ Network $N_{2}$ Tf-KD DML_$N_{1}$ DML_$N_{2}$ RKD_$N_{1}$
RKD_$N_{2}$ CTSL-MKT_$N_{1}$ CTSL-MKT_$N_{2}$ ResNet14 ResNet14 95.08$\pm$0.01
94.78$\pm$0.02 94.92$\pm$0.01 94.95$\pm$0.01 94.83$\pm$0.02 95.28$\pm$0.01
95.22$\pm$0.01 ResNet18 ResNet18 95.20$\pm$0.01 94.88$\pm$0.01 94.99$\pm$0.01
94.98$\pm$0.04 94.92$\pm$0.01 95.29$\pm$0.04 95.33$\pm$0.03 ResNet34 ResNet34
95.41$\pm$0.01 95.42$\pm$0.01 95.32$\pm$0.01 95.45$\pm$0.01 95.45$\pm$0.01
95.69$\pm$0.03 95.59$\pm$0.01 MobileNetV2 MobileNetV2 91.72$\pm$0.01
91.19$\pm$0.07 91.32$\pm$0.04 91.12$\pm$0.03 90.71$\pm$0.06 92.12$\pm$0.02
92.12$\pm$0.02 ShuffleNetV2 ShuffleNetV2 92.47$\pm$0.01 91.97$\pm$0.03
91.92$\pm$0.01 92.08$\pm$0.01 91.59$\pm$0.01 92.64$\pm$0.01 92.49$\pm$0.01
ResNet18 ResNet34 - 95.09$\pm$0.01 95.41$\pm$0.03 95.12$\pm$0.01
95.31$\pm$0.01 95.24$\pm$0.01 95.60$\pm$0.01 ResNet18 VGG19 - 95.11$\pm$0.03
93.49$\pm$0.02 95.03$\pm$0.01 93.50$\pm$0.01 95.16$\pm$0.01 93.91$\pm$0.01
ShuffleNetV2 MobileNetV2 - 91.78$\pm$0.03 91.25$\pm$0.08 91.73$\pm$0.01
90.72$\pm$0.01 92.71$\pm$0.01 92.15$\pm$0.01

Table 3: The average Top-1 accuracy (%) over three individual runs on
CIFAR-10.

###  4.3 Results on CIFAR-100

Table 4 reports the average Top-1 accuracy of all the competing knowledge
distillation methods with various network architectures on CIFAR-100. We have
similar observations as those on CIRFAR-10. Overall, each competing method
improves on the performance of the corresponding baseline, and CTSL-MKT gains
the largest improvement. Compare to Tf-KD, DML and RKD, Top-1 accuracy of each
peer network in CTSL-MKT has been improved by about 1% on average. For
example, the accuracy of the two MobileNetV2 networks in CTSL-MKT has been
increased by 2.46% and 2.68% respectively, compared to those in DML, and
increased by 3.08% and 2.96% Top-1 accuracy compared to those in RKD.

Network $N_{1}$ Network $N_{2}$ Tf-KD DML_$N_{1}$ DML_$N_{2}$ RKD_$N_{1}$
RKD_$N_{2}$ CTSL-MKT_$N_{1}$ CTSL-MKT_$N_{2}$ ResNet14 ResNet14 76.67$\pm$0.02
75.97$\pm$0.01 76.16$\pm$0.11 76.36$\pm$0.03 76.30$\pm$0.01 77.00$\pm$0.05
76.85$\pm$0.04 ResNet18 ResNet18 77.04$\pm$0.12 76.10$\pm$0.10 76.27$\pm$0.07
76.43$\pm$0.01 76.09$\pm$0.01 77.43$\pm$0.10 77.46$\pm$0.01 ResNet34 ResNet34
77.93$\pm$0.01 77.88$\pm$0.12 77.61$\pm$0.03 77.63$\pm$0.03 77.65$\pm$0.05
78.58$\pm$0.01 78.24$\pm$0.02 MobileNetV2 MobileNetV2 70.82$\pm$0.02
68.98$\pm$0.01 68.58$\pm$0.18 68.36$\pm$0.01 68.30$\pm$0.01 71.44$\pm$0.06
71.26$\pm$0.08 ShuffleNetV2 ShuffleNetV2 71.79$\pm$0.02 70.47$\pm$0.15
70.29$\pm$0.04 70.24$\pm$0.01 69.98$\pm$0.03 72.13$\pm$0.02 71.69$\pm$0.05
ResNet18 ResNet34 - 76.15$\pm$0.10 77.71$\pm$0.01 76.41$\pm$0.05
77.83$\pm$0.01 77.61$\pm$0.08 78.15$\pm$0.12 ResNet18 VGG19 - 76.51$\pm$0.02
68.80$\pm$3.74 76.29$\pm$0.02 68.28$\pm$0.87 77.23$\pm$0.02 72.72$\pm$0.06
ShuffleNetV2 MobileNetV2 - 70.47$\pm$0.13 68.83$\pm$0.14 70.50$\pm$0.28
67.87$\pm$0.01 72.46$\pm$0.15 71.34$\pm$0.09

Table 4: The average Top-1 accuracy (%) over three individual runs on
CIFAR-100.

(a) ShuffleNetV2 (b) MobileNetV2

Figure 3: The values of training loss over epochs on the CIFAR-100 training
dataset.

(a) ShuffleNetV2 (b) MobileNetV2

Figure 4: The values of Top-1 accuracy over epochs on the CIFAR-100 testing
dataset.

To further illustrate the learning process of the peer networks in CTSL-MKT,
Figure 3 plots the training loss of ShuffleNetV2 and MobileNetV2 as a function
of epochs, compared to Tf-KD, DML and RKD. It shows that CTSL-MKT and Tf-KD
converge better than DML and RKD. The possible reason is that each network can
self-learn in CTSL-MKT and Tf-KD to overcome the discrepancy in the outputs of
peer networks in DML and RKD during learning. Although CTSL-MKT with multiple
knowledge transfer introduces extra hyper-parameters, it can still converge
faster, achieving comparable training loss with Tf-KD. The loss becomes stable
around 120 epochs in general. Furthermore, Figure 4 displays the corresponding
Top-1 accuracy of each peer network after each epoch on the testing dataset.
It shows that the proposed CTSL-MKT outperforms the others after the
convergence, its performance improves along with the decrement of the training
loss. Overall, the patterns show that two peer networks in CTSL-MKT can work
collaboratively, via teaching and learning from each other at each epoch, and
each network gradually improves itself to achieve a better performance.

###  4.4 Results on Tiny-ImageNet

Table 5 shows the average Top-1 accuracy of the competing methods with five
various peer network architectures on Tiny-ImageNet. From the comparative
results, it can be seen that CTSL-MKT significantly outperforms the baselines,
DML, RKD and Tf-KD. However, on these five peer network architectures, some
peer networks in DML, RKD and Tf-KD achieve poor performance, compared to
their baselines. The possible reason is that the used peer networks are
smaller with less informative knowledge in knowledge transfer and the outputs
of peer networks for the same individual instances migth be different, which
degrades the effectiveness of mutual learning. With multiple kinds of
knowledge and distillation strategies, our CTSL-MKT can well improve the
performance via mutual learning.

Network $N_{1}$ Network $N_{2}$ Tf-KD DML_$N_{1}$ DML_$N_{2}$ RKD_$N_{1}$
RKD_$N_{2}$ CTSL-MKT_$N_{1}$ CTSL-MKT_$N_{2}$ ResNet18 ResNet18 63.29$\pm$0.02
62.30$\pm$0.01 62.39$\pm$0.03 62.80$\pm$0.01 62.42$\pm$0.08 63.63$\pm$0.08
63.64$\pm$0.02 AlexNet AlexNet 49.78$\pm$0.01 44.47$\pm$0.01 44.80$\pm$0.01
43.54$\pm$0.01 42.97$\pm$0.01 51.39$\pm$0.01 51.28$\pm$0.01 SqueezeNet
SqueezeNet 41.66$\pm$0.01 47.16$\pm$0.03 46.95$\pm$0.19 48.22$\pm$0.01
48.55$\pm$0.09 48.60$\pm$0.30 48.86$\pm$0.03 AlexNet SqueezeNet -
44.35$\pm$0.51 46.15$\pm$0.30 44.66$\pm$1.87 46.86$\pm$0.41 50.98$\pm$0.08
47.99$\pm$0.03 ResNet18 AlexNet - 62.62$\pm$0.11 43.53$\pm$0.62 62.37$\pm$0.01
46.64$\pm$0.03 63.37$\pm$0.01 51.56$\pm$0.02

Table 5: The average Top-1 accuracy (%) over three individual runs on Tiny-
ImageNet.

###  4.5 Results on Market-1501

We further compared those methods on Market-1501, which is used for a re-
identification (re-id) task. In this set of experiments, we adopted ResNet50
that is usually used on this dataset to form a peer network architecture.
Figure 5 shows the performance of Tf-KD, DML, RKD and CTSL-MKT, measured by
Rank-1, Rank-5, Rank-10 and mAP. Note that these results were computed for
only one peer network. DML and RKD with collaborative learning consistently
perform better than Tf-KD via self-learning. Our CTSL-MKT outperforms both DML
and RKD across all the metrics. Specifically, in terms of mAP, the improvement
of CTSL-MKT over DML, RKD and Tf-KD is 1.22%, 0.8% and 4.69%, respectively.

Figure 5: Comparative results (%) on Market-1501.

###  4.6 Ablation Study

CTSL-MKT contains three knowledge distillation strategies, i.e., mutual
learning via response-based knowledge transfer from individual instances
(MLI), mutual learning via relation-based knowledge transfer from instance
relationships (MLR) and self-learning via response-based knowledge transfer
from individual instances (SLI). To study how each strategy contributes to the
model performance, we consider the following four variants of CTSL-MKT:

  1. _A_)

the full model using the three strategies altogether, where we used both
online distillation and self-distillation with the two kinds of knowledge;

  2. _B_)

the model using online distillation only with both the response-based
knowledge (MLI) and the relation-based knowledge (MLR);

  3. _C_)

the model using online distillation with the relation-based knowledge (MLR)
and self-distillation with the response-based knowledge (SLI);

  4. _D_)

the model using both online distillation and self-distillation with only the
response-based knowledge, corresponding to MLI + SLI.

Table 6 reports the average Top-1 accuracy of these four variations with
different pairs of peer network architectures. We have the following
observations: 1) Variant A (i.e., the full model) outperforms the other
variants where one knowledge distillation strategy has been removed. It
implies that the use of multiple types of knowledge with both online
distillation and self-distillation plays a crucial role in the performance
gain. 2) Variant B without self-distillation has the largest performance drop,
compared with variants C and D. It indicates that self-distillation
contributes substantially to the overall performance as it could offset the
diversity issue caused by mutual learning and further enhance the knowledge
distillation efficiency. 3) DML, Tf-KD and RKD can be seen as a special case
of CTSL-MKT using only one knowledge distillation strategy. Jointly looking at
Tables 6 and 4 reveals that knowledge distillation methods with two or more
strategies almost always outperform those using only one strategy. Therefore,
it is clear that knowledge distillation via proper multiple knowledge transfer
is very beneficial for improving the performance of model compression.

Case MLI MLR SLI ResNet14 ResNet14 ResNet18 ResNet18 MobileNetV2 MobileNetV2 A
✓ ✓ ✓ 77.00$\pm$0.05 76.85$\pm$0.04 77.43$\pm$0.10 77.46$\pm$0.01
71.44$\pm$0.06 71.26$\pm$0.08 B ✓ ✓ ✗ 76.57$\pm$0.04 76.37$\pm$0.02
76.67$\pm$0.04 76.66$\pm$0.09 69.10$\pm$0.01 69.23$\pm$0.04 C ✗ ✓ ✓
76.69$\pm$0.02 76.70$\pm$0.04 77.35$\pm$0.04 77.29$\pm$0.08 71.18$\pm$0.05
71.10$\pm$0.08 D ✓ ✗ ✓ 76.73$\pm$0.03 76.70$\pm$0.08 77.26$\pm$0.07
77.12$\pm$0.04 71.14$\pm$0.02 71.04$\pm$0.10

(a) Ablation experiments on the same peer network architectures

Case MLI MLR SLI ResNet14 ResNet18 ResNet18 ResNet34 ShuffleNetV2 MobileNetV2
A ✓ ✓ ✓ 77.07$\pm$0.03 77.28$\pm$0.04 77.61$\pm$0.08 78.15$\pm$0.12
72.46$\pm$0.15 71.34$\pm$0.09 B ✓ ✓ ✗ 76.35$\pm$0.01 76.53$\pm$0.07
76.53$\pm$0.02 77.83$\pm$0.01 70.57$\pm$0.03 68.69$\pm$0.01 C ✗ ✓ ✓
76.69$\pm$0.23 77.06$\pm$0.02 77.12$\pm$0.01 77.99$\pm$0.03 72.06$\pm$0.04
71.10$\pm$0.11 D ✓ ✗ ✓ 76.68$\pm$0.03 77.13$\pm$0.02 77.39$\pm$0.01
77.68$\pm$0.01 72.20$\pm$0.09 71.05$\pm$0.15

(b) Ablation experiments on the different peer network architectures

Table 6: Ablation study of CTSL-MKT in terms of the average Top-1 accuracy
over three individual runs on CIFAR-100.

###  4.7 Experiment Discussion

The experimental results reported above have demonstrated the effectiveness of
the proposed CTSL-MKT, while being compared with several state-of-the-art
knowledge distillation methods. We have the following remarks:

  * •

Collaborative learning can make peer networks teach and learn from each other,
and iteratively improve themselves.

  * •

Self-learning of each peer network can further enhance the ability of mutual
learning among peer networks by compensating the loss caused by the diversity
issue.

  * •

Multiple knowledge transfer with more than one types of knowledge and
distillation strategies can significantly improve the KD performance.

  * •

Various peer network architectures (i.e., teacher-student architectures) can
be easily adopted for knowledge transfer via collaborative learning.

##  5 Conclusions

In this paper, we propose a novel knowledge distillation method called
collaborative teacher-student learning via multiple knowledge transfer (CTSL-
MKT). It naturally integrates both self-learning via self-distillation and
collaborative learning via online distillation in a unified framework so that
multiple kinds of knowledge can be transferred effectively in-between
different teacher-student architectures, and CTSL-MKT can achieve individual
instance consistency and instance correlation consistency among the peer
networks. Experimental results on four image classification datasets have
demonstrated CTSL-MKT outperforms the competitors with a noticeable margin,
which proves the necessity of using different distillation schemes to transfer
multiple types of knowledge simultaneously. We believe that our proposed
framework opens a door to design multiple knowledge transfer for knowledge
distillation.

## References

  * [1] J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge Distillation: A Survey,” _arXiv:2006.05525_ , 2020.
  * [2] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning Filters for Efficient ConvNets,” _Int. Conf. Learn. Represent._ , 2017.
  * [3] S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both Weights and Connections for Efficient Neural Networks,” _Adv. Neural Inform. Process. Syst._ , 2015.
  * [4] H. Chen, Y. Wang, C. Xu, C. Xu, and D. Tao, “Learning student networks via feature embedding,” _IEEE Trans. Neur. Net. Lear._ , 2020.
  * [5] F. Tung, and G. Mori, “Similarity-preserving knowledge distillation,” _Int. Conf. Comput. Vis._ , 2019.
  * [6] Y. Liu, C. Shu, J. Wang, and C. Shen, “Structured knowledge distillation for dense prediction,” _IEEE Trans. Pattern Anal. Mach. Intell._ , 2020.
  * [7] K. Li, L. Yu, S. Wang, and P. A. Heng, “Towards Cross-Modality Medical Image Segmentation with Online Mutual Knowledge Distillation,” _AAAI_ , 2020.
  * [8] A. Yao, and D. Sun, “Knowledge Transfer via Dense Cross-Layer Mutual-Distillation,” _Eur. Conf. Comput. Vis._ , 2020.
  * [9] Q. Guo, X. Wang, Y. Wu, Z. Yu, D. Liang, X. Hu, and P. Luo, “Online Knowledge Distillation via Collaborative Learning,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020.
  * [10] C. Yang, L. Xie, C. Su, and A. L. Yuille, “OSnapshot distillation: Teacher-student optimization in one generation,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019.
  * [11] M. Phuong, and C. H. Lampert, “Distillation-based training for multi-exit architectures,” _Int. Conf. Comput. Vis._ , 2019.
  * [12] D. Walawalkar, Z. Shen, and M. Savvides, “Online Ensemble Model Compression using Knowledge Distillation,” _Eur. Conf. Comput. Vis._ , 2020.
  * [13] T. Li, J. Li, Z. Liu, and C. Zhang, “Few sample knowledge distillation for efficient network compression,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020.
  * [14] C. Shen, M. Xue, X. Wang, J. Song, L. Sun, and M. Song, “Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation,” _Int. Conf. Comput. Vis._ , 2019.
  * [15] J. Yim, D. Joo, J. Bae, and J. Kim, “A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2017.
  * [16] N. Passalis, M. Tzelepi, and A. Tefas, “Heterogeneous Knowledge Distillation using Information Flow Modeling,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020.
  * [17] L. Yu, V. O. Yazici, X. Liu, J. Weijer, Y. Cheng, and A. Ramisa, “Learning metrics from teachers: Compact networks for image embedding,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019.
  * [18] K. Xu, L. Rui, Y. Li, and L. Gu, “Feature Normalized Knowledge Distillation for Image Classification,” _Eur. Conf. Comput. Vis._ , 2020.
  * [19] Y. Guan, P. Zhao, B. Wang, Y. Zhang, C. Yao, K. Bian, and J. Tang, “Differentiable Feature Aggregation Search for Knowledge Distillation,” _Eur. Conf. Comput. Vis._ , 2020.
  * [20] G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” _arXiv: 1503.02531_ , 2015.
  * [21] A. Romero, N. Ballas, and S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “FitNets: Hints for Thin Deep Nets,” _Int. Conf. Learn. Represent._ , 2015.
  * [22] S.-I. Mirzadeh, M. Farajtabar, A. Li, and H. Ghasemzadeh, “Improved Knowledge Distillation via Teacher Assistant,” _AAAI_ , 2020.
  * [23] L. Yuan, F. E. Tay, G. Li, T. Wang, and J. Feng, “Revisiting Knowledge Distillation via Label Smoothing Regularization,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020.
  * [24] L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma, “Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation,” _Int. Conf. Comput. Vis._ , 2019.
  * [25] Y. Zhang, T. Xiang, T. M. Hospedales, and H. Lu, “Deep Mutual Learning,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2018.
  * [26] W. Park, D. Kim, Y. Lu, and M. Cho, “Relational Knowledge Distillation,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019.
  * [27] S. Yun, J. Park, K. Lee, and J. Shin, “Regularizing Class-wise Predictions via Self-knowledge Distillation,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020.
  * [28] X. Lan, X. Zhu, and S. Gong, “Knowledge Distillation by On-the-Fly Native Ensemble,” _IAdv. Neural Inform. Process. Syst._ , 2018.
  * [29] G. Wu and S. Gong, “Peer Collaborative Learning for Online Knowledge Distillation,” _IarXiv: 2006.04147_ , 2020.
  * [30] Y. Liu, J. Cao, B. Li, C. Yuan, W. Hu, Y. Li, and Y. Duan, “Knowledge Distillation via Instance Relationship Graph,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019.
  * [31] B. Peng, X. Jin, J. Liu, D. Li, Y. Wu, Y. Liu, S. Zhou, and Z. Zhang, “Correlation Congruence for Knowledge Distillation,” _Int. Conf. Comput. Vis._ , 2019.
  * [32] J. Kim, M. Hyun, I. Chung, and N. Kwak, “Feature Fusion for Online Mutual Knowledge Distillation,” _Int. Conf. Pattern Recog._ , 2019.
  * [33] S. Hou, X. Liu, and Z. Wang, “DualNet: Learn Complementary Features for Image Recognition,” _Int. Conf. Comput. Vis._ , 2017.
  * [34] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks,” _Eur. Conf. Comput. Vis._ , 2016.
  * [35] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized Neural Networks,” _Adv. Neural Inform. Process. Syst._ , 2016.
  * [36] D. Chen, J.-P. Mei, C. Wang, Y. Feng, and C. Chen, “Online Knowledge Distillation with Diverse Peers,” _AAAI_ , 2020.
  * [37] S. You, C. Xu, C. Xu, and D. Tao, “Learning from Multiple Teacher Networks,” _ACM SIGKDD_ , 2017.
  * [38] X. Wu, R. He, Y. Hu, and Z. Sun, “Learning an Evolutionary Embedding via Massive Knowledge Distillation,” _Int. J. Comput. Vis._ , 2020, pp. 2089–2106.
  * [39] T. Xu, and C. Liu, “Data-Distortion Guided Self-Distillation for Deep Neural Networks,” _AAAI_ , 2019.
  * [40] J. H. Cho, and B. Hariharan, “On the Efficacy of Knowledge Distillation,” _Int. Conf. Comput. Vis._ , 2019.
  * [41] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2016.
  * [42] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2018.
  * [43] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, “ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design,” _Eur. Conf. Comput. Vis._ , 2018.
  * [44] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” _Adv. Neural Inform. Process. Syst._ , 2012.
  * [45] K. Simonyan, and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” _Int. Conf. Pattern Recog._ , 2015.
  * [46] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and ¡0.5MB model size,” _Int. Conf. Pattern Recog._ , 2017.
  * [47] A. Krizhevsky, and G. Hinton, “Learning multiple layers of features from tiny images,” _Technical Report._ , 2009.
  * [48] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li, “ImageNet: A large-scale hierarchical image database,” _IEEE Conf. Comput. Vis. Pattern Recog._ , 2009.
  * [49] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, Q. Tian, “Scalable Person Re-Identification: A Benchmark,” _Int. Conf. Comput. Vis._ , 2015.