# Learning Multimodal Contact-Rich Skills from Demonstrations Without Reward
Engineering

Mythra V. Balakuntala1, Upinder Kaur1, Xin Ma∗1, Juan Wachs2, Richard M.
Voyles1 This work was supported by the NSF Center on RObots and SEnsors for
HUman well-Being (RoSe-HuB) under grant CNS-1439717 and by the Office of the
Assistant Secretary of Defense for Health Affairs under Award No.
W81XWH-18-1-0769.∗Corresponding author, 1School of Engineering Technology,
2School of Industrial Engineering, Purdue University, IN 47907, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Everyday contact-rich tasks, such as peeling, cleaning, and writing, demand
multimodal perception for effective and precise task execution. However, these
present a novel challenge to robots as they lack the ability to combine these
multimodal stimuli for performing contact-rich tasks. Learning-based methods
have attempted to model multi-modal contact-rich tasks, but they often require
extensive training examples and task-specific reward functions which limits
their practicality and scope. Hence, we propose a generalizable model-free
learning-from-demonstration framework for robots to learn contact-rich skills
without explicit reward engineering. We present a novel multi-modal sensor
data representation which improves the learning performance for contact-rich
skills. We performed training and experiments using the real-life Sawyer robot
for three everyday contact-rich skills – cleaning, writing, and peeling.
Notably, the framework achieves a success rate of 100% for the peeling and
writing skill, and 80% for the cleaning skill. Hence, this skill learning
framework can be extended for learning other physical manipulation skills.

##  I INTRODUCTION

Seemingly trivial contact-rich skills, such as peeling a cucumber or cleaning
a surface, demand seamless coordination among eyes and hands in humans. While
eyes provide object pose, environment state, and visual feedback; haptic
feedback from hands provides contact information between objects, relative
localization, and alternative control in case of loss of visual feedback.
Visual and tactile modalities – each help resolve ambiguities in the other –
thereby deliver an overall accurate perception of dynamic environments [1].
This synergistic interaction between the two modalities of sensing is
essential for contact-rich tasks, especially in the case of robots.

Robots, aiming to achieve human-level perception and proprioception for a
variety of tasks, need the ability to better understand and model their
interactions with the environment. While numerous published works in robotics
have recognized the utility of a multi-modal system [2, 3, 4], their
implementation has been limited to task-specific solutions. These works have
used extensive task-specific feature engineering coupled with prior knowledge
of task execution in model-based methods, which fairly limits their
generalized use. Subsequently, model-free methods have been introduced which
attempt to learn multi-modal contact-rich skills directly from observations.
These methods are preferred due to the low cost of implementation, re-
programmability, and the ability to model complex tasks previously out of the
scope of model-based methods.

Learning from Demonstration (LfD) is a popular approach which includes model-
free methods to “program” a robot by leveraging demonstrations of the tasks
[5, 6, 7]. Sequences of demonstrations allow the robots to comprehend the
objective, action space, and the manipulation required for completing the task
[8]. Architectures such as Gaussian Mixture Models, Hidden Markov Models, Deep
Neural Networks, and Dynamic Motion Primitives (DMP) have been explored for
learning from demonstrations [9, 5]. These methods can generalize using
extensive demonstrations of kinematic states for tasks such as grasping and
pushing simple objects. However, not only is getting large kinematic samples a
limitation of these methods, but they are also unable to handle perturbations
in the workspace [10, 11].

Figure 1: The task setup with human demonstrations to the robot for cleaning

Reinforcement learning-based methods solve some of these issues, as they do
not require explicit task specification, and can handle greater perturbations,
however, they need well-tuned reward functions [9]. Inverse reinforcement
learning overcomes this limitation by implicitly inferring rewards from
demonstrations [9, 12, 13]. Yet, most of these methods are modeled in closed-
loop systems around a single modality, majorly visual feedback [14, 15, 11,
16]. Using kinesthetic interfaces for teaching by demonstration enables
learning of contact-intensive tasks by guided demonstrations from the user
[17, 18]. However, such interfaces not only limit the domain of tasks that can
be taught but also puts the onus of effective learning on the guiding
expertise of the demonstrator. Moreover, teaching movements and skills by
kinesthetic learning demands long guided intervals that are cost and resource
intensive.

Contact-rich skills, such as peeling and writing, cannot be accomplished with
just a single modality as both force and visual feedback is necessary for
optimum execution. Few prior approaches have exploited the synergy between
vision and touch for contact-rich tasks [14]. Methods combining vision and
touch have been employed to increase the stability of grasp, or learn shape
completion for grasping [19, 20]. Further, other works utilize multiple
modalities for a manipulation task, however, they needed task-specific
strategies or a task-specific specified manipulation graph [21, 22, 23].

In this work, we leverage multiple sensor modalities to learn contact-rich
skills without task-specific reward engineering. Learning contact-rich skills
is a significant step for robots to operate in unstructured environments.
However, designing reward functions to learn such contact-rich skills is not
trivial or in some cases infeasible. In this framework, we infer the reward
from the goal states obtained from expert user demonstrations of the skill.
The demonstrations are semantically segmented to identify human-object
interactions and the goal states. Additionally, the interaction information is
used to define the action space for the policy. A representation for the state
is developed from the multi-sensor data for the policy, which can improve the
learning performance for contact-rich skills. To demonstrate the practicality
of this framework, we implement it using the Sawyer™robot. Three different
contact-rich tasks (writing, cleaning, and peeling) requiring varying force,
contact, and position interactions are demonstrated for testing the
performance of the proposed framework. The contributions of this work are:

  * •

A framework to learn contact-rich skill models from a few demonstrations
without reward engineering, which can remove the need for manual engineering
of rewards.

  * •

A multi-modal sensor data representation for learning contact-rich skills,
which improves the overall execution performance.

  * •

Evaluation of the framework on three varied real-life contact-rich skills –
cleaning, writing, and peeling.

This paper is organized as follows: Section II describes the problem statement
and the overall LfD framework. Section III describes the multi-modal skill
learning framework and the sub-components of this framework, such as the input
semantic segmentation and the execution controller. The experiments and
results are shown in Section IV for the three experiments with the conclusion
presented in Section V along with the discussion of future work.

##  II Problem Statement and LfD Framework

###  II-A Problem Statement

Teaching a robot contact-rich manipulation for a variety of tasks is a non-
trivial endeavor. Contact-rich tasks entail learning precise models of direct
low-level control to reduce contact forces and ensure stability for smooth
task execution [24]. The first step in this is for the robot to be able to
comprehend the multi-modal stimuli of the the skill being taught or executed.
Hence, just position-based control is insufficient; therefore, we use both
position and force control. The input multi-sensory data, consisting of visual
demonstration, depth inputs, force and tactile information, is first learned
by various networks. These learned representations update the skill modeling
policy.

The model executes in a finite bounded horizon, with a Markov decision process
$\mathcal{M}$, with state space $\mathcal{S}$ and action space $\mathcal{A}$.
For a horizon $T$, the state transitions according to the dynamics
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\longrightarrow\mathcal{S}$. A
learning policy $\pi(a|s)$ represents the probability of taking the action $a$
given a state $s$. The cumulative expected reward over horizon $T$ is given by
(1).

$J(\pi)=\mathbb{E}_{\pi}\left[\sum_{t=1}^{T}\phi(s_{t},a_{t})\right]$ (1)

This reward is limited in the range
$\phi:\mathcal{S}\times\mathcal{A}\to\left[-1,0\right]\bigcup\\{1\\}$, where
the negative rewards penalize the policy for sub-optimal convergence, and $1$
being the maximum reward given on optimal convergence. The optimal stochastic
policy $\pi^{*}$, for the model is achieved by maximizing the cumulative
expected reward $J(\pi)$.

Figure 2: The Learning from Demonstration (LfD) Framework

###  II-B LfD Framework

Unlike traditional Learning from Demonstration (LfD) approaches which rely on
large number of demonstrations to learn a task, in our previous work, we
presented a novel one-shot learning from demonstration approach, augmented by
coaching, to learn tasks from one expert demonstration [8], as shown in Figure
2. The demonstration is automatically segmented into a sequence of
parametrized a priori skills using a decision tree classifier. A self
evaluator updates the skill paramters based on the execution performance to
refine the task performance, locally optimizing cumulative performance. The a
priori skills are modelled controllers similar to DMPs [25]. The current paper
proposes a method to learn these contact-rich skills from expert
demonstrations, thereby eliminating the need for modelling the controllers.

Figure 3: The skill learning framework

##  III Multi-Modal Skill Learning Framework

The proposed skill learning framework is designed to learn contact-rich skills
from few visual demonstration without requiring an explicitly defined reward.
One drawback of using RL agents for real-world skills is the large exploration
space and the need for reward shaping. To overcome this issue, we propose to
use videos of expert human demonstrations to infer the action space and the
reward. State representations which capture action relevant information can
benefit from learning control behaviors [14]. For contact-rich tasks, such
representations should include force or contact information. We propose such a
representation combining the data from multiple sensing modalities. The robot
leverages this representation and the learnt reward to learn the execution
policy for the skill. The different components of the skill learning framework
are shown in Figure. 3.

###  III-A Input Semantic Segmentation

The input for learning these skills is a set of few visual human
demonstrations. Identifying human-object interactions is a significant step to
understand goal or intent of a skill [26]. Here, the frames that have such
human-object interactions are referred to as physical interaction keypoints
(PIKs). The PIKs provide information about the goal states of objects.

Instead of training an end-to-end detector for the PIKs, existing object
detection or segmentation networks are utilized to generate features.
Paramount to detecting physical interactions is identifying the hand and the
surfaces of the objects in the scene. The hand pose and the object instance
segmentation features are used as features to train a network to detect
interaction between the hands and any object in each frame. The hand pose,
interaction mode and hand motion can be used to infer whether the user is
performing position or force control along a particular axis. This information
can be leveraged to generate the action space for the skill policy.

The hand pose (6-D) and contours are obtained using a YoloV3 [27] detector.
The object surfaces are obtained using instance segmentation from MaskRCNN
[28]. Features are generated from the hand pose and the instance segments to
identify the PIK. The segmentation mask of dimension $128\times 128$ is passed
thorough a 4-layer convolutional network followed by a 3-layer multilayer
perceptron (MLP) to result in 24 dimension object features. These features are
concatenated with the hand poses of dimension $12$ to result in a $36$
dimension input to a 2-layer MLP network. The network outputs two binary
values, indicating if the left and right hand are interacting with any object
respectively.

The same $36$ features are used as input to another 2-layer MLP which outputs
the class of the object each hand is interacting with. The frames where the
interaction condition switches are denoted as PIKs. The first PIK denotes the
start of the skill and all further PIKs are assumed to be positive goal
states.

The motion of the hand, the hand pose, and the interaction condition in a
moving window of 10 frames is used to form the action space. The action space
for contact tasks is formulated as a tuple $(A_{x},A_{F})$. $A_{x}$ denotes
the domain for kinematic exploration using position changes. For a surface
contact task this will be all degrees of freedom (DoF) other than the contact
direction. $A_{F}$ indicates the force along the contact direction. Clearly,
$A_{F}$ and $A_{x}$ are along perpendicular directions in space. The goal
examples and the action space obtained from segmentation are used for the
policy training.

Figure 4: The skill learning framework

###  III-B Multi-Modal Sensor Representation

Learning contact-rich skills require inputs from force, position and visual
sensors to effectively understand the environment state. Combining data from
multiple sensors can effectively learn tasks such as grasping or peg insertion
[20, 14]. Predictive representations for state based on multi sensor data can
capture action-relevant information [29]. Inspired by this we propose a multi-
modal representation to capture contact conditions, the type of contact and
occlusion to camera based on the multiple sensor inputs.

Our model leverages deep networks to learn a representation for the state
using the information from 5 different inputs. The inputs include an RGB image
$I_{c}$, a depth image $I_{d}$, forces and torques in 6 DoF, tactile
measurements and joint positions. Fig. 4. The state representation consists of
four binary decision values - existence of contact, type of contact, presence
of tactile forces, and binary decision on camera occlusion.

The RGB and depth images are obtained from the Intel Realsense D435i. The
images have a size of $640\times 480$. The RGB images are rescaled to
$224\times 224$ and are passed through a ResNet-50 [30] backend and one fully
connected layer to provide output features of dimension 64. The depth images
capture important scene information regarding occlusions and object positions.
The depth images are rescaled to $128\times 128$, passed through 3
convolutional layers with maxpooling, followed by 3 fully connected layers.
The output depth features have a dimension of 32.

For detecting contact, information from the force-torque sensor and the
tactile sensors are significant. The 6 force-torque values are obtained from
the 6 DoF Force-Torque sensor on the Barrett hand. A $24\times 4$ dimension
tactile data is obtained from the tactile sensors, with 24 values for each
finger and 24 values for the palm. These values are flattened to form a one-
dimensional vector. The force-torque values, tactile values, and 7 joint
position values are concatenated and passed through a 3 layer MLP to produce
12 features. Finally, these 12 features along with the features from the color
and depth images are passed through a 2 layer MLP to predict a vector of 4
binary values $b=(b_{1},b_{2},b_{3},b_{4})$. A cross-entropy loss is used for
training the network.

The first output $b_{1}$ represents the contact state, a binary decision on if
there is contact or not. The second value $b_{2}$ indicates if the contact
type is point or line contact. The third value $b_{3}$ indicates a contact on
the finger tactile sensors. The final value $b_{4}$ is a decision on if the
scene is occluded by the arm. The relationship between vision and force to
detect contact enables inferring these conditions using visual and tactile
data. The contact can be seen and felt on the force torque sensor when the
robot touches the surface. Thus, contact information can be inferred by
combining data from various sensors.

The training of this network is performed using self-supervised methods to
avoid manual labelling. The training is carried out by using both a random
policy and a heuristic policy. The heuristic policy drives the robot to
establish contact in directions seen in the demonstrations. The four outputs
serve as input to the skill policy.

###  III-C Skill Policy

The policy for executing contact-rich skills is learnt using reinforcement
learning. For contact-rich task the action space $A$ is not only kinematic,
but can be forces as well. The action directions where the force control is
necessary is denoted by $A_{F}$. The directions perpendicular to this utilize
position control and the action space for that is indicated by $A_{x}$.

$A=A_{F}\cup A_{x}$ (2)

The contact-rich skill learning is formulated as a model-free reinforcement
learning problem. We use a maximum entropy RL algorithm as they tend to
produce stable policies for practical RL applications [13]. Specifically, we
use the soft actor-critic (SAC) algorithm to learn the policy. SAC uses a
replay buffer $\mathcal{R}$ to store history and trains a critic and actor
sampled from $\mathcal{R}$. The SAC algorithm is presented in Algorithm 1. The
policy network used in MLP with input from the multimodal representation and
produces actions – position for $A_{x}$ and force for $A_{F}$.

Initialize policy $\pi$, critic $Q$, replay buffer $\mathcal{R}$;

for  _$i=1\to$ num_iterations_ do

for  _each step_ do

Sample $a_{t}$ from $\pi_{\theta}(a_{t}|s_{t})$ and execute $a_{t}$;

Observe new state $s_{t+1}$;

Store $(s_{t},a_{t},r(s_{t},a_{t}),s_{t+1})$ in $\mathcal{R}$;

end for

for  _each gradient step_ do

Sample from $\mathcal{R}$;

Update policy $\pi$ and $Q$

end for

end for

Algorithm 1 Soft actor-critic algorithm

Reward description RL policies require designing reward functions which limit
their applicability to complicated manipulation tasks. This is especially true
for contact-rich manipulation tasks where designing the reward is difficult.
One alternative is to learn the reward from the given demonstrations of the
skill. The demonstrations contain information regarding goal of the skill
which can be leveraged to learn the reward function. We propose to extract
frames the demonstrations which represent positive and negative examples of
goals and use them to train a classifier to represent the reward. Let
$\mathcal{D}={(I_{n},y_{n})}$ denote the dataset of goal image frames $I_{n}$
and corresponding binary labels $y_{n}$ indicating positive or negative goals.
The positive goal imply actual goal state, negative goal mean a failed state.
Section III-A describes how the positive goal frames are obtained. To obtain
the negative examples, frames from the beginning of the demonstrations are
used. Frames are randomly sampled from the first 30% of the video before the
second PIK. We make a reasonable assumption that the skill is not completed in
beginning of the demonstration.

Provided: $\mathcal{D}={(I_{n},y_{n})}$;

Classifier $\nu(I)$ to minimize loss $\sum_{n}\mathcal{L}(\nu(I_{n}),y_{n})$

Learn policy using reward $\phi(I_{t})$ where,
$\phi(I_{t},a_{t})=\begin{cases}\nu(I_{t})-1&\text{if~{}}\nu(I_{t})<\kappa\\\
1&\text{if~{}}\nu(I_{t})>\kappa\end{cases}$ (3)

Algorithm 2 Learning rewards from demonstration

Classifier based rewards can enable learning tasks without reward engineering
[13, 31]. Let the binary classifier be denoted as $\nu(I)$. The goal
classifier is trained as a network with a Resnet50 backend to extract image
features and finally a 2 layer MLP with cross-entropy loss as the binary
classification loss $\mathcal{L}$. Training the classifier requires a large
number of examples of negative examples to avoid policy to fool the classifier
by reaching states different from those shown to classifier. As the user
provides multiple demonstration videos, large number of frames can be
extracted to generate negative examples. The reward function is constructed
based on the binary classification label as shown in Algorithm 2. $I_{t}$ is
the image at time step $t$, and $\kappa$ is a threshold to provide high sparse
reward when the performance is close to demonstration.

Figure 5: Execution Controller Model. Here $Q$ is the controller input and
gain parameters, $\tau_{m}$ is the joint torques after clamping to motor
limits.

###  III-D The Robot Execution Controller

Learning contact-rich policies require both position and force control modes.
The controller uses the output position offsets $dx$ and force offsets $df$
provided from the policy and employs a hybrid control scheme to set the joint
torques. The controller layout is shown in Fig. 5. The inputs to the
controller is position and force offsets in Cartesian frame. Let $x$ be the
current pose in 6 DoF and current force along $A_{F}$ direction as $F$.
$x_{t+1}=x_{t}+dx$ and $F_{t+1}=F_{t}+dF$. Let $X_{d}$ be the full 6 DoF
Cartesian pose required and $X$ be the current pose. The required joint
positions, $q_{d}$, can be computed using inverse kinematics. The impedance
controller is used for all directions expect the force direction. In the
direction of $F$, we use a force controller defined as follows,

$F_{d}=K(X_{d}-X)+C(\dot{X}_{d}-\dot{X})$ (4)

where $F_{d}$ is the impedance forces, $K$ is the stiffness and $C$ is the
damping. Let $\tau_{ns}$ be the desired nullspace torques and $q$ the current
joint angles. The required joint torques can be computed as
$\tau=J^{T}(q)F+\tau_{ns}$.

##  IV Experiments and Results

The experiments to evaluate and validate the performance of the skill modeling
framework have been conducted for a) writing, b) cleaning, and c) peeling
skills. All three are everyday contact-rich tasks requiring both vision and
force feedback, particularly useful in robotic applications.

Two baselines were implemented to compare the performance of the proposed
multi-modal skill learning framework. The first method is based on modeled
controllers, derived from the work presented in [8]. The second method is a
naive agent that uses an RL agent with an engineered reward. The parameters
for the modeled controller and the reward for the naive agent are designed
based on the task. These rewards are described in detail in respective
section.

###  IV-A Hardware Setup

Our experiments were conducted using the Rethink Sawyer robotic arm which is
widely used in robotics research, especially collaborative robotics. The
entire setup is shown in Fig. 1. The robot is equipped with a Barrett hand
gripper. The robot is controlled using impedance control mode. The Barrett
BH-282 hand has tactile sensors on the three fingers and the palm, with a 6
DOF Force-Torque sensor at the wrist. An Intel Realsense D435i [32] is used
for providing the color (RGB) and depth image frames. For all the experiments
a region of interest is provided to reduce the background and concentrate
focus on the workspace. The robot is also restricted to operate in a subset of
the workspace in front of it and within the bounds of the region of interest.
The same workspace bounds are used for all the tasks and this region can be
seen in the top image in Fig. 6.

Figure 6: The results of writing the character ‘S’. The top image shows the
results at different stages of training. The bottom figure shows a successful
trial.

###  IV-B Experiment 1: Writing

In this experiment the goal is to write the demonstrated character. A total of
20 demonstrations are provided to the robot with the user writing the
character ‘S’. The policy is trained using soft-actor critic with a 2 layer
multilayer perceptron (MLP) network of size 64 which was trained for 40000
time steps which is approximately 6 hours of real world time.

The modeled controller uses the user hand trajectory from the demonstrations
as the desired trajectory. For the naive RL agent, we use the total pixel
difference between the mean demonstrated image and observed image as the
penalty. If they overlap completely the reward will be 0 else the robot is
penalized. The same policy is used for the execution. For the framework, when
the occlusion parameter is 1, the reward $\phi(I_{t},a_{t})$ is set to 0. The
threshold is set as, $\kappa=0.8$.

The metric for task success is the amount of overlap between the two
trajectories. A mean trajectory is computed from the demonstrated contours
with a standard deviation, across the demonstrations. If the robot achieves a
trajectory within 2 standard deviations from the mean demonstrated trajectory
the execution is considered successful. Fig. 6 shows the performance at
different stages of training and the bottom image in the figure shows a
successful trial. Notably, the framework achieves a success rate of 100% after
just 6 hours of training. The modeled controller achieves a success rate of
78% and the baseline controller has a success rate of 30% after 6 hours as
shown in Table I.

Figure 7: The robot performing the cleaning skill. The sequence of images
shows a successful trial.

###  IV-C Experiment 2: Cleaning

The second experiment is learning a surface cleaning task using a brush. Small
bits of paper were scattered in the region of interest with the goal to clean
the entire area inside this region. A total of 40 demonstrations were
provided, with varying distribution of the particles as the demonstration
data. The same policy, as described previously, was used for training using a
soft-actor critic with a 2 layer MLP network of size 64. The policy was
trained for 50000-time steps which are approximately 8 hours in real-world
time.

The modeled controller approach uses the hand trajectory of the hand of the
user from the demonstration as a feedback. For designing the reward for the
naive RL agent, the area of the particles is used. The particles are detected
in the RGB image using color thresholding methods. The ratio of the area of
the particles to that of the region of interest in the image is used as the
penalty. If $S_{p}$ is area of particles in image and $S_{ROI}$ is the area of
the region of interest, then the reward ($r$) is calculated as:
$r=-S_{p}/S_{ROI}$. As before, for the proposed skill learning framework, the
reward $\phi(I_{t},a_{t})$ is set to 0 when the occlusion parameter is 1. The
threshold $\kappa$ is set as: $\kappa=0.8$.

The task success metric is based on the change in the area cleaned. With the
final area ($S^{f}_{p}$), the task is considered successful if
$S^{f}_{p}/S_{ROI}<0.05$, i.e. 95% of the surface is cleaned. Notably, the
proposed framework achieves a success rate of 80% with the modelled controller
achieving a success rate of 30% and the naive RL agent has a success rate of
43% as shown in Table I. Fig. 7 shows the robot performing a successful trial
using the policy from the proposed framework.

###  IV-D Experiment 3: Peeling

Peeling is a unique contact-rich task that requires the robot to interact with
the object directly instead of a surface. The state change is observed on the
object through vision in the case of peeling a cucumber. The goal of the
experiment is to peel an area of the cucumber. A total of 30 demonstrations of
a user peeling a side of a cucumber were provided to the robot. Using the same
policy as described previously, the trained was accomplished for a soft-actor
critic with a 2 layer MLP network of size 64. The policy was trained for 1000
time steps which are approximately equal to 30 minutes of real-world time.

The modeled controller approach extracts the the trajectory of the hand from
the demonstrations as training input. Force bounds are specified to not to
damage the object. For the naive RL agent, a image based reward is provided
similar to the previous task. The exploration space is limited to the region
of interest. Color thresholding is used to detect the area of the peeled
cucumber and the unpeeled area. The ratio of the areas is provided as the
reward. As before, for the proposed skill learning framework the reward
$\phi(I_{t},a_{t})$ is set to 0 when the occlusion parameter is 1. The
threshold $\kappa$ is set as: $\kappa=0.8$.

Figure 8: A successful trial of robot performing the peeling skill.

Instance segmentation is performed on the scene to detect the unpeeled and
peeled areas of cucumber. The average ratio of peeled to unpeeled areas from
the demonstration is used as the reference. If the ratio for the robot
execution is within 10% of the reference, the trial is considered successful.

The proposed framework achieves a success rate of 100%, the modeled controller
achieves a success rate of 65%, and the naive RL agent has a success rate 15%.
One of the successful trials using the skill learning framework is shown in
Fig. 8. Table I shows the task performance for the three methods.

TABLE I: Results on the success rate for the three contact-rich skills Skill | Proposed Method | Modeled controller | Naive RL
---|---|---|---
Writing | 100% | 78% | 30%
Cleaning | 80% | 30% | 43%
Peeling | 100% | 65% | 15%

##  V Conclusion and Discussion

In this paper, we propose a novel framework to learn policies for contact-rich
skills from few demonstrations without reward engineering. In the framework,
the demonstrations are semantically segmented to identify human-object
interactions and goal states. Subsequently, the reward can be learned from
goals identified in the demonstrations, which removes the need for manual
engineering of reward. Additionally, the interaction modes are used to define
the action space for the policy. A multi-modal representation for the state is
developed from the sensor data for the policy, which improves the learning
performance for contact-rich skills. In the experiments, three different
contact-rich skills requiring varying force and position interactions are
demonstrated to evaluate performance of the proposed framework. Results show
that the skill learning framework achieves a success rate of 100% on writing
and peeling skills and 80% for the cleaning skill. In the future, this work
will be extended to other manipulation skills with physical interaction such
as cutting and scooping.

## References

  * [1] R. Blake, K. V. Sobel, and T. W. James, “Neural synergy between kinetic vision and touch,” _Psychological science_ , vol. 15, no. 6, pp. 397–402, 2004\.
  * [2] H.-C. Song, Y.-L. Kim, and J.-B. Song, “Automated guidance of peg-in-hole assembly tasks for complex-shaped parts,” in _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2014, pp. 4517–4522.
  * [3] J. M. Romano, K. Hsiao, G. Niemeyer, S. Chitta, and K. J. Kuchenbecker, “Human-inspired robotic grasp control with tactile sensing,” _IEEE Transactions on Robotics_ , vol. 27, no. 6, pp. 1067–1079, 2011.
  * [4] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” _The Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 1334–1373, 2016.
  * [5] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” _Robotics and autonomous systems_ , vol. 57, no. 5, pp. 469–483, 2009.
  * [6] S. Calinon and A. G. Billard, “What is the teacher’s role in robot programming by demonstration?: Toward benchmarks for improved learning,” _Interaction Studies_ , vol. 8, no. 3, pp. 441–464, 2007.
  * [7] S. Calinon and A. Billard, “Incremental learning of gestures by imitation in a humanoid robot,” in _Proceedings of the ACM/IEEE international conference on Human-robot interaction_. ACM, 2007, pp. 255–262.
  * [8] M. V. Balakuntala, V. L. Venkatesh, J. P. Bindu, R. M. Voyles, and J. Wachs, “Extending policy from one-shot learning through coaching,” in _2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 2019, pp. 1–7.
  * [9] A. G. Billard, S. Calinon, and R. Dillmann, “Learning from humans,” in _Springer handbook of robotics_. Springer, 2016, pp. 1995–2014.
  * [10] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine, “One-shot visual imitation learning via meta-learning,” in _Conference on Robot Learning_ , 2017, pp. 357–368.
  * [11] C. Finn and S. Levine, “Deep visual foresight for planning robot motion,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 2786–2793.
  * [12] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in _Proceedings of the twenty-first international conference on Machine learning_. ACM, 2004, p. 1.
  * [13] A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine, “End-to-end robotic reinforcement learning without reward engineering,” in _Robotics: Science and Systems_ , 2019.
  * [14] M. A. Lee, Y. Zhu, K. Srinivasan, P. Shah, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, “Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 8943–8950.
  * [15] Y. Chebotar, M. Kalakrishnan, A. Yahya, A. Li, S. Schaal, and S. Levine, “Path integral guided policy search,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 3381–3388.
  * [16] Y. Zhu, Z. Wang, J. Merel, A. Rusu, T. Erez, S. Cabi, S. Tunyasuvunakool, J. Kramár, R. Hadsell, N. de Freitas, _et al._ , “Reinforcement and imitation learning for diverse visuomotor skills,” _arXiv preprint arXiv:1802.09564_ , 2018.
  * [17] M. Kalakrishnan, L. Righetti, P. Pastor, and S. Schaal, “Learning force control policies for compliant manipulation,” in _2011 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2011, pp. 4639–4644.
  * [18] A. X. Lee, H. Lu, A. Gupta, S. Levine, and P. Abbeel, “Learning force-based manipulation of deformable objects from multiple demonstrations,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 177–184.
  * [19] Y. Bekiroglu, R. Detry, and D. Kragic, “Learning tactile characterizations of object-and pose-specific grasps,” in _2011 IEEE/RSJ international conference on Intelligent Robots and Systems_. IEEE, 2011, pp. 1554–1560.
  * [20] R. Calandra, A. Owens, D. Jayaraman, J. Lin, W. Yuan, J. Malik, E. H. Adelson, and S. Levine, “More than a feeling: Learning to grasp and regrasp using vision and touch,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 3300–3307, 2018.
  * [21] D. Watkins-Valls, J. Varley, and P. Allen, “Multi-modal geometric learning for grasping and manipulation,” in _2019 International conference on robotics and automation (ICRA)_. IEEE, 2019, pp. 7339–7345.
  * [22] D. Kappler, P. Pastor, M. Kalakrishnan, M. Wüthrich, and S. Schaal, “Data-driven online decision making for autonomous manipulation.” in _Robotics: Science and Systems_ , 2015.
  * [23] F. J. Abu-Dakka, B. Nemec, J. A. Jørgensen, T. R. Savarimuthu, N. Krüger, and A. Ude, “Adaptation of manipulation skills in physical contact with the environment to reference force profiles,” _Autonomous Robots_ , vol. 39, no. 2, pp. 199–217, 2015.
  * [24] C. C. Beltran-Hernandez, D. Petit, I. G. Ramirez-Alpizar, T. Nishi, S. Kikuchi, T. Matsubara, and K. Harada, “Learning force control for contact-rich manipulation tasks with rigid position-controlled robots,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5709–5716, 2020\.
  * [25] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, and S. Schaal, “Dynamical movement primitives: learning attractor models for motor behaviors,” _Neural computation_ , vol. 25, no. 2, pp. 328–373, 2013.
  * [26] G. Gkioxari, R. Girshick, P. Dollár, and K. He, “Detecting and recognizing human-object interactions,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8359–8367.
  * [27] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [28] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [29] T. Lesort, N. Díaz-Rodríguez, J.-F. Goudou, and D. Filliat, “State representation learning for control: An overview,” _Neural Networks_ , vol. 108, pp. 379–392, 2018.
  * [30] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [31] A. Xie, A. Singh, S. Levine, and C. Finn, “Few-shot goal inference for visuomotor learning and planning,” in _Conference on Robot Learning_ , 2018, pp. 40–52.
  * [32] “intel® realsense™ depth and tracking cameras,” Aug 2020. [Online]. Available: https://software.intel.com/en-us/realsense/d400


11institutetext: Media Integration and Communication Center (MICC),
Dipartimento di Ingegneria dell’Informazione, Università degli Studi di
Firenze
11email<EMAIL_ADDRESS>
# Contrastive Supervised Distillation for
Continual Representation Learning

Tommaso Barletti∗ 0000-0001-7460-4710 Niccoló Biondi∗† 0000-0003-1153-1651
Federico Pernici  0000-0001-7036-6655 Matteo Bruni  0000-0003-2017-1061
and Alberto Del Bimbo  0000-0002-1052-8322

###### Abstract

In this paper, we propose a novel training procedure for the continual
representation learning problem in which a neural network model is
sequentially learned to alleviate catastrophic forgetting in visual search
tasks. Our method, called Contrastive Supervised Distillation (CSD), reduces
feature forgetting while learning discriminative features. This is achieved by
leveraging labels information in a distillation setting in which the student
model is contrastively learned from the teacher model. Extensive experiments
show that CSD performs favorably in mitigating catastrophic forgetting by
outperforming current state-of-the-art methods. Our results also provide
further evidence that feature forgetting evaluated in visual retrieval tasks
is not as catastrophic as in classification tasks. Code at:
https://github.com/NiccoBiondi/ContrastiveSupervisedDistillation.

###### Keywords:

Representation Learning Continual Learning Image Retrieval Visual Search
Contrastive Learning Distillation.

11footnotetext: Tommaso Barletti and Niccoló Biondi contributed
equally.44footnotetext: Corresponding Author.

##  1 Introduction

Deep Convolutional Neural Networks (DCNNs) have significantly advanced the
field of visual search or visual retrieval by learning powerful feature
representations from data [1, 2, 3]. Current methods predominantly focus on
learning feature representations from static datasets in which all the images
are available during training [4, 5, 6]. This operative condition is
restrictive in real-world applications since new data are constantly emerging
and repeatedly training DCNN models on both old and new images is time-
consuming. Static datasets, typically stored on private servers, are also
increasingly problematic because of the societal impact associated with
privacy and ethical issues of modern AI systems [7, 8].

These problems may be significantly reduced in incremental learning scenarios
as the computation is distributed over time and training data are not required
to be stored on servers. The challenge of learning feature representation in
incremental scenarios has to do with the inherent problem of catastrophic
forgetting, namely the loss of previously learned knowledge when new knowledge
is assimilated [9, 10]. Methods for alleviating catastrophic forgetting has
been largely developed in the classification setting, in which catastrophic
forgetting is typically observed by a clear reduction in classification
accuracy [11, 12, 13, 14, 15]. The fundamental differences with respect to
learning internal feature representation for visual search tasks are: (1)
evaluation metrics do not use classification accuracy (2) visual search data
have typically a finer granularity with respect to categorical data and (3) no
classes are required to be specifically learned. These differences might
suggest different origins of the two catastrophic forgetting phenomena. In
this regard, some recent works provide some evidence showing the importance of
the specific task when evaluating the catastrophic forgetting of the learned
representations [16, 17, 18, 19]. In particular, the empirical evidence
presented in [16] suggests that feature forgetting is not as catastrophic as
classification forgetting. We argue that such evidence is relevant in visual
search tasks and that it can be exploited with techniques that learn
incrementally without storing past samples in a memory buffer [20].

According to this, in this paper, we propose a new distillation method for the
continual representation learning task, in which the search performance
degradation caused by feature forgetting is jointly mitigated while learning
discriminative features. This is achieved by aligning current and previous
features of the same class, while simultaneously pushing away features of
different classes. We follow the basic working principle of contrastive loss
[21] used in self-supervised learning, to effectively leverage label
information in a distillation-based training procedure in which we replace
anchor features with the feature of the teacher model.

Our contributions can be summarized as follows:

  1. 1.

We address the problem of continual representation learning proposing a novel
method that leverages label information in a contrastive distillation learning
setup. We call our method Contrastive Supervised Distillation (CSD).

  2. 2.

Experimental results on different benchmark datasets show that our CSD
training procedure achieves state-of-the-art performance.

  3. 3.

Our results confirm that feature forgetting in visual retrieval using fine-
grained datasets is not as catastrophic as in classification.

##  2 Related Works

Continual Learning (CL). CL has been largely developed in the classification
setting, where methods have been broadly categorized based on exemplar [22,
23, 24, 25] and regularization [26, 27, 20, 28]. Only recently, continual
learning for feature representation is receiving increasing attention and few
works pertinent to the regularization-based category has been proposed [17,
18, 19]. The work in [17] proposed an unsupervised alignment loss between old
and new feature distributions according to the Mean Maximum Discrepancy (MMD)
distance [29]. The work [19] uses both the previous model and estimated
features to compute a semantic correlation between representations during
multiple model updates. The estimated features are used to reproduce the
behaviour of older models that are no more available. Finally, [18] addresses
the problem of lifelong person re-identification in which the previously
acquired knowledge is represented as similarity graphs and it is transferred
on the current data through graphs convolutions. While these methods use
labels only to learn new tasks, our method leverages labels information to
both learn incoming tasks and for distillation.

Reducing feature forgetting with feature distillation is also related to the
recent backward compatible representation learning in which newly learned
models can be deployed without the need to re-index the existing gallery
images [30, 31, 32]. This may have an impact on privacy as also the gallery
images are not required to be stored on servers. Finally, the absence of the
cost re-indexing is advantageous in streaming learning scenarios as [33, 34].

Contrastive Learning Contrastive learning has been proposed in [35] for metric
learning and then it is demonstrated to be effective in unsupervised/self-
supervised representation learning [36, 37, 21]. All these works focus on
obtaining discriminative representations that can be transferred to downstream
tasks by fine-tuning. In particular, this is achieved as, in the feature
space, each image and its augmented samples (the positive samples) are grouped
together while the others (the negative samples) are pushed away. However,
[38] observed that, given an input image, samples of the same class are
considered as negative and, consequently, pushed apart from it. We follow a
similar argument which considers as positive also these images.

##  3 Problem Statement

In the continual representation learning problem, a model
$\mathrm{M}(\,\cdot\,;\theta,\mathbf{W})$ is sequentially trained for $T$
tasks on a dataset
$\mathcal{D}=\\{(\mathbf{x}_{i},y_{i},t_{i})\,|\,i=1,2,\ldots,N\\}$, where
$\mathbf{x}_{i}$ is an image of a class $y_{i}\in\\{1,2,\ldots,L\\}$, $N$ is
the number of images, and $t_{i}\in\\{1,2,\ldots,T\\}\,$ is the task index
associated to each image. In particular, for each task $k$, $M$ is trained on
the subset
$\mathcal{T}_{k}=\mathcal{D}|_{t_{i}=k}=\\{(\mathbf{x}_{i},y_{i},t_{i})\,|\,t_{i}=k\\}$
which represents the $k$-th training-set that is composed by $L_{k}$ classes.
Each training-set has different classes and images with respect to the others
and only $\mathcal{T}_{k}$ is available to train the model $\mathrm{M}$
(memory-free).

At training time of task $k$, in response to a mini-batch
$\mathcal{B}=\\{(\mathbf{x}_{i},y_{i},t_{i})\\}_{i=1}^{|\mathcal{B}|}$ of
$\mathcal{T}_{k}$, the model $\mathrm{M}$ extracts the feature vectors and
output logits for each image in the batch, i.e.,
$\mathrm{M}(\mathbf{x}_{i})=C(\phi(\mathbf{x}_{i}))$, where
$\phi(\cdot,\theta)$ is the representation model which extracts the feature
vector $f_{i}=\phi(\mathbf{x}_{i})$ and $C$ is the classifier, which projects
the feature vector $f_{i}$ in an output vector $z_{i}=C(f_{i})$. At the end of
the training phase, $\mathrm{M}$ is used to index a gallery-set
$\mathcal{G}=\\{(\mathbf{x}_{g},y_{g})\,|\,g=1,2,\ldots,N_{g}\\}$ according to
the extracted feature vectors $\\{(f_{g},y_{g})\\}_{g=1}^{N_{g}}$.

At test time, a query-set
$\mathcal{Q}=\\{\mathbf{x}_{q}\,|\,q=1,2,\ldots,N_{q}\\}$ is processed by the
representation model $\phi(\cdot,\theta)$ in order to obtain the set of
feature vectors $\\{f_{q}\\}_{q=1}^{N_{q}}$. According to cosine distance
function $d$, the nearest sample in the gallery-set $\mathcal{G}$ is retrieved
for each query sample $f_{q}$, i.e.,

$f^{*}=\underset{g=1,2,\ldots,N_{g}}{\operatorname{arg}\,\operatorname{min}}\;d(f_{g},f_{q}),$
(1)

##  4 Method

Figure 1: Proposed method is based on the teacher-student framework. During
the training of the student, CE and triplet losses are minimized to learn the
new task data, are KD and CSD are used to preserve the old knowledge using the
teacher (not trainable).

To mitigate the effect of catastrophic forgetting while acquiring novel
knowledge from incoming data, we propose a training procedure that follows the
teacher-student framework, where the teacher is the model before the update
and the student is the model that is updated. The teacher is leveraged during
the training of the student to preserve the old knowledge as old data is not
available.

With reference to Fig. 1, at each task $k$, the student is trained on the
training-set $\mathcal{T}_{k}=\\{(\mathbf{x}_{i},y_{i},t_{i})\,|\,t_{i}=k\\}$
and the teacher is set as frozen, i.e., not undergoing learning. The loss
function that is minimized during the training of the student is the
following:

$\mathcal{L}=\mathcal{L}_{plasticity}+\mathcal{L}_{stability}$ (2)

where $\mathcal{L}_{stability}=0$ during the training of the model on the
first task. In the following, the components of the plasticity and stability
loss are analyzed in detail. In particular, we adopt the following notation.
Given a mini-batch $\mathcal{B}$ of training data, both the student and the
teacher networks produce a set of feature vectors and classifier outputs in
response to training images $\mathbf{x}_{i}\in\mathcal{B}$. We refer to as
$\\{f_{i}\\}$, $\\{z_{i}\\}$ for the feature vectors and classifier outputs of
the student, respectively, with $\\{f_{i}^{\prime}\\}$, $\\{z_{i}^{\prime}\\}$
for the teacher ones, and with $|\mathcal{B}|$ to the number of elements in
the mini-batch.

###  4.1 Plasticity Loss

Following [17], during the training of the updated model, the plasticity loss
is defined as follows:

$\mathcal{L}_{plasticity}=\mathcal{L}_{\rm CE}+\mathcal{L}_{\rm triplet}$

(3)

with

$\mathcal{L}_{\rm
CE}=\frac{1}{|\mathcal{B}|}\sum\limits_{i=1}^{|\mathcal{B}|}\;y_{i}\;\text{log}\bBigg@{3.3}(\frac{\text{exp}\big{(}z_{i}\big{)}}{\sum_{j=1}^{|\mathcal{B}|}\text{exp}\big{(}z_{j}\big{)}}\bBigg@{3.3})$

(4)

$\mathcal{L}_{\rm triplet}={\rm
max}\big{(}||f_{i}-f_{p}||_{2}^{2}-||f_{i}-f_{n}||_{2}^{2}\big{)}.$

(5)

$\mathcal{L}_{\rm CE}$ and $\mathcal{L}_{\rm triplet}$ are the cross-entropy
loss and the triplet loss, respectively. The plasticity loss of Eq. 3 is
optimized during the training of the model and it is used in order to learn
the novel tasks.

###  4.2 Stability Loss

The stability loss preserves the previously acquired knowledge in order to
limit the catastrophic forgetting effect, that is typically performed using
the teacher model for distillation. The stability loss we propose is
formulated as follows:

$\mathcal{L}_{stability}=\lambda_{\rm KD}\;\mathcal{L}_{\rm KD}+\lambda_{\rm
CSD}\;\mathcal{L}_{\rm CSD}$

(6)

where $\lambda_{\rm KD}$ and $\lambda_{\rm CSD}$ are two weights factors that
balance the two loss components, namely Knowledge Distillation (KD) and the
proposed Contrastive Supervised Distillation (CSD). In our experimental
results, we set both $\lambda_{\rm KD}$ and $\lambda_{\rm CSD}$ to 1. An
evaluation of different values is reported in the ablation studies of Sec. 6.

Knowledge Distillation. KD [39] minimizes the log-likelihood between the
classifier outputs of the student and the soft labels produced by the teacher,
instead of the ground-truth labels ($y_{i}$) used in the standard cross-
entropy loss. This encourages the outputs of the updated model to approximate
the outputs produced by the previous one. KD is defined as follows:

$\mathcal{L}_{\rm
KD}=\frac{1}{|\mathcal{B}|}\sum_{i=1}^{|\mathcal{B}|}\;\frac{\text{exp}\big{(}z_{i}^{\prime}\big{)}}{\sum_{j=1}^{|\mathcal{B}|}\text{exp}\big{(}z_{j}^{\prime}\big{)}}\;\text{
log}\bBigg@{3.3}(\frac{\text{exp}\big{(}z_{i}\big{)}}{\sum_{j=1}^{|\mathcal{B}|}\text{exp}\big{(}z_{j}\big{)}}\bBigg@{3.3})$

(7)

Contrastive Supervised Distillation.

Figure 2: Proposed CSD loss. (a) The features of four samples of two classes
are firstly mapped in the feature space by the teacher (blue) and the student
(orange). (b) With CSD samples belonging to the same class (same symbol) are
clustered together and separated from the others.

We propose a new distillation loss, i.e., the Contrastive Supervised
Distillation (CSD) that aligns current and previous feature models of the same
classes while simultaneously pushing away features of different classes. This
is achieved at training time imposing the following loss penalty:

$\mathcal{L}_{\rm
CSD}=-\frac{1}{|\mathcal{B}|}\sum\limits_{i=1}^{|\mathcal{B}|}\frac{1}{|\mathcal{P}(i)|}\sum\limits_{p\in\mathcal{P}(i)}\text{log}\bBigg@{3.3}(\frac{\text{exp}\big{(}f_{i}^{\prime}\cdot
f_{p}\big{)}}{\sum\limits_{\begin{subarray}{c}{a=1}\\\ {a\neq
i}\end{subarray}}^{|\mathcal{B}|}\text{exp}\big{(}f_{i}^{\prime}\cdot
f_{a}\big{)}}\bBigg@{3.3})$

(8)

where $\mathcal{P}(i)=\\{(x_{p},y_{p},t_{p})\in\mathcal{B}\,|\,y_{p}=y_{i}\\}$
is a set of samples in the batch which belong to the same class of
$\mathbf{x}_{i}$, i.e., the positive samples. Eq. 8 encourage for each class,
the alignment of the student representations to the ones of the same class of
the teacher model, which acts as anchors. In Fig. 2, we show the effect of CSD
loss on four samples $\\{(\mathbf{x}_{i},y_{i})\\}_{i=1}^{4}$ with
$y_{i}\in\\{1,2\\}$. Initially (Fig. 2) the feature vectors extracted by the
student $f_{i}$ (orange samples) are separated from the teacher ones
$f_{i}^{\prime}$ (blue samples). CSD clusters together features of the same
class moving the student representations, which are trainable, towards the
fixed ones of the teacher while pushing apart features belonging to different
classes. For the sake of simplicity, this effect is shown just for
$f_{1}^{\prime}$ and $f_{3}^{\prime}$. Indeed, $f_{1}$ and $f_{2}$ become
closer to $f_{1}^{\prime}$, while $f_{3}$ and $f_{4}$ are spaced apart with
respect to $f_{1}^{\prime}$ as they are of class $2$. The same effect is
visible also for $f_{3}^{\prime}$ which attracts $f_{3}$ and $f_{4}$ and push
away $f_{1}$ and $f_{2}$ as shown in Fig. 2.

CSD imposes a penalty on feature samples considering not only the overall
distribution of features of the teacher model with respect to the student one,
but it also clusters together samples of the same class separating from the
clusters of the other classes. Our method differs from KD as the loss function
is computed directly on the features and not on the classifier outputs
resulting in more discriminative representations. CSD also considers all the
samples of each class as positive samples that are aligned with the same
anchor of the teacher and not pairs (teacher-student) of samples as in [40].

##  5 Experimental Results

We perform our experimental evaluation on CIFAR-100 [41] and two fine-grained
datasets, namely CUB-200 [42] and Stanford Dogs [43]. The CIFAR-100 dataset
consists of $60000$ $32\times 32$ images in $100$ classes. The CUB-200 dataset
contains $11788$ $224\times 224$ images of $200$ bird species. Stanford Dogs
includes over $22000$ $224\times 224$ annotated images of dogs belonging to
$120$ species.

The continual representation learning task is evaluated following two
strategies. In CIFAR-100, we evenly split the dataset into $T$ training-set
where the model is trained sequentially, using the open-source Avalanche
library [44]. The experiments are evaluated with $T=2,5,10$. In CUB-200 and
Stanford Dogs, following [45][46], we use half of the data to pre-train a
model and split the remaining data into $T$ training-set. CUB-200 is evaluated
with $T=1,4,10$ while Stanford Dogs with $T=1$.

Implementation Details. We adopt ResNet32
[47]111https://github.com/arthurdouillard/incremental_learning.pytorch as
representation model architecture on CIFAR-100 with 64-dimension feature
space. We trained the model for $800$ epochs for each task using Adam
optimizer with a learning rate of $1\cdot 10^{-3}$ for the initial task and
$1\cdot 10^{-5}$ for the others. Random crop and horizontal flip are used as
image augmentation. Following [19], we adopt pretrained Google Inception [48]
as representation model architecture on CUB-200 and Stanford Dogs with
512-dimension feature space. We trained the model for $2300$ epochs for each
task using with Adam optimizer with a learning rate of $1\cdot 10^{-5}$ for
the convolutional layers and $1\cdot 10^{-6}$ for the classifier. Random crop
and horizontal flip are used as image augmentation. We adopt Recall@K[49][45]
as performance metric using each image in the test-set as query and the others
as gallery.

Table 1: Evaluation on CIFAR-100 of CSD and compared methods.

Method |  Recall@1 (1-50) |  Recall@1 (51-100) |  Recall@1 Average
---|---|---|---
Initial model | 67.6 | 21.7 | 44.7
Fine-Tuning | 37.4 | 64.1 | 50.8
LwF [20] | 64.0 | 59.4 | 61.7
MMD loss [17] | 61.8 | 60.9 | 61.4
CSD (Ours) | 65.1 | 61.6 | 63.4
Joint Training | 70.5 | 71.9 | 71.2

###  5.1 Evaluation on CIFAR-100

We compare our method on CIFAR-100 dataset with the Fine-Tuning baseline, LwF
[20], and [17] denoted as MMD loss. As an upper bound reference, we report the
Joint Training performance obtained using all the CIFAR-100 data to train the
model.

We report in Tab. 1 the scores obtained with $T=2$. In the first row, we show
the Initial Model results, i.e., the model trained on the first half of data
from CIFAR-100. Our approach achieves the highest recall when evaluated on the
initial task and the highest recall on the second task between methods trying
to preserve old knowledge, being second only to Fine-Tuning that focuses only
on learning new data. This results in our method achieving the highest average
recall value with an improvement of $\mathtt{\sim}2\%$ Recall@1 with respect
to LwF and MMD loss and $10.4\%$ with respect to the Fine-Tuning baseline. The
gap between all the continual representation learning methods and Joint
Training is significant ($\mathtt{\sim}8\%$). This underlines the challenges
of CIFAR-100 in a continual learning scenario since there is a noticeable
difference in the appearance between images of different classes causing a
higher feature forgetting.

(a) CIFAR-100 with $T=5$

(b) CIFAR-100 with $T=10$

Figure 3: Evolution of Recall@1 on the first task as new tasks are learned on
CIFAR-100. Comparison between our method (CSD) and compared methods.

Fig. 3(a) and Fig. 3(b) report the evolution of Recall@1 on the initial task
as new tasks are learned with $T=5$ and $T=10$, respectively. In both
experiments, our approach does not always report the highest scores, but it
achieves the most stable trend obtaining the best result as the training end.
This confirms that our approach is effective also when the model is updated
multiple times.

###  5.2 Evaluation on Fine-grained Datasets

We compare our method on CUB-200 and Stanford Dogs datasets with the Fine-
Tuning baseline, MMD loss [17], and [19] denoted as Feature Estimation. As an
upper bound reference, we report the Joint Training performance obtained using
all the data to train the model.

Table 2: Evaluation on Stanford Dogs and CUB-200 of CSD and compared methods.

|  Stanford Dogs |  CUB-200
---|---|---
Method |  Recall@1 (1-60) |  Recall@1 (61-120) |  Recall@1 Average |  Recall@1 (1-100) |  Recall@1 (101-200) |  Recall@1 Average
Initial model | 81.3 | 69.3 | 75.3 | 79.2 | 46.9 | 63.1
Fine-Tuning | 74.0 | 83.7 | 78.8 | 70.2 | 75.1 | 72.7
MMD loss [17] | 79.5 | 83.4 | 81.4 | 77.0 | 74.1 | 75.6
Feat. Est. [19] | 79.9 | 83.5 | 81.7 | 77.7 | 75.0 | 76.4
CSD (Ours) | 80.9 | 83.5 | 82.2 | 78.6 | 78.3 | 78.5
Joint Training | 80.4 | 83.1 | 81.7 | 78.2 | 79.2 | 78.7

(a) CUB-200 with $T=4$

(b) CUB-200 with $T=10$

Figure 4: Evolution of Recall@1 on the first task as new tasks are learned on
CUB-200. Comparison between our method (CSD) and compared methods.

Figure 5: Ablation on loss component on CUB-200 with $T=10$. “+” represents
the combination of components.

$\lambda_{\rm KD}$ | $\lambda_{\rm CSD}$ |  Recall@1 (1-100) |  Recall@1 (101-200) |  Recall@1 Average
---|---|---|---|---
0.1 | 0.1 | 78.24 | 76.82 | 77.53
0.1 | 1 | 79.19 | 77.50 | 78.35
0.1 | 10 | 78.56 | 76.07 | 77.32
1 | 0.1 | 79.32 | 73.82 | 76.57
1 | 1 | 78.62 | 78.34 | 78.48
1 | 10 | 79.12 | 75.32 | 77.22
10 | 0.1 | 78.35 | 76.76 | 77.56
10 | 1 | 79.53 | 76.93 | 78.23
10 | 10 | 78.90 | 75.53 | 77.22

Table 3: Ablation on the weight factors for KD and CSD in Eq. 6 on CUB-200
with $T=1$.

We report in Tab. 2 the scores obtained with $T=1$ on the fine-grained
datasets. On Stanford Dogs, our approach achieves the highest recall when
evaluated on the initial task and comparable result with other methods on the
final task with a gap of only $0.2\%$ with respect to Fine-Tuning that focus
only on learning new data. This results in our method achieving the highest
average recall value with an improvement of $0.5\%$ Recall@1 concerning
Feature Estimation, $0.8\%$ for MMD loss, and $3.4\%$ for Fine-Tuning. On the
more challenging CUB-200 dataset, we obtain the best Recall@1 on both the
initial and the final task outperforming the compared methods. Our method
achieves the highest average recall value with an improvement of $2.1\%$
Recall@1 with respect to Feature Estimation, $2.9\%$ for MMD loss, and $5.8\%$
for Fine-Tuning. Differently from CIFAR-100, on fine-grained datasets, there
is a lower dataset shift between different tasks leading to a higher
performance closer to the Joint Training upper bound due to lower feature
forgetting.

We report in Fig. 4(a) and Fig. 4(b) the challenging cases of CUB-200 with
$T=4$ and $T=10$, respectively. These experiments show, consistently with Tab.
2, how our approach outperforms state-of-the-art methods. In particular, with
$T=10$ (Fig. 4(b)), our method preserves the performance obtained on the
initial task during every update. CSD largely improves over the state-of-the-
art methods by almost $20\%$ \- $25\%$ with respect to [19] and [17] achieving
similar performance to the Joint Training upper bound. By leveraging labels
information for distillation during model updates, CSD provides better
performance and favorably mitigates the catastrophic forgetting of the
representation compared to other methods that do not make use of this
information.

##  6 Ablation Study

Loss Components. In Fig. 5, we explore the benefits given by the components of
the loss in Eq. 2 (i.e., CE, triplet, KD, and CSD) and their combinations in
terms of Recall@1 on CUB-200 with $T=10$. To observe single component
performance, we analyze the trend of Recall@1 on both the current task and
previous ones evaluated jointly. When CSD is used, (i.e., CE+CSD, CE+KD+CSD,
CE+triplet+CSD, CE+triplet+KD+CSD), we achieve higher Recall@1 and maintain a
more stable trend with respect to others. This underlines how CSD is effective
and central to preserve knowledge and limit feature forgetting across model
updates.

Loss Components Weights. Finally, in Tab. 3, we analyze the influence of the
stability loss components varying the parameters $\lambda_{\rm KD}$ and
$\lambda_{\rm CSD}$ of Eq. 6 on CUB-200 with $T=1$. The table shows the
Recall@1 obtained on the first task, on the final task, and the average
between them after training the model. CSD best performs when $\lambda_{\rm
KD}=\lambda_{\rm CSD}=1$, obtaining the highest average Recall@1.

##  7 Conclusions

In this paper, we propose Contrastive Supervised Distillation (CSD) to reduce
feature forgetting in continual representation learning. Our approach tackles
the problem without storing data of previously learned tasks while learning a
new incoming task. CSD allows to minimize the discrepancy of new and old
features belonging to the same class, while simultaneously pushing apart
features from different classes of both current and old data in a contrastive
manner. We evaluate our approach and compare it to state-of-the-art works
performing empirical experiments on three benchmark datasets, namely
CIFAR-100, CUB-200, and Stanford Dogs. Results show the advantages provided by
our method in particular on fine-grained datasets where CSD outperforms
current state-of-the-art methods. Experiments also provide further evidence
that feature forgetting evaluated in visual retrieval tasks is not as
catastrophic as in classification tasks.

Acknowledgments. This work was partially supported by the European Commission
under European Horizon 2020 Programme, grant number 951911 - AI4Media.

The authors acknowledge the CINECA award under the ISCRA initiative (ISCRA-C -
“ILCoRe”, ID: HP10CRMI87), for the availability of HPC resources.

## References

  * [1] Ji Wan, Dayong Wang, Steven Chu Hong Hoi, Pengcheng Wu, Jianke Zhu, Yongdong Zhang, and Jintao Li.  Deep learning for content-based image retrieval: A comprehensive study.  In Proceedings of the 22nd ACM international conference on Multimedia, pages 157–166.
  * [2] H Azizpour, J Sullivan, S Carlsson, et al.  Cnn features off-the-shelf: An astounding baseline for recognition.  In CVPRW, pages 512–519. 2014.
  * [3] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  Advances in Neural Information Processing Systems, 2014.
  * [4] Wei Chen, Yu Liu, Weiping Wang, Erwin Bakker, Theodoros Georgiou, Paul Fieguth, Li Liu, and Michael S Lew.  Deep image retrieval: A survey.  arXiv preprint arXiv:2101.11282, 2021.
  * [5] Giorgos Tolias, Ronan Sicre, and Hervé Jégou.  Particular object retrieval with integral max-pooling of cnn activations.  In ICLR 2016-International Conference on Learning Representations, pages 1–12, 2016.
  * [6] Joe Yue-Hei Ng, Fan Yang, and Larry S Davis.  Exploiting local features from deep networks for image retrieval.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 53–61, 2015.
  * [7] W Nicholson Price and I Glenn Cohen.  Privacy in the age of medical big data.  Nature medicine, 25(1):37–43, 2019.
  * [8] Andrea Cossu, Marta Ziosi, and Vincenzo Lomonaco.  Sustainable artificial intelligence through continual learning.  arXiv preprint arXiv:2111.09437, 2021.
  * [9] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [10] Roger Ratcliff.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.  Psychological review, 97(2):285, 1990.
  * [11] Mochitha Vijayan and SS Sridhar.  Continual learning for classification problems: A survey.  In International Conference on Computational Intelligence in Data Science, pages 156–166. Springer, 2021.
  * [12] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [13] Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer.  Class-incremental learning: survey and performance evaluation on image classification.  arXiv preprint arXiv:2010.15277, 2020.
  * [14] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  Neural Networks, 113:54–71, 2019.
  * [15] Eden Belouadah, Adrian Popescu, and Ioannis Kanellos.  A comprehensive study of class incremental learning algorithms for visual tasks.  Neural Networks, 135:38–54, 2021.
  * [16] MohammadReza Davari and Eugene Belilovsky.  Probing representation forgetting in continual learning.  In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications, 2021.
  * [17] Wei Chen, Yu Liu, Weiping Wang, Tinne Tuytelaars, Erwin M. Bakker, and Michael S. Lew.  On the exploration of incremental learning for fine-grained image retrieval.  In BMVC. BMVA Press, 2020.
  * [18] Nan Pu, Wei Chen, Yu Liu, Erwin M Bakker, and Michael S Lew.  Lifelong person re-identification via adaptive knowledge accumulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7901–7910, 2021.
  * [19] Wei Chen, Yu Liu, Nan Pu, Weiping Wang, Li Liu, and Michael S Lew.  Feature estimations based correlation distillation for incremental image retrieval.  IEEE Transactions on Multimedia, 2021.
  * [20] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  * [21] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [22] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert.  icarl: Incremental classifier and representation learning.  In CVPR, pages 5533–5542. IEEE Computer Society, 2017.
  * [23] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  In CVPR, pages 831–839. Computer Vision Foundation / IEEE, 2019\.
  * [24] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.  Large scale incremental learning.  In CVPR, pages 374–382. Computer Vision Foundation / IEEE, 2019\.
  * [25] Federico Pernici, Matteo Bruni, Claudio Baecchi, Francesco Turchini, and Alberto Del Bimbo.  Class-incremental learning with pre-allocated fixed classifiers.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 6259–6266. IEEE, 2021.
  * [26] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [27] Gido M Van de Ven and Andreas S Tolias.  Three scenarios for continual learning.  arXiv preprint arXiv:1904.07734, 2019.
  * [28] Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim.  Less-forgetting learning in deep neural networks.  arXiv preprint arXiv:1607.00122, 2016.
  * [29] A. Gretton, AJ. Smola, J. Huang, M. Schmittfull, KM. Borgwardt, and B. Schölkopf.  Covariate shift and local learning by distribution matching.  MIT Press, 2009.
  * [30] Yantao Shen, Yuanjun Xiong, Wei Xia, and Stefano Soatto.  Towards backward-compatible representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6368–6377, 2020.
  * [31] Federico Pernici, Matteo Bruni, Claudio Baecchi, and Alberto Del Bimbo.  Regular Polytope Nworks.  IEEE Transactions on Neural Networks and Learning Systems, 2021\.
  * [32] Niccolo Biondi, Federico Pernici, Matteo Bruni, and Alberto Del Bimbo.  CoReS: Compatible Representations via Stationarity.  arXiv preprint arXiv:2111.07632, 2021.
  * [33] Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars.  Task-free continual learning.  In Proceedings of the IEEE/CVF CVPR, pages 11254–11263, 2019.
  * [34] Federico Pernici, Matteo Bruni, and Alberto Del Bimbo.  Self-supervised on-line cumulative learning from video streams.  Computer Vision and Image Understanding, 197:102983, 2020.
  * [35] Sumit Chopra, Raia Hadsell, and Yann LeCun.  Learning a similarity metric discriminatively, with application to face verification.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 539–546. IEEE, 2005.
  * [36] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [37] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707–6717, 2020.
  * [38] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan.  Supervised contrastive learning.  In NeurIPS, 2020.
  * [39] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [40] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  arXiv preprint arXiv:1412.6550, 2014.
  * [41] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [42] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  Computation & Neural Systems Technical Report, 2011.
  * [43] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei.  Novel dataset for fine-grained image categorization.  In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
  * [44] Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Marc Masana, Jary Pomponi, Gido M. van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas S. Tolias, Simone Scardapane, Luca Antiga, Subutai Ahmad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni.  Avalanche: An End-to-End Library for Continual Learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3600–3610, June 2021.
  * [45] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese.  Deep metric learning via lifted structured feature embedding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4004–4012, 2016.
  * [46] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott.  Multi-similarity loss with general pair weighting for deep metric learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [47] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [48] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference CVPR, 2015.
  * [49] Herve Jegou, Matthijs Douze, and Cordelia Schmid.  Product quantization for nearest neighbor search.  IEEE transactions on pattern analysis and machine intelligence, 33(1):117–128, 2010.