# INTERSPEECH 2021 Deep Noise Suppression Challenge

###### Abstract

The Deep Noise Suppression (DNS) challenge was designed to unify the research
efforts in the area of noise suppression targeted for human perception. We
recently organized a DNS challenge special session at INTERSPEECH 2020 and
ICASSP 2021. We open-sourced training and test datasets for the wideband
scenario along with a subjective evaluation framework based on ITU-T standard
P.808, which was used to evaluate participants of the challenge. Many
researchers from academia and industry made significant contributions to push
the field forward, yet even the best noise suppressor was far from achieving
superior speech quality in challenging scenarios. In this version of the
challenge organized at INTERSPEECH 2021, we expanded our training and test
datasets to accommodate fullband scenarios and challenging test conditions. We
used ITU-T P.835 to evaluate the challenge winners as it gives additional
information about the quality of processed speech and residual noise. The two
tracks in this challenge focused on real-time denoising for (i) wideband, and
(ii) fullband scenarios. We also made available a reliable non-intrusive
objective speech quality metric for wideband called DNSMOS for the
participants to use during their development phase.

Index Terms: DNS Challenge, Deep Noise Suppressor, Speech, Noise, Audio,
Speech Quality

##  1 Introduction

With the explosion in the number of people working remotely due to the
pandemic, there has been a surge in the demand for reliable collaboration and
real-time communication tools. Excellent speech quality in our audio calls is
a need during these times as we try to stay connected and collaborate with
people every day. We are easily exposed to a variety of background noises such
as a leaf blower, washing machine, dog barking, a baby crying, kitchen noises,
etc. Background noise significantly degrades the quality and intelligibility
of the perceived speech leading to fatigue. Background noise poses a challenge
in other applications such as hearing aids and smart devices as well.

Real-time Speech Enhancement (SE) for perceptual quality is a decades-old
classical problem and researchers have proposed numerous solutions [1, 2]. In
recent years, learning-based approaches have shown promising results [3, 4,
5]. The Deep Noise Suppression (DNS) Challenge organized at INTERSPEECH 2020
[6] and ICASSP 2021 [7] showed great progress, while also indicating that we
are still about 1.6 Differential Mean Opinion Score (DMOS) away from the ideal
Mean Opinion Score (MOS) of 5 when tested on the challenge test set, which was
reasonably representative of realistic scenarios. The DNS Challenge is the
first contest that we are aware of using the subjective evaluation to
benchmark SE methods using a realistic noisy test set [6].

We open-sourced a large dataset for INTERSPEECH 2020 and ICASSP 2021 DNS
challenge111https://github.com/microsoft/DNS-Challenge. For ease of reference,
we will call the INTERSPEECH 2021 challenge DNS Challenge 3, ICASSP 2021
challenge DNS Challenge 2, and the INTERSPEECH 2020 challenge DNS Challenge 1.
The DNS Challenge 3 was focused on real-time denoising similar to track 1 of
both the DNS challenges 1 and 2. We had 2 tracks in DNS challenge 3 for
wideband (sampling rate = 16000 Hz) and fullband (sampling rate = 48000 Hz)
scenarios. The datasets include over 760 hours of clean speech including
singing voice, emotion data, and non-English languages. Noise data in the
training set remains the same as DNS Challenge 2. Both clean speech and noise
are made publicly available for both wide and fullband scenarios. We provide
over 118,000 room impulse responses (RIR), which includes real and synthetic
RIRs from public datasets for wideband. We provide acoustic parameters:
Reverberation time (T60) and Clarity (C50) for read clean speech and RIR
sample. The test set includes a variety of noisy speech utterances in English
and non-English in a range of reverberant and noisy scenarios. We also include
emotional speech and singing in the presence of background noise.

Unlike ITU-T P.808 that was used for DNS Challenge 1 and 2, we used the
implementation of ITU-T P.835222https://github.com/microsoft/P.808 [8] for the
DNS Challenge 3. In addition to the overall speech quality as in P.808, P.835
provides standalone quality scores of speech and noise. The standalone ratings
will help us focus on the areas that require improvement to achieve better
overall speech quality. Many noise suppressors are very good in suppressing
the background noise but do not improve the quality of speech, which becomes
the bottleneck for improving the overall quality. The results of this
challenge discussed in the later sections reflect the same. We also provided a
non-intrusive objective speech quality metric for wideband scenario called
DNSMOS333https://github.com/microsoft/DNS-Challenge/tree/master/DNSMOS as an
Azure service. We showed that DNSMOS is more reliable than other widely used
objective metrics such as PESQ, SDR, and POLQA [9]. Also, it does not require
reference clean speech and hence can work on real recordings. This paper
describes the datasets, challenge results, and the learnings from the
challenge in more detail.

##  2 Challenge Tracks

The following were the algorithmic and computational requirements that each
participant had to satisfy to be eligible for the challenge.

  1. 1.

Track 1: Real-Time Denoising track for wideband scenario

     * •

The noise suppressor must take less than the stride time $T_{s}$ (in ms) to
process a frame of size $T$ (in ms) on an Intel Core i5 quad-core machine
clocked at 2.4 GHz or equivalent processor. For example, $T_{s}=T/2$ for 50%
overlap between frames. The total algorithmic latency allowed including the
frame size $T$, stride time $T_{s}$, and any look ahead must be $\leq$ 40ms.
For example, for a real-time system that receives 20ms audio chunks, if you
use a frame length of 20ms with a stride of 10ms resulting in an algorithmic
latency of 30ms, then you satisfy the latency requirements. If you use a frame
of size 32ms with a stride of 16ms resulting in an algorithmic latency of
48ms, then your method does not satisfy the latency requirements as the total
algorithmic latency exceeds 40ms. If your frame size plus stride
$T_{1}=T+T_{s}$ is less than 40ms, then you can use up to $(40-T_{1})$ms
future information.

  2. 2.

Track 2: Real-Time Denoising track for fullband scenario

     * •

Satisfy Track 1 requirements.

##  3 Training Datasets

The goal of releasing the clean speech and noise datasets is to provide
researchers with an extensive and representative dataset to train their SE
models. We initially released MSSNSD [10] with a focus on extensibility, but
the dataset lacked the diversity in speakers, emotions, languages, and noise
types. We published a significantly larger and more diverse data set with
configurable scripts for DNS Challenge 1 and 2 [6]. Many researchers found
this dataset useful to train their noise suppression models and achieved good
results. However, the training and the test datasets again lacked more clips
with emotions such as crying, yelling, laughter or singing. Also, the dataset
only included the clips in the English language. For DNS Challenge 3, we added
speech clips with other emotions and included about 10 non-English languages.
The clean speech in the training set resulted in a total of 760.53 hours: read
speech (562.72 hours), singing voice (8.80 hours), emotion data (3.6hours),
Chinese mandarin data (185.41 hours). We have grown clean speech to 760.53
hours as compared to 562.72 hours in DNS Challenge 1. The details about the
clean and noisy dataset are described in the following sections.

###  3.1 Clean Speech

Clean speech consists of three subsets: (i) Read speech recorded in clean
conditions; (ii) Singing clean speech; (iii) Emotional clean speech; and (iv)
Non-English clean speech. The first subset is derived from the public
audiobooks dataset called Librivox444https://librivox.org/. It is available
under the permissive Creative Commons 4.0 license [11]. It has recordings of
volunteers reading over 10,000 public domain audiobooks in various languages,
the majority of which are in English. In total, there are 11,350 speakers.
Many of these recordings are of excellent speech quality, meaning that the
speech was recorded using good quality microphones in silent and less
reverberant environments. But there are many recordings that are of poor
speech quality as well with speech distortion, background noise, and
reverberation. Hence, it is important to clean the data set based on speech
quality. We used the online subjective test framework ITU-T P.808 [12] to sort
the book chapters by subjective quality. The audio chapters in Librivox are of
variable length ranging from few seconds to several minutes. We randomly
sampled 10 audio segments from each book chapter, each of 10 seconds in
duration. For each clip, we had 2 ratings, and the MOS across all clips was
used as the book chapter MOS. Figure 1 shows the results, which show the
quality spanned from very poor to excellent quality. In total, it is 562 hours
of clean speech, which was part of DNS Challenge 1.

The second subset consists of high-quality audio recordings of singing voices
recorded in noise-free conditions by professional singers. This subset is
derived from VocalSet corpus [13] with Creative Commons Attribution 4.0
International License (CC BY 4.0). license. It has 10.1 hours of clean singing
voice recorded by 20 professional singers: 9 males, and 11 females. This data
was recorded on a range of vowels, a diverse set of voices on several
standards and extended vocal techniques, and sung in contexts of scales,
arpeggios, long tones, and excerpts. For wideband, we downsampled the mono
.WAV files from 44.1 kHz to 16 kHz and added them to the clean speech corpus
used by the training data synthesizer.

The third subset consists of emotional speech recorded in noise-free
conditions. This is derived from Crowd-sourced Emotional Multimodal Actors
Dataset (CREMA-D) [14] made available under the Open Database License. It
consists of 7,442 audio clips from 91 actors: 48 male, and 43 female
accounting for a total of 3.5 hours of audio. The age of the actors was in the
range of 20 to 74 years with diverse ethnic backgrounds including African
American, Asian, Caucasian, Hispanic, and Unspecified. Actors read from a pool
of 12 sentences for generating this emotional speech dataset. It accounts for
six emotions: Anger, Disgust, Fear, Happy, Neutral, and Sad at four intensity
levels: Low, Medium, High, Unspecified. The recorded audio clips were
annotated by multiple human raters in three modalities: audio, visual, and
audio-visual. Categorical emotion labels and real-value emotion level values
of perceived emotion were collected using crowd-sourcing from 2,443 raters.
This data was provided as 16 kHz .WAV files so we added it to our wideband
clean speech as it is.

The fourth subset has a clean speech from non-English languages. It consists
of both tonal and non-tonal languages including Chinese (Mandarin), German and
Spanish. Mandarin data consists of OpenSLR18 555http://www.openslr.org/18/
THCHS-30 [15] and OpenSLR33 666http://www.openslr.org/33/ AISHELL [16]
datasets, both with the Apache 2.0 license. THCHS30 was published by the
Center for Speech and Language Technology (CSLT) at Tsinghua University for
speech recognition. It consists of 30+ hours of clean speech recorded at
16-bit 16 kHz in noise-free conditions. Native speakers of standard Mandarin
read text prompts chosen from a list of 1000 sentences. We added the entire
THCHS-30 data in our clean speech for the training set. It consisted of 40
speakers: 9 male, 31 female in the age range of 19-55 years. It has total
13,389 clean speech audio files [15]. The AISHELL dataset was created by
Beijing Shell Shell Technology Co. Ltd. It has clean speech recorded by 400
native speakers ( 47% male and 53% female) of Mandarin with different accents.
The audio was recorded in noise-free conditions using high-fidelity
microphones. It is provided as 16-bit 16kHz .wav files. It is one of the
largest open-source Mandarin speech datasets. We added the entire AISHELL
corpus with 141,600 utterances spanning 170+ hours of clean Mandarin speech to
our training set.

Spanish data is 46 hours of clean speech derived from OpenSLR39, OpenSLR61,
OpenSLR71, OpenSLR73, OpenSLR74 and OpenSLR75 where re-sampled all .WAV files
from 48 kHz to 16 kHz to use them as wideband signals. German data is derived
from four corpora namely (i) The Spoken Wikipedia Corpora [17], (ii)
Telecooperation German Corpus for Kinect [18], (iii) M-AILABS data [19], (iv)
zamia-speech forschergeist corpora. Complete German data constitute 636 hours.
Italian (128 hours), French (190 hours), Russian (47 hours) are taken from
M-AILABS data [19]. M-AILABS Speech Dataset is a publicly available multi-
lingual corpora for training speech recognition and speech synthesis systems.

Figure 1: Sorted near-end single-talk clip quality (P.808) with 95% confidence
intervals.

###  3.2 Noise

The noise clips were selected from Audioset
777https://research.google.com/audioset/ [20] and Freesound
888https://freesound.org/. Audioset is a collection of about 2 million human
labeled 10s sound clips drawn from YouTube videos and belong to about 600
audio events. Like the Librivox data, certain audio event classes are over-
represented. For example, there are over a million clips with audio classes
music and speech and less than 200 clips for classes such as toothbrush,
creak, etc. Approximately 42% of the clips have a single class, but the rest
may have 2 to 15 labels. Hence, we developed a sampling approach to balance
the dataset in such a way that each class has at least 500 clips. We also used
a speech activity detector to remove the clips with any kind of speech
activity, to strictly separate speech and noise data. The resulting dataset
has about 150 audio classes and 60,000 clips. We also augmented an additional
10,000 noise clips downloaded from Freesound and DEMAND databases [21]. The
chosen noise types are more relevant to VoIP applications. In total, there is
181 hours of noise data. The noise files were originally of fullband, which
were resampled to 16 kHz for wideband use case.

###  3.3 Room Impulse Responses

We provide 3076 real and approximately 115,000 synthetic rooms impulse
responses (RIRs) where we can choose either one or both types of RIRs for
convolving with clean speech. Noise is then added to reverberant clean speech
while DNS models are expected to take noisy reverberant speech and produce
clean reverberant speech. Challenge participants can do both de-reverb and
denoising with their models if they prefer. These RIRs are chosen from
openSLR26 [22] 999http://www.openslr.org/26/ and openSLR28 [22]
101010http://www.openslr.org/28/ datasets, both released with Apache 2.0
License.

###  3.4 Acoustic parameters

We provide two acoustic parameters: (i) Reverberation time, T60 [23] and (ii)
Clarity, C50 [24] for all audio clips in clean speech of training set. We
provide T60, C50 and isReal Boolean flag for all RIRs where isReal is 1 for
real RIRs and 0 for synthetic ones. The two parameters are correlated. An RIR
with low C50 can be described as highly reverberant and vice versa [23, 24].
These parameters are supposed to provide flexibility to researchers for
choosing a sub-set of provided data for controlled studies.

##  4 Test set

For DNS Challenge 3, the test set included utterances in English and non-
English languages recording in the presence of a variety of background noises
at different SNR, target levels, and acoustic conditions. Non-English
languages included tonal languages such as Punjabi, Vietnamese, Mandarin, and
Cantonese. Other Non-English languages included Spanish, German, Portuguese
and French. We crowd-sourced the noisy speech collection efforts to get
diversity in terms of languages, acoustic conditions, noise environments,
speaker’s age, ethnicity and to have gender balance. The utterances were
collected at a distance of 1-5 meters from the microphone when they were not
using a headphone to have more reverberation. The development and blind test
sets included utterances with emotions such as laughter, crying, yelling, and
surprise in the presence of background noise. This is to measure the effects
of noise suppressors on human emotions. Many noise suppressors tend to be
aggressive and end up suppressing low-energy emotions and sounds. A small
segment of the clips includes speech in the presence of musical instruments
such as guitar, piano, violin playing in the background. This is to ensure
that noise suppression methods do well in the presence of musical tones
overlapping with speech. We also included speech collected in the presence of
stationary noise as it is the most common use case scenario. All the clips
were originally collected at a sampling rate of 48 kHz and were resampled to
16 kHz.

(a) Speech MOS

(b) Background Noise MOS

(c) Overall MOS

Figure 2: Track 1 results

(a) Speech MOS

(b) Background Noise MOS

(c) Overall MOS

Figure 3: Track 2 results

##  5 Challenge Results and Key Takeaways

###  5.1 Evaluation set up and results

The final evaluation was done on the blind test set using the crowdsourced
subjective evaluation framework based on ITU-T P.835 [8] to determine the DNS
quality. Each clip was rated by 5 qualified raters, which gave the maximum 95%
Confidence Interval (CI) of 0.05 DMOS per model. A total of 19 teams
participated in track 1 and only 3 teams participated in track 2. Figures 2
and 3 shows the Speech MOS, Background Noise MOS and the Overall MOS results
for track 1 and 2 respectively. The tables contain Differential MOS (DMOS)
which is the difference between the MOS of the processed clips by a model (or
the team) and the MOS of the original noisy speech. The original noisy speech
MOS is shown inside the parenthesis in the row containing DMOS for ”Noisy”.
The absolute MOS can be computed by adding the MOS of ”Noisy” with the DMOS of
a particular model of interest. The participants will be ranked based on
Overall MOS given they satisfy other challenge requirements. Participants are
required to submit the number of operations per second of their model. This
will be used as a tie-breaker. The challenge also requires the participating
teams to submit a paper to INTERSPEECH 2021 explaining their method and get
the paper accepted.

###  5.2 Key takeaways

  1. 1.

We can see from figure 2a that all the teams except for the top 2 did worse
than noisy in terms of speech quality measured by Speech MOS. Most of the
noise suppressors introduce speech distortion when they get aggressive in
suppressing noise and end up suppressing speech components. This is likely due
to an inaccurate estimation of noise power in each frame.

  2. 2.

The Background Noise MOS in 2b shows that almost all the models are trained
well to suppress the background noise. The top 3 models achieved a DMOS of
almost 2 (MOS of 4.6), which is a significant improvement over noisy speech.

  3. 3.

According to figure 2c, the best model achieved an Overall MOS of about 3.78,
which is about 1.2 DMOS less than the perfect quality of MOS 5. This shows
that achieving superior noise suppression without distorting speech is a
challenging problem. Analysis in [8] shows that the best theoretical Overall
MOS that can be achieved is 3.92 assuming Bacground Noise MOS of 5 and Speech
MOS same as that of noisy speech. Hence, the field of speech enhancement
optimized for human perception is still in its nascent phase.

  4. 4.

Researchers can get the biggest bang for the buck in terms of Overall MOS by
improving the Speech MOS by maintaining excellent Background Noise MOS.

##  6 Conclusions

The INTERSPEECH 2021 DNS Challenge was organized to help researchers from
academia and industry to come together and tackle this challenging problem in
speech enhancement. Large inclusive and diverse training and test datasets
with supporting scripts were open sourced along with other tools such as
perceptual objective metrics. Many participants from both industry and
academia found the datasets very useful.

## References

  * [1] Y. Ephraim and D. Malah, “Speech enhancement using a minimum-mean square error short-time spectral amplitude estimator,” _IEEE TASLP_ , 1984.
  * [2] C. Karadagur Ananda Reddy, N. Shankar, G. Shreedhar Bhat, R. Charan, and I. Panahi, “An individualized super-gaussian single microphone speech enhancement for hearing aid users with smartphone as an assistive device,” _IEEE Signal Processing Letters_ , vol. 24, no. 11, pp. 1601–1605, 2017.
  * [3] S. Fu, Y. Tsao, X. Lu, and H. Kawai, “Raw waveform-based speech enhancement by fully convolutional networks,” in _2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_.
  * [4] H.-S. Choi, H. Heo, J. H. Lee, and K. Lee, “Phase-aware single-stage speech denoising and dereverberation with U-net,” _arXiv preprint arXiv:2006.00687_ , 2020.
  * [5] Y. Koyama, T. Vuong, S. Uhlich, and B. Raj, “Exploring the best loss function for DNN-based low-latency speech enhancement with temporal convolutional networks,” _arXiv preprint arXiv:2005.11611_ , 2020.
  * [6] C. K. Reddy _et al._ , “The INTERSPEECH 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results,” in _ISCA INTERSPEECH_ , 2020.
  * [7] C. K. Reddy, H. Dubey, V. Gopal, R. Cutler, S. Braun, H. Gamper, R. Aichner, and S. Srinivasan, “Icassp 2021 deep noise suppression challenge,” _arXiv preprint arXiv:2009.06122_ , 2020.
  * [8] B. Naderi and R. Cutler, “A crowdsourcing extension of the itu-t recommendation p. 835 with validation,” _arXiv preprint arXiv:2010.13200_ , 2020.
  * [9] C. K. Reddy, V. Gopal, and R. Cutler, “Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,” _arXiv preprint arXiv:2010.15258_ , 2020.
  * [10] C. K. Reddy _et al._ , “A scalable noisy speech dataset and online subjective test framework,” _arXiv preprint arXiv:1909.08050_ , 2019.
  * [11] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in _IEEE ICASSP_ , 2015\.
  * [12] B. Naderi and R. Cutler, “An open source implementation of ITU-T recommendation P.808 with validation,” in _ISCA INTERSPEECH_ , 2020.
  * [13] J. Wilkins, P. Seetharaman, A. Wahl, and B. Pardo, “Vocalset: A singing voice dataset.” in _ISMIR_ , 2018.
  * [14] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “CREMA-D: Crowd-sourced emotional multimodal actors dataset,” _IEEE Trans. on Affective Computing_ , vol. 5, no. 4, pp. 377–390, 2014.
  * [15] Z. Z. Dong Wang, Xuewei Zhang, “THCHS-30 : A free chinese speech corpus,” 2015\. [Online]. Available: http://arxiv.org/abs/1512.01882
  * [16] H. Bu, J. Du, X. Na, B. Wu, and H. Zheng, “Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline,” in _2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA)_. IEEE.
  * [17] “The Spoken Wikipedia Corpora,” https://nats.gitlab.io/swc/, [Online; accessed 2020-09-01].
  * [18] “Telecooperation German Corpus for Kinect,” http://www.repository.voxforge1.org/downloads/de/german-speechdata-TUDa-2015.tar.gz, [Online; accessed 2020-09-01].
  * [19] “M-AILABS Speech Multi-lingual Dataset,” https://www.caito.de/2019/01/the-m-ailabs-speech-dataset/, [Online; accessed 2020-09-01].
  * [20] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _IEEE ICASSP_ , 2017.
  * [21] J. Thiemann, N. Ito, and E. Vincent, “The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings,” _The Journal of the Acoustical Society of America_ , p. 3591, 05 2013.
  * [22] T. Ko, V. Peddinti, D. Povey, M. L. Seltzer, and S. Khudanpur, “A study on data augmentation of reverberant speech for robust speech recognition,” in _IEEE ICASSP_ , 2017.
  * [23] P. Antsalo _et al._ , “Estimation of modal decay parameters from noisy response measurements,” in _Audio Engineering Society Convention 110_ , 2001\.
  * [24] H. Gamper, “Blind C50 estimation from single-channel speech using a convolutional neural network,” in _Proc. IEEE MMSP_ , 2020, pp. 136–140.


# Family Shopping Recommendation System
Using User Profile and Behavior Data

Jiacheng Xu  School of Computer Science and
Software Engineering
East China Normal University
Email<EMAIL_ADDRESS>
###### Abstract

With the arrival of the big data era, recommendation system has been a hot
technology for enterprises to streamline their sales. Recommendation
algorithms for individual users have been extensively studied over the past
decade. Most existing recommendation systems also focus on individual user
recommendations, however in many daily activities, items are recommended to
the groups not one person. As an effective means to solve the problem of group
recommendation problem, we extend the single user recommendation to group
recommendation. Specifically we propose a novel approach for family-based
shopping recommendation system. We use the dataset from the real shopping mall
consisting of shopping records table, client-profile table and family
relationship table. Our algorithm integrates user behavior similarity and user
profile similarity to build the user based collaborative filtering model. We
evaluate our approach on a real-world shopping mall dataset.

##  I Introduction

With the arrival of the big data era, recommendation system [1, 2] is a kind
of big data technology for enterprises to gain more profit. Recently,
recommendation systems are becoming one of the hotspots in the field of
recommendation systems.

In recent years, with the recommendation system research and developments very
rapid, there are more and more various types of recommendation systems such as
mobile recommendation systems, news recommendation system, music
recommendation system and so on. However, these recommendation systems are
only for individual users. In fact, many daily activities are make up of many
users in groups (e.g. watching movies or TV shows, going to a restaurant,
vacationing etc.). Therefore, the recommendation system needs to consider the
preferences of each user in the group for recommendation. This kind of system
is called group recommendation system [3]. It extends the recommendation
object from a single user to a group, posing new challenges to recommendation
systems. The interests of the group members can be diverse in reality. How to
get the common preference of the group members, alleviating the preference
conflict among the group members, and make the recommendation result meet the
needs of all group members as much as possible, are the key problems [4] to be
solved in the group recommendation system. But at present, the research of
group recommendation system has received more and more attention.

Nowadays, group recommendation system is more and more common such as Let’s
Browse (news group recommendation system) [5], Masthoff’s Group Recommender
(music group recommendation system) [6], Intrigue (tourism group
recommendation system) [7], Where2Eat (Food group recommendation system) [8]
and so on. From the above example, group recommendation system can be seen
everywhere in our lives.

In this paper, we proposed a novel family recommendation system algorithm. We
use the dataset from the real shopping mall consisting of shopping records
table, client-profile table and family relationship table. Its aim is to bring
huge profits to the shopping mall. In fact, the entire data set acquisition
process is very difficult. Because when users go shopping in the supermarket,
they do not leave their family information. In other words, they do not tell
us who is his wife or who is his children. For solving the problem, we
determined their family relationship by telephone in the process of collecting
the data. Preprocessing is performed to do data cleaning and curation, which
leads to the systematic experiments conducted in the paper.

In a nutshell, the main contributions of this work are:

1) To our best knowledge, this is the first work on family level personalized
recommendation system, which is useful for the case when a family share a
common account or family members will purchase items for the other members in
the family.

2) We propose a novel approach for integrating the information from
user/family purchase data, activity data and shopping mall event data into a
unified model for the family based recommendation problem.

3) We verify our approach on via an empirical study on a real-world dataset
and showcase the efficacy of our method.

In Section I, we present the background of recommendation system and group
recommendation system and propose a new group recommendation algorithm. The
rest of paper is structured as follows: Section II presents the commonly used
algorithms such as content-based collaborative filtering [9] and group-based
collaborative filtering. In section III, we specifically explain the detail of
our group recommendation system architecture. Section IV shows the
experimental result by peer algorithms. Finally Section V concludes this
paper.

##  II Related Work

###  II-A Memory-based Collaborative Filtering

Memory-based collaborative filtering (CF) algorithms are popular techniques
for recommender systems. It involves the user-item database to predict which
items users really like. According to every user who belongs to a group of
people with similarity interests, we can find a group of people who share
similar interests. By doing so, we can recommend the item to a user by it. In
fact, it is neighborhood-based recommendation system [10] and very prevalent.

The neighborhood-based CF algorithm uses the following steps. Firstly,
calculate the similarity represented by $W_{i,j}$ which reflects correlation
or distance between two uses or two items ,$i$ and $j$. Secondly, predict the
unknown ratings of the user or item on a certain item or user by aggregating
the neighbors who are $k$ most similar to get the top-N most frequent items as
the recommendation, when the task is to generate a top-N recommendation
result.

####  II-A1 Similarity Computation

Similarity computation is one of the most important steps in the process of
the recommendation system. Different methods that get the similarity lead to
different results. For a user-based collaborate filtering algorithm, we
calculate the similarity, $W_{u,v}$, between the users $u$ and $v$ who have
both rated the same items. While for an item-based collaborate filtering
algorithm [11], we calculate the similarity, $W_{i,j}$, between the items $i$
and $j$ who have both been rated by the same users. Now, correlation-based
similarity and vector cosine-based similarity are fluent and efficient
methods.

Vector cosine-based similarity is a useful method to calculate the similarity
as popular as the correlation-based similarity we discuss below. The
similarity between two items or users can be calculated by treating each items
or users as a vector and computing the cosine of the angle between two
vectors[12]. Hence the element inside the vector of the item is the user’s
rating of the item. Vector cosine similarity between items $i$ and $j$ is
given by

$\displaystyle
W_{i,j}=\cos(\vec{i},\vec{j})=\frac{\vec{i}\cdot\vec{j}}{\left\|\vec{i}\right\|*\left\|\vec{j}\right\|}$
$\displaystyle=\frac{\sum_{u\in U}r_{u,i}r_{u,j}}{\sqrt{\sum_{u\in
U}r_{u,i}^{2}}\sqrt{\sum_{u\in U}r_{u,j}^{2}}}$

where $\cdot$ denotes the dot-product of the two item vectors $i$, $j$ and $U$
is the whole set of users and $\bar{r_{u,i}}$ is the rating that user $u$ give
the item $i$, which is shown in Fig.1.

But the vector cosine-based similarity algorithm has some limitations. In
practice, different users may rate some items using different rating standard.
As a result, the vector cosine similarity may not represent the similarity
between two items. Taking this into account, we adjust the vector cosine-based
similarity algorithm to address this shortcoming. Simply, we can subtract the
corresponding user average from each co-rated pair. In fact, the adjusted
vector cosine similarity has the same formula as the Pearson correlation
similarity. In the following, correlation-based similarity will be introduced
in detail.

Correlation-based similarity is the method that we calculate the similarity by
Pearson correlation. As we known, Pearson correlation denotes that two
variables linearly relate with each other. For the user-based algorithm, the
Pearson correlation between users $u$ and $v$:

$\displaystyle W_{u,v}=\frac{\sum_{i\in
I}(r_{u,i}-\bar{r_{u}})(r_{v,i}-\bar{r_{v}})}{\sqrt{\sum_{i\in
I}(r_{u,i}-\bar{r_{u}})^{2}}\sqrt{\sum_{i\in I}(r_{v,i}-\bar{r_{v}})^{2}}}$

where $I$ is the whole set of items, $\bar{r_{u}}$ is the average rating of
the items of user $u$ and $r_{u,i}$ is the rating that use $u$ give to the
item $i$. It show in Fig.1.

Figure 1: Users-items rating matrix.

For the item-based algorithm, the Pearson correlation between users $u$ and
$v$:

$W_{i,j}=\frac{\sum_{u\in
U}(r_{u,i}-\bar{r_{i}})(r_{u,j}-\bar{r_{j}})}{\sqrt{\sum_{u\in
U}(r_{u,i}-\bar{r_{u}})^{2}}\sqrt{\sum_{u\in U}(r_{u,j}-\bar{r_{u}})^{2}}}$

where $U$ is the whole set of users and $\bar{r_{i}}$ is the average rating of
the items $i$. It show in Fig.1.

In a word, the Pearson correlation-based CF algorithm as a typical collaborate
filtering algorithm is so efficient that it is widely used in the
Recommendation system [13, 14].

But the above two methods need the rating matrix about users and items. In a
real case, the rating data is not easily available and often is very sparse.
However, we can still obtain useful information from implicit data. For
example, if two users buy the same goods, they have some common preferences in
a shopping mall. The next method can address such a problem by calculating the
precise similarity.

Jaccard similarity [15] is very simple algorithm. Its main idea is that the
more same goods purchased by the two users and the fewer amount of goods
bought by the two users, the higher the similarity between them. It is very
useful to process the shopping records without ratings. Jaccard similarity
formula between items $i$ and $j$ is given by:

$W_{u,v}=\frac{\left|N_{u}\bigcap N_{v}\right|}{\left|N_{u}\bigcup
N_{v}\right|}$

where $N_{u}$ is the set of items which user $u$ purchase and $N_{v}$ is the
set of items which user $v$ purchase. $\left|N_{u}\bigcap N_{v}\right|$
denotes the same items which are bought by users $u$, $v$. And
$\left|N_{u}\bigcup N_{v}\right|$ means that the amount items which are bought
by users $u$, $v$. If users $u$, $v$ do not buy the same items, their
similarity is 0. If users $u$, $v$ do not buy the different items, their
similarity is 1. So we find it can reflects the similarity between users $u$,
$v$ effectively.

Euclidean distance similarity[16] is another similarity algorithm. It does not
need the relationship between users and items. Each item or user has its own
attributes. Therefore we can calculate the user similarity by users’ profile
by Euclidean distance. Each user attribute can be composed of a vector. For
example, the vector of use $u$ can be denoted by
$\vec{u}=(a_{1},a_{2},...,a_{n})$. In brief, the similarity problem is
transformed into two vector distance problems. Then we decide to choose the
Euclidean distance. But what we need to pay attention to is that the more
distance, the less similarity.

####  II-A2 Prediction Computation

Prediction computation is also one of critical parts in a collaborative
filtering system [17]. In fact, many methods share similar ideas. In the
neighborhood-based CF algorithms, one can choose a subset of the nearest
neighbors of the user according to their similarity with another one and
aggregate their ratings to generate predictions for user. We can take a
weighted average of all the ratings on that item $i$ to make a precise
prediction for the rating of the user $k$, on a certain item $i$. The
following formula is given by [13]:

$r_{k,i}=\bar{R}_{k}+\frac{\sum_{u\in U}(r_{u,i}-\bar{r}_{u})\cdot
w_{k,u}}{\sum_{u\in U}\left|w_{k,u}\right|}$

where $w_{k,u}$ is the weight between the user $k$ and user $u$, $\bar{r}_{k}$
is the average ratings for the user $k$ on all other rated items as same as
$\bar{r}_{u}$. $U$ is the set of users who have rated the item $i$. For
memory-based prediction, we can also predict the rating simply by the
following formula:

$r_{k,i}=\frac{\sum_{u\in U}r_{u,i}\cdot w_{k,u}}{\sum_{u\in
U}\left|w_{k,u}\right|}$

####  II-A3 Top-N Recommendations

The Top-N Recommendations are that a set of N top-ranked items, which attract
the specific users, are recommended. For example, when you use NetEase cloud
music, if you are an old user, you will be recommended a list of songs which
you may be interested in. The Top-N Recommendations can mainly divided into
two classes, User-Based Top-N Recommendation Algorithms and Item-Based Top-N
Recommendation Algorithms.

For the former, we first choose the $k$ most similar users to the specific
user by using the Pearson Correlation Similarity or Vector Cosine-based
Similarity [18]. After the $k$ most similar users have been chosen, we can
aggregate their corresponding rows in the user-item rating matrix $R$ to
identify a set of items, $A$. With the set $A$, user-based collaborate
filtering techniques recommend the items with the top-N highest scores in $A$
which the specific user has not purchased.

For the latter, we firstly choose the $k$ most similar items for each item
according to their similarities predicted by Pearson Correlation Similarity or
Vector Cosine-based Similarity. Then we can identify the set, $A$, by taking
the union of the k most similar items and removing each of the items in the
set, $U$, which the specific user has already purchased. The set $A$ is
candidates of recommended items. Then we predict the similarities between each
item of the set $A$ and the set $U$. We can recommend the Top-N items [19]
according to the resulting set of the items which are sorted in decreasing
order of the ratings in $A$. The above is the summary concept of the Top-N
Recommendations.

###  II-B Group-based Collaborative Filtering

Group-based Collaborative Filtering is the important point in this paper. It
is different from the single user recommendation system, especially the
calculating similarity method. Sometimes, it can solve the Cold-Start problem
[20]. Group detection and group preference aggregation are two important part
of the Group-based Collaborative Filtering.

####  II-B1 Group Detection

In most of the existing group recommendation systems, it has identified the
grouping [21, 22]. However, in many group recommendation systems, the grouping
is not given. So we should add a user to a certain group by some method. We
can call it group detection. There are two commonly used methods: the first is
based on user preferences and the second is based on demographic methods. In
fact, it is the cluster problem. In this field, the K-means is the most famous
method. For example, in [23], the user-item scoring matrix is directly input
to the K-means clustering algorithm, and the users are clustered according to
user preferences. And demographic information can also be the input to the
K-means clustering algorithm. For example, the group recommendation system
Intrigue [7] divides the group into different subgroups according to the
demographic characteristics of the user. But in my experience, the grouping is
given by investigating families.

####  II-B2 Group Preference Aggregation Strategy

Group preference aggregation which is proposed by Dyer et al. [24] is the most
important part of the group recommendation system. In the group recommendation
systems, preference aggregation refers to the extraction of group preference
according to the preferences of group members [24]. In an actual world, group
recommendation systems need to meet the different requirements by the
different group preference aggregation strategies. Meanwhile, it also need to
meet the overall satisfaction, fairness, understandability and other
requirements.

Recently, some basic group preference aggregation strategies in group
recommendation systems have been studied [25]. In [26], ten group preference
aggregation strategies are analyzed in detail. Then five commonly used
strategies are discussed below.

Average Strategy: The average of the group membership ratings is taken as the
rating of this group.

Most Pleasure Strategy: The maximum of the group membership ratings is taken
as the rating of this group.

Least Misery Strategy: The minimum of the group membership ratings is taken as
the rating of this group [27].

Average Without Misery Strategy: The average of the group membership ratings
which is higher than the certain threshold is taken as the rating of this
group[28].

Most Respected Person Strategy: The rating of group is decided by the ratings
of the most respected person in this group.

####  II-B3 Group Preference Aggregation Method

Group Preference Aggregation which occurs at different stage of the
recommendation is also different. Paper [29] summarizes Group Preference
Aggregation Method into three categories: recommendation result aggregation,
rating aggregation and group preference modeling.

For recommendation result aggregation and rating aggregation, we first
generate the recommendation for each user of the group or calculates the
predictive ratings, and then generate a group recommendation aggregation or
group prediction ratings.

For the group preference model, it first generates a group preference model by
aggregating the group members preferences models. Then we can recommend the
items by the group preference model.

In this paper, we mainly aggregate the attributes of the numbers of the family
group by a certain strategy and then recommend the items to the family group.

##  III Proposed Approach

###  III-A Datasets

The dataset consist of client profile, transaction dataset, visit dataset,
family dataset and participation dataset from the real shopping mall.

Client profile dataset has 4505 client profiles with seven attributes: join
days, sex, age, phone, email, neighborhood, register source and income. Its
main contents are the client profile which are used to calculate the user
similarity.

Transaction dataset contains 25550 shopping records with categories:
_productbrand_ , _producttype_ and _maincategory_.The items distribution with
three categories show in the Table I, II, III. It is used to calculate the
user similarity by Jaccard similarity algorithm.

Visit dataset has 60427 records with check-in timestamp and checkout
timestamp.

Participation dataset records the activities each user participates.

Family dataset has 6378 family groups. Every family group data contains the
information about the composition of the family.

TABLE I: Item Distribution-Product Brand Product Brand | Amount
---|---
DXNIKE | 2435
DXDISNEY | 2003
JEEP SPIRIT | 1629
G&H | 1578
K&C | 1437
BAB | 1258
DBN | 1138
NEW BALANCE | 1005
L | 939
… | …
TABLE II: Item Distribution-Product Type. Product Type | Amount
---|---
Miscellaneous | 6108
CLOTHING | 5715
INFANT & KIDS | 3838
SHOES | 2808
SPORTSWEAR | 1404
ACCESSORIES | 1314
TOYS | 1198
FOOD | 664
BAB TOY | 643
CASUALWEAR | 609
… | …
TABLE III: Item Distribution-Main Category. Main Category | Amount
---|---
INFANT & KIDS | 8030
LIFESTYLE | 4021
MENS WEAR | 2861
Miscellaneous | 2301
KIDS | 1805
HOME & LIFESTYLE | 1332
TOYS | 766
KNITWEAR | 620
ACCESSORIES | 448
TC WOVEN SHIRTS | 394
… | …

###  III-B Data Preprocessing

Data Preprocessing is the important stage in the family recommendation system.
It can not only help us with numerical calculations but also make the
experiment results better. Data Preprocessing can be divided into two parts.

####  III-B1 Data Cleaning

Some information of the items with the missing values are deleted or replaced
by the certain value such as zero, the average value and so on. It can make
our dataset more complete.

####  III-B2 Data Normalization

The form of the elements of the dataset is various, so it’s hard to deal
directly with them.

For the user-behavior feature, we can normalize the data items of the
participation dataset and the transaction dataset to triples (_memberid_ ,
_productbrand_`\`_producttype_`\`_maincategory_`\`_activity_ , the quantity of
the items).

For the client profile dataset, we can separate the attributes of client into
numeric variable and category variable. The category variables such as sex,
neighborhood, register source and so on can be encoded by the one-shot
encoding. For example, the attribute _sex_ with three values can be encoded to
the vector( Female: $(0,1)$ Male: $(1,0)$ ) and the numeric variables can be
mapped to 0 to 1 by the normalization method. For the attribute _telephone_
and _email_ , we can encode them to 0 or 1 according to whether user left the
information.

###  III-C User Behavior-based Similarity Matrix

According to the user-behavior feature with four categories (_productbrand_ ,
_producttype_ , _maincategory_ , _activity_), we use the triples which we
propose in the section III-B. In order to get all the set of items which the
user have bought as the inputs to the Jaccard similarity algorithm proposed in
the section II-A1, then can get four user similarity matrix[30], which are
respectively denoted as $W_{brand}$, $W_{type}$, $W_{category}$,
$W_{activity}$.

###  III-D User Profile-based Similarity Matrix

In the client profile dataset, using the user profile, we can calculate the
user similarity matrix[31, 32]. Firstly, we can generate the user profile
vectors by the method introduced in the section III-B. For example, we can
formula the user attributes to the vector
$\vec{a}=(1,1,0,0,1,0,0,0,0,2,2,1.907638,0.567901,0.468739,...)$. Then we can
use the user profile vector to calculate the user distance matrix by the
Euclidean distance proposed in the section III-B. Then we normalize all the
elements of the user distance matrix to interval$(0,1)$, denoted as $D$.
Finally, we get the user similarity matrix by the formula:

$W_{uv}=1-D_{uv}\quad s.t.\quad u,v\in U$

where the dimensions of the matrix $W$ and matrix $B$ are the same and
$W_{uv}$ is the similarity between user $u$ and user $v$ which belong to the
set of users denoted as $U$. To distinguish the similarity matrix above, we
denote the similarity matrix as $W_{profile}$.

###  III-E Hybrid Similarity Matrix Algorithm

The five similarity matrix ($W_{profile}$, $W_{brand}$, $W_{type}$,
$W_{category}$, $W_{activity}$) has already gotten. Then we can aggregate them
to get more better similarity matrix. Because aggregating them which are
gotten from different categories can fully represent the similarity between
users. The empirical experimental results suggest the usefulness of our model.

###  III-F Family Recommendation Algorithm

The single user similarity matrix algorithms have been discussed above. For
the group similarity matrix, the group discovering can be stepped, because we
have known the member of the family according to the Family dataset.

Then for the family behavior-based similarity matrix, we combine the triples
mentioned in the section III-B2 with family units. For example, user $i$
triple is $(i,1,1)$ and user $j$ triple is $(j,4,1)$. The members of the
family $f$ are user $i$ and user $j$, so the family $f$ triples are $(f,1,1)$
and $(f,4,1)$. Then we use the triples to get all the set of items the family
have bought as the inputs to the Jaccard similarity algorithm proposed in the
section II-A1. Finally we can get four user similarity matrix, which are
respectively denoted as $W_{brand}^{F}$, $W_{type}^{F}$, $W_{category}^{F}$,
$W_{activity}^{F}$.

For the family profile-based similarity matrix, we firstly sum the vectors of
all the member of the family $f$ as the profile vector of the family $f$. We
can calculate the similarity matrix like the user profile-based similarity
matrix, denoted as $W_{profile}^{F}$. It can make the dissimilar vectors
farther apart and the similar vector more closer. Finally, we aggregate the
five similarity matrix to get the final similarity matrix denoted as
$W_{Final}^{F}$ and recommend the item with the family unit using
$W_{Final}^{F}$. The whole procedure is shown in the Fig.2.

Figure 2: Overview of the family recommendation approach.

##  IV Experiments

###  IV-A Models

We mainly propose three different models and compare them by the evaluation
metric. Finally, we can find that the third model is better than the first one
and the second one according to experiment results.

User Model:

User Behavior-based + User Profile-based Recommendation Method

Hybrid User Model:

User Behavior-based+User Profile-base + Hybrid Similarity Matrix Algorithm
Recommendation Method

Hybrid Family Model:

User Behavior-based+User Profile-based + Hybrid Similarity Matrix Algorithm +
Family Recommendation Method

The recommended items in this experiment have three categories (_productbrand_
, _producttype_ , _maincategory_). (The details are told in Section IV-D.
_Experimental Procedure_.)

As a result, for the _User Model_ , we recommend the item _productbrand_ by
aggregating the $W_{brand}$, $W_{activity}$ and $W_{profile}$,recommend the
item _producttype_ by aggregating the $W_{type}$, $W_{activity}$ and
$W_{profile}$ and recommend the item _maincategory_ by aggregating the
$W_{category}$, $W_{activity}$ and $W_{profile}$.

The _Hybrid User Model_ and _Hybrid Family Model_ recommend the item with
three categories by using one similarity matrix. For the _Hybrid User Model_ ,
we recommend the item by aggregating the $W_{brand}$, $W_{type}$,
$W_{category}$, $W_{activity}$ and $W_{profile}$. For the _Hybrid Family
Model_ ,we recommend the item by aggregating the $W_{brand}^{F}$,
$W_{type}^{F}$, $W_{category}^{F}$, $W_{activity}^{F}$ and $W_{profile}^{F}$.

###  IV-B Dataset Splitting

Dataset splitting is the important stage when we split the transaction dataset
into two parts, one part as testing set and the other part as training set.
Because it is the shopping records with timestamps, we decide to split the
dataset by the time point. By observing the time and space distribution of the
data which is shown in Fig.3., we choose the timestamp ‘2016-07-15 00:00:00’
as the splitting point of the testing set and the training set. Testing set
and training set account for 20 percent and 80 percent respectively. Because
it is shopping data, testing set and training set split by the time point will
be better for us to evaluate the model performances according to the data
characteristics.

Figure 3: Time and space distribution of the data.

###  IV-C Evaluation Metric

There are many evaluation metric for recommender system [33], such as RMSE
(Root Mean Square Error), MAE (Mean Absolute Error), recall rate, precision
rate and so on. Here, we finally choose the recall rate and the precise rate
for their popularity in related works.

The recall rate describes how many percentage of the user’s shopping records
are included in the final recommendation list, and the precise rate describes
how many the final recommended lists are user shopping records that have
occurred.

Recall Rate is given by

$Recall=\frac{\sum_{u\in U}\left|R(u)\bigcap T(u)\right|}{\sum_{u\in
U}\left|T(u)\right|}$

Precision Rate is given by

$Recall=\frac{\sum_{u\in U}\left|R(u)\bigcap T(u)\right|}{\sum_{u\in
U}\left|R(u)\right|}$

where $R(u)$ is the set of the items which are recommended to the user $u$ and
$T(u)$ is the set of the items which the user $u$ have already bought.

###  IV-D Experimental Procedure

Our experiment is to recommend the items from three categories (_productbrand_
, _producttype_ , _maincategory_) by the Top-$k$ recommendation which $k$ is
increasing from 1 to 10 and to calculate the recall rate and precise rate of
them. In addition, we propose three models. In summary, we can get the 60 data
points drawn in the Fig.4 and the Fig.5 to compare the performances between
the different models.

###  IV-E Experimental Results

As Fig.4 shows, the dashed line represents the _User Model_ and the straight
line represent the _Hybrid User Model_. The recall rate curve of the _User
Model_ and _Hybrid User Model_ both increase with $k$ increasing and The
precise rate curve of the _User Model_ and _Hybrid User Model_ , when k is
larger than 3, both decrease with $k$ increasing. The _Hybrid User Model_ ,
whether from accuracy or recall, is better than the _User Model_.

As Fig.5 shows, the straight line represents the _Hybrid User Model_ and the
dashed line represent the _Hybrid Family Model_. The recall rate curve of the
_Hybrid User Model_ and the _Hybrid User Model_ both increase with $k$
increasing and The precise rate curve of the _Hybrid User Model_ and the
_Hybrid Family Model_ , when $k$ is larger than 3 both decrease with $k$
increasing. The _Hybrid Family Model_ , whether from accuracy or recall, is
better than the _Hybrid User Model_.

According to the conclusion above, the _Hybrid Family Model_ , for both
accuracy and recall, is better than the _User Model_ and _Hybrid User Model_.

##  V Conclusion and Future Work

Firstly, we discuss the single user recommendation system and the popular
group recommendation systems from multiple perspectives. After researching the
dataset , we propose a novel approach for integrating the information from
user/family purchase data, activity data and shopping mall event data into an
unified model for the family based recommendation problem. we verify our
approach on via an empirical study on a real-world dataset and showcase the
efficacy of our method. Finally, the algorithm works well according to the
experimental results.

In the future, this algorithm can be combined with the matrix factorization or
other machine learning technology. This would be a good starting point for
further research.

Figure 4: User Model and Hybrid User Model. Figure 5: Hybrid User Model and
Hybrid Family Model.

## References

  * [1] X. Su and T. M. Khoshgoftaar, “A survey of collaborative filtering techniques,” _Advances in Artificial Intelligence_ , vol. 2009, no. 12, 2009\.
  * [2] P. Resnick and H. R. Varian, “Recommender systems,” _Communications of the ACM_ , vol. 40, no. 3, pp. 56–58, 1997.
  * [3] L. Boratto, “Group recommender systems,” in _Proceedings of the 10th ACM Conference on Recommender Systems_. ACM, 2016, pp. 427–428.
  * [4] J. Masthoff, “Group recommender systems: Combining individual models,” in _Recommender systems handbook_. Springer, 2011, pp. 677–702.
  * [5] H. Lieberman, N. Van Dyke, and A. Vivacqua, “Let’s browse: a collaborative browsing agent,” _Knowledge-Based Systems_ , vol. 12, no. 8, pp. 427–431, 1999.
  * [6] J. Masthoff and A. Gatt, “In pursuit of satisfaction and the prevention of embarrassment: affective state in group recommender systems,” _User Modeling and User-Adapted Interaction_ , vol. 16, no. 3-4, pp. 281–319, 2006.
  * [7] L. Ardissono, A. Goy, G. Petrone, M. Segnan, and P. Torasso, “Intrigue: personalized recommendation of tourist attractions for desktop and hand held devices,” _Applied Artificial Intelligence_ , vol. 17, no. 8-9, pp. 687–714, 2003.
  * [8] F. Guzzi, F. Ricci, and R. Burke, “Interactive multi-party critiquing for group recommendation,” in _Proceedings of the fifth ACM conference on Recommender systems_. ACM, 2011, pp. 265–268.
  * [9] M. Balabanović and Y. Shoham, “Fab: content-based, collaborative recommendation,” _Communications of the ACM_ , vol. 40, no. 3, pp. 66–72, 1997.
  * [10] R. M. Bell and Y. Koren, “Improved neighborhood-based collaborative filtering,” in _KDD cup and workshop at the 13th ACM SIGKDD international conference on knowledge discovery and data mining_. Citeseer, 2007, pp. 7–14.
  * [11] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, “Item-based collaborative filtering recommendation algorithms,” in _Proceedings of the 10th international conference on World Wide Web_. ACM, 2001, pp. 285–295.
  * [12] G. Salton and M. J. McGill, “Introduction to modern information retrieval,” 1986\.
  * [13] P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl, “Grouplens: an open architecture for collaborative filtering of netnews,” in _Proceedings of the 1994 ACM conference on Computer supported cooperative work_. ACM, 1994, pp. 175–186.
  * [14] M. R. McLaughlin and J. L. Herlocker, “A collaborative filtering algorithm and evaluation metric that accurately model the user experience,” in _Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval_. ACM, 2004, pp. 329–336.
  * [15] H. Seifoddini and M. Djassemi, “The production data-based similarity coefficient versus jaccard’s similarity coefficient,” _Computers & industrial engineering_, vol. 21, no. 1, pp. 263–266, 1991.
  * [16] F. Fouss, A. Pirotte, J.-M. Renders, and M. Saerens, “Random-walk computation of similarities between nodes of a graph with application to collaborative recommendation,” _IEEE Transactions on knowledge and data engineering_ , vol. 19, no. 3, pp. 355–369, 2007.
  * [17] J. L. Herlocker, J. A. Konstan, A. Borchers, and J. Riedl, “An algorithmic framework for performing collaborative filtering,” in _Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval_. ACM, 1999, pp. 230–237.
  * [18] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, “Analysis of recommendation algorithms for e-commerce,” in _Proceedings of the 2nd ACM conference on Electronic commerce_. ACM, 2000, pp. 158–167.
  * [19] G. Karypis, “Evaluation of item-based top-n recommendation algorithms,” in _Proceedings of the tenth international conference on Information and knowledge management_. ACM, 2001, pp. 247–254.
  * [20] Z. Miao, J. Yan, K. Chen, X. Yang, H. Zha, and W. Zhang, “Joint prediction of rating and popularity for cold-start item by sentinel user selection,” vol. PP, no. 99, pp. 1–1, 2016.
  * [21] M. O’connor, D. Cosley, J. A. Konstan, and J. Riedl, “Polylens: a recommender system for groups of users,” in _ECSCW 2001_. Springer, 2001, pp. 199–218.
  * [22] J. Freyne and B. Smyth, “Cooperating search communities,” in _International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems_. Springer, 2006, pp. 101–110.
  * [23] L. Boratto, S. Carta, and M. Satta, “Groups identification and individual recommendations in group recommendation algorithms,” in _Workshop on the Practical Use of Recommender Systems, Algorithms and Technologies (PRSAT), Held in Conjunction with RecSys_ , 2010, pp. 27–34.
  * [24] J. S. Dyer and R. K. Sarin, “Group preference aggregation rules based on strength of preference,” _Management Science_ , vol. 25, no. 9, pp. 822–832, 1979.
  * [25] L. Baltrunas, T. Makcinskas, and F. Ricci, “Group recommendations with rank aggregation and collaborative filtering,” in _Proceedings of the fourth ACM conference on Recommender systems_. ACM, 2010, pp. 119–126.
  * [26] J. Masthoff, “Group modeling: Selecting a sequence of television items to suit a group of viewers,” in _Personalized Digital Television_. Springer, 2004, pp. 93–141.
  * [27] S. Berry, S. Fazzio, Y. Zhou, B. Scott, and L. Francisco-Revilla, “Netflix recommendations for groups,” _Proceedings of the American Society for Information Science and Technology_ , vol. 47, no. 1, pp. 1–3, 2010.
  * [28] J. F. McCarthy and T. D. Anagnost, “Musicfx: an arbiter of group preferences for computer supported collaborative workouts,” in _Proceedings of the 1998 ACM conference on Computer supported cooperative work_. ACM, 1998, pp. 363–372.
  * [29] A. Jameson and B. Smyth, “Recommendation to groups,” in _The adaptive web_. Springer, 2007, pp. 596–627.
  * [30] D. Li, C. Chen, Q. Lv, J. Yan, L. Shang, and S. Chu, “Low-rank matrix approximation with stability,” in _Proc. of_ , vol. 951, 2016, p. 16.
  * [31] C. Chen, D. Li, Q. Lv, J. Yan, S. M. Chu, and L. Shang, “Mpma: Mixture probabilistic matrix approximation for collaborative filtering.” in _IJCAI_ , 2016.
  * [32] C. Chen, D. Li, Q. Lv, J. Yan, and S. Chu, “Gloma: Embedding global information in local matrix approximation models for collaborative filtering.” in _AAAI_ , 2017.
  * [33] C. Basu, H. Hirsh, W. Cohen _et al._ , “Recommendation as classification: Using social and content-based information in recommendation,” in _Aaai/iaai_ , 1998, pp. 714–720.