# Privileged Features Distillation at Taobao Recommendations

Chen Xu*, Quan Li*, Junfeng Ge, Jinyang Gao, Xiaoyong Yang, Changhua Pei, Fei
Sun, Jian Wu, Hanxiao Sun, and Wenwu Ou  Alibaba Group, Beijing, China
chaos.xc, liquan.quanli, beili.gjf, jinyang.gjy, xiaoyong.yxy, changhua.pch,
<EMAIL_ADDRESS>
(2018)

###### Abstract.

Features play an important role in the prediction tasks of e-commerce
recommendations. To guarantee the consistency of off-line training and on-line
serving, we usually utilize the same features that are both available.
However, the consistency in turn neglects some discriminative features. For
example, when estimating the conversion rate (CVR), i.e., the probability that
a user would purchase the item if she clicked it, features like dwell time on
the item detailed page are informative. However, CVR prediction should be
conducted for on-line ranking before the click happens. Thus we cannot get
such post-event features during serving.

We define the features that are discriminative but only available during
training as the privileged features. Inspired by the distillation techniques
which bridge the gap between training and inference, in this work, we propose
privileged features distillation (PFD). We train two models, i.e., a student
model that is the same as the original one and a teacher model that
additionally utilizes the privileged features. Knowledge distilled from the
more accurate teacher is transferred to the student, which helps to improve
its prediction accuracy. During serving, only the student part is extracted
and it relies on no privileged features. We conduct experiments on two
fundamental prediction tasks at Taobao recommendations, i.e., click-through
rate (CTR) at coarse-grained ranking and CVR at fine-grained ranking. By
distilling the interacted features that are prohibited during serving for CTR
and the post-event features for CVR, we achieve significant improvements over
their strong baselines. During the on-line A/B tests, the click metric is
improved by $\mathbf{+5.0\%}$ in the CTR task. And the conversion metric is
improved by $\mathbf{+2.3\%}$ in the CVR task. Besides, by addressing several
issues of training PFD, we obtain comparable training speed as the baselines
without any distillation.

Privileged Features, Distillation, E-commerce Recommendations, CTR, CVR

*Both authors contributed equally to this work

††journalyear: 2018††copyright: acmlicensed††conference: Woodstock ’18: ACM
Symposium on Neural Gaze Detection; June 03–05, 2018; Woodstock,
NY††booktitle: Woodstock ’18: ACM Symposium on Neural Gaze Detection, June
03–05, 2018, Woodstock, NY††price: 15.00††doi: 10.1145/1122445.1122456††isbn:
978-1-4503-9999-9/18/06††ccs: Information systems Information retrieval††ccs:
Computing methodologies Neural networks

##  1\. Introduction

In recent years, deep neural networks (DNNs) (Cheng et al., 2016; Covington et
al., 2016; Guo et al., 2017; Lian et al., 2018; Zhou et al., 2018b; Ni et al.,
2018) have achieved very promising results in the prediction tasks of
recommendations. However, most of these works focus on the model aspect. While
there are limited works except (Cheng et al., 2016; Covington et al., 2016)
paid attention to the feature aspect in the input, which essentially determine
the upper-bound of the model performance. In this work, we also focus on the
feature aspect, especially the features in e-commerce recommendations.

To ensure the consistency of off-line training and on-line serving, we usually
use the same features that are both available in the two environments in real
applications. However, a bunch of discriminative features, which are only
available at training time, are thus ignored. Taking conversion rate (CVR)
prediction in e-commerce recommendations as an example, here we aim to
estimate the probability that the user would purchase the item if she clicked
it. Features describing user behaviors in the clicked detail page, e.g., the
dwell time on the whole page, can be rather helpful. However, these features
cannot be utilized for on-line CVR prediction in recommendations, because it
has to be done before any click happens. Although such post-event features can
indeed be recorded for off-line training. In consistent with the learning
using privileged information (Vapnik and Izmailov, 2015; Vapnik and Vashist,
2009), here we define the features that are discriminative for prediction
tasks but only available at training time, as the privileged features.

A straightforward way to utilize the privileged features is multi-task
learning (Ruder, 2017), i.e., predicting each feature with an additional task.
However, in the multi-task learning, each task does not necessarily satisfy a
no-harm guarantee (i.e., privileged features can harm the learning of the
original model). More importantly, the no-harm guarantee will very likely be
violated since estimating the privileged features might be even more
challenging than the original problem (Lambert et al., 2018). From the
practical point of view, when using dozens of privileged features at once, it
would be a challenge to tune all the tasks.

Figure 1. Illustration of model distillation (MD) (Hinton et al., 2015) and
privileged features distillation (PFD) proposed in this work. In MD, the
knowledge is distilled from the more complex model. While in PFD, the
knowledge is distilled from both the privileged and the regular features. PFD
also differs from the original learning using privileged information (LUPI)
(Lopez-Paz et al., 2016), where the teacher only processes the privileged
features.

Inspired by learning using privileged information (LUPI) (Lopez-Paz et al.,
2016), here we propose privileged features distillation (PFD) to take
advantage of such features. We train two models, i.e., a student and a teacher
model. The student model is the same as the original one, which processes the
features that are both available for off-line training and on-line serving.
The teacher model processes all features, which include the privileged ones.
Knowledge distilled from the teacher, i.e., the soft labels in this work, is
then used to supervise the training of the student in addition to the original
hard labels, i.e., $\\{0,1\\}$, which additionally improves its performance.
During on-line serving, only the student part is extracted, which relies on no
privileged features as the input and guarantees the consistency with training.
Compared with MTL, PFD mainly has two advantages. On the one hand, the
privileged features are combined in a more appropriate way for the prediction
task. Generally, adding more privileged features will lead to more accurate
predictions. On the other hand, PFD only introduces one extra distillation
loss no matter what the number of privileged features is, which is much easier
to balance.

PFD is different from the commonly used model distillation (MD) (Hinton et
al., 2015; Buciluǎ et al., 2006). In MD, both the teacher and the student
process the same inputs. And the teacher uses models with more capacity than
the student. For example, the teachers can use deeper networks to instruct the
shallower students (Zhou et al., 2018a; Kim and Rush, 2016). Whereas in PFD,
the teacher and the student use the same models but differ in the inputs. PFD
is also different from the original LUPI (Lopez-Paz et al., 2016), where the
teacher network in PFD additionally processes the regular features. Figure 1
gives an illustration on the differences.

In this work, we apply PFD to Taobao recommendations. We conduct experiments
on two fundamental prediction tasks by utilizing the corresponding privileged
features. The contributions of this paper are four-fold:

  * •

We identify the privileged features existing at Taobao recommendations and
propose PFD to leverage them. Compared with MTL to predict each privileged
feature independently, PFD unifies all of them and provides a one-stop
solution.

  * •

Different from the traditional LUPI, the teacher of PFD additionally utilizes
the regular features, which instructs the student much better. PFD is in
complementary to MD. By combining both of them, i.e., PFD+MD, we can achieve
further improvements.

  * •

We train the teacher and the student synchronously by sharing common input
components. Compared to training them asynchronously with independent
components as done in tradition, such training manner achieves even or better
performance, while the cost of time is much reduced. Thus the technique is
adoptable in online learning, where real-time computation is in high demand.

  * •

We conduct experiments on two fundamental prediction tasks at Taobao
recommendations, i.e., CTR prediction at coarse-grained ranking and CVR
prediction at fine-grained ranking. By distilling the interacted features that
are prohibited due to efficiency requirement for CTR at coarse-grained ranking
and the post-event features for CVR as introduced above, we achieve
significant improvements over their strong baselines. During the on-line A/B
tests, the click metric is improved by $\mathbf{+5.0\%}$ in the CTR task. And
the conversion metric is improved by $\mathbf{+2.3\%}$ in the CVR task.

##  2\. Related Distillation Techniques

Before giving detailed description of our PFD, we will firstly introduce the
distillation techniques (Hinton et al., 2015; Buciluǎ et al., 2006). Overall,
the techniques are aiming to help the non-convex student models to train
better. For model distillation, we can typically write the objective function
as follows:

(1)
$\min_{\bm{W}_{s}}~{}~{}(1-\lambda)*L_{s}\left(\bm{y},f_{s}(\bm{X};\bm{W}_{s})\right)+\lambda*L_{d}\left(f_{t}(\bm{X};\bm{W}_{t}),f_{s}(\bm{X};\bm{W}_{s})\right),$

where $f_{t}$ and $f_{s}$ are the teacher model and the student model,
respectively. $L_{s}$ denotes the student pure loss with the known hard labels
$\bm{y}$ and $L_{d}$ denotes its loss with the soft labels produced by the
teacher. $\lambda\in[0,1]$ is the hyper-parameter to balance the two losses.
Compared with the original function that minimizes $L_{s}$ alone, we are
expecting that the additional loss $L_{d}$ in Eq.(1) will help to train
$\bm{W}_{s}$ better by distilling the knowledge from the teacher. In the work
of (Pereyra et al., 2017), Pereyra et. al. regard the distillation loss as
regularization on the student model. When training $f_{s}$ alone by minimizing
$L_{s}$, it is prone to get overconfident predictions, which overfit the
training set (Szegedy et al., 2016). By adding the distillation loss, $f_{s}$
will also approximate the soft predictions from $f_{t}$. By softening the
outputs, $f_{s}$ is more likely to achieve better generalization performance.

Typically, the teacher model is more powerful than the student model. Teachers
can be the ensembles of several models (Buciluǎ et al., 2006; Hinton et al.,
2015; Zhang et al., 2018), or DNNs with more neurons (Tang and Wang, 2018),
more layers (Zhou et al., 2018a; Kim and Rush, 2016), or even broader
numerical precisions (Mishra and Marr, 2018) than students. There are also
some exceptions, e.g., in the work of (Anil et al., 2018), both of the two
models are using the same structure and learned from each other, with
difference only in the initialization and the orders to process the training
data.

As indicated in Eq.(1), the parameter $\bm{W}_{t}$ of the teacher is fixed
across the minimization. We can generally divide the distillation technique
into two steps: firstly train the teacher with the known labels $\bm{y}$, then
train the student by minimizing Eq.(1). In some applications, the models could
take rather long time to converge, thus it is impractical to wait for the
teacher to be ready as Eq.(1). Instead, some works try to train the teacher
and the student synchronously (Anil et al., 2018; Zhou et al., 2018a; Zhang et
al., 2018). Besides distilling from the final output as Eq.(1), it is possible
to distill from the middle layer, e.g., Romero et al. (Romero et al., 2015)
try to distill the intermediate feature maps, which help to train a deeper and
thinner network.

In addition to distilling knowledge from more complex models, Lopez-Paz et al.
(Lopez-Paz et al., 2016) propose to distill knowledge from privileged
information $\bm{X}^{*}$, which is also known as learning using privileged
information (LUPI). The loss function then becomes:

(2)
$\min_{\bm{W}_{s}}~{}~{}(1-\lambda)*L_{s}\left(\bm{y},f(\bm{X};\bm{W}_{s})\right)+\lambda*L_{d}\left(f(\bm{X}^{*};\bm{W}_{t}),f(\bm{X};\bm{W}_{s})\right).$

In the work of (Wang et al., 2018), Wang et al. apply LUPI to image tag
recommendation. Besides the teacher and the student, they additionally learn a
discriminator, which ensures the student to learn the true data distribution
at the equilibrium faster. Chen et al. apply LUPI to review-based
recommendation. They also utilize adversarial training to select informative
reviews. Although achieving better performance, both of the works are only
validated on relatively small datasets. It still remains unknown whether these
techniques can reach equilibrium of the min-max game in industry-scale
datasets.

Figure 2. Overview of Taobao recommendations. We adopt a cascaded learning
framework to select/rank items. At coarse-grained ranking, the interacted
features, although being discriminative, are prohibited as they greatly
increase the latency at serving. Some representative features are illustrated
in the lower part.

##  3\. Privileged Features at Taobao Recommendations

To better understand the privileged features exploited in this work, we
firstly give an overview of Taobao recommendations in Figure 2. As usually
done in industry recommendations (Covington et al., 2016; Liu et al., 2017),
we adopt the cascaded learning framework. There are overall three stages to
select/rank the items before presenting to the user, i.e., candidate
generation, coarse-grained ranking, and fine-grained ranking. To make a trade-
off between efficiency and accuracy, more complex and effective model is
adopted as the cascaded stage goes forward, with the expense of higher latency
to scoring the items. In the candidate generation stage, we choose around
$10^{5}$ items that are most likely to be clicked or purchased by a user from
the huge-scale corpus. Generally, the candidate generation is mixed from
several sources, e.g., collaborative filtering (Deshpande and Karypis, 2004),
the DNN models (Covington et al., 2016), etc. After the candidate generation,
we adopt two stages for ranking, where PFD is applied in this work.

In the coarse-grained ranking stage, we are mainly to estimate the CTRs of all
items selected by the candidate generation stage, which are then used to
select the top-$k$ highest ranked items for the next stage. The inputs of the
prediction model mainly consist of three parts. The first part consists of the
user behavior, which records the history of her clicked/purchased items. As
the user behavior is in sequential, RNNs (Hochreiter and Schmidhuber, 1997;
Hidasi et al., 2016) or self-attention(Vaswani et al., 2017; Kang and McAuley,
2018) is usually adopted to model the user’s long short-term interests. The
second part consists of the user features, e.g., user id, age, gender, etc.
and the third part consists of the item features, e.g., item id, category,
brand, etc. Across this work, all features are transformed into categorical
type and we learn an embedding for each one111Numerical features are
discretized with pre-defined boundaries..

In the coarse-grained ranking stage, the complexity of the prediction model is
strictly restricted, in order to grade tens of thousands of candidates in
milliseconds. Here we utilize the inner product model (Huang et al., 2013) to
measure the item scores:

(3)
$f\left(\bm{X}^{u},\bm{X}^{i};\bm{W}^{u},\bm{W}^{i}\right)\triangleq\left<\mathbf{\Phi}_{{\bm{W}}^{u}}\left(\bm{X}^{u}\right),\mathbf{\Phi}_{\bm{W}^{i}}\left(\bm{X}^{i}\right)\right>,$

where the superscript $u$ and $i$ denote the user and item, respectively.
$\bm{X}^{u}$ denotes a combination of user behavior and user features.
$\mathbf{\Phi}_{\bm{W}}(\cdot)$ represents the non-linear mapping with learned
parameter $\bm{W}$. $\langle\cdot,\cdot\rangle$ is the inner product
operation. As the user side and the item side are separated in Eq.(3), during
serving, we can compute the mappings
$\mathbf{\Phi}_{\bm{W}^{i}}\left(\cdot\right)$ of all items off-line in
advance222In order to capture the real-time user preference, the user mappings
cannot be stored.. When a request comes, we only need to execute one forward
pass to get the user mapping $\Phi_{\bm{W}^{u}}\left(\bm{X}^{u}\right)$ and
compute its inner product with all candidates, which is extremely efficient.
For more details, see the illustration in Figure 4.

As shown in Figure 2, the coarse-grained ranking does not utilize any
interacted features, e.g., clicks of the user in the item category during the
last $24$ hours, clicks of the user in the item shop during the last $24$
hours, etc. As verified by the experiments below, adding these features can
largely enhance the prediction performance. However, it in turn greatly
increases the latency during serving, since the interacted features are
depending on the user and the specific item. In other words, the features vary
with different items or users. If putting them either at the item or the user
side of Eq.(3), the inference of the mappings
$\Phi_{\bm{W}}\left(\cdot\right)$ need to be executed as many times as the
number of candidates, i.e., $10^{5}$ here. Generally, the non-linear mapping
$\Phi_{\bm{W}}\left(\cdot\right)$ costs several orders more computation than
the simple inner product operation. It is thus unpractical to use the
interacted features during serving. Here we regard them as the privileged
features for CTR prediction at coarse-grained ranking.

Figure 3. Illustrative features describing the user behavior in the detailed
page of the clicked item. Including the dwell time that is not shown, these
features are rather informative for CVR prediction. However, during serving,
we need to use CVR to rank all candidate items as shown in left sub-figure
before any item being clicked. We thus denote these features as the privileged
features for CVR prediction.

In the fine-grained ranking stage, besides estimating the CTR as done in the
coarse-grained ranking, we need to estimate the CVR for all candidates, i.e.,
the probability that the user would purchase the item if she clicked it. In
e-commerce recommendations, the main aim is to maximize the Gross Merchandise
Volume (GMV), which can be decomposed into CTR $\times$ CVR $\times$ Price.
Once estimating the CTR and CVR for all items, we can rank them by the
expected GMVs to maximize them. Under the definition of CVR, it is obvious
that user behaviors on the detailed page of the clicked item, e.g., dwell
time, whether viewing the comment or not, whether communicating with the
seller or not, etc., can be rather helpful for the prediction. However, CVR
needs to be estimated for ranking before any future click happens. Features
describing user behaviors on the detailed page are not available during
inference. Here we thus denote these features as the privileged features for
CVR prediction. To better understand that, we give an illustration in Figure
3.

##  4\. Privileged Feature Distillation

Algorithm 1 Minimizing the student, distillation, and teacher loss in Eq.(5)
synchronously with SGD.

0: Hyper-parameter $\lambda$, swapping step $k$, and learning rate $\eta$

1: Initialize $(\bm{W}_{s},\bm{W}_{t})$ and let $i=0$.

2: while not converged do

3: Get training data $\left(\bm{y},\bm{X},\bm{X}^{*}\right).$

4: if $i<k$ then

5: $\bm{W}_{s}=\bm{W}_{s}-\eta\nabla_{\bm{W}_{s}}L_{s}.$

6: else

7:
$\bm{W}_{s}=\bm{W}_{s}-\eta\nabla_{\bm{W}_{s}}\\{(1-\lambda)*L_{s}+\lambda*L_{d}\\}.$

8: end if$\bm{W}_{t}=\bm{W}_{t}-\eta\nabla_{\bm{W}_{t}}L_{t}.$ //No
distillation loss $L_{d}$

9: Update $i=i+1$

10: end while

10: $(\bm{W}_{s},\bm{W}_{t})$

In the original LUPI as Eq.(2), the teacher relies on the privileged
information $\bm{X}^{*}$. Although being informative, the privileged features
in this work only partially describe the user’s preference. The performance of
which using these features can even be inferior to that of which using regular
features. Besides, the predictions based on privileged features can sometimes
be misleading. For example, it generally takes more time for customers to
decide on expensive items, while the conversion rate of these items is rather
low. When conducting CVR estimation, the teacher of LUPI makes predictions
relying on the privileged features, e.g., dwell-time, but not considering the
regular features, e.g., item price, which may result in false positive
predictions on the expensive items. To alleviate that, we additionally feed
the regular features to the teacher model. The original function of Eq.(2) is
thus modified as follows:

(4)

$\displaystyle\min_{\bm{W}_{s}}~{}~{}(1-\lambda)*L_{s}\left(\bm{y},f(\bm{X};\bm{W}_{s})\right)+\lambda*L_{d}\left(f(\bm{X},\bm{X}^{*};\bm{W}_{t}),f(\bm{X};\bm{W}_{s})\right).$

Generally, adding more information, e.g., more features, will lead to more
accurate predictions. The teacher $f(\bm{X},\bm{X}^{*};\bm{W}_{t})$ here is
thus expected to be stronger than the student $f(\bm{X};\bm{W}_{s})$ or the
teacher of LUPI $f(\bm{X}^{*};\bm{W}_{t})$. In the above scenario, by taking
both the privileged and the regular features into consideration, the dwell
time feature instead can be used to distinguish the extent of preference on
different expensive items. The teacher is thus more knowledgeable to instruct
the student rather than mislead it. As verified by the experiments below,
adding regular features to the teacher is non-trivial and it greatly improves
the performance of LUPI. Thereafter we denote this technique as PFD to
distinguish it from LUPI.

As Eq.(4) indicates, the teacher $f(\bm{X},\bm{X}^{*};\bm{W}_{t})$ is trained
in advance. However, it takes a long time to train the teacher model alone in
our applications. It is thus quite unpractical to apply distillation as
Eq.(4). A more plausible way is to train the teacher and the student
synchronously as done in (Anil et al., 2018; Zhou et al., 2018a; Zhang et al.,
2018). The objective function is then modified as follows:

(5)

$\displaystyle\begin{split}\min_{\bm{W}_{s},\bm{W}_{t}}~{}~{}&(1-\lambda)*L_{s}\left(\bm{y},f(\bm{X};\bm{W}_{s})\right)+\lambda*L_{d}\left(f(\bm{X},\bm{X}^{*};\bm{W}_{t}),f(\bm{X};\bm{W}_{s})\right)\\\
&+L_{t}\left(\bm{y},f(\bm{X},\bm{X}^{*};\bm{W}_{t})\right).\end{split}$

Figure 4. Illustration of training the inner-product model with PFD+MD and (b)
its deployment during serving. At the training time, the privileged features,
i.e., interaction $\bm{X}^{*}$ between $\bm{X}^{u}$ and $\bm{X}^{i}$, and the
more complex DNN model together form a strong teacher to instruct the student.
During serving, we compute the mappings
$\mathbf{\Phi}_{\bm{W}^{i}}\left(\cdot\right)$ of all items off-line in
advance. When a request comes, we only need to execute one forward pass to
derive the user mapping $\mathbf{\Phi}_{\bm{W}^{u}}\left(\bm{X}^{u}\right)$.

Although saving time, synchronous training can be un-stable. In the early
stage when the teacher is not well-trained, the distillation loss $L_{d}$ may
distract the student and slow the training down. Here we alleviate it by
adopting a warm up scheme. We set $\lambda$ of Eq.(5) to $0$ in the early
stage and fix it to the pre-defined value thereafter, with the swapping step
being a hyper-parameter. In our huge scale datasets, we find that such simple
scheme works well. Different from mutual learning (Zhang et al., 2018), we
only enable the student to learn from the teacher here. Otherwise the teacher
will co-adapt with the student, which degenerates its performance. When
computing the gradient with respect to the teacher parameters $\bm{W}_{t}$, we
thus omit the distillation loss $L_{d}$. The update with SGD is illustrated in
Algorithm 1.

Across this work, all models are trained in the parameter sever systems (Dean
et al., 2012), where all parameters are stored in the servers and most
computation is executed in the workers. The training speed is mainly depending
on the computation load in the workers and the communication volume between
the workers and the servers. As indicated in Eq.(5), we train the teacher and
the student together. The number of parameters and the computation are roughly
doubled. Training using PFD can thus be much slower than training on the
student alone, which is unpractical in industry. Especially for on-line
learning where real-time computation is in high demand, adopting distillation
can add much burden. Here we alleviate that by sharing all common input
components in the teacher and the student. Since the embeddings of all
features occupy most of the storage in the severs333For the student model
alone, all embeddings take up to $150$ Gigabytes., the communication volume is
almost halved by sharing. The computation can also be reduced by sharing the
components of processing the user clicked/purchased behavior, which is known
to be costly. As verified by the experiments below, we can achieve even or
better performance by sharing. Besides, we increase only a little extra time
compared to training the student alone, which makes PFD adoptable for online
learning.

Extension: PFD+MD. As illustrated in Figure 1, PFD distills knowledge from the
privileged features. In comparison, MD distills knowledge from the more
complex teacher model. The two distillation techniques are complementary. A
natural extension is to combine them by forming a more accurate teacher to
instruct the student.

In the CTR prediction at coarse-grained ranking, as Eq.(3) shows, we use the
inner product model to increase the efficiency during serving. In fact, the
inner product model can be regarded as the generalized matrix factorization
(Covington et al., 2016). Although we are using non-linear mapping
$\mathbf{\Phi}_{\bm{W}}(\cdot)$ to transform the user and item inputs, the
model capacity is intrinsically limited by the bi-linear structure at the
inner product operation. DNNs, with the capacity to approximate any function
(Cybenko, 1989; Hornik, 1991), are considered as a substitution for the inner
product model in the teacher. In fact, as proved in Theorem 1 of (Lin et al.,
2017), the product operation can be approximated arbitrarily well by a two-
layers neural network with only $4$ neurons in the hidden layer. Thus the
performance of using DNN is supposed to be lower-bounded by that of using the
inner-product model.

Figure 5. Illustration of input components used for CTR at coarse-grained
ranking and CVR at fine-grained ranking. We adopt the multi-head self-
attention structure (Vaswani et al., 2017) to model the user clicked/purchased
history. The common input components between the teacher and the student are
shared during training.

In PFD+MD, we thus adopt the DNN model as the teacher network. In fact, the
teacher model here is the same as the model used for CTR prediction at fine-
grained ranking. PFD+MD in this task can be regarded as distilling knowledge
from the fine-grained ranking to improve the coarse-grained ranking. For
better illustration, we give the whole framework in Figure 4. During serving,
we extract the student part only, which relies on no privileged features. As
the mappings $\mathbf{\Phi}_{\bm{W}^{i}}\left(\bm{X}^{i}\right)$ of all items
are independent of the users, we compute them off-line in advance. When a
request comes, the user mapping
$\mathbf{\Phi}_{\bm{W}^{u}}\left(\bm{X}^{u}\right)$ is firstly computed. After
that, we compute its inner-product with the mappings of all items produced
from the candidate generation stage. The top-$k$ highest scored items are then
chosen and fed to the fine-grained ranking. On the whole, we only execute one
forward pass to derive the user mapping and conduct efficient inner product
operations between the user and all candidates, which are rather friendly in
the aspect of computation.

##  5\. Experiments

In this section, we conduct experiments at Taobao recommendations with the aim
of answering the following research questions:

  * •

RQ1: What is the performance of PFD on the tasks of CTR at coarse-grained
ranking and CVR at fine-grained ranking?

  * •

RQ2: Compared to individual PFD, can we achieve additional improvements by
combining PFD with MD?

  * •

RQ3: Is PFD sensitive to the hype-parameter $\lambda$ in Eq.(5)?

  * •

RQ4: What is the effect of training the teacher and student synchronously by
sharing common input components?

###  5.1. Experimental Settings

To better understand the network structure, we give an illustration of all
input components used in this work in Figure 5. As mentioned earlier, all
features are transformed into categorical type and we learn an embedding for
each one. The entities, e.g., user and item, are then represented by the
concatenations of their corresponding feature embeddings. Here we adopt the
self-attention structure (Vaswani et al., 2017) to model the user
clicked/purchased history. Let $\mathbf{V}\in\mathbb{R}^{n\times k}$ denote
the input values of the sequence, with $n$ being the combined embedding
dimension and $k$ being the sequence length. The transformed values
$\mathbf{\tilde{V}}$ are then derived as weighted combinations of input
$\mathbf{V}$, i.e.,
$\mathbf{\tilde{V}}=\text{Softmax}\left(\mathbf{Q}^{T}\mathbf{K}\right)\mathbf{V}$,
where the queries $\mathbf{Q}$ and the keys $\mathbf{K}$ are the same as the
values $\mathbf{V}$. To improve the effective resolution of combinations, the
multi-head structure is adopted, where $\mathbf{Q}$, $\mathbf{K}$, and
$\mathbf{V}$ are projected linearly into several subspaces and the
transformations are executed, respectively. Note that we also add feed-forward
network, skip connections (He et al., 2016), layer normalization (Ba et al.,
2016) to the attention mechanism as done in the original work (Vaswani et al.,
2017). For simplicity, we omit the details here. The only difference with the
original self-attention is that we do not adopt position encodings in the
input. We instead insert several extra features describing the user behavior
on the particular item, e.g., the clicked/purchased time from now, the dwell
time on the item, etc. Besides encoding the relative positions of the items,
the extra features also reflect the importance of the items for future
predictions. According to our experiments, adding these features can greatly
improve the performance.

After deriving $\mathbf{\tilde{V}}$ with one-layer’s self-attention, we adopt
the mean pooling operation over the sequence length $k$. Here we set $k$ to
$50$, the number of heads to $4$, and the subspace dimension to $32$. As
illustrated in Figure 5, the input components are fed to the corresponding
teacher and student networks, which consists of several fully-connected
layers. We use LeakyReLU (Maas et al., 2013) as the activation and insert
batch normalization (Ioffe and Szegedy, 2015) before it. The models are
trained in the parameter servers with the asynchronous Adagrad optimizer
(Duchi et al., 2011). In the first one million steps, the learning rate is
increased linearly to the pre-defined value $0.01$, which is then kept fixed.
We set the batch size to $1024$ and the number of epoch to $1$. We train the
teacher and student synchronously by sharing common input components. Unless
stated otherwise, $\lambda$ is set to $0.5$ and the swapping step $k$ in
Algorithm 1 is set to $10^{6}$.

As the labels are in $1$ or $0$, i.e., whether the users clicked/purchased the
item or not, we use the log-loss for both the teacher and the student, i.e.,

(6)

$\displaystyle
L_{t/s}\triangleq\frac{1}{N}\sum_{i=1}^{N}\left(y_{i}\mbox{log}\left(1+e^{-f_{t/s,i}}\right)+(1-y_{i})\mbox{log}\left(1+e^{f_{t/s,i}}\right)\right),$

where $f_{t/s,i}$ denotes the output of the $i$-th sample from the teacher or
student model. For the distillation loss $L_{d}$, we use the cross entropy,
i.e., by replacing $y_{i}$ in the above equation with $1/(1+e^{-f_{t,i}})$.
Here we measure the performance of models with the widely-used areas under the
curve (AUC) in the next-day held-out data.

###  5.2. CTR at Coarse-grained Ranking (RQ1-2)

Table 1. CTR dataset description at coarse-grained ranking of Taobao recommendations. Datasets |  $\\#$ Users |  $\\#$Items |  $\\#$Clicks |  $\\#$ Impressions
---|---|---|---|---
$1$ Day | $9.35\times 10^{7}$ | $2.67\times 10^{7}$ | $5.03\times 10^{8}$ | $1.09\times 10^{10}$
$10$ Days | $2.88\times 10^{8}$ | $4.45\times 10^{7}$ | $4.57\times 10^{9}$ | $9.90\times 10^{10}$
Table 2. Testing AUC of different methods for CTR at coarse-grained ranking. We do not include MTL as it is too cumbersome to predict dozens of privileged features. Due to the huge training cost, we only compare PFD+MD with the baseline in the dataset of 10 days.  Methods | Dataset of $1$ Day | Dataset of $10$ Days
---|---|---
Student | Teacher | Student | Teacher
Baseline | $0.6625$ | $-$ | $0.7042$ | $-$
LUPI (Lopez-Paz et al., 2016) | $0.6637$ | $0.6687$ | $-$ | $-$
MD (Hinton et al., 2015) | $0.6704$ | $0.6892$ | $-$ | $-$
PFD | $0.6712$ | $0.6921$ | $-$ | $-$
PFD+MD | $\mathbf{0.6745}$ | $0.7110$ | $\mathbf{0.7160}$ | $0.7411$

Across this work, we conduct experiments using the traffic logs of the Guess
You Like scenario in the front page of Taobao app. The dataset description of
CTR at coarse-grained ranking is summarized in Table 1. For the inner-product
model, the user mapping $\mathbf{\Phi}_{\bm{W}^{u}}\left(\cdot\right)$ and the
item mapping $\mathbf{\Phi}_{\bm{W}^{i}}\left(\cdot\right)$ in Eq.(3) are
formulated as: Input-¿$512$-¿ Act.-¿$256$-¿Act.-¿$128$-¿$\ell_{2}$-normalize.
When executing inner-product between
$\mathbf{\Phi}_{\bm{W}^{u}}\left(\cdot\right)$ and
$\mathbf{\Phi}_{\bm{W}^{i}}\left(\cdot\right)$, we additionally multiply it
with a scalar, i.e., $5$, in compensation for the shrinkage of value after
normalization. In LUPI (Lopez-Paz et al., 2016), MD (Hinton et al., 2015), and
PFD+MD, the teacher networks use $3$-layers’ MLP, with the number of hidden
neurons being $512$, $256$, and $128$, respectively. In PFD, we use the inner-
product model for both the teacher and the student, where the privileged
features are put at the user side of the teacher. The teacher and the student
of PFD (and PFD+MD) share all common input components except user id. We do
not include MTL as it is too cumbersome to predict dozens of privileged
features.

Off-line and on-line performance. The testing AUC of all compared distillation
techniques are shown in the left part of Table 2. Compared the teacher of PFD
with the baseline, we confirm the effectiveness of the interacted features. By
distilling knowledge from these features, we improve the testing AUC from
$0.6625$ to $0.6712$. Although both utilizing the same privileged features,
PFD is superior to LUPI. This is because that the teacher of PFD additionally
processes the regular features, which results in more accurate predictions to
instruct the student than using privileged features alone as done in LUPI
($0.6921$ v.s. $0.6687$). Similarly, we achieve further improvements with
PFD+MD by forming a more accurate teacher. In order to validate whether the
superiority of PFD+MD could still holds when training longer with more data.
We conduct experiments on the traffic logs of $10$ days. Due to the huge
training cost, we only compare PFD+MD with the baseline. As shown in the right
part of Table 2, the student of PFD+MD surpasses the baseline AUC with a
margin, i.e., $+0.0118$. We also conduct on-line A/B tests to validate its
effectiveness. Compared with the baseline, PFD+MD consistently improves the
click metric by $\mathbf{+5.0\%}$. And we have fully deployed the technique
into production.

Cost of directly utilizing interacted features. As discussed earlier, the
interacted features are prohibited for the inner-product model during
inference. With such features, we need to compute the mappings
$\Phi_{\bm{W}}\left(\cdot\right)$ as many times as the number of candidates,
i.e., $10^{5}$. In contrast, without such features, we only need to compute
the mapping once and execute its inner product with all candidates
efficiently. Suppose that the input to
$\mathbf{\Phi}_{\bm{W}}\left(\cdot\right)$ has $1024$ dimensions.
Theoretically, we need $1024\times 512+512\times 256+256\times 128\approx
6.9\times 10^{5}$ fused multiply-add flops to get one mapping. In comparison,
executing one inner product operation needs $128$ flops, which is $\sim
5400\times$ less. We also conduct simulated experiments in the personal
computer. We repeat $10^{5}$ times to simulate the mapping inference and the
inner product operation, which totally costs $89.695$s and $0.108$s,
respectively. Executing the mapping is $\sim 830\times$ slower than the inner
product operation.

###  5.3. CVR at fine-grained ranking (RQ1-2)

Table 3. CVR dataset description at fine-grained ranking of Taobao recommendations Datasets |  $\\#$Users |  $\\#$Items |  $\\#$Purchases |  $\\#$Clicks
---|---|---|---|---
$30$ Days | $2.78\times 10^{8}$ | $3.74\times 10^{7}$ | $6.97\times 10^{7}$ | $1.40\times 10^{10}$
$60$ Days | $3.36\times 10^{8}$ | $5.27\times 10^{7}$ | $1.32\times 10^{8}$ | $2.71\times 10^{10}$
Table 4. Testing AUC of different methods for CVR prediction at fine-grained ranking. Methods | Dataset of $30$days | Dataset of $60$days
---|---|---
Student | Teacher | Student | Teacher
Baseline | $0.9040$ | $-$ | $0.9082$ | $-$
MTL (Ruder, 2017) | $0.9045$ | $-$ | $0.9077$ | $-$
LUPI (Lopez-Paz et al., 2016) | $0.8965$ | $0.9651$ | $0.9003$ | $0.9659$
MD (Hinton et al., 2015) | $0.9052$ | $0.9058$ | $0.9093$ | $0.9103$
PFD | $\mathbf{0.9084}$ | $0.9901$ | $0.9135$ | $0.9923$
PFD+MD | $0.9082$ | $0.9911$ | $\mathbf{0.9138}$ | $0.9929$

In the above CTR prediction, we use the traffic logs of all impressions. While
in the CVR prediction, we extract the traffic logs of all clicks. The datasets
are summarized in Table 3. We adopt $3$-layers’ MLP for the baseline and all
of the student networks, with the number of hidden neurons being $512$, $256$,
and $128$, respectively. The teacher networks of PFD and LUPI also have the
same structure as their students. In MD and PFD+MD, their teacher networks are
expanded to $7$-layers’ MLP, with the number of hidden neurons being $8192$,
$4096$, $2048$, $1024$, $512$, $256$, and $128$, respectively. We also compare
with MTL. Here we adopt the hard parameter sharing version of MTL (Ruder,
2017), where all tasks share first hidden layer and independently predict each
task with $2$-layers’ MLP. We adopt the mean squared error for the auxiliary
continuous regression tasks, e.g., predicting the dwell time, and the log-loss
for the auxiliary binary prediction tasks, e.g., predicting whether viewing
the user comments or not. The hyper-parameters are chosen empirically from
$\\{0.01,0.02,0.05,0.1,0.2,0.5,1,2\\}$ with the principle of keeping all
auxiliary losses balanced.

Table 5. Students’ testing AUC of different hyper-parameters $\lambda$ in the $1$ day’s dataset of CTR. The superscript +/- indicates the highest/lowest AUC of each method among all chosen hyper-parameters. $\lambda$ | $0.1$ | $0.3$ | $0.5$ | $0.7$ | $0.9$
---|---|---|---|---|---
LUPI (Lopez-Paz et al., 2016) | $0.6648^{+}$ | $0.6640$ | $0.6637$ | $0.6631$ | $0.6624^{-}$
MD (Hinton et al., 2015) | $0.6695^{-}$ | $0.6697$ | $0.6704$ | $0.6706^{+}$ | $0.6700$
PFD | $0.6711$ | $0.6709$ | $0.6712^{+}$ | $0.6700$ | $0.6696^{-}$
PFD+MD | $0.6741$ | $0.6740$ | $0.6745$ | $0.6747^{+}$ | $0.6739^{-}$
Table 6. Students’ testing AUC of different hyper-parameters $\lambda$ in the $30$ days’ dataset of CVR. $\lambda$ | $0.1$ | $0.3$ | $0.5$ | $0.7$ | $0.9$
---|---|---|---|---|---
LUPI (Lopez-Paz et al., 2016) | $0.9024^{+}$ | $0.8998$ | $0.8965$ | $0.8876$ | $0.8613^{-}$
MD (Hinton et al., 2015) | $0.9047^{-}$ | $0.9054^{+}$ | $0.9052$ | $0.9050$ | $0.9049$
PFD | $0.9081^{-}$ | $0.9082$ | $0.9084^{+}$ | $0.9082$ | $0.9082$
PFD+MD | $0.9081$ | $0.9085^{+}$ | $0.9082$ | $0.9083$ | $0.9080^{-}$

Off-line and on-line performance. The results of all testing methods are shown
in Table 4. Among them, LUPI gets the worst performance, although the testing
AUC of its teacher is rather high. The reason of the inferiority has been
discussed in Section 4. The experiments also confirm the positive effects of
adding regular features to the teacher model, where PFD improves the baseline
by $+0.0044$ AUC in the dataset of $30$ days and $+0.0053$ AUC in the dataset
of $60$ days, respectively. Compared with PFD, PFD+MD has no distinct
superiority. This is mainly due to that the improvement of MD over the
baseline is moderately small. In practice, PFD is thus preferred over PFD+MD
as costing much less computation resources. We conduct on-line A/B tests to
validate the effectiveness of PFD. Compared with the baseline, PFD steadily
improves the conversion metric by $\mathbf{+2.3\%}$ over a long period of
time.

###  5.4. Ablation Study (RQ3-4)

Sensitivity of Hyper-parameter. In the above experiments, we fix the the
hyper-parameter $\lambda$ at $0.5$ for all distillation techniques. Here we
test the sensitivity of $\lambda$. The results of CTR prediction are shown in
Table 5. Most of the distillation techniques surpass the baseline, i.e.,
$0.6625$, among all chosen hyper-parameters, except LUPI with $\lambda=0.9$.
MD, PFD, and PFD+MD are all robust to varying $\lambda$. Even their worst
results improve the baseline by margins. We also conduct experiments in the
CVR dataset. As shown in Table 6, LUPI narrows the gap with the baseline,
i.e., $0.9040$ as $\lambda$ decreases, while it is still inferior to the
baseline. MD, PFD, and PFD+MD are again robust to varying $\lambda$ in this
huge-scale dataset.

Table 7. Effects of training PFD+MD in different manners in the $1$ day’s dataset of CTR. Ind&Async denotes that the teacher and the student are trained asynchronously with independent input components. Share&Sync denotes that the teacher and the student are trained synchronously with shared common input components. The superscript ∗ means that all common input components are shared except user id. We also record the wall-clock time in hours in the forth column. | Student | Teacher | Time | Relative
---|---|---|---|---
Baseline | $0.6625$ | $-$ |  $9.24$ h | $0\%$
Ind&Async | $0.6751$ | $0.7112$ |  $18.43$ h | $+99.5\%$
Ind&Sync | $0.6748$ | $0.7112$ |  $14.32$ h | $+55.0\%$
Share&Sync | $0.6717$ | $0.7108$ |  $9.51$ h | $+2.9\%$
Share∗&Sync | $0.6745$ | $0.7110$ |  $10.29$ h | $+11.4\%$
Table 8. Effects of training PFD+MD in different manners in the $30$ days’ dataset of CVR. The row with superscript † is the result of PFD. | Student | Teacher | Time | Relative
---|---|---|---|---
Baseline | $0.9040$ | $-$ |  $12.22$ h | $0\%$
Ind&Async | $0.9067$ | $0.9887$ |  $26.85$ h | $+119.7\%$
Ind&Sync | $0.9069$ | $0.9887$ |  $20.56$ h | $+67.4\%$
Share&Sync | $0.9082$ | $0.9911$ |  $14.97$ h | $+22.5\%$
Share&Sync† | $0.9084$ | $0.9901$ |  $12.67$ h | $+3.6\%$

Effects of Training Manner. In the above experiments, the teacher and the
student are trained synchronously by sharing common input components. Here we
test the effects of such training manner. The results of CTR prediction are
shown in Table 7. Training the teacher and the student synchronously achieves
almost the same performance as training asynchronously (Ind&Sync v.s.
Ind&Async). When training the two models by sharing all common input
components, the performance of the student degenerates. As the privileged
features, i.e., the interacted features between the user and her
clicked/purchased items, reflect the user’s personal interests, we allocate an
independent user id embedding to the student, in order to absorb the extra
preferences distilled from the privileged features. The result after such
modification is shown in the row of Share∗&Sync, where the performance
degeneration is much alleviated and only a little extra wall-clock time is
introduced. We also test the effects of different training manners in the CVR
dataset. The results are shown in Table 8. As indicated in the rows of
Share&Sync and Ind&Sync, the teacher can be additionally improved by sharing
common input components. Consequently, the student is improved by distilling
knowledge from the more accurate teacher. In the last row of Table 8, we
report the result of PFD. Compared with the baseline, it adds almost no extra
wall-clock time while achieves similar performance to PFD+MD in the row of
Share&Sync. Overall, by adopting PFD+MD for CTR and PFD for CVR, we can
achieve much better performance while add no burden to training time.
Therefore, the technique is even adoptable for online learning on the
streaming data (McMahan et al., 2013), where real-time computation is in high
demand.

##  6\. Conclusion

In this work, we identify the privileged features existing at Taobao
recommendations, i.e., the interacted features for CTR at coarse-grained
ranking and the post-event features for CVR at fine-grained ranking. And we
propose PFD to leverage them. Different from the traditional LUPI, PFD
additionally processes the regular features in the teacher, which is shown to
be the core for its success. We also propose PFD+MD to utilize the
complementary feature and model capacities to better instruct the student. And
it achieves further improvements. The effectiveness is validated on two
fundamental prediction tasks at Taobao recommendations, where the baselines
are greatly improved by the proposed distillation techniques. During the on-
line A/B tests, the click metric is improved by $\mathbf{+5.0\%}$ in the CTR
task. And the conversion metric is improved by $\mathbf{+2.3\%}$ in the CVR
task. We also address several issues of training PFD, which lead to comparable
training speed as the baselines without any distillation.

## References

  * (1)
  * Anil et al. (2018) Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E Dahl, and Geoffrey E Hinton. 2018.  Large scale distributed neural network training through online distillation. In _ICLR_.
  * Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016\.  Layer normalization.  _arXiv:1607.06450_ (2016).
  * Buciluǎ et al. (2006) Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. 2006\.  Model compression. In _Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. ACM, 535–541.
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. ACM, NY, USA, 7–10.
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016\.  Deep neural networks for youtube recommendations. In _RecSys_. ACM, 191–198.
  * Cybenko (1989) George Cybenko. 1989\.  Approximation by superpositions of a sigmoidal function.  _Mathematics of Control, Signals and Systems_ 2, 4 (1989), 303–314.
  * Dean et al. (2012) Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. 2012\.  Large scale distributed deep networks. In _Advances in Neural Information Processing Systems_. 1223–1231.
  * Deshpande and Karypis (2004) Mukund Deshpande and George Karypis. 2004.  Item-based top-n recommendation algorithms.  _ACM Transactions on Information Systems (TOIS)_ 22, 1 (2004), 143–177.
  * Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011.  Adaptive subgradient methods for online learning and stochastic optimization.  _Journal of machine learning research_ 12, Jul (2011), 2121–2159.
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: A factorization-machine based neural network for CTR prediction. In _AAAI_. AAAI Press, 1725–1731.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _CVPR_. 770–778.
  * Hidasi et al. (2016) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016.  Session-based recommendations with recurrent neural networks. In _ICLR_.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015\.  Distilling the knowledge in a neural network.  _arXiv:1503.02531_ (2015).
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Hornik (1991) Kurt Hornik. 1991\.  Approximation capabilities of multilayer feedforward networks.  _Neural Networks_ 4, 2 (1991), 251–257.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data. In _CIKM_. ACM, 2333–2338.
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift. In _ICML_. 448–456.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _ICDM_. IEEE, 197–206.
  * Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016.  Sequence-Level Knowledge Distillation. In _EMNLP_. 1317–1327.
  * Lambert et al. (2018) John Lambert, Ozan Sener, and Silvio Savarese. 2018.  Deep learning under privileged information using heteroscedastic dropout. In _CVPR_. 8886–8895.
  * Lian et al. (2018) Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. 2018\.  xDeepFM: Combining explicit and implicit feature interactions for recommender systems. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. ACM, NY, USA, 1754–1763.
  * Lin et al. (2017) Henry W Lin, Max Tegmark, and David Rolnick. 2017.  Why does deep and cheap learning work so well?  _Journal of Statistical Physics_ 168, 6 (2017), 1223–1247.
  * Liu et al. (2017) Shichen Liu, Fei Xiao, Wenwu Ou, and Luo Si. 2017\.  Cascade ranking for operational e-commerce search. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. ACM, 1557–1565.
  * Lopez-Paz et al. (2016) David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. 2016.  Unifying distillation and privileged information. In _ICLR_.
  * Maas et al. (2013) Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013\.  Rectifier nonlinearities improve neural network acoustic models. In _ICML_ , Vol. 30. 3.
  * McMahan et al. (2013) H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. 2013\.  ftrl. In _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_. 1222–1230.
  * Mishra and Marr (2018) Asit Mishra and Debbie Marr. 2018.  Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. In _ICLR_.
  * Ni et al. (2018) Yabo Ni, Dan Ou, Shichen Liu, Xiang Li, Wenwu Ou, Anxiang Zeng, and Luo Si. 2018.  Perceive your users in depth: Learning universal user representations from multiple e-commerce tasks. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. ACM, 596–605.
  * Pereyra et al. (2017) Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. 2017.  Regularizing neural networks by penalizing confident output distributions.  _arXiv:1701.06548_ (2017).
  * Romero et al. (2015) Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015.  Fitnets: Hints for thin deep nets. In _ICLR_.
  * Ruder (2017) Sebastian Ruder. 2017\.  An overview of multi-task learning in deep neural networks.  _arXiv:1706.05098_ (2017).
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016.  Rethinking the inception architecture for computer vision. In _CVPR_. 2818–2826.
  * Tang and Wang (2018) Jiaxi Tang and Ke Wang. 2018.  Ranking distillation: Learning compact ranking models with high performance for recommender system. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. ACM, 2289–2298.
  * Vapnik and Izmailov (2015) Vladimir Vapnik and Rauf Izmailov. 2015.  Learning using privileged information: similarity control and knowledge transfer.  _Journal of Machine Learning Research_ 16, 2023-2049 (2015), 2\.
  * Vapnik and Vashist (2009) Vladimir Vapnik and Akshay Vashist. 2009.  A new learning paradigm: Learning using privileged information.  _Neural Networks_ 22, 5-6 (2009), 544–557.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in Neural Information Processing Systems_. 5998–6008.
  * Wang et al. (2018) Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. 2018\.  Kdgan: Knowledge distillation with generative adversarial networks. In _Advances in Neural Information Processing Systems_. 775–786.
  * Zhang et al. (2018) Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018\.  Deep mutual learning. In _CVPR_. 4320–4328.
  * Zhou et al. (2018a) Guorui Zhou, Ying Fan, Runpeng Cui, Weijie Bian, Xiaoqiang Zhu, and Kun Gai. 2018a.  Rocket launching: A universal and efficient framework for training well-performing light net. In _AAAI_.
  * Zhou et al. (2018b) Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018b.  Deep interest network for click-through rate prediction. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. ACM, 1059–1068.


# Strings from large charged fields

Carlos A. Cardona<EMAIL_ADDRESS>IFT-UNESP. R. Dr. Bento Teobaldo Ferraz,
271 - Bloco II, São Paulo, Brasil

###### Abstract

It has been known long time ago Polchinski (1996) that the low energy dynamics
of open-strings is described by non-abelian gauge theories in the same way
that the low energy effective description of closed strings is governed by
Einstein’s gravity. In this short note, we review and comment on some examples
where conversely, an effective behavior of non-abelian gauge theories in some
particular limits are described by some type of extended objects like strings
or branes. We constrain the discussion to a few examples sharing some
similarities.

###### keywords:

Gauge Theory , String Theory , AdS/CFT Correspondence,

††journal: Nuclear Physics B

##  1 Introduction

This short note is based on a talk given in the X Latin American Symposium of
High Energy Physics held in Medellin-Colombia from 24th to 28th of November of
2014, and was intended to address a non-string theory target audience. The
main aim is to collect a few known scenarios where the physics of a particular
set of operators in a field theory, can be effectively be described by string-
like or even brane-like configurations at some particular limits.

Stringy behaviors embedded in field theory are not a new topic. A classical
example corresponds to vortex in $D=4$ Abelian-Higgs model. There we can
consider static topological solutions confined on two spatial dimension
Abrikosov (1957); Nielsen and Olesen (1973), known as Abrikosov-Nielsen-Olesen
vortex, which in three spatial dimensions looks like a rigid string. Those
solutions can also be found in supersymmetric extensions where they saturate
the BPS condition of the supersymmetric algebra and hence can be protected
from quantum corrections, allowing access to the strong coupling regime of the
supersymmetric theory. It is also possible to find an effective action
describing the dynamics of those strings.

A related example is given by flux tubes between a quark-antiquark pair in
QCD. Lately, there has been a renewed interest on QCD-strings (see for example
Aharony and Komargodski (2013); Shifman and Yung (2015)) mainly motivated for
recent improvements measuring the energies of various states of such strings
via lattice simulations (see Teper (2009) for a review and references
therein).

Another interesting embedding of strings into gauge theory observables concern
to the scattering matrix of massless particles in non-abelian gauge theory. It
was noticed by Witten (2004) that the perturbative expansion of ${\cal N}=4$
super Yang-Mills theory with U(N) gauge group is equivalent to the instanton
expansion of a topological string theory whose target space is the Calabi-Yau
supermanifold ${\mathbb{C}P}^{3|4}$. Using this string theory it was possible
to reproduce the Maximal Helicity Violating Amplitudes at tree level for
gluons in QCD. After this breakthrough and its generalization to include loops
Britto et al. (2005), a huge revolution for the computation of scattering
amplitudes in non-abelian gauge theories has started, including the discovery
(or confirmation) that a scattering amplitude of gravitons can be related to a
scattering amplitude of gluons. Moreover, the progress in this direction have
allowed to apply this techniques to the computation of realistic scattering
amplitudes to be search at the LHC (see for example van Deurzen et al.
(2014)).

Along the lines of this note we are going to comment on other stringy
appearances in field theory. A particularity of the chosen examples comes form
the similarities into the mechanism how extended objects appear, such as that
they usually appear as a collective phenomena for a large charged set of
states in the field theory as well as they are associated to non-perturbative
effects.

Despite that we are going to see different examples, we emphasize that we will
be bearing in mind the AdS/CFT correspondence Maldacena (1999) relating non-
abelian conformal field theories with gravity theories (aka string theories).
Since the original formulation of the correspondence Maldacena (1999), an
impressive amount of work have been done, including many tests of the duality
as well as the exact formulation of it for non-abelian conformal field
theories in dimensions $D=2,3,4$. It also have been a big deal of progress in
the application of the correspondence to more realistic scenarios, such as QCD
or Condense Matter Systems.

##  2 Stringy behavior of the Hubbard Model

Let us start considering a simplified model which will share some
characteristics with the gauge theories we are going to consider later on. Let
us take a system which transform like a spin-$s$ representation of SU(2) and
interact with a external magnetic field $B$ through a Zeeman-like potential
$V_{z}=B\cdot S$, being $S$ the spin-vector $S=(s_{x},s_{y},s_{z})$, Fradkin
(1991). The usual highest weight state representation is defined up to a
normalization as the set of states diagonalizing at the same time both
$S_{z},S^{2}$,

$S_{z}|s,s\rangle=s|s,s\rangle\,,\quad S^{2}|s,s\rangle=s(s+1)|s,s\rangle$ (1)

The idea is to write the evolution operator of the system in terms of the
following set of coherent states,

$|n\rangle={\rm e}^{i\theta(n_{0}\times n)\cdot S}|s,s\rangle\,,$ (2)

where $n_{0}$ is a unitary vector in the $z$ direction and $\theta$ is the
angle between $n_{0}$ and the arbitrary unitary vector $n$. The above coherent
state is an eigenvector of the spin operator $S|n\rangle=s|n\rangle$ and has
an expansion in terms of the spins-$s$ representation of SU(2) as,

$|n\rangle=\sum_{m=-s}^{s}D^{(s)}(n)_{m\,s}|s,m\rangle\,,$ (3)

the expansion coefficients satisfies the important property Perelomov (1986),

$D^{(s)}(n_{1})D^{(s)}(n_{2})=D^{(s)}(n_{3}){\rm
e}^{i\Phi(n_{1},n_{2},n_{3})S_{3}}\,,$ (4)

where $n_{i},\quad i=1,2,3$ are three arbitrary unit vectors and
$\Phi(n_{1},n_{2},n_{3})$ is the area enclosed by the spherical triangle with
vertices at $n_{i}$. The normalization of the coherent states is,

$\langle n_{1}|n_{2}\rangle={\rm
e}^{i\Phi(n_{1},n_{2},n_{3})s}\left(\frac{1+n_{1}\cdot
n_{2}}{2}\right)^{s}\,.$ (5)

The completes relation in the space of coherent states is given in this case
by,

$\mathcal{I}=\left(\frac{2s+1}{4\pi}\right)\int
d^{3}n\delta(n^{2}-1)|n\rangle\langle n|\,.$ (6)

The above relations are sufficient to write the evolution operator

$\displaystyle Z$ $\displaystyle=$ $\displaystyle{\rm Tr}{\rm
e}^{iV_{z}T}\sim$ (7) $\displaystyle\sim$
$\displaystyle\lim_{N\to\infty}\left(\prod_{j=1}^{N}\int
d\mu(n)\right)\left(\prod_{j=1}^{N}\langle n(t_{j})|{\rm e}^{-\delta
tV_{z}}|n(t_{j+1})\right)$ $\displaystyle\sim$
$\displaystyle\int\mathcal{D}n\,{\rm e}^{iS_{M}[n]}\,,$

where $d\mu(n)=\left(\frac{2s+1}{4\pi}\right)d^{3}n\,\delta(n^{2}-1)$ and the
action is defined by

$S_{M}[n]=s\,\mathcal{A}[n]+m\int_{0}^{T}dx_{0}\left(\partial_{0}n(x_{0})\right)^{2}-s\int_{0}^{T}dx_{0}B\cdot
n(x_{0})\,,$ (8)

with $m=s\delta t/4\to 0$. The $\mathcal{A}[n]$ corresponds to the area
enclosed by the closed trajectory $n(x_{0})$ on the unit-sphere surface, and
can be rewrote as a Wess-Zumino action on the two-sphere. Hence the action (8)
can be interpreted as a string on the surface of a sphere whose end point are
coupled to a $B-$field. Moreover, it scales with $s$ and hence for large
values of $s$, the action is dominated by the classical solutions of the
string over the sphere. This stringy description should be replicated in
higher dimensions as well as higher symmetrical systems.

##  3 Chern-Simons plus matter

Let us now consider a theory in three dimensions which consist of double
Chern-Simons matter theory in three dimension with level $(k,-k)$ and gauge
group $U(N)\times U(N)$. The theory becomes weakly coupled when the level $k$
is large, hence in the large $N$ limit the coupling analogous to ‘t Hooft
coupling is given by $\lambda\equiv N/k$, which is kept finite. The gauge
fields are coupled to four chiral superfields in the bifundamental
representation of the gauge group $U(N)\times U(N)$, and in the fundamental
representation of the SU(4) R-symmetry. We denote the complex scalars in these
4 chiral multiplets as ($A_{1},A_{2},\bar{B}_{1},\bar{B}_{2})$. Here
$A_{1},A_{2}$ are in the $(N,\bar{N})$ representation of $U(N)\times U(N)$,
whereas $B_{1},B_{2}$ are in the conjugate, $(\bar{N},N)$. Under the SU(4)R
R-symmetry group ($A_{1},A_{2},\bar{B}_{1},\bar{B}_{2}$) transform in the
$\mathbf{4}$ representation. There is also a $U(1)_{R}$ under which all of
$(A_{1},A_{2},\bar{B}_{1},\bar{B}_{2})$ have charge $+1$.

We would like to build a particular operator with large charge. In order an
operator to have large charge it need to be composite of a large amount of
fundamental fields, but in order to do that, we need to use as a building
block an operator in the same $U(N)$, since an index in the left $U(N)$ of the
gauge group, cannot be contracted with an index in the right $U(N)$. Without
loss of generality, let us take two particular building blocks of that type as

$Z\equiv(A_{2}B_{2})\,,\,X\equiv(A_{2}B_{1})\,,$ (9)

which have indexes only on the left $U(N)$ and let us take a particular highly
charged operator such as,

$\mathcal{O}_{J,l}=\left[ZZ\cdots ZZ\,X\,ZZ\cdots Z\right]^{m_{N}}_{p_{N}}.$
(10)

Where the index $J$ denotes the total amount of $Z-$fields in it, i.e, the
total charge of the composite operator under a diagonal action of the global
symmetry SU(4), and $l$ denotes the position of the “impurity” $X$ into the
chain of $Z$’s. Denoting by $\Delta$ the dimension of the operator (10), it
satisfies $\Delta-J=1/2$ classically, but this quantity get quantum
corrections. We are particularly interested in computing the first quantum
correction to the twist $\tau\equiv\Delta-J$ of the operator (10), but even at
first order it has a huge mixing whose diagonalization becomes a daunting
task. Even so, we would like to do it in a ‘t Hooft expansion where we need
$N$ to be large. It turns out that in that limit, only planar diagrams
contribute as the ones displayed in the figure below

therefore the operator (10) with an impurity $X$ at position $l$ only get
mixed with operators of the same type with impurities at position $l+1$ and
$l-1$. It looks very like a Heisenberg near-neighborhood spin chain, which as
is well known can be diagonalized by the following Fourier transformation,

$\mathcal{O}_{J,n}=\times\sum_{l=0}^{J}\left[ZZ\cdots ZZ\,X\,ZZ\cdots
Z\right]^{m_{N}}_{p_{N}}{\rm cos}\left(\frac{\pi nl}{J}\right).$ (11)

The twist of the operator (11) at leading order is then given by

$(\Delta-
J_{\chi})=\frac{1}{2}\left[1+\frac{8\pi^{2}n^{2}}{J^{2}}\lambda^{2}\right].$
(12)

From the above result, we can see that we could explore the strong coupling
($\lambda>>1$) regime of operators of the type (11) by taking the limit
$J-$large and $\lambda-$large but keeping $\lambda/J$ finite. On the other
hand, by doing $J-$large, we could take the continuous limit of (11), where
the chain of $Z$’s become a string and the impurity behaves as a single
excitation of the string. By using the $AdS/CFT$ correspondence, it has been
proved that it is the actual picture for both, $\mathcal{N}=4$ SYM in four
dimensions Berenstein et al. (2002) and for Chern-Simons matter with
$\mathcal{N}=6$ in three dimensions Gaiotto et al. (2009) (for open strings in
the three-dimensional case see also Cardona and Nastase (2014)) . In both
cases, the large$-\lambda$, large $-J$ limit of the subset of operators just
discussed behaves as a string moving really fast on a sphere and on a complex
projective space, respectively.

##  4 Strings from DIS in QCD

As a last example, let us consider a more realistic scenario, such as a
process of deeply inelastic scattering DIS of a very energetic hadron off of
virtual photon. DIS processes have played a very important role in the
understanding of QCD in the early days, in particular it was fundamental in
the formulation of the parton model.

The scattering matrix for an hadron scattered off a virtual photon to an
hadron plus a virtual photon, can be written as,

$i\mathcal{M}(\gamma p\to\gamma
p)=(ie)^{2}\epsilon_{\mu}^{*}(q)\epsilon_{\nu}(q)(-iW^{\mu\nu}(P,q))\,,$ (13)

where $q$ and $\epsilon_{\mu}$ are the momentum and polarization vector of the
virtual photon respectively, $P$ is the momentum of the hadron and the matrix

$W^{\mu\nu}=i\int d^{4}x{\rm e}^{iq\cdot x}\langle
P|T\\{J^{\mu}(x)J^{\nu}(0)\\}|P\rangle\,,$ (14)

is known as the Forward Compton Amplitude. $J^{\mu}$ is the quark
electromagnetic current

$J^{\mu}=\sum_{f}Q_{f}\bar{q}_{f}\gamma^{\mu}q_{f}$ (15)

Since we are interested in a very high energetic scattering process, we would
like to see at short distances $x\to 0$ and hence we can use the Operator
Product Expansion between the electromagnetic currents as follows

$\int d^{4}x\,{\rm
e}^{ipx}J_{\mu}(x)J_{\nu}(0)\sim\sum_{f}\sum_{n=2}c_{n}^{\mu_{1}\cdots\mu_{n-2}}{\cal
O}^{(n)}_{f\,\mu\nu,\mu_{1}\cdots\mu_{n-2}}\,,$

where

${\cal
O}^{(n)}_{f\,\mu_{1}\cdots\mu_{n}}=\bar{q}\gamma^{\\{\mu_{1}}D^{\mu_{2}}\cdots
D^{\mu_{n}\\}}q\,,$ (16)

Hence the computation is reduced to the expectation value $\langle P|{\cal
O}^{(n)}_{f\,\mu_{1}\cdots\mu_{n}}|P\rangle$, which should satisfies the
renormalization group equations (Callan-Symanzik),

$\left(\mu^{2}\frac{d}{d\mu^{2}}+\beta(g^{2})\frac{\partial}{\partial
g^{2}}\right){\mathbb{O}}=-[{\mathbb{H}}_{2}(g^{2}){\mathbb{O}}]\,,$

Since we would like to focus on high energetic particles, we can neglect the
masses of all quarks. In that case, it is convenient to use a “light-cone
formalism” J.B. Kogut (1970); Brink et al. (1983); Mandelstam (1983) in which
one integrates non-propagating degrees of freedom and only keeps the physical
ones. It ultimately means that we define every field on a null vector. Calling
that vector $n$, it should satisfy that $n^{2}=1$.

On the light-cone, the evolution kernel appearing in (4) is given at one-loop
by I.I. Balitsky (1989)

${\mathbb{H}}_{2}=\frac{g^{2}\,C_{F}}{8\pi^{2}}[H_{12}+2\gamma_{q}]\,,$ (17)

where $\gamma_{q}$ is the anomalous dimension of the quark field at one-loop
and $H_{12}$ can be represented as an integral operator as

$\displaystyle[H_{12}\cdot{\mathbb{O}}](z_{1},z_{2})=\int_{0}^{1}\frac{d\alpha}{\alpha}\bar{\alpha}[2{\mathbb{O}}(z_{1},z_{2})$
$\displaystyle~{}~{}~{}~{}~{}~{}-{\mathbb{O}}(\bar{\alpha}z_{1}+\alpha
z_{2},z_{2})-{\mathbb{O}}(z_{1},\alpha z_{1}+\bar{\alpha}z_{2})]\,,$

where $\bar{\alpha}=1-\alpha$. These operators have the physical
interpretation that they displace two particles along the light-cone in the
direction of each other. In order to solve the Callan-Symanzik equation (4) we
should diagonalize ${\mathbb{H}}_{2}$.

Since we are studying a hard processes at large energies, we could use the
fact that QCD is approximately conformal at that limit, providing that
$\alpha_{s}\to 0$ Braun et al. (2003). On the light-cone trajectories, the
conformal symmetry reduce to $SL(2,\mathbb{R})$, acting on the coordinate $z$
parametrizing the light-rays as,

$z\rightarrow\frac{az+b}{cz+d},\quad\mathbb{O}(\alpha
z)\rightarrow(cz+d)^{-2j}\mathbb{O}\left(\alpha\frac{az+b}{cz+d}\right)\,,$
(18)

with $ad-bc=1$ and $j$ being the conformal weight of the field. The
infinitesimal generators of the light-cone conformal symmetry
$SL(2,\mathbb{R})$ are given by the global Virasoro generators,

$L^{-}=-\partial_{z},\quad L^{+}=2jz+z^{2}\partial_{z},\quad
L^{0}=j+z\partial_{z}\,.$ (19)

The conformal symmetry tell us that the operators ${\mathbb{H}}_{2}$ have to
commute with the two particle conformal spin $L_{12}^{a}\equiv
L_{1}^{a}+L_{2}^{a},\,\,a=\pm,0$ (since the operator (16) is composed of two
quarks). It in turn implies that ${\mathbb{H}}_{2}$ should be a function of
the casimir operator of $SL(2,\mathbb{R})$

$L_{12}^{2}=L_{12}^{a}L_{12\,a}=J_{12}(J_{12}-1)\,.$ (20)

Acting with the two operators ${\mathbb{H}}_{2}$ and $L_{12}^{2}$ over a
representation carrying spin $J_{12}=n+2$ it is found Korchemsky (2012) that,

$H_{12}=2[\psi(J_{12})-\psi(2)]\,,$ (21)

where

$\psi(x)=\frac{d{\rm ln}\Gamma(x)}{dx}$ (22)

is the Euler psi-function. For large values of $x$ the $\psi(x)$ function can
be approximate by,

$\psi(x)={\rm ln}(x)-\frac{1}{2x}+\mathcal{O}(x^{-2})\,,$ (23)

therefore, for operators with large spins $J_{12}$ the value under the
evolution kernel can be approximate by

$H_{12}\sim 2{\rm ln}J_{12}\,.$ (24)

which means that at one loop, the twist $\tau$ of the operator (16) scales as
${\rm ln}{J_{12}}$. In the next section, we are going to see that this
behavior can be reproduced from a classical string.
Rotating string in $AdS$
Let us take a spinning closed string in $AdS$ Gubser et al. (2002); Kruczenski
(2005). Without loss of generality, let us consider $AdS$ space in three
dimensions with the following metric,

$ds^{2}=-{\rm cosh}^{2}dt^{2}+d\rho^{2}+{\rm sinh}^{2}d\theta^{2}\,.$ (25)

We choose an ansatz for a particular rotating string,

$t=\tau,\quad\theta=\omega\tau+\sigma,\quad\rho=\rho(\sigma)\,.$ (26)

A classical solution should minimize the Nambu-Goto action on $AdS_{3}$,

$S=-\frac{\sqrt{\lambda}}{2\pi}\int\sqrt{-\dot{X}^{2}X^{\prime
2}+(\dot{X}X^{\prime})^{2}}\,.$ (27)

By means of the rotating string ansatz and the action above one can found the
equation of motion for $\rho(\sigma)$ given by,

$\rho^{\prime}(\sigma)=\frac{1}{2}\frac{{\rm sinh}2\rho}{{\rm
sinh}2\rho_{0}}\frac{\sqrt{{\rm sinh}^{2}2\rho-{\rm
sinh}^{2}2\rho_{0}}}{\sqrt{{\rm cosh}^{2}\rho-\omega^{2}{\rm
sinh}^{2}\rho}}\,,$ (28)

where $\rho_{0}$ is a integration constant which corresponds to the minimum
value of $\rho$ for the given solution. Since we would like to compute the
energy and spin of the rotating string, it is not needed to know the exact
solution. The quantity of interest is $E-J$ when $J$ is large. In that limit,
the above quantity can be approximated as

$E-J\sim\sqrt{\lambda}\,{\rm ln}(4\pi J)\,.$ (29)

In the above expression, $\sqrt{\lambda}$ should be proportional to the string
tension, which in the solutions we are considering must be very big, since the
string is strongly stretched when $J$ is large. So we see that the classical
”twist” $E-J$ of the rotating string in $AdS$ scales with the angular momentum
in the same way as the twist of the composite operators appearing at the
small-$x$ DIS scattering amplitude.

According to the AdS/CFT correspondence, the above rotating string in $AdS$
should corresponds to the strong coupling description of a composite operator
with $J-$derivatives in ${\cal N}=4$ SYM.

##  5 Conclusions

In this note we have collected a few known examples of stringy effective
descriptions for subsets of large charged operators in non-abelian gauge
theories. We mainly intended to show how extended-like objects appear into
field theory, as an effective description of collective phenomena happening at
large energies and strong coupling.

In our opinion, it shows how, among other several situations, string theory
becomes not only a nice tool to explore interesting limits of field theory but
also how it is ubiquitous for a large set of phenomena at high energies.

Acknowledgements

I would like to thank to the organizer of the X SILAFAE for the hospitality
during the event. The research of CC is supported in part by CNPQ grant
501043/2012-8.

## References

  * Abrikosov (1957) Abrikosov, A., 1957. On the Magnetic properties of superconductors of the second group. Sov.Phys.JETP 5, 1174–1182.
  * Aharony and Komargodski (2013) Aharony, O., Komargodski, Z., 2013. The Effective Theory of Long Strings. JHEP 1305, 118.
  * Berenstein et al. (2002) Berenstein, D. E., Maldacena, J. M., Nastase, H. S., 2002. Strings in flat space and pp waves from N=4 superYang-Mills. JHEP 0204, 013.
  * Braun et al. (2003) Braun, V., Korchemsky, G., Mueller, D., 2003. The Uses of conformal symmetry in QCD. Prog.Part.Nucl.Phys. 51, 311–398.
  * Brink et al. (1983) Brink, L., Lindgren, O., Nilsson, B. E., 1983. N=4 Yang-Mills Theory on the Light Cone. Nucl.Phys. B212, 401.
  * Britto et al. (2005) Britto, R., Cachazo, F., Feng, B., 2005. New recursion relations for tree amplitudes of gluons. Nucl.Phys. B715, 499–522.
  * Cardona and Nastase (2014) Cardona, C., Nastase, H., 2014. Open strings on D-branes from ABJM.
  * Fradkin (1991) Fradkin, E., 1991. Field theories of condensed matter systems.
  * Gaiotto et al. (2009) Gaiotto, D., Giombi, S., Yin, X., 2009. Spin Chains in N=6 Superconformal Chern-Simons-Matter Theory. JHEP 0904, 066.
  * Gubser et al. (2002) Gubser, S., Klebanov, I., Polyakov, A. M., 2002. A Semiclassical limit of the gauge / string correspondence. Nucl.Phys. B636, 99–114.
  * I.I. Balitsky (1989) I.I. Balitsky, V. B., 1989. Evolution equations for QCD string operators. Nucl.Phys. B311, 541.
  * J.B. Kogut (1970) J.B. Kogut, D. S., 1970. Quantum Electrodynamics in the Innite-Momentum Frame. Phys. Rev. D. 1, 2901.
  * Korchemsky (2012) Korchemsky, G., 2012. Review of AdS/CFT Integrability, Chapter IV.4: Integrability in QCD and N=4 SYM. Lett.Math.Phys. 99, 425–453.
  * Kruczenski (2005) Kruczenski, M., 2005. Spiky strings and single trace operators in gauge theories. JHEP 0508, 014.
  * Maldacena (1999) Maldacena, J. M., 1999. Large N field theories, string theory and gravity, 47–116.
  * Mandelstam (1983) Mandelstam, S., 1983. Light Cone Superspace and the Ultraviolet Finiteness of the N=4 Model. Nucl.Phys. B213, 149–168.
  * Nielsen and Olesen (1973) Nielsen, H. B., Olesen, P., 1973. Vortex Line Models for Dual Strings. Nucl.Phys. B61, 45–61.
  * Perelomov (1986) Perelomov, A., 1986. Generalized coherent states and their applications.
  * Polchinski (1996) Polchinski, J., 1996. Tasi lectures on D-branes, 293–356.
  * Shifman and Yung (2015) Shifman, M., Yung, A., 2015. Critical String from Non-Abelian Vortex in Four Dimensions.
  * Teper (2009) Teper, M., 2009. Large N and confining flux tubes as strings - a view from the lattice. Acta Phys.Polon. B40, 3249–3320.
  * van Deurzen et al. (2014) van Deurzen, H., Greiner, N., Huston, J., Luisoni, G., Mastrolia, P., et al., 2014\. Recent progress in higher order computations for the physics of the Higgs boson. PoS LL2014, 038.
  * Witten (2004) Witten, E., 2004. Perturbative gauge theory as a string theory in twistor space. Commun.Math.Phys. 252, 189–258.