# Science Driven Innovations Powering Mobile Product: Cloud AI vs. Device AI
Solutions on Smart Device

Deguang Kong  Yahoo Research701 1st Ave, Sunnyvale, California, 94089
<EMAIL_ADDRESS>
(2016)

###### Abstract.

Recent years have witnessed the increasing popularity of mobile devices (such
as iphone) due to the convenience that it brings to human lives. On one hand,
rich user profiling and behavior data (including per-app level, app-
interaction level and system-interaction level) from heterogeneous information
sources make it possible to provide much better services (such as
recommendation, advertisement targeting) to customers, which further drives
revenue from understanding users’ behaviors and improving user’ engagement. In
order to delight the customers, intelligent personal assistants (such as
Amazon Alexa, Google Home and Google Now) are highly desirable to provide
real-time audio, video and image recognition, natural language understanding,
comfortable user interaction interface, satisfactory recommendation and
effective advertisement targeting.

This paper presents the research efforts we have conducted on mobile devices
which aim to provide much smarter and more convenient services by leveraging
statistics and big data science, machine learning and deep learning, user
modeling and marketing techniques to bring in significant user growth and user
engagement and satisfactions (and happiness) on mobile devices. The developed
new features are built at either cloud side or device side, harmonically
working together to enhance the current service with the purpose of increasing
users’ happiness. We illustrate how we design these new features from system
and algorithm perspective using different case studies, through which one can
easily understand how science driven innovations help to provide much better
service in technology and bring more revenue liftup in business. In the
meantime, these research efforts have clear scientific contributions and
published in top venues, which are playing more and more important roles for
mobile AI products.

engagement, growth, big data science, recommender, targeting, forecasting,
effectiveness, deep, convolution, GoogLeNet, optimization, embedding, LSTM,
natural language, malicious, adversarial, privacy

††copyright: rightsretained††doi: 10.475/123_4††isbn:
123-4567-24-567/08/06††conference: ACM Mobile AI conference; Dec 2017;
USA††journalyear: 2018††article: 4††price: 15.00

##  1\. introduction

Artificial intelligence (AI) is the technology driving the new revolution. AI
on cloud is running AI algorithms on cloud side, and it has demonstrated
overwhelming performance in image recognition, speech recognition, and video
understanding, etc. AI on device is running powerful AI algorithms on the
device. Due to the limited resources on device side, AI algorithm must be more
economic and efficient to satisfy real-time serving requirement.

New challenges come with the exponentially growing markets of mobile device
applications. We need to address many new problems, for example, diversified
app markets, heterogeneous users’ behaviors and limited computational
resources, in order to provide better service and improve user engagement on
mobile devices. AI is playing more and more important roles in mobile product,
which is highly demanded by customers to provide more intelligent, smart and
convenient services. In the paper next, we will show our research efforts
towards building smarter and more convenient mobile systems in the following
aspects:

  * •

Cloud-AI solutions: The majority of computing and learning tasks are performed
at cloud side, which provides rich resources and computational powers for
computing. Cloud is necessary and very helpful for pooling of big data and
training huge amount of history observations using sophisticated machine/deep
learning and parameter tuning strategies.

    * –

Mobile App Understanding: Risk Assessment and Malicious App detection

    * –

Marketing for mobile app User Growth: Attraction new users, retention,
campaign effectiveness analysis

    * –

Mobile App User engagement: User modeling, profiling and Recommendation

    * –

Mobile App Monetization: native advertisement serving, advertisement
targeting, bid optimization and pricing

  * •

Device AI solutions: The major learning and recognition tasks are performed at
device side. Running AI on device has brought many advantages, such as
immediate response, enhanced reliability, increased privacy, and efficient use
(Li and Chuah, 2015) of network bandwidth. AI inference algorithm can be
mainly running on mobile devices.

    * –

Image Recognition and Image Privacy on Mobile Devices

    * –

Deep learning model compression on mobile devices

  * •

Cloud AI and Device AI interactions: Some parts of learning and recognition
tasks are performed at device side while others are put on cloud side. One
particular example is personal assistant, such as Amazon Alexa and Google
Home. In particular, AI inference running entirely in the cloud will have
issues for real-time serving that are usually latency-sensitive and mission-
critical (e.g., autonomous driving). The real-world system is expected to
benefit from the cloud-side sophisticated training and high-performance device
processing and inference (Li et al., 2016c), leading to the best overall
system performance.

    * –

Personal Assistant Engine on Mobile Device

      * *

Dialog System

      * *

Speech recognition

      * *

Speech synthesis

      * *

NLP understanding

      * *

Chatbot

      * *

Recommender system

We provide scientific leaderships and provide practical solutions to solving
these challenging problems because we strongly believe that, pure engineering
is not enough in practice. These techniques will be very helpful for
addressing the challenging problems in solving mobile big data science and AI
problems in real world if (a big “if”) the technology can be accurate, robust
and scalable enough. We need research breakthrough to improve the state-of-
the-art and we have diligently worked on them. Fortunately, some of them have
been graduated into product.

Figure 1. Overview of cloud AI solution: (a) left panel: from application
perspective, understanding apps and detection of malicious apps; (b) right
panel: user growth, user engagement and monetization.

##  2\. Cloud AI solution

In this section, we provide cloud-AI solutions from the following four
perspectives:

  * •

Mobile App Understanding: Risk Assessment and Malicious App detection

  * •

Marketing for mobile app User Growth: Attraction new users, retention,
Campaign effectiveness analysis

  * •

Mobile App User engagement: User modeling, profiling and Recommendation

  * •

Mobile App Monetization: advertisement serving, advertisement targeting, bid
optimization and pricing

In particular, Fig.1 shows an overview of the framework.

###  2.1. Mobile App Understanding

Figure 2. Ranking the risk of mobile apps using multi-modal features such as
descriptions, user review, permission access and ad library.  Figure 3.
Flowchart of risk assessment from user comments. Given an app, the user
comments are used to evaluate the risk of apps in two steps: (a)
“crowdsourcing” is used to accumulate user comments into app-level features to
accumulate user comments into app-level features (shown as “feature
extraction”, “auto annotation” and “crowdsourcing”); (b) “learning to rank”
model is used to predict risk scores by utilizing these latent features, where
pairwise constraints are enforced between pairwise apps (shown as relative
scores of two apps).  Figure 4. Demonstration of app risk assessment app.

A mobile app is a software application developed specifically for the use on
smart devices. The mobile can provide the convenience to users to achieve the
users’ purpose and satisfy users’ interest, almost solving everything you may
image. For example, one can get news from the news app, one can purchase a
Swiss watch while sitting at any office, one can even talk with the lover who
is actually far away for thousands of miles. In this section, we will provide
the mobile app understanding from several perspectives:

$\bullet$ App risk assessment

$\bullet$ Malicous app detection

$\bullet$ App maturity rating

####  2.1.1. App Risk Assessment

Comparing with traditional software markets, markets like Google Play and
Apple Store have lower entry threshold for developers and faster financial
payback, hence greatly encouraging more and more developers to invest in this
thriving business. Therefore controlling the quality of apps, especially the
security risk of them across the whole markets, becomes an important issue to
all that involved. On the other hand, public concerns about privacy issues
with online activity and mobile phones are also elevating, demanding a mobile
environment with more respect to users’ privacy. In mobile apps, permissions
indicate the resources that the apps can access, and thus can be viewed as a
privacy indicator. From users’ perspective, the meta data such as user reviews
and developer descriptions reflect users’ perceptions and developer
expectations for the apps, and thus are also correlated with risks of apps.

Our idea is to explore heterogeneous privacy indicators (Kong and etc., 2015)
(Kong and Cen, 2017) for app risk ranking, which, we believe, is very
important to internet company to improve user engagement in mobile platforms
(as shown in Fig. 2). The risk ranking problem is formulated as a multi-view
feature learning problem by exploring group LASSO and exclusive group LASSO
techniques (Kong et al., 2014), (Kong and Ding, 2013), (Kong et al., 2016)
which can automatically select the most discriminant features by considering
both inter-view feature competitions, and also intra-view feature
correlations. In particular, we solve the following optimization problem,
given feature ${\mathbf{x}}_{i}\in\Re^{p}$ for each app $i$, $Y_{ki}$ for
label of app $i$ with category label $k$ ($1\leq k\leq K$), we aim to find the
feature weight ${\mathbf{w}}^{v}_{k}$ for class $k$ regarding $v$-th ($1\leq
v\leq V$) view feature, i.e.,

(1) $\displaystyle\min_{{\mathbf{W}}\in\Re^{p\times K}}$
$\displaystyle\sum_{i=1}^{n}\sum_{k=1}^{K}\Big{(}Y_{ki}log\sum_{k=1}^{K}\exp^{{\mathbf{w}}^{\top}_{k}{\mathbf{x}}_{i}}-Y_{ki}{\mathbf{w}}^{\top}_{k}{\mathbf{x}}_{i}\Big{)}$
$\displaystyle+\alpha\sum_{k=1}^{K}\sum_{v=1}^{V}\|{\mathbf{w}}^{v}_{k}\|_{2}+\beta\sum_{k=1}^{K}\sum_{v=1}^{V}\|{\mathbf{w}}^{v}_{k}\|^{2}_{1}$

Correspondingly, we derive an efficient iteratively re-weighted algorithm to
tackle the resultant optimization problem, which can handle any group
structure, regardless of coherent or exclusive group structures. It
demonstrates very good performance in real-world datasets (totally 13, 174
apps, 34, 514 descriptions, 9,986, 568 user reviews and 100 ads libraries).

We also derive a crowdsourcing ranking approach (Kong et al., 2015) (Cen et
al., 2015) (see Fig.3) to rank risk of apps from user comments by combining
feature learning and ranking SVM methods, which also provides good solutions
in practice. The problem we solve is formalized as:

(2) $\displaystyle\min_{{\mathbf{w}}\in\Re^{p},\theta,{\mathbf{Y}}_{\ell}}$
$\displaystyle-log\Pr(D_{n}|\theta,{\mathbf{Y}}_{\ell})-log\Pr(\theta)$
$\displaystyle\lambda\|{\mathbf{w}}\|^{2}+C\|({\mathbf{e}}-{\mathbf{B}}{\mathbf{Y}}_{\ell}{\mathbf{w}}\|^{2},$

based on the maximum a posteriori probability (MAP) estimation, where
${\mathbf{w}}$ is the feature weight, ${\mathbf{Y}}_{\ell}$ is the labels
learned from the feature pre-processing step and $\theta$ is the prior
distribution of parameters in crowdsourcing process by aggregating different
user reviews, and $logPr(D_{n}|\theta,{\mathbf{Y}}_{\ell})$ gives the
likelihood of objective function given the current parameters while
$\|({\mathbf{e}}-{\mathbf{B}}{\mathbf{Y}}_{\ell}{\mathbf{w}}\|^{2}$ is the
hinge-loss function in SVM ranking.

Lessons Learned We do need multi heterogeneous models to find the most
discriminant features. User reviews and ads libraries play import roles in
understanding the risk of apps except the permissions. A demo system is shown
in Fig.4.

####  2.1.2. Malicious App Detection

Figure 5. Enhanced malicious app detection process: (i) for benign apps, we
generate adversarial examples; (ii) for malicious apps, we generate the
mutated and evolved apps.  Figure 6. Phylogenetic tree generated for the
DroidKungFu family. Each leaf in the graph denotes a malware sample in
DroidKungFu family, where leaf nodes (1–4) belong to droidkungfu.ab, (5–9)
belong to droidkungfu.aw, (10) belongs to droidkungfu.bb, (11– 12) belong to
droidkungfu.bl, (13–15) belong to droidkungfu.c, (16–22) belong to
droidkungfu.g, (23–28) belong to droidkungfu. m.

Existing techniques on adversarial malware generation employ feature mutations
based on feature vectors extracted from malware. However, most (if not all) of
these techniques suffer from a common limitation: feasibility of these attacks
is unknown. The synthesized mutations may break the inherent constraints posed
by code structures of the malware, causing either crashes or malfunctioning of
malicious payloads. To address the limitation, Yang et. al. (Yang et al.,
2017) present Malware Recomposition Variation (MRV), an approach that conducts
semantic analysis of existing malware to systematically construct new malware
variants for malware detectors to test and strengthen their detection
signatures/models. In particular, we use two variation strategies (i.e.,
malware evolution attack and malware confusion attack) following structures of
existing malware to enhance feasibility of the attacks. Upon the given
malware, we conduct semantic-feature mutation analysis and phylogenetic
analysis to synthesize mutation strategies. Based on these strategies, we
perform program transplantation to automatically mutate malware byte-code to
generate new malware variants. We evaluate our MRV approach on actual malware
variants, and our empirical evaluation on 1,935 Android benign apps and 1,917
malware shows that MRV produces malware variants that can have high likelihood
to evade detection while still retaining their malicious behaviors. We also
propose and evaluate three defense mechanisms to counter MRV.

Fig.5 gives an overview of the technique we used for malicious app detection.
The major differences between our work and the existing works are:

Mutated samples We mutate the malware features from the original feature
values to the ones that are less differentiable for malware detection (Kong
and Yan, 2013). The newly generated samples are fed into the classification
model again for building the discriminant classifier.

Evolved samples We mimic and automate the evolution of malware based on the
insight that the evolution process of malware reflects the strategies employed
by malware authors to achieve a malicious purpose while evading detection.
Fig.6 gives an example of phylogenetic tree generated for the DroidKungFu
family. The newly generated samples are fed into the classification model
again for building the discriminant classifier.

Adversarial samples We explore the malware features to identify much more
blind spots of existing detection, and generate the adversarial samples from
benign apps that are actually labeled as malicious ones. The newly generated
adversarial samples are fed into the classification model again for building
the discriminant classifier.

Our work also has strong connections with adversarial learning (Li et al.,
2016). To make the discriminant classifier more robust, actually we generate
the synthetic apps that fools the discriminator into accepting it as the true
apps. Similar to Generative adversarial network (GAN) (Goodfellow et al.,
2014), these adversarial and mutated/evolved sample generation process is like
a generative process, heavily relying on the feature model used in malware
detection, instead of the “random noises” as in standard GAN. The generative
process actually enforces “data augmentation” operations, which is a key
strategy used in deep learning process.

Semi-supervised learning is applied for automatic generation of Android
security policies in (Wang et al., 2015). However, it suffers from the inherit
limitation of semi-supervised learning (Kong and Yan, 2014): the algorithm
performance degrades significantly at the situation when the mislabeled
samples tend to propagate the errors instead of “belief” along the similarity
measurement path computed from domain knowledge or underlying data manifold
information, which is well known in machine learning community.

####  2.1.3. App maturity rating

Figure 7. App maturity rating: from word embedding to app embedding.

App maturity rating concerns how to protect children from inappropriate
content in mobile apps. Apps may contain sexual, violence and drug usage in
their content. Therefore, mobile platforms provide rating policies to label
the maturity levels of Apps and the reasons why an App has a given maturity
level, which enables parents to select maturity-appropriate Apps for their
children. However, existing approaches to implement these maturity rating
policies are either costly (because of expensive manual labeling) or
inaccurate (because of no centralized controls). In this work (Hu et al.,
2015) we aim to design and build a machine learning framework to automatically
predict maturity levels for mobile Apps and the associated reasons with a high
accuracy and a low cost using machine learning (and deep learning) techniques.

Specifically, we extract novel features from App descriptions by leveraging
the semantic embedding of words (a.k.a word embedding) to automatically
capture the semantic similarity between words and adapt Support Vector Machine
to capture label correlations with Pearson correlation in a multi-label
classification setting. In particular, in embedding step, given a sequence of
training words $w_{1}$ , $w_{2}$ , $w_{3}$, $\cdots$, $w_{T}$ , the skip-gram
model (Mikolov et al., 2013) is used to maximize the average log likelihood,
i.e.,

(3) $\displaystyle\max\frac{1}{T}\sum_{t=1}^{T}\sum_{-c\leq j\leq c,j\neq
0}log\Pr(w_{t+j}|w_{j}),$

where c is the size of training context. and $\Pr(w_{t+j}|w_{j})$ is the
probability of occurrence of $w_{t+j}$ given $w_{t}$ which is usually defined
using a softmax function, i.e.,

(4)
$\displaystyle\Pr(w_{O}|w_{I})=\frac{\exp\Big{(}({\mathbf{v}}^{\prime}_{w_{O}})^{\top}{\mathbf{v}}_{w_{I}}\Big{)}}{\sum_{o=1}^{W}\exp\Big{(}({\mathbf{v}}^{\prime}_{w_{o}})^{\top}{\mathbf{v}}_{w_{I}}\Big{)}}$

where ${\mathbf{v}}_{w_{I}}$ and ${\mathbf{v}}^{\prime}_{w_{O}}$ are the
“input” and “output” word embedding, and $W$ is the number of words in
vocabulary.

In app maturity rating application, all app descriptions and textual comments
are pre-processed as word-embedding. Then they can be fed into the embedding
model perfectly to learn the app embedding. The key idea is to aggregate the
semantics from word level to app level, which actually leverages the semantics
from _word embedding_ to _App embedding_. Fig.7 shows the flowchart of design.
Essentially, the framework infers the app maturity using the following logics:

App description and User comment $\rightarrow$ word embedding $\rightarrow$
App level embedding $\rightarrow$ Predictive model $\rightarrow$ Maturity
contents labeling $\rightarrow$ maturity level

In experiment, we evaluate our approach and various baseline methods using
datasets that we collected from both Apple store and Google Play. We
demonstrate that, with only App descriptions, our approach already achieves
85% Precision for predicting mature contents and 79% Precision for predicting
maturity levels, which substantially outperforms baseline methods.

###  2.2. Marketing for mobile app User Growth

Marketing the app is one of the most common ways used for driving user growth
and improving user engagement. In this section, we will discuss the business
intelligence techniques used for attracting new users, promoting the retention
of users and improving ad campaign effectiveness.

####  2.2.1. Attract more users

There are several typical ways used for attracting new users in business
intelligence:

  * •

Concentrating on user experience: the best way to build customer relationships
is to delight the customers. In a simple word, leave everyone who uses the app
feeling good.

  * •

Cross promotion One need to target people in different segments. Segment the
potential customers and figure out how likely one can bring the customer to
daily active users (DAU) or monthly active users (MAU). Targeting model is
widely adopted in Facebook, Google and Yahoo business to targeting the look-
alike users.

  * •

Develop a Content/functionality Strategy The app is desired to provide the
interesting contents and functionalities that will hit the target audience at
the right places. The content and functionality features should cater to
targeting’s requirement. This process is usually optimized using A/B tests to
generate even more virality. The “viral factor” can be used to measure how
effectively the new feature can attract new customers, which is widely adopted
in Facebook, Instagram, Snapchat, etc.

  * •

Media exposure Tell a great story to the media and spend money on different
publishers in ad campaigns. The return on investment can be measured using
campaign effectiveness analysis.

  * •

Purchasing more traffics At user-level, different coupons, rewards and
discounts are exciting ways to attract new customers. At business side,
jointly work with the major internet service providers in bundled sales
approach will promote the user growth111https://hbr.org/2016/02/every-company-
needs-a-growth-manager. This is also known as _paid acquisition_.

  * •

Search engine optimization Promote the product using the search engine from
creating contents that are more favorable by search engine. The content
includes Q&A, articles, long-form reviews, etc. The goal is to increase the
page-views from attracting the new users after searching.

In practice, all these different scenarios can be combined together to promote
user growth based on the budget limit. The mathematic optimization is easy to
be obtained before reaching the ceiling on saturation, i.e.,

$\displaystyle\max_{\theta,\Theta}\frac{\Delta\text{\\#User}(\Theta)}{\Delta{\text{Money\\_Spent}(\theta)}},$
(5) $\displaystyle s.t.\text{\\#User}(\Theta)\leq\text{Market\\_capacity}$

where $\text{\\#User}(\Theta)$ and $\text{Money\\_Spent}(\theta)$ denote the
number of users and current money spend in the marketplace, and
$\Theta,\theta$ are parameters, respectively. In real world, the optimization
is performed either empirically from history experience or based on the
forecasting results using machine learning and statistical models. Although
the model itself can be highly biased due to the censored and noisy data
observations, we believe robust statistical modeling is still an effective and
automated way for internet marketing compared to empirical analysis (or simple
cohort analysis) from prior knowledge, i.e.,

$\displaystyle\Theta=argmin_{\Theta}\mathcal{L}\Big{(}\text{User
number},\hat{f}(\text{user feature set, app feature set},\Theta)\Big{)}$
$\displaystyle\theta=argmin_{\theta}\mathcal{L}\Big{(}\text{Money
spent},\hat{g}(\text{user feature set, app feature set},\theta)\Big{)},$

where $\mathcal{L}(x,y)$ is the loss function (e.g., least square loss or
cross-entropy loss) that captures the difference between $x$ and $y$,
functions $f(.)$ and $g(.)$ can be learned using machine learning/deep
learning model in distributed big data science environment.

####  2.2.2. Improve user retention

Figure 8. User segmentation (cohort) based on time-varying features.

A retention analysis allows one to see the numbers like this:

  * •

What percentage users are coming back after a week?

  * •

What percentage of users are paying after a month?

  * •

How long users stick around for a week or for a month?

  * •

Did the new feature released last month increase retention or degrade it?

  * •

Why some users churn and others do not?

  * •

How to form growth hypotheses based on quantitative data?

Essentially, retention analysis tells a compelling story about “users doing A
in a period” more than “number of users doing A”. In cohort analysis, users
are grouped into different segmentation (e.g., based on sign-up date) for
behavioral analytics. For example, one can easily observe the percentage of
the accounts that still use the service in the following weeks since they
signed up at different time. Also, stickiness is a measure of engagement,
which looks at how many times a user performs a particular action in a weekly
or monthly interval. In practice, after setting the analytic goals (such as
the number of weeks and the cohort of users one wants to track), one can
easily write funnel queries or use Google Analytics (or other tools) to
achieve the goal.

Retention analysis can help people to figure out how many people we have lost
and drive the insight from such analysis. The numbers shown in retention
analysis is not the final goal, but the insight and solutions to improve user
retentions are what we really want. Therefore, it is necessary to figure out
the relationship between user actions and retention one you can identify the
main behaviors that are correlated with long-term use. The most widely used
correlation measurement between retention and an action is pearson correction,
i.e.,

(6) $\displaystyle
r=\frac{\sum_{i}(x_{i}-\bar{x})(y_{i}-\bar{y})}{\sqrt{\sum_{i}(x_{i}-\bar{x})^{2}}\sqrt{\sum_{i}(y_{i}-\bar{y})^{2}}},$

where $x_{i}$ is sample of retention observations and $y_{i}$ is sample of
user action observations. For example, for facebook users, users who already
added 7 friends within the first 10 days after they used facebook are more
likely to continue to use Facebook long-term while users who added less than 7
friends are more likely to churn out. For this situation, the user action of
“adding 7 friends in a certain timeframe” is highly correlated with retention.
Please keep in mind that “correlation” does not mean “causality”. The user
action may not be responsible for the retention. Further causality analysis is
needed to connect the cause (i.e., user action) with the effect (i.e., user
retention). Fig.8 shows the user segmentation result based on user behaviors.

Further we clarity the relations between retention rate, survival function
(Rupert G. Miller, 2001), hazard function (or hazard rate) widely used in
statistic data science .

Hazard Rate is usually defined as the probability $\lambda_{i}$ of object that
does not survive in the $i$-th time interval ($t_{i}-t_{i-1}$).

Retention Rate is defined as the probability (=$1-\lambda_{i}$) of object that
survives in the $i$-th time interval $(t_{i}-t_{i-1})$.

Survival Function $S(t_{i})$ is defined as the probability of object that
survives up to $t_{i}$.

Essentially we have:

###### Lemma 2.1.

Retention rate = 1 - Hazard Rate. Survival function $S({t_{i}})$ is given by:

(7) $\displaystyle\text{Kaplan-Meier Method}:\quad\quad
S(t_{i})=\prod_{i}(1-\lambda_{i})$ (8) $\displaystyle\text{Nelson-Aalen
Method}:\quad\quad S(t_{i})=\exp{(-\sum_{i}\lambda_{i})}$

####  2.2.3. Marketing funnel and campaign effectiveness

Figure 9. Marketing funnel (from standard marketing science book). The
advertising campaigns are started to target different customer cohort. The
app-level campaign is generally initiated for brand-level advertising, while
user-level advertising is started for lifting up the direct response from
users.

The marketing funnel is widely adopted in marketing research for understanding
the chance of turning leads into customers from marketing and sales
perspective. The general idea is similar to a _funnel_ , i.e., marketers first
give very broad net in order to capture the potential customers as many as
possible, and then slowly nurture prospective customers through the purchasing
(or conversion) decision, by narrowing down the candidate pools in each stage
of the funnel. Fig. shows the different stages in marketing funnel, i.e.,

Awareness $\rightarrow$ interest $\rightarrow$ consideration $\rightarrow$
intent $\rightarrow$ evaluation $\rightarrow$ purchase.

Campaign Name | Targeting audience | Rewards
---|---|---
Appolo | first time user | New York travel
Light burn | MAU | iphone 8
Fantastic day | Infrequent users | Dogfood
Table 1. Campaign examples (only for demonstration purpose).

In practice, at different stages of marketing funnel, the promotion campaign
would be different based on targeting customers. For example, at “awareness”
stage, the campaign goal is to discover the customer net via establishing the
trust and thoughts with events, advertising, trade shows, blog posts,
infographics, etc. Therefore, the campaign effectiveness measurement should be
defined to measure the reaches of customers at different information sources.
At “evaluation” stage, one should convince the buyers to make a final
decision, then the purchase rate will be an important factor to evaluate the
campaign effectiveness.

In fact, marketing campaign can be performed at different stages of funnels
for B2C or B2B business. As illustrated before, the campaign effectiveness
analysis actually depends on the targeting customers at different stages of
funnel, which can be viewed as external variables to _personalize_ the
campaign effectiveness analysis. We show the _generalized_ version of campaign
effectiveness analysis, where personalized version can be obtained
correspondingly by incorporating the external variable.

We show how to compute the campaign effectiveness using generalized analysis.
Here we give several examples of targeting campaigns in Table.1.

Let $Y_{1i}$ and $Y_{0i}$ be potential benefits (e.g., bonus, rewards, credit,
discount, etc) for individual $i$ when $i$ receives the treatment or does not
receive treatment, respectively. The fundamental problem of making a _casual
inference_ is how to reconstruct the results that are not observed. For each
individual $i$, what if individual $i$ does not receive treatment? Basically,
we do analysis at aggregation-level using average treatment effect (ATE)
analysis and average treatment effect on the treated group (ATT) analysis. ATE
is defined as:

(9) $\displaystyle ATE=E(Y_{1i}|T_{i}=1,0)-E(Y_{0i}|T_{i}=1,0),$

where $E(.)$ represents the expectations on the aggregation level, and $T_{i}$
denotes the treatment with value $1$ for the treated group while value $0$ for
the control group, i.e., the average effect that would be observed in the
treated and control group received treatment, compared with if none in both
groups received treatment. Correspondingly,

(10) $\displaystyle ATT=E(Y_{1i}|T_{i}=1)-E(Y_{0i}|T_{i}=1)$

which refers to the average differences if the treated group received
treatment if none of these in the treated group received treatment. Then the
propensity score (Mccaffrey et al., 2004) is defined as the conditional
probability of receiving a treatment given pre-treatment characteristics:

(11) $\displaystyle\Pr(X)=\Pr(T=1|X)=E(T|X),$

where $T=\\{0,1\\}$ is the indicator of exposure to treatment and $X$ is the
multi-dimensional representation of pre-treatment characteristics. The key
idea is that treated and control units should be on average observationally
identical (a.k.a _balancing hypothesis_). In other words, for a given
propensity score, exposure to treatment is random and therefore any standard
probability model can be used to estimate the propensity score, e.g.,

(12) $\displaystyle\Pr(T_{i}=1|X_{i})=\phi(h(X_{i})),$

where $\phi$ denotes the normal cumulative distribution function, and
$h(X_{i})$ is a function of covariates with linear and higher order terms.
Please keep in mind the choice of estimate of propensity score of function
$h(.)$ must satisfy the balancing hypothesis.

###  2.3. Mobile App User engagement

Figure 10. Upper panel: privacy aware app recommendation; Lower panel: context
aware app recommendation.

In this section, we discuss how to improve the user engagement using user
modeling, profiling and recommendation techniques. As the beginning of June
2014, App Store had 1.2 million Apps and a cumulative of 75 billion downloads.
Therefore it is urgent to develop effective personalized App recommendation
systems. In particular, we discuss how to do _privacy aware app
recommendation_ and _context aware app recommendation_ for mobile users.
Recommendation is useful and helpful since it can capture users’ preference
and interest, and it strongly connects with targeting in technology although
differs in business logic. Fig.10 gives an overview of the app recommendation
introduced in this section.

####  2.3.1. Privacy aware app recommendation

Recent years have witnessed a rapid adoption of mobile devices and a dramatic
proliferation of mobile applications (Apps for brevity). However, the large
number of mobile Apps makes it difficult for users to locate relevant Apps.
Therefore, recommending Apps becomes an urgent task. Traditional
recommendation approaches focus on learning the interest of a user and the
functionality of an item (e.g., an App) from a set of user- item ratings, and
they recommend an item to a user if the item’s functionality well matches the
user interest. However, Apps could have privileges to access a user’s
sensitive resources (e.g., contact, message, and location). As a result, a
user chooses an App not only because of its functionality, but also because it
respects the users’ privacy preference. To the best of our knowledge, this
work presents the first systematic study on incorporating both interest-
functionality interactions and user privacy preferences to perform
personalized App recommendations (Liu et al., 2015). Specifically, we first
construct a new model to capture the trade-off between functionality and user
privacy preference. In particular, in this work, it leverages the state-of-
the-art Poisson factorization technique and optimizes the objective

$\displaystyle\max_{{\mathbf{u}},{\mathbf{v}}}\Pr(y_{ij}|{\mathbf{u}}_{i},{\mathbf{v}}_{j},{\mathbf{p}}_{s}),$
(13)
$\displaystyle=Poisson(y_{ij},{\mathbf{u}}_{i}({\mathbf{v}}_{j}+\lambda\sum_{s\in\Sigma_{j}}{\mathbf{p}}_{s})),$

where $y_{ij}$ is the rating score for a particular user $i$ for app $j$,
${\mathbf{u}}_{i}$ is the user $i$ latent factor, ${\mathbf{v}}_{j}$ is the
app $j$ latent factor and ${\mathbf{p}}_{s}$ is app privacy latent factor with
respect to app $j$. Then we crawled a real-world dataset (16, 344 users, 6,
157 Apps, and 263, 054 ratings) from Google Play and use it to comprehensively
evaluate our model and previous methods. We find that our method consistently
and substantially outperforms the state-of-the-art approaches, which implies
the importance of user privacy preference on personalized App recommendations.
Moreover, we explore the impact of different levels of privacy information on
the performances of our method, which gives us insights on what resources are
more likely to be treated as private by users and influence user behaviors at
selecting Apps.

User | Location Semantics | Recommended Service
---|---|---
John | Safeway | Apple Pay or Chase Pay
Damao | Bank | Get a free coffee
Amy | Mall | Use Banana Coupon
Table 2. Use cases of context aware recommendation.

####  2.3.2. Context aware app recommendation

In many practical applications, in practice, the recommendation depends on
context. Here “context” is a very generic concept that can denote location,
gender, age, interest or other segmentations. In other words, recommendation
is performed on different buckets based on an attribute or a combination of a
group of attributes. Similar to app recommendation, we solve this problem
using tensor bilinear factorization technique (Kong, 2017). In particular, we
solve the following problem:

$\displaystyle\max_{{\mathbf{U}},{\mathbf{V}},{\mathbf{P}}}\Pr({\mathbf{X}}_{ijk}|{\mathbf{U}}_{ir},{\mathbf{V}}_{js},{\mathbf{P}}_{kt})$
(14)
$\displaystyle=Poisson({\mathbf{X}}_{ijk},{\mathbf{U}}_{ir}{\mathbf{V}}_{jr}+{\mathbf{U}}_{it}{\mathbf{P}}_{kt}+{\mathbf{V}}_{js}{\mathbf{P}}_{ks})$

where $X_{ijk}$ is the rating score for a particular user $i$ for app $j$ in
context $k$, ${\mathbf{U}}_{i:}$ is the user $i$ latent factor,
${\mathbf{V}}_{j:}$ is the app $j$ latent factor and ${\mathbf{P}}_{k:}$ is
app context latent factor for context $k$. Similarly this framework can be
easily extended for generating the context aware service recommendations. The
use cases are shown in Table. 2.

Lessons Learned. In this section, we use “app” as demonstrated examples for
recommendation purpose. Our approach can be easily adapted for recommendations
on purchase and others.

###  2.4. Mobile App Monetization

The basic idea is how to make more money from mobile apps. Therefore, this
section includes _advertisement serving, advertisement targeting, bid
optimization and pricing._ Although the technology has been graduated into
product, this section is intentionally left blank until our research works are
published.

##  3\. Device-AI solution

In this section, we provide device-AI solutions, which aims to protect image
privacy on mobile devices.

Figure 11. Private photo privacy protection. The sensitive regions are marked
using bounding box.  Figure 12. Private photo privacy protection pipeline:
original image $I_{x}$ is encrypted to $E(Ix)$, after transformation it
becomes $T(E(Ix))$, and finally after decryption operations it is in the form
of $D[T(E(Ix))]$.

###  3.1. Image Recognition and Image Privacy on Mobile Devices

Every second, nearly 4,000 photos uploaded to Facebook, around 4,600 photos
exchanged through Snapchat Photos are uploaded, saved and shared on cloud,
e.g., centralized photo sharing platforms (PSPs). In cloud side, sensitive
regions are exposed to public, etc. What is the security and privacy risk?
Photo owners worry about privacy leakage on cloud/PSPs, also Cloud/PSPs may
access and process user photos without explicitly asking for user agreement
and share the unprotected photos. In this work we propose _image perturbation
technique_ to protect the image privacy. Ideally, given encryption function
$E(.)$, transformation function $T(.)$, the goal is to find decryption
function $D(.)$, such that for an image $I_{X}$ , it exist

(15) $\displaystyle D\Big{[}T(E(I_{X}))\Big{]}=T(X)$

Our system design is guided by the following theorem. Let $P(.)$ be the image
perturbation technique, we have:

###### Theorem 3.1.

Using Image perturbation technique $E=P(.)$ for “encryption” of photos, it can
exactly “decrypt” the photos and recover the original one, i.e.,

(16) $\displaystyle D\Big{[}T(E(I_{X}))\Big{]}=T(X)$

where $D=f(T,E)$ can be easily calculated given function $E(.)=P(.)$ and
$T(.)$.

Note that “crypto” based technique (including symmetric and public key
encryption) may not work since it is not compatible with transformation $T(.)$
and also $D$ is impossible to be computed given $E(.)$ and $T(.)$, and
therefore $X$ cannot be recovered (see Fig.12). For exactly the same reason,
differential privacy (Li et al., 2016a) added Laplacian noises to the image,
which is, in fact, irreversible although privacy preserving. Finally one
cannot recover anything given image transformation $T(.)$ as well.

In our approach (He et al., 2016), it supports different linear
transformations (e.g., Rotation, Cropping, Scaling) and also non-linear
transformation such as compression. The key idea of our approach is to perturb
DC and AC components discriminant in FFT domain, which achieves the same
purpose as crypto but is compatible with different image transformation. Also,
our approach has advantages due to its simple, fast and effective
implementation. In our solution, applying a transformation on perturbed image
is equal to applying transformation on original image plus applying
transformation on virtual image (generated from perturbation). An example is
shown in Fig.13.

Figure 13. Image perturbation process: the perturbed image is equal to the
original image plus the perturbed parameters. The perturbed image is stored on
cloud, while the original image is collected from the user.

###  3.2. Deep learning based techniques for protecting image privacy

Figure 14. CNN pipeline for privacy detection that consists of two pipelines:
(a) object feature learning pipeline (upper panel); (b) convolution feature
learning pipeline (lower panel). $h(x)$, $\ell(x)$ are activation functions

This section provides a deep learning based technique to protect image
privacy. Several examples are shown in Fig. 11. Photo privacy is a very
important problem in the digital age where photos are commonly shared on
social networking sites and mobile devices. The main challenge in photo
privacy detection is how to generate discriminant features to accurately
detect privacy at risk photos. Existing photo privacy detection works, which
rely on low-level vision features, are non-informative to the users regarding
what privacy information is leaked from their photos. In this section, we
propose a new framework called PrivacyCNH (Tran et al., 2016) that utilizes
hierarchical features which include both object and convolutional features in
a deep learning model to detect privacy at risk photos. In particular, given
the joint deep learning structure (i.e., Alexnet (Krizhevsky et al., 2012))
$\mathcal{V}=\\{V^{1},V^{2},V^{3},V^{4}\\}$ and
$\mathcal{W}=\\{W^{1},W^{2},\cdots,W^{5}\\}$, the posterior probability of
privacy risk for an image $i$ is given by the sigmoid function using the
learned features, i.e.,

(17)
$\displaystyle\Pr(y_{i}=1|X_{i};\mathcal{V},\mathcal{W})=\frac{1}{1+\exp{(-z)}},$
(18) $\displaystyle
z=(V^{4}_{k})^{\top}h_{4}(X_{i})+{(W^{6}_{\ell})}^{\top}\ell_{5}(X_{i})+\beta,$

where $V^{i}$ and $W^{j}$ are the CNN network structure parameters with $i$
and $j$ indicating the layer number, $k$ indexes the hidden unit in layer $i$,
$\ell$ indexes the hidden unit in layer $j$, $h_{i}$ and $\ell_{j}$ are the
activation functions for object CNN and low-level CNN respectively and $\beta$
is the biased scalar term.

The generation of object features enables our model to better inform the users
about the reason why a photo has privacy risk. The combination of
convolutional and object features provides a richer model to understand photo
privacy from different aspects, thus improving photo privacy detection
accuracy. Experimental results demonstrate that the proposed model outperforms
the state-of-the-art work and the standard convolutional neural network (CNN)
with convolutional features on photo privacy detection tasks. Fig. 14
demonstrates the pipeline of our method.

###  3.3. Model compression on mobile devices

Figure 15. Distributed Deep Learning execution framework.

CNN model has been widely and successfully used in many computer vision tasks,
such as object detection, fine-grained image classification, age estimation
(Wang et al., 2017), etc. The popularity of mobile phone brings the great
convenience to people life due to the existence of many practical and
excellent apps. However, to run CNN models (even in testing phrase) for a
typical vision task is a luxury for most devices due to the high computational
cost and limited memory space and power resources. To accelerate CNN models is
highly desirable to facilitate mobile vision applications that highly depends
on the performance of CNN models.

Our investigation on AlexNet indicates that not only full-connected layers and
convolution layers consume a lot of time, but also some non-tensor layers
(such as Pooling layer and LRN layers) that do not contain any high-order
tensor-type weight parameter are also time-consuming. However, current
researches focus on rank approximation or parameter compression in fully
connected and convolution layers. Although helpful, the acceleration and
compression of non-tensor layers are totally ignored.

To address this limitation, this paper (Li et al., 2018) proposes a unified
framework to compress CNN models by dismembering non-tensor layers, to
simultaneously accelerate the CNN model testing performance with neglect
performance degradation. With re-trained new network parameters in “re-birth”
layers, the functionality of non-tensor layers is equivalently implemented in
the new merged layers with significant efficiently improvement. The standard
least square error is used to minimize the error function in re-training
process where the new parameters are essentially the “quantized” old
parameters (in some sense). The framework includes both _streaming merge_ and
_branch merge_ that are able to conduct fast computations easily adapted for
current mainstream CNN models and potential new CNN pipelines. In the
meantime, in order to run deep learning on mobile devices, we provide an
_elastic_ approach to run deep learning in a distributed fashion (shown in
Fig. 15).

Theoretical Analysis

The convolution layer transforms the input feature map $X\in\Re^{M\times
N\times K}\rightarrow Y\in\Re^{M^{\prime}\times N^{\prime}\times K^{\prime}}$,
i.e.

$\displaystyle f_{\text{conv}}:X\mapsto
Y,\;\;\;Y_{i^{\prime}j^{\prime}k^{\prime}}=\sum_{i=1}^{d_{k}}\sum_{j=1}^{d_{k}}\sum_{k=1}^{K}W_{ijkk^{\prime}}X_{i+i^{\prime},j+j^{\prime},k}\;\;(1\leq
k^{\prime}\leq K^{\prime})$

where $K,K^{\prime}$ are the number of feature map channels, and
$M,N;M^{\prime},N^{\prime}$ are the size of the images, which is actually
regular linear convolution by a filter bank, $d_{k}\times d_{k}$ is the kernel
size and feature map $Y$ is essentially the sum of inner product by traversing
along different locations with $d_{k}\times d_{k}$ kernel (e.g., $d_{k}=3$)
and the output response $Y$is obtained by enforcing linear transformation
${\mathbf{W}}$ on feature map $X$.

The local response normalization (LRN) layer performs “lateral inhibition”
based on the fact that the activated neurons will have impact on those neurons
in its local input regions. Therefore, it usually performs normalizing over
local input regions from $\Re^{M\times N\times K}\rightarrow\Re^{M\times
N\times K^{\prime}}$, i.e.,

(19) $\displaystyle f_{\text{LRN}}:X\mapsto
Y,\;\;\;Y_{ijk^{\prime}}=\frac{X_{ijk}}{\left(\kappa+\alpha\sum_{k\in
G(k^{\prime})}X_{ijk}^{2}\right)^{\beta}},$

where
$G(k)=\left[k-\lfloor\frac{\rho}{2}\rfloor,k+\lceil\frac{\rho}{2}\rceil\right]\cap\\{1,2,\dots,K\\}$
is a group of $\rho$ consecutive feature channels in the input map. Clearly,
if $\kappa=0,\alpha=1,\beta=1$, this gives $\ell_{2}$ normalization. A batch
normalization operation (Ioffe and Szegedy, 2015) is usually applied to change
the distributions of activations to avoid “Internal covariate shift”. During
SGD training, each activation of the mini-batch is centered to zero-mean and
unit variance where the mean and variance are measured over the whole mini-
batch. Then a learned offset $\beta$ and multiplicative factor $\gamma$ are
then applied, i.e., given values of $X$ over a mini-batch:
$\\{X^{1},X^{2},\cdots,X^{m}\\}$, batch normalization projects them into
$\\{Y^{1},Y^{2},\cdots,Y^{m}\\}$ using the following steps:

$\displaystyle\mu_{{\mathbf{B}}}\leftarrow\frac{1}{m}\sum_{i=1}^{m}X^{i};\;\;\;\sigma^{2}_{{\mathbf{B}}}\leftarrow\frac{1}{m}(X^{i}-\mu_{{\mathbf{B}}})^{2};\;\;\;$
(20)
$\displaystyle\hat{X^{i}}\leftarrow\frac{X^{i}-\mu_{{\mathbf{B}}}}{\sqrt{\sigma^{2}_{{\mathbf{B}}}+\epsilon}};\;\;\;Y^{i}\leftarrow\gamma\hat{X^{i}}+\beta;$

where $\mu_{{\mathbf{B}}}$ and $\sigma^{2}_{{\mathbf{B}}}$ are the mean and
variance of the data in the mini-batch.

Then a pooling operator operates on individual feature channels, coalescing
nearby feature values into one by the application of a suitable operator such
as max pooling from $\Re^{M\times N\times K}\rightarrow\Re^{M^{\prime}\times
N^{\prime}\times K^{\prime}}$, i.e.,

$\displaystyle f_{\text{Pooling}}:X\mapsto
Y,\;\;\;Y_{ijk}=\max\\{Y_{i^{\prime}j^{\prime}k}:i\leq i^{\prime}<i+p,j\leq
j^{\prime}<j+p\\},$

where $p$ denotes the nearby $p$ regions222Sum pooling can be similarly done..

To achieve the desired functionality with acceleration, the idea is to find a
mapping function $\mathcal{F}:X\in\Re^{M\times N\times K}\rightarrow
Y\in\Re^{M^{\prime\prime}\times N^{\prime\prime}\times K^{\prime}}$ such that
it can get the same feature map value $Y^{i}$ given the same input feature map
$X^{i}$ for any image $i$. Recall that convolution operation can be viewed as
enforcing linear transformation $W$ on the input feature maps in the fully
connected layers, and therefore we aim to build a single convolution operation
($*$) that replaces several non-tensor layers by setting a new optimization
goal, i.e.,

(21) $\displaystyle\forall
i:\;\;\;Y^{i}=Y_{\text{COM}}^{i};\;\;\;Y_{\text{COM}}^{i}\simeq\hat{W}*X^{i}+\hat{b};$

While the type and sequence of functions is usually handcrafted, the
parameters $W$ and bias $b$ can be learned from our experiments for solving a
least square problem using SGD, i.e.,

(22)
$\displaystyle(\hat{W^{*}},\hat{b^{*}})=argmin_{\hat{W},\hat{b}}\sum_{i}\|Y_{\text{COM}}^{i}-(\hat{W}*X^{i}+\hat{b})\|^{2},$

Note here $\hat{W^{*}}\in\Re^{d_{s}\times d_{s}\times K\times K^{\prime}}$,
the size of which is quite similar to the original convolution steps except
new kernel size changes to $d_{s}\times d_{s}$ by considering pooling
operations.

##  4\. Cloud and Device-AI interactions

Figure 16. Key component in “personal intelligent assistant”.

A typical example is “Intelligent personal assistant”. The ideal personal
assistant can talk to you in speech conversation, understand your words, get
information for you and even do something for you (such as write a letter).
The major several products in markets include Amazon
Echo333https://en.wikipedia.org/wiki/Amazon_Echo, Google
Home444https://en.wikipedia.org/wiki/Google_Home, Apple’s
Siri555https://en.wikipedia.org/wiki/Siri, and Microsoft
Cortana666https://en.wikipedia.org/wiki/Cortana. For example, Amazon Alexa has
been sold more than 25 millions since
2015777https://en.wikipedia.org/wiki/Amazon_Alexa. The personal assistant can
work in the following scenarios:

  * •

Online chat (such as some instant message app

  * •

Speech and voice recognition

  * •

Taking and uploading images.

The personal assistant can provide a wide variety of services, such as
providing weather information, playing music from
Spotify888https://www.spotify.com/us/, playing videos, buying items from
Amazon, completing the customer service tasks, etc. Technically, personal
assistant needs six major components to support all its functionalities, i.e.,

  * •

Speech Recognition

  * •

Speech Synthesis

  * •

Natural Language Understanding (NLU)

  * •

Chatbot

  * •

Dialog system

  * •

Recommender System

which will be illustrated in detail in the following subsections.

###  4.1. Speech recognition

Essentially, speech recognition component learns a function $\hat{g}$ which
can automatically label the speech signal to its corresponding labels (such as
transcript) in a structured-input structured-output way, i.e.,

(23) $\displaystyle\hat{g}:\text{speech signal}\rightarrow\text{label of
signal}$

The goal is to learn function $\hat{g}$ with high accuracy. $\hat{g}$ can be
modeled using HMM, LSTM and other deep learning models.

The tradition phonetic-based (e.g., HMM-based method) (Gales and Young, 2007)
required feature engineer (e.g. n-gram) and separate training component for
pronunciation, acoustic and language model. The benefit of the current end-to-
end deep learning pipeline essentially jointly learns different components of
speech recognizer, which facilitates the deep learning training and deployment
on mobile device. Attention-based automatic speech recognition models (a.k.a
”Listen, Attend and Spell”) (Chorowski et al., 2015) based on deep neural
network can literally listen to the acoustic signal, pay attention to
different parts of the signal and spell out the transcript one character at a
time.

From product perspective, the automatic speech recognition (ASR) module needs
the support of the far-field technology, speaker adaption, noise filtering
techniques (Li et al., 2014) to make the ASR system work in practice. For
example, far-field E (electric) and B (magnetic) field strength decreases
inversely with distance from the source, resulting in an inverse-square law
for the radiated power intensity of electromagnetic radiation.

Figure 17. The trigged functionalities provided by the most popular personal
assistant: Amazon Alexa. The Alexa and app pictures are obtained from
internet.

###  4.2. Speech Synthesis

Speech synthesis (Taylor, 2009) aims to synthesize the human speech from the
textual descriptions or from symbolic linguistic representations. A typical
text-to-speech (TTS) system (or ”engine”) (Dutoit, 1997) has several
components, and the learned function $\hat{h}$ can automatically generate the
speech accurately given the input text, i.e.,

(24) $\displaystyle\hat{h}:\text{text characters}\rightarrow\text{speech}$

The front end first performs tokenization and normalization, which converts
raw text (including numbers and abbreviations) into the written-out words.
Then each word was assigned the phonetic transcription, which divides the text
into prosodic units such as clauses, phrases, and sentences. This actually
finishes the text-to-phoneme conversion. After this process, the output is the
symbolic linguistic representation consisting of phonetic transcriptions and
prosody information.

The next thing concerns how to convert the symbolic linguistic representation
into sound. The general way is to compute the target prosody (pitch contour,
phoneme durations) on the output speech.

It seems the process is very complicated. Fortunately many APIs from big
giants (e.g., Apple Siri, AT&T) have been offered to accelerate the
development process.

###  4.3. Natural Language understanding (NLU)

Natural language understanding module processes the natural language input
using disassembling and parsing techniques (Huang et al., 2001). Given the
utterance, the system needs to identify the proper name, part-of-speech (POS),
named entity and finally parses it into the object and predicator sets.
Natural language understanding is performed from syntax level, semantic level
to pragmatic level of understanding in linguistic analysis, which lays
foundation for understanding the sentiment and emotions (Liang et al., 2006),
uncovering insights from structured and unstructured data. The three most
important features desired for NLU are:

  * •

Proper name identification: identify the “proper name” from utterance

  * •

Part of speech tagging: labelling the part of speech as a category of words
that have similar grammatical properties. For example, in english, the labels
are noun, verb, adjective, adverb, pronoun, preposition, conjunction,
interjection, etc.

  * •

Syntactic/semantic parser: taking input data and building the data structure -
in the format of parse tree, such as abstract syntax tree or other
hierarchical structure to represent the input and check the correctness of
syntax or semantics.

###  4.4. Chatbot

Chatbot is a computer program that can conduct the conversation using textual
or audio methods with human being. The chatbot simulates how a human would
behave in a conversation, which is widely used in a dialogue system or
customer service for practical purpose. The simple chatbot uses keyword-based
matching for the input, and then queries the answers from database using
keyword matching from the prior knowledge database. Nowadays, more
sophisticated natural language processing techniques (e.g., recurrent neural
networks/LSTM) are used in current Chatbot systems (Serban et al., 2017).

###  4.5. Dialog System

The dialog system (Lowe et al., 2015) intends to have conversations with human
like a conversion agent from employing text, speech, graphics, haptics,
gestures and others for communication.

One typical dialog system using reinforcement learning first triggers the
language understanding component to understand the use input, and then dialog
manager queries the dialog policy, e.g.,

(25) $\displaystyle a=\pi(s)$

where $a$ is the action given the current sate $s$ distribution $\pi(s)$, then
it collects the rewards using

(26) $\displaystyle\mathcal{L}=(s,a,r,s^{\prime}),$

where $r$ is the reward and $s^{\prime}$ is the state after transition. The Q
function can be induced by applying Q-learning updates over mini batches such
that $Q$-function is optimized using

(27) $\displaystyle\max Q(s,a),$

before making a decision on the dialog policy. After the dialogue policy is
determined, the dialog system manages the general flow of the conversation
based on the history and state of the dialog, and produces output using the
output generator, including natural language generator and trigger text-to-
speech engine (TTS) as well. When the $Q$-function is learned, to achieve the
better performance, using multilayer deep learning (e.g., GoogLeNet (Szegedy
et al., 2015)) networks, we in fact, adopt deep reinforcement learning (Li et
al., 2016b) to build the more robust and accurate system.

###  4.6. Recommender System

A typical way is to incorporate more information into both Item features and
user feature modeling process, such as the content information of the items
(including title, artists, genre, year, etc), user demographics (including
income, age, gender), geo-location, social network profile and other relevant
features. Given millions of features and data samples, a natural way is using
distributed big machine learning framework to derive the corresponding
solutions using “learning to ranking” functions, i.e.,

$\displaystyle\hat{f}:(\text{visit, item title, user
income,...})\rightarrow\text{likelihood of users' like or dislike}$

Google implemented this type of recommendation system using deep and wide
recommendation (Cheng et al., 2016).

Summary Imagine that one can easily build his own personal assistant, given
the far-field recognition technique on device, Amazon cloud service (AWS),
state-of-the-art speech recognition and natural language understanding
techniques, chatbot and dialog system, recommender techniques. It is the great
opportunity for everyone.

##  5\. Conclusion

This paper presents our research efforts on mobile data science, which
provides a scientific approach to drive innovations on different mobile AI
applications on both cloud side and device side. The paper presents very
detailed case studies regarding how to apply machine learning and optimization
techniques to solve the real-world challenging problem on mobile devices. We
are delivering more intelligent mobile innovations for mobile AI applications.
Stay tuned for our future works.

Acknowledgement  The majority of paper contents are based on the author’s
_published_ papers. Thanks for all co-authors who have contributed to this
work as listed in references, and I really appreciate their strong support and
help. Any opinions, findings or recommendations expressed in this material are
those of the authors and do not necessarily reflect the views of any company.
Any plagiarism of this work is forbidden (especially for bad guys).

## References

  * (1)
  * Cen et al. (2015) Lei Cen, Deguang Kong, Luo Si, and etc. 2015\.  Mobile App Security Risk Assessment: A Crowdsourcing Ranking Approach from User Comments. In Proceedings of the 2015 SIAM International Conference on Data Mining, Vancouver, BC, Canada, April 30 - May 2, 2015. 658–666.  https://doi.org/10.1137/1.9781611974010.74
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016\.  Wide & Deep Learning for Recommender Systems.  CoRR abs/1606.07792 (2016).  arXiv:1606.07792 http://arxiv.org/abs/1606.07792
  * Chorowski et al. (2015) Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015\.  Attention-based Models for Speech Recognition. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 (NIPS’15). MIT Press, Cambridge, MA, USA, 577–585.  http://dl.acm.org/citation.cfm?id=2969239.2969304
  * Dutoit (1997) Thierry Dutoit. 1997\.  An Introduction to Text-to-speech Synthesis.  Kluwer Academic Publishers, Norwell, MA, USA.
  * Gales and Young (2007) Mark Gales and Steve Young. 2007.  The Application of Hidden Markov Models in Speech Recognition.  Found. Trends Signal Process. 1, 3 (Jan. 2007), 195–304.  https://doi.org/10.1561/2000000004
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative Adversarial Nets.  In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 2672–2680.  http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf
  * He et al. (2016) Jianping He, Bin Liu, Deguang Kong, Xuan Bao, and etc. 2016.  PUPPIES: Transformation-Supported Personalized Privacy Preserving Partial Image Sharing. In 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2016, Toulouse, France, June 28 - July 1, 2016. 359–370.  https://doi.org/10.1109/DSN.2016.40
  * Hu et al. (2015) Bing Hu, Bin Liu, Neil Zhenqiang Gong, Deguang Kong, and etc. 2015.  Protecting Your Children from Inappropriate Content in Mobile Apps: An Automatic Maturity Rating Framework. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management, CIKM 2015, Melbourne, VIC, Australia, October 19 - 23, 2015. 1111–1120.  https://doi.org/10.1145/2806416.2806579
  * Huang et al. (2001) Xuedong Huang, Alex Acero, and Hsiao-Wuen Hon. 2001\.  Spoken Language Processing: A Guide to Theory, Algorithm, and System Development (1st ed.).  Prentice Hall PTR, Upper Saddle River, NJ, USA.
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), David Blei and Francis Bach (Eds.). JMLR Workshop and Conference Proceedings, 448–456.  http://jmlr.org/proceedings/papers/v37/ioffe15.pdf
  * Kong (2017) Deguang Kong. 2017\.  Context aware recommendation via bilinear tensor factorization. In Technical Report.
  * Kong and Cen (2017) Deguang Kong and Lei Cen. 2017.  Mobile App Risk Assessment using Multi-modal Learning. In Technical Report.
  * Kong et al. (2015) Deguang Kong, Lei Cen, and etc. 2015.  AUTOREB: Automatically Understanding the Review-to-Behavior Fidelity in Android Applications. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, CO, USA, October 12-6, 2015. 530–541.  https://doi.org/10.1145/2810103.2813689
  * Kong and Ding (2013) Deguang Kong and Chris H. Q. Ding. 2013.  Efficient Algorithms for Selecting Features with Arbitrary Group Constraints via Group Lasso. In 2013 IEEE 13th International Conference on Data Mining, Dallas, TX, USA, December 7-10, 2013. 379–388.  https://doi.org/10.1109/ICDM.2013.168
  * Kong and etc. (2015) Deguang Kong and etc. 2015\.  Towards Permission Request Prediction on Mobile Apps via Structure Feature Learning. In Proceedings of the 2015 SIAM International Conference on Data Mining, Vancouver, BC, Canada, April 30 - May 2, 2015. 604–612.  https://doi.org/10.1137/1.9781611974010.68
  * Kong et al. (2014) Deguang Kong, Ryohei Fujimaki, Ji Liu, Feiping Nie, and Chris H. Q. Ding. 2014.  Exclusive Feature Learning on Arbitrary Structures via \ell_{1, 2}-norm. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada. 1655–1663.  http://papers.nips.cc/paper/5631-exclusive-feature-learning-on-arbitrary-structures-via-ell_12-norm
  * Kong et al. (2016) Deguang Kong, Ji Liu, Bo Liu, and Xuan Bao. 2016\.  Uncorrelated Group LASSO. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA. 1765–1771.  http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12528
  * Kong and Yan (2013) Deguang Kong and Guanhua Yan. 2013.  Discriminant malware distance learning on structural information for automated malware classification. In The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA, August 11-14, 2013. 1357–1365.  https://doi.org/10.1145/2487575.2488219
  * Kong and Yan (2014) Deguang Kong and Guanhua Yan. 2014.  Transductive malware label propagation: Find your lineage from your neighbors. In 2014 IEEE Conference on Computer Communications, INFOCOM 2014, Toronto, Canada, April 27 - May 2, 2014. 1411–1419.  https://doi.org/10.1109/INFOCOM.2014.6848075
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  ImageNet Classification with Deep Convolutional Neural Networks.  In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097–1105.  http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
  * Li et al. (2016) Bo Li, Yining Wang, Aarti Singh, and Yevgeniy Vorobeychik. 2016\.  Data Poisoning Attacks on Factorization-Based Collaborative Filtering.  In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, Inc., 1885–1893.  http://papers.nips.cc/paper/6142-data-poisoning-attacks-on-factorization-based-collaborative-filtering.pdf
  * Li and Chuah (2015) Dawei Li and Mooi Choo Chuah. 2015.  EMOD: an efficient on-device mobile visual search system. In Proceedings of the 6th ACM Multimedia Systems Conference. ACM, 25–36.
  * Li et al. (2016c) Dawei Li, Theodoros Salonidis, Nirmit V Desai, and Mooi Choo Chuah. 2016c.  DeepCham: Collaborative Edge-Mediated Adaptive Deep Learning for Mobile Object Recognition. In Edge Computing (SEC), IEEE/ACM Symposium on. IEEE, 64–76.
  * Li et al. (2018) Dawei Li, Xiaolong Wang, and Deguang Kong. 2018.  DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices. In AAAI.
  * Li et al. (2014) Jinyu Li, Li Deng, Yifan Gong, and Reinhold Haeb-Umbach. 2014\.  An Overview of Noise-robust Automatic Speech Recognition.  IEEE/ACM Trans. Audio, Speech and Lang. Proc. 22, 4 (April 2014), 745–777.  https://doi.org/10.1109/TASLP.2014.2304637
  * Li et al. (2016b) Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016b.  Deep Reinforcement Learning for Dialogue Generation.  CoRR abs/1606.01541 (2016).  arXiv:1606.01541 http://arxiv.org/abs/1606.01541
  * Li et al. (2016a) Ninghui Li, Min Lyu, Dong Su, and Weining Yang. 2016a.  Differential Privacy: From Theory to Practice.  Morgan & Claypool Publishers.  https://doi.org/10.2200/S00735ED1V01Y201609SPT018
  * Liang et al. (2006) Percy Liang, Alexandre Bouchard-Côté, Dan Klein, and Ben Taskar. 2006.  An End-to-end Discriminative Approach to Machine Translation. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics (ACL-44). Association for Computational Linguistics, Stroudsburg, PA, USA, 761–768.  https://doi.org/10.3115/1220175.1220271
  * Liu et al. (2015) Bin Liu, Deguang Kong, Lei Cen, Neil Zhenqiang Gong, and etc. 2015.  Personalized Mobile App Recommendation: Reconciling App Functionality and User Privacy Preference. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM 2015, Shanghai, China, February 2-6, 2015. 315–324.  https://doi.org/10.1145/2684822.2685322
  * Lowe et al. (2015) Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015\.  The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems.  CoRR abs/1506.08909 (2015).  arXiv:1506.08909 http://arxiv.org/abs/1506.08909
  * Mccaffrey et al. (2004) Daniel F. Mccaffrey, Greg Ridgeway, and Andrew R. Morral. 2004\.  Propensity Score Estimation with Boosted Regression for Evaluating Causal Effects in Observational Studies.  Psychological Methods 9 (2004), 403–425.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013.  Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States. 3111–3119.  http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality
  * Rupert G. Miller (2001) Jr. Rupert G. Miller. 2001\.  Survival Analysis, 2nd Edition. In 978-1-118-03106-3.
  * Serban et al. (2017) Iulian Vlad Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang, Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, Sarath Chandar, Nan Rosemary Ke, Sai Mudumba, Alexandre de Brébisson, Jose Sotelo, Dendi Suhubdy, Vincent Michalski, Alexandre Nguyen, Joelle Pineau, and Yoshua Bengio. 2017\.  A Deep Reinforcement Learning Chatbot.  CoRR abs/1709.02349 (2017).  arXiv:1709.02349 http://arxiv.org/abs/1709.02349
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going Deeper with Convolutions. In Computer Vision and Pattern Recognition (CVPR).  http://arxiv.org/abs/1409.4842
  * Taylor (2009) Paul Taylor. 2009\.  Text-to-Speech Synthesis (1st ed.).  Cambridge University Press, New York, NY, USA.
  * Tran et al. (2016) Lam Tran, Deguang Kong, and Ji Liu. 2016.  Privacy-CNH: A Framework to Detect Photo Privacy with Convolutional Neural Network using Hierarchical Features. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA. 1317–1323.  http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12232
  * Wang et al. (2015) Ruowen Wang, William Enck, Douglas S. Reeves, Xinwen Zhang, Peng Ning, Dingbang Xu, Wu Zhou, and Ahmed M. Azab. 2015\.  EASEAndroid: Automatic Policy Analysis and Refinement for Security Enhanced Android via Large-Scale Semi-Supervised Learning. In 24th USENIX Security Symposium, USENIX Security 15, Washington, D.C., USA, August 12-14, 2015. 351–366.  https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/wang-ruowen
  * Wang et al. (2017) Xiaolong Wang, Yin Zhou, Deguang Kong, Jon Currey, Dawei Li, and Jiayu Zhou. 2017\.  Unleash the Black Magic in Age: A Multi-Task Deep Neural Network Approach for Cross-Age Face Verification. In 12th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2017, Washington, DC, USA, May 30 - June 3, 2017. 596–603.  https://doi.org/10.1109/FG.2017.75
  * Yang et al. (2017) Wei Yang, Deguang Kong, Tao Xie, and Carl A. Gunter. 2017\.  Malware Detection in Adversarial Se ings: Exploiting Feature Evolutions and Confusions in Android Apps. In ACSAC.