# Hybrid Encoder: Towards Efficient and Precise Native Ads Recommendation via
Hybrid Transformer Encoding Networks

Junhan Yang  University of Science and Technology of China
<EMAIL_ADDRESS>,  Zheng Liu  Microsoft Research Asia
<EMAIL_ADDRESS>,  Bowen Jin  Tsinghua University
<EMAIL_ADDRESS>,  Jianxun Lian  Microsoft Research Asia
<EMAIL_ADDRESS>,  Defu Lian  University of Science and Technology
of China<EMAIL_ADDRESS>,  Akshay Soni  Microsoft
<EMAIL_ADDRESS>,  Eun Yong Kang  Microsoft<EMAIL_ADDRESS>,
Yajun Wang  Microsoft<EMAIL_ADDRESS>,  Guangzhong Sun  University of
Science and Technology of China<EMAIL_ADDRESS>and  Xing Xie  Microsoft
Research Asia<EMAIL_ADDRESS>
###### Abstract.

Transformer encoding networks have been proved to be a powerful tool of
understanding natural languages. They are playing a critical role in native
ads service, which facilitates the recommendation of appropriate ads based on
user’s web browsing history. For the sake of efficient recommendation,
conventional methods would generate user and advertisement embeddings
independently with a siamese transformer encoder, such that approximate
nearest neighbour search (ANN) can be leveraged. Given that the underlying
semantic about user and ad can be complicated, such independently generated
embeddings are prone to information loss, which leads to inferior
recommendation quality. Although another encoding strategy, the cross encoder,
can be much more accurate, it will lead to huge running cost and become
infeasible for realtime services, like native ads recommendation.

In this work, we propose hybrid encoder, which makes efficient and precise
native ads recommendation through two consecutive steps: retrieval and
ranking. In the retrieval step, user and ad are encoded with a siamese
component, which enables relevant candidates to be retrieved via ANN search.
In the ranking step, it further represents each ad with disentangled
embeddings and each user with ad-related embeddings, which contributes to the
fine-grained selection of high-quality ads from the candidate set. Both steps
are light-weighted, thanks to the pre-computed and cached intermedia results.
To optimize the hybrid encoder’s performance in this two-stage workflow, a
progressive training pipeline is developed, which builds up the model’s
capability in the retrieval and ranking task step-by-step. The hybrid
encoder’s effectiveness is experimentally verified: with very little
additional cost, it outperforms the siamese encoder significantly and achieves
comparable recommendation quality as the cross encoder.

Native Ads Recommendation, User and Ad Understanding, Transformer Encoding
Networks

##  1\. Introduction

Native ads are important ways of online advertising, where advertisements are
imperceptibly injected into the webpages being browsed by the users. In
contrast to the conventional search ads where queries are explicitly specified
by user, native ads are recommended implicitly based on user’s web-browsing
history. Apparently, high-quality recommendation calls for deep understanding
of user behavior and ad content. In recent years, transformer-based models
turn out to be popular solutions due to their superior capability on natural
language processing, e.g., Bert (Devlin et al., 2018) and RoBerta (Liu et al.,
2019). The existing methods would represent a pair of input sequences
independently with a siamese encoder (Reimers and Gurevych, 2019; Lu et al.,
2020) (as the two-tower architecture in Figure 1.A), and measure their
relevance based on the embeddings’ similarity, such as cosine and inner-
product. With these approaches, relevant ads can be efficiently acquired via
ANN search, like PQ (Jegou et al., 2010; Ge et al., 2013) and HNWS (Malkov and
Yashunin, 2018). However, one obvious defect is that there is no interaction
between the user and ad encoding process. Given that the underlying semantic
about user and ad can be complicated, such independently generated embeddings
is prone to information loss, thus cannot reflect the user-ad relationship
precisely. In contrast, another form of transformer encoding networks, the
cross-encoder (as the one-tower structure in Figure 1.B), can be much more
accurate (Luan et al., 2020; Lee et al., 2019), where the input sequences may
fully attend to each other during the encoding process. Unfortunately, the
cross-encoder is highly time-consuming and hardly feasible for realtime
services like native ads recommendation, as every pair of user and ad needs to
be encoded and compared from scratch. The above challenges bring us to a
challenging problem: whether some sort of encoding architecture can be both
efficient and precise. Motivated by this challenge, we propose hybrid encoder,
which addresses the above problem with the following featurized designs.

$\bullet$ Two-stage workflow. Instead of making recommendation in one shot,
hybrid encoder selects appropriate ads through two consecutive steps:
retrieval and ranking, with two sets of user and ad representations. First of
all, it represents user and ad independently with a siamese encoder, such that
relevant candidates can be retrieved via ANN search. Secondly, it generates
disentangled ad embeddings and ad-related user embeddings, based on which
high-quality ads can be identified from the candidate set. Both steps are
highly efficient, which makes the time cost affordable in practice.

$\bullet$ Fast ranking. Unlike the conventional time-consuming cross-encoder,
hybrid-encoder establishes the user-ad interaction by intensively re-using the
cached hidden-states and pre-computation result. On the one hand, it caches
the transformer’s last-layer hidden-states for the user, which will
comprehensively preserve the information about user’s web-browsing behaviors.
On the other hand, it pre-computes the disentangled embeddings for each ad.
While an ad is chosen as a user’s candidate, the disentangled ad embeddings
will attend to the user’s cached hidden-states, which gives rise to the “ad-
related” user embeddings. With such embeddings, necessary information for
analyzing user-ad relationship can be effectively extracted, which in return
contributes to the selection of best ads from the candidate set. Apparently,
most of the involved operations are light-weighted, e.g., inner-product and
summation, whose time cost is almost neglectable compared with the feed-
forward inference of transformer. Therefore, the running time of the ranking
step is approximately up-bounded by one pass of transformer encoding.

$\bullet$ Progressive Training Pipeline. Although both the retrieval and
ranking step aim to find out appropriate advertisements w.r.t. the presented
user, they are actually deployed for distinct purposes, thus need to be
learned in different ways. In this work, a progressive learning strategy is
adopted to optimize the overall performance of the two-stage workflow. For one
thing, the retrieval step is to select coarse-grained candidates from the
whole advertisement corpus; therefore, it is trained with global contrast
learning, which discriminates the ground-truth from the negative ones sampled
from the global scope. For another thing, the ranking step is to select the
best advertisements from the candidate set generated by a certain retrieval
policy. Therefore, the model is further trained with the local contrast
learning, where a set of candidate advertisements are sampled based on the
retrieval policy learned in the first stage, and the model is required to
identify the ground-truth given such “hard negative samples”.

Comprehensive empirical investigation are performed on Microsoft Audience
Ads111https://about.ads.microsoft.com/en-us/solutions/microsoft-audience-
network/microsoft-audience-ads platform, where the hybrid encoder’s
effectiveness is verified from the following aspects. I. It achieves
comparable recommendation quality as the accurate but expensive cross-encoder,
given that the running time is sufficiently large. II. It significantly
outperforms the conventional siamese encoder when the time cost is limited
within a tolerable range as required by the production. Additional studies are
also made with public-available dataset, which further confirms the above
findings.

To sum up, the following contributions are made in this work.
$\bullet$ We propose hybrid encoder, which recommends native ads with a two-
stage workflow: the retrieval step collects coarse-grained candidates from the
whole corpus; the ranking step selects the best ads from the given candidates.
Both steps are highly efficient, which collaboratively produce precise
recommendation for the user.
$\bullet$ The proposed model is trained progressively through global contrast
learning and local contrast learning, which optimizes the overall performance
of the two-stage recommendation workflow.
$\bullet$ The effectiveness of hybrid encoder is verified with large-scale
data from native ads production. Compared with conventional transformer
encoders, hybrid encoder is competitive in terms of both recommendation
precision and working efficiency.

Figure 1. Illustration of conventional transformer encoders. (A) siamese
encoder: user’s browsed webs and ad’s description are encoded independently,
whose relevance is calculated as the embedding similarity. (B) cross encoder:
user’s browsed webs and ad’s description are concatenated into one token list
and jointly encoded by a unified transformer; the relevance is computed based
on final hidden-states, usually the one corresponding to [CLS] token.

##  2\. Related Work

In this section, related works are discussed from the following perspectives:
1) native ads recommendation, and 2) transformer encoding networks.

###  2.1. Native Ads Recommendation

Native ads is a relatively new form of online advertising, where
advertisements are recommended to user based on their web-browsing behaviors.
A typical recommendation workflow is composed of the following operations.
Particularly, each advertisement is characterized with a compact description,
e.g., new apple ipad mini. When an advertising request is presented, it turns
user’s browsed web-pages into “implicit queries” and finds out the most
relevant ads-descriptions. Finally, the selected ads will be displayed
imperceptibly on the web-pages being browsed by the user. In the early days,
the recommendation process would intensively make use of the retrieval
algorithms built on explicit features: each ads-description is further split
into a set of keywords, based on which an inverted index can be built to
organize the whole advertisements. To make the index more accurate and
compact, conventional IR techniques like TF-IDF and BM25 are adopted so as to
identify the most important keywords for index (Robertson and Zaragoza, 2009;
Manning et al., 2008). Given an implicit query from user, the system will
extract its keywords in the first place, and then join with the inverted index
so as to collect the advertisements within the matched entries.

Apparently, the old-fashioned approaches can be severely limited by the ratio
of keyword-overlap. To overcome such a defect, more and more effort has been
made on latent retrieval (Fan et al., 2019; Huang et al., 2020), where the
semantic relationship between user and ad can be measured more precisely with
compact embeddings. Early representative works along this direction includes
DSSM & CDSSM (Huang et al., 2013; Shen et al., 2014), where multi-layer-
perception and convolutional neural networks are adopted; in recent years,
much more advanced approaches are continuously developed, especially based on
transformer encoders and pretrained language models, such as (Reimers and
Gurevych, 2019; Lu et al., 2020; Karpukhin et al., 2020; Guu et al., 2020; Lee
et al., 2019).

###  2.2. Transformer Encoding Networks

Transformer is a powerful network structure for text encoding (Vaswani et al.,
2017), as long range interactions can be established for the input sequence on
top of multi-head self-attention. In recent years, multi-layer transformer
encoders have become the foundation of almost every large-scale pretrained
language model, like BERT (Devlin et al., 2018) and RoBerta (Liu et al.,
2019), with which the underlying semantic can be captured in-depth for the
presented text. Given its superior capability in various NLP applications,
transformer is also used as the backbone network structure for text retrieval
(Reimers and Gurevych, 2019; Lu et al., 2020; Karpukhin et al., 2020).
Typically, a two-tower transformer encoding network (also known as siamese
encoder or bi-encoder) are deployed, which generates the latent
representations for the source and target text segments; after that, the
source and target relevance can be measured with the embedding similarity,
e.g., cosine or inner-product. The siamese encoder naturally fits the need of
large-scale retrieval/recommendation, as relevant targets can be efficiently
acquired for the given source via those well-known ANN paradigms, e.g., PQ
(Jegou et al., 2010; Ge et al., 2013) and HNSW (Malkov and Yashunin, 2018).
However, one obvious limitation about the siamese encoder is that the source
and target are encoded independently; in other words, there is no interaction
between the source and target’s encoding process. Such a problem may lead to
the ignoring of important information for the generated embeddings, which
harms the model’s accuracy. According to (Luan et al., 2020), this problem
will be more severe as the input sequence becomes longer.

An intuitive way of addressing the above problem is to make use of the one-
tower transformer encoder (Luan et al., 2020), usually referred as cross-
encoder, where the inputs are combined into one sequence and jointly processed
by a unified encoding networks. However, this architecture needs to encode
every pair of inputs from scratch for comparison, whose running time is
prohibitive for most of the realtime services, like native ads recommendation
(where hundreds or up-to-thousand comparisons need to be made within
millisecond-scale). Recently, it becomes more and more emphasized on how to
acquire accurate matching result in an temporally-efficient manner. One
promising option is to establish the interaction between inputs on top of the
transformer’s hidden-states (Humeau et al., 2019; Luan et al., 2020). For
example, in (Humeau et al., 2019), the hidden-states of the source sequence
are mapped into a total of $K$ embeddings called context vectors; the context
vectors will be further aggregated w.r.t. the target embedding via attentive
pooling; finally, the aggregation result will be compared with the target
embedding so as to reflect the source-target relevance. Our work also takes
advantage of the hidden-states. Unlike the conventional methods, the
interaction between inputs is established based on raw hidden-states of the
source and disentangled embeddings of the target, which turns out to be both
efficient and more effective.

##  3\. Methodology

###  3.1. Preliminaries

Transformer encoding networks have been proved to be a powerful tool for
natural language understanding, which significantly benefit downstream tasks,
like recommendation and information retrieval. Such networks consist of
multiple layers of transformer blocks (Vaswani et al., 2017), which are built
upon multi-head self-attention (MHSA). For each particular layer, the input
embeddings (used as $Q$, $K$ and $V$) are interacted with each other via the
following operations:

(1)
$\begin{gathered}\mathrm{MHSA}(Q,K,V)=\mathrm{Concat}(head_{1},...,head_{h})W^{O},\\\
head_{i}=\mathrm{Attention}(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}).\end{gathered}$

where $W^{O}$, $W_{i}^{Q}$, $W_{i}^{K}$, $W_{i}^{V}$ are the projection
matrices to be learned. In transformers, the scaled dot-product attention is
adopted, which computes the weighted-sum of $V$ with the following equation:

(2) $\mathrm{Attention}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d}})V,$

where $d$ indicates the value of hidden vector’s dimension. Apparently, MHSA
has the advantage of introducing long-range interaction for the input
sequence, which brings about stronger context-awareness for the generated
representations.

While applied for IR/recommendation-related tasks, transformer encoding
networks usually adopt the following two forms: the siamese encoder and the
cross encoder. In the following part, we will demonstrate how the above forms
of encoders are applied for the native ads recommendation scenario.

####  3.1.1. siamese Encoder

The siamese encoder leverages a pair of identical sub-networks to process the
user history and ads description, as shown in Figure 1 (A). Each sub-network
will aggregate the last layer’s hidden states for the representation of its
input sequence; as a result, two embeddings $E_{u}$ and $E_{a}$ will be
generated, which are of the same hidden dimension. The user-ad relevance is
measured based on the similarity between the embeddings:

(3) $Rel(u,a)=\langle E_{u},E_{a}\rangle,$

where $\langle\cdot\rangle$ stands for the similarity measurement, e.g.,
inner-product. The good thing about siamese encoder is that the user history
and ad description can be encoded once and used for relevance computation
everywhere. More importantly, the generated embeddings can be indexed for ANN-
search (e.g., MIPS (Jegou et al., 2010)), which naturally fits the need of
large-scale retrieval/recommendation. However, the siamese encoder is prone to
information loss, as user and ad are independently encoded. Therefore, it is
usually limited by its inferior accuracy.

Figure 2. Infrastructure of Hybrid Encoder. In the middle and left side, the
user embedding $E_{u}$ and ad embedding $E_{a}$ are generated independently;
both embeddings are used for the ANN search in the retrieval step. On the
middle and right side, the disentangled ad embeddings $M_{a}$ are offline
generated, which are used to attend user’s cached hidden-states for the ad-
related user embeddings; both embeddings are flattened into 1-D vecs and
compared to measure the fine-grained user-ad relationship.

####  3.1.2. Cross Encoder

The cross encoder makes use of a unified transformer to analyze the relevance
between the user and ad. In particular, the input sequences are concatenated
into the following joint string marked with [CLS] and [SEP]:

(4) $[CLS]t_{0}^{user},...,t_{M}^{user}[SEP]t_{0}^{ad},...,t_{N}^{ad},$

where $t_{\cdot}^{user}$ and $t_{\cdot}^{ad}$ indicate tokens from user’s
browsed web-pages and ad description, respectively. The joint string will be
encoded by the transformer. And based on the last layer’s hidden-state
corresponding to [CLS], denoted as $H_{[CLS]}$, the web-ad relevance is
calculated as:

(5) $Rel(user,ads)=\sigma(H_{[CLS]}W_{A}),$

where $W_{A}$ is the $d\times 1$ linear projection vector for the output
logit. The cross encoder is of much higher model capacity compared with the
siamese encoder, as user and ad may fully get attended by each other within
the encoding process. Therefore, cross encoder can be much more accurate, as
important information is preserved more effectively. However, the cross
encoder is highly limited in working efficiency, as every pair of user and ad
need to be encoded from scratch so as to find out the most relevant result.

###  3.2. Overview of Hybrid Encoder

####  3.2.1. Problem Formulation

We consider the scenario where natives ads recommended are based on user’s
web-browsing behaviors. In this place, the following definition is presented,
which clarifies the objective of native ads recommendation (NAR).

###### Definition 3.1.

(NAR) Given user’s web-browsing history, NAR looks for the advertisement $a$,
which gives rise to the maximum probability of being clicked.

It is apparent that the NAR problem can be quantitatively expressed by the
following optimization problem:

(6) $\max\mathbb{E}_{(u,a)\sim P(u,a)}\log\pi(a|u),$

where $P(u,a)$ stands for the joint probability of user-$u$-click-ad-$a$, and
$\pi(a|u)$ refers to the advertisement’s probability of being recommended by
NAR conditioned on the user’s web-browsing history. By optimizing the above
objective, the recommendation algorithm is expected to produce the maximum
amount of ad-clicks, which contributes to the revenue growth of the whole
market.

####  3.2.2. Two-Stage Processing

To facilitate efficient and precise native ads recommendation, a two-stage
workflow is employed, which contains the consecutive steps of retrieval and
ranking. The retrieval step is to select relevant candidate ads w.r.t. the
presented user. Given that the size of the whole advertisement set can be huge
(usually as large as a few millions), the retrieval steps must be highly
efficient. Thus, approximate nearest neighbour search turns out to be a
natural choice. To make use of it, user history and ad description are
independently encoded as $E_{u}$ and $E_{a}$, and the user-ad relevance is
measured with the embedding similarity. As a result, we’ll efficiently get the
candidate ads for the retrieval step via ANN search:

(7) $\Omega_{retr}=\mathrm{ANN}(E_{u},\\{E_{a}|\Omega_{A}\\})$

where $\\{E_{a}|\Omega_{A}\\}$ denotes the embeddings of whole ads. Following
the retrieval step, the ranking step is carried out so as to select the best
set of ads out of the coarse-grained candidates. For the sake of better
recommendation quality, a ranking function $f(\cdot)$ will be utilized, which
is able to establish the interaction between user and each of the candidate
ads. Finally, the fine-grained set of ads for recommendation can be determined
based on the ranking scores:

(8) $\Omega_{rank}=\mathrm{Top}\text{-}K\\{f(u,a)|\Omega_{retr}\\}$

####  3.2.3. Online Cost Analysis

The running time of the above two-stage processing is constituted of three
parts: the user and ad embedding generation, the ANN search, and the ranking
of candidates. However, because the in-market ads are temporally invariant and
given beforehand, the ad embeddings can all be generated offline; besides,
compared with the model’s inference cost, the ANN search is fast and its time
complexity is constant (or sub-linear). Therefore, the online cost will be
determined mainly by two factors: the generation of user embedding and the
ranking of candidates. The generation of user embedding takes one-pass of
transformer encoding. And with the conventional cross-encoder, it takes
$|\Omega_{retr}|$ rounds of transformer encoding to rank the candidates.
Finally, there will be $|\Omega_{retr}|+1$ rounds of transformer encoding in
total, which is still temporally infeasible for native ads production.
Apparently, the ranking step turns out to be the efficiency bottleneck of the
two-stage workflow. In the following discussion, a significantly accelerated
ranking operation will be introduced, where the time cost is reduced to
$\epsilon$ transformer encoding ($\epsilon\leq 1$). As a result, it will
merely take $1+\epsilon$ transformer encoding, which makes hybrid encoder
competitive in working efficiency.

###  3.3. Model Architecture

The architecture of hybrid encoder is shown in Figure 2, where multiple
transformers are jointly deployed for the hybrid encoding process. On the left
and middle side of the model, the deployed transformers take the form of a
siamese encoder, where user and ad embeddings, $E_{u}$ and $E_{a}$, are
generated independently. These embeddings are used for the ANN search of the
retrieval step. On the middle and right side, the disentangled ad embeddings
$M_{a}$ are generated, which may capture ad semantic more comprehensively;
meanwhile, the disentangled ad embeddings will attend to the cached hidden-
states of the ua-interaction network, which gives rise to a total of $|M_{a}|$
“ad-related” user embeddings $M_{u}$. Finally, the disentangled ad embeddings
and ad-related user embeddings are flatten into 1-D vectors and compared,
which helps to make fine-grained analysis of user-ad relationship for the
ranking step.

####  3.3.1. User and Ad Embedding Generation

The user’s browsed web-pages and ads’ descriptions are independently feed into
the user embedding network and the ad embedding network. Unlike the default
practice where the [CLS] token’s hidden-state in the last-layer is taken as
the representation result, we use the mean-pooling of all tokens’ hidden-
states in the last-layer as the user and ad embeddings, which is suggested to
be more effective (Reimers and Gurevych, 2019; Lu et al., 2020):

(9)
$\begin{gathered}E_{u}=\text{mean}(H^{u}),~{}where~{}H^{u}=\text{U-Net}(\text{user's
browsed webs});\\\
E_{a}=\text{mean}(H^{a}),~{}where~{}H^{a}=\text{A-Net}(\text{ad
description}).\end{gathered}$

As the user and ad embeddings need to support the ANN search operation in the
retrieval step, $E_{u}$ and $E_{a}$ must be within the same latent space,
whose similarity is directly comparable. As a result, the user-ad relevance
for the retrieval step is computed as:

(10) $Rel_{retr}(u,a)=\langle E_{u},E_{a}\rangle.$

In our implementation, we use inner-product as the similarity measure and
adopt HNSW for the search of approximate neighbours.

####  3.3.2. Fast Ranking with UA Interaction

The interaction between user and ad is established via ua-interaction network,
as shown on the right side of Figure 2. In this stage, the following three
operations are carried out step-by-step. First of all, it generates $K$
disentangled ad embeddings $M_{a}$ with multi-head attention (Lin et al.,
2017):

(11)
$\begin{gathered}M_{a}:\\{M_{a}^{i}=\sum\nolimits_{H^{a}}\alpha^{i}_{j}H^{a}_{j}|i=1,...,K\\},\\\
~{}where~{}\boldsymbol{\alpha}^{i}=\text{softmax}(H^{a},\boldsymbol{\theta}^{i}).\end{gathered}$

In the above equation, $\\{\boldsymbol{\theta}^{i}\\}_{1}^{K}$ stands for the
attention heads, which aggregate the hidden-states of the advertisement from
different views for more comprehensive representation. Secondly, each of the
disentangled ad embeddings is used to attend user’s last-layer hidden-states
$H^{ua}$ from the ua-interaction network. As a result,the following ad-related
user embeddings $M_{u}$ will be generated:

(12)
$\begin{gathered}M_{u}:\\{M_{u}^{i}=\sum\nolimits_{H^{ua}}\alpha^{i}_{j}H^{ua}_{j}|i=1,...,K\\},\\\
~{}where~{}\boldsymbol{\alpha}^{i}=\text{softmax}(H^{ua},M_{a}^{i}).\end{gathered}$

Finally, the disentangled ad embeddings $M_{a}$ and the ad-related user
embeddings $M_{u}$ are flattened and concatenated into 1-D vectors, whose
similarity is used as the measurement of user-ad relevance for the ranking
step:

(13)
$\begin{gathered}Rel_{rank}(u,a)=\langle\bar{M}_{u},\bar{M}_{a}\rangle,\\\
where~{}\bar{M}_{u}=\text{cat}([M_{u}^{1},...,M_{u}^{K}]),~{}\bar{M}_{a}=\text{cat}([M_{a}^{1},...,M_{a}^{K}]).\end{gathered}$

$\bullet$ Remarks. Compared with the expensive cross-encoding process, there
are two notable characters about the proposed ua-interaction: 1) the
interaction is uni-directional: with only user’s hidden-states attended by the
representation of ad. And 2) the interaction is introduced not from the very
bottom, but at the last-layer of the transformer. Despite of its simplicity,
the proposed method is desirable from the following perspectives.

First of all, it leverages the pre-computation result (i.e., the disentangled
ad embeddings) and the cached hidden-states (from ua-interaction network) to
establish interaction between user and ad. Therefore, it merely incurs one
pass of feed-forward inference of the ua-interaction network. Besides, it’s
empirically found that a relatively small-scale network (compared with the
user embedding network) may already be sufficient for making high-quality
interaction. Therefore, it only brings about $\epsilon$ ($\epsilon\leq 1$) of
the transformer encoding cost as the generation of user embedding, which makes
the ranking step highly competitive in time efficiency.

Secondly, the proposed ua-interaction is more effective in preserving the
underlying information of user and ad. On the one hand, the ad descriptions
are usually simple and compact; even without being interacted by the user, the
disentangled embeddings may still comprehensively represent the advertisement.
On the other hand, despite that user history can be complicated, the necessary
information for measuring user-ad relationship will be fully extracted when
attended by the disentangled ad embeddings. As a result, the relevance between
user and ad will be measured more precisely based on such informative
embeddings, which significantly benefits the identification of the best
candidates.

Figure 3. Illustration of the progressive training pipeline, where the blue
and red dots indicate the user embedding and the ground-truth ad embedding,
while the green triangles stands for the irrelevant ad embeddings. In the 1st
stage, the global contrast learning discriminates the ground-truth from the
global negatives, which leads to a relatively coarse-grained scope of
candidates (the yellow region); in the 2nd stage, the local contrast learning
identifies the ground-truth within the candidate scope, which pushes the user
embedding even closer to the ground-truth ad embedding.

###  3.4. Progressive Training Pipeline

Hybrid encoder is trained progressively with two consecutive steps (as Figure
3). Firstly, the global contrast learning is carried out for the user and ad
embedding networks; in this step, the model learns to discriminate the ground-
truth ad from the global negatives. As a result, it makes the ground-truth to
be confined within the neighbourhood of user embedding, which can be retrieved
via ANN-search. Secondly, the local contrast learning is performed for the ua-
interaction network, where the model continues to learn the selection of
ground-truth from the presented candidates. With this operation, the user and
ad representations (more specifically, the flattened ad-related user
embeddings and disentangled ad embeddings) will be further pushed together,
which improves the recommendation accuracy of the ranking step.

####  3.4.1. Global Contrast Learning

The global contrast learning is carried out in the form of binary
classification. Firstly, a positive user-ad case $(u,a)$ is sampled from the
production log; then, a total of $n$ negatives ads $\\{a^{\prime}_{i}\\}_{n}$
are sampled from the global ad corpus $A$. As a result, the following training
objective is formulated:

(14) $\begin{gathered}\mathrm{E}_{(u,a)\sim
P(u,a)}\Big{(}\log\big{(}\sigma(u,a)\big{)}-\sum\nolimits_{\\{a^{\prime}_{i}\\}_{n}\sim
A}\log\big{(}\sigma(u,a^{\prime}_{i})\big{)}\Big{)},\\\
~{}where~{}\sigma(u,a)=\text{exp}\big{(}Rel_{retr}(u,a)\big{)}.\end{gathered}$

With the optimization of the above objective, the user and ground-truth ad
embeddings will come close towards each other; meanwhile, the global
negatives’ embeddings will be pushed away from the user embedding. Therefore,
it enables the retrieval step to find out relevant candidate ads via ANN
search.

####  3.4.2. Local Contrast Learning

The local contrast learning is performed w.r.t. the well trained retrieval
strategy. For each positive $(u,a)$ sample, the negative cases are sampled
within the scope of candidates, i.e., the neighbour of user embedding $E_{u}$:
$\text{N}(E_{u})$, based on which the training objective is formulated as
follows:

(15) $\begin{gathered}\mathrm{E}_{(u,a)\sim
P(u,a)}\Big{(}\log\big{(}\sigma(u,a)\big{)}-\sum\nolimits_{\\{a^{\prime}_{i}\\}_{n}\sim\text{N}(E_{u})}\log\big{(}\sigma(u,a^{\prime}_{i})\big{)}\Big{)},\\\
~{}where~{}\sigma(u,a)=\text{exp}\big{(}Rel_{rank}(u,a)\big{)}.\end{gathered}$

Notice that $\text{N}(E_{u})$ does not necessarily mean the nearest neighbours
of $(E_{u})$. In fact, it is empirically found that the using the nearest
neighbours will lead to inferior training outcome, where the underlying
reasons are two-fold: firstly the nearest neighbours may give rise to a high
risk of false-negatives, i.e., the sampled ads are non-clicked but relevant to
user; secondly, the nearest neighbours can be monotonous, i.e., the sampled
negatives are probably associated with similar semantics, thus reducing the
information which can be learned form the negative samples. In our
implementation, we will firstly collect a large number of neighbours of
$E_{u}$, and then randomly sample a handful of cases as the training
negatives.

##  4\. Experimental Studies

Table 1. Specs about the datasets. #Train/Valid/Test: number of samples in
train/valid/test set; #Query: number of input queries (ads: #user, news:
#headline); #Target: number of recommendation targets (ads: #ad, news:
#annotation). Table 2. Basic Comparison section 4.2.1, I. & II.: results on
Native Ads and Google News, respectively. The hybrid encoder outperforms the
siamese encoder on both datasets, and it generates comparable performance as
the cross encoder on Native Ads dataset. Table 3. Hit-Time Ratio Analysis
section 4.2.2 (time measured in ms). The hybrid encoder achieves comparable
performance as the cross encoder, but it merely incurs small additional cost
compared with the siamese encoder.

###  4.1. Experiment Settings

The settings about the experimental studies are specified as follows.

$\bullet$ Datasets. The experimental studies are carried out based on
production data collected from Microsoft Audience Ads
platform222https://about.ads.microsoft.com/en-us/solutions/microsoft-audience-
network/microsoft-audience-ads. The dataset is composed of large-scale user-ad
pairs, in which users are characterized by their web-browsing behaviors, and
web-pages/advertisements are associated with their textual descriptions. As
discussed in the methodology, the recommendation algorithms are required to
predict user’s next-clicked ad based on her web-browsing behavior. The
training, validation and testing sets are strictly partitioned in time in
order to avoid data leakage: the training data is from 2020-05-01 to
2020-05-31, while the validation/testing data is from 2020-07-14 to
2020-08-11. The detailed specifications about the dataset are illustrated with
Table 1.

Although the proposed method is designed for native ads recommendation, it is
also necessary to explore its performance on more generalized scenarios, which
select semantic-close targets for the input query from a large-scale corpus.
In this place, the Google News dataset333https://github.com/google-research-
datasets/NewSHead is adopted for further evaluation. This dataset contains the
headlines of news articles (parsed from the provided urls) and their human
annotated descriptions, e.g., news headline “nba playoffs 2019 milwaukee bucks
vs toronto raptors conference finals schedule” and human annotation “NBA
playoff Finals”. In our experiment, the news headline is used as the input
query, to which the related human annotation needs to be selected from the
whole annotation corpus. The training and validation/testing set partition is
consistent with (Gu et al., 2020) where the dataset was originally created.

$\bullet$ Baselines. As discussed, the hybrid encoder is compared with the two
representative forms of transformer encoding networks (Luan et al., 2020). One
is the siamese-encoder, which is the conventional way of making use of
transformers for native ads recommendation (Lu et al., 2020); as hybrid
encoder already includes a siamese component, we’ll directly take that part
out of the well-trained hybrid encoder for testing such that impact from
inconsistent settings can be excluded. The other one is the cross-encoder,
which is the most accurate but temporally infeasible; the cross encoder is
also learned through the same training workflow as the hybrid encoder’s
ranking component section 3.4.2, thus we can fairly compare the effect
resulted from model architecture. The computation of user-ad relevance with
both encoders is consistent with the previous introduction made in section
3.3.1 and section 3.1. In our experiments, we adopt BERT-base (Devlin et al.,
2018), the 12-layer-768-dim bi-directional pretrained transformer, as our
backbone structure. Besides, as BERT-base is still a too large-scale model for
realtime service, another model based on a light-weighted 2-layer-128-dim
distilled BERT is trained for production usage.

$\bullet$ Performance Evaluation. The quality of natives ads recommendation is
evaluated in terms of Hit@N, which reflects how likely the ground-truth ad can
be included within the top-N recommendation. To focus on the effect resulted
from each model itself, the top-N recommendation is made based on the same
two-stage workflow as discussed in section 3.2.2, where a shared set of
candidate ads (a total of $k$N, which equals to 100 by default) are provided
from approximate nearest neighbour search. As introduced in section 3.3.1, the
ANN search is performed with HNSW444https://github.com/nmslib/hnswlib, where
the embedding similarity is measured in terms of inner-product. With the
shared candidate set, the siamese encoder, cross encoder and hybrid encoder
will generate their recommendation results based on the ranking scores
calculated with Eq.10, 5 and 13, respectively.

$\bullet$ System Settings. All the models are implemented based on PyTorch
1.6.0 and Python 3.6.9, and trained on Nvidia V100 GPU clusters. The time cost
is tested with a CPU machine (Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, with
112 GB MEM), which is consistent with the production environment555GPU
inference is not feasible in production environment due to factors like huge
online traffic, and the requirement of realtime service (user’s request must
be processed right away instead of being buffered and processed periodically).

###  4.2. Experiment Analysis

####  4.2.1. Basic Comparison

In basic comparison, a total of 100 candidate ads are firstly generated via
ANN search; then each encoder makes the top-N recommendation based on its
computation of user-ad relevance. The experimental results are reported in
Table 2, where both datasets and both types of backbone networks are included
for comparison. It can be observed that hybrid encoder outperforms siamese
encoder in all settings; and while used for native ads, it achieves comparable
recommendation quality as the cross encoder. In Google News dataset, gaps
between hybrid encoder and cross encoder are relatively larger, which may
attribute to the following reason. The Google News dataset has much less
training instances than the native ads dataset; as a result, the model’s
performance could be limited by insufficient training. However, the cross
encoder can be more robust to this adverse situation because data can be
utilized with higher efficiency. Specifically, as the input query and target
are jointly encoded from the bottom-up, it becomes easier to figure out their
underlying relevance; therefore, it enables the model to be learned more
quickly and less affected by the data sparsity. To verify the above analysis,
additional experiments are carried out in which models are trained with less
data on native ads dataset (with 1/3 of the original training data). It is
found that the cross encoder’s performance becomes “Hit@1:0.1054,
Hit@3:0.2397”, while the hybrid encoder’s performance becomes “Hit@1:0.0891,
Hit@3:0.2089” (comprehensive results are included in Table 8 in Appendix). In
other words, the cross encoder is less affected, which further confirms our
previous analysis. More discussion about both encoders’ relationship will be
made in section 4.2.4.

Table 4. Asymmetric Model Deployment section 4.2.3. The light-weighted ua-
interaction network leads to very limited loss in recommendation quality.
Table 5. Progressive Training section 4.2.4. The progressive training
significantly improves hybrid encoder’s performance.

####  4.2.2. Hit-Time Ratio

More detailed performance analysis is made for the model trained for real-
world production (i.e., the one based on distilled BERT). Particularly, the
top-1 recommendation quality is measured with variant number of candidate ads
from ANN search, together with the inference time needed to process every
single user. (In the previous experiment, it is set to be 100 regardless of
time cost; but in production, it’s required strictly to work in millisecond
scale). As shown in Table 3, the hybrid encoder merely incurs limited
additional cost compared with the siamese encoder, meanwhile it improves the
recommendation quality by a large-margin. As discussed in section 3.2.3, the
siamese encoder takes one transformer encoding to serve each user, while the
hybrid encoder merely introduces another $\epsilon$ pass of feed-forward
inference ($\epsilon=1$ for this experiment as ua-interaction network and user
embedding network are of the same size). While in reality, the increment of
running time is even humble as the inference of user embedding network and ua-
interaction network are called simultaneously. In contrast, the cross encoder
is far more expensive, as user needs to be encoded from scratch with each of
the candidate ads. (Although in experiment environment, the encoding process
can be run in parallel given a sufficient number of machines, it’s not
realistic in production as online traffic is tremendous). It can be observed
that the model’s running time grows rapidly and soon becomes over 10 ms, which
is a default threshold for many realtime services, like native ads
recommendation. Despite the expensive running cost, the performance gain from
cross encoder is very limited: similar to the observation in section 4.2.1,
the hybrid encoder and cross encoder’s performances are quite close to each
other in most of the settings.

####  4.2.3. Asymmetric Model Deployment

Recall that the hybrid encoder’s working efficiency can be further improved if
a smaller scale ua-interaction network is utilized (which makes $\epsilon$
lower than 1), we further test the model’s performance with ua-interaction
network and user/ad embedding network deployed asymmetrically. In particular,
for user/ad embedding network with BERT-base (12-layer), the ua-interaction
network is switch to a smaller 2-layer transformer; while for user/ad
embedding network with distilled BERT (2-layer), the ua-interaction network is
built on a single-layer transformer (the hidden-dimensions stay the same). The
experiment result is demonstrated in Table 4. Despite that the ua-interaction
network’s scale is reduced significantly, the loss in quality is relatively
small, especially when a large number of ads are recommended (Top-20). The
underlying reasons are twofold. Firstly, the disentangled ad embeddings are
pre-computed with the same network, whose quality remains the same. Secondly,
although the ua-interaction network is simplified, user’s information for
analyzing user-ad relevance may still be effectively extracted thanks to its
being attended by the ad.

####  4.2.4. Progressive Learning

The ablation study is made to evaluate the progressive training’s effect on
native ads recommendation section 3.4. The experiment results are demonstrated
in Table 5, where * means that the model is trained only with the global
contrast learning. It can be observed that the hybrid encoder’s performance is
improved significantly when the progressive training is utilized. We further
compare the progressive training pipeline’s effect on cross encoder, and find
that the cross encoder’s performance may also benefit from it; e.g., in native
ads dataset with BERT-base backbone, the performance is improved from
“Hit@1:0965, Hit@3:0.2178” to “Hit@1:0.1111, Hit@3:0.2497” (more results are
included in Table 7 in Appendix). However, it is not as significant as hybrid
encoder, and such a phenomenon may still be resulted from the cross-encoder’s
higher data efficiency in learning user-ad relevance, as discussed in section
4.2.1. These observations do not conflict with our basic argument; in fact,
the cross encoder is always more capable of making fine-grained prediction,
while the hybrid encoder may get very close to the cross encoder’s performance
if it is trained properly and sufficiently.

####  4.2.5. Degree of Disentanglement

The disentanglement degree’s impact on native ads recommendation is also
studied (i.e., the number of disentangled ad embeddings to be generated). The
experiment results are demonstrated in Table 6, where hybridi means that the
degree is set to be $i$. It can be observed that the best performance is
achieved at degree 1 and 3, when the backbone structure is BERT-base
(12-layer-768-dim) and distilled BERT (2-layer-128-dim), respectively.
Meanwhile, further increasing the disentanglement degree will not help the
model’s performance. Such a phenomenon indicates that a large-scale model
(e.g., BERT-base) may fully represent ad’s information with one-single high-
dimension vector; and when the model’s scale is reduced (e.g., distilled
BERT), it may require a few more vectors to fully extract ad’s information.
Because of the simplicity about ad’s description (usually a short sequence of
keywords indicating one particular product or service), a small
disentanglement degree will probably be sufficient in practice.

####  4.2.6. Summarization

The major findings of the experimental studies are summarized as follows.
$\bullet$ The hybrid encoder significantly outperforms the siamese encoder
with little additional cost; besides, it achieves comparable recommendation
quality as the cross encoder, but works with much higher efficiency.
$\bullet$ The progressive training pipeline contributes substantially to the
model’s performance. Because of the simplicity of ad description, a small
disentanglement degree can be sufficient in practice. Besides, a largely
simplified ua-interaction network is able to reduce the model’s inference cost
with very limited loss in quality.

Table 6. Degree of Disentanglement on Native Ads section 4.2.5.

##  5\. Conclusion

In this paper, we proposed hybrid encoder for the efficient and precise native
ads recommendation. With hybrid encoders, the recommendation is made through
two consecutive steps: retrieval and ranking. In the retrieval step, user and
ad are independently encoded, based on which candidate ads can be acquired via
ANN search. In the ranking step, user and ad are collaboratively encoded,
where ad-related user embeddings and disentangled ad embeddings are generated
to select high-quality ads from the given candidates. Both the retrieval step
and ranking step are light-weighted, which makes hybrid encoder competitive in
working efficiency. Besides, a progressive training pipeline is proposed to
optimize the overall performance of the two-stage workflow, where the model is
learned to learned to “make candidate retrieval from the global scope” and
“select the best ads from the given candidates” step-by-step. The experimental
studies demonstrate that hybrid encoder significantly improves the quality of
native ads recommendation compared with conventional siamese encoder;
meanwhile, it achieves comparable recommendation quality as the cross encoder
with highly reduced time consumption.

## References

  * (1)
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Fan et al. (2019) Miao Fan, Jiacheng Guo, Shuai Zhu, Shuo Miao, Mingming Sun, and Ping Li. 2019\.  MOBIUS: Towards the Next Generation of Query-Ad Matching in Baidu’s Sponsored Search. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2509–2517.
  * Ge et al. (2013) Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. 2013\.  Optimized product quantization for approximate nearest neighbor search. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 2946–2953.
  * Gu et al. (2020) Xiaotao Gu, Yuning Mao, Jiawei Han, Jialu Liu, Hongkun Yu, You Wu, Cong Yu, Daniel Finnie, Jiaqi Zhai, and Nicholas Zukoski. 2020.  Generating Representative Headlines for News Stories. In _Proc. of the the Web Conf. 2020_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_ (2020).
  * Huang et al. (2020) Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020.  Embedding-based Retrieval in Facebook Search. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2553–2561.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data. In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_. 2333–2338.
  * Humeau et al. (2019) Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019.  Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring.  _arXiv preprint arXiv:1905.01969_ (2019).
  * Jegou et al. (2010) Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010\.  Product quantization for nearest neighbor search.  _IEEE transactions on pattern analysis and machine intelligence_ 33, 1 (2010), 117–128.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense Passage Retrieval for Open-Domain Question Answering.  _arXiv preprint arXiv:2004.04906_ (2020).
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019\.  Latent retrieval for weakly supervised open domain question answering.  _arXiv preprint arXiv:1906.00300_ (2019).
  * Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017.  A structured self-attentive sentence embedding.  _arXiv preprint arXiv:1703.03130_ (2017).
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Lu et al. (2020) Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020.  TwinBERT: Distilling knowledge to twin-structured BERT models for efficient retrieval.  _arXiv preprint arXiv:2002.06275_ (2020).
  * Luan et al. (2020) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2020.  Sparse, Dense, and Attentional Representations for Text Retrieval.  _arXiv preprint arXiv:2005.00181_ (2020).
  * Malkov and Yashunin (2018) Yury A Malkov and Dmitry A Yashunin. 2018.  Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs.  _IEEE transactions on pattern analysis and machine intelligence_ (2018).
  * Manning et al. (2008) Christopher D Manning, Hinrich Schütze, and Prabhakar Raghavan. 2008.  _Introduction to information retrieval_.  Cambridge university press.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  _The probabilistic relevance framework: BM25 and beyond_.  Now Publishers Inc.
  * Shen et al. (2014) Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014.  A latent semantic model with convolutional-pooling structure for information retrieval. In _Proceedings of the 23rd ACM international conference on conference on information and knowledge management_. 101–110.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.

##  Appendix A Appendix

Additional experimental results (shown in the next page) are included in the
Appendix. According to Table 7, the models’ performances are consistently
improved with the adoption of progressive training. And in Table 8: cross
encoder’s performance is less affected when the volume of training data is
reduced.

Table 7. The comprehensive result of hybrid encoder w./w.o. progressive
training pipeline. (I.: Native Ads, II: Google News) Table 8. Hybrid Encoder’s
performance with 1/3 of the original training data.


http://www.math.polytechnique.fr/ sabbah ††thanks: (C.S.) This research was
supported by the grants ANR-08-BLAN-0317-01 and ANR-13-IS01-0001-01 of the
Agence nationale de la recherche.

http://homepage.ntu.edu.tw/ jdyu/ ††thanks: (J.-D.Y.) This work was partially
supported by the NCTS and the MoST of Taiwan.

# On the irregular Hodge filtration of exponentially twisted mixed Hodge
modules

Claude Sabbah UMR 7640 du CNRS
Centre de Mathématiques Laurent Schwartz
École polytechnique
F–91128 Palaiseau cedex
France<EMAIL_ADDRESS>Jeng-Daw Yu Department of Mathematics
National Taiwan University
Taipei 10617
Taiwan<EMAIL_ADDRESS>
###### Abstract.

Given a mixed Hodge module $\EuScript{N}$ and a meromorphic function $f$ on a
complex manifold, we associate to these data a filtration (the irregular Hodge
filtration) on the exponentially twisted holonomic module
$\EuScript{N}\otimes\EuScript{E}^{f}$, which extends the construction of
[E-S-Y13]. We show the strictness of the push-forward filtered
$\mathscr{D}$-module through any projective morphism
$\pi:X\mathchoice{\longrightarrow}{\rightarrow}{\rightarrow}{\rightarrow}Y$,
by using the theory of mixed twistor $\mathscr{D}$-modules of T. Mochizuki. We
consider the example of the rescaling of a regular function $f$, which leads
to an expression of the irregular Hodge filtration of the Laplace transform of
the Gauss-Manin systems of $f$ in terms of the Harder-Narasimhan filtration of
the Kontsevich bundles associated with $f$.

###### Key words and phrases:

Irregular Hodge filtration, twisted de Rham complex

###### Key words and phrases:

Irregular Hodge filtration, twisted de Rham complex, holonomic D-module,
exponential twist, mixed Hodge module, mixed twistor D-module, Kontsevich
bundle, Gauss-Manin system, Laplace transformation

###### 1991 Mathematics Subject Classification:

14F40, 32S35, 32S40

###### 1991 Mathematics Subject Classification:

14F40, 32S35, 32S40