# ContentCTR: Frame-level Live Streaming Click-Through Rate Prediction with
Multimodal Transformer

Jiaxin Deng  National Laboratory of Pattern Recognition, Institute of
Automation, Chinese Academy of Sciences95 Zhongguancun East RoadHaidian
QuBeijing ShiChina100190<EMAIL_ADDRESS>,  Dong Shen  KuaiShou Inc.6
Shangdi West RoadHaidian QuBeijing ShiChina100085<EMAIL_ADDRESS>,
Shiyao Wang  KuaiShou Inc.6 Shangdi West RoadHaidian QuBeijing ShiChina100085
<EMAIL_ADDRESS>,  Xiangyu Wu  KuaiShou Inc.6 Shangdi West
RoadHaidian QuBeijing ShiChina100085<EMAIL_ADDRESS>,  Fan Yang
KuaiShou Inc.6 Shangdi West RoadHaidian QuBeijing ShiChina100085
<EMAIL_ADDRESS>,  Guorui Zhou  KuaiShou Inc.6 Shangdi West RoadHaidian
QuBeijing ShiChina100085<EMAIL_ADDRESS>and  Gaofeng Meng  National
Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of
Sciences95 Zhongguancun East RoadHaidian QuBeijing ShiChina100190
<EMAIL_ADDRESS>
(2023)

###### Abstract.

In recent years, live streaming platforms have gained immense popularity as
they allow users to broadcast their videos and interact in real-time with
hosts and peers. Due to the dynamic changes of live content, accurate
recommendation models are crucial for enhancing user experience. However, most
previous works treat the live as a whole item and explore the Click-through-
Rate (CTR) prediction framework on item-level, neglecting that the dynamic
changes that occur even within the same live room. In this paper, we proposed
a ContentCTR model that leverages multimodal transformer for frame-level CTR
prediction. First, we present an end-to-end framework that can make full use
of multimodal information, including visual frames, audio, and comments, to
identify the most attractive live frames. Second, to prevent the model from
collapsing into a mediocre solution, a novel pairwise loss function with
first-order difference constraints is proposed to utilize the contrastive
information existing in the highlight and non-highlight frames. Additionally,
we design a temporal text-video alignment module based on Dynamic Time Warping
to eliminate noise caused by the ambiguity and non-sequential alignment of
visual and textual information. We conduct extensive experiments on both real-
world scenarios and public datasets, and our ContentCTR model outperforms
traditional recommendation models in capturing real-time content changes.
Moreover, we deploy the proposed method on our company platform, and the
results of online A/B testing further validate its practical significance.

stream highlight detection, multi-modal learning, click-through rate
prediction

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††submissionid: 183††ccs: Information
systems Data mining

##  1\. Introduction

Figure 1. (a) shows the screenshots of the platform mobile app. There exist
four kinds of modality: the visual feature of streaming frames, the speech
from streamer, the ID embedding of streamer (red box) and the bullet comments
from the user (green box). (b) The grey line shows the ground truth CTR
changes of a dancing streamer while the green line represents the predicted
CTR of ID-based recommendation model. And three distinct highlight moments are
marked by red, yellow, and blue points.

Live streaming platform represent a new type of online interaction and
experienced rapid growth in recent years. On such platform, more and more
users are attracted to live streaming for entertainment, while streamers
receive rewards from both the audiences and the platform. This new form of
interaction and entertainment has motivated researchers to study emerging
issues such as gift-sending mechanisms (Zhu et al., 2017), E-Commerce events
(Yu et al., 2021), and other live streaming practices. Due to the large number
of streamers and the constantly changing content, an accurate recommendation
algorithm is crucial for enhancing user experience. (Zhang et al., 2021b, a)
focus on capturing mutual information between streamers and viewers to obtain
better representations. (Rappaz et al., 2021) propose a self-attentive model
to address the dynamic availability and repeat consumption of live streaming.
(Yu et al., 2021) present a novel Live Stream E-Commerce Graph Neural Network
framework to learn the tripartite interaction among streamers, users and
products.

The broadcast of real-time content implies the live is constantly changing,
and the performance of various segments may differ significantly. Although
most previous work (Zhang et al., 2021b, a; Yu et al., 2021; Rappaz et al.,
2021) treat live broadcasts as items and model them at the item-level, we
believe that understanding the content of individual frames and making more
refined, frame-level estimates is crucial. Additionally, identifying the most
captivating frames from the stream and presenting them to potentially
interested users can considerably enhance user engagement, attract more users,
and increase revenue for live-streaming platforms. To fully capture the
changes in live content, three challenges must be addressed: (1) As depicted
in Figure 1 (a), live-streaming scenarios involve several modalities,
including real-time visual frames, speech from the streamer, bullet comments
from audiences, and other categorical inputs (e.g., streamer IDs and live
IDs). Consequently, a model that can leverage various modalities and their
historical information is crucial. (2) Traditional pointwise estimation may
lead to the model converging on a suboptimal solution. Apart from that, as
illustrated in Figure 1 (b), the traditional ID based model may learn a
mediocre solution (shown in green line), neglecting multiple singular points
that could represent live streaming highlights. Thus, an appropriate loss
function that models the contrastive relationship between highlight and non-
highlight frames is beneficial for generating the final CTR prediction
results. (3) In live-streaming scenarios, the streamer’s speech and audiences’
comments may be ambiguous and not sequentially aligned with the visual frames,
necessitating a module to filter out the noise caused by misalignment.

In this paper, we propose an end-to-end transformer-based network called
ContentCTR, which leverages multi-modality features for frame-level Click-
Through Rate (CTR) prediction. To the best of our knowledge, this is the first
research that explores frame-level live streaming recommendation. First, the
ContentCTR exploits the information contained in visual, speech, comment, and
ID embedding and uses an attention mechanism to adaptively find correlations
and interactions from different modalities. Second, to alleviate the
misalignment between visual and textual modality, we develop a dynamic time
warping algorithm to address potential temporal discrepancies that may arise
during live streaming events. Finally, to better exploit the contrastive
information between highlight frames and no-highlight frames, we design a
pairwise loss function with first-order difference constraints. We find that
the constraints are essential when jointly optimizing pointwise and pairwise
losses to avoid collisions and model collapse. It is worth noting that our
work focuses on modeling the relationship between real-time live content and
CTR, without introducing user personalization factors. The model can be
utilized not only for online traffic mechanism but also for helping existing
recommendation models enhance content understanding. We perform comprehensive
experiments on both a large-scale real-world live streaming dataset and a
public PHD dataset (Garcia del Molino and Gygli, 2018) and achieve the state-
of-the-art performance.

In summary, the main contributions made in this work are as follows:

  * •

We propose ContentCTR, an end-to-end transformer-based network for frame-level
CTR prediction in live streaming platform. It efficiently utilizes the
features of different modalities and captures the dynamic highlight pattern.

  * •

We design a pairwise-based loss function with first-order difference
constraints to exploit the contrastive information of highlight frames and no-
highlight frames. In addition, a dynamic time warping (DTW) based alignment
strategy is present to alleviate misalignment in streaming scenarios.

  * •

We conduct comprehensive experiments on both a large-scale real-world live
streaming dataset and a public PHD dataset (Garcia del Molino and Gygli,
2018). Our method outperforms all baseline models. We also perform online A/B
testing on the live streaming platform of company, achieving a 2.9% lift in
CTR and a 5.9% improvement in terms of live play duration.

Figure 2. The framework of our proposed method. Part (a) and (d) show the
architecture of Multi-Modal Transformer and Casual Attention Mask which are
discussed in Section 3.2. Part (b) shows the proposed Dynamic Time Warping
Alignment Loss which is discussed in Section 3.3. Part (c) shows the
motivation of Boundary-aware Pairwise Loss which is discussed in Section 3.4.

##  2\. Related Work

###  2.1. Deep Click-Through Rate Prediction

Based on the input modality, the existing methods for Click-Through Rate (CTR)
prediction can be classified into two categories: ID-based and multi-modal-
based CTR prediction models. For the ID-based CTR prediction approachs,
techniques such as Wide&Deep (Cheng et al., 2016), DeepFM (Guo et al., 2017),
and DCN-M (Wang et al., 2021) aim to capture high-order feature interactions
and complex feature dependencies. On the other hand, DIFM (Lu et al., 2021)
addresses the fixed feature representation problem by learning the input
vector level weights for feature representations. For multi-modal-based CTR
prediction models, the majority of the research has focused on video Click-
Through Rates prediction, which is a fundamental task in the field of
multimedia and information retrieval. For example, AutoFIS (Liu et al., 2020)
propose a two-stage algorithm called Automatic Feature Interaction Selection
while UBR4CTR (Qin et al., 2020) propose User Behavior Retrieval for CTR
prediction framework to tackle the problem that sequential patterns such as
periodicity or long-term dependency are not embedded in the recent several
behaviors but in far back history. Both of these approaches are based on the
Factorization Machine (FM) (Rendle, 2010) method. For DNNs based methods, DSTN
(Ouyang et al., 2019) propose a Deep Spatiotemporal Translation Network
(DSTN), which utilize the auxiliary data in spatial and temporal domain. In
order to better capture user preferences, HyperCTR (He et al., 2021) design a
Hypergraph Click-Through Rate prediction framework, built upon the hyperedge
notion of hypergraph neural networks, which can yield modal-specific
representations of users and micro-videos. However, all these methods only
focus on video-level CTR prediction. Compared with traditional video CTR
prediction task, stream highlight CTR prediction needs frames level CTR
prediction network which can leverage multi-modal feature to address its
unique challenges, i.e. the dynamic patterns of live-streaming and non-
sequential alignment of different modality.

###  2.2. Live Streaming Recommendation

As an emerging form of social media, increasing attention has being given to
live streaming recommendations. Current works typically treat streamers or
audiences as items, and develop recommendation systems to predict interactions
between users and items. These methods can be broadly categorized into two
types: Filtering-based methods and Deep learning-based methods. For Filtering-
based methods, HyPAR (Yang et al., 2013) proposed a Collaborative and Content-
based filtering hybrid approach to recommend streamers to audiences. For Deep
learning-based methods, LiveRec (Rappaz et al., 2021) and (Zhang et al.,
2021a) leverages self-attentive mechanisms based on historical interactions
and repeat consumption behavior, while DRIVER (Gao et al., 2023) learns
dynamic representations by leveraging users’ highly dynamic behaviors. (Zhang
et al., 2021c) utilizes LSTM to extract the representation and capture the
preference of streamers and audiences. However, these works only focus on the
interaction between streamers and audiences at the item level, neglecting the
importance of highly dynamic multi-modal content in live streaming.
Additionally, they cannot provide frame-level recommendations.

###  2.3. Video Highlight Detection

The task most closely related to stream CTR prediction is Personalized Video
Highlight Detection (P-VHD), as both aim to recognize different content
patterns in temporal sequences. The primary objective of P-VHD is retrieving a
subset of video frames that capture a person’s main attention or interests
from the original video. Recently, several works (Garcia del Molino and Gygli,
2018; Rochan et al., 2020; Chen et al., 2021; Wang et al., 2022; Bhattacharya
et al., 2022) are proposed for P-VHD task, which aims at extracting user-
adaptive highlight predictions guided by annotated user history. For example,
PHD-GIFs (Garcia del Molino and Gygli, 2018) is the first personalized video
highlight detection technique that also creates a large-scale dataset called
PHD. The P-VHD task and stream CTR prediction both need to consider
personalized behavior, while P-VHD only needs to predict the label of frames
to highlight or no-highlight one. On the other hand, stream CTR prediction
needs to regress the correct order of all frames, which means that stream
highlight CTR prediction is more challenging.

##  3\. Methodology

As illustrated in Figure 2, we propose the Transformer Click-Through Rate
prediction framework, named ContentCTR. ContentCTR utilizes multi-modal
features to predict the CTR at frame level. In particular, we first define the
problem of predicting CTR for stream highlights in Section 3.1. Then the
Multi-Modal Transformer backbone is introduced to fuse and interact with
different modalities in Section 3.2. Additionally, we propose the Dynamic Time
Warping strategy for aligning text and visual feature in Section 3.3. Finally,
boundary-aware pairwise loss for exploring contrasting information is
presented in Section 3.4.

###  3.1. Problem Formulation

The main goal of the CTR prediction is to estimate the probability of the
current content of the live broadcast being clicked. In this work, we simplify
this problem for predicting the Click-Through Rate of frame $\delta_{i}$ at
timestamp $i$. We denote $M_{i}=\\{v_{i},a_{i},x_{i}\\}$ as the multi-modal
tuple, where $v_{i},a_{i}$ and $x_{i}$ represent the visual, speech and
comments of frame $\delta_{i}$, respectively. We hypothesize that different
streamers have distinct talents and attract different audiences who are
typically interested in specific types of highlight moments in a streaming
room. For instance, as shown in Figure 1 (b), some audiences may be attracted
to a streamer’s dancing, while others may be attracted to the PK between
streamers. Therefore, we denote $E_{u}$ as the ID embedding of streamer $u$.
The ID embedding of streamer is extracted by a pre-trained SIM (Pi et al.,
2020) model that reflects the rough classification of a streamer.
Additionally, we consider that the highlight pattern and audience taste change
over time, and thus, the model should use information from the $n-1$ lookahead
windows $W_{M}=\\{M_{i-n+1},M_{i-n+2},\cdots,M_{i-1}\\}$ of previous frames to
predict the CTR $y_{i}$ of frame $\delta_{i}$. The prediction problem can be
formulated as follows,

(1) $\operatorname{Prob}\left(\delta_{i}\mid
W_{M},E_{u}\right)\sim\Gamma\left(W_{M},E_{u},\delta_{i}\right)$

where the frame $\delta_{i}$ is represented by multi-modal feature of
lookahead windows $W_{M}$ and the ID embedding $E_{u}$. The probability that
audiences will click on the frame $\delta_{i}$ is denoted by
$\operatorname{Prob}\left(\delta_{i}\mid W_{M},E_{u}\right)$. Moreover,
$\Gamma\left(W_{M},E_{u},\delta_{i}\right)$ is the model used to estimate the
probability $\operatorname{Prob}\left(\delta_{i}\mid W_{M},E_{u}\right)$.

###  3.2. Multi-Modal Transformer

The proposed Multi-Modal Transformer backbone includes the following three
basic components, i.e., feature fusion layer, Perceiver block, and casual
sequence decoder.

####  3.2.1. Feature Fusion Layer

As depicted in Figure 2 (b), given the historical window $W_{M}$ of streamer
$u$, we extract multi-modal features for every timestamp, including the
streaming frames $v_{i}$, Auto Speech Recognition (ASR) text $a_{i}$ from the
streamer, comment $x_{i}$ from the audiences. The streaming frames are
tokenized by the pre-trained swin (Liu et al., 2021) $f(\cdot)$ while the ASR
and comment text are tokenized with the BERT (Devlin et al., 2018) Chinese
Large model $g(\cdot)$ and we only use the hidden feature of <CLS> token as
the text embedding. Since the embedding dimensions for streaming frames and
language tokens are different, two MLP heads $proj_{1}(\cdot)$ and
$proj_{2}(\cdot)$ are set to map the frames embedding and text embedding to
the same dimension $d$, denoted by $S_{p}\in\mathcal{R}^{b\times n\times
1\times d}$ and $S_{a}\in\mathcal{R}^{b\times n\times 1\times d}$, where $n$
is the length of lookahead window and $b$ is the batch size. The above process
can be formulated as follows:

(2) $\displaystyle{S}_{{p}}$ $\displaystyle={proj}_{1}(f(\boldsymbol{v}))$
$\displaystyle{S}_{{a}}$
$\displaystyle={proj}_{2}\left({g}_{\texttt{<{CLS}>}}([\boldsymbol{a},\boldsymbol{x}])\right)$

where
$\boldsymbol{v}=\left[v_{1},\cdots,v_{n}\right],\boldsymbol{a}=\left[a_{1},\cdots,a_{n}\right]$
and $\boldsymbol{x}=\left[x_{1},\cdots,x_{n}\right]$.

For frames $\delta_{i}$ at timestamp $i$, the corresponding frames feature and
text feature are denoted by $S_{p}^{i}\in\mathcal{R}^{b\times 1\times 1\times
d}$ and $S_{a}^{i}\in\mathcal{R}^{b\times 1\times 1\times d}$ respectively. We
concatenate $S_{p}^{i}$ and $S_{a}^{i}$ as the final input tokens
$E_{f}^{i}\in\mathcal{R}^{b\times 1\times 2\times d}$ of the Perceiver block.
Finally, we obtain $E_{f}\in\mathcal{R}^{b\times n\times 2\times d}$ by
concatenating all features of the timestamps.

####  3.2.2. Perceiver Block

The Perceiver block is connected to the casual sequence decoder, as shown in
Figure 2 (a). It takes the repeated ID embedding $n\times
E_{u}\in\mathcal{R}^{b\times n\times 1\times d}$ of streamer $u$ and the
fusion multi-modal feature $E_{f}\in\mathcal{R}^{b\times n\times 2\times d}$
as input, where $n$ is the length of lookahead window, $b$ is the batch size
and $d$ is the input dimension. The motivation of Perceiver block is the
success of multi-head attention mechanism (Vaswani et al., 2017) in sequential
recommendation (Chen et al., 2019) and video understanding (Alayrac et al.,
2022). We apply it to capture streamer’s highlight patterns on lookahead
sequences with the query-based retrieval problem.

First, we initialize learned latent query $Q\in\mathcal{R}^{bn\times 1\times
d}$ with flattened $n\times E_{u}$. Next, we concatenate the flattened $E_{f}$
and $n\times E_{u}$ at the second dimension and take $K=V=[E_{f},n\times
E_{u}]\in\mathcal{R}^{bn\times 3\times d}$ as key and value. We linearly
project the input vector to latent vectors with $d_{h}$ dimensions, by
different linear projections. Then we perform the scaled dot-product multi-
head attention as follows,

(3) $\displaystyle\operatorname{MultiHead}(Q,K,V)$
$\displaystyle=\operatorname{Concat}\left(\text{head}_{1},\text{head}_{2},\cdots,\text{head}_{n_{h}}\right),$
$\displaystyle\text{head}_{i}$
$\displaystyle=\operatorname{Attention}\left(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}\right),$

where $n_{h}$ is the number of attention head and
$W_{i}^{Q}\in\mathcal{R}^{d\times d_{h}},W_{i}^{K}\in\mathcal{R}^{d\times
d_{h}}$ and $W_{i}^{V}\in\mathcal{R}^{d\times d_{h}}$ are the learnable
parameters. The scaled dot-product attention function is defined as follows,

(4)
$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{\top}}{\sqrt{d_{h}}}\right)$

Then, the Perceiver block employs a feed-forward network with residual
connections for performance boosting. The Perceiver block is stacked for $L$
layers and the outputs of previous block are fed into the Perceiver block
alternately. The output of Perceiver block is denoted by
$E_{p}\in\mathcal{R}^{bn\times 1\times d_{n}}$. The pseudocode of the
Perceiver block is shown in Algorithm 1.

Algorithm 1 A Pytorch-style Pseudocode for Perceiver Block.

⬇

def perceiver_block(

x_f, # The [b, n, 2, d] multi-modal feature

x, # The learned latent query with shape [b, n, 1, d]

num_layers, # The number of layers

):

x_f = flatten(x_f) # [b, n, 2, d] -> [b * n, 2, d]

x = flatten(x) # [b, n, 1, d] -> [b * n, 1, d]

for i in range(num_layers):

# Attention

x = x + attention_i(q=x, kv=concat([x_f, x]))

# Feed forward with residual connection

x = x + ffw_i(x)

return x

####  3.2.3. Casual Sequence Decoder

We first unflatten and squeeze the output of Perceiver block
$E_{p}\in\mathcal{R}^{bn\times 1\times d}$ as the dimension of
$\mathcal{R}^{b\times n\times d}$. Then we initialize the query, key and value
of decoder as $Q=K=V=E_{p}\in\mathcal{R}^{b\times n\times d}$. After that, we
apply the scaled dot-product multi-head attention which is the same as shown
in Equation 3, but with the scaled dot-product attention function and casual
attention as follows:

(5)
$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^{\top}}{\sqrt{d_{h}}}+M\right)$

where $M$ is the casual attention mask. As shown in Figure 2 (d), it is an
$n\times n$ matrix filled with -inf and its upper triangular sub-matrices are
filled with 0. By applying the casual attention mask $M$ the problem of future
information leakage in the temporal dimension is avoided. The pseudocode of
the Decoder is shown in Algorithm 2.

Algorithm 2 A Pytorch-style Pseudocode for Decoder.

⬇

def decoder(

x, # The input with shape [b*n, 1, d_n]

num_layers, # The number of layers

):

x = unflatten(x).squeeze() #[b*n,1,d_n] -> [b,n,d_n]

for i in range(num_layers):

# Attention

x = x + attention_with_mask_i(q=x, k=x, v=x))

# Feed forward with residual connection

x = x + ffw_i(x)

return x

The output $H$ from the final attention layer is then fed into a fully-
connected layer, followed by a sigmoid transformation to produce the scalar
prediction of CTR $s$. The parameters of the fully-connected layer are
represented by $W\in\mathcal{R}^{d_{n}\times 1}$.

The main objective is to maximize the log-likelihood between the predicted CTR
$s$ and the actual CTR $y$, which is achieved using the following pointwise
model to optimize the standard LogLoss (Liu et al., 2009):

(6)
$L_{Point}=-\frac{1}{n}\sum_{i=1}^{n}\left[y_{i}\cdot\log\left(s_{i}\right)+(1-y_{i})\cdot\log\left(1-s_{i}\right)\right]$

###  3.3. DTW Alignment

The motivation behind for alignment is to address potential temporal
discrepancies that may arise during live streaming events. For example, the
streamer may describe the plan before taking action, or explain detailed
information after action. Additionally, the comments from the audiences may
experience some time lag, which further exacerbates the misalignment issue.
Therefore, it is essential to train text and visual encoders that can handle
misalignment to alleviate that problem. Inspired by previous works (Xu et al.,
2021; Ko et al., 2022), in this section we present our contrastive learning
based framework for visual and text sequences alignment.

Given the text sequence $S_{a}$ and visual sequence $S_{p}$ from a lookahead
window $W_{M}$, we assume that when timestamps $i$ in the text sequence and
$j$ in the visual sequence represent the same pattern, then $S_{a}^{i}$ and
$S_{p}^{j}$ should have a semantic similarity. To train a network that can
minimized the distance between text sequences $S_{a}$ and visual sequences
$S_{p}$, we use the Dynamic Time Warping (Müller, 2007) (DTW) which calculates
the minimum cumulative matching costs over units as the sequence distance to
measure the similarity. First, we computes a pairwise distance matrix
$D\left(S_{a},S_{p}\right)\coloneqq\left[\mathrm{sim}\left(S_{a}^{i},S_{p}^{j}\right)\right]_{ij}\in\mathcal{R}^{n\times
n}$ with a distance measure $\mathrm{sim}(\cdot)$. In our work we applied the
cosine similarity as $\mathrm{sim}(\cdot)$. Then, we employs dynamic
programming and sets a matrix $C\in\mathcal{R}^{n\times n}$ to record the
minimum cumulative cost between $S_{a}^{i}$ and $S_{p}^{j}$ (Dixit, 1990):

(7) $C_{i,j}=D_{i,j}+\min\left\\{C_{i-1,j-1},C_{i-1,j},C_{i,j-1}\right\\}$

where $1\leq i,j\leq n$. Then, the distance
$d_{\left\\{\mathbf{S}_{a},\mathbf{S}_{p}\right\\}}$ between sequences $S_{a}$
and $S_{p}$ is set to the last element of matrix $C$:

(8) $d_{\left\\{\mathbf{S}_{a},\mathbf{S}_{p}\right\\}}=C_{n,n}$

Contrastive Learning (CL) is a self-supervised learning technique that learns
a representation of data by comparing different views of same samples. The
basic idea of CL is to bring together similar pairs and push away dissimilar
pairs. We hypothesize that the positive pair
$\left\\{{S}_{a},{S}_{p}\right\\}$ should be similar. To construct the
negative sample ${S}_{n}$, we randomly shuffle the temporal order of video
sequence ${S}_{p}$. Therefore, the negative pair
$\left\\{{S}_{a},{S}_{n}\right\\}$ should be dissimilar. Then, we can derive
the training objective to minimize the InfoNCE loss (Oord et al., 2018):

(9)
$\displaystyle\mathcal{L}_{align}\left(S_{a},S_{p},\boldsymbol{S}_{n}\right)=-\log\frac{\exp\left(d_{\left\\{S_{a},S_{p}\right\\}}/\tau\right)}{\exp\left(d_{\left\\{S_{a},S_{p}\right\\}}/\tau\right)+\sum_{S_{n}\in\boldsymbol{S}_{n}}{\exp}\left(d_{\left\\{S_{a},S_{n}\right\\}}/\tau\right)}$

where $\boldsymbol{S}_{n}=\left\\{S_{ni}\right\\}_{i=1}^{N}$ is the set of $N$
negative samples shuffled from $S_{p}$. As shown in Figure 2 (b), by
minimizing the align loss $\mathcal{L}_{align}$, the visual encoder $f(\cdot)$
and text encoder $g(\cdot)$ should be able to learn a good representation from
aligned and misaligned pairs.

###  3.4. Boundary Aware Pairwise Loss

Figure 3. Loss function for pairwise optimization without any constraints.

In this section, we present some intriguing discoveries on the traditional
pairwise logistic ranking loss (Burges et al., 2005) and propose modifications
to the loss function by integrating the boundary-aware first-order difference
constraint, thereby enhancing the optimization. The addition of the pairwise
loss aids in exploiting the underlying contrasting information present in both
the highlight and no-highlight frames.

Consider the following pairwise loss function, which has no constraints:

(10)
$L_{Pair}^{0}=\sum_{y_{i}>y_{j}}{\log\left(1+e^{-\sigma\left(s_{i}-s_{j}\right)}\right)}$

where $y_{i}$ and $y_{j}$ represent the ground truth CTR at timestamps $i$ and
$j$, while $s_{i}$ and $s_{j}$ represent the predicted CTR from the model.
Figure 3 illustrates the changing of the loss function $L_{Pair}$ with respect
to $s_{i}-s_{j}$, which reveals that when $y_{i}-y_{j}$ holds, minimizing
$L_{Pair}$ tends to cause $s_{i}-s_{j}$ to overtake the optimal value of
$y_{i}-y_{j}$. This leads to over-optimization.

Based on above findings, we propose a revised pairwise loss that incorporates
a boundary-aware first-order difference constraint:

(11)
$L_{Pair}^{1}=\sum_{y_{i}>y_{j}}{\log\left(1+e^{-\sigma\left(s_{i}-s_{j}\right)}\right)},\left(y_{i}-y_{j}\right)-\left(s_{i}-s_{j}\right)\geqslant
0$

where $\left(y_{i}-y_{j}\right)-\left(s_{i}-s_{j}\right)\geqslant 0$ denotes
the boundary, and solely those samples that reside within the boundary will
calculate $L_{Pair}^{1}$. Any samples located outside the boundary will result
in $L_{Pair}^{1}$ being set to 0.

Without losing generality, the original pairwise loss function $L_{Pair}^{0}$
presented in Equation 10 can be divided into three distinct parts:

  * •

Part 1: As shown in Figure 2 (c1), when $s_{i}-s_{j}\leq 0$, the model’s
assessment of the significance between timestamp $i$ and $j$ is entirely
incorrect, given that timestamp $i$ is more "highlighting" than timestamp $j$.

  * •

Part 2: As shown in Figure 2 (c2), when $0<s_{i}-s_{j}<y_{i}-y_{j}$, it
implies that the model has distinguished that timestamp $i$ is more
"highlighting" than timestamp $j$, but it still fails to accurately predict
the difference in CTR values between the two timestamps, and thus, it is still
not optimal.

  * •

Part 3: As shown in Figure 2 (c3), when $y_{i}-y_{j}<s_{i}-s_{j}$, it
indicates that the model’s predicted CTR value is too aggressive, resulting in
over-optimization.

In order to verify above scenarios, we design the following loss functions:

(12)
$L_{Pair}^{2}=\sum_{y_{i}>y_{j}}{\log\left(1+e^{-\sigma\left(s_{i}-s_{j}\right)}\right)},s_{i}-s_{j}\leq
0$

where $s_{i}-s_{j}\leq 0$ means that it only optimizes on Part 1.

(13) $\displaystyle
L_{Pair}^{3}=\sum_{y_{i}>y_{j}}{\log\left(1+e^{-\sigma\left(s_{i}-s_{j}\right)}\right)},y_{i}-y_{j}>s_{i}-s_{j}>0$

where $y_{i}-y_{j}>s_{i}-s_{j}>0$ means that it only optimizes on Part 2. The
ablation study among $L_{Pair}^{0}$, $L_{Pair}^{1}$, $L_{Pair}^{2}$ and
$L_{Pair}^{3}$ is discuss in Section 4. In this work, we apply the
$L_{Pair}^{1}$ for optimization.

By combining pointwise loss, align loss and pairwise loss, our final loss used
to learn the model parameters is defined as:

(14)
$\mathcal{L}=\lambda_{1}\mathcal{L}_{Point}+\lambda_{2}\mathcal{L}_{align}+\lambda_{3}\mathcal{L}_{Pair}^{1}$

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are the tradeoff
parameters.

##  4\. Experiments

In this section, we first introduce the datasets, evaluation metrics, and
implementation details. Next, we present the experimental results and some
analysis of them. Specifically, our experiments aim to answer the following
research questions:

  * •

RQ1: How does the modality impact the model?

  * •

RQ2: To what extent can the proposed Perceiver Block enhance the model’s
performance?

  * •

RQ3: How effective is the proposed DTW alignment block in alleviating
misalignment cases?

  * •

RQ4: How much can the model’s performance be enhanced by the proposed pairwise
loss?

  * •

RQ5: How well does the proposed model perform on public datasets, such as
video highlights?

###  4.1. Dataset

To comprehensively evaluate the model’s performance, we present experimental
results on both KLive Dataset and a public video highlighting dataset.
Detailed information regarding these datasets is provided as follows.

####  4.1.1. KLive Dataset

We have constructed a large-scale dataset based on our company’s short video
and live streaming platform, which boasts over three million daily active
users. Our methodology involved selecting 39,000 high-quality live rooms over
a period of three days, based on user rewards and viewing time indicators.
Each live room is then divided into multiple consecutive 30s live segments,
with three pictures evenly sampled for each segment. The streamer’s ASR and
audiences’ comments are extracted for each segment, and the CTR is calculated
by dividing the number of clicked users by all watched users. We then
construct consecutive 20 streaming segments into a dataset sample, filtering
out samples with a low number of watched users of the last live segment to
ensure a reliable CTR. In the end, we obtained 1,436,979 and 286,510 samples
for our training and test datasets, respectively. Each sample is comprised of
the streamer’s item ID, three video frames for 20 streaming segments, audience
comments, the streamer’s ASR speech, and the ground truth CTR for the all live
segments.

####  4.1.2. PHD Dataset

To verify the generality of our method, we evaluate on the publicly available
personalized video highlight detection dataset (Garcia del Molino and Gygli,
2018) (PHD). This dataset comprises of URLs of YouTube videos, IDs of
evaluators or "users," and the segments they designated as highlight frames
based on their preferences. The most recent video that a user annotated is
considered as the target video for that particular user. Since PHD only
provides YouTube URLs, we download the original videos and crawl the captions
from YouTube to carry out the experiments. Our training and testing sets
include up to 20 history highlight frames and one target video per user. The
duration of the history highlight frames varies from 1 to 672.19 seconds, with
an average length of 5.12 seconds. The target videos range from 1 to 37,434
seconds, with an average length of 431.79 seconds. The training set comprises
of a total of 12,541 users and 81,056 videos, while the testing set has 833
users and 7,595 videos that do not overlap with any user or video in the
training set. Consistent with (Chen et al., 2021), we divide the target video
into fixed-length segments (192 frames) and only train those segments that
contain highlight frames. During testing, we make the entire video segments as
input for inference.

###  4.2. Evaluation Metrics

For the experiments on KLive dataset, we employed Kendall’s tau $\tau$
(Kendall, 1945) to measure the correlation between our predicted click-through
rates (CTR) $s$ and the ground truth CTR $y$. It is defined as:

(15) $\tau=\frac{P-Q}{\sqrt{\left(P+Q+T\right)\cdot\left(P+Q+U\right)}}$

where $P$ represents the number of concordant pairs, $Q$ denotes the number of
discordant pairs, $T$ indicates the number of ties only in $s$, and $U$ is the
number of ties only in $y$. If the same pair experiences a tie in both $s$ and
$y$, it is not included in either $T$ or $U$. Value of $\tau$ close to 1
indicate strong agreement, while values approaching -1 indicate strong
disagreement.

Regarding the experiment on the PHD dataset, we utilized the widely adopted
mean Average Precision (mAP) as a metric to evaluate the performance of our
method, which is also applied in previous works (Bhattacharya et al., 2022;
Wang et al., 2022; Chen et al., 2021; Rochan et al., 2020) in video highlight
detection. We report the mAP on the test set and follow the way in (Wang et
al., 2022) to calculate the mAP.

###  4.3. Implementation Details

####  4.3.1. Baseline:

On KLive dataset, we compare our method with ID-based recommendation methods
and several variants of our proposed approach as follows:

  * •

W&D (Cheng et al., 2016): This method trains wide linear models and deep
learning network together to integrate the advantages of memorization and
generalization in recommendation, but requires feature engineering in addition
to raw features.

  * •

DeepFM (Guo et al., 2017): This method utilizes the benefits of both
factorization machines and deep learning by sharing input for the deep and
wide components.

  * •

DIFM (Lu et al., 2021): This approach leverages the benefit of input-aware
factorization machines to address the issue of standard FMs that generate a
fixed representation for varying input instances.

  * •

DCN-M (Wang et al., 2021): This approach is an enhanced version of DCN (Wang
et al., 2017), with greater effectiveness in learning both explicit and
implicit feature crosses through the application of low-rank techniques to
approximate such crosses.

  * •

SelfAttention-plain: We remove the ID embedding $E_{u}$ of the streamer of
ContentCTR and replace the Perceiver block with a self-attention block.

  * •

SelfAttention-input: We replace the Perceiver block with a self-attention
block and concatenate the ID embedding $E_{u}$ of the streamer with $E_{f}$ to
be used as input for the self-attention.

  * •

CrossAttention-q: We replace the Perceiver block with a cross-attention block.
The query of every cross-attention layer is set as $E_{u}$, while the key and
value are initialized with $E_{f}$.

  * •

CrossAttention-kv: We replace the Perceiver block with a cross-attention
block. The key and value of every cross-attention layer are set as $E_{u}$,
while the query is initialized with $E_{f}$.

  * •

CrossAttention-qkv: We replace the Perceiver block with a cross-attention
block. The query is initialized with $E_{u}$, while the key and value are
initialized with $E_{f}$.

Figure 4. Visualization of the CTR prediction results for ContentCTR (red
line), DIFM (green line) and ground truth CTR (grey line).

On the PHD dataset, we make slight modifications to our proposed ContentCTR
method and compare it with several classic baselines for personalized video
highlight detection. Specifically, they are:

  * •

Adaptive-H-FCSN (Rochan et al., 2020): This method employs a convolutional
highlight detection network with a history encoder to learn user-specific
highlight patterns.

  * •

PR-Net (Chen et al., 2021): This method proposes a reasoning framework to
explicitly learn frame-level patterns. It also employs contrastive learning to
alleviate annotation ambiguity.

  * •

PAC-Net (Wang et al., 2022): This method introduces a Decision Boundary
Customizer (DBC) module and a Mini-History (Mi-Hi) mechanism to capture more
fine-grained user-specific preferences.

  * •

ShowMe (Bhattacharya et al., 2022): This method leverages the content of both
user history and target videos, using pre-trained features of YOLOv5 (Jocher,
2020).

  * •

ContentCTR-HL: We modify our proposed ContentCTR model to adapt it to the
personalized video highlight detection task. First, we extract YOLOv5 features
from images for training and testing. The text comes from the captions crawled
from the YouTube video. Second, the ID embedding is replaced with the user
history feature for personalized highlighting. Then, since there is no need to
consider future information leakage in the video highlighting task, the causal
attention mask is set to fully visible. Finally, when optimizing the network,
only $\mathcal{L}_{Point}$ and $\mathcal{L}_{align}$ are reserved for
optimization.

####  4.3.2. Setup Detail:

During the training of ContentCTR on the KLive dataset, the layer number for
the Pereciver block and Decoder block is set to 3, the input dimension
$d=512$, the hidden dimension of the transformer $d_{h}$ is set to 64 and the
number of attention head $n_{h}=8$. We utilize pre-trained swin as the Encoder
$f(\cdot)$ and pre-trained Bert Chinese Large as the Encoder $g(\cdot)$. The
swin and BERT are both pre-trained on image and text data from 39,000
streaming rooms. The tradeoff parameters, $\lambda_{1}$, $\lambda_{2}$, and
$\lambda_{3}$, are respectively set to 0.65, 0.15, and 0.20. The $\sigma$ in
$L_{Pair}^{1}$ is set to 10 and the number of negative sample $N$ is set to 8.
We optimize ContentCTR for 12 epochs with a learning rate of $5\times 10^{-5}$
using the Adam optimizer, and the global batch size is set to 48. During
training, the gradient of swin and BERT feature extractors are open for fine-
tuning, while the parameters of two MLP heads are updated, too. The training
process takes about 34 hours on 8 Nvidia Tesla V100 GPUs. During the training
and testing of ID-based recommendation methods on the KLive dataset, the input
sparse features for these methods include the streamer item ID, live ID,
timestamp ID, exposure count, comment count, gift count, click count, like
count, follow count, room entry and exit count for each streaming segment. The
objective of these methods is to regress the CTR of each segment. When
training ContentCTR-HL on the PHD dataset, we first extract the YOLOv5
features for each frame of the historical highlight frames in the training and
testing set. We use pre-trained YOLOv5 as the Encoder $f(\cdot)$ and pre-
trained Bert English Large as the Encoder $g(\cdot)$. Then we train our
network using the Adam optimizer with a batch size of 8 and an initial
learning rate of $5\times 10^{-5}$, with a cosine decay scheduler. The
training is performed for 20 epochs on 8 NVIDIA Tesla V100 GPUs, which takes
approximately 16 hours.

###  4.4. Overall Performance Comparison

Table 1.  Performances of different methods on KLive dataset

Model |  Tau $\tau$
---|---
ID-based Recommendation Methods
W&D [DLRS’16] |  0.5619
DeepFM [IJCAI’17] |  0.5542
DIFM [IJCAI’21] |  0.5697
DCN-M [WWW’21] |  0.5629
Variants of ContentCTR
SelfAttention-plain |  0.5868
SelfAttention-input |  0.5718
CrossAttention-q |  0.5909
CrossAttention-kv |  0.5780
CrossAttention-qkv |  0.5883
ContentCTR |  0.5919

####  4.4.1. Quantitative Results:

Table 1 summarizes the CTR prediction performances achieved by various methods
on KLive dataset, indicating that ContentCTR surpasses all ID-based
recommendation methods and the variations of ContentCTR. The findings reveal
that the interaction approach among different modalities plays a crucial role
in the final CTR prediction performance. Furthermore, compared to ID-based
recommendation methods, ContentCTR demonstrates superior CTR prediction
performance which suggests that the transformer based encoder-decoder
architecture and Perceiver block is more efficient in capturing dynamic
patterns.

Table 2. Comparison with state-of-the-art alternative on PHD dataset.

Model |  |  Param.(M) | mAP(%)
---|---|---|---
Adaptive-H-FCSN (Rochan et al., 2020) |  [ECCV’20] |  197.35 | 15.65
PR-Net (Chen et al., 2021) |  [ICCV’21] |  - | 18.66
PAC-Net (Wang et al., 2022) |  [ECCV’22] |  5.89 | 17.51
ShowMe (Bhattacharya et al., 2022) |  [MM’22] |  - | 16.40
ContentCTR-HL |  [Ours] |  66.35 | 21.89

In order to answer RQ5 and further evaluate the effectiveness of our method,
we report the performance comparison of ContentCTR-HL with various state-of-
the-art alternatives on PHD dataset. As shown in Table 2, our approach
outperforms all other alternatives on PHD dataset, outperforming PR-Net (Chen
et al., 2021) by +3.23%. This result undoubtedly demonstrates the practicality
and effectiveness of our method. The key improvement of our method can be
derived from the following three aspects. Firstly, in contrast to other
approaches that solely rely on visual features, our method leverages the
multi-modal features of video frames and captions. Secondly, the sequence-to-
sequence encoder-decoder architecture is proficient in detecting frame-level
highlights. Thirdly, the incorporation of DTW alignment effectively alleviates
the possible misalignments between captions and video frames.

####  4.4.2. Qualitative Results:

Moreover, we visualize several typical cases on KLive dataset in Figure 4.
Figure 4 illustrates the prediction results of ContentCTR in red line, DIFM in
green line and the ground truth CTR in grey line. It is obvious that when the
ground truth CTR arises abrupt fluctuations (shark rises or fails) due to the
dynamic changing of content, ContentCTR is able to capture real-time content
changes more effectively. However, DIFM tends to change more gradually, which
is improper for high dynamic streaming highlight recommendation scenarios.

###  4.5. Modality Impact(RQ1)

We study the impact of different modalities on ContentCTR’s performance on
KLive dataset, as shown in Table 3. The results demonstrate that ContentCTR
achieves a tau of $0.5919$ when all the modalities are engaged. We find that
the visual modality has the most important impact on the model performance,
causing a performance degradation of -4.72% when removed. Apart from that, the
text modality is the second most significant factor, leading to a -2.01%
degradation. Lastly, the ID embedding has the smallest but still significant
effect on the model’s performance, with a -0.51% degradation when removed.
This ablation study suggests that incorporating different modalities is
essential for the model to predict CTR accurately.

Table 3.  Ablation study on different modality impact. We study the effect of
visual modality $v$, speech modality $a$, comment modality $x$ and item
modality $u$.

Model |  $v$ |  $a$ |  $x$ |  $u$ | Tau $\tau$
---|---|---|---|---|---
ContentCTR |  ✓ |  ✓ |  ✓ |  ✓ | 0.5919
ContentCTR w/o item |  ✓ |  ✓ |  ✓ |  - | 0.5868 $\downarrow$ 0.51%
ContentCTR w/o text |  ✓ |  - |  - |  ✓ | 0.5718 $\downarrow$ 2.01%
ContentCTR w/o visual |  - |  ✓ |  ✓ |  ✓ | 0.5447 $\downarrow$ 4.72%

###  4.6. Perceiver Block(RQ2)

As demonstrated in Table 1, the Perceiver block surpasses all the variants of
ContentCTR in terms of performance. Table 1 provides the following insights.

Firstly, the results indicate that inputting the ID embedding directly into
the transformer model fails to fully leverage personalized information. The
self-attention architecture is not suitable for detecting the correlation
between different modalities.

Secondly, concerning cross-attention architectures, it is critical to
carefully design the interaction approach between the ID embedding and other
content-based modalities. The traditional query-key-value-based cross-
attention mechanism proves to be less effective than the Perceiver block.

###  4.7. DTW Alignment and Pairwise Loss(RQ3&RQ4)

We design Model1-Model5 to investigate the impact of different loss functions
on KLive dataset, which include $L_{Point}$, $L_{Pair}^{0}$, $L_{Pair}^{1}$,
$L_{Pair}^{2}$, $L_{Pair}^{3}$, and $L_{align}$. According to Table 4, when we
optimized Model4 with $L_{Point}$ and $L_{Pair}^{2}$, its performance
significantly degraded by -5.05%. We hypothesize that the gradient changes are
too drastic when optimizing only Part 1, which is detrimental to modeling the
contrastive information of highlight frames and non-highlight frames. However,
when optimizing only Part 2 with $L_{Pair}^{3}$, Model5 showed a performance
improvement of +0.66%, but it still falls behind Model3, which is jointly
optimized on Part 1 and Part 2 with $L_{Pair}^{1}$, resulting in a significant
improvement of +1.11%.

Table 4.  Ablation study of ContentCTR on pairwise loss and DTW align loss. We
study the effect of loss function $L_{Point}$, $L_{Pair}^{0}$, $L_{Pair}^{1}$,
$L_{Pair}^{2}$, $L_{Pair}^{3}$ and $L_{align}$.

Model |  $L_{Point}$ |  $L_{Pair}^{0}$ |  $L_{Pair}^{1}$ |  $L_{Pair}^{2}$ |  $L_{Pair}^{3}$ |  $L_{align}$ | Tau $\tau$ | avg. $s/y$
---|---|---|---|---|---|---|---|---
Model1 |  ✓ |  - |  - |  - |  - |  - | 0.5761 | 1.2789
Model2 |  ✓ |  ✓ |  - |  - |  - |  - | 0.5857 $\uparrow$ 0.96% | 1.3164
Model3 |  ✓ |  - |  ✓ |  - |  - |  - | 0.5872 $\uparrow$ 1.11% | 1.3021
Model4 |  ✓ |  - |  - |  ✓ |  - |  - | 0.5256 $\downarrow$ 5.05% | 1.3414
Model5 |  ✓ |  - |  - |  - |  ✓ |  - | 0.5824 $\uparrow$ 0.66% | 1.2194
Ours |  ✓ |  - |  ✓ |  - |  - |  ✓ | 0.5919 $\uparrow$ 1.58% | 1.3011

Figure 5. The change of $L_{Point}$ and $L_{Pair}$ of Model2 (blue line) and
Model3 (red line) during training. It is obvious that the pairwise loss of
Model2 is over-optimized which causes the pointwise loss to collapse in Model2
while both losses remain normal in Model3.

Although Model2 with $L_{Pair}^{0}$ has also shown performance improvement, we
have noticed that during training, the pointwise loss tends to collapse due to
the over-optimization of pairwise loss on Part 3, which is shown in Figure 5.

As expected, when Model3 incorporates $L_{align}$ for additional optimization,
the DTW alignment loss shows further improvement by +0.47%. Figure 6 shows
some visualizations of DTW alignment results for the KLive dataset, which
reveal that misalignment cases do exist in streaming scenarios. This also
indicates that the involvement of $L_{align}$ helps ContentCTR in training
better visual and text encoders which reduces the possible misalignment
between the two.

Figure 6. Visualization of Dynamic Time Warping (DTW) alignment results for
the KLive dataset. Each subfigure represents the cosine similarity between
video and text feature sequences. The red point denotes the DTW alignment path
found by our algorithm.

###  4.8. Ablation Study on PHD

We also conducted an additional ablation study on the PHD dataset based on
ContentCTR-HL, analyzing the impact of highlight frames, captions, DTW align
loss and input frames number. As shown in Table 5, the performance of
ContentCTR-HL decreased by -2.34% when the history feature is removed,
indicating that the history feature of highlight frames is crucial for
personalized video highlight detection. Similarly, the absence of captions
resulted in a -1.11% decrease in ContentCTR-HL’s performance, highlighting the
usefulness of text modality for the model’s detection capability.
Interestingly, ContentCTR-HL is found to be less sensitive to DTW alignment
loss. We hypothesize that this phenomenon is caused because most captions
generated by the official YouTuber are accurate enough. Apart from that, since
$L_{align}$ is a self-learning contrastive loss, it may require a larger data
scale to take effect, but currently, only 30% of all videos in PHD have
downloadable captions. We also find that the performance of ContentCTR-HL
improves with the increase of the input frames number. However, the increase
in performance is insignificant when the number of frames is increased from 96
to 192.

Table 5. Ablation Study on ContentCTR-HL.

Model | Frames Number $n$
---|---
32 |  64 |  96 |  192
ContentCTR-HL(Ours) |  18.29 |  19.54 |  21.59 |  21.89
Ours w/o history |  - |  - |  - |  19.55 $\downarrow$ 2.34%
Ours w/o caption |  - |  - |  - |  20.06 $\downarrow$ 1.11%
Ours w/o DTW |  - |  - |  - |  21.75 $\downarrow$ 0.14%

###  4.9. Online Experiments

We test the proposed framework in real-world live streaming scenarios through
online A/B testing. The experiment is conducted over four consecutive days,
with traffic randomly assigned to either the baseline method or our method. In
the baseline group, candidate live rooms are sorted by scores produced by a
traditional recommendation model, e.g., Click-Through Rate $s_{ctr}$, Long-
View-Through Rate $s_{lvtr}$. Note that these scores focus on capturing the
long-term relationship between streamers and users. In contrast, our method
utilizes content-based CTR as an additional factor like $s_{exp}$. This term
is capable of catching the highlight moment and show the most attractive live
contents to users. The results show that our method achieve a 2.9% and 5.9%
improvements in terms of CTR and live play duration, respectively, which
demonstrate the effectiveness of our content based model.

(16) $\displaystyle s_{base}$ $\displaystyle=s_{ctr}+s_{lvtr}+...$
$\displaystyle s_{exp}$ $\displaystyle=s_{ctr}+s_{lvtr}+...+s_{ContentCTR},$

##  5\. Conclusion

In this paper, we study the task of Click-Through Rates prediction in live
streaming scenario. We introduce ContentCTR, which utilizes a multimodal
transformer to achieve frame-level CTR prediction. Specifically, we propose a
streamer-personalized Perceiver Block that fuses ID embedding, visual, audio,
and comment embedding. The decoder network outputs the final CTR prediction
for each frame. To address the possible misalignment between video frames and
texts, we carefully design a Dynamic Time Warping (DTW) alignment loss for
optimization. Additionally, the boundary-aware constrained pairwise loss
demonstrates better performance when combined with the pointwise loss. We
conduct comprehensive experiments on both the KLive dataset and the public PHD
dataset, which demonstrate the effectiveness of our methods in both streaming
CTR prediction and video highlight detection tasks, compared with state-of-
the-art methods. Moreover, the proposed method has been deployed online on the
company’s short video platform and serves over three million daily users.

## References

  * (1)
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022\.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ 35 (2022), 23716–23736.
  * Bhattacharya et al. (2022) Uttaran Bhattacharya, Gang Wu, Stefano Petrangeli, Viswanathan Swaminathan, and Dinesh Manocha. 2022\.  Show Me What I Like: Detecting User-Specific Video Highlights Using Content-Based Multi-Head Attention. In _Proceedings of the 30th ACM International Conference on Multimedia_. 591–600.
  * Burges et al. (2005) Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005.  Learning to rank using gradient descent. In _Proceedings of the 22nd international conference on Machine learning_. 89–96.
  * Chen et al. (2019) Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, and Wenwu Ou. 2019.  Behavior sequence transformer for e-commerce recommendation in alibaba. In _Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data_. 1–4.
  * Chen et al. (2021) Runnan Chen, Penghao Zhou, Wenzhe Wang, Nenglun Chen, Pai Peng, Xing Sun, and Wenping Wang. 2021.  Pr-net: Preference reasoning for personalized video highlight detection. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7980–7989.
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. 7–10.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dixit (1990) Avinash K Dixit. 1990\.  _Optimization in economic theory_.  Oxford University Press, USA.
  * Gao et al. (2023) Ge Gao, Hongyan Liu, and Kang Zhao. 2023.  Live streaming recommendations based on dynamic representation learning.  _Decision Support Systems_ 169 (2023), 113957.
  * Garcia del Molino and Gygli (2018) Ana Garcia del Molino and Michael Gygli. 2018.  Phd-gifs: personalized highlight detection for automatic gif creation. In _Proceedings of the 26th ACM international conference on Multimedia_. 600–608.
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: a factorization-machine based neural network for CTR prediction.  _arXiv preprint arXiv:1703.04247_ (2017).
  * He et al. (2021) Li He, Hongxu Chen, Dingxian Wang, Shoaib Jameel, Philip Yu, and Guandong Xu. 2021\.  Click-through rate prediction with multi-modal hypergraphs. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 690–699.
  * Jocher (2020) Glenn Jocher. 2020\.  _YOLOv5 by Ultralytics_.  https://doi.org/10.5281/zenodo.3908559
  * Kendall (1945) Maurice G Kendall. 1945\.  The treatment of ties in ranking problems.  _Biometrika_ 33, 3 (1945), 239–251.
  * Ko et al. (2022) Dohwan Ko, Joonmyung Choi, Juyeon Ko, Shinyeong Noh, Kyoung-Woon On, Eun-Sol Kim, and Hyunwoo J Kim. 2022.  Video-text representation learning via differentiable weak temporal alignment. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5016–5025.
  * Liu et al. (2020) Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xiuqiang He, Zhenguo Li, and Yong Yu. 2020.  Autofis: Automatic feature interaction selection in factorization models for click-through rate prediction. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2636–2645.
  * Liu et al. (2009) Tie-Yan Liu et al. 2009\.  Learning to rank for information retrieval.  _Foundations and Trends® in Information Retrieval_ 3, 3 (2009), 225–331.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Lu et al. (2021) Wantong Lu, Yantao Yu, Yongzhe Chang, Zhen Wang, Chenhui Li, and Bo Yuan. 2021\.  A dual input-aware factorization machine for CTR prediction. In _Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence_. 3139–3145.
  * Müller (2007) Meinard Müller. 2007\.  Dynamic time warping.  _Information retrieval for music and motion_ (2007), 69–84.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Ouyang et al. (2019) Wentao Ouyang, Xiuwu Zhang, Li Li, Heng Zou, Xin Xing, Zhaojie Liu, and Yanlong Du. 2019.  Deep spatio-temporal neural networks for click-through rate prediction. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2078–2086.
  * Pi et al. (2020) Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. 2020\.  Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 2685–2692.
  * Qin et al. (2020) Jiarui Qin, Weinan Zhang, Xin Wu, Jiarui Jin, Yuchen Fang, and Yong Yu. 2020\.  User behavior retrieval for click-through rate prediction. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2347–2356.
  * Rappaz et al. (2021) Jérémie Rappaz, Julian McAuley, and Karl Aberer. 2021\.  Recommendation on live-streaming platforms: Dynamic availability and repeat consumption. In _Proceedings of the 15th ACM Conference on Recommender Systems_. 390–399.
  * Rendle (2010) Steffen Rendle. 2010\.  Factorization machines. In _2010 IEEE International conference on data mining_. IEEE, 995–1000.
  * Rochan et al. (2020) Mrigank Rochan, Mahesh Kumar Krishna Reddy, Linwei Ye, and Yang Wang. 2020.  Adaptive video highlight detection by learning from user history. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16_. Springer, 261–278.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2022) Hang Wang, Penghao Zhou, Chong Zhou, Zhao Zhang, and Xing Sun. 2022.  PAC-Net: Highlight Your Video via History Preference Modeling. In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIV_. Springer, 614–631.
  * Wang et al. (2017) Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. 2017\.  Deep & cross network for ad click predictions.  In _Proceedings of the ADKDD’17_. 1–7.
  * Wang et al. (2021) Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021.  Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In _Proceedings of the web conference 2021_. 1785–1797.
  * Xu et al. (2021) Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021.  Videoclip: Contrastive pre-training for zero-shot video-text understanding.  _arXiv preprint arXiv:2109.14084_ (2021).
  * Yang et al. (2013) Tzu-Wei Yang, Wen-Yuah Shih, Jiun-Long Huang, Wei-Chih Ting, and Pin-Chuan Liu. 2013.  A hybrid preference-aware recommendation algorithm for live streaming channels. In _2013 Conference on Technologies and Applications of Artificial Intelligence_. IEEE, 188–193.
  * Yu et al. (2021) Sanshi Yu, Zhuoxuan Jiang, Dong-Dong Chen, Shanshan Feng, Dongsheng Li, Qi Liu, and Jinfeng Yi. 2021.  Leveraging tripartite interaction information from live stream E-commerce for improving product recommendation. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 3886–3894.
  * Zhang et al. (2021a) Shuai Zhang, Hongyan Liu, Jun He, Sanpu Han, and Xiaoyong Du. 2021a.  A deep bi-directional prediction model for live streaming recommendation.  _Information Processing & Management_ 58, 2 (2021), 102453\.
  * Zhang et al. (2021b) Shuai Zhang, Hongyan Liu, Jun He, Sanpu Han, and Xiaoyong Du. 2021b.  Deep sequential model for anchor recommendation on live streaming platforms.  _Big Data Mining and Analytics_ 4, 3 (2021), 173–182.
  * Zhang et al. (2021c) Shuai Zhang, Hongyan Liu, Jun He, Sanpu Han, and Xiaoyong Du. 2021c.  Deep sequential model for anchor recommendation on live streaming platforms.  _Big Data Mining and Analytics_ 4, 3 (2021), 173–182.
  * Zhu et al. (2017) Zhenhui Zhu, Zhi Yang, and Yafei Dai. 2017.  Understanding the gift-sending interaction on live-streaming video websites. In _Social Computing and Social Media. Human Behavior: 9th International Conference, SCSM 2017, Held as Part of HCI International 2017, Vancouver, BC, Canada, July 9-14, 2017, Proceedings, Part I 9_. Springer, 274–285.


p_n(y, z)= p_n(ᾰ(0|y, z),0 |z,y, s),
If ∑_n ∈𝒩 p_n(ᾰ(0|y, z),0 |z,y, s) > p_max, then  b_n(y, z)  is a solution of $\beta$ to
∑_n ∈𝒩 p_n(ᾰ(β|z, y, s),β|y, z_n) = p_max,
α(y, z) = a(b_n(y, z)|y, z),
b_n(y, z)= b_n(a(b_n(y, z)|y, z),0 |y, z_n),
p_n(y, z)= p_n(a(b_n(y, z)|y, z),0 |y, z_n),

we let $\overline{\alpha}(y, \boldsymbol{z})$ (abbreviated as $\overline{\alpha}$ sometimes for simplicity) be
a solution of $\alpha> 0$ to $ \sum_{n \in \mathcal{N}} \widetilde{b}_n(\alpha,0 \mid y, z_n)  = b_{\textnormal{max}}$; i.e.,

∑_n ∈𝒩 b_n(α,0 |y, z_n)  = b_max ,
∑_n ∈𝒩 p_n(α,0 |y, z_n) ≤p_max.

Then satisfies $\sum_{n \in \mathcal{N}} \widetilde{p}_n(\alpha,0 \mid y, z_n) \leq p_{\textnormal{max}}$.

Case 2: If $\beta > 0$,
then we let $\widehat{\alpha}(y, \boldsymbol{z}) > 0$ and $\widehat{\beta}(y, \boldsymbol{z}) > 0$ (abbreviated as $\widehat{\alpha}$ and $\widehat{\beta}$ sometimes for simplicity) be
solutions of $\alpha$ and $\beta$ to

∑_n ∈𝒩 b_n(α,β|z,y)  = b_max ,
∑_n ∈𝒩 p_n(α, β,T,ζ|z,y, s) = p_max.

a solution of $\alpha$ to $ \sum_{n \in \mathcal{N}} \widetilde{b}_n(\alpha,0 \mid y, z_n)  = b_{\textnormal{max}}$ satisfies $\sum_{n \in \mathcal{N}} \widetilde{p}_n(\alpha,0 \mid y, z_n) > p_{\textnormal{max}}$

Let $\overline{\alpha}$ be
a solution of $\alpha$ to $ \sum_{n \in \mathcal{N}} \widetilde{b}_n(\alpha,0 \mid y, z_n)  = b_{\textnormal{max}}$

\begin{align}
\textnormal{Problem $\mathbb{P}_{66}(\iffalse T,\fi\boldsymbol{\zeta}\mid\boldsymbol{z},y,\boldsymbol{s})$}:\min_{\boldsymbol{b},\boldsymbol{p},\boldsymbol{f}^{\textnormal{MS}},\boldsymbol{f}^{\textnormal{VU}}} \quad & -H_{\mathbb{P}_{5}}(\boldsymbol{b},\boldsymbol{p},\boldsymbol{f}^{\textnormal{MS}},\boldsymbol{f}^{\textnormal{VU}},0\mid \boldsymbol{z},y,\boldsymbol{s})  +\sum_{n \in \mathcal{N}} \Big(\zeta_n  t_n(b_n,p_n,s_n,f_n^{\textnormal{MS}},f_n^{\textnormal{VU}}) \Big) \label{eqP5} \\
\textrm{s.t.} \quad &  \textnormal{(\ref{constraintbn})}, \textnormal{(\ref{constraintpn})}, \textnormal{(\ref{constraintfMS})}, \textnormal{(\ref{constraintfn})}
\end{align}

§.§ Proof of Lemma <ref>

§.§ Proof of Lemma <ref>

§.§ Proof of Proposition <ref>

Relationships between the notations for Steps 2.1–2.5.

\begin{align}
\begin{array}{l}
\textnormal{Given $[ \boldsymbol{z},y, \boldsymbol{s}]$,  \textbf{if} $\boldsymbol{\zeta}$ satisfies that with  $[\boldsymbol{b},\boldsymbol{p},\boldsymbol{f}^{\textnormal{MS}},\boldsymbol{f}^{\textnormal{VU}}, \alpha,\beta,\gamma,\boldsymbol{\delta}  ]$ being set as }  \\ \textnormal{$[\acute{\boldsymbol{b}}(\blacktriangle),\acute{\boldsymbol{p}}(\blacktriangle),\acute{\boldsymbol{f}}^{\textnormal{MS}}(\blacktriangle),\acute{\boldsymbol{f}}^{\textnormal{VU}}(\blacktriangle),\acute{\alpha}(\blacktriangle),\acute{\beta}(\blacktriangle),\acute{\gamma}(\blacktriangle),\acute{\boldsymbol{\delta}}(\blacktriangle)]$, there exists $T$ such that}  \\ \textnormal{(\ref{partialLpartialT}), (\ref{Complementarytime}), (\ref{constraintTtau}) and (\ref{Dualfeasibility}e) all hold, where ``$(\blacktriangle)$'' means ``$(\iffalse T,\fi\boldsymbol{\zeta}\mid \boldsymbol{z},y, \boldsymbol{s})$'',} \\ \textnormal{\textbf{then} $[\acute{\boldsymbol{b}}(\blacktriangle),\acute{\boldsymbol{p}}(\blacktriangle),\acute{\boldsymbol{f}}^{\textnormal{MS}}(\blacktriangle),\acute{\boldsymbol{f}}^{\textnormal{VU}}(\blacktriangle),T,\acute{\alpha}(\blacktriangle),\acute{\beta}(\blacktriangle),\acute{\gamma}(\blacktriangle),\acute{\boldsymbol{\delta}}(\blacktriangle),\boldsymbol{\zeta}]$ will become}\\ \textnormal{$[\boldsymbol{b}^{\#}(\textnormal{\small$\bigstar$}),\boldsymbol{p}^{\#}(\textnormal{\small$\bigstar$}),(\boldsymbol{f}^{\textnormal{MS}})^{\#}(\textnormal{\small$\bigstar$}),(\boldsymbol{f}^{\textnormal{VU}})^{\#}(\textnormal{\small$\bigstar$}),T^{\#}(\textnormal{\small$\bigstar$}), \alpha^{\#}(\textnormal{\small$\bigstar$}),\beta^{\#}(\textnormal{\small$\bigstar$}),\gamma^{\#}(\textnormal{\small$\bigstar$}),\boldsymbol{\delta}^{\#}(\textnormal{\small$\bigstar$})  ,\boldsymbol{\zeta}^{\#}(\textnormal{\small$\bigstar$})]$,}\\ \textnormal{where ``{\small$ (\bigstar)$}'' means ``$(\boldsymbol{z},y,\boldsymbol{s})$''. }
\end{array} \label{relation-bigstar-blacksquare}
\end{align}

\begin{align}
\begin{array}{l}
\textnormal{Given $[\iffalse T,\fi\boldsymbol{\zeta}, \boldsymbol{z},y, \boldsymbol{s}]$, \textbf{if} $\beta$ satisfies that setting $[\boldsymbol{b},\boldsymbol{p},\boldsymbol{f}^{\textnormal{MS}},\boldsymbol{f}^{\textnormal{VU}},\alpha,\gamma,\boldsymbol{\delta}]$ as }  \\ \textnormal{$[\breve{\boldsymbol{b}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{p}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{f}}^{\textnormal{MS}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{f}}^{\textnormal{VU}}(\textnormal{\small$\blacksquare$}),\breve{\alpha}(\textnormal{\small$\blacksquare$}),\breve{\gamma}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{\delta}}(\textnormal{\small$\blacksquare$})]$ ensures (\ref{Complementarybeta}), (\ref{constraintpn}) and (\ref{Dualfeasibility}b),}  \\ \textnormal{where ``$(\textnormal{\small$\blacksquare$})$'' means ``$(\beta,\iffalse T,\fi\boldsymbol{\zeta}\mid \boldsymbol{z},y, \boldsymbol{s})$'',} \\ \textnormal{\textbf{then} $[\breve{\boldsymbol{b}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{p}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{f}}^{\textnormal{MS}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{f}}^{\textnormal{VU}}(\textnormal{\small$\blacksquare$}),\breve{\alpha}(\textnormal{\small$\blacksquare$}),\beta,\breve{\gamma}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{\delta}}(\textnormal{\small$\blacksquare$})]$ will become}\\ \textnormal{$[\acute{\boldsymbol{b}}(\blacktriangle),\acute{\boldsymbol{p}}(\blacktriangle),\acute{\boldsymbol{f}}^{\textnormal{MS}}(\blacktriangle),\acute{\boldsymbol{f}}^{\textnormal{VU}}(\blacktriangle),\acute{\alpha}(\blacktriangle),\acute{\beta}(\blacktriangle),\acute{\gamma}(\blacktriangle),\acute{\boldsymbol{\delta}}(\blacktriangle)]$,}\\ \textnormal{where ``$(\blacktriangle)$'' means ``$(\iffalse T,\fi\boldsymbol{\zeta}\mid \boldsymbol{z},y, \boldsymbol{s})$''.}
\end{array} \label{relation-blacksquare-blacktriangle}
\end{align}

\begin{align}
\begin{array}{l}
\textnormal{Given $[\beta,\iffalse T,\fi\boldsymbol{\zeta}, \boldsymbol{z},y, \boldsymbol{s}]$, \textbf{if} $\alpha$ satisfies that setting $[\boldsymbol{b},\boldsymbol{p},\boldsymbol{f}^{\textnormal{MS}},\boldsymbol{f}^{\textnormal{VU}},\gamma,\boldsymbol{\delta}]$ as }  \\ \textnormal{$[\widetilde{\boldsymbol{b}}(\blacktriangledown),\widetilde{\boldsymbol{p}}(\blacktriangledown),\widetilde{\boldsymbol{f}}^{\textnormal{MS}}(\blacktriangledown),\widetilde{\boldsymbol{f}}^{\textnormal{VU}}(\blacktriangledown),\widetilde{\gamma}(\blacktriangledown),\widetilde{\boldsymbol{\delta}}(\blacktriangledown) ]$ ensures (\ref{Complementaryalpha}), (\ref{constraintbn}) and (\ref{Dualfeasibility}a),}  \\ \textnormal{where ``$(\blacktriangledown)$'' means ``$(\alpha, \beta,\iffalse T,\fi\boldsymbol{\zeta}\mid \boldsymbol{z},y, \boldsymbol{s})$'',} \\ \textnormal{\textbf{then} $[\widetilde{\boldsymbol{b}}(\blacktriangledown),\widetilde{\boldsymbol{p}}(\blacktriangledown),\widetilde{\boldsymbol{f}}^{\textnormal{MS}}(\blacktriangledown),\widetilde{\boldsymbol{f}}^{\textnormal{VU}}(\blacktriangledown),\alpha, \widetilde{\gamma}(\blacktriangledown),\widetilde{\boldsymbol{\delta}}(\blacktriangledown) ]$ will become}\\ \textnormal{$[\breve{\boldsymbol{b}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{p}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{f}}^{\textnormal{MS}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{f}}^{\textnormal{VU}}(\textnormal{\small$\blacksquare$}),\breve{\alpha}(\textnormal{\small$\blacksquare$}),\breve{\gamma}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{\delta}}(\textnormal{\small$\blacksquare$})]$,}\\ \textnormal{where ``$(\textnormal{\small$\blacksquare$})$'' means ``$(\beta,\iffalse T,\fi\boldsymbol{\zeta}\mid \boldsymbol{z},y, \boldsymbol{s})$''.}
\end{array} \label{relation-blacktriangledown-blacksquare}
\end{align}

After obtaining $\acute{\beta}(\blacktriangle)$ by solving (<ref>)–(<ref>), we let the $\beta$ parameter in $[\breve{\boldsymbol{b}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{p}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{f}}^{\textnormal{MS}}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{f}}^{\textnormal{VU}}(\textnormal{\small$\blacksquare$}),\breve{\alpha}(\textnormal{\small$\blacksquare$}),\breve{\gamma}(\textnormal{\small$\blacksquare$}),\breve{\boldsymbol{\delta}}(\textnormal{\small$\blacksquare$})]$ be $\acute{\beta}(\blacktriangle)$, and the corresponding results become$[\acute{\boldsymbol{b}}(\blacktriangle),\acute{\boldsymbol{p}}(\blacktriangle),\acute{\boldsymbol{f}}^{\textnormal{MS}}(\blacktriangle),\acute{\boldsymbol{f}}^{\textnormal{VU}}(\blacktriangle),\acute{\alpha}(\blacktriangle),\acute{\gamma}(\blacktriangle),\acute{\boldsymbol{\delta}}(\blacktriangle)]$.

Section <ref>-4: Solving for the “$(\textnormal{\small$\bigstar$})$” notations (i.e., Details for Page substeps's “Step 2.4” and “Step 2.5”). Based on (<ref>), we obtain the “$(\textnormal{\small$\bigstar$})$” (i.e., “$(\boldsymbol{z},y,\boldsymbol{s})$”) notations by enforcing (<ref>), (<ref>), (<ref>) and (<ref>e) on the “$(\textnormal{$\blacktriangle$})$” (i.e., “$(\iffalse T,\fi\boldsymbol{\zeta}\mid \boldsymbol{z},y, \boldsymbol{s})$”) notations; i.e.,

Given “$\textnormal{\small$\bigstar$}$”, the solution of $[T,\boldsymbol{\zeta}]$ to (<ref>)–(<ref>) is just $[T^{\#}(\textnormal{\small$\bigstar$}),\boldsymbol{\zeta}^{\#}(\textnormal{\small$\bigstar$})]$ (i.e., $[T^{\#}(\boldsymbol{z},y,\boldsymbol{s}),\boldsymbol{\zeta}^{\#}(\boldsymbol{z},y,\boldsymbol{s})]$).

To better understand  the  bisection search process,

we present Proposition <ref> below.

Given $[\beta,\iffalse T,\fi\boldsymbol{\zeta}, \textnormal{\small$\bigstar$}]$, we have that $\sum_{n \in \mathcal{N}} \widetilde{b}_n(\alpha, \beta,\iffalse T,\fi\boldsymbol{\zeta}\mid \textnormal{\small$\bigstar$})$ is  as $\alpha$ increases.


# Changes in the nature of the spectral continuum and the stability of the
cyclotron line in the X-ray pulsar GRO J2058+42

A. S. Gorban1,2∗, S. V. Molkov1, S.S.Tsygankov3,1, A.A. Mushtukov4,1 and A. A.
Lutovinov1,2

(1) Space Research Institute, Russian Academy of Sciences, Profsoyuznaya ul.
84/32, Moscow, 117997 Russia
(2) Higher School of Economics, National Research University, Myasnitskaya ul.
20, Moscow, 101100 Russia
(3) Department of Physics and Astronomy, FI-20014 University of Turku, Finland
(4) Leiden Observatory, Leiden University, NL-2300RA, Leiden, The Netherland

The results of the broadband spectral and timing study of the transient X-ray
pulsar GRO J2058+42 in a wide energy range at a low luminosity $L_{x}\simeq
2.5\times 10^{36}$ erg s-1 are reported. The data revealed that the pulse
profile and pulse fraction of the source are significantly changed in
comparison with previous NuSTAR observations, when the source was ten times
brighter. The cyclotron absorption line at $\sim 10$ keV in the narrow phase
interval is consistent with the high state observations. Spectral analysis
showed that at high luminosities $L_{x}\simeq(2.7-3.2)\times 10^{37}$ erg s-1
the spectrum has a shape typical of accreting pulsars, while when the
luminosity drops by about an order of magnitude, to $2.5\times 10^{36}$ erg
s-1 a two-component model is necessary to its describing. This behavior fits
into a model in which the low-energy part of the spectrum is formed in a hot
spot, and the high-energy part is formed as a result of resonant Compton
scattering by incident matter in an accretion channel above the surface of a
neutron star.

Keywords: GRO J2058+42 , X-ray sources, X-ray binaries, accretion, magnetic
field.
∗ E-mail<EMAIL_ADDRESS>
##  1 INTRODUCTION

The transient X-ray pulsar GRO J2058+42 was discovered in September 1995
during its giant outburst with the BATSE instrument BATSE on board the Compton
Gamma Ray Observatory (Wilson et al., 1995). The type II outburst (bright
outbursts in Be systems independent of the binary orbital phase with a peak
luminosity reaching the Eddington limit for a neutron star) lasted about 46
days with the maximum flux about 300 mCrab in the 20-100 keV energy range. The
pulse period is decreased from 198 s to 196 s during these days. After the
main burst GRO J2058+42 had remained active for approximately two years. The
pulsar demonstrated a series of weaker type I bursts (weak bursts associated
with the passage of a neutron star through the periastron of a binary system)
with the maximum pulsed flux of 10-15 mCrab (20-50 keV). The intervals between
bursts was about 110 days, which was interpreted as an orbital period in the
binary system. Small increases in brightness up to $\sim$8 mCrab were also
observed between the main flares (Wilson et al. 1995). These outbursts were
confirmed by the RXTE/ASM instrument and indicated a possible shorter orbital
period of 55 days (Corbet et al. 1997; Wilson et al. 2005).

After the period of activity GRO J2058+42 went into quiescence and exhibited
no outburst activity in X-rays until March 22, 2019, when the Burst Alert
Telescope (BAT) onboard the Neil Gehrels Swift observatory (Gehrels et al.,
2004) and the Gamma-ray Burst Monitor (GBM) onboard the Fermi observatory
(Meagan et al. 2009) detected a new giant outburst (Bartelmy et al. 2019;
Malakaria et al. 2019). The peak of luminosity this outburst was similar to
the previous outburst. During the 2019 outburst the source was monitored by
Swift, NICER, Fermi observatories. Two observations were performed also with
the NuSTAR observatory near the maximum luminosity (Molkov et al. 2019).

The periodic bursts, which follow the giant outburst, point that the optical
companion of binary system is a Be star. The optical companion was accurately
determined with the optical photometry and spectroscopy (Reig et al., 2005)
and was identified with the star of the O9.5-B0IV-Ve type, which is located at
a distance of $9\pm 1$ kpc. The distance to the binary system was estimated as
${8.0_{-1.2}^{+0.9}}$ kpc from the Gaia telescope data (Arnason et al. 2021).
This value will be used later in the article.

Molkov et al. (2019) discovered a cyclotron resonant scattering line at $\sim
10$ keV in the source spectrum and its harmonics at $\sim 20$ and $\sim 30$
keV by using data from the NuSTAR observatory. It was shown that the cyclotron
line and its harmonics were registered only in a narrow range of pulsar
rotation phases. Based on these measurements the magnetic strength of the
pulsar was estimated as $\sim 10^{12}$ G.

An analysis of the AstroSat observatory data confirmed the presence of the
cyclotron line in the GRO J2058+42 spectrum (Mukerjee et al., 2020) and
significant variations its parameters with the pulse phase. Moreover, authors
reported about a discovery of quasi-periodic oscillations with the frequency
of 0.09 Hz in the source power spectrum.

In order to study the properties of the pulsar at low accretion rates (see,
for example, the review by Tsygankov et al. 2017 of the properties of pulsars
in Be systems in low states), we asked another observation with the NuSTAR
observatory, which was carried out approximately 150 days after the maximum of
the main outburst. The flux $F_{3}=(3.3_{-2.8}^{+0.4})\times 10^{-10}$ erg
cm-2 s-1 was measured from the source in the energy range of 3-79 keV. This
value is about an order of magnitude lower than the fluxes detected from the
source during the first two NuSTAR observations: $F_{1}=(3.6\pm 0.1)\times
10^{-9}$ erg cm-2 s-1 and $F_{2}=(4.3\pm{0.1)\times 10^{-9}}$ erg cm-2 s-1
(Molkov et al. 2019).

This paper is devoted to studying the characteristics of a pulsar in the low
state and comparing them with those previously obtained for the bright state.

##  2 OBSERVATIONS AND DATA ANALYSIS

Table 1: Observations of GRO J2058+42, used in this paper Telescope | Date | ObsID | Exposure
---|---|---|---
NuSTAR | 25.03.2019 (MJD 58567) | 90501313002 | 20 ksec
NuSTAR | 28.08.2019 (MJD 58723) | 90501336002 | 38 ksec
Swift/XRT | 28.08.2019 (MJD 58723) | 00088982001 | 2 ksec

The NuSTAR observation of source with low state luminosity was carried out on
August 28, 2019 (MJD 58723) with an exposure time 38 ksec (ObsID 90501336002).
The observatory NuSTAR (Nuclear Spectroscopic Telescope ARray) is sensitive to
X-rays in the energy range of 3 - 79 keV. It consists of two co-aligned
identical telescopes (FPMA and FPMB) with an energy resolution of $\sim 400$
eV at 10 keV (Harrison et al., 2013)

The NuSTAR data were analyzed with the standard software (NuSTARDAS v1.9.7),
which is supplied as part of the HEASOFT v6.29 software with CALDB v20211202
calibrations.

Firstly, data processing was carried out with the NUPIPELINE procedure.
Further, we used NUPRODUCTS to obtain energy spectra and light curves of the
source. The GRO J2058+42 data of both modules were extracted from a circular
region of radius 50’’ centred on the source position. The background was
extracted with a similar radius of 50’’ on the same detector chip. To compare
properties of the pulsar in different intensity states, we used NuSTAR data
from one of the previous observation, which was carried out at March 25, 2019
(ObsID 90501313002) with the exposure of $\sim 20$ ksec (see Table 1). The
background was extracted on the adjacent chip due to a high brightness of the
source. Photons from the source and background were chosen from a circular
region of radius 70’’.

Figure 1: The light curve of X-ray pulsar GRO J2058+42 in several energy
ranges obtained with different instruments. The fluxes are given in different
units of measurement and re-normalized for the better clarity (see the inset).
Numbers point to the NuSTAR observations. Luminosity is shown in the energy
range of 1-10 keV to visualize the light curve profile in this picture.

To expand our spectral and timing analysis we used a Swift/XRT observation
with the exposure time of 2 ksec (ObsID 00088982001), which was made
simultaneously with the NuSTAR observation. The telescope Swift/XRT works in
the energy range of 0.3-10 keV. The source spectra were obtained with the
online tools (Evans et al. 2009) provided by the UK Swift Science Data Centre
at the University of Leicester.111http://www.swift.ac.uk/user\\_objects/ We
used the Photon Counting (PC) mode data for the analysis. The energy spectra
were also binned by 25 counts per bin for spectral analysis with $\chi^{2}$
statistic. The approximation of the spectra was carried out in the XSPEC
v12.12.0 package (Arnaud et al., 1999).

##  3 RESULTS

Several GRO J2058+42 observations made by the NuSTAR observatory during its
outburst activity in 2019 allowed us to make a detailed comparative analysis
of the temporal and spectral properties of the pulsar in states with an
accretion rate that differs by about order of magnitude: bright state
luminosity is $L_{x}\simeq 2.7\times 10^{37}$ erg s-1 (ObsID 90501313002), low
state is $L_{x}\simeq 2.5\times 10^{36}$ erg s-1 (ObsID 90501336002) (here and
further, value for energy range 3-79 keV uses). The light curve obtained from
the different instruments is shown in Fig.1.

Figure 2: Pulse profiles GRO J2058+42 in wide energy range 3-79 keV from
NuSTAR observatory. Red lines show bright state observation in March 25, 2019
(ObsID: 90501313002), green lines - low state observation in August 28, 2019
(ObsID: 90501336002).

###  3.1 Timing Analysis of the Emission from GRO J2058+42

For timing analysis of the NuSTAR data, photon arrival times were first
corrected for the barycenter of the Solar System using standard HEASOFT
package tools. Due to the lack of well-measured orbital parameters of the
system, the photon arrival time was not corrected for the orbital motion of
the neutron star. All light curves were obtained with a time resolution of 0.1
s, then the background was subtracted from them, and the light curves of the
two modules were combined using the lcmath tool (FTOOLS V6.29). Further, we
determined the pulse period of 193.375$\pm$0.001s using the epoch folding
technique (program efsearch from the HEASOFT package). The error in the period
was estimated using Monte Carlo simulations of the light curve (see Boldin et
al. 2013 for details)

Figure 3: Pulse profiles of GRO J2058+42 in low state in ten energy bands from
the NuSTAR data in August 2019. Profiles are shifted along the Y axis for
clarity.

The measured pulse period was used to obtain the pulse profile in a wide
energy range of 3-79 keV and comparison with the pulse profile in the bright
state (Fig. 2). It is clearly see that pulse profile was changed between two
observations. The main differences in the profiles are expressed in the main
peak – in the bright state at phases 0.25-0.35 its relative intensity is much
higher than in the low state. At the same time, a clearly visible deep narrow
minimum appears in the low state at phases 0.65-0.75. Also, an additional peak
is observed in the bright state at phases of 0.1, which possibly associated
with a more complex structure of emission regions on the surface of a neutron
star.

Figure 4: Pulsed fraction for GRO J2058+42 as a function of energy for two
NuSTAR observations: red crosses correspond to the bright state (March 25,
2019; ObsID: 90501313002), green color – low state (August 28, 2019; ObsID:
90501336002). Dashed lines indicate positions of the cyclotron line and its
harmonics.

At the next step we extracted light curves in the energy ranges of 3-5, 5-7,
7-9, 9-11, 11-15, 15-19, 19-23, 23-29, 29-41 and 41-79 keV and constructed
corresponding pulse profiles (Fig. 3). The figure demonstrates that the pulse
profile is changed significantly with the energy. Particularly, in the energy
range from 3 keV to $\sim$10 keV, the profile shows four separate peaks at
phases of 0.1, 0.3, 0.6, as in the bright state (see Molkov et al. 2019).
Also, a broad peak near phases 0.7-0.9 is appeared in the low state. As the
energy increases, the peak at phase 0.3 expands as in the bright state, the
peak at phase 0.7-0.9 persists at all energies up to $\sim 40$ keV. At higher
energies, the secondary peak disappears, which leads to the appearance of a
relatively flat region in a wide phase range of 0.5–1.0. This shape of the
pulse profile was also observed previously at energies of 20–70 keV (Wilson et
al. 1998).

We have plotted the dependence of the pulse fraction (PF) on the energy for
both NuSTARobservations. The PF was defined as the ratio
(Fmax$-$Fmin)/(Fmax+Fmin), where Fmax and Fmin are the maximum and minimum
fluxes in the pulse profile consisting of 20 bins, respectively. According to
Fig. 4, the PF in both states increases with the energy, which is typical of
most X-ray pulsars (Lutovinov and Tsygankov 2009). However, this growth is
non-monotonous. In particular, in the bright state (red dots), some feature
(local increase of the PF) is clearly visible around 10 keV, where the
cyclotron absorption line is located. In a low state (blue dots), a broken
dependence is also shown around energy. Peculiarities at energies
corresponding to the higher harmonics of the line are not visible due to
insufficient statistics and the large width of the energy channels. It is
important to note that the presence of similar features in the PF near the
cyclotron energy was shown earlier for a number of X-ray pulsars (Tsygankov et
al. 2007; Lutovinov and Tsygankov 2009; Tsygankov et al. 2010; Lutovinov et
al. 2017).

###  3.2 Spectral Analysis of the Source GRO J2058+42

We used continuum model in the form of thermal Comptonization (compTT in the
XSPEC package, Titarchuk, 1994). This model was applied by Molkov et al.
(2019) for description of bright state spectra, that allow to compare the
spectral properties of GRO J2058+42 in different intensity states. In
addition, the iron emission line at $\sim 6.4$ keV was added to the model. Its
width was frozen at 0.24 keV determined by Molkov et al. (2019).

The interstellar absorption was taken into account by adding the component
phabs into the model. An obtained value of absorption $N_{\rm H}\simeq 7\times
10^{21}$ cm-2 is in good agreement with the Galactic absorption in the
direction to the source $\sim 6.2\times 10^{21}$ cm-2 (HI4PI Collaboration, N.
Ben Bekhti, L. Floer, et al., 2016). In order to take into account the
different calibrations of the FPMA and FMPB modules aboard NuSTAR, as well as
the nonsimultaneity of the observations of NuSTAR and Swift, normalization
coefficients were added as well (see Table 2). The rest of the model
parameters for different instruments were fixed among themselves.

Figure 5: (a) Energy spectrum GRO J2058+42 measured in August 2019 by NuSTAR
(ObsID 90501336002) (green and black dots) and Swift/XRT (red dots), the solid
lines show the best-fit model; gray dots show the energy spectrum obtained by
NuSTAR in March 2019 (ObsID 90501313002). Panel (b) shows the residuals of the
observational data from the phabs*(gaussian+compTT) model, panel (c) is for
the phabs*(gaussian+compTT)*gabs model, panel (d) shows the residuals of the
observed data from the phabs*(gaussian+compTT+compTT) model.

In contrast to the data obtained in the bright state, for the low state, the
phabs*(gaussian+compTT) model shows an unsatisfactory quality of the fit with
$\chi^{2}$ = 1443.92 for 1218 degrees of freedom (d.o.f.) for the GRO J2058+42
broadband spectrum. Also, there are noticeable deviations in around the energy
of $\sim$27 keV (Fig. 5, b). Formally, to describe this feature, we added a
wide absorption line with a Gaussian profile gabs to the model. This resulted
in a significant improvement in the quality of the fit with $\chi^{2}$ =
1262.05 (1216 d.o.f.) (Fig. 5, c). However, a sufficiently large width of the
line (see Table 2) and the cyclotron absorption lines discovered earlier by
Molkov et al. (2019) at other energies cast doubt on the physical meaning of
this component. Therefore, we took a different approach to describe the
spectrum and used a combination of two Comptonization components (the compTT +
compTT model in XSPEC) as a continuum. The temperatures of the seed photons of
both components were tied to follow the approach of Tsygankov et al. (2019).
This model phabs*(gaussian+compTT+compTT) showed best quality fit with
$\chi^{2}$ = 1219.86 for 1214 d.o.f. (fig. 5, d). The model parameters are
presented in Table 2.

Previously, Molkov et al. (2019) showed that the phase-resolved spectra in the
bright state contain a cyclotron line at $\sim$10 keV at pulse phases
0.05-0.15, as well as its harmonics at $\sim$20 and $\sim$30 keV. We also
carried out phase-resolved spectroscopy of the third NuSTAR observation. We
used model phabs*(gaussian+compTT+compTT) for phase-resolved spectra, just as
for the averaged spectrum. This model without additional components showed the
adequate description of all phased spectra except for phases 0.05-0.15. The
fit of the spectrum at these phases revealed unsatisfactory quality with
$\chi^{2}$ = 495.61 for 406 d.o.f. due to the feature around 10 keV (Fig. 6,
b). To describe this feature we added a gabs component of about 10 keV (fig.
6, c), that significantly improved quality of fit with $\chi^{2}$ = 451.09 for
403 d.o.f. (see Table 3). Thus, the cyclotron absorption line at $\sim 10$ keV
was confirmed in the source spectrum at the low state at the same phases as in
previous observations. We were unable to register harmonics at energies of 20
and 30 keV due to the lack statistics in the phase-resolved spectrum.

Table 2: Parameters of the spectrum for GRO J2058+42 with the continuum described by the compTT+gabs and compTT+compTT Model parameters | compTT+gabs | compTT+compTT
---|---|---
$N_{H}$, $10^{22}$ cm-2 | ${0.2\pm{0.1}}$ | ${0.7_{-0.2}^{+0.2}}$
$E_{Fe}$, keV | 6.4, frozen | $6.32_{-0.08}^{+0.08}$
$\sigma_{Fe}$, keV | 0.24 | 0.24, frozen
$W_{Fe}$, keV | $0.03_{-0.01}^{+0.01}$ | $0.04_{-0.01}^{+0.02}$
$E_{Cycl}$, keV | $27.2\pm{0.7}$ | $-$
$\sigma_{Cycl}$, keV | $10\pm{2}$ | $-$
$\tau_{Cycl}$ | $0.53\pm{0.15}$ | $-$
$T_{0,Comptt},low$, keV | $1.27\pm{0.03}$ | $1.01_{-0.06}^{+0.05}$
$kT_{Comptt},low$, keV | $10.9\pm{0.7}$ | $3.9_{-0.4}^{+0.4}$
$\tau_{Comptt},low$ | $3.7\pm{0.4}$ | $8.0_{-1.8}^{+16.8}$
$T_{0,Comptt},high$, keV | $-$ | = $T_{0,Comptt},low$
$kT_{Comptt},high$, keV | $-$ | $14.6_{-1.2}^{+1.1}$
$\tau_{Comptt},high$ | $-$ | $2.9_{-0.5}^{+0.9}$
$Flux$ (3 - 79 keV), $10^{-10}$ | $3.2_{-0.1}^{+0.1}$ | $3.3_{-2.8}^{+0.4}$
erg cm-2 $s^{-1}$ |  |
$C_{NuSTAR}$ | $0.99_{-0.01}^{+0.01}$ | $0.99_{-0.01}^{+0.01}$
$C_{XRT}$ | $0.89_{-0.04}^{+0.04}$ | $0.89_{-0.04}^{+0.04}$
$\chi^{2}$ (d.o.f.) | 1262.05 (1216) | 1219.86 (1214)
Figure 6:  (a) Energy spectrum of GRO J2058+42 at the pulse phases 0.05-0.15 for NuSTAR observation in August 2019 (ObsID 90501336002). Both FPMA and FPMB data are shown as blue and black dots. The solid blue line shows the best-fit model, and the black and green lines show the contribution of different components. Panel (b) shows residuals of the observational data from the phabs*(gaussian+comptt+comptt) model similar to the average spectrum model, (c) for the model with the cyclotron line phabs*(gaussian+comptt+comptt)* gabs. Table 3: Parameters of the phased spectrum at 0.05-0.15 for GRO J2058+42 with the continuum described by the constanta*phabs*(gaussian+compTT+compTT)*gabs model Model parameters | Value
---|---
$N_{H}$, $10^{22}$ cm-2 |  ${0.7},$ fixed
$E_{Fe}$, keV |  $6.32$, fixed
$\sigma_{Fe}$, keV |  $0.24$, fixed
$W_{Fe}$, keV | $0.08_{-0.04}^{+0.04}$
$E_{Cycl}$, keV | $10.6_{-0.2}^{+0.2}$
$\sigma_{Cycl}$, keV | $1.3_{-0.2}^{+0.3}$
$\tau_{Cycl}$ | $0.31_{-0.02}^{+0.03}$
$T_{0,Comptt},low$, keV |  $1.01,$ fixed
$kT_{Comptt},low$, keV | $4.9_{-0.5}^{+0.7}$
$\tau_{Comptt},low$ | $4.8_{-0.5}^{+0.6}$
$T_{0,Comptt},high$, keV | = $T_{0,Comptt},low$
$kT_{Comptt},high$, keV | $14.5_{-1.6}^{+2.3}$
$\tau_{Comptt},high$ | >15
$C_{NuSTAR}$ | $0.98_{-0.02}^{+0.02}$
$\chi^{2}$ (d.o.f.) | 336.49 (362)

##  4 DISCUSSION

In the present work, it was shown that when the luminosity of the X-ray pulsar
GRO J2058+42 decreased to a few by $10^{36}$ erg s-1, the energy spectrum of
the source changed significantly. In particular, the approximation by the
thermal Comptonization model, which works well for the bright state of a
pulsar, no longer describes the spectrum. This happened in connection with a
clear change in the shape of the spectrum, expressed in its flattening in the
region of 10-30 keV. As shown above, an acceptable approximation of the
spectrum can be obtained by modifying the model by adding either an absorption
line at an energy of about 27 keV, or by adding a second emission component
with a high temperature. However, the too large width of the absorption
feature and the presence of true cyclotron lines in the phase-resolved spectra
of the source at other energies exclude the interpretation of spectrum
flattening through the cyclotron absorption.

The appearing of second emission component in the high energy of low
luminosity pulsars is a well-established fact (Tsygankov et al., 2019 a, b;
Lutovinov et al., 2021; Doroshenko et al., 2021). This behavior is associated
with the Comptonization of cyclotron photons in the overheated atmosphere of a
neutron star (Mushtukov et al., 2021; Sokolova-Lapa et al., 2021). In the case
of GRO J2058+42, the cyclotron line is at the lowest energy ($\sim 10$ keV) to
compare with all sources where such a change in the spectrum was observed. The
high-energy component of the spectrum cannot be associated with the emission
and following Comptonization of cyclotron photons in the hot atmosphere of a
neutron star due to the fact that the cyclotron line in the GRO J2058+42
spectrum is observed at an energy of $\sim 10$ keV.

On the other hand, the relatively low cyclotron energy makes possible and even
necessary another mechanism for the formation of a high-energy component in
the spectrum. Namely, in the subcritical accretion regime, which we assume for
GRO J2058+42 at the considered luminosity, the accretion channel has an
optical thickness of $<1$ with respect to Thomson scattering. However, at
resonance, where the scattering cross section is much larger than Thomson
scattering, the accretion channel is optically thick.

The position of the resonance in the reference frame of the neutron star
surface is redshifted due to the Doppler effect. The red shift changes
depending on the direction of photon escape from the accretion channel, and
resonant scattering is expected to cover a wide spectral band from $\sim 6$ to
$\sim 10$ keV. Photons that even once experienced resonant scattering by the
incident matter, whose velocity is $\sim 0.5c$, leave the accretion channel
with an energy much higher than before scattering, which forms the high-energy
part of the spectrum. Such a mechanism of high-energy part formation of the
spectrum implies a different beam pattern of the outgoing radiation of the
low-energy and high-energy parts of the spectrum. As the high-energy part of
the spectrum is the result of scattering in the accretion channel, it has to
have a beam pattern close to fan-shaped. Probably, the observed evolution of
the pulse profile with the photon energy is associated with this difference in
the beam patterns at high and low energies (see Fig. 3) (Mushtukov et al.,
2018).

##  5 CONCLUSIONS

In this paper we presented the results of timing and spectral analysis of the
X-ray pulsar GRO J2058+42 in a wide energy range based on the data obtained by
NuSTAR and Swift observatories in August 2019. According to the results, the
source was in the low luminosity state with $L_{x}\simeq 2.5\times 10^{36}$
erg s-1 which is an order of magnitude lower than previously observed. Timing
analysis has shown the changes in the energy-averaged pulse profile in
comparison with the bright state. The PF gradually increases with the energy,
that is typical of X-ray pulsars. Additionally, the PF indicates the presence
of a feature around the cyclotron energy in both low and bright states.

It has been also shown that the energy spectrum has changed greatly upon
transition to a low accretion rate. To describe it, we proposed a continuum
model consisting of two Comptonization components. The observed shape of the
spectrum in the low state can be interpreted in terms of a model where the
low-energy part of the spectrum is formed in a hot spot, while the high-energy
part is formed as a result of resonant Compton scattering of photons in the
range of $\sim 6-10$ keV by the incident matter in the accretion channel above
the surface of a neutron star.

The phase-resolved spectroscopy also showed a photon deficit around 10 keV in
the pulse phase interval 0.05-0.15. This feature can be interpreted as a
cyclotron absorption line. The position of the cyclotron line at $\sim$10 keV
in a narrow phase region is consistent with results of observations in the
high state.

##  6 ACKNOWLEDGMENTS

This study was carried out using the data obtained with NuSTAR, a Caltech
project, funded by NASA and operated by NASA/JPL and the data provided by the
UK Swift Science Data Centre (XRT data analysis). In this study we also used
the software provided by the High-Energy Astrophysics Science Archive Research
Center (HEASARC), which is a service of the NASA/GSFC Astrophysics Science
Division. This work was supported by RSF grant no. 19-12-00423.

### References

1. Arnason, R. M., Papei, H., Barmby, P., Bahramian, A., Gorski, M.D., MNRAS, 502, 4, 5455-5470 (2021)

2. Arnaud, K., Dorman, B., Gordon, C., Astrophysics Source Code Library, 10005 (1999)

3. Barthelmy, S. D., Evans, P. A., Gropp, J. D., et al., GRB Coordinates Network, 23985, 1 (2019)

4. HI4PI Collaboration: Ben Bekhti, N., Floer, L., et al., Astronomy & Astrophysics, 594, A116 (2016)

5. Boldin, P. A., Tsygankov, S. S., Lutovinov, A. A., Astron. Lett., 39, 375 (2013)

6. Gehrels, N., Chincarini, G., Giommi, P., et al., The Astrophysical Journal, 611, 1005 (2004)

7. Doroshenko, V., Santangelo, A., Tsygankov, S. S., and Ji, L., Astronomy & Astrophysics, 647, A165 (2021)

8. Corbet, R., Peele, A., and Remillard, R., International Astronomical Union Circular, 6556, 3 (1997)

9. Lutovinov, A., Tsygankov, S., Molkov, S., et al., The Astrophysical Journal, 912, 17 (2021)

10. Lutovinov, A. A. and Tsygankov, S. S., Astronomy Letters, 35, 433 (2009)

11. Lutovinov, A. A., Tsygankov, S. S., Postnov, K. A., et al., MNRAS, 466, 593 (2017)

12. Malacaria, C., Jenke, P., Wilson-Hodge, C. A., and Roberts, O. J., The Astronomer’s Telegram, 12614, 1 (2019)

13. Meegan, C., Lichti, G., Bhat, P. N., et al., The Astrophysical Journal, 702, 791 (2009)

14. Molkov, S., Lutovinov, A., Tsygankov, S., Mereminskiy, I., and Mushtukov, A., The Astrophysical Journal, 883, L11 (2019)

15. Mukerjee, K., Antia, H. M., and Katoch, T., The Astrophysical Journal, 897, 73 (2020)

16. Mushtukov, A. A., Suleimanov, V. F., Tsygankov, S. S., and Portegies Zwart, S., Monthly Notices of the Royal Astronomical Society, 503, 5193 (2021)

17. Reig, P., Negueruela, I., Papamastorakis, G., Manousakis, A., and Kougentakis, T., Astronomy & Astrophysics, 440, 637 (2005)

18. Sokolova-Lapa, E., Gornostaev, M., Wilms, J., et al., Astronomy & Astrophysics, 651, A12 (2021)

19. Titarchuk, L., AIP Conf. Proc., No. 304, 380 (1994)

20. Wilson, C. A., Finger, M. H., Harmon, B. A., Chakrabarty, D., and Strohmayer, T., The Astrophysical Journal, 499, 820 (1998)

21. Wilson, C. A., Weisskopf, M. C., Finger, M. H., et al., The Astrophysical Journal, 622, 1024 (2005)

22. Harrison, F. A., Craig, W. W., Christensen, F. E., et al., The Astrophysical Journal, 770, 103 (2013)

23. Tsygankov, S. S., Doroshenko, V., Mushtukov, A. A., et al., Monthly Notices of the Royal Astronomical Society: Letters, 487, L30 (2019a)

24. Tsygankov, S. S., Lutovinov, A. A., Churazov, E. M., and Sunyaev, R. A., Astronomy Letters, 33, 368 (2007)

25. Tsygankov, S. S., Lutovinov, A. A., and Serber, A. V., Monthly Notices of the Royal Astronomical Society, 401, 1628 (2010)

26. Tsygankov, S. S., Wijnands, R., Lutovinov, A. A., et al., Monthly Notices of the Royal Astronomical Society, 470, 126 (2017)

27. Tsygankov, S. S., Rouco Escorial, A., Suleimanov, V. F., et al., Monthly Notices of the Royal Astronomical Society: Letters, 483, L144 (2019b)

28. Evans, P. A., Beardmore, A. P., Page, K. L., et al., MNRAS, 397, 1177 (2009)