# PUNR: Pre-training with User Behavior Modeling for News Recommendation

Guangyuan Ma1,2, Hongtao Liu3, Xing Wu1,2, Wanhui Qian3, Zhepeng Lv3, Qing
Yang3, Songlin Hu1,2222Corresponding authors.
1Institute of Information Engineering, Chinese Academy of Sciences, Beijing,
China
2School of Cyber Security, University of Chinese Academy of Sciences, Beijing,
China
3Du Xiaoman Finance, Beijing, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

News recommendation aims to predict click behaviors based on user behaviors.
How to effectively model the user representations is the key to recommending
preferred news. Existing works are mostly focused on improvements in the
supervised fine-tuning stage. However, there is still a lack of PLM-based
unsupervised pre-training methods optimized for user representations. In this
work, we propose an unsupervised pre-training paradigm with two tasks, i.e.
user behavior masking and user behavior generation, both towards effective
user behavior modeling. Firstly, we introduce the user behavior masking pre-
training task to recover the masked user behaviors based on their contextual
behaviors. In this way, the model could capture a much stronger and more
comprehensive user news reading pattern. Besides, we incorporate a novel
auxiliary user behavior generation pre-training task to enhance the user
representation vector derived from the user encoder. We use the above pre-
trained user modeling encoder to obtain news and user representations in
downstream fine-tuning. Evaluations on the real-world news benchmark show
significant performance improvements over existing baselines.

##  1 Introduction

News recommendation is an essential technology for predicting candidate news
via the history of user behaviors. It is widely used in online news feeds and
platforms to model user behaviors and push personalized news to their users.

Existing news recommendation methods often employ Pre-trained Language Models
(PLMs) to produce news vectors and user vectors, and match personalized
candidate news based on the dot product distances of these two vectors. For
example, NRMS-BERT Wu et al. (2019c, 2021) replaces multi-head self-attention
with pre-trained PLM, such as BERT Devlin et al. (2019), RoBERTa Liu et al.
(2019) and UniLM Dong et al. (2019), to get news embeddings of user behaviors.
Then a hierarchical shallow user encoder is used on top of the news encoder,
in order to aggregate the user behaviors to get a user representation. Such
dual-tower PLM-based user modeling architecture is widely adopted in current
works of news recommendations. However, the separated forward of the
individual user behaviors (i.e. users’ clicked news titles) and hierarchical
aggregation result in potentially weak user modeling ability and complicated
model architectures.

Figure 1:  The architecture of our user modeling method for new
recommendations in downstream fine-tuning. A Siamese encoder is used for both
modeling the news and user embeddings. Note that we use the textual titles of
user-clicked news as inputs in pre-training, rather than a pure news id.

On the contrary, UNBERT Zhang et al. (2021b) proposes to concatenate the
candidate news and user behavior news together, and predict the matching score
via a classification head with the word-level vector and the news-level
vector. AMM Zhang et al. (2021a) concatenates the cross pairs of title,
abstract, and body together between the users’ history news and candidate
news, to extract multi-field matching representation in both within-field and
cross-field. Such single-tower classification techniques result in good user-
candidate behavior modeling but are relatively computationally expensive. In
order to achieve joint modeling of user behaviors while maintaining a balanced
computational offload, as is shown in Figure 1, we simplify the architecture
of the user encoder and only use a Siamese encoder for modeling both news
vectors and user vectors. The concatenation of user history news brings better
alignment of user behaviors. And we refer to this joint modeling of users’
clicked histories as User Modeling.

Effectively recommending content to users relies heavily on the ability to
accurately model their behaviors Sun et al. (2019). PTUM Wu et al. (2020a)
proposes pre-training multi-head self-attention layers from scratch to predict
masked behaviors and the next K behaviors. However, PTUM only utilizes a small
amount of in-domain data and does not leverage the power of PLMs.
Unfortunately, pre-training for news recommendation has not yet been
extensively explored, as most existing methods focus on improving the fine-
tuning stage by adapting an off-the-shelf PLM as the news encoder. We argue
that pre-training a PLM specifically tailored for news recommendation is
essential for achieving optimal performance. Moreover, research on modeling
user behaviors for news recommendations is still in its early stages.

Figure 2:  Pre-training design of our proposed method, PUNR. We incorporate a
special user behavior masking strategy and a user behavior generation task in
the pre-training stage of the user encoder.

In our work, we propose PUNR, Pre-training with User behavior modeling for
News Recommendation, seeking to incorporate unsupervised pre-training signals
for better user modeling ability. On top of the Siamese architecture, we
incorporate two pre-training tasks tailored for user modeling. Detailed
designs are shown in Figure 2.

$\bullet$ User behavior masking task. We randomly mask the whole behavior
spans in the inputs of the user encoder. The task for the user encoder is to
recover the masked behavior spans with its contextual user history news texts.
The modeling ability of the user encoder is improved and makes stronger
recommending performance.

Inspired by recent representation generative pre-training technology in IR
community Lu et al. (2021); Wu et al. (2022), the information can be
compressed into the representation vector through a generative pre-training
task with a shallow Transformer decoder. This auto-regression pre-training
schema enlights us to explore new pre-training paradigms for better modeling a
user vector in news recommendations.

$\bullet$ User behavior generation task. A single-layer auto-regression
Transformers decoder is added on top of the user encoder. The decoder takes
the user vector and user history news as input. The task for the decoder is to
generate the whole user history news in an auto-regression way assisted by the
user vector. This auto-regression pretask enables a more powerful user
representation learning.

To the best of our knowledge, we make the first effort to bring unsupervised
pre-training tasks to the PLM-based news recommender system. The joint use of
user behavior generation and user behavior masking design for user vector
optimization makes a steady recommendation improvement. We also propose a
simple yet effective Siamese architecture for encoding both the news vectors
and user vectors. Experiment results on the real-world news recommendation
task show considerable performance gains over multiple baselines. In addition,
we conduct several ablation studies to verify the robustness of our method.

Our contribution can be summarized as follows:
$\bullet$ We propose an effective user modeling Siamese encoder architecture
for both encoding the news and user representations.
$\bullet$ We incorporate user behavior generation and user behavior masking
tasks into the unsupervised pre-training of the PLM-based news recommendation
system.
$\bullet$ Experiments show that our work achieves considerable performance
gains over multiple baselines.

Figure 3:  The implementation of user model embeddings for PUNR. The title
fields of the clicked news are concatenated together for joint modeling of
each user’s behavior. The joint user modeling not only simplifies the
architecture of the user encoder to a single BERT model but also enables
stronger recommendation effects.

##  2 Approach

In this section, we describe the problem statement of news recommendation and
the detailed implementations of the pre-training and fine-tuning.

###  2.1 Problem Statement

Given a user with a set of clicked news
$\\{{D}_{1}^{u},{D}_{2}^{u},...,{D}_{|U|}^{u}\\}\in\mathcal{U}$ (denoted as
behaviors), the task for news recommendation is to distinguish preferable
(positive) news from candidate sets
$\\{{D}_{1}^{v},{D}_{2}^{v},...,{D}_{|V|}^{v}\\}\in\mathcal{V}$ based on
user’s behaviors.

###  2.2 Pre-training with User Behavior Modeling

We employ a Transformers-based Pretrained Language Model (PLM), e.g. BERT, to
serve as a user encoder to generate the representation of the user’s
behaviors. Specifically, given a clicked news set $\mathcal{U}$, we tokenize
each news input111Following previous works, we use the title of news as
inputs. and concatenate them together as a line of input tokens
$\mathbb{T}=\\{t_{0},t_{1},...,t_{n}\\}$ to produce token embeddings
$E_{token}$. Following Zhang et al. (2021b), we employ positional embeddings
$E_{pos}$ and news segment embeddings $E_{seg}$ to represent positional and
segmenting information. The input embeddings $E$ is the sum of the above three
embeddings, as is shown in Figure 3.

$E(\mathbb{T})=E_{token}+E_{pos}+E_{seg}$ (1)

#### User Behavior Masking

During the pre-training stage, we employ a user span masking strategy for
modeling the user behaviors with token-level Mask Language Modeling (MLM)
task. Given a total mask ratio $\alpha$, we especially use [MASK] tokens to
replace the whole piece of the news segment. The percentage of mask tokens
used by user behavior masking is $\beta$. And the rest of the tokens are
randomly masked. We denote the set of masked tokens as $\mathbb{M}$.

The masked text $m(\mathbb{T})$ is firstly converted to user embeddings
$E(m(\mathbb{T}))$, and then forwarded through the L-layer Transformers-based
user encoder ($Enc$) to recover all masked positions with the MLM loss. The
loss is formulated as:

$\mathcal{L}_{mlm}=-\sum_{i\in\mathbb{M}}\log p(t_{i}|Enc(E(m(\mathbb{T})))$
(2)

During the recovery of user behavior masking, the user model predicts the
entirely masked behaviors with unmasked or partially masked contexts. This
masking strategy will bring consistent user behavior alignment in user
modeling.

#### User Behavior Generation

For each layer $l\in\\{1,...,L\\}$ of the user encoder, the output hidden
states are denoted as follows:

$\mathbf{H}^{l}=\\{\mathbf{h}_{0}^{l},\mathbf{h}_{1}^{l},...,\mathbf{h}_{N}^{l}\\}$
(3)

Here we simply use the last hidden states of the [CLS] position, i.e.
${h}_{0}^{last}$ as the user vector. Based on the concatenated input of user
vector CLS and user embeddings, a single-layer Transformers-based auto-
regression decoder ($Dec$) is introduced to perform the generation task of the
entire user behavior texts.

$\mathcal{L}_{dec}=-\sum_{i\in\mathbb{T}}\log
p(t_{i+1}|Dec({h}_{0}^{last},E(\mathbb{T})))$ (4)

Similar to the discussion in Lu et al. (2021); Wu et al. (2022), the single-
layer decoder has limited modeling capacity. So it needs to rely on the
information from the user vector to finish the generation of entire textual
user behaviors. Thus the token-level behavior information is compressed into
the user vector, resulting in stronger user behavior modeling.

The pre-training loss is the sum of the encoder MLM loss and the decoder CLM
loss.

$\mathcal{L}=\mathcal{L}_{mlm}+\mathcal{L}_{dec}$ (5)

###  2.3 Fine-tuning

Fine-tuning is conducted on news recommendation benchmarks for verifying the
effectiveness of our work. For simplicity of downstream design, we share the
parameters between the news encoder and the user encoder, for both modeling
the candidate news and the user’s clicked news. The encoder is initialized
from the above pre-trained PLM. The clicked news is jointly forwarded through
the user encoder to obtain the user vector $u$ at the CLS position. And the
candidate news is separately forwarded through the news encoder to obtain the
individual news vectors $v$. Following NRMS-BERT Wu et al. (2021), we use the
negative sampling strategy, utilizing a cross-entropy loss for minimizing the
distance between the user vector $u$ and the positive news vector ${v}^{+}$
while pushing away the negative news vectors ${v}^{-}$.

$\mathcal{L}_{\mathrm{ft}}=-\log\frac{\exp({d(u,{v}^{+})}))}{\sum\exp(({d(u,{v}^{+})}+{d(u,{v}^{-})}))}$
(6)

where $d$ means the dot product operation.

##  3 Experiments

In this section, we introduce the experimental settings of our work, including
the pre-training, fine-tuning, and baseline methods. Then we analyze the main
experimental results.

###  3.1 Pre-training

#### Datasets

The MIND dataset Wu et al. (2020b) is used as the pre-training corpus. We use
2.2 million click behaviors from the training set of MIND for the pre-training
of our model. The target max length is set to 512.

Since the auto-regression decoder is not initialized, we use the original BERT
Devlin et al. (2019) pre-training corpus Wiki+BookCorpus for the
initialization of the decoder. It contains about 5.6 million documents. NLTK
sentence tokenizer is used to split the corpus and dynamically padded to the
target max length.

#### Implementation

The user encoder in the pre-training stage is initialized from the pre-trained
BERT-base checkpoint. We first freeze the BERT encoder and only pre-train the
single-layer auto-regression with general Wiki+BookCorpus. The AdamW optimizer
is used with a batch size of 512, a learning rate of 3e-4, and pre-training
steps of 150k for the full initialization of the decoder on the general
corpus. Then we pre-train the full model on MIND corpus using AdamW optimizer
with a batch size of 256, a learning rate of 1e-5, and pre-training steps of
10k. A linear scheduler with a warmup ratio of 0.1 is used for the learning
rate schedule. We set the total masking ratio $\alpha$ to 0.3 and the user
behavior masking ratio in the total masked token sets $\beta$ to 0.3 because
this combination brings the best results. We use 8 Tesla V100 GPUs and takes
about 1 day to finish the whole pre-training. We vary the pre-training seeds
and take an average of 3 scores as our main results.

###  3.2 Fine-tuning

#### Datasets

The MIND benchmark Wu et al. (2020b) is used to evaluate the effectiveness of
our method. It is a read-world English news recommendation dataset, which
contains 160k news texts, 1 million users, 15.8 million impressions, and 24
million click behaviors collected from Microsoft News in six weeks. MIND has
two versions of datasets with different sizes. MIND-large includes 2.2 million
samples in the training set and 365K samples in the development set. MIND-
small is a randomly sampled version of the whole dataset, which contains 50k
users and their behavior logs from the MIND dataset.

| MIND-Small | MIND-Large
---|---|---
Methods | AUC | MRR | nDCG@5 | nDCG@10 | AUC | MRR | nDCG@5 | nDCG@10
LibFM | 59.74 | 26.33 | 27.95 | 34.29 | 61.85 | 29.45 | 31.45 | 37.13
DeepFM | 59.89 | 26.21 | 27.74 | 34.06 | 61.87 | 29.3 | 31.35 | 37.05
DKN | 61.75 | 27.05 | 28.9 | 35.38 | 64.07 | 30.42 | 32.92 | 38.66
NPA | 63.21 | 29.11 | 31.7 | 37.81 | 65.92 | 32.07 | 34.72 | 40.37
NAML | 65.5 | 30.39 | 33.08 | 39.31 | 66.46 | 32.75 | 35.66 | 41.4
LSTUR | 64.38 | 29.46 | 31.89 | 38.17 | 67.08 | 32.36 | 35.15 | 40.93
NRMS | 64.83 | 30.01 | 32.52 | 38.92 | 67.66 | 33.25 | 36.28 | 41.98
FIM | 65.02 | 30.26 | 32.91 | 39.1 | 67.87 | 33.46 | 36.53 | 42.21
NRMS-BERT | 65.52† | 31.00† | 33.87† | 40.38† | 69.5 | 34.75 | 37.99 | 43.72
UNBERT | 67.62 | 31.72 | 34.75 | 41.02 | 70.68 | 35.68 | 39.13 | 44.78
AMM | 67.96 | 32.98 | 36.64 | 42.77 | - | - | - | -
PUNR | 68.89±0.17* | 33.33±0.08* | 36.94±0.11* | 43.10±0.09* | 71.03±0.04* | 35.17±0.05 | 39.04±0.04 | 45.41±0.05*

Table 1:  Main results on MIND datasets. The best scores are marked in bold.
*Two-tailed t-tests demonstrate statistically significant improvements in our
method over baselines ( p-value $\leq$ 0.05 ). †The scores are based on our
reproduction.

#### Implementation

The pre-trained encoder is used to initialize the downstream news and user
encoders. The parameters are shared across these two encoders because we found
that using a Siamese encoder improves performance. We fine-tune the encoder
using AdamW optimizer with a batch size of 128, and a learning rate of 3e-5
for 3 epochs. The linear scheduler with a warmup ratio of 0.1 is also used in
the fine-tuning stage. Following Wu et al. (2021), 1 positive and 4 negative
news are randomly sampled from the candidate sets. The max number of user
behaviors for the user encoder is set to 50, and the max length for each news
title is set to 30. The seed for fine-tuning is fixed to 42 for
reproducibility.

#### Evaluation Metrics

Following Wu et al. (2020b), several evaluation metrics, including AUC, MRR,
nDCG@5, and nDCG@10, are used for evaluating the performance. The scores are
averaged across all impression logs. We train the model on MIND training sets
and report the results on MIND development sets.

###  3.3 Baselines

We compare the performance of our model with multiple baselines, including
general TF-IDF Ramos et al. (2003) feature-based methods, convolutional neural
network (CNN-based) methods, deep neural network (DNN-based) methods, and PLM-
based methods. Most of the experiment results come from Zhang et al. (2021b),
while results of the PLM-based methods come from their original papers.
LibFM Rendle (2012) utilizes TF-IDF features of users’ clicked news and
candidate news in factorization machines.
DeepFM Guo et al. (2017) also uses TF-IDF features in deep factorization
machines.
DKN Wang et al. (2018) uses information from a knowledge graph and CNN
networks for news feature extraction.
NPA Wu et al. (2019b) utilizes the CNN network for news feature extracting and
personalized attention network incorporating user IDs.
NAML Wu et al. (2019a) applies the neural news recommendation with attentive
multi-view learning for news representation extraction.
LSTUR An et al. (2019) applies the neural news recommendation with long- and
short-term user representations for news recommendation.
NRMS Wu et al. (2019c) utilizes multi-head self-attention for news
recommendation.
FIM Wang et al. (2020) employs a fine-grained interest matching for neural
news recommendation.
NRMS-BERT Wu et al. (2021) is a dual-tower PLM-based neural news
recommendation method that uses PLM, such as BERT, for news feature
extraction.
UNBERT Zhang et al. (2021b) is a single-tower PLM-based neural news
recommendation method that concatenates candidate news and users’ clicked news
together and utilizes word-level and news-level features for click probability
prediction.
AMM Zhang et al. (2021a) is a single-tower PLM-based neural news
recommendation method that concatenates the cross pairs of title, abstract,
and body together from candidate news and users’ clicked news to extract
multifield matching representation for click probability prediction.

| AUC | MRR | nDCG@5 | nDCG@10
---|---|---|---|---
NRMS-BERT (Baseline) | 69.5 | 34.75 | 37.99 | 43.72
w/ User Modeling | 70.52 (+1.02) | 34.60 | 38.38 (+0.39) | 44.81 (+1.09)
w/ Only User Behavior Masking | 70.79 (+1.29) | 34.86 (+0.11) | 38.65 (+0.66) | 45.11 (+1.39)
w/ Only User Behavior Generation | 70.92 (+1.42) | 35.02 (+0.27) | 38.86 (+0.87) | 45.28 (+1.56)
w/ Both Pre-training Tasks | 71.07 (+1.57) | 35.19 (+0.44) | 39.07 (+1.08) | 45.45 (+1.73)
Table 2:  Ablation study for user modeling and different pre-training tasks on
MIND-Large benchmarks. Performance gains compared to the NRMS-BERT baseline
are marked in red.

###  3.4 Main Results

The main results are presented in Table 1. Scores on MIND-Small and MIND-Large
show a significant improvement over multiple baselines, including recent PLM-
based neural recommendation methods like NRMS-BERT, UNBERT, and AMM. Our
method completely beats the previous dual-tower methods NRMS-BERT. When
comparing with the single-tower baseline, although AMM utilizes multifield
matching information with a single-tower encoder for click probability
prediction, our dual-tower method still outperforms it by +0.93 on AUC, +0.35
on MRR, +0.3 on nDCG@5, +0.33 on nDCG@10 of MIND-Small benchmark. Our method
outperforms UNBERT by +0.35 on AUC and +0.63 on nDCG@10.

These performance gains could be attributed to our special user modeling
design with user behavior masking and user behavior generation tasks. These
tasks are tailored for user modeling and user vector pre-training, thus
bringing continuous unsupervised signals for effective user representation
learning. We will perform a detailed analysis in the next section.

Figure 4:  Performances with different total mask ratios $\alpha$ on MIND-
Large Benchmark.

##  4 Analysis

Given the performance gains of our user behavior pre-training methods, we
systematically analyze the following scientific questions in this section. Q1.
How does user modeling contribute to the performances? Q2. How do user
behavior masking and user behavior generation contribute to the performances?
Q3. How do total mask ratio $\alpha$ and behavior mask ratio $\beta$ influence
the performances? Q4. Is the initialization of the auto-regression decoder
necessary for the pre-training? Q5. How does performance change when choosing
a Siamese or fully separated encoder in fine-tuning? Q6. Will different
pooling methods affect the performances?

Detailed analyses (A) are conducted below one by one. The seed for all
experiments in this section is fixed to 42 for reproducibility.

Figure 5:  Performances with different user behavior masking ratios $\beta$ on
MIND-Large Benchmark.

###  4.1 Influence of User Modeling and User Behavior Pre-training Tasks

Table 2 shows the ablation results for user modeling and user behavior pre-
training tasks.

#### A1.

NRMS-BERT is a typical PLM-based news recommendation baseline method. After we
simplify the architecture of the user encoder and switch to the Siamese
encoder for user modeling fine-tuning, our model achieves +1.02 points on AUC
over the previous baseline.

We attribute this increment to better user modeling in the user encoder. The
concatenation of user-clicked histories brings token-level deep interaction
within Transformers blocks, thus bringing joint modeling and better alignment
to the user representation.

#### A2.

Besides the user modeling, we discuss the impact of different pre-training
tasks. After incorporating the user behavior masking and user behavior
generation tasks into the pre-training stage, our model achieves +1.29 and
+1.42 on AUC compared to the baseline. The usage of both two tasks achieves
+1.57 on AUC, which demonstrates the effectiveness of both pre-training tasks.

As discussed in Section 2, the user behavior masking task encourages the model
to predict the entirely masked textual span with other unmasked contextual
information, thus bringing a stronger signal for user behavior modeling. The
user behavior generation task utilizes a single-layer auto-regression decoder
to fully generate the entire user behaviors based on the user vector, thus
enriching the ability of the user representation during the backpropagation
update from the decoder layer.

AR-Dec | AUC | MRR | nDCG@5 | nDCG@10
---|---|---|---|---
Pre-trained | 71.07 | 35.19 | 39.07 | 45.45
Random init | 70.43 | 34.68 | 38.48 | 44.93

Table 3:  Ablation study for the initialization of Auto-Regression Decoder
(AR-Dec) with general corpus on MIND-Large benchmarks.

| AUC | MRR | nDCG@5 | nDCG@10
---|---|---|---|---
Siamese | 71.07 | 35.19 | 39.07 | 45.45
Separated | 70.32 | 34.61 | 38.37 | 44.84

Table 4:  Ablation study for the Siamese or separated encoder in fine-tuning
on MIND-Large benchmarks.

###  4.2 Influence of the Total Mask Ratio and User Behavior Masking Ratio

#### A3.

Performances with different total mask ratios are shown in Figure 4. We fix
the user behavior mask ratio $\beta$ at 30% and change the total mask ratio
between 15% and 60%. We empirically found that our pre-training method can
tolerate a wide range of total mask ratios, and slightly increasing it to 30%
or 45% will be helpful to achieve higher performance.

The results "w/ Only User Behavior Generation" in Table 2 show that a suitable
user behavior masking is helpful for higher performance. Thus we also adjust
the user behavior masking ratio $\beta$ to test a suitable setting. The total
mask ratio is fixed at 30%. We found that keeping the user behavior masking
ratio less than 30% achieves higher performance. We believe that making the
user behavior masking ratio too high will cause a much more challenging task,
thus unsuitable for the reconstruction of masked behaviors.

###  4.3 Impact for the Initialization of the Auto-Regression Decoder

#### A4.

Since the single-layer Auto-Regression Decoder (AR-Dec) can not be initialized
from a pre-trained checkpoint, we freeze the encoder and specifically train
the AR-Dec with the same general pre-training corpus as BERT. We compare the
pre-trained and random initialized AR-Dec in Table 3. Results show that the
decoder needs a proper initialization on the general corpus. A randomly
started decoder will corrupt the further pre-training with user behavior
generation task.

| AUC | MRR | nDCG@5 | nDCG@10
---|---|---|---|---
CLS | 71.07 | 35.19 | 39.07 | 45.45
Average | 70.95 | 35.03 | 38.92 | 45.28
Attention | 70.77 | 34.92 | 38.79 | 45.20

Table 5:  Ablation study for different pooling methods on MIND-Large
benchmarks.

###  4.4 Impact of Siamese or Fully Separated Encoder

#### A5.

Performances of using a Siamese or fully separated encoder for fine-tuning is
presented in Table 4. Similar to the observation in the IR community Dong et
al. (2022), sharing parameters between the news and user encoder brings better
alignment of news and user vectors, thus achieving higher performance.

###  4.5 Impact of Different Pooling Methods

#### A6.

Performances with different pooling methods are listed in Table 5. Following
Wu et al. (2021), three different pooling methods are compared in our works,
including 1) CLS: Directly using representation at CLS position as user
vector, which is widely used to obtain a sentence embedding. We use this
method in our main results. 2) Average: Using the average of hidden states
from the last layer. 3) Attention: Using a parameterized MLP weighting network
to aggregate a sentence embedding from the hidden states of the last layer.

Different from previous works Wu et al. (2021), our method achieves higher
scores on parameter-free pooling methods, e.g. CLS and Average, but has a
lower performance on Attention pooling methods. We believe that our pre-
training method with user behavior generation is tailored for direct user
representation learning. And a parameterized pooling method will weaken the
effort of this pre-training process.

##  5 Related Works

In this section, we introduce traditional neural news recommendation systems
and PLM-based neural news recommendation systems.

###  5.1 Neural News Recommendations

News recommendation systems aim to predict users’ preferred news based on
their browsing histories. Early manual feature-based recommendation systems
Rendle (2012); Guo et al. (2017) often employ deep factorization machines for
click-through rate prediction. In recent years, traditional news
recommendation systems utilize neural networks, e.g. convolutional neural
network (CNN) and deep neural network (DNN), for neural-based news and user
feature extraction. DKN Wang et al. (2018) utilizes knowledge-graph-enhanced
CNN networks for news feature extraction. NPA Wu et al. (2019b) also utilizes
CNN networks with personalized attention networks incorporating user IDs. NAML
Wu et al. (2019a) is a classical DNN-based neural news recommendation with
attentive multi-view learning for news representation learning. LSTUR An et
al. (2019) applies long and short-term user representations into DNN-based
neural news recommendation. NRMS Wu et al. (2019c) is another classical DNN-
based neural news recommendation method that utilizes multi-head self-
attention. FIM Wang et al. (2020) incorporates a fine-grained interest
matching into DNN-based methods. Traditional neural news recommendation
systems are mostly lightweight networks, designed for online inference of news
recommendations.

###  5.2 PLM-based News Recommendations

The bloom of Pre-trained Language Models (PLMs), such as BERT Devlin et al.
(2019), RoBERTa Liu et al. (2019) and UniLM Dong et al. (2019), invigorates
the development of PLM-based neural recommendation systems for higher
recommendation performances. With sufficient unsupervised pre-training corpus,
PLMs are powerful for representation extraction. Thus they are used as the
backbone of PLM-based news recommendation systems. PLM-NR Wu et al. (2021)
designs a dual-tower architecture that utilizes PLM for news embedding
extraction. A hierarchical user encoder is used for aggregating the user
embeddings. The distances between the news embeddings and the user embeddings
are calculated with dot product or cosine similarities. PLM-NR works as a
classical PLM-based news recommendation baseline. Similar to PLM-NR, we also
use a dual-tower architecture for inferencing. But we use concatenated user
clicked histories, rather than separated encoding, for effective user
modeling.

UNBERT Zhang et al. (2021b) proposes to concatenate candidate news and users’
clicked news together and use a single-tower BERT encoder for direct click
probability prediction. It aggregates word-level and news-level features and
achieves competing recommendation performances. AMM Zhang et al. (2021a)
pushes the steps further and proposes to concatenate the cross pairs of title,
abstract, and body together from candidate news and users’ clicked news. Thus
it extracts multifield matching representation for click probability
prediction. Similar to the above methods, we jointly model the user-clicked
behaviors. But we do not concatenate the candidates with the users’ behaviors,
because our work focuses on user representation optimization. Our work remains
a dual-tower architecture for balancing effectiveness and efficiency.
Furthermore, to the best of our knowledge, we are the first to bring
unsupervised pre-training tasks tailored for user modeling into the pre-
training stage of the user model. Our pre-training methods bring better
modeling ability to the user encoder, which is proven to be effective with
extensive experiments.

##  6 Conclusion

This paper proposes to pre-train with user behavior modeling for better news
recommendations. A simple yet efficient user modeling encoder is used for
better alignment of user behaviors. On top of this, we further incorporate the
user behavior masking task and the user behavior generation task for effective
unsupervised pre-training. Experiments on real-world news benchmarks show that
our works significantly improve the performances of PLM-based news
recommendations.

## Limitations

As our pre-training method is tailored for PLM-based news recommendations, it
requires relatively larger GPU resources than non-PLM-based methods. Moreover,
the joint modeling of the user encoder brings increased performance, but the
news embeddings from user behaviors cannot be pre-encoded and thus
uncacheable. This hinders fast inferencing in low-resource scenarios. We will
investigate the effect of downscaling PLMs or distillation for faster
inferencing, and leave this to further works.

## Ethics Statement

The authors declare that they have no conflict of interest.

## References

  * An et al. (2019) Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019.  Neural news recommendation with long- and short-term user representations.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 336–345. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 13042–13054.
  * Dong et al. (2022) Zhe Dong, Jianmo Ni, Dan Bikel, Enrique Alfonseca, Yuan Wang, Chen Qu, and Imed Zitouni. 2022.  Exploring dual encoder architectures for question answering.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9414–9419, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  Deepfm: A factorization-machine based neural network for CTR prediction.  In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017_ , pages 1725–1731. ijcai.org.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Lu et al. (2021) Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, and Arnold Overwijk. 2021.  Less is more: Pretrain a strong Siamese encoder for dense text retrieval using a weak decoder.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2780–2791, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ramos et al. (2003) Juan Ramos et al. 2003.  Using tf-idf to determine word relevance in document queries.  In _Proceedings of the first instructional conference on machine learning_ , volume 242, pages 29–48. Citeseer.
  * Rendle (2012) Steffen Rendle. 2012.  Factorization machines with libfm.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 3(3):1–22.
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019\.  Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer.  In _Proceedings of the 28th ACM international conference on information and knowledge management_ , pages 1441–1450.
  * Wang et al. (2020) Heyuan Wang, Fangzhao Wu, Zheng Liu, and Xing Xie. 2020.  Fine-grained interest matching for neural news recommendation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 836–845, Online. Association for Computational Linguistics.
  * Wang et al. (2018) Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. 2018.  DKN: deep knowledge-aware network for news recommendation.  In _Proceedings of the 2018 World Wide Web Conference on World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018_ , pages 1835–1844. ACM.
  * Wu et al. (2019a) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019a.  Neural news recommendation with attentive multi-view learning.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ , pages 3863–3869. ijcai.org.
  * Wu et al. (2019b) Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019b.  NPA: neural news recommendation with personalized attention.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_, pages 2576–2584. ACM.
  * Wu et al. (2019c) Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019c.  Neural news recommendation with multi-head self-attention.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6389–6394, Hong Kong, China. Association for Computational Linguistics.
  * Wu et al. (2021) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021.  Empowering news recommendation with pre-trained language models.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21, page 1652–1656, New York, NY, USA. Association for Computing Machinery.
  * Wu et al. (2020a) Chuhan Wu, Fangzhao Wu, Tao Qi, Jianxun Lian, Yongfeng Huang, and Xing Xie. 2020a.  PTUM: Pre-training user model from unlabeled user behaviors via self-supervision.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1939–1944, Online. Association for Computational Linguistics.
  * Wu et al. (2020b) Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020b.  MIND: A large-scale dataset for news recommendation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3597–3606, Online. Association for Computational Linguistics.
  * Wu et al. (2022) Xing Wu, Guangyuan Ma, Meng Lin, Zijia Lin, Zhongyuan Wang, and Songlin Hu. 2022\.  Contextual masked auto-encoder for dense passage retrieval.
  * Zhang et al. (2021a) Qi Zhang, Qinglin Jia, Chuyuan Wang, Jingjie Li, Zhaowei Wang, and Xiuqiang He. 2021a.  AMM: attentive multi-field matching for news recommendation.  In _SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021_ , pages 1588–1592. ACM.
  * Zhang et al. (2021b) Qi Zhang, Jingjie Li, Qinglin Jia, Chuyuan Wang, Jieming Zhu, Zhaowei Wang, and Xiuqiang He. 2021b.  UNBERT: user-news matching BERT for news recommendation.  In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021_ , pages 3356–3362. ijcai.org.


# Highly strain-tunable charge valley transport in bismuth

Suguru Hosoi1,† Fumu Tachibana1 Mai Sakaguchi1 Kentaro Ishida1 Masaaki
Shimozawa1 Koichi Izawa1 Yuki Fuseya2 Yuto Kinoshita3 Masashi Tokunaga3
1Graduate School of Engineering Science, Osaka University, Toyonaka, Osaka,
560-8531, Japan2Department of Engineering Science, University of Electro-
Communications,Chofu, Tokyo, 182-8585, Japan3Institute for Solid State
Physics, The University of Tokyo, Kashiwa, Chiba, 277-8581, Japan
†To whom correspondence should be addressed.
E-mail<EMAIL_ADDRESS>
###### Abstract

The manipulation of the valley degree of freedom can boost the technological
development of novel functional devices based on valleytronics. The current
mainstream platform for valleytronics is to produce a monolayer with inversion
asymmetry, in which the strain-band engineering through the substrates can
serve to improve the performance of valley-based devices. However, pinpointing
the effective role of strain is inevitable for the precise design of the
desired valley structure. Here, we demonstrate the valley-dependent charge
transport continuously controllable via the external strain for bulk bismuth
crystals with three equivalent electron valleys. The strain response of
resistance, namely elastoresistance, exhibits the evolutions in both
antisymmetric and symmetric channels with decreasing temperature. The
elastoresistance behaviors mainly reflect the significant changes in valley
density depending on the symmetry of induced strain, evidenced by our
developed semiclassical transport theory and strain-dependent quantum
oscillation measurements. These facts suggest the successful tune and
evaluation of the valley populations through strain-dependent charge valley
transport.

Quantum degrees of freedom provide a central ingredient for the applications
of functional electronic devices. Among them, the local conduction-band
minimum, valley, is attracting attention as a key element for high-profile
valleytronics, subsequently to charge for electronics and spin for
spintronicsSchaibleyNatRevMat2016 . A fundamental step for exploiting the
valley degrees of freedom is the development of the method for lifting and
monitoring the degenerated energy of valleys at different positions in
momentum space. Successful valley selection has so far been demonstrated by
various strategies: strain for 2D electron-gas systems in AlAs
heterostructurePhysRevLett.97.186404 , electric field for
diamondsIsbergNatMater2013 , polarized light for transition metal
dichalcogenidesZengNatNanotech2012 ; MakNanotecqh2012 ; CaoNC2012 , and
magnetic field for bismuthZhuNatPhys2012 ; KulcherNatMater2014 . In addition,
direct assessments of valleys have been reported in sophisticated spectroscopy
measurementsZengNatNanotech2012 ; MakNanotecqh2012 ; CaoNC2012 . However, it
requires more simple methods that serve as both a controller and a barometer
of valley degrees of freedom for potential valleytronic applications.

Figure 1: Schematic images of controlling valley structures by strain. a,
Valley polarized states selected by symmetry-breaking (antisymmetric) strain.
b, Uniform energy shift of valleys by symmetry-preserved (symmetric) strain.
Dashed and solid lines represent unstrained Brillouin zone and one deformed by
each type of strain, respectively. Figure 2: Two experimental geometry of
elastoresistance measurements in bismuth. a-c, Strain $\varepsilon_{xx}$
dependence of resistance $R_{xx}$ along binary (b) and temperature dependence
of elastoresistance ER|| (c) in the parallel geometry (a). Sample is glued on
the platform for applying strain. d-f, Strain $\varepsilon_{yy}$ dependence of
resistance (e) and temperature dependence of elastoresistance ER⟂ (f) in the
perpendicular geometry (d). #A represents the badge number of the samples.
Figure 3: Symmetry-resolved elastoresistance response of bismuth. a, Symmetric
components of elastoresistance for left axis. Right axis represents the
symmetric valley susceptibility based on the relation ER${}_{\rm
sym}=-\chi_{\rm sym}$. To ensure the reproducibility, we shows the results of
two samples, #A and #B. Inset shows a schematic picture of changes in valley
structure induced by tensile symmetric strain. Blue dashed and solid line
depict original and strain-decreased Brillouin zone (BZ), respectively. Orange
arrows represent the direction of the change in BZ by symmetric strain. b,
Antisymmetric components of elastoresistance represented by pink circles for
left axis. Purple diamonds represent valley susceptibility evaluated by
$\chi_{\rm anti}={\rm ER}_{\rm anti}/\gamma(T)$, which scale is shown in the
right axis. Inset represents schematics of valley polarizations induced by
antisymmetric strain. Blue dashed and solid line depict original and strain-
deformed Brillouin zone, respectively. Red arrows indicate the direction of
the change in BZ by antisymmetric strain.

One of the practical approaches is to control and evaluate valley degrees of
freedom through electrical transport. An appropriate material for this
situation is a single-element semimetal bismuth with three equivalent electron
valleys and one hole valleyJPIssi ; doi:10.7566/JPSJ.84.012001 ; Zhu_2018 . A
strong magnetic field ($B>40$ T) can completely polarize their electron
valleys depending on the direction of the applied magnetic field. For example,
under a magnetic field along the binary direction, one electron valley
survives, whereas the other two electron valleys disappear; this is completely
opposite to the case for the field along the bisectrixZhuNatCommun2017 ;
IwasaSciRep2019 . Furthermore, bismuth exhibits characteristic field-angle
dependent orbital magnetoresistance that can be captured by the semiclassical
transport theory with assumed ellipsoid shape of mobility tensors for hole and
one of the three equivalent electron valleysPhysRevX.5.021022 , respectively.
Therefore, bismuth is a good platform to describe the valley-dependent charge
transport. However, even a few Tesla of magnetic field that is enough to
induce finite valley polarization secondarily causes prominent quantum
oscillations, which makes it complicated beyond the scope of this
semiclassical treatment. Alternatively, our focused strain is expected to be
an effective tool to simply lift valley degeneracyBrandtSovPhys .

The potential roles of strain in valley materials are not only limited to
produce valley polarization, but expand to band engineering to acquire an
ideal valley structure; the schematic description of the strain-controlled
valley structure is described in Fig.1. As mentioned above, symmetry-breaking
anisotropic strain can directly break the degeneracy of valleys, as shown in
Fig.1b. On the other hand, in-plane symmetric strain does not induce valley
polarization but alternatively serves as tuning the band gap that determines
the capability of potential device applications by shifting the energy level
of the valleys, as shown in Fig.1a. In fact, an enormous effort mainly using
epitaxial strain has been directed to increasing the band gap for valley
materials: grapheneZhouNatMat , germaniumJurgenNatPhoto , and transition metal
dichalcogenidesPhysRevLett.105.136805 . Utilizing each symmetry channel of
strain can be useful for the precise design of the valley structure, and it is
highly desirable to elucidate how strain can control the valley structures.
Here, we demonstrate simultaneous control and evaluation of valleys in bismuth
via charge valley transport under the uniaxial stress for bismuth. The
application of uniaxial stress induces both these symmetric and antisymmetric
strain in the material. In order to clarify the effective role of the induced
strain depending on each symmetry channel, we performed symmetry-resolved
elastoresistance measurements of bismuth. Especially, since the crystal
structure of bismuth has two bilayers within the hexagonal
structureHOFMANN2006191 , the demonstration of strain-engineerable valleys in
bismuth can provide the significant insights for tuning valley profiles of
promising valleytronics candidates via epitaxial strain, such as graphene and
atomically thin transition metal dichalcogenides with a hexagonal crystal
structure.

Response in resistance of bismuth against the applied strain is summarized in
Fig. 2. Elastoresistance of bismuth exhibits contrasting results between two
experimental geometries. First, we measured longitudinal elastoresistance in
the parallel geometry depicted in Fig. 2a, where the current $j$ and induced
strain $\varepsilon_{xx}$ are along the binary ($x$) direction as
$j\,||\,\varepsilon_{xx}\,||\,{\rm binary}$, and it is defined as
ER${}_{||}=\frac{{\rm d}\Delta
R_{xx}(\varepsilon_{xx})/R_{xx}(\varepsilon_{xx}=0)}{{\rm
d}\varepsilon_{xx}}$. Here, $\Delta R_{xx}(\varepsilon_{xx})$ represents
strain-induced changes in binary-direction resistance $R_{xx}$: $\Delta
R_{xx}(\varepsilon_{xx})=R_{xx}(\varepsilon_{xx})-R_{xx}(\varepsilon_{xx}=0)$.
ER|| changes its sign from negative to positive on cooling with a broad
minimum structure, as shown in Figs. 2b,c. On the other hand, transverse one
ER${}_{\perp}=\frac{{\rm d}\Delta
R_{xx}(\varepsilon_{yy})/R_{xx}(\varepsilon_{yy}=0)}{{\rm d}\varepsilon_{yy}}$
in the perpendicular geometry (Fig. 2d), where $j$ is also along the binary
but the induced strain $\varepsilon_{yy}$ is along the bisectrix ($y$)
direction as $j\,\perp\,\varepsilon_{yy}\,||\,{\rm bisectirx}$, monotonically
increases with decreasing temperature, as shown in Figs. 2e,f. The observed
strain directional dependence is consistent with the previous study around the
room temperature using thin film samples, including each sign and
amplitudeKoike_1966 . The essential different temperature dependences between
ER|| and ER⟂ may reflect the mixing contributions from two symmetry channels.

To elucidate the origin of these strain direction-dependent behaviors, we
resolved two components of elastoresistance by combing the results of both
experimental geometries: symmetric component ER${}_{\rm
sym}=\frac{1}{1-\nu_{p}}({\rm ER}_{\rm||}+{\rm ER}_{\perp})$ and antisymmetric
component ER${}_{\rm anti}=\frac{1}{1+\nu_{p}}({\rm ER}_{\rm||}-{\rm
ER}_{\perp})$, as shown in Figs. 3a,b. Here, $\nu_{\rm p}$ represents the
Poisson ratio of the platform, where the sample is glued. We have performed
several samples to ensure the reproducibility of the results(see also
Supplementary information Section I). In high temperature regions, ERanti
dominates over ERsym. However, the magnitude of both ERanti and ERsym exhibits
the enhancement on cooling with opposite signs, indicating the sensitivity of
bismuth against multiple symmetry channel of the strain. This ERanti is
saturated roughly around $T\sim 50\,{\rm K}$, while ERsym shows continuous
enhancements. This symmetry crossover from antisymmetric to symmetric response
reflects a broad minimum in ER|| with sign change, as shown in Fig. 2c. By
contrast, in the perpendicular geometry, both two channels of elastoresistance
give cooperative contributions, leading to the strong enhancements of ER⟂, as
shown in Fig. 2f.

Figure 4: Strain-dependent quantum oscillations of sample #B along $B$ $||$
binary under two experimental geometries. a,b, Shubnikov-de Haas oscillations
under the parallel (a) and perpendicular geometry (b). The black arrow
indicates quantum limit (QL) for electron valleys e2/e3. c, Schematic
illustration of the Fermi surface area originating from the quantum
oscillations indicated by yellow lines on the valleys e2/e3. d, Both strain
$\varepsilon_{xx}$ and $\varepsilon_{yy}$ dependent shifts of quantum limits
of electron valleys e2/e3.

Here, we address the microscopic mechanism behind this strain-sensitive charge
transport of bismuth. We start from the semiclassical framework introduced in
Ref.PhysRevX.5.021022 . In this theory, the conductivity is described by the
summation of each valley contribution: $\sigma=\sum_{i}n_{i}e\mu_{i}$, where
$n_{i}$, $e$, and $\mu_{i}$ represent carrier density, the elementary charge,
and mobility of each valley with index $i$, respectively. Once the mobility
tensor of one electron valley and hole valley is fixed, this basic and simple
framework successfully captures the complicated field-angle dependences of
magnetoresistance in bismuthPhysRevX.5.021022 . However, the application of
strain alters both carrier density and mobility, describing that the
conductivity under strain is generally complicated. To solve this problem, we
propose the simplified effective carrier-based model under strain
$\varepsilon$ for each valley:
$\sigma_{i}(\varepsilon)=(n_{i}(\varepsilon=0)+\Delta
n_{i}(\varepsilon))e\mu_{i}=(1+\sum_{\Gamma}\chi^{i}_{\Gamma}\varepsilon_{\Gamma})n_{i}e\mu_{i}$,
where we only consider the strain-induced changes in carrier density $\Delta
n_{i}(\varepsilon)=n_{i}(\varepsilon)-n_{i}(\varepsilon=0)$. $\Delta
n_{i}(\varepsilon)$ are described by strain-valley susceptibility
$\chi^{i}_{\Gamma}=(1/n_{i}(\varepsilon=0)){\rm d}n_{i}/{\rm
d}\varepsilon_{\Gamma}$ that can be decomposed by the symmetry channel
$\Gamma$ of the induced strain: symmetric strain $\varepsilon_{\rm
sym}=(\varepsilon_{xx}+\varepsilon_{yy})/2$ and antisymmetric strain
$\varepsilon_{\rm anti}=(\varepsilon_{xx}-\varepsilon_{yy})/2$. This
simplified model can essentially capture the elastoresistance of bismuth. The
importance of this carrier density term for describing elastoresistance
behaviors has also been acknowledged for the WTe2doi:10.1073/pnas.1910695116 ,
which is one of the well-known semimetals with a small carrier concentration,
just like bismuth. We additionally extend the symmetry-dependent changes in
valley density to reflect the valley degree of freedom of bismuth, which
cannot be explored in WTe2 due to the low crystal symmetry and lack of valley
degrees of freedom.

The modification of valley structures is constrained by the symmetry of the
lattice deformation. Isotropic symmetric strain $\varepsilon_{\rm sym}$
preserves the rotational symmetry underlying the crystal lattice, leading to
the uniform change of valley population without breaking the equivalence of
three electron valleys. Adding the charge neutrality condition, the valley
population varies with symmetric strain as $\Delta n_{\rm e1}=\Delta n_{\rm
e2}=\Delta n_{\rm e3}=\Delta n_{\rm hole}/3=n\chi_{\rm sym}\varepsilon_{\rm
sym}$, where $n$ represents the valley density for one electron valley at
ambient stress. In this situation, symmetric strain only changes the total
carrier number described as $\chi_{\rm sym}$, and hence straightforwardly
connects with elastoresistance as ER${}_{\rm sym}=-\chi_{\rm sym}$ within the
carrier-based model neglecting the strain-induced modification of
mobility(Supplementary information
Section II). Based on this model, the observed positive sign of the symmetric
elastoresistance ERsym indicates that tensile strain, which reduces the size
of the Brillouin zone, decreases the carrier density of each valley, as shown
in the inset of Fig. 3a. This behavior is consistent with the first principle
study reporting that the overlap of the in-direct band gap between the hole
and electron becomes small by the expansion of the trigonal plane crystal
latticePhysRevB.91.125129 . This band modification reflects the charge
neutrality of semimetal and becomes significant particularly at low
temperatures, where only low energy bands near the Fermi level become
relevant. In fact, the elastoresistance of WTe2 at low temperatures is also
attributed to the charge neutral band modificationdoi:10.1073/pnas.1910695116
. Thus, ERsym clearly visualizes the temperature evolutions of uniform energy
shifts of electron and hole valleys, as shown in Fig. 3a.

On the other hand, symmetry-breaking antisymmetric strain $\varepsilon_{\rm
anti}$ can make a difference in the valley polarization in one valley e1 and
the other two valleys e2/e3: $\Delta n_{\rm e1}=n\chi_{\rm
anti}\varepsilon_{\rm anti}$ and $\Delta n_{e2/e3}=-n\chi_{\rm
anti}\varepsilon_{\rm anti}/2$. Now, $\chi_{\rm anti}$ represents the valley
susceptibility that evaluates the sensitivity of the valley polarization
against applied symmetry-breaking antisymmetric strain, which corresponds to
so-called nematic susceptibility applied to various iron-based
superconductorsdoi:10.1126/science.1221713 ; doi:10.1126/science.aab0103 ;
doi:10.1073/pnas.1605806113 ; doi:10.1073/pnas.2110501119 . In contrast to the
case of $\chi_{\rm sym}$, the relationships between $\chi_{\rm anti}$ and
ERanti depend on the anisotropy of valley mobility $\gamma$: ER${}_{\rm
anti}=-\frac{\mu^{\rm e1}_{xx}-\mu^{\rm e2}_{xx}/2-\mu^{\rm
e3}_{xx}/2}{\mu^{\rm e1}_{xx}+\mu^{\rm e2}_{xx}+\mu^{\rm
e3}_{xx}+3\nu_{xx}}\chi_{\rm anti}=\gamma\chi_{\rm anti}$, where $\mu^{\rm
e1}_{xx},\mu^{\rm e2}_{xx}$, and $\mu^{\rm e3}_{xx}$ represent the relevant
mobility tensor component of each electron valley, respectively, and
$\nu_{xx}$ is hole valley mobility tensor component. The detailed derivation
of this relation is provided in the Supplementary information Section II. The
relevant anisotropic factor in this experimental geometry is evaluated as
$\gamma\sim-0.35$ at low temperatures based on the previous
studiesPhysRevX.5.021022 ; PhysRev.181.1070 . The negative sign of $\gamma$
comes from the fact that valley e1 has higher mobility along binary than the
other electron valleys e2/e3, leading to the conductivity improvements by
positive antisymmetric strain-induced increases of the e1 valley density.
Figure 3b depicts the overall temperature dependence of $\chi_{\rm anti}$,
which incorporates the temperature dependence of $\gamma$PhysRevX.5.021022 ;
PhysRev.181.1070 ; J_PMichenaud_1972  (see also Supplementary information
Section II). The estimated $\chi_{\rm anti}$ is comparable to or even larger
than $\chi_{\rm sym}$, suggesting the strain sensitivity in the valley
densities against both symmetric and antisymmetric strains.

To strengthen our discussions, we try to evaluate the strain-induced changes
in valley populations through strain-dependent quantum oscillations of
bismuth. A magnetic field is applied along the binary direction in both strain
geometry, as shown in Figs. 4a,b. Three clear peaks are observed in the second
field derivative of resistivity derived from Shubnikov-de Haas(SdH)
oscillations of electron valleys e2/e3 shown in Fig. 4c. These peaks exhibit
strain sensitivity; for the parallel geometry, positive strain
$\varepsilon_{xx}$ shifts the peak position to a lower field side, as shown in
Fig. 4a, evidencing the shrink of the electron valleys e2/e3; by contrast,
opposite trends of each peak exhibit shifts toward higher field region under
positive strain $\varepsilon_{yy}$ for the perpendicular geometry, as shown in
Fig. 4b. Here, due to the small number of observable peaks, we focus on the
quantum limits to estimate the strain dependence of the valley density instead
of using the conventional fast Fourier transformation (FFT) analysis. Both
strain $\varepsilon_{xx}$ and $\varepsilon_{yy}$ dependent quantum limits for
valleys e2/e3 are shown in Fig. 4d. Combing these results gives another
evaluation of valley susceptibility: $\chi_{\rm sym}^{\rm QO}\sim-100$ and
$\chi_{\rm anti}^{\rm QO}\sim 280$(Supplementary information Section III).
These values are qualitatively consistent with the evaluations by
elastoresistance, including each magnitude and sign (See Figs. 3a,b).
Therefore, quantum oscillation measurements demonstrate that strain induced
valley density change coincides with valley susceptibility described by
elastoresistance, supporting the validity of our proposed simple carrier-based
transport model under strain. Returning to our original motivation,
antisymmetric strain successfully tunes electron valley degeneracy, which can
be evaluated by $\chi_{\rm anti}$. In addition, $\chi_{\rm anti}$ develops
with cooling, suggesting the manipulation capability of valley degrees of
freedom at low temperatures. Furthermore, large $\chi_{\rm sym}$ suggests that
tensile symmetric strain efficiently suppresses the indirect gap of bismuth,
which contributes to enhance another aspect of valley capability.

Finally, we discuss the possibility of a relationship with the nematic aspects
of bismuth. Bismuth, where three equivalent valleys are degenerate, can be
classified into $Z_{3}$ states. Thus, the nematic state of bismuth are
described as valley polarized states, which can be classified into the novel
$Z_{3}$ nematicity recently discussed in various materials such as
magnetismLittleArielleNatMater2020 , charge density waveLinpengNature2022 ,
and nematic superconductivityChangwooNC2020 . In fact, the possibility of
valley nematic states in bismuth has been discussed in low-temperature regions
under magnetic field in both bulk ZhuNatPhys2012 ; PhysRevX.5.021022  and
surface statesdoi:10.1126/science.aag1715 , although the former results are
recently attributed to the extrinsic effects due to the boundary
conductanceKangNC2022 . In that sense, the direct evaluation of valley density
in this study demonstrates the effective role of strain in controlling these
$Z_{3}$ orders. Increasing $\chi_{\rm anti}$ at low temperatures does not deny
the putative nematic state in bismuth. In fact, iron-based superconductors are
the representative metals that exhibit such a large ERanti comparable to
bismuth, owing to the critical divergence of nematic
susceptibilitiesdoi:10.1126/science.1221713 ; doi:10.1126/science.aab0103 ;
doi:10.1073/pnas.1605806113 ; doi:10.1073/pnas.2110501119 (see Supplementary
information
Section IV). However, such an enhancement in iron-based superconductors
generally occurs in only one symmetry channel since ordinary nematic materials
are only sensitive to the specific direction of strain that couples with the
symmetry of their own nematicity. Therefore, the evolution of $\chi_{\rm
sym}$, which reaches a comparable magnitude to $\chi_{\rm anti}$, clearly
demarcates bismuth from simple nematicity. Observed $\chi_{\rm anti}$ of
bismuth does not necessarily pinpoint rotational symmetry breaking field.
Rather, simultaneously large $\chi_{\rm sym}$ and $\chi_{\rm anti}$ describe
the sensitivity to any external perturbative stress field. The realization of
this strain sensitivity over multiple symmetry channels possibly originates
from the nature of semimetal with the smallness of fermi energy and charge
neutrality. The mixing ratio of induced symmetric strain to antisymmetric
strain by applied uniaxial pressure can change through the aspect ratio of the
crystal, and the fabrication of sample dimension can be one tool to tune the
desirable valley profiles.

## Methods

Sample preparations
Sample specimens were firstly spark-cut from the ingot of single-crystal
bismuth grown by the Czochoralski method. Then, those specimens were cleaved
and cut to achieve suitable dimensions for elastoresistance measurements:
typically 1 mm $\times$ 400 µm $\times$ 60 µm. Electric currents are applied
along a binary axis for all measured samples. Residual-Resistivity Ratio (RRR)
is 5-10 for their small-dimension specimens. Electrical resistance was
measured with a standard four-probe technique. Gold wires were attached with
Dupont 4929N silver paste to the electrodes.

Elastoresistance measurements
Elastoresistance measurements were carried out by using home-built piezo-
driven strain apparatus based on the design originally reported in
10.1063/1.4881611 . Uniaxial stress was applied to samples attached on the
rigid platform made of titanium to achieve large strain without the
destruction of the samples10.1063/5.0008829 . The resistive strain gauge was
attached to the backside of the platform to evaluate the amount of induced
strain. The bridge circuits with one active gauge and three dummy gauges were
built to measure the strain-induced changes in resistance of the strain gauge.

For decomposing the symmetric and antisymmetric responses in elastoresistance,
we have measured two different strain geometries, whose technique was
originally suggested in iron-based superconductorsPhysRevB.88.085113 . In this
study, we have measured binary($x$)-direction resistance $R_{xx}$ under stress
along binary($x$) or bisectrix($y$):

${\rm ER}_{||}=\frac{{\rm d}\Delta
R_{xx}(\varepsilon_{xx})/R_{xx}(\varepsilon_{xx}=0)}{{\rm
d}\varepsilon_{xx}},$ ${\rm ER}_{\perp}=\frac{{\rm d}\Delta
R_{xx}(\varepsilon_{yy})/R_{xx}(\varepsilon_{yy}=0)}{{\rm
d}\varepsilon_{yy}},$

where $\varepsilon_{xx}$ and $\varepsilon_{yy}$ represent strain along each
applied stress, respectively, and the strain-induced changes in resistance are
described as $\Delta
R_{xx}(\varepsilon)=R_{xx}(\varepsilon)-R_{xx}(\varepsilon=0)$. For
simplicity, we consider only components within the binary($x$)-bisectrix($y$)
plane. Symmetric and antisymmetric elastoresistance are given respectively:

${\rm ER}_{\rm sym}=\frac{1}{(1-\nu_{p})}({\rm ER}_{||}+{\rm ER}_{\perp}),$

${\rm ER}_{\rm anti}=\frac{1}{(1+\nu_{p})}({\rm ER}_{||}-{\rm ER}_{\perp}).$

Here, $\nu_{p}$ is an effective Poisson ratio of the platform directly
measured by strain gauges ($\nu_{p}\sim 0.197$).

## Acknowledgements

We thank M. Hecker and J. Scmalian for helpful discussions. We also thank N.
Miura for the provided bismuth samples. S.H. thanks J. Bartlett, A. Steppke,
and C. W. Hicks for making him aware of the importance of studying strain-
response of bismuth and sharing the the technique for developing the strain
cell through the other collaboration works. This work was supported by Grants-
in-Aid for Scientific Research (KAKENHI) (Nos. JP18H01167, JP20K20901,
JP22H01939, JP22K03522, JP22K18690, 23H04862, JP23K17879) and Grand-in-Aid for
Scientific Research on innovative areas “Quantum Liquid Crystals” (No.
JP20H05162) from Japan Society for the Promotion of Science. S.H. and M.
Shimozawa were supported by the Multidisciplinary Research Laboratory System
(MRL), Osaka University, respectively.

## Author contributions

S.H. conceived and supervised this project. S.H., F.T., and K.Ishida,
developed strain apparatus. S.H., F.T., M.Sakaguchi, and M.Shimozawa performed
transport measurements under strain with the help of K.Izawa. S.H., F.T., and
M.Sakaguchi analyzed data. S.H. and Y.F. established transport model under
strain, and S.H. and F.T. performed numerical calculations based on this
model. Y.K. and M.T. prepared the single crystal of bismuth. S.H. and
M.Shimozawa prepared the manuscript with theoretical inputs from Y.F.. All
authors commented on the manuscript.

## References

  * (1) J. R. Schaibley, H. Yu, G. Clark, P. Rivera, J. S. Ross, K. L. Seyler, W. Yao, X. Xu, Valleytronics in 2D materials. Nature Reviews Materials 1, 16055 (2016).
  * (2) O. Gunawan, Y. P. Shkolnikov, K. Vakili, T. Gokmen, E. P. De Poortere, M. Shayegan, Valley susceptibility of an interacting two-dimensional electron system. Phys. Rev. Lett. 97, 186404 (2006).
  * (3) J. Isberg, M. Gabrysch, J. Hammersberg, S. Majdi, K. K. Kovi, D. J. Twitchen, Generation, transport and detection of valley-polarized electrons in diamond. Nature Materials 12, 760–764 (2013).
  * (4) H. Zeng, J. Dai, W. Yao, D. Xiao, X. Cui, Valley polarization in MoS2 monolayers by optical pumping. Nature Nanotechnology 7, 490–493 (2012).
  * (5) K. F. Mak, K. He, J. Shan, T. F. Heinz, Control of valley polarization in monolayer MoS2 by optical helicity. Nature Nanotechnology 7, 494–498 (2012).
  * (6) T. Cao, G. Wang, W. Han, H. Ye, C. Zhu, J. Shi, Q. Niu, P. Tan, E. Wang, B. Liu, J. Feng, Valley-selective circular dichroism of monolayer molybdenum disulphide. Nature Communications 3, 887 (2012).
  * (7) Z. Zhu, A. Collaudin, B. Fauqué, W. Kang, K. Behnia, Field-induced polarization of dirac valleys in bismuth. Nature Physics 8, 89–94 (2012).
  * (8) R. Küchler, L. Steinke, R. Daou, M. Brando, K. Behnia, F. Steglich, Thermodynamic evidence for valley-dependent density of states in bulk bismuth. Nature Materials 13, 461–465 (2014).
  * (9) J.-P. Issi, Low temperature transport properties of the group V semimetals. Australian Journal of Physics 32, 585–628 (1979).
  * (10) Y. Fuseya, M. Ogata, H. Fukuyama, Transport properties and diamagnetism of dirac electrons in bismuth. Journal of the Physical Society of Japan 84, 012001 (2015).
  * (11) Z. Zhu, B. Fauqué, K. Behnia, Y. Fuseya, Magnetoresistance and valley degree of freedom in bulk bismuth. Journal of Physics: Condensed Matter 30, 313001 (2018).
  * (12) Z. Zhu, J. Wang, H. Zuo, B. Fauqué, R. D. McDonald, Y. Fuseya, K. Behnia, Emptying dirac valleys in bismuth using high magnetic fields. Nature Communications 8, 15297 (2017).
  * (13) A. Iwasa, A. Kondo, S. Kawachi, K. Akiba, Y. Nakanishi, M. Yoshizawa, M. Tokunaga, K. Kindo, Thermodynamic evidence of magnetic-field-induced complete valley polarization in bismuth. Scientific Reports 9, 1672 (2019).
  * (14) A. Collaudin, B. Fauqué, Y. Fuseya, W. Kang, K. Behnia, Angle dependence of the orbital magnetoresistance in bismuth. Phys. Rev. X 5, 021022 (2015).
  * (15) N. B. Brandt, V. A. Kul’bachinkii, N. Y. Minina, V. D. Shirokikh, Change of the band structure and electronic phase transitions in Bi and Bi1-xSbx alloys under uniaxial tension strains. Sov. Phys. JETP 51, 562 (1980).
  * (16) S. Y. Zhou, G. H. Gweon, A. V. Fedorov, P. N. First, W. A. de Heer, D. H. Lee, F. Guinea, A. H. Castro Neto, A. Lanzara, Substrate-induced bandgap opening in epitaxial graphene. Nature Materials 6, 770–775 (2007).
  * (17) J. Michel, J. Liu, L. C. Kimerling, High-performance Ge-on-Si photodetectors. Nature Photonics 4, 527–534 (2010).
  * (18) K. F. Mak, C. Lee, J. Hone, J. Shan, T. F. Heinz, Atomically thin $\mathrm{MoS}_{2}$: A new direct-gap semiconductor. Phys. Rev. Lett. 105, 136805 (2010).
  * (19) P. Hofmann, The surfaces of bismuth: Structural and electronic properties. Progress in Surface Science 81, 191-245 (2006).
  * (20) R. Koike, H. Kurokawa, Elastoresistance effects in evaporated bismuth films. Japanese Journal of Applied Physics 5, 503 (1966).
  * (21) N. H. Jo, L.-L. Wang, P. P. Orth, S. L. Bud’ko, P. C. Canfield, Magnetoelastoresistance in WTe2: Exploring electronic structure and extremely large magnetoresistance under strain. Proceedings of the National Academy of Sciences 116, 25524-25529 (2019).
  * (22) I. Aguilera, C. Friedrich, S. Blügel, Electronic phase transitions of bismuth under strain from relativistic self-consistent $GW$ calculations. Phys. Rev. B 91, 125129 (2015).
  * (23) J.-H. Chu, H.-H. Kuo, J. G. Analytis, I. R. Fisher, Divergent nematic susceptibility in an iron arsenide superconductor. Science 337, 710-712 (2012).
  * (24) H.-H. Kuo, J.-H. Chu, J. C. Palmstrom, S. A. Kivelson, I. R. Fisher, Ubiquitous signatures of nematic quantum criticality in optimally doped Fe-based superconductors. Science 352, 958-962 (2016).
  * (25) S. Hosoi, K. Matsuura, K. Ishida, H. Wang, Y. Mizukami, T. Watashige, S. Kasahara, Y. Matsuda, T. Shibauchi, Nematic quantum critical point without magnetism in FeSe1-xSx superconductors. Proceedings of the National Academy of Sciences 113, 8139-8143 (2016).
  * (26) K. Ishida, Y. Onishi, M. Tsujii, K. Mukasa, M. Qiu, M. Saito, Y. Sugimura, K. Matsuura, Y. Mizukami, K. Hashimoto, T. Shibauchi, Pure nematic quantum critical point accompanied by a superconducting dome. Proceedings of the National Academy of Sciences 119, e2110501119 (2022).
  * (27) R. Hartman, Temperature dependence of the low-field galvanomagnetic coefficients of bismuth. Phys. Rev. 181, 1070–1086 (1969).
  * (28) J. P. Michenaud, J. P. Issi, Electron and hole transport in bismuth. Journal of Physics C: Solid State Physics 5, 3061 (1972).
  * (29) A. Little, C. Lee, C. John, S. Doyle, E. Maniv, N. L. Nair, W. Chen, D. Rees, J. W. F. Venderbos, R. M. Fernandes, J. G. Analytis, J. Orenstein, Three-state nematicity in the triangular lattice antiferromagnet Fe1/3NbS2. Nature Materials 19, 1062–1067 (2020).
  * (30) L. Nie, K. Sun, W. Ma, D. Song, L. Zheng, Z. Liang, P. Wu, F. Yu, J. Li, M. Shan, D. Zhao, S. Li, B. Kang, Z. Wu, Y. Zhou, K. Liu, Z. Xiang, J. Ying, Z. Wang, T. Wu, X. Chen, Charge-density-wave-driven electronic nematicity in a kagome superconductor. Nature 604, 59–64 (2022).
  * (31) C.-w. Cho, J. Shen, J. Lyu, O. Atanov, Q. Chen, S. H. Lee, Y. S. Hor, D. J. Gawryluk, E. Pomjakushina, M. Bartkowiak, M. Hecker, J. Schmalian, R. Lortz, $Z_{3}$-vestigial nematic order due to superconducting fluctuations in the doped topological insulators NbxBi2Se3 and CuxBi2Se3. Nature Communications 11, 3056 (2020).
  * (32) B. E. Feldman, M. T. Randeria, A. Gyenis, F. Wu, H. Ji, R. J. Cava, A. H. MacDonald, A. Yazdani, Observation of a nematic quantum hall liquid on the surface of bismuth. Science 354, 316-321 (2016).
  * (33) W. Kang, F. Spathelf, B. Fauqué, Y. Fuseya, K. Behnia, Boundary conductance in macroscopic bismuth crystals. Nature Communications 13, 189 (2022).
  * (34) C. W. Hicks, M. E. Barber, S. D. Edkins, D. O. Brodsky, A. P. Mackenzie, Piezoelectric-based apparatus for strain tuning. Review of Scientific Instruments 85, 065003 (2014).
  * (35) J. Park, J. M. Bartlett, H. M. L. Noad, A. L. Stern, M. E. Barber, M. König, S. Hosoi, T. Shibauchi, A. P. Mackenzie, A. Steppke, C. W. Hicks, Rigid platform for applying large tunable strains to mechanically delicate samples. Review of Scientific Instruments 91, 083902 (2020).
  * (36) H.-H. Kuo, M. C. Shapiro, S. C. Riggs, I. R. Fisher, Measurement of the elastoresistivity coefficients of the underdoped iron arsenide Ba(Fe0.975Co0.025)2As2. Phys. Rev. B 88, 085113 (2013).