#  ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT

Rui Pan<EMAIL_ADDRESS>
Department of Computer Science and Engineering
The Hong Kong University of Science and Technology, Hong Kong Shizhe
Diao11footnotemark: 1<EMAIL_ADDRESS>
Department of Computer Science and Engineering
The Hong Kong University of Science and Technology, Hong Kong Jianlin
Chen11footnotemark: 1<EMAIL_ADDRESS>
Department of Social Sciences
University of Southampton, UK Tong Zhang<EMAIL_ADDRESS>
Department of Computer Science and Engineering
The Hong Kong University of Science and Technology, Hong Kong Equal
contribution.

###### Abstract

In this paper, we present ExtremeBERT, a toolkit for accelerating and
customizing BERT pretraining. Our goal is to provide an easy-to-use BERT
pretraining toolkit for the research community and industry. Thus, the
pretraining of popular language models on customized datasets is affordable
with limited resources. Experiments show that, to achieve the same or better
GLUE scores, the time cost of our toolkit is over $6\times$ times less for
BERT Base and $9\times$ times less for BERT Large when compared with the
original BERT paper. The documentation and code are released at
https://github.com/extreme-bert/extreme-bert under the Apache-2.0 license.

Keywords: BERT, Pretraining, Easy-to-Use, Acceleration, Customized Datasets,
Language Models.

##  1 Introduction

The field of Natural Language Processing (NLP) has made significant progress
since the introduction of large language models, such as BERT (Devlin et al.,
2018), GPT-3 (Brown et al., 2020) and T5 (Raffel et al., 2020). However, most
large language models require a stage of pretraining before being applied to
the target NLP task, where the pretraining cost could be staggering.

Several libraries have been proposed to alleviate the above problem. For
instance, HuggingFace (Wolf et al., 2020) 111Code repository:
https://github.com/huggingface/transformers provides a huge number of
pretrained models, allowing users to download the model directly from the
model repository. It also encourages users to upload their own pretrained
models, so that others can utilize the same model without repeating any
expensive pretraining process. Nevertheless, those features are still
insufficient for researchers or engineers with privacy concerns or
customization needs. For example, in most hospitals or banks, the data has its
own value. Releasing the model trained on those data will not only compromise
user privacy, but can also incur serious security issues. Due to this very
reason, those pretrained models are no longer available online, forcing one to
pretrain the whole model from scratch. However, HuggingFace provides slight
acceleration for pretraining models under the limited-resource scenario. Hence
the time cost still remains a huge obstacle.

On the other hand, several other libraries provide significant speed-up for
pretraining BERT (Devlin et al., 2018). For instance, Academic BERT (Izsak et
al., 2021) summarizes a list of tricks to reduce the pretraining time cost,
making it possible to surpass BERT Base on GLUE tasks (Wang et al., 2018) at
significantly less cost. However, this toolkit restricts itself to the
original BERT settings and lacks sufficient flexibility for customized models
or datasets. Furthermore, its official repository 222Code repository:
https://github.com/IntelLabs/academic-budget-bert misses an easy-to-use
pipeline and requires users to manually build the working environment, modify
its codes and run the programs stage by stage in order to reproduce the
reported results.

To resolve those pain points, we propose ExtremeBERT, a toolkit that obtains
an-order-of-magnitude acceleration in BERT pretraining. In the meantime, it
supports an easy-to-use pipeline and dataset customization.

##  2 Toolkit Overview

ExtremeBERT is a toolkit designed to support fast language model pretraining
under a limited-resource scenario. It can significantly speed up the
pretraining process of BERT through combining several recent accelerating
techniques. Meanwhile, ExtremeBERT supports a wide range of datasets,
including customized datasets provided by users and all Huggingface
pretraining datasets online 333Huggingface dataset repository:
https://huggingface.co/datasets. Those features are seamlessly incorporated in
the toolkit and can be customized simply by specifying a YAML configuration
file, where running the whole pipeline only requires a one-line command.

The whole toolkit is built upon the code repository of Izsak et al. (2021),
where most of its acceleration features have been inherited. On top of that,
ExtremeBERT introduces its own acceleration techniques and further improves
its already superior speed. Several missing features and pain points in Izsak
et al. (2021) are also well handled in ExtremeBERT, leading to a much more
user-friendly interface.

###  2.1 Feature: Easy-to-use Pipeline

Figure 1: Five stages of the ExtremeBERT pipeline and the one-line command.
Figure 2: Left: An example YAML configuration file for BERT Large bert-
large.yaml; Right: An example YAML configuration file for a customized BERT
model bert-customized.yaml.

Many frameworks, including Huggingface (Wolf et al., 2020) and Academic BERT
(Izsak et al., 2021), require users to manually type input commands or write
codes for each stage of the pipeline. However, this is not only painstaking,
it is also inefficient. This pain point is solved in ExtremeBERT by
introducing an one-line command pipeline. It asks users to provide a
configuration file that only includes the necessary information of BERT
pretraining.

In addition, ExtremeBERT solely relies on python packages, freeing users from
complex environment preparation that is common in many other accelerated
frameworks, e.g. the APEX C++ toolkit dependency problem in Izsak et al.
(2021).

###  2.2 Feature: Acceleration

Figure 3: Illustration of the modified ESD scheduler.

ExtremeBERT inherits all acceleration techniques of Academic BERT (Izsak et
al., 2021), including the DeepSpeed Package (Rasley et al., 2020), mixed-
precision training, large batch training, insufficient large model training,
STILT tricks (Phang et al., 2018) and so on.

On top of that, ExtremeBERT adopts a modified Elastic Step Decay (ESD)
learning rate scheduler (Pan et al., 2021) to further accelerate the
pretraining process.

$\displaystyle\eta_{t}=$
$\displaystyle\begin{cases}\eta_{0},\quad&t\in\left[0,1-r^{\ell}\right]\cdot
T,\\\
\eta_{0}\cdot\left(\frac{1}{2r}\right)^{i-\ell}\quad&t\in\left(1-r^{i-1},1-r^{i}\right]\cdot
T,i\geq\ell+1.\end{cases}$ (1)

Experiments show that this technique results in $\sim 0.4$ improvements over
the linear decay scheduler in the final GLUE score.

###  2.3 Feature: Datasets

ExtremeBERT supports two type of datasets: customized datasets provided by
users and datasets supported by Huggingface.

For customized datasets, users only need to convert their corpus into raw text
files, where in each text file, articles are assumed to be separated by blank
lines. Then by grouping those text files in folders and specifying the
folders’ paths in the pipeline configuration file, ExtremeBERT will
automatically handle the data pre-processing on given datasets, i.e. pre-
masking, shuffling, sharding and tokenization.

For Huggingface datasets, as shown in Figure 2, only require the dataset name
and split name to be specified in the pipeline configuration file, e.g.
bookcorpus and plain_text, respectively. ExtremeBERT will then download the
corresponding datasets from the Huggingface dataset repository and handle the
pre-processing.

On top of that, ExtremeBERT supports large dataset processing on small RAM
servers. In the toolkit provided by Izsak et al. (2021), it is observed that
the pre-processing of Wikipedia and bookcorpus datasets costs more than 60GB
memory, rendering it impossible for researchers to use them on small servers
such as Google Colab, which typically supports $<25$ GB memory. To solve this
issue, ExtremeBERT introduces a time-memory tradeoff mechanism to enable
small-RAM pre-processing, which basically utilizes disk serialization to save
RAM usage. By specifying the SYSTEM:MAX_MEMORY_IN_GB field in the pipeline
configuration file, as shown in Figure 2, ExtremeBERT will control the RAM
usage for data pre-processing and make sure no out-of-memory issue occurs.

##  3 Performance

ExtremeBERT provides significant speedup over BERT’s original toolkit (Devlin
et al., 2018). To quantify this acceleration, we compare the cost of reaching
or surpassing the benchmark GLUE score, as shown in Table 1. Here all
empirical experiments are conducted on BERT Large models with different
hyperparameters, the same as the convention in Izsak et al. (2021). For more
experimental details, please refer to Appendix A.

Benchmark | Toolkit |  GPU resource |  GLUE score |  Pretraining Time (days) |  Cost (V100 $\times$ days) | Speedup
---|---|---|---|---|---|---
BERT Base | Original Paper |  TPU chips $\times 16$ | 79.6 | $4.0$ | $>64$ | -
ExtremeBERT |  GeForce RTX 3090 $\times 8$ | 81.1 | $1.3$ | $\sim 10.4$ | $\bm{6.1\times}$
BERT Large | Original Paper |  TPU chips $\times 64$ | 82.1 | $4.0$ | $>256$ | -
ExtremeBERT |  GeForce RTX 3090 $\times 8$ | 82.4 | 3.4 | $\sim 27.2$ | $\bm{9.4\times}$
Table 1: GLUE: results on 9 tasks selected by (Devlin et al., 2018).

Experiment results demonstrate that our toolkit offers roughly one order of
magnitude speedup over the original BERT toolkit. This renders it possible for
researchers and engineers to pretrain a BERT model from scratch with limited
resources.

##  4 Conclusion and Future Work

In this paper, we introduce ExtremeBERT, a resource-friendly framework for
BERT pretraining, which supports customized datasets and offers roughly one
order of magnitude acceleration over the original BERT pretraining process. To
obtain the desired customized pretrained model, users are only a configuration
file and one-line command away, where the remaining part will be handled
smoothly by ExtremeBERT.

In the future, ExtremeBERT will support more features, including more diverse
models, multi-server dataset pre-processing, and SuperGLUE (Wang et al., 2019)
for finetuning tasks. Our vision is to make the pretraining of language models
possible for every researcher and engineer.

##  Appendix A Experimental Details in Section 3

For the BERT Base benchmark, we follow the convention of Izsak et al. (2021),
where a BERT Large model is insufficiently trained to obtain the performance
of BERT Base. We first use $6\%$ portions of iterations to linearly “warm-up”
the learning rate from $0$ to the peak learning rate $\eta_{0}=10^{-3}$. Then
we invoke a modified version of Elastic Step Decay (Pan et al., 2021) for the
rest of the iterations

$\displaystyle\eta_{t}=$
$\displaystyle\begin{cases}\eta_{0},\quad&t\in\left[0,1-r^{\ell}\right]\cdot
T,\\\
\eta_{0}\cdot\left(\frac{1}{2r}\right)^{i-\ell}\quad&t\in\left(1-r^{i-1},1-r^{i}\right]\cdot
T,i\geq\ell+1.\end{cases}$ (2)

The default hyperparameters are set to be $\eta_{0}=2\times 10^{-3}$,
$r=2^{-1/2}$ and $\ell=6$. As for the number of iterations, we run 23k
iterations instead of a 1-day budget, since it is invariant to the machine
settings and more reproducible. Other pretraining/finetuning settings remain
the same as (Izsak et al., 2021).

For the BERT Large benchmark, the only difference is the number of iterations
and the peak learning rate. For the number of iterations, we use 57.5k
iterations instead of 23k iterations, thus the model is more sufficiently
trained. For the peak learning rate, we use a smaller learning rate
$\eta_{0}=10^{-3}$ to make the training process more stable.

In all experiments, we limit the sequences to 128 tokens and use single-
sentence training without next sentence prediction (NSP) to optimize the
pretraining process. We also hold out only 0.5% of the data and compute the
validation loss less frequently to increase the training time.

Moreover, regarding the cost estimation in Table 1, we refer to the statistics
in https://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transformers-bert/ to
make the pretraining costs on different hardwares comparable.

##  Appendix B User Manual

Version 0.1.0
Date 2022-12-01
Title A Tooklit for Accelerating Pretraining of Customized BERT
Author Rui Pan (rpan@connect.ust.hk), Shizhe Diao (sdiaoaa@connect.ust.hk),
Jianlin Chen (jc6g20@soton.ac.uk), Tong Zhang<EMAIL_ADDRESS>
Maintainer Rui Pan (rpan@connect.ust.hk), Shizhe Diao<EMAIL_ADDRESS>
Depends Python ($\geq 3.6.0$), CUDA ($\geq 11.2$), CUDA Driver ($\geq 460.67$)
Description ExtremeBERT is a toolkit that accelerates the pretraining of
customized language models on customized datasets, described in the paper
“ExtremeBERT: A Toolkit for Accelerating Pretraining of Customized BERT”.
License Apache-2.0 license
NeedsCompilation no
Repository https://github.com/extreme-bert/extreme-bert
Date/Publication 2022-12-01 10:00:00
System Requirements

  * •

Linux Ubuntu $\geq$ 18.04

  * •

At least 20 GB GPU memory, e.g. GPU 3090 $\times$ 1, or 2080Ti GPU $\times$ 2,
GeForce RTX 3070 $\times$ 4, etc.

  * •

At least 200GB disk space.

###  B.1 Getting started

####  B.1.1 Installation

⬇

1$ source install.sh

####  B.1.2 Configurable Pipeline

First, one may refer to configs/24h-bert-simple.yaml and make suitable configs
for the pipeline, including datasets, number of gpus available, etc. Then, by
simply running the following command, the whole pipeline will be executed
stage by stage,

⬇

1$ python main.py --config configs/24h-bert-simple.yaml

This command will run environment installation, dataset prepration,
pretraining, finetuning and test result collection one by one and generate the
.zip file for GLUE test server submission under
output_test_translated/finetune/*/*.zip.

To run each stage separately, one may choose to enable or disable each stage
in the configuration file. For example,

Figure 4: Left: An example YAML configuration file data-preprocess.yaml for
data preprocessing only ; Right: An example YAML configuration file train.yaml
for pretraining + finetuning + result collection when the dataset is
preprocessed.

To obtain more detailed configurations for each stage, one may refer to
following sections.

####  B.1.3 Preprocessing

The dataset directory includes scripts to pre-process the datasets we used in
our experiments (Wikipedia, Bookcorpus), which can be used as follows. The
main pipeline script main.py is actually invoking those two scripts with
different arguments during data preprocessing.

⬇

1$ python shard_data.py \

2 --output_dir {tmp_dir} \

3 --num_train_shards 256 \

4 --num_test_shards 128 \

5 --frac_test 0.1 \

6 --max_memory 64 \

7 --dataset wikipedia 20220301.en \

8 --dataset bookcorpusopen plain_text

⬇

1$ python generate_samples.py \

2 --dir {tmp_dir} \

3 -o {processed_dataset_dir} \

4 --dup_factor 10 \

5 --seed 42 \

6 --do_lower_case 1 \

7 --masked_lm_prob 0.15 \

8 --max_seq_length 128 \

9 --tokenizer_name bert-large-uncased \

10 --model_name bert-large-uncased \

11 --max_predictions_per_seq 20 \

12 --n_processes 8

####  B.1.4 Pretraining

During pretraining, the main pipeline script main.py first invokes
pretrain_search.sh, which further calls run_pretraining.sh and finally leads
to run_pretraining.py. For fully configurable settings, please refer to the
following command.

⬇

1$ deepspeed run_pretraining.py \

2 --dataset_path {processed_dataset_dir} \

3 --output_dir {output_dir} \

4 --model_type bert-mlm \

5 --tokenizer_name bert-large-uncased \

6 --hidden_act gelu \

7 --hidden_size 1024 \

8 --num_hidden_layers 24 \

9 --num_attention_heads 16 \

10 --intermediate_size 4096 \

11 --hidden_dropout_prob 0.1 \

12 --attention_probs_dropout_prob 0.1 \

13 --encoder_ln_mode pre-ln \

14 --lr 1e-3 \

15 --train_batch_size 4096 \

16 --train_micro_batch_size_per_gpu 32 \

17 --lr_schedule time \

18 --curve linear \

19 --warmup_proportion 0.06 \

20 --gradient_clipping 0.0 \

21 --optimizer_type adamw \

22 --weight_decay 0.01 \

23 --adam_beta1 0.9 \

24 --adam_beta2 0.98 \

25 --adam_eps 1e-6 \

26 --total_training_time 24.0 \

27 --early_exit_time_marker 24.0 \

28 --print_steps 100 \

29 --num_epochs_between_checkpoints 10000 \

30 --job_name pretraining_experiment \

31 --project_name budget-bert-pretraining \

32 --validation_epochs 3 \

33 --validation_epochs_begin 1 \

34 --validation_epochs_end 1 \

35 --validation_begin_proportion 0.05 \

36 --validation_end_proportion 0.01 \

37 --validation_micro_batch 16 \

38 --deepspeed \

39 --data_loader_type dist \

40 --do_validation \

41 --use_early_stopping \

42 --early_stop_time 180 \

43 --early_stop_eval_loss 6 \

44 --seed 42 \

45 --fp16

After pretraining, the pretrained model will be saved to {output_dir}, which
by default is saved_models/pretrain/{dataset_name} in our pipeline, where
{dataset_name} is automatically generated based on dataset content and will be
displayed in the pretraining log. Users can also manually specify
{dataset_name} in the pipeline configuration file, which corresponds to
DATASET.ID.

In addition, the intermediate logs will have a backup in
log/pretrain/{dataset_name}/ for both standard output stream and error stream.

####  B.1.5 Finetuning

During finetuning, the main pipeline script main.py first invokes
finetune_search.sh, which further calls run_glue.sh and finally leads to
run_glue.py. For fully configurable settings, please refer to the following
command. Please notice that finetune_search.sh will automatically handle the
standard hyperparameter search process and the STILT trick (Phang et al.,
2018).

⬇

1$ python run_glue.py \

2 --model_name_or_path {path_to_model_checkpoint} \

3 --task_name MRPC \

4 --max_seq_length 128 \

5 --output_dir /tmp/finetuning \

6 --overwrite_output_dir \

7 --do_train --do_eval \

8 --evaluation_strategy steps \

9 --per_device_train_batch_size 32 \

10 --gradient_accumulation_steps 1 \

11 --per_device_eval_batch_size 32 \

12 --learning_rate 5e-5 \

13 --weight_decay 0.01 \

14 --eval_steps 50 \

15 --evaluation_strategy steps \

16 --max_grad_norm 1.0 \

17 --num_train_epochs 5 \

18 --lr_scheduler_type polynomial \

19 --warmup_steps 50

The finetune result of each hyperparameter will be displayed in the log on
screen. In our pipeline, this log will be redirected and saved in
log/finetune/{dataset_name}/{task_name}/*/. The test set prediction will also
be generated during finetuning, which will be saved in
output/finetune/{dataset_name}/{task_name}/*/. Those files will be parsed
later in result collection stage to generate the final validation result and
submission zip for https://gluebenchmark.com/.

###  B.2 Functions

BasePretrainModel Base pretraining model definition

##### Description

The base class definition of pretraining model.

Usage

⬇

1model = BasePretrainModel(args)

##### Arguments

args: configurations

##### Value

an object of pretraining model.

get_default_config Get default configuration

##### Description

Get default configuration for pipeline training.

Usage

⬇

1config = get_default_config()

##### Arguments

N/A

##### Value

A dictionary tree is returned, which contains the following components:

  * •

config.SYSTEM: system configurations

  * •

config.WANDB: wandb configurations

  * •

config.DATASET: dataset configurations

  * •

config.PRETRAIN: pretrain configurations

  * •

config.FINETUNE: finetune configurations

  * •

config.TOKENIZER: tokenizer configurations

prepare_dataset Prepare dataset

##### Description

Prepare dataset for pretraining.

Usage

⬇

1prepare_dataset(config)

##### Arguments

config: configurations

##### Value

N/A

pretrain Pretraining entry point

##### Description

Pretraining function.

Usage

⬇

1pretrain(config)

##### Arguments

config: configurations

##### Value

N/A

train Pretraining function

##### Description

Pretraining function including iterations over datasets.

Usage

⬇

1eval_loss, scale_counter = train(

2 args,

3 index,

4 model,

5 optimizer,

6 lr_scheduler,

7 pretrain_dataset_provider,

8 validation_dataset,

9)

##### Arguments

  * •

args: arguments

  * •

index: index of epoch

  * •

model: training model

  * •

optimizer: optimizer

  * •

lr_scheduler: learning rate scheduler

  * •

pretrain_dataset_provider: the pretraining dataset

  * •

validation_dataset: validation dataset

##### Value

  * •

eval_loss: evaluation loss

  * •

scale_counter_at_1: if stuck at scale 1, add to scale counter

finetune Finetuning entry point

##### Description

Finetuning on downstream tasks.

Usage

⬇

1finetune(config)

##### Arguments

config: configurations

##### Value

N/A

collect_test_result Collect test results

##### Description

Collect test results.

Usage

⬇

1collect_test_result(config)

##### Arguments

config: configurations

##### Value

N/A

##### Details

It will run ‘summarize_val.sh’ to summarize the validation results for each
downstream tasks, ‘collect_best_val.sh’ to collect best validation results,
‘translate_test_result.sh’ to translate the prediction to the desired format
according to GLUE benchmark website.

⬇

1dataset_name = config.DATASET.ID

2run_bash(f’./summarize_val.sh {dataset_name}’)

3run_bash(f’./collect_best_val.sh {dataset_name}’)

4run_bash(f’./translate_test_result.sh {dataset_name}’)

## References

  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Izsak et al. (2021) Peter Izsak, Moshe Berchansky, and Omer Levy.  How to train bert with an academic budget, 2021.  URL https://arxiv.org/abs/2104.07705.
  * Pan et al. (2021) Rui Pan, Haishan Ye, and Tong Zhang.  Eigencurve: Optimal learning rate schedule for SGD on quadratic objectives with skewed hessian spectrums.  _CoRR_ , abs/2110.14109, 2021.  URL https://arxiv.org/abs/2110.14109.
  * Phang et al. (2018) Jason Phang, Thibault Févry, and Samuel R Bowman.  Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks.  _arXiv preprint arXiv:1811.01088_ , 2018.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505–3506, 2020.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_ , 2018.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _Advances in neural information processing systems_ , 32, 2019.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, October 2020. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.emnlp-demos.6.


# Spin-orbit coupling effects on the stability of two competing structures in
Pb/Si(111) and Pb/Ge(111)

Xiao-Yan Ren1,2, Hyun-Jung Kim3, Seho Yi2, Yu Jia1,4∗, and Jun-Hyung Cho2,1∗ 1
International Laboratory for Quantum Functional Materials of Henan, and School
of Physics and Engineering, Zhengzhou University, Zhengzhou 450001, China
2 Department of Physics and Research Institute for Natural Sciences, Hanyang
University, 17 Haengdang-Dong, Seongdong-Ku, Seoul 133-791, Korea
3 Korea Institute for Advanced Study, 85 Hoegiro, Dongdaemun-gu, Seoul
130-722, Korea
4 Key Laboratory for Special Functional Materials of Ministry of Education,
School of Physics and Electronics, Henan University, Kaifeng 475004, China

###### Abstract

Using first-principles density-functional theory (DFT) calculations, we
investigate the 4/3-monolayer structure of Pb on the Si(111) or Ge(111)
surface within the two competing structural models termed the H3 and T4
structures. We find that the spin-orbit coupling (SOC) influences the relative
stability of the two structures in both the Pb/Si(111) and Pb/Ge(111) systems:
i.e., our DFT calculation without including the SOC predicts that the T4
structure is energetically favored over the H3 structure by ${\Delta}E$ = 25
meV for Pb/Si(111) and 22 meV for Pb/Ge(111), but the inclusion of SOC
reverses their relative stability as ${\Delta}E$ = $-$12 and $-$7 meV,
respectively. Our analysis shows that the SOC-induced switching of the ground
state is attributed to a more asymmetric surface charge distribution in the H3
structure, which gives rise to a relatively larger Rashba spin splitting of
surface states as well as a relatively larger pseudo-gap opening compared to
the T4 structure. By the nudged elastic band calculation, we obtain a sizable
energy barrier from the H3 to the T4 structure as ${\sim}$0.59 and
${\sim}$0.27 eV for Pb/Si(111) and Pb/Ge(111), respectively. It is thus likely
that the two energetically competing structures can coexist at low
temperatures.

###### pacs:

71.70.Ej, 73.20.At, 73.20.-r

##  I INTRODUCTION

Two-dimensional (2D) electronic systems have attracted intensive attention due
to their many exotic phenomena and potential applications in nanoelectronic
devices. brun ; zhang ; kurosaki ; car ; coldea ; wei ; meng ; shimizu
Recently, metal-atom adsorption on semiconductor surfaces has been employed to
generate spin-polarized current on the basis of the Rashba spin splitting,
where the spin-orbit coupling (SOC) lifts the spin degeneracy due to the
broken inversion symmetry. kim ; matet ; barke ; gierz ; frant ; stolw ; gluz
; stol  We here focus on the 4/3 monolayer (ML) adsorption of Pb atoms on the
Si(111) or Ge(111) surface, forming a dense phase with the
${\sqrt{3}}{\times}{\sqrt{3}}R30^{\circ}$ unit cell [see Figs. 1(a) and 1(b)].
It was recently reported yaji  that this phase displays a large spin splitting
of metallic surface bands on the semiconducting Ge(111) substrate, thereby
promising for surface spin transport/accumulation because surface spin signals
can be preserved over long time within the substrate band gap.

Earlier reflection high energy electron diffraction (RHEED), x-ray, and low
energy electron diffraction (LEED) measurements for Pb/Ge(111) suggested that
the 4/3 ML coverage has four Pb atoms per unit cell, three of which is located
at off-centered T1 sites and the fourth Pb atom at an H3 site. dev ; feid ;
huang  This structural model is termed the H3 structure [see Fig. 1(a)], which
preserves the $C_{3v}$ symmetry. Recently, angle-resolved photoelectron
spectroscopy (ARPES) and spin-resolved ARPES for Pb/Ge(111) demonstrated that
a metallic surface-state band with a dominant Pb 6$p$ character exhibits a
large Rashba spin splitting of 200 meV. yaji  This observed spin splitting of
surface state agreed well with the surface band structure of the H3 structure
(equivalently the close-packed structure), obtained using a density-functional
theory (DFT) calculation with including the SOC yaji ; yaji2 . However,
earlier DFT calculations without SOC for Pb/Si(111) chan ; hupa  and
Pb/Ge(111) anci  predicted that the T4 structure [see Fig. 1(b)] or the so-
called chain structure with the broken $C_{3v}$ symmetry is slightly more
stable than the H3 structure by less than a few tens of meV. This conflict of
the ground state between experiments dev ; feid ; huang  and previous DFT
calculations chan ; hupa ; anci  may imply that the SOC influences the
relative stability of the H3 and T4 structures in the Pb/Si(111) and
Pb/Ge(111) surface systems. Therefore, it is very challenging to examine the
energetics of the two structures by using the DFT calculations with/without
including the SOC.

Figure 1: (Color online) Top and side views of the optimized (a) H3 and (b) T4
structures of Pb/Si(111). The green and gray circles represent Pb and Si
atoms, respectively. For distinction, the Si atoms in the subsurface layers
are drawn with small circles. The dashed line indicates the
${\sqrt{3}}{\times}{\sqrt{3}}R30^{\circ}$ unit cell, where the four Pb atoms
are numbered. The Pb4 atom in (a) and (b) is located at the H3 and T4 sites,
respectively. The x, y, and z axes point along the [11${\overline{2}}$],
[1${\overline{1}}$0], and [111] directions, respectively. The height
difference between the H3 (T4) Pb atom and the first Si-substrate layer is
labeled as ${\Delta}h$.

In the present study, we optimize the geometries of Pb/Si(111) and Pb/Ge(111)
within the H3 and T4 structural models. Our DFT calculation without SOC
demonstrates that the T4 structure is energetically favored over the H3
structure by ${\Delta}E$ = 25 meV for Pb/Si(111) and 22 meV for Pb/Ge(111),
consistent with previous DFT calculations. chan ; hupa ; anci  However, the
inclusion of SOC reverses their relative stability as ${\Delta}E$ = $-$12 and
$-$7 meV for Pb/Si(111) and Pb/Ge(111), respectively. We find that the H3
structure has a more asymmetric surface charge distribution compared to the T4
structure, which in turn results in a relatively larger Rashba spin splitting
as well as a relatively larger pseudo-gap opening along the $\overline{KM}$
symmetry line. These results provide an explanation for the SOC-driven
switching of the ground state. Moreover, our nudged elastic band calculation
obtains an energy barrier for the transition on going from the H3 to the T4
structure as large as ${\sim}$0.59 and ${\sim}$0.27 eV for Pb/Si(111) and
Pb/Ge(111), respectively. This suggests that the two energetically competing
structures can coexist at low temperatures.

##  II CALCULATIONAL METHODS

The present DFT calculations were performed using the Vienna ab initio
simulation package with the projector-augmented wave method and a plane wave
basis set. vasp1 ; vasp2  For the treatment of exchange-correlation energy, we
employed the GGA functional of Perdew-Burke-Ernzerhof. pbe  The Pb/Si(111) and
Pb/Ge(111) surfaces were modeled by a periodic slab geometry consisting of the
twelve Si and Ge atomic layers with ${\sim}$25 Å of vacuum in between the
slabs. The bottom of the Si or Ge substrate was passivated by one H atom. We
employed a dipole correction that cancels the artificial electric field across
the slab. neugebauer  The kinetic energy cutoff of the plane-wave basis set
was taken to be 400 eV, and the ${\bf k}$-space integration was done with the
15${\times}$15 Monkhorst-Pack meshes in the surface Brillouin zones (SBZ) of
the ${\sqrt{3}}{\times}{\sqrt{3}}R30^{\circ}$ unit cell. All atoms except the
bottom two substrate layers were allowed to relax along the calculated forces
until all the residual force components were less than 0.01 eV/Å.

##  III RESULTS

We begin to optimize the H3 and T4 structures of the Pb/Si(111) and Pb/Ge(111)
surfaces using the DFT calculation in the absence of SOC. The optimized H3 and
T4 structures of Pb/Si(111) are displayed in Fig. 1(a) and 1(b), respectively.
We find that the T4 structure is energetically more stable than the H3
structure by ${\Delta}E$ = 25 and 22 meV per
${\sqrt{3}}{\times}{\sqrt{3}}R30^{\circ}$ unit cell for Pb/Si(111) and
Pb/Ge(111), respectively. This preference of the T4 structure over the H3
structure is consistent with previous DFT calculations chan ; hupa ; anci .
The calculated bond lengths $d_{\rm Pb-Pb}$ and $d_{\rm Pb-Si}$ (or $d_{\rm
Pb-Ge}$) at the interface are given in Table I. It is noticeable that the H3
structure of Pb/Si(111) [Pb/Ge(111)] has an equal value of $d_{\rm Pb-Pb}$ =
3.137 (3.224) Å between three nearest Pb$-$Pb atoms, thereby preserving the
$C_{3v}$ symmetry. Meanwhile, in the T4 structure of Pb/Si(111) [Pb/Ge(111)],
$d_{\rm Pb-Pb}$ = 3.183 (3.265) Å between two nearest Pb$-$Pb atoms is
slightly shorter than the third one by ${\sim}$0.01 (0.02) Å, indicating the
broken $C_{3v}$ symmetry. This rotational symmetry breaking in the T4
structure is also reflected by the position of the T4 Pb adatom which is
located at the off-centered T4 site with a lateral displacement of
${\sim}$0.05 Å along the x direction [see Fig. 1(b)].

Table 1: Bond lengths (in Å) $d_{\rm Pb-Pb}$ and $d_{\rm Pb-Si}$ (or $d_{\rm Pb-Ge}$) as well as ${\Delta}h$ (see Fig. 1) in the H3 and T4 structures, obtained using DFT and DFT+SOC. Two different values of $d_{\rm Pb-Pb}$ and $d^{{}^{\prime}}_{\rm Pb-Pb}$ are given in the T4 structure. The results for Pb/Ge(111) are given in parentheses. |  | DFT | DFT+SOC
---|---|---|---
H3 | $d_{\rm Pb-Pb}$ | 3.137 (3.224) | 3.101 (3.188)
| d${}_{{\rm Pb_{1}}-{\rm Si}}$ | 2.888 (2.893) | 2.882 (2.897)
| d${}_{{\rm Pb_{2}}-{\rm Si}}$ | 2.888 (2.893) | 2.882 (2.897)
| d${}_{{\rm Pb_{3}}-{\rm Si}}$ | 2.888 (2.893) | 2.882 (2.897)
| ${\Delta}h$ | 2.596 (2.737) | 2.649 (2.761)
T4 | $d_{\rm Pb-Pb}$ | 3.183 (3.265) | 3.133 (3.218)
| $d^{{}^{\prime}}_{\rm Pb-Pb}$ | 3.193 (3.281) | 3.142 (3.237)
| d${}_{{\rm Pb_{1}}-{\rm Si}}$ | 2.877 (2.893) | 2.867 (2.887)
| d${}_{{\rm Pb_{2}}-{\rm Si}}$ | 2.882 (2.914) | 2.871 (2.901)
| d${}_{{\rm Pb_{3}}-{\rm Si}}$ | 2.882 (2.914) | 2.871 (2.901)
| ${\Delta}h$ | 2.534 (2.537) | 2.689 (2.672)

Figures 2(a) and 2(b) show the band structures of the H3 and T4 structural
models of Pb/Si(111), respectively, obtained using the DFT calculation without
SOC: For Pb/Ge(111), see Figs. 1S(a) and 1S(b) of the Supplemental Material.
supp  The band projection onto the Pb $p_{x}$, $p_{y}$, and $p_{z}$ orbitals
is also displayed in Figs. 2(a) and 2(b). It is seen that the band structures
of the H3 and T4 structures are very similar to each other. Especially, the
band dispersion of the surface states crossing the Fermi level $E_{F}$ is
nearly parabolic along the $\overline{{\Gamma}M}$ and $\overline{{\Gamma}K}$
lines, indicating 2D electronic states in the Pb overlayer. As shown in Figs.
2(a) and 2(b), the orbital character of the surface states exhibits a strong
${\bf k}$-dependence along the symmetry lines. Consequently, the SOC may
induce an efficient hybridization of the Pb $p_{x}$, $p_{y}$, and $p_{z}$
orbitals, as discussed below.

Figure 2: (Color online) Band structures of the (a) H3 and (b) T4 structural
models, obtained using DFT. The corresponding ones obtained using DFT+SOC are
given in (c) and (d), respectively. The bands projected onto the Pb $p_{x}$,
$p_{y}$, and $p_{z}$ orbitals are displayed with circles whose radii are
proportional to the weights of such orbitals. The energy zero represents
$E_{F}$. The inset in (a) shows the surface Brillouin zones of the
${\sqrt{3}}{\times}{\sqrt{3}}R30^{\circ}$ unit cell. In (c) and (d), the
contour plots of the charge density of the lowest unoccupied surface state at
the $K$ point are drawn in the vertical plane containing the H3 and T4 Pb
atoms, respectively, where the first line is at 0.3${\times}$10-3 electrons/Å3
with spacings of 0.5${\times}$10-3 electrons/Å3. A close up of the spin
splitting of the surface state at the $K$ point (just above $E_{F}$) and the
$M$ point (just below $E_{F}$) is displayed for the H3 and T4 structures in
(c) and (d), respectively, together with including their time-reversal
counterparts.

Next, we examine the SOC effects on the stability of the H3 and T4 structures
using the DFT+SOC calculation. Interestingly, the inclusion of SOC changes the
relative stability of the two structures: i.e., the H3 structure becomes more
stable than the T4 structure by 12 and 7 meV for Pb/Si(111) and Pb/Ge(111),
respectively. As shown in Table I, for the H3 (T4) structure of Pb/Si(111),
the SOC decreases $d_{\rm Pb-Pb}$ and $d_{\rm Pb-Si}$ by ${\sim}$0.03 (0.05)
and ${\sim}$0.01 (0.01) Å, respectively, but increases ${\Delta}h$ by
${\sim}$0.05 (0.16) Å. On the other hand, for the H3 (T4) structure of
Pb/Ge(111), the SOC produces decreases of ${\sim}$0.04 (0.05) and ${\sim}$0.01
(0.01) Å for $d_{\rm Pb-Pb}$ and $d_{\rm Pb-Si}$, respectively, and an
increase of ${\sim}$0.03 (0.14) Å for ${\Delta}h$. To understand the SOC-
induced switching of the ground state, we plot the band structures of the H3
and T4 models of Pb/Si(111) in Fig. 2(c) and 2(d), respectively: For
Pb/Ge(111), see Figs. 1S(c) and 1S(d) of the Supplemental Material. supp  The
spin degeneracy of surface states as well as other states is found to be
lifted over the SBZ except at the high-symmetry points (i.e., ${\Gamma}$ and
$M$ points). Specifically, it is seen that the H3 structure has a spin
splitting of 300 meV for the surface states at the $K$ point just below
$E_{F}$, larger than the corresponding one (136 meV) of the T4 structure. The
insets of Figs. 2(c) and 2(d) show a close up of the spin splitting of the
surface state at the $K$ point along the $\overline{K{\Gamma}}$ direction
(just above $E_{F}$) and the $M$ point along the $\overline{MK}$ direction
(just below $E_{F}$), obtained for the H3 and T4 structures, respectively.
Obviously, these spin-split subbands illustrate the typical dispersion of the
Rashba-type spin splitting. We fit the $k$-dependent dispersion of such spin-
split subbands with the characteristic parameters such as the momentum offset
${\Delta}k_{R}$ and the Rashba energy $E_{R}$ [see Fig. 2(c)] by using the
Rashba spin-splitting eigenvalues ${\epsilon}_{\pm}$ =
$\frac{{{\hbar}^{2}}k^{2}}{2m^{*}}{\pm}{{\alpha}_{R}}k$, where $m^{*}$ is the
electron effective mass and ${\alpha}_{R}$ the Rashba parameter. We find that
the H3 structure has ${\alpha}_{R}$ = 2.775 (0.495) eV Å along
$\overline{K{\Gamma}}$ ($\overline{MK}$), larger than 0.523 (0.093) eV Å for
the T4 structure. It is also noticeable that the H3 structure exhibits a
relatively larger pseudo-gap opening of 385 meV along the $\overline{KM}$ line
compared to that (275 meV) of the T4 structure: see Figs. 2(c) and 2(d). Thus,
these relatively larger spin splitting and pseudo-gap opening of the H3
structure compared to the T4 structure can lead to the SOC-driven switching of
the ground state.

From the DFT+SOC calculation, the band projection of surface states shows a
strong hybridization between the Pb $p_{x}$, $p_{y}$, and $p_{z}$ orbitals
[see Fig. 2(c) and 2(d)]. This SOC-induced hybridization is known to give not
only a gap opening sunwoo  but also an asymmetric surface charge distribution
kim ; nagan  which in turn determines the size of the Rashba spin splitting
through the integral of the charge density times the potential gradient along
the direction perpendicular to the surface, ${\alpha}_{R}$ ${\propto}$
${\int}dV/dz\,{\rho}({\bf{r}})d{\bf{r}}$. As shown in the inset of Figs. 2(c)
and 2(d), the H3 structure has a more asymmetric charge character of the
surface state compared to the T4 structure, thereby giving rise to the above-
mentioned relatively larger values of ${\alpha}_{R}$. Indeed, a recent DFT
calculation for the H3 structure of Pb/Ge(111) reported that the observed
large Rashba spin splitting is due to an asymmetric charge distribution in the
vicinity of the H3 Pb atom. yaji2  It is noteworthy that the SOC changes more
dominantly the position of the T4 Pb atom, which moves outward from the
surface by ${\sim}$0.16 and ${\sim}$0.14 Å (see Table I) for Pb/Si(111) and
Pb/Ge(111), respectively. This outward movement possibly contributes to the
rather symmetric charge character of the surface state in the vicinity of the
T4 Pb atom, as shown in the inset of Fig. 2(d).

Figures 3(a) and 3(b) show the helical spin textures of the H3 and T4
structures of Pb/Si(111) along the Fermi surface, respectively. It is seen
that the spin angular momentum (SAM) direction rotates anti-clockwise
(clockwise) along the outer (inner) Fermi surface. The total spin polarization
$|{\bf\it S}|$ = $\sqrt{{S_{x}}^{2}+{S_{y}}^{2}+{S_{z}}^{2}}$ and the
longitudinal spin component $S_{z}$ along the outer and inner Fermi surfaces
are given in Fig. 3(c) and 3(d), respectively. We find that the value of
$|{\bf\it S}|$ in the H3 structure is slightly larger than that in the T4
structure. It is also found that the magnitude of the transverse spin
component $S_{x}$ or $S_{y}$ is larger in the H3 structure compared to the T4
structure, while that of $S_{z}$ is reversed between the two structures. These
different features between the two structures may be caused by a relatively
larger in-plane potential gradient in the T4 structure due to its broken
$C_{3v}$ symmetry. As shown in Figs. 3(c) and 3(d), the spin vectors are
highly polarized along the $z$ direction, reflecting that the Pb overlayer on
the Si(111) substrate has a large in-plane component of potential gradients.
It is noted that the H3 structure has the $C_{3v}$ symmetry while the T4
structure has one mirror-plane ${\sigma}_{v}$ symmetry with the $xz$ plane
[see Fig. 1(b)]. Consequently, the whole spin textures along the Fermi surface
satisfy the symmetries involved in the H3 and T4 structures, respectively [see
Figs. 3(a) and 3(b)]. Since both the H3 and T4 structures have a mirror
symmetry of the $xz$ plane, the $S_{x}$ and $S_{z}$ components change their
sign but the $S_{y}$ component remains unchanged when the spin vectors within
the irreducible part of SBZ are reflected through the mirror plane along the
Fermi surface. However, the spin vectors at the points crossing the
$\overline{{\Gamma}K}$ line are oriented perpendicular to the mirror plane.
Note that this mirror-plane symmetry is a combination of the proper rotation
of 180∘ (about the $y$ axis) with the inversion. Meanwhile, the whole spin
textures shown in Figs. 3(a) and 3(b) also satisfy the time-reversal symmetry
that reverses simultaneously the wavevector and spin.

Figure 3: (Color online) Helical spin textures of the (a) H3 and (b) T4
structures of Pb/Si(111) along the Fermi surface. The SAM vectors with the
negative, zero, and positive $S_{z}$ components are represented with
increasing brightness of the arrows. The total spin polarization $|{\bf\it
S}|$ = $\sqrt{{S_{x}}^{2}+{S_{y}}^{2}+{S_{z}}^{2}}$ and the $S_{z}$ component
along the outer and inner Fermi surfaces are given in (c) and (d),
respectively.

To find the energy barrier for the phase transition between the H3 and T4
structures, we calculate the energy profile along the transition path by using
the nudged elastic-band method neb . The calculated energy profile for the
transition state ($TS$) is displayed in Fig. 4. We find that $TS$ is higher in
energy than the H3 structure, yielding an energy barrier of ${\sim}$0.59
(0.27) eV on going from the H3 to the T4 structure in Pb/Si(111) [Pb/Ge(111)].
The presence of a sizable energy barrier between the H3 and T4 structures
suggests that the two energetically competing structures can coexist in
Pb/Si(111) or Pb/Ge(111) at low temperatures. However, as temperature
increases, thermal fluctuation between the two structures is plausible. Based
on an Arrhenius-type activation process with the usual attempt frequency of
${\sim}$1013 Hz, we estimate the thermal fluctuation temperature as
${\sim}$270 and ${\sim}$125 K for Pb/Si(111) and Pb/Ge(111), respectively.
This peculiar feature of the two competing structures separated by a sizable
energy barrier is likely to cause not only the existence of complicated mixed
phases at low temperatures but also the order-disorder transition at high
temperatures, as observed in the Pb/Si(111) system. hori ; step

Figure 4: (Color online) Calculated energy profile along the transition
pathway from the H3 to the T4 structure in Pb/Si(111). The atomic geometry of
the transition state (TS) is given. The numbers denote the total energies of
TS and the T4 structure relative to the H3 structure. The corresponding energy
values for Pb/Ge(111) are given in parentheses.

##  IV SUMMARY

We have investigated the 4/3-ML Pb overlayer structures on the Si(111) and
Ge(111) surfaces within the H3 and T4 models using the DFT calculations
with/without including the SOC. For both Pb/Si(111) and Pb/Ge(111) systems,
the DFT calculation without SOC showed that the T4 structure is energetically
favored over the H3 structure, but the DFT+SOC calculation reverses their
relative stability. This SOC-driven switching of the ground state is accounted
for in terms of a more asymmetric surface charge distribution in the H3
structure compared to the T4 structure, which in turn gives rise to a
relatively larger Rashba spin splitting as well as a relatively larger pseudo-
gap opening along the $\overline{KM}$ line. In addition, our nudged elastic
band calculations for Pb/Si(111) and Pb/Ge(111) showed the presence of a
sizable energy barrier between the H3 and T4 structures, so that the two
energetically competing structures can coexist at low temperatures. Further
experiments are needed for identifying such a close proximity of
thermodynamically phase-separated ground states.

ACKNOWLEDGEMENTS

This work was supported by National Research Foundation of Korea (NRF) grant
funded by the Korea Government (MSIP) (NRF-2014M2B2A9032247) and by the
National Basic Research Program of China (No.2012CB921300), the National
Natural Science Foundation of China (No.11274280), and Innovation Scientists
and Technicians Troop Construction Projects of Henan Province. The
calculations were performed by KISTI supercomputing center through the
strategic support program (KSC-2015-C3-044) for the supercomputing application
research.

∗ Corresponding authors<EMAIL_ADDRESS>and<EMAIL_ADDRESS>
## References

  * (1) C. Brun, T. Cren, V. Cherkez, F. Debontridder, S. Pons, D. Fokin, M. C. Tringides, S. Bozhko, L. B. Ioffe, B. L. Altshuler and D. Roditchev, Nat. Phys. 10,444 (2014).
  * (2) T. Zhang, P. Cheng, W.-J. Li, Y.-J. Sun, G. Wang, X.-G. Zhu, K. He, L. Wang, X. Ma, X. Chen, Y. Wang, Y. Liu, H.-Q. Lin, J.-F. Jia and Q.-K. Xue, Nat. Phys. 6, 104 (2010)
  * (3) Y. Kurosaki, Y. Shimizu, K. Miyagawa, K. Kanoda, and G. Saito, Phys. Rev. Lett. 95, 177001 (2005).
  * (4) J. M. Carpinelli, H. H. Weitering, M. Bartkowiak, R. Stumpf, and E. W. Plummer, Phys. Rev. Lett. 79, 2859 (1997).
  * (5) R. Coldea, D. A. Tennant, A. M. Tsvelik, and Z. Tylczynski, Phys. Rev. Lett. 86, 1335 (2001).
  * (6) H. H. Weitering, J. M. Carpinelli, A. V. Melechko, J. Zhang, M. Bartkowiak, and E. W. Plummer, Science 285, 2107 (1999).
  * (7) Z. Y. Meng, T. C. Lang, S. Wessel, F. F. Assaad, and A. Muramatsu, Nature 464, 847 (2010).
  * (8) Y. Shimizu, K. Miyagawa, K. Kanoda, M. Maesato, and G. Saito, Phys. Rev. Lett. 91, 107001 (2003).
  * (9) A. V. Matetskiy, S. Ichinokura, L. V. Bondarenko, A. Y. Tupchaya, D. V. Gruznev, A. V. Zotov, A. A. Saranin, R. Hobara, A. Takayama and S. Hasegawa, Phys. Rev. Lett. 115, 147003 (2015).
  * (10) I. Barke, F. Zheng, T. K. Rügheimer, and F. J. Himpsel, Phys. Rev. Lett. 97, 226405 (2006).
  * (11) I. Gierz, T. Suzuki, E. Frantzeskakis, S. Pons, S. Ostanin, A. Ernst, J. Henk, M. Grioni, K. Kern and C. R. Ast, Phys. Rev. Lett. 103, 046803 (2009).
  * (12) E. Frantzeskakis, S. Pons, and M. Grioni, Phys. Rev. B 82, 085440 (2010).
  * (13) S. D. Stolwijk, A. B. Schmidt, M. Donath, K. Sakamoto, and P. Krüger, Phys. Rev. Lett. 111, 176402 (2013).
  * (14) D. V. Gruznev, L. V. Bondarenko, A. V. Matetskiy, A. Y. Tupchaya, A. A. Alekseev, C. R. Hsing, C.M. Wei, S. V. Eremeev, A. V.Zotov, and A. A. Saranin, Phys. Rev. B 91, 035421 (2015).
  * (15) S. D. Stolwijk, K. Sakamoto, A. B. Schmidt, P. Krüger, and M. Donath, Phys. Rev. B 90, 161109 (2014).
  * (16) H.-J. Kim and J.-H. Cho, Phys. Rev. B 92, 085303 (2015).
  * (17) K. Yaji, Y. Ohtsubo, S. Hatta, H. Okuyama, K. Miyamoto, T. Okuda, A. Kimura, H. Namatame, M. Taniguchi and T. Aruga, Nat Commun, 1, 17 (2010).
  * (18) B. N. Dev, F. Grey, R. L. Johnson, and G. Materlik, Europhys. Lett. 6, 311 (1988).
  * (19) R. FeidenhansÌ, J.S. Pedersen, M. Nielsen, F. Grey, and R.L. Johnson, Surf. Sci. 178, 927 (1986).
  * (20) H. Huang, C. M. Wei, H. Li, B. P. Tonner and S. Y. Tong, Phys. Rev. Lett. 62, 559 (1989).
  * (21) K. Yaji, S. Hatta, T. Aruga and H. Okuyama, Phys. Rev. B 86, 235317 (2012).
  * (22) T.-L. Chan, C. Z. Wang, M. Hupalo, M. C. Tringides, Z.-Y. Lu and K. M. Ho, Phys. Rev. B 68 045410 (2003).
  * (23) M. Hupalo, T. L. Chan, C. Z. Wang, K. M. Ho and M. C. Tringides, Phys. Rev. B 66, 161410 (2002).
  * (24) F. Ancilotto, A. Selloni, and R. Car, Phys. Rev. Lett. 71, 3685 (1993).
  * (25) G. Kresse and J. Hafner, Phys. Rev. B 48, 13115 (1993).
  * (26) G. Kresse and J. Furthmüller, Comput. Mater. Sci. 6, 15 (1996).
  * (27) J. P. Perdew, K. Burke, and M. Ernzerhof. Phys. Rev. Lett. 77, 3865 (1996); 78, 1396 (1997).
  * (28) J. Neugebauer and M. Scheffler, Phys. Rev. B 46, 16067 (1992).
  * (29) See Supplemental Material at http://link.aps.org/supplemental/xxxx for the band structures of the Pb/Ge(111)surface, obtained using DFT and DFT+SOC.
  * (30) S.-W Kim, C. Liu, H.-J Kim, J.-H. Lee, Y. Yao, K.-M. Ho, and J.-H. Cho, Phys. Rev. Lett. 115, 096401 (2015).
  * (31) M. Nagano, A. Kodama, T. Shishidou, and T. Oguchi, J. Phys. : Condens. Matter 21, 064239 (2009).
  * (32) G. Henkelman, B. P. Uberuaga, and H. Jónsson. J. Chem. Phys. 113, 9901 (2000).
  * (33) K. Horikoshi, X. Tong, T. Nagao and S. Hasegawa. Phys. Rev. B 60, 13287 (1999).
  * (34) S. Stepanovsky, M. Yakes, V. Yeh, M. Hupalo and M. C. Tringides. Surf. Sci. 600, 1417 (2006).