# FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding

Bill Tuck Weng Pung*, Alvin Chan
School of Computer Science and Engineering, Nanyang Technological University
<EMAIL_ADDRESS>

###### Abstract

Inducing latent tree structures from sequential data is an emerging trend in
the NLP research landscape today, largely popularized by recent methods such
as Gumbel LSTM and Ordered Neurons (ON-LSTM). This paper proposes FastTrees, a
new general-purpose neural module for fast sequence encoding. Unlike most
previous works that consider recurrence to be necessary for tree induction,
our work explores the notion of parallel tree-induction, i.e., imbuing our
model with hierarchical inductive biases in a parallelizable, non-
autoregressive fashion. To this end, our proposed FastTrees achieves
competitive or superior performance to ON-LSTM on four well-established
sequence modeling tasks, i.e., language modeling, logical inference, sentiment
analysis and natural language inference. Moreover, we show that the FastTrees
module can be applied to enhance Transformer models, achieving performance
gains on three sequence transduction tasks (machine translation, subject-verb
agreement and mathematical language understanding), paving the way for modular
tree-induction modules. Overall, we outperform existing state-of-the-art
models on logical inference tasks by $+4\%$ and mathematical language
understanding by $+8\%$. Code is available on
Github111https://github.com/billptw/FastTrees.

##  1 Introduction

Inducing and imposing hierarchical tree-like inductive biases in sequential
models has garnered increasingly significant attention Shen et al. (2020,
2018, 2017); Yogatama et al. (2016); Jacob et al. (2018); Havrylov et al.
(2019), largely owing to the promise of automatically capturing intrinsic
syntactic and linguistic structures prevalent in many forms of sequential data
(e.g., language, mathematics and music). This is reflected in recent works,
i.e., Ordered Neurons (ON-LSTM Shen et al. (2018)) and Gumbel LSTM Choi et al.
(2018), which have shown that imbuing sequential models with hierarchical
inductive biases is a fruitful endeavor. After all, learning with well-suited
architectural inductive biases generally improves representation learning.

While most prior work relies on autoregressive methods for tree induction Shen
et al. (2018); Choi et al. (2018), this paper investigates the notion of a
separable, modular tree-induction module which learns to induce latent trees
in a parallelizable, non-autoregressive fashion. The key idea is to de-couple
tree induction from the recursive loop and parameterize the tree induction
module with parallelizable methods, such as convolution or position-wise feed-
forward layers, for efficient computation on GPU-enabled machines. To this
end, the tree-induction module no longer relies on the hidden-to-hidden
transition. The key motivation is that tree composition decisions are not
necessarily global (i.e., access to all previous tokens via a compressed
hidden state memory may not be necessary). Conversely, learning from local
neighborhoods may generate higher performing induced trees.

In light of the inherent speed and efficiency benefits, we coin our proposed
methods FastTrees. Empirically, we show that parallel tree induction is highly
effective in speed and performance, with $20\%-30\%$ faster inference than the
recent ON-LSTM while achieving competitive performance on a suite of NLP
tasks. Moreover, we find that parallel tree induction shines in processing
formal language, exceeding the state-of-the-art performance by $+4\%$
(absolute percentage) on the logical inference task Bowman et al. (2015b). On
the unsupervised parsing task, we show that FastTrees is capable of producing
high fidelity trees, with competitive performance when compared to the
autoregressive tree induction method in ON-LSTM. Additionally, we move beyond
parallel tree induction and propose Faster FastTrees, a quasi-recurrent model
that dispenses its reliance on hidden-to-hidden transitions, achieving
$\approx 40\%$ speed gain over ON-LSTM.

Aside from its enhanced inference speed over the ON-LSTM, the feasibility of
non-autoregressive tree induction opens up new avenues of research on modular
tree-induction modules. Beyond just faster ON-LSTM units, we propose a new
general-purpose FastTrees module that learns to induce trees on a sequence-
level. This allows us to learn tree-structured gating functions for
Transformer models Vaswani et al. (2017). Our experiments show the efficacy of
the FastTrees-enhanced Transformers, ameliorating vanilla Transformers
performance on a variety of tasks like machine translation, subject-verb
agreement, and even attaining $+8\%$ absolute improvement in accuracy for
mathematical language understanding.

#### Our contributions

Overall, the prime contributions of this work can be summarized as follows:

  * •

We propose that tree induction can be made parallel and non-autoregressive. We
show the general effectiveness of parallel trees on a variety of NLP tasks,
demonstrating that recurrence is not necessary for tree induction.

  * •

We propose FastTrees, adapted from the recent state-of-the-art ON-LSTM Shen et
al. (2018) for improved efficiency. Our proposed FastTrees outperforms ON-LSTM
on sentiment analysis, natural language inference and logical inference tasks.
This is achieved while enjoying $20\%-40\%$ in speed gains. Additionally, we
propose an even faster variation of FastTrees that completely dispenses with
hidden-to-hidden transitions altogether, also achieving similar and
competitive performance across several tasks.

  * •

We propose FastTrees Transformer models that outperform regular Transformers
on Mathematical Language Understanding tasks by $+8\%$.

##  2 Related Work

Learning to induce hierarchical structures from sequential data has shown
immense potential in many recent works Shen et al. (2018); Choi et al. (2018);
Drozdov et al. (2019); Shen et al. (2017); Bowman et al. (2015b); Jacob et al.
(2018). After all, many forms of sequential data, especially language, are
intrinsically hierarchical in nature. Latent tree induction can potentially
benefit the representation learning process, especially since finding suitable
inductive biases forms the cornerstone of deep learning research.

Recurrent neural networks (RNN) and its variants (e.g., LSTMs Hochreiter and
Schmidhuber (1997) and/or GRUs Cho et al. (2014) have been highly effective
inductive biases for reasoning with sequences. Over the years, there had been
many algorithmic innovations and improvements to the recurrent unit. A clear
and promising line of research is to incorporate recursive structures and
trees into sequence models. An early work, the Tree-LSTM Tai et al. (2015),
explicitly composes sequences by leveraging syntactic information, just like
many of the predecessor work, e.g., recursive neural networks Socher et al.
(2013). Subsequently, Bowman et al. (2016) proposed a stack-augmented neural
network trained by syntax information.

Learning latent tree structures without explicit syntactic supervision has
demonstrated recent success. Shen et al. (2017) proposed Parse-Read-Predict
Network (PRPN), a model that leverages self-attention and learned syntactic
distance to induce latent trees. Similarly, the Gumbel LSTM Choi et al. (2018)
learns the discrete composition of tokens in a sequence using the Gumbel
Softmax operator. Learning to compose with reinforcement learning is also a
prominent approach Yogatama et al. (2016). Recently, Havrylov et al. (2019)
proposed to jointly learn syntax and semantics via Proximal Policy
Optimization (PPO).

The most directly relevant work, the ON-LSTM Shen et al. (2018) proposed a new
activation function (cumulative Softmax) in order to imbue ordered hierarchies
in the gating functions of the LSTM. The tree-inducing gating functions are
parameterized in a similar fashion to the main LSTM unit and are also
conditioned on the previous hidden state.

The landscape of sequence encoding has recently shifted towards parallel
paradigms, instead of relying on the relatively slower, step-by-step
recurrence. This is made notable by the inception of models such as the
Transformer Vaswani et al. (2017) or Convolutional models Wu et al. (2019).
This work investigates the relative importance of the autoregressive induction
of tree structure. Our work is closely related to recent works that attempt to
parallelize the recurrent unit, e.g., Quasi-Recurrent Networks Bradbury et al.
(2016) or Simple Recurrent Units Lei et al. (2017). To this end, this work
brings forward a novel perspective of investigating the nature of
autoregressive versus non-autoregressive models within the context of latent
tree-induction.

##  3 Our Proposed Method

This section describes our proposed method. Our key innovation lies in
proposing sequence encoders that typically accept $\ell$ vectors of $d$
dimensions. For recurrent models, the recursive loop is defined as
$h_{t},c_{t}=\text{RNN}(x_{t},h_{t-1},c_{t-1})$. The output representation at
each time step is the hidden state $h_{t}$. On the other hand, $c_{t}$ is the
internal state of the RNN unit at timestep $t$.

Figure 1: FastTrees model, showing the parallel tree-induction module (in
purple) in an unrolled LSTM unit (vanilla LSTM connections in black). The
master forget gate $\hat{f}_{t}$ and master input gate $\hat{i}_{t}$ now rely
only on the input of the current time step ($x_{t}$), and thus can be computed
in parallel.

###  3.1 FastTrees

Our FastTrees unit accepts an input sequence of $X\in\mathbb{R}^{\ell\times
d}$ where $\ell$ is the length of the sequence and $d$ is the input dimension.
FastTrees can be interpreted as a modification of the standard LSTM cell. As a
necessary exposition within the context of this work, the details of the
standard LSTM cell are described as follows:

$\displaystyle f_{t}$ $\displaystyle=\sigma(W_{f}x_{t}+U_{f}h_{t-1}+b_{f})$
$\displaystyle i_{t}$ $\displaystyle=\sigma(W_{i}x_{t}+U_{i}h_{t-1}+b_{i})$
$\displaystyle o_{t}$ $\displaystyle=\sigma(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})$
$\displaystyle\hat{c}_{t}$ $\displaystyle=tanh(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})$
$\displaystyle c_{t}$ $\displaystyle=f_{t}\odot c_{t-1}+i_{t}\odot\hat{c}_{t}$
$\displaystyle h_{t}$ $\displaystyle=o_{t}\odot tanh(c_{t})$

where $x_{t}$ is the input token at timestep $t$. $h_{t}$ is the hidden state
of the LSTM unit at timestep $t$. $W_{*},U_{*},b_{*}$ where $*=\\{f,i,o,c\\}$
are the parameters of the unit. $\sigma$ is the sigmoid activation function
and $\odot$ is the element-wise (Hadamard) product. Gating functions are
parameterized by linear transformations applied on both the input token and
previous hidden state. Our modification lies in the construction of
structured, hierarchical gating functions, described as follows:

#### Parallel Tree-Induction

In order to learn tree-structures in a non-autoregressive fashion, we compute
the master forget gate $\hat{f}_{t}$ and master input gate $\hat{i}_{t}$ as
follows:

$\displaystyle\hat{f}_{t}$ $\displaystyle=\text{cumax}(F_{f}(x_{t}))_{t}$
$\displaystyle\hat{i}_{t}$ $\displaystyle=1-\text{cumax}(F_{i}(x_{t}))_{t}$

where $F_{f}(.)$ and $F_{i}{}(.)$ are parameterized functions. cumax() is the
cumulative Softmax function which essentially applies the cumulative sum
function right after a Softmax operator. Note that this is in contrast with
ON-LSTM Shen et al. (2018) which considers
$\hat{f}_{t}=\text{cumax}(W_{\hat{f}}x_{t}+U_{\hat{f}}h_{t-1}+b_{\hat{f}})$
and
$\hat{i}_{t}=1-\text{cumax}(W_{\hat{i}}x_{t}+U_{\hat{i}}h_{t-1}+b_{\hat{i}})$.
Instead, our formulation allows parallel computation of $\hat{f}_{t}$ and
$\hat{i}_{t}$. The standard choice of $F_{f}(.)$ and $F_{i}{}(.)$ are 2
layered position-wise feed-forward layers:

$\displaystyle F(x_{t})=W{x_{t}}+b_{t}$

Alternatively, we may also consider incorporating local information into the
tree induction mechanism using causal 1D convolutions followed by a single
positional feed-forward layer.

$\displaystyle F(X)=F_{P}(\text{CausalConv}(X))$

In this case, the sequence token $x_{t}$ should not have access to tokens
$>t$. $F_{P}(.)$ is a position feed-forward layer. The two variants are named
FastTrees and Conv. FastTrees respectively.

Next, to learn tree-structured hidden representations, the structured gating
mechanism is defined as follows:

$\displaystyle\omega_{t}$ $\displaystyle=\hat{f}_{t}\odot\hat{i}_{t}$
$\displaystyle\hat{f}_{t}$
$\displaystyle=f_{t}\odot\omega_{t}+(\hat{f}_{t}-\omega_{t})$
$\displaystyle\hat{i}_{t}$
$\displaystyle=i_{t}\odot\omega_{t}+(\hat{i}_{t}-\omega_{t})$ $\displaystyle
c_{t}$ $\displaystyle=\hat{f}_{t}\odot c_{t-1}+\hat{i}_{t}\odot\hat{c}_{t}$

where $c_{t}$ is the new cell state at time step $t$. The gates are now imbued
with a hierarchical structure, due to the splitting of points defined by the
cumax function. We refer interested readers to Shen et al. (2018) for more
details.

###  3.2 Faster FastTrees

We move beyond parallel tree induction and propose an even faster variant of
FastTrees. We completely remove the reliance of the hidden-to-hidden
transition within the recurrent unit. The Faster FastTrees unit is described
as follows:

$\displaystyle f_{t}$
$\displaystyle=\sigma(F_{f}(x_{t})),\>i_{t}=\sigma(F_{i}(x_{t})),\>o_{t}=\sigma(F_{o}(x_{t}))$
$\displaystyle\hat{f}_{t}$
$\displaystyle=\text{cumax}(F_{f}(x_{t}))_{t},\>\>\hat{i}_{t}=1-\text{cumax}(F_{i}(x_{t}))_{t}$
$\displaystyle\omega_{t}$ $\displaystyle=\hat{f}_{t}\odot\hat{i}_{t}$
$\displaystyle\hat{f}_{t}$
$\displaystyle=f_{t}\odot\omega_{t}+(\hat{f}_{t}-\omega_{t}),\>\>\hat{i}_{t}=i_{t}\odot\omega_{t}+(\hat{i}_{t}-\omega_{t})$
$\displaystyle\hat{c}_{t}$
$\displaystyle=tanh(W_{c}x_{t}+b_{c}),\>\>c_{t}=\hat{f}_{t}\odot
c_{t-1}+\hat{i}_{t}\odot\hat{c}_{t}$ $\displaystyle h_{t}$
$\displaystyle=o_{t}\odot tanh(c_{t})$

where $F_{f}(x_{t}),F_{i}(x_{t}),F_{o}(x_{t})$ are linear transformations.
This formulation is similar in spirit to QRNNs Bradbury et al. (2016) and can
be interpreted as the quasi-recurrent adaptation of ON-LSTM. Naturally, this
formulation gains speed, as more gate construction functions are now
parallelizable.

Model | Parameters | Val ppl | Test ppl | Time (sec)
---|---|---|---|---
PRPN-LM Shen et al. (2017) | - | - | 62.00 | -
4-Layer Skip Connection Melis et al. (2017) | 24M | 60.90 | 58.30 | -
AWD-LSTM Merity et al. (2017) | 24M | 60.00 | 57.30 | -
ON-LSTM Shen et al. (2018) | 25M | 58.32 | 56.25 | 296
Faster FastTrees (This work) | 25M | 61.93 | 59.10 |  173 $(+42\%)$
Conv. FastTrees (This work) | 25M | 59.88 | 57.30 | 204 $(+31\%)$
FastTrees (This work) | 25M | 58.47 | 56.35 | 209 $(+29\%)$
Table 1: Model perplexity (lower is better) on validation and test sets for
language modeling task on PTB. Train times measured by wall-clock in seconds
per epoch. Percentage speed improvements of FastTrees over ON-LSTM denoted in
parenthesis.

###  3.3 FastTree Transformers

To demonstrate the general-purpose utility of our method, we show how our
FastTrees module can be added to boost the performance of Transformers models.
The key idea is to show that this hierarchical inductive bias can be useful
when added to other sequence transduction models. This module accepts an input
$X\in\mathbb{R}^{\ell\times d}$ and produces an output representation of equal
shape $Y\in\mathbb{R}^{\ell\times d}$. Note that we switch to matrix notation
since our operations now operate at sequence-level.

$\displaystyle A$ $\displaystyle=\text{cumax}(F_{a}(X))$ $\displaystyle B$
$\displaystyle=(1-\text{cumax}(F_{b}(X)))$ $\displaystyle F$
$\displaystyle=((\sigma(F_{h}(X)))\odot(A\odot B))+(A-(A\odot B))$
$\displaystyle Y$ $\displaystyle=F\odot X$

$F_{a}(.),F_{b}(.),F_{h}(.)$ are parameterized functions which may be
sequence-level operations (e.g., convolution or position-wise feed-forward
layers). The key idea behind this approach is to convert the token-level
ordered hierarchical inductive bias Shen et al. (2018) into sequence level
operations. The interpretation of FastTrees Transformer is reminiscent of the
$\hat{f},\hat{i}$ master gating functions in ON-LSTM and/or FastTrees. Here,
we simply learn a single structured forget gate, directly masking the output
representations of this layer in a tree-structured fashion.

#### Overall Architecture

We place the tree induction module denoted FastTrees(.) right after the self-
attention layer. A single Transformer block is now written as:

$\displaystyle\hat{X}$
$\displaystyle=\frac{1}{\sqrt{d}}\>\text{Softmax}(W_{Q}(X^{\prime})W_{K}(X^{\prime}))^{\top}(W_{V}(X^{\prime})$
$\displaystyle Y^{\prime}$ $\displaystyle=\textsc{FastTrees}(\hat{X})$
$\displaystyle\hat{Y}$ $\displaystyle=F_{\phi}(Y^{\prime}))$

where $F_{\phi}(.)$ is a two-layered position-wise feed-forward network with
ReLU activations. Notably, our extension inherits all other properties from
the base Transformer, including its multi-headed and multi-layered nature.

##  4 Experiments

We evaluate FastTrees units on four well-established sequence modeling tasks
(language modeling, logical inference, sentiment analysis and natural language
inference), and FastTrees-augmented Transformers on three tasks (machine
translation, subject-verb agreement and mathematical language understanding).
All timed comparisons were tested by training the models on NVIDIA Tesla V100
GPUs for fair comparison, and averaging the wall clock time per epoch over
several cycles.

###  4.1 Word-level Language Modeling

Word-level language modeling is an important test of the modeling capabilities
of a neural network in various linguistic phenomena. In evaluating our model’s
representational abilities, we test it by measuring perplexity on the Penn
Treebank (PTB) Marcus et al. (1993) dataset
222https://catalog.ldc.upenn.edu/LDC99T42.

#### Experimental Setup

To evaluate our model performance fairly, we follow the model hyper-
parameters, regularization and optimization techniques used in ON-LSTM Shen et
al. (2018). We use a three-layer model with 1150 units in the hidden layer,
and an embedding size of 400. We apply dropout on the word vectors, the output
between LSTM layers, the output of the final LSTM layer, and embedding dropout
of (0.5, 0.3, 0.45, 0.1) respectively. We perform grid search on hidden layer
sizes in the range of [600,1500] with the step size of 50, and found 1150 to
have the best validation performance.

#### Results

Table 1 reports our results on the word-level language modeling task. Keeping
the number of layers, hidden state and embedding dimensions constant, we show
that the performance attained by FastTrees is competitive to ON-LSTM while
training $29-42\%$ faster, and also outperforms AWD-LSTM Merity et al. (2017).

| % Accuracy on each sequence length |
---|---|---
Model | 7 | 8 | 9 | 10 | 11 | 12 | Time (sec)
TreeLSTM† Tai et al. (2015) | 94.0 | 92.0 | 92.0 | 88.0 | 97.0 | 86.0 | -
LSTM | 88.0 | 84.0 | 80.0 | 78.0 | 71.0 | 69.0 | -
RRNet Jacob et al. (2018) | 84.0 | 81.0 | 78.0 | 74.0 | 72.0 | 71.0 | -
ON-LSTM Shen et al. (2018) | 91.0 | 87.0 | 85.0 | 81.0 | 78.0 | 75.0 | 85
Faster FastTrees | 66.0 | 62.0 | 57.0 | 55.0 | 53.0 | 53.0 |  49.0 $(+42\%)$
FastTrees | 91.0 | 88.0 | 83.0 | 80.0 | 76.0 | 74.0 | 75.0 $(+11\%)$
Conv. FastTrees | 93.0 | 90.0 | 86.0 | 83.0 | 80.0 | 79.0 | 58.0 $(+32\%)$
Accuracy Gain (abs.) over ON-LSTM | $+2\%$ | $+3\%$ | $+1\%$ | $+2\%$ | $+2\%$ | $+4\%$ |
Table 2: Percentage accuracy on test set of models on each sequence length
(from 7 to 12) for logical inference task. $\dagger$ denotes with ground truth
syntax. Train times measured by wall-clock in seconds per epoch. Percentage
speed improvements of FastTrees over ON-LSTM denoted in parenthesis.

###  4.2 Logical Inference

The logical inference task Bowman et al. (2015b) tests for a model’s ability
to exploit a recursively defined language to generalize sentences with complex
unseen structures. The key idea is that models with appropriate hierarchical
inductive bias will do well on this task.

Sentences in this language use a combination of up to six word types
$(p_{1},p_{2},p_{3},p_{4},p_{5},p_{6})$ and three logical operations
$(and,or,not)$. The length of a sentence pair (termed as the sequence length)
is defined by the number of logical operators within the longer of the two
sentences. The relationship between the two sentences are described using
seven mutually exclusive logical relations: two directions of entailment
$(\sqsubset,\sqsupset)$, equivalence $(\equiv)$, exhaustive and non-exhaustive
contradiction $(\wedge,\mid)$, and two types of semantic independence
$(\\#,\smile)$. The task is to predict the correct logical relationship given
a sequence.

#### Experimental Setup

We train the model with sequence lengths varying up to 6, while evaluating the
model separately on each sequence length (up to 12). Notably, performance on
sequence lengths between 7 and 12 is representative of its ability to
generalize to unseen sentence structures, given their absence in the training
data. We use a 80/20% train/test split, with 10% of the training set reserved
for the validation set. We parameterize the RNN models with a hidden layer of
size 400, and the input embedding of size 128. A dropout of 0.2 was applied
between different layers.

#### Results

Table 2 illustrates the performance improvements of Conv. FastTrees over ON-
LSTM. Conv. FastTrees outperforms ON-LSTM, especially in longer sentence
sequences with a $+4\%$ absolute improvement for sequence length 12. This
shows its ability to learn recursive data structures while operating $+32\%$
faster. The key idea here is to show that learning trees in a parallel, non-
autoregressive fashion can be sufficiently powerful as a model hierarchical
structure.

Model | Parameters | Acc (%) | Time (sec)
---|---|---|---
LSTM | 2.3M | 87.86 | 132
ON-LSTM | 2.3M | 88.30 | 275
Faster FastTrees | 2.3M | 86.99 |  161 $(+42\%)$
Conv. FastTrees | 2.3M | 88.46 | 218 $(+21\%)$
FastTrees | 2.3M | 88.69 | 194 $(+30\%)$

Table 3: Sentiment classification task with percentage accuracy evaluated on
test set of SST-2. Time measured by wall-clock in seconds per training epoch.
Percentage speed improvements of FastTrees over ON-LSTM denoted in
parenthesis.

###  4.3 Sentiment Analysis

In our experiments, we train our models using the binary Stanford Sentiment
Treebank (SST-2) Socher et al. (2013), a 70k sentence dataset where the model
predicts a positive or negative sentiment label given an input
sentence333https://www.kaggle.com/atulanandjha/stanford-sentiment-
treebank-v2-sst2.

#### Experimental Setup

We initialize the word vectors using two well-established pre-trained
embedding types, GloVe 300D Pennington et al. (2014) and FastText Grave et al.
(2018). Embedding projections of size $256$ are fed into an encoder with
hidden size of $512$, and subsequently an MLP with a hidden layer $D_{c}$ of
size $512$ for the classifier layer. We vary the architecture of the middle
encoder layer between ON-LSTM and varying FastTrees models for the experiment.
The initial learning rate is set to $0.0004$ and dropped by a factor of $0.2$
when accuracy plateaus, with dropout set to $0.5$. The loss used is standard
cross-entropy, and Adam is used for optimization Kingma and Ba (2014).

#### Results

High performance in the sentiment analysis task is indicative of hierarchical
representation induced, as the successful classification of sentence sentiment
is more likely when the model learns a strategy in identifying context and
word relations within the entire sentence. From table 3, we see how both
FastTrees and Conv. FastTrees outperforms ON-LSTM and LSTM on the task, with
significant speed-up in training time. This increase in performance supports
our hypothesis of the induced tree representations in FastTrees.

###  4.4 Natural Language Inference

Natural language inference is the task of predicting whether two sentences, a
premise sentence and a hypothesis sentence, are neutral, entailing or
contradictory. Inference of the latter two is essential in natural language
understanding, indicative of a model’s ability for semantic representation. We
use the Stanford Natural Language Inference (SNLI) dataset Bowman et al.
(2015a), which consists of about 570k human-generated, manually-labeled
English sentence pairs444https://nlp.stanford.edu/projects/snli/.

#### Experimental Setup

The input to the model is a sentence pair, they are represented individually
as the premise sentence vector $h^{pre}$ and the hypothesis sentence vector
$h^{hyp}$ by the sentence encoder. The vectors are then concatenated, with
embedding projections of size $256$ that are fed into an encoder with hidden
size of $512$, and subsequently an MLP with a hidden layer of size $1024$ for
the classifier layer. We vary the architecture of the middle encoder layer
between ON-LSTM and each FastTrees model for the experiment. The initial
learning rate is set to $0.0004$ and dropped by a factor of $0.2$ when
accuracy plateaus, with dropout set to $0.2$. The MLP classifier has a dropout
of $0.2$. The loss used is standard cross-entropy, and Adam is used for
optimization.

Model | Parameters | Acc (%) | Time (sec)
---|---|---|---
LSTM | 4.9M | 86.12 | 657
Gumbel Tree-LSTM | 10.3M | 86.00 | -
ON-LSTM | 4.9M | 85.82 | 1348
Faster FastTrees | 4.9M | 85.21 |  850 $(+37\%)$
FastTrees | 4.9M | 86.04 | 1062 $(+21\%)$
Conv. FastTrees | 4.9M | 86.23 | 1052 $(+22\%)$

Table 4: Percentage accuracy of models evaluated on test set of SNLI on the
natural language inference task. Corresponding size of models in number of
parameters denoted in parenthesis. Train times in seconds per epoch generated
from wall-clock time. Percentage speed improvements of FastTrees over ON-LSTM
denoted in parenthesis.

#### Results

From table 4, we note the improved performance of FastTrees and Conv.
FastTrees over ON-LSTM, while training >$20\%$ faster. Conv. FastTrees also
outperforms LSTM and Gumbel Tree-LSTM in this task, with a model size half
that of the Gumbel Tree-LSTM.

###  4.5 Neural Machine Translation

This task involves translating between one language and another. More
concretely, we utilize the IWSLT’15 English-Vietnamese (En-Vi)
dataset555https://nlp.stanford.edu/projects/nmt/.

#### Experimental Setup

We implement our model with the
Tensor2Tensor666https://github.com/tensorflow/tensor2tensor framework, using
Transformer Base as the key baseline model. We train both the base Transformer
and our proposed models for $50K$ steps using the default hyperparameters.
During inference, the length penalty is set to $0.6$ and the beam size is set
to $4$. We average the parameters of the last 8 checkpoints.

#### Results

Table 5 illustrates how our proposed method achieves state-of-the-art
performance on the IWSLT’15 En-Vi dataset, outperforming not only the base
transformer model but also prior work. More importantly, we demonstrate the
utility of the FastTrees module, achieving a $1.88\%$ absolute improvement in
BLEU score. Both variants of FastTrees are able to enhance the base
Transformers model performance in this task.

Model | BLEU
---|---
Luong & Manning (2015) | 23.30
Seq2Seq Attention | 26.10
Neural Phrase-based MT | 27.69
Neural Phrase-based MT + LM | 28.07
Transformer Vaswani et al. (2017) | 28.43
Transformer + FastTrees | 30.31
Transformers + Conv. FastTrees | 29.72
Abs. Improvement | +$1.88\%$
Table 5: BLEU Papineni et al. (2002) scores on machine translation task using
IWSLT’15 English-Vietnamese dataset.

###  4.6 Subject-Verb Agreement and Mathematical Language Understanding

We include additional experiments on subject-verb agreement (SVA) Linzen et
al. (2016) and mathematical language understanding (MLU) Wangperawong (2018)
to demonstrate the effectiveness of FastTrees. The SVA task is a binary
classification problem, determining if a sentence, _e.g._ , ‘The keys to the
cabinet _____ .’ is followed by a plural or singular verb. This involves
language understanding pertaining to learning syntax-sensitive dependencies.
On the other hand, the MLU task involves input sequences such as
$x=85,y=-523,x*y$ in which the expected decoding output should be $-44455$.

#### Experimental Setup

Experiments are conducted on the Tensor2Tensor framework, using the tiny
default hyperparameter setting. Models are trained for $10K$ steps for SVA and
$100K$ steps for MLU. The evaluation metric is accuracy for the SVA task and
accuracy per sequence for MLU.

Model | SVA | MLU
---|---|---
Transformer | 94.8 | 76.10
Transformer + FastTrees | 94.8 | 84.26
Transformer + Conv. FastTrees | 95.7 | 82.62
Abs. Improvement | +$0.9\%$ | +$8.16\%$

Table 6: Accuracy scores on subject-verb agreement prediction task and
accuracy per sequence on mathematical language understanding task.

#### Results

Table 6 presents our experimental results on SVA and MLU. Once again, we
observe improvements over the base Transformer model. While we observe modest
improvements on the SVA task with Conv. FastTrees, the performance gain on the
MLU task is very promising. Notably, we achieve an $+8.16\%$ absolute
improvement on accuracy per sequence. To this end, we posit that the naturally
hierarchical nature of mathematical expressions enables our inductive bias to
shine on this task.

##  5 Qualitative Analysis

First, we perform an analysis on the quality of trees produced by models
trained on the language modeling tasks. This is determined by the parsing F1
score on the unsupervised constituency parsing task. Second, we visualize the
output trees composed by the language models and that of human annotators;
these visualizations serve to examine the fidelity of trees produced by
FastTrees.

###  5.1 Unsupervised Constituency Parsing

The unsupervised constituency parsing task compares the latent tree structure
induced by a pre-trained model with parse outputs annotated by human experts.
To ensure reproducibility, we follow the experiment setup proposed in Htut et
al. (2018). We first train the models on the language modeling task using the
PTB dataset and save those with sufficiently low perplexities. Subsequently,
we test them on the Wall Street Journal 10 (WSJ10) dataset and WSJ test set
for this task.

Model | Parsing F1 | % Accuracy
---|---|---
WSJ10 | WSJ | ADJP | NP | PP
Random Trees | 32.2 | 18.6 | 17.4 | 22.3 | 16.0
Balanced Trees | 43.4 | 24.5 | 22.1 | 20.2 | 9.3
Left Branching | 19.6 | 9.0 | 17.4 | - | -
Right Branching | 56.6 | 39.8 | - | - | -
3300D ST-Gumbel† | - | 20.1 | 15.6 | 18.8 | 9.9
w/o Leaf GRU† | - | 25 | 18.9 | 24.1 | 14.2
300D RL-SPINN† | - | 13.2 | 1.7 | 10.8 | 4.6
w/o Leaf GRU† | - | 13.2 | 1.6 | 10.9 | 4.6
PRPN-LM | 71.3 | 38.1 | 26.2 | 63.9 | 24.4
ON-LSTM | 66.8 | 49.4 | 46.2 | 61.4 | 55.4
FastTrees | 65.5 | 44.3 | 43.1 | 54.8 | 52.9

Table 7: Unlabeled parsing F1 scores evaluated on full WSJ10 and WSJ test
sets. † denotes models trained on NLI task, and evaluated on full WSJ. The %
Accuracy columns represent percentage of ground truth constituents of a given
type that corresponds to constituents in the model parses.

Table 7 reports the parsing F1 scores obtained from the $2^{nd}$ layer of the
3-layered ON-LSTM and FastTrees. On WSJ10, FastTrees outperforms the random
and standard branching baselines, and closely matches ON-LSTM. On WSJ,
FastTrees achieves 44.3, outperforming ST-Gumbel Choi et al. (2018) and RL-
SPINN Yogatama et al. (2016) models and variants. We also note that FastTrees
performs well on phrase detection, including adjective phrases (ADJP), noun
phrases (NP), and prepositional phrases (PP). From this analysis, we show that
with the removal of hidden-to-hidden transition on FastTrees to obtain speed-
ups of $20-40\%$, the induced latent trees are still of relatively high
fidelity; FastTrees outperforms the standard branching trees, ST-Gumbel, and
RL-SPINN models in parsing F1 score, closely resembling the ground truth.

###  5.2 Tree Visualization

We illustrate samples of the parse output from the unsupervised constituency
parsing task in figures 2 and 3. FastTrees is able to generate parse trees
very close to the human-annotated ground truth, as shown in figure 2,
achieving an F1 score of $85.7\%$.

Figure 2: Ground Truth (top), FastTrees output (bottom)

Figure 3 shows how the model output can resemble the ground truth even for
longer sequences of $24$ tokens, with an F1 score of $71.0\%$ versus ON-LSTM
at $15.4\%$. We surmise that these examples demonstrate the abilities of
FastTrees in learning non-trivial tree composition schemes that are useful for
many downstream NLP tasks.

Figure 3: Parse tree from Ground Truth (top), FastTrees (middle), and ON-LSTM
Shen et al. (2018) (bottom)

##  6 Conclusion

We propose FastTrees, a model that induces latent trees in a parallel, non-
autoregressive fashion. Our proposed model outperforms ON-LSTM on logical
inference, natural language inference and sentiment analysis while training up
to 40% faster. FastTrees also achieves state-of-the-art results on logical
inference, demonstrating that it can learn effectively on intrinsically
hierarchical data. Finally, we show that parallel FastTrees can be used to
enhance Transformer models, bringing about $8\%$ advancement over MLU.

## References

  * Bowman et al. (2015a) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015a.  A large annotated corpus for learning natural language inference.  _arXiv preprint arXiv:1508.05326_.
  * Bowman et al. (2016) Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. 2016.  A fast unified model for parsing and sentence understanding.  _arXiv preprint arXiv:1603.06021_.
  * Bowman et al. (2015b) Samuel R Bowman, Christopher D Manning, and Christopher Potts. 2015b.  Tree-structured composition in neural networks without tree-structured architectures.  _arXiv preprint arXiv:1506.04834_.
  * Bradbury et al. (2016) James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2016.  Quasi-recurrent neural networks.  _arXiv preprint arXiv:1611.01576_.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_.
  * Choi et al. (2018) Jihun Choi, Kang Min Yoo, and Sang-goo Lee. 2018.  Learning to compose task-specific tree structures.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Drozdov et al. (2019) Andrew Drozdov, Pat Verga, Mohit Yadav, Mohit Iyyer, and Andrew McCallum. 2019.  Unsupervised latent tree induction with deep inside-outside recursive autoencoders.  _arXiv preprint arXiv:1904.02142_.
  * Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018.  Learning word vectors for 157 languages.  _arXiv preprint arXiv:1802.06893_.
  * Havrylov et al. (2019) Serhii Havrylov, Germán Kruszewski, and Armand Joulin. 2019.  Cooperative learning of disjoint syntax and semantics.  _arXiv preprint arXiv:1902.09393_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Htut et al. (2018) Phu Mon Htut, Kyunghyun Cho, and Samuel R Bowman. 2018.  Grammar induction with neural language models: An unusual replication.  _arXiv preprint arXiv:1808.10000_.
  * Jacob et al. (2018) Athul Paul Jacob, Zhouhan Lin, Alessandro Sordoni, and Yoshua Bengio. 2018.  Learning hierarchical structures on-the-fly with a recurrent-recursive model for sequences.  In _Proceedings of The Third Workshop on Representation Learning for NLP_ , pages 154–158.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Lei et al. (2017) Tao Lei, Yu Zhang, and Yoav Artzi. 2017.  Training rnns as fast as cnns.  _arXiv preprint arXiv:1709.02755_.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of lstms to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993.  Building a large annotated corpus of English: The Penn Treebank.  _Computational Linguistics_ , 19(2):313–330.
  * Melis et al. (2017) Gábor Melis, Chris Dyer, and Phil Blunsom. 2017.  On the state of the art of evaluation in neural language models.  _arXiv preprint arXiv:1707.05589_.
  * Merity et al. (2017) Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017.  Regularizing and optimizing lstm language models.  _arXiv preprint arXiv:1708.02182_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting on association for computational linguistics_ , pages 311–318. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Shen et al. (2017) Yikang Shen, Zhouhan Lin, Chin-Wei Huang, and Aaron Courville. 2017.  Neural language modeling by jointly learning syntax and lexicon.  _arXiv preprint arXiv:1711.02013_.
  * Shen et al. (2018) Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2018.  Ordered neurons: Integrating tree structures into recurrent neural networks.  _arXiv preprint arXiv:1810.09536_.
  * Shen et al. (2020) Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, and Aaron Courville. 2020.  Structformer: Joint unsupervised induction of dependency and constituency structure from masked language modeling.  _arXiv preprint arXiv:2012.00857_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015.  Improved semantic representations from tree-structured long short-term memory networks.  _arXiv preprint arXiv:1503.00075_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wangperawong (2018) Artit Wangperawong. 2018.  Attending to mathematical language with transformers.  _CoRR_ , abs/1812.02825.
  * Wu et al. (2019) Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. 2019.  Pay less attention with lightweight and dynamic convolutions.  _arXiv preprint arXiv:1901.10430_.
  * Yogatama et al. (2016) Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Wang Ling. 2016\.  Learning to compose words into sentences with reinforcement learning.  _arXiv preprint arXiv:1611.09100_.


# Mixed singlet-triplet superconducting state within the moiré $t$-$J$-$U$
model as applied to the description of twisted WSe2 bilayer

M. Zegrodnik<EMAIL_ADDRESS>Academic Centre for Materials and
Nanotechnology, AGH University of Krakow, Al. Mickiewicza 30, 30-059 Krakow,
Poland  A. Biborski  Academic Centre for Materials and Nanotechnology, AGH
University of Krakow, Al. Mickiewicza 30, 30-059 Krakow, Poland

###### Abstract

We analyze an analog of the $t$-$J$-$U$ model as applied to the description of
a single moiré flat band of twisted WSe2 bilayer. To take into account the
correlation effects induced by a significant strength of the Coulomb
repulsion, we use the Gutzwiller approach and compare it with the results
obtained by the Hartree-Fock method. We discuss in detail the graduate
appearance of a two dome structure of the superconducting state in the phase
diagram by systematically increasing the Coulomb repulsion integral, $U$. The
two superconducting domes residing on both sides of a Mott insulating state
can be reproduced for a realistic parameter range in agreement with the
available experimental data. According to our analysis the paired state has a
highly unconventional character with a mixed $d+id$ (singlet) and $p-ip$
(triplet) symmetry. Both components of the mixed paired state are of
comparable amplitudes. However, as shown here, a transition between pure
singlet and pure triplet pairing should be possible in the considered system
by tuning the top and bottom gate voltage, which controls the magnitude of
valley-dependent spin-splitting in the system.

##  I Introduction

In recent years the twisted moiré structures have gathered a significant
amount of interest due to their rich physics on one side and high degree of
tunability on the other. The development in the field has been mostly
triggered by the discovery of both correlated insulating state and
unconventional superconductivity in the magic angle twisted bilayer graphene
(TBG)Cao  _et al._ (2018a, b). Since than, many other moiré structures have
been intensively studied which are bilayer boron nitrideXian  _et al._ (2019),
multilayer grapheneZhang  _et al._ (2021); Chen  _et al._ (2019); Shen  _et
al._ (2020) as well as both homo- and hetero-bilayer transition metal
dichalcogenides (TMD)Wang  _et al._ (2020); Liao  _et al._ (2020); Ulstrup
_et al._ (2020); Ghiotto  _et al._ (2021).

In moiré systems, for certain twist angles between the atomic layers, flat
energy bands are created. This feature points to a significant role of inter-
electronic interactions which is a situation similar to that known from the
so-called strongly correlated electron systems such as copper based high-
temperature superconductorsSpałek  _et al._ (2022). In fact, both the well
known cuprates and the novel moiré systems exhibit similar feature in the
phase diagram which is the appearance of superconducting (SC) domes residing
in close proximity of the correlated insulating stateCao  _et al._ (2018b);
Wang  _et al._ (2020).

For the case of TBG the flat bands and the resulting correlation induced
phenomena are fragile when it comes to deviations of the twist angle around
the so-called magic values. On the other hand, in TMD moiré systems the effect
is more robust and the flat bands are observed for larger spectrum of
anglesWang  _et al._ (2020); An  _et al._ (2020); Ghiotto  _et al._ (2021).
What also distinguisches the TMD systems from those graphene based is the
appearance of significant spin-orbit coupling.

As it has been shown both by the DFT calculations and continuum model based
approach, the twisted homo-bilayer WSe2 (tWSe2) can be described by a single-
band tight-binding model on a triangular lattice with spin-orbit coupled
hoppings which reproduce the flat electronic band with a valley-dependant
spin-splittingWang  _et al._ (2020); Pan  _et al._ (2020). In such an approach
the so-called displacement field controlled by the bias voltage across the
bilayer, tunes the spin-dependent phase of the hopping amplitudes. This allows
for changing the magnitude of the spin-splitting in the system. In order to
take into account the electron-electron interactions, such approach can be
supplemented with the onsite Coulomb repulsion term leading to the Hubbard
model. It has been estimated that the value of the Coulomb integral $U$ places
tWSe2 in the moderate correlations regimeWang  _et al._ (2020). However, the
magnitude of electronic correlations in those systems can be controlled by the
twist angle which changes the $U/W$ parameter, with $W$ being the bare band
width. The Hubbard model has been studied in the context of both magnetically
ordered states and $d+id$ spin-singlet supercondcutivity of tWSe2Zang  _et
al._ (2021); Bélanger  _et al._ (2022); Zang  _et al._ (2022); Pan  _et al._
(2020); Pan and Das Sarma (2022). In such picture the SC pairing comes as a
result of the electronic correlations which become significant for large
enough value of the Hubbard $U$. Another canonical model which can be analyzed
in the context of electron-electron correlation effects in TMD moiré bands is
the $t$-$J$ modelWu  _et al._ (2018); and Sheng (2022); Chen and Sheng (2023)
within which the Cooper pairing comes naturally from the exchange interaction
term $\sim J$. However, it should be noted that the situation in this context
is not completely resolved so far and a well established description of the
paired state in those systems has not been reached.

Here, we propose an analog of the $t$-$J$-$U$ model as applied to the
description of a single moiré flat band of tWSe2. The $t$-$J$-$U$ hamiltonian
has been initially proposed for the description of copper based high-TC
superconductorsDaul  _et al._ (2000); Basu  _et al._ (2001); Xiang  _et al._
(2009) and as shown by us in previous years it leads to good quantitative
agreement between theory and experiment for selected fundamental features of
the paired stateSpałek  _et al._ (2017). This fact together with similarities
between the cuprates and twisted moiré van der Waals structures constitutes
the main motivation for the analysis of unconventional superconducting phase
in tWSe2 with the use of the $t$-$J$-$U$ based approach. Due to the
significant magnitude of the onsite Coulomb repulsion in such an analysis one
has to take into account the electronic correlation effects beyond the level
of a simple Hartree-Fock method (HF). Here, we use the Gutzwiller approach
(GA), which is efficient and catches the basic effects related with electron-
electron interactions. As we show the two superconducting domes residing on
both sides of the insulating state can be reproduced with the use of the
presented method in agreement with the experimental indicatorsWang  _et al._
(2020). The character of the obtained superconducting state is highly
unconventional with a mixed singlet-triplet Cooper pairing. Interestingly,
according to our analysis the balance between the singlet and triplet
contributions to the pairing could be controlled by the gate voltage in a real
experimental setup. In particular, in proper gate voltage range, a transition
between pure singlet and pure triplet pairing can be achieved within a single
non-hybrid system.

##  II Model and method

Here we apply a moiré band $t$-$J$-$U$ hamiltonian of the form

$\hat{H}=\hat{H}_{0}+\hat{H}_{J}+\hat{H}_{U},$ (1)

where $\hat{H}_{0}$ represents a tight-binding model with spin-orbit coupled
hoppings on a triangular lattice. As shown in Refs. Wang  _et al._ , 2020; Pan
_et al._ , 2020, such single particle hamiltonian can well describe the top
moiré band of tWSe2. Namely,

$\hat{H}_{0}=\sum_{<ij>\sigma}t_{ij\sigma}\;\hat{c}^{\dagger}_{i\sigma}\hat{c}_{j\sigma},$
(2)

where $t_{ij\sigma}$ are the hopping parameters between the nearest-
neighboring sites of the form,

$t_{ij\sigma}=|t|e^{i\sigma\phi_{ij}},$ (3)

where $\phi_{ij}=-\phi_{ji}$ due to hermiticity condition and $\sigma=+1,-1$
represents the spin up and down states associated with $+K$ and $-K$ valleys,
respectively. The hopping parameters fulfill the $C_{3}$ rotational symmetry
and reconstruct the spin-splitted Fermi surfaces. This is a manifestation of
the fact that the system is non-centrosymmetric and spin-orbit coupling
appears. In order to simplify the situation, here we limit to the nearest-
neighbor hopping terms (summation over $<ij>$) what is justified by the fact
that the longer-ranged hoppings are at least one order of magnitude
smallerWang  _et al._ (2020); Pan  _et al._ (2020). In the further analysis we
take the values of hoppings provided in Ref. Wang  _et al._ , 2020 for the
twist angle $5.08^{\circ}$ which is close to the one for which the signatures
of superconductivity have been identified experimentally. As shown in Fig. 1,
by changing the value of the so-called displacement field one can tune both
$|t|$ and $\phi$, where the latter determines the magnitude of the valley-
dependent spin-splitting. In real experimental setup the displacement field is
controlled by the bias voltage across the bilayer.

Figure 1: The amplitude (a) and phase (c) of the nearest neighbor complex
hopping parameter ($t_{\uparrow}=|t|e^{i\phi}$) as a function of the
displacement field as calculated from the data provided in Ref. Wang  _et al._
, 2020. The dispersion relations [(c) and (d)] and the Fermi surfaces at half-
filling [(e) and (f)] for two selected values of the displacement field
provided in the Figures. Note that by increasing the displacement field one
also increases the phase of the hopping parameters which in turn enhances the
spin-splitting of the Fermi surfaces.

The intersite exchange interaction term appearing in Hamiltonian (1) has the
following form

$\begin{split}\hat{H}_{J}&=J\sideset{}{{}^{\prime}}{\sum}_{<ij>}\bigg{(}\hat{S}^{z}_{i}\hat{S}^{z}_{j}+\cos{(2\phi_{ij})}\sum_{\alpha=x,y}\hat{S}^{\alpha}_{i}\hat{S}^{\alpha}_{j}\\\
&+\sin{(2\phi_{ij})}(\mathbf{\hat{S}}_{i}\times\mathbf{\hat{S}}_{j})\cdot\hat{z}\bigg{)},\end{split}$
(4)

where $\hat{S}^{\alpha}_{i}$ for $\alpha=x,y,z$ are the three components of
the electron spin operator and $\phi_{ij}$ is the phase of the spin up hopping
between $i$ and $j$ lattice sites. The summation above runs over the nearest
neighbor sites only. The prime indicates that each bond between $i$ and $j$ is
taken only once. Due to the appearance of the complex spin-dependant hoppings,
the exchange term is slightly modified with respect to the one originally
appearing within the $t$-$J$-$U$ model as applied for the cupratesSpałek  _et
al._ (2017). In particular, here the Dzyaloshinskii-Moriya (DM) term appears
which, however, might be significantly suppressed for very low values of the
displacement fields for which $\phi\approx 0$. In such a case the exchange
interaction takes the standard form,
$\sum_{ij}\mathbf{\hat{S}}_{i}\mathbf{\hat{S}}_{j}$.

The last term in Hamiltonian (1) corresponds to the onsite Coulomb interaction

$\hat{H}_{U}=U\sum_{i}\hat{n}_{i\uparrow}\hat{n}_{i\downarrow},$ (5)

where $\hat{n}_{i\uparrow}$ ($\hat{n}_{i\downarrow}$) are the number of
particle operators on given site with spin up (down), while $U$ corresponds to
the Coulomb interaction strength.

In order to take into account the correlation effects induced by a significant
value of the Hubbard $U$ we apply the zeroth order diagrammatic expansion of
the Gutzwiller wave function (DE-GWF)Spałek  _et al._ (2022) which is
equivalent to the $Statistically$ $consistant$ $Gutzwiller$ $Approximation$
(SGA)Abram  _et al._ (2013). As we show here, for the considered problem the
zeroth order expansion provides a good enough approximation to reach a
qualitative agreement with fundamental experimental observations for tWSe2.
Within this method the many-body effects are captured by taking the
variational correlated wave function in the following form

$|\Psi_{G}\rangle=\hat{P}|\Psi_{0}\rangle,$ (6)

where $|\Psi_{0}\rangle$ is the non-correlated (mean-field) state to be
defined below and $\hat{P}$ is the Gutzwiller-like correlation operator of the
following form

$\hat{P}=\prod_{i}\sum_{\Gamma}\lambda_{i,\Gamma}|\Gamma\rangle_{i\;i}\langle\Gamma|,$
(7)

where $|\Gamma\rangle$ are the four states from the local basis
$|\emptyset\rangle_{i}$, $|\uparrow\rangle_{i}$, $|\downarrow\rangle_{i}$,
$|\uparrow\downarrow\rangle_{i}$ and $\lambda_{i,\Gamma}$ are the
corresponding variational parameters. The $|\Psi\rangle_{0}$ state is taken
here as the uncorrelated superconducting state which is defined by the pairing
and hopping mean-field parameters, as well as the electron occupation
expectation value

$\begin{split}S_{ij}^{\sigma\bar{\sigma}}=\langle\hat{c}_{i\sigma}\hat{c}_{j\bar{\sigma}}&\rangle_{0},\quad
P_{ij\sigma}=\langle\hat{c}^{\dagger}_{i\sigma}\hat{c}_{j\sigma}\rangle_{0},\\\
&n_{i\sigma}=\langle\hat{n}_{i\sigma}\rangle_{0},\end{split}$ (8)

respectively. In the above $\langle\hat{o}\rangle_{0}$ stands for the
expectation value of the $\hat{o}$ operator in the state $|\psi\rangle_{0}$,
and $\bar{\sigma}$ represents spin orientation opposite to $\sigma$.

Here we focus on the unconventional features of the paired state, therefore,
for simplicity we assume no magnetic or charge ordering. Hence, we take
$\lambda_{i,\Gamma}\equiv\lambda_{\Gamma}$,
$\lambda_{\uparrow}=\lambda_{\downarrow}=\lambda_{s}$,
$n_{i\uparrow}=n_{i\downarrow}\equiv n_{s}$. Due to the appearance of the
spin-orbit coupling in the model both singlet and triplet pairings are
possible which means that we need to treat $S_{ij}^{\uparrow\downarrow}$ and
$S_{ij}^{\downarrow\uparrow}$ separately allowing both for pure spin-singlet
pairing ($S_{ij}^{\uparrow\downarrow}=-S_{ij}^{\downarrow\uparrow}$), pure
spin-triplet pairing
($S_{ij}^{\uparrow\downarrow}=S_{ij}^{\downarrow\uparrow}$) as well as their
mixture. As can be shown the equal spin state (ESP) with
$S_{ij}^{\uparrow\uparrow}=S_{ij}^{\downarrow\downarrow}$ and
$S_{ij}^{\sigma\bar{\sigma}}=0$ is equivalent to the $m_{s}=0$ spin triplet
state with $S_{ij}^{\uparrow\downarrow}=S_{ij}^{\downarrow\uparrow}$ and
$S_{ij}^{\sigma\sigma}=0$ and as such is taken into account in our
analysisSmidman  _et al._ (2017).

The main task within our approach is to calculate the ground-state energy per
lattice site of the system in the correlated state within the zeroth order
diagrammatic expansion

$E_{G}=\frac{\langle\Psi_{G}|\hat{H}|\Psi_{G}\rangle}{N\langle\Psi_{G}|\Psi_{G}\rangle}=\frac{1}{N}\langle\hat{H}\rangle_{G},$
(9)

where $N$ is the number of moiré lattice sites. As we show in more detail in
Appendix A, the ground state energy in the correlated state can be expressed
as a function of the variational parameters, $\lambda_{\Gamma}$, the mean-
field parameters, $S_{ij}^{\sigma\bar{\sigma}}$ and $P_{ij\sigma}$, as well as
$n_{s}$. In order to carry out the minimization procedure effectively with the
self-consistency condition fulfilled we apply the Effective Hamiltonian
SchemeKaczmarczyk (2015), which we show in some more detail in Appendix B as
applied to the considered situation.

After determining the mean-field and the variational parameters one can
calculate the correlated superconducting gap amplitude as well as the hopping
expectation values in the correlated state

$\begin{split}\Delta^{\sigma\bar{\sigma}}_{ij}&=\langle\hat{c}_{i\sigma}\hat{c}_{j\bar{\sigma}}\rangle_{G}=q^{2}\langle\hat{c}_{i\sigma}\hat{c}_{j\bar{\sigma}}\rangle_{0},\\\
\tilde{P}_{ij\sigma}&=\langle\hat{c}^{\dagger}_{i\sigma}\hat{c}_{j\sigma}\rangle_{G}=q^{2}\langle\hat{c}^{\dagger}_{i\sigma}\hat{c}_{j\sigma}\rangle_{0},\end{split}$
(10)

where $q=\lambda_{s}(\lambda_{d}\;n_{s}+\lambda_{\emptyset}(1-n_{s}))$. Since
both the hopping and the exchange interactions are taken into account to the
nearest neighbors only, we also calculate the six nearest neighbor real space
pairing amplitudes both for $\uparrow\downarrow$ and $\downarrow\uparrow$ spin
configurations. We use the following notation

$\Delta^{\sigma\bar{\sigma}}_{\theta}=\langle\hat{c}_{i\sigma}\hat{c}_{j(\theta)\bar{\sigma}}\rangle_{G},$
(11)

where $j(\theta)$ is the nearest neighboring lattice site of site $i$, while
$\theta$ corresponds to the angle between the positive half-$x$ axis and the
$\mathbf{R}_{ij}=\mathbf{R}_{i}-\mathbf{R}_{j}$ vector. The angles
$\theta\in\\{0,\;\pi/3,\;2\pi/3,\;\pi,\;4\pi/3,\;5\pi/3\\}$ correspond to six
subsequent nearest neighbor lattice sites, $j$. Due to the homogeneity of the
system, the $\Delta^{\sigma\bar{\sigma}}_{\theta}$ gap amplitude does not
depend on the position of the $i$ lattice site. In order to identify the
symmetry of the obtained paired state for a given spin configuration we
transform the six nearest neighbor pairing amplitudes into a six symmetry
resolved pairing amplitudes of the following form

$\Delta^{\sigma\bar{\sigma}}_{M,p}=\frac{i^{p}}{6}\sum_{\theta}e^{-iM\theta}\Delta^{\sigma\bar{\sigma}}_{\theta},$
(12)

where the summation runs over the angles
$\theta\in\\{0,\;\pi/3,\;2\pi/3,\;\pi,\;4\pi/3,\;5\pi/3\\}$, and $M$ is the
symmetry factor, which takes integer values and corresponds to six possible
pairing symmetries (cf. Table 1). The $p$ parameter corresponds to the parity
of the particular symmetry. Namely, for even parity symmetries we have $p=0$
and for odd parities we have $p=1$.

Table 1: The six possible values of the symmetry factor, $M$, and the corresponding values of the parity factor $p$ (second column), appearing in Eq. (12). The corresponding six symmetries of the superconducting gap together with their parities are provided in the third and fourth columns. The Cooper pair spin state compatible with a given symmetry is provided in the fifth column. M | p | gap symmetry | parity | spin state
---|---|---|---|---
0 | 0 |  $extended$ $s$ | even | singlet
1 | 1 | $p_{x}+i\;p_{y}$ | odd | triplet
2 | 0 | $d_{x^{2}-y^{2}}+i\;d_{xy}$ | even | singlet
3 | 1 | $f$ | odd | triplet
4 | 0 | $d_{x^{2}-y^{2}}-i\;d_{xy}$ | even | singlet
5 | 1 | $p_{x}-i\;p_{y}$ | odd | triplet

Finally, the correlated gap amplitudes corresponding to subsequent symmetries
can be transformed to those corresponding to singlet and triplet spin states
of the Cooper pairs

$\begin{split}\Delta^{s}_{M,p}&=(\Delta^{\uparrow\downarrow}_{M,p}-\Delta^{\downarrow\uparrow}_{M,p})/\sqrt{2},\\\
\Delta^{t}_{M,p}&=(\Delta^{\uparrow\downarrow}_{M,p}+\Delta^{\downarrow\uparrow}_{M,p})/\sqrt{2}.\\\
\end{split}$ (13)

Only three of the considered symmetries are compatible with singlet ($d+id$-,
$d-id$-, and $extended$ $s$-$wave$) which means that they always give zero for
the triplet component ($\Delta^{t}_{M,p}=0$ for $M=0,2,4$). The remaining
$p+ip$-, $p-ip$-, and $f$-$wave$ symmetries are compatible with the spin-
triplet pairing, therefore, $\Delta^{s}_{M,p}=0$ for $M=1,3,5$. In our
calculation scheme we determine all the twelve nearest neighbor gap
amplitudes, $\Delta^{\uparrow\downarrow}_{\theta}$,
$\Delta^{\downarrow\uparrow}_{\theta}$, in the correlated state
$|\Psi_{G}\rangle$ without any constraints which would limit us only to one
selected symmetry among those provided in Table 1. Next, we transform them to
the final symmetry resolved singlet and triplet gap amplitudes by using Eqs.
(12), (LABEL:eq:gap_symmetries_spin). In such a manner we can encompass both
the pure superconducting symmetries as well as their mixtures. For the sake of
clarity, in the following Sections, we use the symmetry names ($p\pm ip$,
$d\pm id$, $f$ etc.) in the subscripts of the symmetry resolved
superconducting gaps instead of the values of the $M$ and $p$ factors.

It should be noted that within the so-called diagrammatic expansion method
(DE-GWF) one starts from the renormalized mean-field theory as the zeroth-
order result and proceed with incorporating systematically the nonlocal
correlations of increased range in higher orders. Therefore, within the higher
order calculations the correlated gaps and hopping expectation values are no
longer related with their non-correlated counterparts by a simple
renormalization given by Eqs. (10) (cf. Ref. Spałek  _et al._ , 2017). Also,
in such case the hopping and pairing between longer-range neighboring sites
needs to be included which makes the analysis much more involved. This allows
to take into account the correlation effects with better accuracy.
Nevertheless, previous calculations carried out within the SGA approach
(equivalent to the zeroth order DE-GWF) and the higher order DE-GWF method,
both has lead to the stability of the $d$-$wave$ paired state, and the dome
like behavior of the SC gap for the case of the single band $t$-$J$-$U$ model
on the square-latticeAbram  _et al._ (2013); Spałek  _et al._ (2017).
Therefore, one would expect that also here the inclusion of the higher order
terms would not change the main result of the analysis, which is related to
the symmetry of the gap and the general form of the phase diagram.

The code which was written to carry out the numerical calculations as well as
the data behind the figures presented in the following Sections are available
in the open repositoryZegrodnik (2023).

##  III Results

In the following we analyze the features of the unconventional superconducting
state for the WSe2 bilayer with the twist angle $5.08^{\circ}$ and the
displacement field $D=0.45$ V/nm for which the signatures of both the
correlated insulating state and superconductivity have been identified
experimentallyWang  _et al._ (2020). The bare band width for the corresponding
hopping parameters is $W\approx 90$ meV. It has been estimated the onsite
Coulomb repulsion integral, $U$, is comparable to $W$ for the considered
systemWang  _et al._ (2020); Pan  _et al._ (2020). Unless stated otherwise, we
take $U=120$ meV which places the system close to the moderately correlated
regime. Within the present approach we take the value of $J$ as determined
from the formula $J=4|t|^{2}/U=2.54$ meV corresponding to the mapping of the
Hubbard model to the spin Heisenberg model shown in Ref. Pan  _et al._ , 2020.

Figure 2: (a) Superconducting gap amplitudes for the $d+id$ (spin singlet) and
$p-ip$ (spin triplet) symmetries as a function of band filling $n$; (b)
Absolute value of the nearest neighbor hopping expectation value as a function
of $n$; (c) Density of states as a function of $n$. The gap amplitudes
corresponding to all four remaining symmetries (cf. Table 1) have turned out
to be zero in the considered band filling range. The interaction parameters of
the model are set to $U=120$ meV, $J=2.54$ meV.

As one can see in Fig. 2, for the chosen model parameters the symmetry
resolved gap amplitudes create two superconducting domes residing on both
sides of the half-filled situation ($n=1$). Here, we define the band filling
as the number of electrons in the narrow band per single moiré lattice site,
$n=2n_{s}$. According to our calculations the paired state has a mixed
singlet-triplet character with comparable magnitudes of both components of the
pairing. The singlet (triplet) pairing component is realized by the $d+id$
($p-ip$) symmetry of the superconducting gap. Apart from the two gap
amplitudes shown in the Figure, all the remaining ones listed in Table 1 have
turned out to be zero for the given range of model parameters. One should note
that pairing in the mixed $d/p$ channel has been proposed to appear due to
density waves fluctuations very recently according to functional
renormalization group calculationsKlebl  _et al._ (2022). Also, the
singlet/triplet pairing has been analyzed within the $t$-$J$ model showing the
dominant singlet $d$-wave contributionChen and Sheng (2023). The singlet-
triplet mixing obtained here is a consequence of spin-orbit coupling, due to
which parity is no longer a good quantum numberSmidman  _et al._ (2017). Among
the possible pairing symmetries the $d\pm id$ and $p\pm ip$ optimize the
condensation energy due to the fact that they provide a fully gapped situation
in contradiction to the $extended$ $s$-$wave$ and $f$-$wave$ symmetries, for
which the line nodes would cross the Fermi surface, in the considered doping
range (cf. Appendix C).

In Fig. 2 (b) we show that the nearest neighbor hopping expectation value
$\tilde{P}$ is being significantly suppressed as one approaches the half-
filled situation ($n=1$) from both sides, which indicates the appearance of
the correlation induced insulating state (Mott insulator). The insulating
behavior suppresses superconductivity in the close proximity of $n=1$ creating
the double-dome structure seen in experimentsWang  _et al._ (2020). The
resulting two SC domes are not symmetric with respect to half-filling, which
is caused by the appearance of the van Hove singularity for $n\approx 0.8$,
which in turn enhances the pairing at the hole-doped side of the phase diagram
[cf. Fig. 2 (c)]. One should note that the approach based on the onsite
Gutzwiller-like correlator results in a trivial character of the Mott
transition. For the complete description of this effect one should take into
account more involved Jastrov and/or doublon-holon correlatorsCapello  _et
al._ (2005); Miyagawa and Yokoyama (2011).

Figure 3: The single particle energy $\langle\hat{H}_{0}\rangle/N$ (a), the
Coulomb interaction energy $\langle\hat{H}_{U}\rangle/N$ (b), the exchange
interaction energy $\langle\hat{H}_{J}\rangle/N$ (c), and the total energy of
the system $\langle\hat{H}\rangle/N$ (d) all as functions of band filling $n$
for the case of Hartree-Fock and Gutzwiller approach calculations. The results
have been obtained for the same model parameters as those corresponding to
Fig. 2.

In Fig. 3 we show the subsequent contributions to the total system energy as
well as their sum, all as functions of band filling as obtained from the
Gutzwiller approach and the Hartree-Fock method. The former takes into account
the correlation effects induced by the Hubbard $U$ while the latter neglects
them. As one can see the two methods deviate significantly close to half-
filling where the correlations are most pronounced. Again, within the
Gutzwiller approach one can see the signature of the insulating behaviour for
$n=1$ where the kinetic energy approaches zero, together with the Coulomb
energy as both the electron hopping and the double occupancies are being
suppressed. It should be noted that the appearance of the insulating state is
concomitant with a significant decrease of the exchange interaction energy as
shown in Fig. 3 (b). This effect is caused by the fact that the $\lambda_{s}$
parameter, over which the expectation value of the $\sim J$ term is
premultiplied [cf. Eq. (17)], increases close to half filling as shown in Fig.
6 (a) in Appendix A. One may understand this as an electron correlation
induced increase of the renormalized exchange interaction integral
$J^{\prime}=\lambda_{s}^{4}\;J$. As shown in Fig. 3 (d), the Gutzwiller method
gives total energy lower than HF as it should be since the former is of better
quality.

The appearance of the mixed singlet-triplet pairing seen in Fig. 2 is a
consequence of the spin-orbit coupled hoppings and the resulting spin-
splitting of the Fermi surface. It should be noted that with the use of the
displacement field one can tune the phase of the hoppings, $\phi$ [cf. Fig. 1
(b)], which in turn determines the magnitude of the spin splitting as well as
the strength of the DM interaction [cf Eq. 4]. The influence of the DM
interaction on the paired state is briefly discussed in Appendix D. Since the
spin-splitting plays an important role when it comes to the appearance of the
mixed spin-triplet pairing, it is tempting to check if the balance between the
singlet and triplet contributions can be modified by changing $\phi$. In Fig.
4 we show the $\phi$-dependance of the two SC gaps, $\Delta^{s}_{d+id}$
(singlet) and $\Delta^{t}_{p-ip}$ (triplet), for two selected band fillings.
For the sake of simplicity in these calculations we keep $|t|$ constant and
change only the phase of the hopping. As one can see by changing
$\phi=\pi\rightarrow\pi/2\rightarrow 0$, we also carry out the transition
between singlet$\rightarrow$triplet$\rightarrow$singlet pairing, respectively.
Purely real (imaginary) hopping values result in purely singlet (triplet)
pairing. In between those three distinguished situations a mixed singlet-
triplet state appears. The presented data indicate the possibility of tuning
the balance between the singlet and triplet contributions by changing the
displacement field. The important question is if it is possible to tune the
gate voltage in a real experimental setup so as to reach the displacement
field values for which $\phi=\pi/2$ in order to generate pure spin-triplet
pairing. This would allow for switching between singlet and triplet pairing in
an $in$ $situ$ manner by simply adjusting the top and bottom gate voltage.

Figure 4: (a) Superconducting gap amplitudes for the $d+id$ (spin singlet) and
$p-ip$ (spin triplet) symmetries as a function of the phase of the hopping
parameters (cf. Eq. 3). For the sake of simplicity here we keep the hopping
absolute value, $|t|$, constant. (a) and (b) correspond to two different band
fillings specified in the Figure.

As the last part of our analysis we consider the calculated phase diagram in
the $(U,n)$ plane. It can be seen from Fig. 5, that large enough $U$ is
crucial in order to induce the paired state in the system. As the $U$ value is
increased, first, a weak superconducting state appears in the range of band
fillings which correspond to the van Hove singularity ($n\approx 0.8$) where
the high values of density of states create favorable conditions for the
pairing. When $U$ approaches the value of $W\approx 90$ meV, the pairing is
strengthen around half-filling. This effect is caused by the exchange
interaction enhancement which is due to $\lambda_{s}>1$, as discussed earlier
(cf. Eq. 17). For $U\gtrsim W$, the SC state stability range becomes divided
and two SC domes are created which is a direct result of the insulating state
appearance at $n=1$. It should be noted that even though the electronic
structure of the model and the features of the superconducting state obtained
here are significantly different from those corresponding the copper based
superconductors there are some clear similarities between the two systems.
Namely, in both systems the two superconducting domes divided by a correlated
insulating state appear. With this respect, the structure of the $(U,n)$-phase
diagram calculated here resembles the one obtained by us previously for the
case of a single-band model as applied for the description of the
cupratesSpałek  _et al._ (2017); Zegrodnik and Spałek (2017). However, it
should be noted that also other approaches have been reported in recent years
which represent different methods and points of view to the problem of pairing
in single band models of copper based superconductorsGunnarsson  _et al._
(2015); Qin  _et al._ (2020); Dong  _et al._ (2022a, b). In particular, the
approach presented here does not take into account the dynamical screening
effects which leads to frequency dependent $U$ parameter and may affect the
strength of the pairing as well as the resulting critical temperatureNilsson
_et al._ (2019).

Figure 5: (a) Correlated superconducting gap amplitudes for the $d+id$ (spin
singlet) and $p-ip$ (spin triplet) symmetries as a function the Coulomb
repulsion integral, $U$, and the band filling $n$ for $J=2.54$ meV. For the
chosen model parameters the bare band width is $W\approx 90$ meV. Note that
for $U\gtrsim W$ the two SC domes are created in the phase diagram. The narrow
white region is where problems with convergence of the numerical procedure
appeared.

##  IV Conclusions

We have applied the moiré $t$-$J$-$U$ model to the description of
unconventional superconducting state in tWSe2. In order to take into account
the electron-electron correlations we have used the Gutzwiller method. As we
show our approach leads to the reconstruction of two superconducting domes
residing on both sides of the correlation induced insulating state appearing
at half-filling, which is in agreement with the experimental data shown in
Ref. Wang  _et al._ , 2020. A similar structure of the phase diagram as the
one obtained here has also been discussed by us recently in a quantum-dot
square lattice system with the use of the variational Monte Carlo
methodBiborski  _et al._ (2021). According to our analysis the suppression of
the electron hopping processes near the half-filled situation is concomitant
with the enhanced role of the exchange interaction term as the corresponding
renormalization factor becomes larger than unity, $\lambda_{s}>1$ (cf. Fig. 6
and Eq. 17). This effect also enhances the superconducting phase around the
$n=1$ case, however, for large enough value of the Coulomb repulsion integral,
$U\gtrsim W$, the appearance of the Mott insulating behaviour separates the SC
state stability regime into two superconduting domes (cf. Fig. 5). It is worth
mentioning that it was proposed very recently within a two-orbital model that
the Mott-Hubbard behavior in tWSe2 can be switched into Hund correlated state
by tuning the twist angleRyee and Wehling (2023). Such effect is not taken
into account here as it requires a two-band extension of the considered model.

The character of the paired state obtained here is highly unconventional with
a mixed spin singlet-spin triplet pairing and non-zero values of the the
corresponding $d+id$ and $p-ip$ components of the gap symmetry, respectively.
As we show the balance between the singlet and triplet contributions can be
tuned by adjusting the gate voltage which determines the valley-dependent spin
splitting of the electronic structure (cf. Fig. 4). In particular, if the
hopping phase $\phi=\pi/2$ can be reached experimentally, the transition
between pure spin-singlet and pure spin-triplet pairing can be realized in the
system.

It should be noted that various magnetic and charge ordered states have been
analyzed by a number of methods within the Hubbard and Heisenberg models as
applied to the description of tWSe2Zang  _et al._ (2021); Bélanger  _et al._
(2022); Zang  _et al._ (2022); Pan  _et al._ (2020); Pan and Das Sarma (2022);
Biderang  _et al._ (2022); Kiese  _et al._ (2022). Furthermore, the pair-
density-wave state has been shown to appear in weak out-of-plane Zeeman
fieldWu  _et al._ (2022). It is expected that some of those symmetry-broken
states may also appear in the present $t$-$J$-$U$ based approach. In
particular, in analogy with the cuprates one would expect that close to half-
filling antiferromagnetic phase should become stable. As argued in Refs. Pan
_et al._ , 2020; Zang  _et al._ , 2021 the DM interaction acts as an
anisotropy that favors in-plane spin ordering and should select the subset of
$120^{\circ}$ antiferromagnetic states. In the considered model the DM
interaction is affected by the hopping phase [cf. Eq. 4], which in turn can be
tuned by the displacement field. Therefore, for particular range of the
displacement fields also a tendency towards ferromagnetism can appear. It
would be interesting to analyze the interplay between the magnetic/charge
ordering and the mixed superconducting pairing considered here. In particular,
the appearance of charge density waves may lead to modulation of the pairing
amplitudes in space as has been shown by us in recent years for the analogical
approach as applied to cupratesZegrodnik and Spałek (2018). Such study
requires much more involved calculations and is beyond the scope of the
present analysis. We should be see progress along this line soon.

Finally, the spin-triplet $p\pm ip$ paired state is interesting when it comes
to its topological features. Namely, if a gate-defined nanowire structure
based on tWSe2 could be realized, the $p\pm ip$ pairing could give rise to
Majorana bound states at the edges of such nanowire. What is very attractive
in such concept is that all the ingredients required for such effect can
appear in a single material. This is in opposition to other considered systems
in which subsequent factors like spin-orbit coupling or superconducting
pairing are introduced with the use of proximity effect in a hybrid systemOreg
_et al._ (2010); Lutchyn  _et al._ (2010).

##  V Acknowledgement

Discussions with P. Wójcik are gratefully acknowledged. This research was
founded by National Science Centre, Poland (NCN) according to decision
2021/42/E/ST3/00128. For the purpose of Open Access, the author has applied a
CC-BY public copyright licence to any Author Accepted Manuscript (AAM) version
arising from this submission.

##  Appendix A Expectation value of the $t$-$J$-$U$ Hamiltonian in the
correlated state

Here we show how to express the expectation value of our $t$-$J$-$U$
Hamiltonian in the variational correlated state defined by Eq. (6) within the
zeroth order diagrammatic expansion (DE-GWF method). As shown in Ref. Bünemann
_et al._ , 2012, it is convenient to impose the following condition on the
correlation operator

$\hat{P}^{2}_{i}=1+x\hat{d}_{i}^{HF},$ (14)

where $\hat{d}^{HF}_{i}=\hat{n}^{HF}_{i\uparrow}\hat{n}^{HF}_{i\downarrow}$,
$\hat{n}^{HF}_{i\sigma}=\hat{n}_{i\sigma}-n_{s}$, and $x$ is yet another
variational parameter. One can express the variational parameters
$\lambda_{\Gamma}$ as a function of $x$ in the following manner

$\begin{split}\lambda^{2}_{d}&=1+x(1-n_{s})^{2},\\\
\lambda^{2}_{s}&=1-xn_{s}(1-n_{s})^{2},\\\
\lambda^{2}_{\emptyset}&=1+xn_{s}^{2},\end{split}$ (15)

therefore, in fact we are left with only one variational parameter. In such
case the expectation values of subsequent terms of our Hamiltonian take the
form

$\langle\hat{H}_{0}\rangle_{G}=\sum_{ij}q^{2}t_{ij\sigma}\langle\hat{c}^{\dagger}_{i\sigma}\hat{c}_{j\sigma}\rangle_{0},$
(16)

$\begin{split}\langle\hat{H}_{J}\rangle_{G}&=\lambda^{4}_{s}\;J\sum_{<ij>}\bigg{(}\frac{1}{2}\;\sum_{\sigma}\;e^{i\sigma
2\phi_{ij}}\langle\hat{c}^{\dagger}_{i\sigma}\hat{c}_{i\bar{\sigma}}\hat{c}^{\dagger}_{j\bar{\sigma}}\hat{c}_{j\sigma}\rangle_{0}\\\
&+\frac{1}{4}\;\sum_{\sigma\sigma^{\prime}}\sigma\sigma^{\prime}\langle\hat{n}^{HF}_{i\sigma}\hat{n}^{HF}_{j\sigma^{\prime}}\rangle_{0}\bigg{)},\end{split}$
(17)

$\langle\hat{H}_{U}\rangle=N\lambda_{d}^{2}\;U\;n_{s}^{2},$ (18)

where
$q=\lambda_{s}\big{(}\lambda_{d}n_{s}+\lambda_{\emptyset}(1-n_{s})\big{)}$.
Since all the four-operator expectation values which appear on the right-hand
side of Eq. (17) are in the non-correlated state, one can carry out the
standard Wick’s theorem decomposition in order to express them in terms of the
mean-field parameters $S_{ij}^{\sigma\bar{\sigma}}$, $P_{ij\sigma}$. For the
sake of clarity in Fig. 6 we provide the band filling dependence of the
renormalization factor $q$ and the variational parameters $\lambda_{s}$,
$\lambda_{d}$, and $\lambda_{\emptyset}$, appearing in expressions (16-18). It
should be noted that the Hartree-Fock calculations are equaivalent to the
situation for which $\lambda_{\Gamma}\equiv 1$, $q\equiv 1$.

Figure 6: The renoralization factor $q$ and the variational parameters
$\lambda_{s}$, $\lambda_{d}$, and $\lambda_{\emptyset}$, as functions of the
band filling, $n$, for the same model parameters as those corresponding to
Fig. 2.

##  Appendix B The effective Hamiltonian scheme

From the minimization condition of the ground state energy (9) one can derive
the effective Hamiltonian, which for the case of superconducting phase has the
form

$\begin{split}\hat{\mathcal{H}}_{\textrm{eff}}&=\sum_{ij\sigma}t^{\textrm{eff}}_{ij\sigma}\hat{c}^{\dagger}_{i\sigma}\hat{c}_{j\sigma}-\mu^{\textrm{eff}}\sum_{i\sigma}\hat{n}_{i\sigma}\\\
&+\sideset{}{{}^{\prime}}{\sum}_{<ij>\sigma}\big{(}\Delta^{\textrm{eff}}_{ij\sigma\bar{\sigma}}\hat{c}_{j\sigma}\hat{c}_{i\bar{\sigma}}+H.c.\big{)},\end{split}$
(19)

where the effective hopping, effective chemical potential, and the effective
superconducting gap parameters are defined through the corresponding relations

$t^{\textrm{eff}}_{ij\sigma}\equiv\frac{\partial\mathcal{F}}{\partial
P_{ij\sigma}},\quad(\Delta^{\textrm{eff}}_{ij\sigma\bar{\sigma}})^{*}\equiv\frac{\partial\mathcal{F}}{\partial
S^{\sigma\bar{\sigma}}_{ji}},\quad\mu^{\mathrm{eff}}\equiv-\frac{\partial\mathcal{F}}{\partial
n_{s}},$ (20)

where $\mathcal{F}=E_{G}-\mu_{G}n_{s}$ with $\mu_{G}$ being the chemical
potential determined in the correlated state. By using Eqs. (16-18) one can
express the ground state energy, $E_{G}$ in the correlated state with the use
of the mean-field parameters, variational parameters, as well as the band
filling. This allows to calculate the values of the effective parameters from
Eq. (20). Next, the effective Hamiltonian can be diagonalyzed by introducing
the generalized Bogolubov de Gennes transformation and the self-consistent
equations for $P_{ij\sigma}$, $S^{\sigma\bar{\sigma}}_{ij}$, $\mu_{G}$ can be
derived in a standard manner. Additionally, the procedure of solving the self-
consistent equations has to be supplemented with the minimization of the
ground state energy with respect to the variational parameter $x$.

##  Appendix C Gap symmetry in the $\mathbf{k}$-space

Here we show explicitly the $\mathbf{k}$-dependency of the superconducting gap
for selected pairing symmetries. It should be noted that in order to obtain
the $\mathbf{k}$-dependant superconducting gaps one can transform the real-
space pairing amplitudes given by equations (11) in the following manner

$\Delta^{\sigma\sigma^{\prime}}(\mathbf{k})=\sum_{\theta}\Delta^{\sigma\sigma^{\prime}}_{\theta}e^{i\mathbf{k}\mathbf{R}_{ij(\theta)}},$
(21)

where $j(\theta)$ is the nearest neighboring lattice site of site $i$, while
$\theta$ corresponds to the angle between the positive half-$x$ axis and the
$\mathbf{R}_{ij}=\mathbf{R}_{i}-\mathbf{R}_{j}$ vector. The summation is over
the angles $\theta\in\\{0,\;\pi/3,\;2\pi/3,\;\pi,\;4\pi/3,\;5\pi/3\\}$, which
correspond to six subsequent nearest neighbor lattice sites, $j$. In analogy
to the real space gap amplitudes [cf. Eq. (LABEL:eq:gap_symmetries_spin)],
also here we can define the singlet and triplet superconducting gaps in
$\mathbf{k}$-space.

$\begin{split}\Delta^{s}(\mathbf{k})&=\big{(}\Delta^{\uparrow\downarrow}(\mathbf{k})-\Delta^{\downarrow\uparrow}(\mathbf{k})\big{)}/\sqrt{2}\\\
\Delta^{t}(\mathbf{k})&=\big{(}\Delta^{\uparrow\downarrow}(\mathbf{k})+\Delta^{\downarrow\uparrow}(\mathbf{k})\big{)}/\sqrt{2}\end{split}$
(22)

Figure 7: The absolute value of the superconducting gaps for spin-singlet
symmetries ($d\pm id$-wave and $extended$ $s$-wave) and spin-triplet
symmetries ($p\pm ip$-wave and $f$-wave), all as functions of momentum,
$\mathbf{k}$. In (e) and (f) we show the spin up and down dispersion relations
and the Fermi surfaces for exemplary value of the band filling $n=0.9$. Note
that the $d\pm id$ and $p\pm ip$ pairing symmetries provide significantly
larger values of the superconducting gaps at the Fermi surface than the
$exteded$ $s$\- and $f$-$wave$ pairings.

In Fig. 7 we show the absolute value of the superconducting gaps for spin-
singlet symmetries ($d\pm id$-$wave$ and $extended$ $s$-$wave$) and two spin-
triplet symmetries ($p\pm ip$-$wave$ and $f$-$wave$), all as functions of
momentum, $\mathbf{k}$. Additionally in Fig. 7 (f) we show the Fermi surface
for one exemplary band filling value, $n=0.9$, for which the mixture of $d+id$
and $p-ip$ pairing has been shown to be stable in the considered model (cf.
Fig. 2).

As one can see from the figures, the $d\pm id$ and $p\pm ip$ pairing
symmetries provide a fully gapped situation at the Fermi surface in
contradiction to the $extended$ $s$-$wave$ and $f$-$wave$ symmetries. This
leads to the optimization of the condensation energy for the case of the $d\pm
id$ and $p\pm ip$ mixing.

##  Appendix D The effect of DM interaction on the paired state

For the sake of completeness, in Fig. 8 we show the results obtained for the
case when the spin-splitting is still present in the electronic structure,
however, the exchange interaction term is taken in the standard form without
the DM term. Namely,

$\hat{H}_{J}=J\sum_{ij}\mathbf{\hat{S}}_{i}\mathbf{\hat{S}}_{j}.$ (23)

As one can see in such case the triplet component is still present, however it
is significantly smaller with respect to the results presented in Fig. 2 where
the DM term has been taken into account.

Figure 8: (a) Superconducting gap amplitudes for the $d+id$ (spin singlet) and
$p-ip$ (spin triplet) symmetries as a function of band filling $n$; (b)
Absolute value of the nearest neighbor hopping amplitude as a function of $n$.
Here, the $\sim J$ term is taken in the form given by Eq. 23. The model
parameters are the same as those corresponding to Fig. 2.

## References

  * Cao  _et al._ (2018a) Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, R. C. Ashoori, and P. Jarillo-Herrero, Nature 556, 80 (2018a).
  * Cao  _et al._ (2018b) Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Nature 556, 43 (2018b).
  * Xian  _et al._ (2019) L. Xian, D. M. Kennes, N. Tancogne-Dejean, M. Altarelli, and A. Rubio, Nano Lett. 19, 4934 (2019).
  * Zhang  _et al._ (2021) X. Zhang, K.-T. Tsai, Z. Zhu, W. Ren, Y. Luo, S. Carr, M. Luskin, E. Kaxiras, and K. Wang, Phys. Rev. Lett. 127, 166802 (2021).
  * Chen  _et al._ (2019) G. Chen, A. L. Sharpe, P. Gallagher, E. J. Rosen, Ilan T. Fox, L. Jiang, B. Lyu, H. Li, K. Watanabe, T. Taniguchi, J. Jung, Z. Shi, D. Goldhaber-Gordon, Y. Zhang, and F. Wand, Nature 572, 215 (2019).
  * Shen  _et al._ (2020) C. Shen, Y. Chu, Q. Wu, N. Li, S. Wang, Y. Zhao, J. Tang, J. Liu, J. Tian, K. Watanabe, T. Taniguchi, R. Yang, Z. Y. Meng, D. Shi, O. V. Yazyev, and G. Zang, Nature Physics 16, 520 (2020).
  * Wang  _et al._ (2020) L. Wang, E.-M. Shih, A. Ghiotto, L. Xian, D. A. Rhodes, C. Tan, M. Claassen, D. M. Kennes, Y. Bai, B. Kim, K. Watanabe, T. Taniguchi, X. Zhu, J. Hone, A. Rubio, and C. R. Pasupathy, Abhay N. amd Dean, Nature Materials 19, 861 (2020).
  * Liao  _et al._ (2020) M. Liao, Z. Wei, L. Du, Q. Wang, J. Tang, H. Yu, F. Wu, J. Zhao, X. Xu, B. Han, K. Liu, P. Gao, T. Polcar, Z. Sun, D. Shi, R. Yang, and G. Zhang, Nature Communications 11, 6104 (2020).
  * Ulstrup  _et al._ (2020) S. Ulstrup, R. J. Koch, S. Singh, K. M. McCreary, B. T. Jonker, J. T. Robinson, C. Jozwiak, E. Rotenberg, A. Bostwick, J. Katoch, and J. A. Miwa, Science Advances 6 (2020), https://doi.org/10.1126/sciadv.aay6104.
  * Ghiotto  _et al._ (2021) A. Ghiotto, E.-M. Shih, G. S. S. G. Pereira, D. A. Rhodes, J. Kim, Bumho Zang, A. J. Millis, K. Watanabe, T. Taniguchi, J. C. Hone, L. Wang, C. R. Dean, and A. N. Pasupathy, Nature 597, 345 (2021).
  * Spałek  _et al._ (2022) J. Spałek, M. Fidrysiak, M. Zegrodnik, and A. Biborski, Physics Reports 959, 1 (2022).
  * An  _et al._ (2020) L. An, X. Cai, D. Pei, M. Huang, Z. Wu, Z. Zhou, J. Lin, Z. Ying, Z. Ye, X. Feng, R. Gao, C. Cacho, M. Watson, Y. Chen, and N. Wang, Nanoscale Horiz. 5, 1309 (2020).
  * Pan  _et al._ (2020) H. Pan, F. Wu, and S. Das Sarma, Phys. Rev. Res. 2, 033087 (2020).
  * Zang  _et al._ (2021) J. Zang, J. Wang, J. Cano, and A. J. Millis, Phys. Rev. B 104, 075150 (2021).
  * Bélanger  _et al._ (2022) M. Bélanger, J. Fournier, and D. Sénéchal, Phys. Rev. B 106, 235135 (2022).
  * Zang  _et al._ (2022) J. Zang, J. Wang, J. Cano, A. Georges, and A. J. Millis, Phys. Rev. X 12, 021064 (2022).
  * Pan and Das Sarma (2022) H. Pan and S. Das Sarma, Phys. Rev. B 105, 041109 (2022).
  * Wu  _et al._ (2018) F. Wu, T. Lovorn, E. Tutuc, and A. H. MacDonald, Phys. Rev. Lett. 121, 026402 (2018).
  * and Sheng (2022) Y. and D. N. Sheng, Phys. Rev. X 12, 031009 (2022).
  * Chen and Sheng (2023) F. Chen and D. Sheng, “Singlet, triplet and pair density wave superconductivity in the doped triangular-lattice moiré system,”  (2023).
  * Daul  _et al._ (2000) S. Daul, D. J. Scalapino, and S. R. White, Phys. Rev. Lett. 84, 4188 (2000).
  * Basu  _et al._ (2001) S. Basu, R. J. Gooding, and P. W. Leung, Phys. Rev. B 63, 100506 (2001).
  * Xiang  _et al._ (2009) T. Xiang, H. G. Luo, D. H. Lu, K. M. Shen, and Z. X. Shen, Phys. Rev. B 79, 014524 (2009).
  * Spałek  _et al._ (2017) J. Spałek, M. Zegrodnik, and J. Kaczmarczyk, Phys. Rev. B 95, 024506 (2017).
  * Abram  _et al._ (2013) M. Abram, J. Kaczmarczyk, J. Jędrak, and J. Spałek, Phys. Rev. B 88, 094502 (2013).
  * Smidman  _et al._ (2017) M. Smidman, M. B. Salamon, H. Q. Yuan, and D. F. Agterberg, Reports on Progress in Physics 80, 036501 (2017).
  * Kaczmarczyk (2015) J. Kaczmarczyk, Philosophical Magazine 95, 563 (2015), https://doi.org/10.1080/14786435.2014.965235 .
  * Zegrodnik (2023) M. Zegrodnik, “Unconventional superconductivity in twisted bilayer wse2: single band t-j-u model [data set]. zenodo. https://doi.org/10.5281/zenodo.8081872,”  (2023).
  * Klebl  _et al._ (2022) L. Klebl, A. Fischer, L. Classen, M. M. Scherer, and D. M. Kennes, “Competition of density waves and superconductivity in twisted tungsten diselenide,”  (2022).
  * Capello  _et al._ (2005) M. Capello, F. Becca, M. Fabrizio, S. Sorella, and E. Tosatti, Phys. Rev. Lett. 94, 026406 (2005).
  * Miyagawa and Yokoyama (2011) T. Miyagawa and H. Yokoyama, Journal of the Physical Society of Japan 80, 084705 (2011), https://doi.org/10.1143/JPSJ.80.084705 .
  * Zegrodnik and Spałek (2017) M. Zegrodnik and J. Spałek, Phys. Rev. B 96, 054511 (2017).
  * Gunnarsson  _et al._ (2015) O. Gunnarsson, T. Schäfer, J. P. F. LeBlanc, E. Gull, J. Merino, G. Sangiovanni, G. Rohringer, and A. Toschi, Phys. Rev. Lett. 114, 236402 (2015).
  * Qin  _et al._ (2020) M. Qin, C.-M. Chung, H. Shi, E. Vitali, C. Hubig, U. Schollwöck, S. R. White, and S. Zhang (Simons Collaboration on the Many-Electron Problem), Phys. Rev. X 10, 031016 (2020).
  * Dong  _et al._ (2022a) X. Dong, E. Gull, and A. J. Millis, Nat. Phys. 18, 1293 (2022a).
  * Dong  _et al._ (2022b) X. Dong, L. Del Re, A. Toschi, and E. Gull, PNAS 119, e2205048119 (2022b).
  * Nilsson  _et al._ (2019) F. Nilsson, K. Karlsson, and F. Aryasetiawan, Phys. Rev. B 99, 075135 (2019).
  * Biborski  _et al._ (2021) A. Biborski, M. P. Nowak, and M. Zegrodnik, Phys. Rev. B 104, 245430 (2021).
  * Ryee and Wehling (2023) S. Ryee and T. O. Wehling, Nano LEtt. 23, 573 (2023).
  * Biderang  _et al._ (2022) M. Biderang, M.-H. Zare, and J. Sirker, Phys. Rev. B 105, 064504 (2022).
  * Kiese  _et al._ (2022) D. Kiese, Y. He, C. Hickey, A. Rubio, and D. M. Kennes, APL Materials 10, 031113 (2022).
  * Wu  _et al._ (2022) Y.-M. Wu, Z. Wu, and H. Yao, “Pair-density-wave and chiral superconductivity in twisted bilayer transition-metal-dichalcogenides,”  (2022).
  * Zegrodnik and Spałek (2018) M. Zegrodnik and J. Spałek, Phys. Rev. B 98, 155144 (2018).
  * Oreg  _et al._ (2010) Y. Oreg, G. Refael, and F. von Oppen, Phys. Rev. Lett. 105, 177002 (2010).
  * Lutchyn  _et al._ (2010) R. M. Lutchyn, J. D. Sau, and S. Das Sarma, Phys. Rev. Lett. 105, 077001 (2010).
  * Bünemann  _et al._ (2012) J. Bünemann, T. Schickling, and F. Gebhard, Europhysics Letters 98, 27006 (2012).