# Table-based Fact Verification with Self-adaptive
Mixture of Experts

Yuxuan Zhou1, Xien Liu1, Kaiyin Zhou2,3, Ji Wu1
1Department of Electronic Engineering, Tsinghua University, Beijing 100084,
China
2THiFLY Research, Tsinghua University, Beijing 100084, China
3State Key Laboratory of Cognitive Intelligence, Hefei, Anhui 230088, China

###### Abstract

The table-based fact verification task has recently gained widespread
attention and yet remains to be a very challenging problem. It inherently
requires informative reasoning over natural language together with different
numerical and logical reasoning on tables (e.g., count, superlative,
comparative). Considering that, we exploit mixture-of-experts and present in
this paper a new method: Self-adaptive Mixture-of-Experts Network (SaMoE).
Specifically, we have developed a mixture-of-experts neural network to
recognize and execute different types of reasoning—the network is composed of
multiple experts, each handling a specific part of the semantics for
reasoning, whereas a management module is applied to decide the contribution
of each expert network to the verification result. A self-adaptive method is
developed to teach the management module combining results of different
experts more efficiently without external knowledge. The experimental results
illustrate that our framework achieves 85.1% accuracy on the benchmark dataset
TabFact, comparable with the previous state-of-the-art models. We hope our
framework can serve as a new baseline for table-based verification. Our code
is available at https://github.com/THUMLP/SaMoE.

##  1 Introduction

Fact Verification, aiming to determine the consistency between a statement and
given evidence, has become a crucial part of various applications such as fake
news detection, rumor detection (Rashkin et al., 2017; Thorne et al., 2018;
Goodrich et al., 2019; Vaibhav et al., 2019; Kryscinski et al., 2020). While
most existing research focuses on verification based on unstructured text, a
new trend is extending the scope to structured evidence (e.g., tables), which
is informative and ubiquitous in our daily lives. Table-based verification
faces different challenges than unstructured-text-based due to the complexity
of the requirements, including sophisticated textual, numerical, and logical
reasoning across evidence tables; even for some statements, multiple types of
reasoning are indispensable to complete the verification. An example is
presented in Figure 1.

Figure 1: An Example of table-based fact verification.

To tackle the challenges above, previous work established two kinds of
methods: (1) program-enhanced methods (Chen et al., 2020; Zhong et al., 2020;
Shi et al., 2020; Yang et al., 2020) and (2) table-based pre-trained models
(Eisenschlos et al., 2020; Liu et al., 2021). The program-enhanced methods
mainly leverage programs generated by the semantic parser. Specifically,
statements are parsed into executable programs to extract the
logical/numerical semantics, which is further be leveraged together with
contextual semantics learned by a language model (e.g., BERT) in inference.
However, the semantic parsers that generate semantic-consistent programs must
be trained in a weak supervision setting, which brings difficulties in
training. Furthermore, generalizing this method to other datasets is almost
impossible without the API set modification according to the reasoning
requirements on the new datasets.

Figure 2: An overview of Self-adaptive Mixture-of-Experts Network (SaMoE) for
table-based fact verification.

The table-based pre-trained models leverage elaborate model structure (Herzig
et al., 2020) and pre-training tasks (Eisenschlos et al., 2020; Liu et al.,
2021) to enhance the reasoning skills on structured data. Nevertheless, two
significant shortcomings remain. Firstly, the process is demanding due to the
tremendous computing resources required by pre-training. Moreover, the
effectiveness of pre-training to its downstream tasks mainly depends on the
adaptability between these two tasks. Therefore, implementing pre-training
tasks may fail to meet the requirements when facing the unseen reasoning types
demanded by new datasets.

In this paper, we introduce an innovative framework, Self-adaptive Mixture of
Experts (SaMoE), to address the previously mentioned problems. The entire
framework is illustrated in Figure 2. SaMoE consists of 3 components: feature
extractor, experts, and management module, which is the combination of manager
and supervisor networks. Each expert initially takes the same feature as input
and then learns to deal with different parts of the reasoning types (e.g.,
contextual/logical/numerical) required by table-based verification. A
management module is designed to guide the training of experts and combine
experts’ verification results effectively. The manager network in this module
assigns each expert a unique attention score, allowing each individual to
focus on different kinds of reasoning types and summarizes experts’ entire
outputs as the final verification result. However, managers failed to allocate
the highest attention score to the expert who performs best on the current
reasoning type in most circumstances. To alleviate this problem, we introduce
a supervisor network to adjust the attention score given by the manager. The
supervisor network is trained self-adaptively (i.e., it learns directly from
experts’ performance on the train set) without prior knowledge of the task or
dataset. Extensive experiments are conducted to show that our proposed
framework, implemented with a general pre-trained language model RoBERTa (Liu
et al., 2019), outperforms previous state-of-the-art methods, including table-
based pre-trained models. The main contributions of this work are as follows:

  * •

We innovatively implement mixture-of-experts for table-based verification,
aiming to arrange each expert to different types of reasoning. This method can
also be easily generalized to other datasets.

  * •

We investigate a self-adaptive method to adjust suitable attention score to
each expert according to its performance on different reasoning types,
achieving more efficient cooperation across experts.

  * •

Our framework achieves better performance on the TabFact dataset without the
assistance of table-based pre-trained models.

##  2 Research Question

The table-based verification task expects one to determine whether a statement
$S$ is entailed or refuted by an evidence table $T$. The process above can be
regarded as a binary classification task and thus denoted as $f(S,T)=\hat{y}$,
where $f$ is the verification model and $\hat{y}\in\\{0,1\\}$ its prediction.

##  3 Methods

We present the proposed framework (SaMoE), which leverages a set of experts to
deal with different parts of the reasoning types involved in table-based
verification. This section is organized as follows. Sec.3.1 introduces the
feature extractor that extracts the joint semantics of the table-statement
pair. Sec.3.2 describes experts that verify the statements separately based on
the same extracted semantics. Sec.3.3 describes the management module that
guides the experts’ training and combines their verification results
effectively; two components of this module, the manager and the supervisor,
are introduced in this section individually.

###  3.1 Feature Extractor

Feature extractor parses the statement-table pair and learns the joint table-
statement semantics. Tables are initially pruned and serialized into a
sequence. Subsequently, the serialized tables are transmitted into the
language model together with the statements for joint representation learning.
These two processes will be further interpreted in the following subsections.

####  3.1.1 Table Pre-processing

As for Tables, the pre-processing (pruning and serializing) before the joint
representation learning provides convenience for subsequent processing of the
existing language model.

##### Table Pruning

Table pruning discards some parts of the table that do not participate in the
verification, according to the input size limit of the language model. We take
advantage of the table-pruning algorithm proposed in Chen et al. (2020) and
further enhance its performance. The original algorithm matches the entities
in statements with cells in tables by a heuristic method and selects the
columns that include matched cells to form the pruned table. Noticed that the
algorithm always fails to select the critical columns of verification while
there is still room left for the input sequence of the language model, we
further add a greedy strategy on the algorithm that keeps adding columns that
are not selected to the pruned table until reaching the maximum input size of
the downstream model to make the best use of its capacity.

##### Table Serializing

Tables are further serialized to a 1-D sequence after pruning to be compatible
with the input format of the language model. We follow the serializing method
used in TABLE-BERT (Chen et al., 2020) that paraphrases tables with a natural
language template. Specifically, a table with $m$ rows and $n$ columns is
paraphrased as “row 1 is: $h_{1}$ is $T_{11}$; … ; $h_{n}$ is $T_{1n}$. row 2
is: … row $m$ is: $h_{1}$ is $T_{m1}$; … ; $h_{n}$ is $T_{mn}$.", where
$h_{i}$ refers the $i^{th}$ header and $T_{ij}$ the value in the $(i,j)-th$
cell of table $T$. We find that such template-serialized tables are more
suitable for language models pre-trained on unstructured text to process.

####  3.1.2 Joint Representation Learning

After the table pre-processing, the serialized table and the statement are
further passed to a language model to learn the joint contextual
representation of each token. The learned representation vectors are then
transmitted to the experts and the management module for inference and
management. Specifically, the serialized table and the statement are initially
tokenized into two token sequences T̃ and S. Then the joint token sequence X
is formed as $\textbf{X}=[\langle s\rangle$, S, $\langle/s\rangle$, T̃,
$\langle/s\rangle]$, where $\langle s\rangle$ and $\langle/s\rangle$ are the
separators that identify the beginning and the end of each token sequence. The
token sequence X will be padded or truncated to fit the maximum input length
of the language model. Finally, a transformer model is applied to learn the
contextual representation of X :

$\textbf{H}=f_{LM}(\textbf{X})$ (1)

where $\textbf{H}\in\mathbb{R}^{n\times d}$ refers to the learned joint
representation, $n$ is the maximum input length and $d$ the dimension of the
representation vector. $f_{LM}$ denotes the contextual representation learning
process of the language model. In this paper, we implement it with transformer
(Vaswani et al., 2017), the most popular contextual representation model in
recent years.

###  3.2 Experts

A group of experts is applied to verify the statements separately based on the
same statement-table joint semantics extracted by the feature extractor
module. Experts share the same model structure, while the parameter learning
strategy of SaMoE gives expert differentiation. Specifically, each expert is
implemented with a stack of transformer encoding layers. An MLP classifier
that calculates the probability of the statement is entailed by the evidence
table based on the encoded semantics. We implement experts with the same
general structure rather than different structures specially designed for
certain reasoning types since we anticipate that the proposed framework can be
smoothly generalized to other datasets. The process above can be formulated as
follows:

$\textbf{h}_{i}=f_{Enc_{i}}(\textbf{H})$ (2)
$\textbf{p}_{i}=softmax(tanh(\textbf{h}_{i}\textbf{W}_{1}^{i})\textbf{W}_{2}^{i})$
(3)

where $\textbf{h}_{i}\in\mathbb{R}^{d}$ is the token $\langle s\rangle$’s
final representation vector encoded by the $i^{th}$ expert’s encoder
$Enc_{i}$. It implies the $i^{th}$ expert’s whole understanding to the
statement-table pair. $\textbf{W}_{1}^{i}\in\mathbb{R}^{d\times d}$ and
$\textbf{W}_{2}^{i}\in\mathbb{R}^{d\times 2}$ are the trainable parameters of
$i^{th}$ expert’s classifier, which projects $\textbf{h}_{i}$ to the
probabilities $\textbf{p}_{i}\in\mathbb{R}^{2}$ that the statement is
entailed/refuted by the table. $tanh$ and $softmax$ are activation functions.
$n_{e}$ refers to the number of experts.

###  3.3 Management Module

Learning the joint semantics parsed in Sec.3.1, the management module intends
to generate attention scores to bias experts’ training and combine experts’
results efficiently. The module consists of two components: manager and
supervisor, both of them are implemented based on transformer model. The
manager is mainly designed to guide experts’ training, while the supervisor is
applied to combine experts’ results efficiently.

##### Manager

The manager guides the training of experts and forms a preliminary assumption
to the expert combination. It encodes the joint representation matrix and
generates attention scores $\textbf{a}_{M}$ to guide the experts’ training
process:

$\textbf{h}_{M}=f_{Enc_{M}}(\textbf{H})$ (4)
$\textbf{e}_{M}=tanh(\textbf{h}_{M}\textbf{W}_{1}^{M})\textbf{W}_{2}^{M}$ (5)
$\textbf{a}_{M}=softmax(\textbf{e}_{M})$ (6)

where $Enc_{M}$ denotes the manager’s encoder,
$\textbf{W}_{1}^{M}\in\mathbb{R}^{d\times d}$ and
$\textbf{W}_{2}^{M}\in\mathbb{R}^{d\times n_{e}}$ are trainable parameters.
The network structures of the manager and experts are basically the same, only
different in the layers of the encoder and the output dimension.

After preceding calculation, the normalized attention scores $\textbf{a}_{M}$
are used to guide the training of experts by a specially designed verification
loss, which will be introduced in Sec.4.1.1.

##### Supervisor

The supervisor adjusts the attention scores submitted by the manager to
improve the cooperative efficiency among experts (i.e. assigning higher
weights to experts who perform better on the current input pair). The network
predicts the deviation between the preliminary assumption (i.e., the
attention) and the ideal combination weights based on the knowledge encoded in
the joint representation matrix H:

$\textbf{h}_{S}=f_{Enc_{S}}(\textbf{H})$ (7)
$\textbf{e}_{S}=tanh(\textbf{h}_{S}\textbf{W}_{1}^{S})\textbf{W}_{2}^{S}$ (8)
$\textbf{a}_{S}=softmax(\textbf{e}_{M}+\textbf{e}_{S})$ (9)

where $\textbf{W}_{1}^{S}\in\mathbb{R}^{d\times d}$,
$\textbf{W}_{2}^{S}\in\mathbb{R}^{d\times n_{e}}$ are trainable parameters and
$Enc_{S}$ refers to the encoder of the supervisor. Parameters of the
supervisor are optimized self-adaptively based on experts’ performance on the
train set. More details of this learning strategy will be presented in
Sec.4.2.

##  4 Parameter Learning

Parameters in SaMoE are learned in two consecutive stages: 1) Multi-expert
training: parameters in the feature extractor, experts and the manager are
end-to-end optimized under the supervision of labels; 2) Self-adaptive
learning: parameters in the supervisor are self-optimized by observing
experts’ performance on the train set (other parameters are fixed
simultaneously). A weighted sum of two losses is minimized in the first stage
to achieve diverse and balanced training of experts. For the second stage, we
minimize a self-adaptive loss calculated based on the experts’ classification
loss. Subsequent sections introduce these two learning stages in detail.

###  4.1 Multi-expert Training

Multi-expert training guides each expert on dealing with different reasoning
types while maintaining balanced training across experts. To achieve the goals
above, we develop two loss functions: 1) verification loss $\mathcal{L}_{V}$
that measures the weighted sum of each expert’s classification loss,
differentiating experts’ learning with different attention scores assigned by
the manager; 2) manager assumption loss $\mathcal{L}_{M}$ that is applied to
prevent the occurrence of imbalanced training across experts. The overall loss
of this state is calculated by a weighted sum of these two terms:
$\mathcal{L}_{1}=\mathcal{L}_{V}+\lambda\mathcal{L}_{M}$, where $\lambda$ is a
hyperparameter that controls the ratio of $\mathcal{L}_{M}$. Subsequent
sections provide detailed introduction to these two terms.

####  4.1.1 Verification Loss

The verification loss $\mathcal{L}_{V}$ is designed based on the loss function
proposed in Jacobs et al. (1991). It is calculated by the weighted sum of each
expert’s cross-entropy:

$\mathcal{L}_{V}=\sum_{i=1}^{n_{e}}(\textbf{a}_{M})_{i}\cdot
CE(\textbf{p}_{i},l)$ (10)

where $(\textbf{a}_{M})_{i}$ is the $i^{th}$ element of the attention scores
$\textbf{a}_{M}$, $l\in\\{0,1\\}$ is the label of the statement-table pair and
$CE(\cdot,\cdot)$ the cross-entropy function. Note that it is necessary to
calculate each expert’s cross-entropy independently. We want each expert to
behave like an independent expert (i.e., complete the verification without the
help of other experts). The attention vector $\textbf{a}_{M}$ acts as a
"training scheduler" in this loss function: experts that are assigned with
larger attention scores receive a larger gradient than other experts on the
current input, resulting in diverse experts’ performance.

####  4.1.2 Manager Assumption Loss

We have trained the MoE with only the verification loss $\mathcal{L}_{V}$ and
observe a severe "imbalanced experts" phenomenon that only one expert is well-
trained (i.e., the expert performance is improved by training) and the manager
keeps assigning a close-to-1 attention score to this expert, which is also
reported in previous research (Eigen et al., 2013; Shazeer et al., 2017). To
avoid this problem, we develop another loss function that forces the manager
to assign reasonable attention scores to experts:

$\mathcal{L}_{M}=D(\textbf{a}_{P}||\textbf{a}_{M})$ (11)

where $D(\cdot||\cdot)$ denotes the Kullback–Leibler divergence and
$\textbf{a}_{P}$ a prior assumption that is generated with a simple heuristic
algorithm (to be introduced in the next paragraph) which requires limited
prior knowledge of the reasoning types. By minimizing $\mathcal{L}_{M}$, the
manager learns to assign each expert with a reasonable attention score,
resulting in a balanced training across experts.

##### Prior Assumption Generation

The prior assumption $\textbf{a}_{P}$ is generated to represent the
probabilities that the statement involves different reasoning types that we
are interested in. Specifically, we develop a trigger-word-based heuristic
algorithm to form the prior assumption for each statement automatically:

  1. 1.

Initialize the prior assumption with $\textbf{e}_{0}\in\mathbb{R}^{n_{e}}$,
which is empirically set as $(0.1,0.1,...,0.6)^{T}$. The
$(\textbf{e}_{0})_{n_{e}}$ represents the probability that the statement does
not involve any predefined reasoning types and thus is set higher than other
values in advance.

  2. 2.

Traverse the trigger-word set111a set of words that suggest a specific
reasoning type, see Appendix C for more information. of each reasoning type
($n_{e}-1$ types in total). If a trigger word/pattern $w$ that belongs to
$i^{th}$ reasoning type is detected in the statement, the trigger’s weight
$s_{w}$ (set empirically) is accumulated to the $i^{th}$ dimension of a zero-
initialized bias vector $\delta\in\mathbb{R}^{n_{e}}$:
$\delta_{i}\leftarrow\delta_{i}+s_{w}$.

  3. 3.

Add the bias vector $\delta$ to the prior assumption $\textbf{e}_{0}$ and
normalize to get the prior assumption:
$\textbf{a}_{P}=softmax(\textbf{e}_{0}+\delta)$.

Figure 3 presents an example of this process. Learning to imitate the prior
assumption, the manager guides each expert to focus on different reasoning
types and thus achieves diverse experts. We implement a relatively small
trigger-word pool in experiments and find the method works effectively,
indicating that the method can be smoothly generalized to other datasets with
little modification to the predefined reasoning types and trigger-word pool.

Figure 3: An example of prior assumption generation with $n_{e}=5$ and 4
predefined reasoning types.

###  4.2 Self-adaptive Learning

Self-adaptive learning aims to enhance further the expert combining efficiency
with only the knowledge of the expert’s performance on the train set.
Specifically, an “expert ability" vector $\textbf{a}_{E}\in\mathbb{R}^{n_{e}}$
is calculated based on the “expert loss" vector
$\textbf{m}\in\mathbb{R}^{n_{e}}$, where $\textbf{m}_{i}=CE(\textbf{p}_{i},l)$
is the cross-entropy loss of the $i^{th}$ expert. Note that the cross-entropy
of the expert is negatively correlated with its performance. Then the expert
ability vector $\textbf{a}_{E}$ is calculated as follows:

$\textbf{a}_{E}=softmax(-\alpha\cdot\textbf{m})$ (12)

where $\alpha=\sqrt{\beta/Var(\textbf{m}})$ is a variance-normalizing
coefficient and $\beta$ is a hyperparameter that decides the variance of the
expert ability vector before the activation (i.e.,
$Var(-\alpha\cdot\textbf{m})=\beta$). Such normalization is designed based on
the observation that m tends to have a extreme small variance and
$softmax(-\textbf{m})$ often generates a close-to-uniform distribution. Note
that the generated $\textbf{a}_{E}$ is positively correlated with the experts’
performance (e.g., if the $i^{th}$ expert outperforms the $j^{th}$ expert on
the input pair then we have $(\textbf{a}_{E})_{i}>(\textbf{a}_{E})_{j}$).

Based on $\textbf{a}_{E}$, we develop the loss function that has the same form
with $\mathcal{L}_{M}$ in Sec 4.1.2:

$\mathcal{L}_{S}=D(\textbf{a}_{E}||\textbf{a}_{S})$ (13)

By minimizing the loss above, the higher attention scores are assigned to the
best-performed experts after the supervisor’s adjustment, resulting in more
efficient cooperation across experts.

##  5 Experiment Setup

###  5.1 Data and Metric

We conduct the experiments on TabFact, a large scale benchmark dataset of the
table-based fact verification task222We did not conduct experiments on other
datasets such as SEM-TAB-FACTS (Wang et al., 2021) and InfoTabs (Gupta et al.,
2020), since there is little work and comparisons have been made on these
datasets.. TabFact contains a total of 117k statements and 16k Wikipedia
tables. The test set is further divided into a simple and complex subset based
on verification difficulty, for verifying some statements on TabFact requires
more logical/numerical reasoning skills. We choose accuracy as the evaluation
metric following the existing work to make our experiment results comparable.
More details of TabFact are presented in Appendix A.

###  5.2 Implementation Details

##### Training Details

We set $n_{e}=5$ expert networks in our implementation of SaMoE. The
transformer layers are 12 for encoders in the feature extractor and experts
and 2 for encoders in the manager and supervisor. The hidden states’ dimension
$d$, the maximum input length $n$, the $\lambda$ in Sec.4.1, and the $\beta$
in Sec.4.2 are set to 1024, 512, 0.1 and 0.1 respectively. We applied RoBERTa-
Large (Liu et al., 2019) to initialize the feature extractor and experts in
our framework. The details of parameter initialization can be found in
Appendix B.

We apply Adam optimizer (Kingma and Ba, 2015) in training with learning rate
2e-5, dropout rate 0.1, warmup step 17,304, and batch size 32. SaMoE is first
trained in the Multi-expert training stage for 57,680 steps (20 epochs). Then
the supervisor is trained in the self-adaptive learning stage for another
5,000 steps, while the best parameters of other parts in the framework are
loaded and fixed. The model is evaluated every 1000 steps, and the model that
achieves the highest performance on the development set is saved. All the
codes are implemented with Pytorch (Paszke et al., 2019) and the transformers
package (Wolf et al., 2020). We train all our models on a single GeForce RTX
3090.

##### Settings of Prior Assumption

We choose the top 4 types of reasoning types that appear most frequently in
TabFact333We follow the statistics in Chen et al. (2020) for the frequency of
different reasoning types. (count, comparative, superlative, negation). We
apply a small trigger-word pool containing only 26 trigger words, injecting
limited prior knowledge of the dataset. More details of this part are
presented in Appendix C.

###  5.3 Baselines

We compared our proposed framework with different kinds of baselines on
TabFact: (1) Program-enhanced methods: LPA (Chen et al., 2020),
LogicalFactChecker (Zhong et al., 2020), HeterTFV (Shi et al., 2020), ProgVGAT
(Yang et al., 2020) and Decomp (Yang and Zhu, 2021); (2) Table-based pre-
trained models: TaPas (Eisenschlos et al., 2020) and TaPEx (Liu et al., 2021);
(3) Other methods: Table-BERT (Chen et al., 2020) and SAT (Zhang et al.,
2020).

Model | Val | Test | Testsimple | Testcomplex | Small Test
---|---|---|---|---|---
TABLE-BERT | 66.1 | 65.1 | 79.1 | 58.2 | 68.1
LPA | 65.1 | 65.3 | 78.7 | 58.5 | 68.9
LogicalFactChecker | 71.8 | 71.7 | 85.4 | 65.1 | 74.3
HeterTFV | 72.5 | 72.3 | 85.9 | 65.7 | 74.2
SAT | 73.3 | 73.2 | 85.5 | 67.2 | -
ProgVGAT | 74.9 | 74.4 | 88.3 | 67.6 | 76.2
Decomp-Large | 82.7 | 82.7 | 93.6 | 77.4 | 84.7
TaPas-Large | 81.5 | 81.2 | 93.0 | 75.5 | 84.1
TaPEx | 84.6 | 84.2 | 93.9 | 79.6 | 85.9
SaMoE | 84.2 | 85.1 | 93.6 | 80.9 | 86.7
Human Performance | - | - | - | - | 92.1
Table 1: Comparative performance (accuracy) on TabFact.

##  6 Results

###  6.1 Overall Performance

We compare the proposed SaMoE with different kinds of baselines, and the
results are listed in Table 1. Baselines are presented with the best
performance reported in the corresponding papers. SaMoE obtains an accuracy of
85.1% on the test set, achieving a new state-of-the-art on the dataset.
Results show that our method consistently outperforms all the program-enhanced
methods with a significant 2.4% improvement compared with the Decomp method
(the best performed program-enhanced method). Note that SaMoE performs similar
with Decomp-Large on the simple subset of the test set (93.6% vs. 93.6%) while
outperforms Decomp-Large with a remarkable 3.5% on the complex subset (80.9%
vs. 77.4%). Such analysis indicates that the performance improvement is mainly
derived from successfully verifying complex statements, which required more
sophisticated reasoning than statements in the simple set. SaMoE even shows
comparable performance with the previous SOTA TaPEx that is pre-trained to
execute SQL queries on tables. Our method outperforms TaPEx with a 0.9%
improvement on the test set and a further 1.3% improvement on the complex
subset, indicating that SaMoE, based on a text-based pre-trained model,
performs even better than table-based pre-trained models on a variety of
complex reasoning types demanded by the table-based verification.

Model | Val | Test
---|---|---
SaMoE | 84.2 | 85.1
SaMoE w/o Sa | 84.0 | 84.7
SaMoE w/o Sa ($n_{e}=1$) | 83.6 | 84.0
Table 2: Ablation results that shows the effectiveness of the proposed MoE and
self-adaptive learning methods. SaMoE w/o Sa denotes that the framework
without self-adaptive learning, and $n_{e}=1$ denotes that the framework
involves only one expert, where the management module does not work in this
situation.

###  6.2 Ablation Study

We further investigate the effectiveness of the MoE structure and self-
adaptive learning with an ablation study. We conduct two experiments: one
reduces the number of experts to 1 to disable the contribution from the MoE
structure (SaMoE w/o Sa ($n_{e}=1$)); the other trains the proposed framework
with only the Multi-expert training stage (SaMoE w/o Sa). Results are
presented in Table 2. The MoE structure achieves a 0.7% improvement on the
test set (84.7% vs. 84.0%), and self-adaptive learning further improves the
performance slightly (85.1% vs. 84.7%). Note that the slight improvement of
self-adaptive learning is expected since the experts and the feature extractor
are fixed in this stage. The results demonstrate the effectiveness of both the
MoE structure and the self-adaptive learning.

###  6.3 Effectiveness Analysis

We show in this section that the effectiveness of the proposed framework is
derived from two aspects: the differentiation of experts (each expert
outperforms others on a specific part of reasoning types) and the effective
attention assignment by the management module (the best-performed experts are
assigned with higher attention scores).

(a) Trained with $\mathcal{L}_{V}$

(b) Trained with $\mathcal{L}_{V}+\mathcal{L}_{M}$

Figure 4: Comparison of models trained with/without the manager assumption
loss $\mathcal{L}_{M}$.

####  6.3.1 Expert Differentiation

We first investigate the proposed manager assumption loss $\mathcal{L}_{M}$
and find that it achieves balanced training across experts, which is the
premise of expert differentiation. Figure 4 compares the two models trained
with and without $\mathcal{L}_{M}$, with the performance curves of different
experts on the development set presented in each sub-figure. Once
$\mathcal{L}_{M}$ is applied, four experts that fail to be trained (the
performance stays around 50% as training steps increase) achieve comparable
performance with the rest expert (expert 5 in sub-figure (a)). The result
indicates that the proposed $\mathcal{L}_{M}$ leads balanced training across
experts.

Figure 5: The proportion of statements in the test set that at least $k$
experts verify them correctly ($k\in[1,5]$).

We further show that the proposed framework achieves differentiation across
experts. Figure 5 presents the proportion of statements in the test set that
are verified correctly by at least $k$ experts ($k$ varies from 1 to 5). Note
that the proportion increases rapidly as $k$ decreases (76.2% to 90.7% for $k$
from 5 to 1), which illustrates that experts behave differently on a large
proportion of statements. The results indicate that SaMoE successfully
achieves expert differentiation, which expands the original performance upper
bound considerably (90.7%).

Model | Accuracy
---|---
Top 1 | Top 2 | Top 3
SaMoE | 32.0 | 59.0 | 76.0
SaMoE w/o Sa | 25.4 | 44.8 | 67.6
Table 3: The top-k accuracy of the management module that predicts the best-
performed experts on the test set.

####  6.3.2 Effective Attention Assignment

We conduct a detailed analysis to investigate whether the management module
assigns higher attention scores to experts with the best performance after
self-adaptive learning. To achieve this goal, we regard the management module
as a $n_{e}$-class classifier and calculate the top-k accuracy of predicting
the best-performed expert (the one with the smallest cross-entropy) on the
test set where k is chosen in [1, 2, 3]. The results of the analysis are
presented in Table 3. The top-k accuracy is improved significantly after self-
adaptive learning (+6.6%, +14.2%, +8.4% respectively), indicating that the
management module successfully assigns higher attention scores to the best-
performed experts by self-adaptive learning.

Based on the significant performance upper bound expanded by the expert
differentiation, the effective attention assignment achieves more efficient
cooperation across these diverse experts, thus improving the verification
performance.

##  7 Related Works

##### Table-Based Fact Verification

Most of the current models utilize programs to improve the model’s ability to
handle various types of numerical and logical reasoning (Chen et al., 2020;
Zhong et al., 2020; Shi et al., 2020; Yang et al., 2020; Yang and Zhu, 2021),
while Eisenschlos et al. (2020); Liu et al. (2021) leverage table-based pre-
trained models to parse the structural and numerical semantics of tables
better. Unlike previous works, we use a novel mixture-of-experts framework to
handle different logical and numerical semantics without semantic parsing and
table-based pre-training.

##### Mixture of Experts

Mixture of experts is a special model combining method. Jacobs et al. (1991)
first introduces this method and proposes a loss that encourages competitive
learning across expert models. Afterwards, it is applied in various fields,
including dialog system (Le et al., 2016), content recommendation(Ma et al.,
2018; Zhu et al., 2020), image classification(Wang et al., 2020; Riquelme et
al., 2021), etc. In this paper, we develop a self-adapted mixture-of-experts
framework that achieves a more effective combination of experts by learning
from the experts’ performance on the train set.

##  8 Conclusion

This paper proposes a new method that exploits the mixture of experts to
recognize and execute different types of reasoning required for table-based
fact verification. We propose an MoE model guided with limited prior knowledge
to handle different parts of the reasoning types required by table-based
verification with diverse experts. Moreover, we design a supervisor network to
adjust the imprecise attention score and achieve a more efficient combination
across experts. A self-adaptive learning strategy is further applied to train
the proposed supervisor network without prior knowledge of the task or
dataset. The experiments show that the proposed model achieves a new state-of-
the-art performance of 85.1% accuracy on the benchmark dataset TABFACT. The
ablation studies and analysis further indicate the effectiveness of the
proposed MoE structure and self-adaptive learning strategy. We hope our work
is helpful for those who aim to further exploit the power of mixture-of-
experts on table-based reasoning in the future.

## Acknowledgments

We thank the anonymous reviewers for their valuable comments. This work is
supported by Scientific and Technological Innovation 2030–“New Generation
Artificial Intelligence” Major Project (No.2021ZD0113400), National Key
Research and Development Program of China (No.2021YFC250083), and the Beijing
Municipal Natural Science Foundation (No.L192026).

## References

  * Chen et al. (2020) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020.  Tabfact: A large-scale dataset for table-based fact verification.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Eigen et al. (2013) David Eigen, Marc’Aurelio Ranzato, and Ilya Sutskever. 2013.  Learning factored representations in a deep mixture of experts.  _arXiv preprint arXiv:1312.4314_.
  * Eisenschlos et al. (2020) Julian Eisenschlos, Syrine Krichene, and Thomas Müller. 2020.  Understanding tables with intermediate pre-training.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 281–296, Online. Association for Computational Linguistics.
  * Goodrich et al. (2019) Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019.  Assessing the factual accuracy of generated text.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 166–175.
  * Gupta et al. (2020) Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020.  INFOTABS: Inference on tables as semi-structured data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2309–2324, Online. Association for Computational Linguistics.
  * Herzig et al. (2020) Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020.  TaPas: Weakly supervised table parsing via pre-training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4320–4333, Online. Association for Computational Linguistics.
  * Jacobs et al. (1991) Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991\.  Adaptive mixtures of local experts.  _Neural computation_ , 3(1):79–87.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the factual consistency of abstractive text summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9332–9346, Online. Association for Computational Linguistics.
  * Le et al. (2016) Phong Le, Marc Dymetman, and Jean-Michel Renders. 2016.  Lstm-based mixture-of-experts for knowledge-aware dialogues.  In _1st Workshop on Representation Learning for NLP_ , pages 94–99. ACL Anthology.
  * Liu et al. (2021) Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, and Jian-guang Lou. 2021.  Tapex: Table pre-training via learning a neural sql executor.  _arXiv preprint arXiv:2107.07653_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Ma et al. (2018) Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1930–1939.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 8024–8035.
  * Rashkin et al. (2017) Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017\.  Truth of varying shades: Analyzing language in fake news and political fact-checking.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2931–2937, Copenhagen, Denmark. Association for Computational Linguistics.
  * Riquelme et al. (2021) Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. 2021.  Scaling vision with sparse mixture of experts.  _Advances in Neural Information Processing Systems_ , 34.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Shi et al. (2020) Qi Shi, Yu Zhang, Qingyu Yin, and Ting Liu. 2020.  Learn to combine linguistic and symbolic information for table-based fact verification.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5335–5346, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _NAACL-HLT_.
  * Vaibhav et al. (2019) Vaibhav Vaibhav, Raghuram Mandyam, and Eduard Hovy. 2019.  Do sentence interactions matter? leveraging sentence level representations for fake news classification.  In _Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13)_ , pages 134–139, Hong Kong. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wang et al. (2021) Nancy X. R. Wang, Diwakar Mahajan, Marina Danilevsky, and Sara Rosenthal. 2021.  SemEval-2021 task 9: Fact verification and evidence finding for tabular data in scientific documents (SEM-TAB-FACTS).  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 317–326, Online. Association for Computational Linguistics.
  * Wang et al. (2020) Xin Wang, Fisher Yu, Lisa Dunlap, Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, and Joseph E Gonzalez. 2020.  Deep mixture of experts via shallow embedding.  In _Uncertainty in artificial intelligence_ , pages 552–562. PMLR.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yang et al. (2020) Xiaoyu Yang, Feng Nie, Yufei Feng, Quan Liu, Zhigang Chen, and Xiaodan Zhu. 2020\.  Program enhanced fact verification with verbalization and graph attention network.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7810–7825, Online. Association for Computational Linguistics.
  * Yang and Zhu (2021) Xiaoyu Yang and Xiaodan Zhu. 2021.  Exploring decomposition for table-based fact verification.  _arXiv preprint arXiv:2109.11020_.
  * Zhang et al. (2020) Hongzhi Zhang, Yingyao Wang, Sirui Wang, Xuezhi Cao, Fuzheng Zhang, and Zhongyuan Wang. 2020.  Table fact verification with structure-aware transformer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1624–1629, Online. Association for Computational Linguistics.
  * Zhong et al. (2020) Wanjun Zhong, Duyu Tang, Zhangyin Feng, Nan Duan, Ming Zhou, Ming Gong, Linjun Shou, Daxin Jiang, Jiahai Wang, and Jian Yin. 2020.  LogicalFactChecker: Leveraging logical operations for fact checking with graph module network.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6053–6065, Online. Association for Computational Linguistics.
  * Zhu et al. (2020) Ziwei Zhu, Shahin Sefati, Parsa Saadatpanah, and James Caverlee. 2020.  Recommendation for new users and new items via randomized training and mixture-of-experts transformation.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1121–1130.

##  Appendix A Statistics of TabFact

Table 4 shows the basic statistics of TabFact. As the table shows, the whole
dataset is randomly divided into three subsets with the ratio be 8:1:1. The
average numbers of rows and columns in tables keep approximately the same
across three subsets, which reflects the consistency of data distribution.

Split | #Sentence | #Table | Avg.row | Avg.col
---|---|---|---|---
Train | 92,283 | 13,182 | 14.1 | 5.5
Dev | 12,792 | 1,696 | 14.0 | 5.4
Test | 12,779 | 1,695 | 14.2 | 5.4
Table 4: Statistics of TabFact, including the number of statements, tables,
and the average number of rows and columns in tables.

##  Appendix B Parameter Initialization

For parameter initialization, We leverage RoBERTa-Large, a pre-trained
language model that has 24 transformer encoding layers. We initial parameters
of the feature extractor with the embedding layer and the bottom 12 encoding
layers of RoBERTa-Large and each expert with the upper 12 encoding layers of
RoBERTa-Large, respectively. We use PyTorch to initialize other parameters
randomly.

##  Appendix C Specific Setting of Prior Assumption Generation

We choose four reasoning types that appear most frequently in TabFact: count,
comparative, superlative, and negation. The detailed definitions of four
reasoning types chosen in our implementation are listed below:

  1. 1.

Count: counting the number of specific rows in the table, such as “xxx be
listed a total of 3 times", “xxx win only 1 time in …", etc.

  2. 2.

Comparative: comparing two values in the statement or cells, such as “xxx play
in more than 1 game during …", “xxx has a larger yyy than zzz", etc.

  3. 3.

Superlative: finding the highest/lowest value of the specific column, such as
“the longest xxx be yyy", “the lowest score at xxx be yyy", etc.

  4. 4.

Negation: negating the original semantics of the statement, such as “xxx has
never lost a game in …", “xxx never score 0 points", etc.

Type | Trigger | Weight
---|---|---
Count | only+[number] | 1.6
Count | [number]+times | 2
Count | [number]+of | 1.6
Count | there be+[number] | 1.6
Negation | no | 1.5
Negation | not | 1.5
Negation | never | 1.5
Negation | didn’t | 1.5
Comparative | [JJS] or [RBS] | 1.5
Superlative | [JJR] or [RBR] | 1.5
Table 5: Some trigger words/patterns applied in the generation of the prior
assumption on TabFact.

A small trigger-word pool that contains only 26 trigger words/patterns is
applied for the prior assumption generation: 11 triggers for the "count" type,
15 for "negation"; and for the rest types (i.e., "comparative" and
"superlative" types), the NLTK package is employed to recognize the
comparative and superlative words automatically. Such a small trigger-word
pool injects limit prior knowledge of the dataset, indicating that the
proposed method can be generalized to other datasets by simply modifying the
pool of trigger words. Table 5 presents some words/patterns in the trigger-
word pool applied in our experiments. x+[number] denotes a combination of a
word and a number that is served as a trigger (e.g., for the statement “xxx
win 3 times in …", we match the phrase “3 times" with the trigger
“[number]+times").


††thanks: Equally contributing authors††thanks: Equally contributing authors

# Cooling of a Zero-Nuclear-Spin Molecular Ion to a Selected Rotational State

Patrick R. Stollenwerk  Argonne National Laboratory, Lemont, Illinois 60439,
USA  Ivan O. Antonov  Department of Physics and Astronomy, Northwestern
University, Evanston, Illinois 60208, USA  Sruthi Venkataramanababu  Graduate
Program in Applied Physics, Northwestern University, Evanston, Illinois 60208,
USA  Yen-Wei Lin  Department of Physics and Astronomy, Northwestern
University, Evanston, Illinois 60208, USA  Brian C. Odom
<EMAIL_ADDRESS>Department of Physics and Astronomy, Northwestern
University, Evanston, Illinois 60208, USA

###### Abstract

We demonstrate rotational cooling of the silicon monoxide cation via optical
pumping by a spectrally filtered broadband laser. Compared with diatomic
hydrides, SiO+ is more challenging to cool because of its smaller rotational
interval. However, the rotational level spacing and large dipole moment of
SiO+ allows direct manipulation by microwaves, and the absence of hyperfine
structure in its dominant isotopologue greatly reduces demands for pure
quantum state preparation. These features make 28Si16O+ a good candidate for
future applications such as quantum information processing. Cooling to the
ground rotational state is achieved on a 100 ms time scale and attains a
population of 94(3)%, with an equivalent temperature $T=0.53(6)$ K. We also
describe a novel spectral-filtering approach to cool into arbitrary rotational
states and use it to demonstrate a narrow rotational population distribution
($N\pm 1$) around a selected state.

##  I Introduction

A broad range of physics and chemistry interest motivates developing tools for
robust control over molecular internal degrees of freedom. Applications
include study of cold collisions Zhang and Willitsch (2017); Dörfler et al.
(2019), quantum state-dependent chemistry Bell and P. Softley (2009); Hutson
(2010); Balakrishnan (2016), astrochemistry Ian (2008), many-body physics Yan
et al. (2013); Hazzard et al. (2014); Schmidt and Lemeshko (2015, 2016),
quantum information processing Ni et al. (2018); Hudson and Campbell (2018);
Campbell and Hudson (2019), and precision spectroscopy Safronova et al.
(2018); Cairncross et al. (2017); Andreev et al. (2018); Alighanbari et al.
(2020); Chou et al. (2020). Molecules, in contrast to free atoms, have
rotational and vibrational degrees of freedom via their chemical bonds. On one
hand, these extra degrees of freedom extend the scope of possible control and
provide the rich structure that generates their appeal. On the other hand,
their level structure can be quite complicated and thus challenging for state
control. Despite this increased complexity, substantial progress on state
preparation of molecules has been made in the past decade using several
techniques including optical pumping Viteau et al. (2008); Staanum et al.
(2010); Schneider et al. (2010); Cournol et al. (2018); Lien et al. (2014),
buffer-gas cooling Rellergert et al. (2013); Hansen et al. (2014), state-
selective photoionization Tong et al. (2010), projective preparation using
quantum logic Chou et al. (2017), supersonic expansion of molecular beams
Shagam and Narevicius (2013), and photoassociation Ni et al. (2008).

Trapped molecular ions in Coulomb crystals can be isolated from the
environment and are well-suited for precision spectroscopy, quantum
information processing, and other applications requiring uninterrupted
dynamics over long time scales. State preparation by optical pumping allows
rapid resetting of the molecular state, often desired in these applications.
Optical pumping of trapped molecular ion rotations has previously been
demonstrated for diatomic hydrides Staanum et al. (2010); Schneider et al.
(2010); Lien et al. (2014). However, the non-zero nuclear spin of hydrogen (or
deuterium) couples with the rotational degree of freedom and any nuclear spin
of the other atom, making optical pumping to a pure state still a challenge.
The only demonstration of simultaneous rotational and nuclear spin optical
pumping achieved $\sim$20% hyperfine state purity of HD+ in a few tens of
seconds Bressel et al. (2012). A somewhat different quantum-logic approach
determines with high confidence into which pure state a single trapped CaH+
ion is non-destructively projected, but the _a priori_ probability of being in
a given rotational manifold is currently limited to the $\sim$10% thermal
population Chou et al. (2017).

In contrast to hydrides, high natural-abundance oxide isotopologues exist
where both atoms have nuclear spin $I=0$, circumventing the challenge of
hyperfine structure in quantum state preparation. Taking advantage of this
simplification, we demonstrate here broadband optical pumping of 28Si16O+ to
its ground rotational level, with well-defined total angular momentum, on a
timescale of 100 ms with 94(3)% fidelity. We also demonstrate, although with
lower state purity, optical pumping to a selected rotational state with $N>0$.

The rotational spacing of oxides (43 GHz for the lowest SiO+ interval) is
smaller than the few to several hundred GHz typical of hydrides. This makes
cooling them more challenging, both because the optical pumping spectrum is
more congested and because more levels are thermally populated; 95% of the 300
K population is distributed over the rotational states $N\leq 30$ for SiO+,
compared with $\sim$10 for typical hydrides. Although technically more
challenging for cooling, the smaller rotational interval of oxides is
favorable for future applications. SiO+ has a sizeable body-frame dipole
moment of $>4$ Debye Werner et al. (1982); Cai and François (1999);
Chattopadhyaya et al. (2003); Li et al. (2019), so coherent rotational
transitions can be driven with convenient microwave sources. Also rotational
transitions of oxides are further from the peak of the 300 K blackbody
spectrum, so ultimate lifetimes and coherence times for polar oxides can be
one to three orders of magnitude longer, depending on other limiting factors.

Figure 1:  Transitions for pumping and state readout. (a) Potential energy
curves (not to scale) for the lowest three electronic states along with the
higher-lying C state, with representative vibrational and rotational
substructure. Black arrows show the primary pumping excitation and spontaneous
emission channel, and gray arrows a possible parity flip sequence. (b)
Rotational pumping from $N=2\rightarrow 0$. (c) Dissociative transitions from
$N={0,1,2}$.

##  II Broadband Optical Pumping

At 300 K, 99.6% of SiO+ molecules are in the ground vibrational state $v=0$,
but population is spread over $\sim$30 rotational levels. We desire rotational
pumping which minimizes unwanted incidental vibrational excitations. Several
groups have noted the advantage of optical pumping with the class of molecules
whose ground and excited electronic states have similar bond equilibrium
distances, i.e. molecules with nearly diagonal Franck-Condon Factors (FCFs) Di
Rosa (2004); Sofikitis et al. (2009); Nguyen and Odom (2011); Nguyen et al.
(2011); Cournol et al. (2018). Such molecules have electronic excitation
largely decoupled from vibrational excitation. In SiO+ the diagonal FCFs of
the $B$-$X$ transition (Fig 1(a)) allow on average more than 30 optical
pumping cycles before $v$ changes Stollenwerk et al. (2017).

Diagonal FCFs also imply the states have nearly identical rotational
constants, resulting in a spectrum well separated according to angular
momentum selection rules. A broadband laser with relatively simple spectral
filtering tuned to a diagonal transition in AlH+ was used to achieve high-
fidelity cooling to the ground rotational state Lien et al. (2014). Since the
rotational constant of SiO+ is an order of magnitude smaller, rotational
cooling of this new species to a similar degree requires significantly better
spectral filtering.

State preparation of SiO+ was achieved by spectrally filtering a frequency
doubled Spectra-Physics MaiTai HP laser tuned to the
$B^{2}\Sigma^{+}-X^{2}\Sigma^{+}$ electronic transition near 385 nm. Spectral
filtering was done by using the 2-$f$ configuration of the pulse-shaping setup
described in Lin and Odom (2016). For ground state preparation the spectral
filtering mask requires pumping of only the $P$-branch transitions ($\Delta
N=-1$) which is accomplished by blocking the high frequency components at the
Fourier plane with a razor blade (Fig. 2) Lien et al. (2011, 2014).

Figure 2:  (Center) Spectral masks overlayed onto the Fortrat diagram of the
SiO+ $B-X(0,0)$ transition. $P$ and $R$-branch transitions are indicated by
blue and red dots respectively. The blocked spectrum for each mask is
indicated by the dark hashes. The measured optical pumping spectra which have
been filtered to target the ground rotational state (Bottom) and $N=10$ (Top)
are shown with the relevant $P$ and $R$-branch transitions indicated by blue
and red vertical lines.

To extend preparation to arbitrary $N>0$ rotational states, a mask on the
$P$-branch must be introduced to only pump down to the target state, and the
mask on the $R$-branch needs to be shifted to allow pumping up to the target
state (Fig. 2). This is accomplished with the removal of a band in the middle
of the spectrum in addition to the removal of the high frequency components.
This band is filtered using a thin metal ribbon (0.038 $\times$ 3 mm) whose
profile is adjusted by rotating to match the required bandwidth at the Fourier
plane. In this way, each rotational level is exclusively pumped toward the
target state, which is intentionally left dark.

##  III Trapping and Detection

Quantum state control experiments were performed at room temperature under
ultra-high vacuum conditions ($7(4)\times 10^{-10}$ Torr). For each data point
a sample of 10 to 100 SiO+ was co-loaded with 500 to 1000 laser cooled barium
ions in a linear Paul trap. Loading of SiO+ was performed using the SiO
$A^{1}\Pi-X^{1}\Sigma^{+}(5,0)$ 1+1 REMPI transition following ablation of a
solid SiO sample Stollenwerk et al. (2019). Translational energy is rapidly
cooled sympathetically by Ba+, however molecular internal degrees of freedom
are decoupled from translational motion. We expect to load SiO+ between $N=4$
and $N=15$ Stollenwerk et al. (2019). The interval between loading and
dissociation is typically 30 s at which time, without optical pumping, BBR and
spontaneous emission have redistributed population from a single $N$ into
nearby levels, but full thermal equilibration has not yet occurred. We do not
observe significant population in $N=0$ without optical pumping.

The trapped SiO+ were detected using an in-situ laser cooled fluorescence mass
spectrometry (LCFMS) Baba and Waki (2001) technique. Briefly, the radial
secular motion of the SiO+ was resonantly excited using a low amplitude (0.5-1
V) RF waveform applied to one of the radial trapping rods. Motional excitation
causes Doppler broadening of the Ba+ resonance and a decrease of Ba+
fluorescence in proportion to the number of SiO+ in the trap.

State detection of SiO+ was performed destructively using single-photon
resonance-enhanced photodissociation spectroscopy via the predissociative
$C^{2}\Pi$ state, which our preliminary linewidth measurements indicate has a
lifetime of order of several hundred picoseconds for $v=0$ and less for higher
$v$. These lifetimes are sufficiently long to provide rotational resolution
and sufficiently short for efficient dissociation. The state was previously
reported in theoretical studies as the $2^{2}\Pi$ state Cai and François
(1999); Chattopadhyaya et al. (2003); Honjou (2003); Shi et al. (2012); Li et
al. (2019). It had not been observed experimentally prior to this work. A
manuscript detailing spectroscopy of the $C^{2}\Pi$ state is currently in
preparation Sci (manuscript in preparation).

A pulsed dye laser with frequency-doubled output near 226 nm was used for
dissociation. Predissociation of SiO+ via the $C^{2}\Pi$ state leads to Si+ \+
O products, and we monitored the LCFMS SiO+ signal to measure the dissociated
fraction. We performed two types of measurements to characterize state
control.

The first method uses slow steady-state depletion, where the rate constant
yields relative populations. The LCFMS signal was monitored while optical
pumping and concurrently firing the 10 Hz dissociation laser tuned to
frequency $f$ for 30 s. If dissociation is slow enough, steady-state
population is maintained, and the number of SiO+ molecules as a function of
pulse number $m$ is given by $N_{SiO^{+}}=N_{0}e^{-\Gamma(f)m}$. Each pulse
dissociates a fraction of remaining molecules given by

$d(f)=p_{N}(f)\,n_{N}\approx-\frac{dN_{SiO^{+}}}{dm}\frac{1}{N_{SiO^{+}}}=\Gamma(f),$
(1)

where $p_{N}(f)$ is the probability per pulse of dissociating a molecule which
is in the probed state $N$, and $n_{N}$ is the fraction of remaining molecules
in that state. To ensure that Eqn. (1) is valid, we require $p_{N}(f)\ll 1$
and $\Gamma\ll\Gamma_{eq}$, where $\Gamma_{eq}$ is the equilibration rate of
the probed state. Experimentally, we reduce the dissociation laser fluence
until these conditions are met. A fit to the LCFMS decay at each $f$ yields
the dissociation spectrum $d(f)$, in which the peak heights depend both on
line strength and population. Although these spectra show only relative
populations, this first method provides a good signal-to-noise ratio (SNR),
since the entire sample contributes toward statistics even for probed states
with low population. Also, spectra can be taken with constant SNR over a
dynamic range of more than two orders of magnitude.

Figure 3:  Dissociation spectra of SiO+ optically pumped (a) towards $N=0$ and
(b) $N=10$. (c) Relative rotational state populations inferred from the
spectra and their $\pm 1\sigma$ uncertainties, along with a 300 K thermal
distribution (red).

The second method is a single-shot depletion technique Staanum et al. (2010);
Schneider et al. (2010); Lien et al. (2011), which yields absolute
populations. We recorded the LCFMS signal before and after a single intense
pulse tuned to dissociate from state $N$, where $p_{N}(f)\sim 1$. Because the
predissociation lifetime of the upper state is much shorter than the 10 ns
pulse duration, 100% dissociation probability is achievable. The fractional
population in $N$ is given by $F_{N}=(D_{i}-D_{f})/D_{i}$, where $D_{i}$ and
$D_{f}$ are the (positive-valued) LCFMS fractional fluorescence dips before
and after the dissociation pulse.

##  IV Relative Cooling Efficiency

Fig. 1(c) shows the dipole-allowed
$\mbox{$|X^{2}\Sigma^{+},v=0\rangle$}\rightarrow\mbox{$|C^{2}\Pi_{3/2},v=1\rangle$}$
dissociative readout transitions. This vibronic transition was chosen because
it exhibits good separation between lines originating from $N=0$ and $N=1$ as
well as from other isotopologues. Each originating $N$ has up to four
resolvable lines labeled as $x(N)$. The branch type $x$ is characterized by
$\Delta N=(J^{\prime}+1/2)-N$, where $J^{\prime}$ is the upper rotational
quantum number; e.g. ${}^{s}R_{21}(0.5)$ is denoted $s(0)$.

Fig. 3 shows the spectrum after the population has been pumped toward $N=0$.
We simulated the spectrum using PGOPHER Western (2017) and fit the spectral
envelope to obtain a ratio of population in $N=1$ to $N=0$ of 0.075(3). We
also demonstrate cooling into an excited rotational state by applying the
spectral mask for $N=10$ (Fig. 2). Both the $N=0$ and $N=10$ spectra are in
sharp contrast with a thermal distribution at 300 K.

Although the steady-state analysis technique does not directly yield absolute
populations, some qualitative conclusions can be drawn about populations in
other states. Scans searching for transitions originating from $N\geq 2$ of
$v=0$ and for any $N$ of $v=1$ did not show any discernable peaks. The states
in the $A^{2}\Pi$ manifold are too short-lived for significant population
accumulation. A quantitative measurement setting bounds on these and other
populations is discussed in the section below.

##  V Absolute Cooling Efficiency and Timescale

Fig. 4 shows the measured population accumulation in $N=0$ when pumping toward
that state, analyzed using the single-shot method. Here, we used the
$\mbox{$|X^{2}\Sigma^{+},v=0,N=0\rangle$}\rightarrow\mbox{$|C^{2}\Pi_{1/2},v=0,J=0.5\rangle$}$
transition at 44044.5 cm-1. Technical noise, which dominates over SiO+
counting noise, is primarily due to laser fluctuations affecting Ba+ cooling
efficiency and fluorescence. The anomalous point at very short times is
understood to be a spurious signal from population in $N=11$, which has an
overlapping line.

Figure 4:  Single-shot analysis of $N=0$ population versus optical pumping
time. Error bars account for technical noise and SiO+ counting statistics.
Since LCFMS involves difference measurements, points and error bars outside of
the range of 0 to 1 are expected. The solid line is a fit to the model, and
the horizontal line indicates the corrected asymptote accounting for
systematic shifts. (Inset) Measured longevity of molecules prepared in $N=0$,
after the pumping laser is turned off.

Two time scales are present. The faster time scale is for cooling of the
separate parity populations independently. Photon absorption and then
spontaneous emission on the diagonal $B$-$X$ transitions is a parity-
preserving process, so approximately half of the population is unable to be
directly pumped into the even-parity ground rotational state. The slower time
scale is determined by the rate of parity flips. The details of this process
have not been determined, but one possible pathway for obtaining the requisite
odd number of electric dipole transitions is shown in Fig. 1(a) (also see
Supplemental Material).

An analytic model (see Supplemental Material) is used to fit the data. To
assess absolute cooling efficiency, an offset must be applied to correct
systematic shifts from reaction with the background hydrogen, isotopic
impurities, and partial overlap of the laser linewidth with other dissociating
transitions. We conclude that the steady-state $N=0$ absolute population
fraction of 28Si16O+ is 0.94(3), which corresponds to a temperature of 0.53(6)
K. This result is in good agreement with the relative population analysis.

Thermalization of SiO+ out of $N=0$, after the cessation of optical pumping,
is shown in the inset of Fig. 4. The population loss is well fit by an
exponential decay with a time constant of 35(4) s. This time scale is much
faster than blackbody-induced pure rotational (400 s) or vibrational
excitations (380 s), and also faster than the observed reaction rate with H2
($\sim$600 s). However, it could be consistent with inelastic collisions with
H2, which have a Langevin collision time of 40(20) s, with uncertainty coming
from the H2 pressure. Blackbody redistribution via the $A$ state is also a
possible mechanism, expected to occur on a time scale of 70 - 130 s given the
predicted $A$ state lifetimes Nguyen and Odom (2011); Li et al. (2019).

##  VI Conclusions

This work demonstrates the extension of a broadband rotational cooling
technique from a trapped diatomic hydride to an oxide. Furthermore, we show
that the class of diagonal molecules amenable to rotational cooling can be
extended to include those with an intermediate electronic state not involved
in the dominant optical pumping cycle.

Higher resolution spectral filtering is possible, for example by using a
virtual imaged phased array (VIPA) where sub-GHz resolution has been achieved
Willits et al. (2012). VIPA could significantly enhance the preparation
fidelity of the $N=0$ and $N=10$ states shown here. Consequently, it might be
advantageous to use $N>0$ states for quantum information processing, since
dominant decoherence mechanisms are reduced for higher rotational states
Hudson and Campbell (2018). Optical pumping to $N\gg 0$, not explored in this
work, can also be useful for spectroscopic studies and can provide insights
into the molecular Hamiltonian at high energies Sci (manuscript in
preparation).

The limiting timescale is currently the parity cooling step. Electronic decay
from $A$ to $X$ is predicted to be of order 5 ms Li et al. (2019), thus we
expect that the cooling rate could be increased by more than an order of
magnitude with increased spectral fluence of the pump laser. An alternative
would be to use microwaves to drive parity flips, for instance at 86 GHz
connecting $N=1$ with $N=2$ Scholl et al. (1995) for cooling to $N=0$. Use of
a microwave drive could equalize the timescales for cooling the two parities,
and with a more intense cooling laser 28Si16O+ population could be cooled to a
pure internal quantum state in $<10~{}\mu$s, limited by the spontaneous
emission rate of $B$. Further refinements may enable fluorescence imaging or
direct Doppler cooling of SiO+ Nguyen and Odom (2011); Li et al. (2019) and
could help realize multi-ion molecular clocks with canceling Stark and second
order Doppler shifts Stollenwerk et al. (2018).

We have demonstrated straightforward pumping of 28Si16O+ to a state of well-
defined angular momentum. Since 28Si16O+ also has microwave-accessible
rotational transitions for quantum manipulation, this species could play a
similar role in a wide range of applications, as do only a relatively small
handful of popular atomic species.

###### Acknowledgements.

We gratefully acknowledge Jason Nguyen for proposing rotational cooling of
SiO+, along with his and David Tabor’s early work on SiO+ in the lab.
Development of dissociative analysis techniques were funded by NSF Grant No.
PHY-1806861, LCFMS techniques were funded by ARO Grant No. W911NF-14-0378, and
optical state control techniques were funded by AFOSR Grant No.
FA9550-17-1-0352.

## References

  * Zhang and Willitsch (2017) D. Zhang and S. Willitsch, in _Cold Chemistry_ (2017), pp. 496–536.
  * Dörfler et al. (2019) A. D. Dörfler, P. Eberle, D. Koner, M. Tomza, M. Meuwly, and S. Willitsch, Nat. Commun. 10, 1 (2019).
  * Bell and P. Softley (2009) M. T. Bell and T. P. Softley, Mol. Phys. 107, 99 (2009).
  * Hutson (2010) J. M. Hutson, Science 327, 788 (2010).
  * Balakrishnan (2016) N. Balakrishnan, J. Chem. Phys. 145, 150901 (2016).
  * Ian (2008) W. S. Ian, _Low temperatures and cold molecules_ (World Scientific, 2008).
  * Yan et al. (2013) B. Yan, S. A. Moses, B. Gadway, J. P. Covey, K. R. A. Hazzard, A. M. Rey, D. S. Jin, and J. Ye, Nature 501, 521 (2013).
  * Hazzard et al. (2014) K. R. A. Hazzard, B. Gadway, M. Foss-Feig, B. Yan, S. A. Moses, J. P. Covey, N. Y. Yao, M. D. Lukin, J. Ye, D. S. Jin, et al., Phys. Rev. Lett. 113, 195302 (2014).
  * Schmidt and Lemeshko (2015) R. Schmidt and M. Lemeshko, Phys. Rev. Lett. 114, 203001 (2015).
  * Schmidt and Lemeshko (2016) R. Schmidt and M. Lemeshko, Phys. Rev. X 6, 011012 (2016).
  * Ni et al. (2018) K.-K. Ni, T. Rosenband, and D. D. Grimes, Chem. Sci. 9, 6830 (2018).
  * Hudson and Campbell (2018) E. R. Hudson and W. C. Campbell, Phys. Rev. A 98, 040302(R) (2018).
  * Campbell and Hudson (2019) W. C. Campbell and E. R. Hudson, arXiv preprint arXiv:1909.02668 (2019).
  * Safronova et al. (2018) M. S. Safronova, D. Budker, D. DeMille, D. F. J. Kimball, A. Derevianko, and C. W. Clark, Rev. Mod. Phys. 90, 025008 (2018).
  * Cairncross et al. (2017) W. B. Cairncross, D. N. Gresh, M. Grau, K. C. Cossel, T. S. Roussy, Y. Ni, Y. Zhou, J. Ye, and E. A. Cornell, Phys. Rev. Lett. 119, 153001 (2017).
  * Andreev et al. (2018) V. Andreev, D. G. Ang, D. DeMille, J. M. Doyle, G. Gabrielse, J. Haefner, N. R. Hutzler, Z. Lasner, C. Meisenhelder, B. R. O’Leary, et al., Nature 562, 355 (2018).
  * Alighanbari et al. (2020) S. Alighanbari, G. S. Giri, F. L. Constantin, V. I. Korobov, and S. Schiller, Nature 581, 152 (2020).
  * Chou et al. (2020) C. W. Chou, A. L. Collopy, C. Kurz, Y. Lin, M. E. Harding, P. N. Plessow, T. Fortier, S. Diddams, D. Leibfried, and D. R. Leibrandt, Science 367, 1458 (2020).
  * Viteau et al. (2008) M. Viteau, A. Chotia, M. Allegrini, N. Bouloufa, O. Dulieu, D. Comparat, and P. Pillet, Science 321, 232 (2008).
  * Staanum et al. (2010) P. F. Staanum, K. Højbjerre, P. S. Skyt, A. K. Hansen, and M. Drewsen, Nat. Phys. 6, 271 (2010).
  * Schneider et al. (2010) T. Schneider, B. Roth, H. Duncker, I. Ernsting, and S. Schiller, Nat. Phys. 6, 275 (2010).
  * Cournol et al. (2018) A. Cournol, P. Pillet, H. Lignier, and D. Comparat, Phys. Rev. A 97, 031401(R) (2018).
  * Lien et al. (2014) C.-Y. Lien, C. M. Seck, Y.-W. Lin, J. H. Nguyen, D. A. Tabor, and B. C. Odom, Nat. Commun. 5, 4783 (2014).
  * Rellergert et al. (2013) W. G. Rellergert, S. T. Sullivan, S. J. Schowalter, S. Kotochigova, K. Chen, and E. R. Hudson, Nature 495, 490 (2013), ISSN 1476-4687.
  * Hansen et al. (2014) A. K. Hansen, O. Versolato, S. B. Kristensen, A. Gingell, M. Schwarz, A. Windberger, J. Ullrich, J. C. López-Urrutia, M. Drewsen, et al., Nature 508, 76 (2014).
  * Tong et al. (2010) X. Tong, A. H. Winney, and S. Willitsch, Phys. Rev. Lett. 105, 143001 (2010).
  * Chou et al. (2017) C.-w. Chou, C. Kurz, D. B. Hume, P. N. Plessow, D. R. Leibrandt, and D. Leibfried, Nature 545, 203 (2017).
  * Shagam and Narevicius (2013) Y. Shagam and E. Narevicius, The Journal of Physical Chemistry C 117, 22454 (2013).
  * Ni et al. (2008) K.-K. Ni, S. Ospelkaus, M. De Miranda, A. Pe’Er, B. Neyenhuis, J. Zirbel, S. Kotochigova, P. Julienne, D. Jin, and J. Ye, Science 322, 231 (2008).
  * Bressel et al. (2012) U. Bressel, A. Borodin, J. Shen, M. Hansen, I. Ernsting, and S. Schiller, Phys. Rev. Lett. 108, 183003 (2012).
  * Werner et al. (1982) H.-J. Werner, P. Rosmus, and M. Grimm, Chem. Phys. 73, 169 (1982).
  * Cai and François (1999) Z.-L. Cai and J.-P. François, J. Mol. Spectrosc. 197, 12 (1999).
  * Chattopadhyaya et al. (2003) S. Chattopadhyaya, A. Chattopadhyay, and K. K. Das, J. Mol. Struct. THEOCHEM 639, 177 (2003).
  * Li et al. (2019) R. Li, X. Yuan, G. Liang, Y. Wu, J. Wang, and B. Yan, Chem. Phys. 525, 110412 (2019).
  * Di Rosa (2004) M. D. Di Rosa, Eur. Phys. J. D 31, 395 (2004), ISSN 1434-6079.
  * Sofikitis et al. (2009) D. Sofikitis, S. Weber, A. Fioretti, R. Horchani, M. Allegrini, B. Chatel, D. Comparat, and P. Pillet, New J. Phys. 11, 055037 (2009).
  * Nguyen and Odom (2011) J. H. V. Nguyen and B. Odom, Phys. Rev. A 83, 053404 (2011).
  * Nguyen et al. (2011) J. H. V. Nguyen, C. R. Viteri, E. G. Hohenstein, C. D. Sherrill, K. R. Brown, and B. Odom, New J. Phys. 13, 063023 (2011).
  * Stollenwerk et al. (2017) P. R. Stollenwerk, B. C. Odom, D. L. Kokkin, and T. Steimle, J. Mol. Spectrosc. 332, 26 (2017).
  * Lin and Odom (2016) Y.-W. Lin and B. C. Odom, arXiv:1610.04324 (2016).
  * Lien et al. (2011) C.-Y. Lien, S. R. Williams, and B. Odom, Phys. Chem. Chem. Phys. 13, 18825 (2011).
  * Stollenwerk et al. (2019) P. R. Stollenwerk, I. O. Antonov, and B. C. Odom, J. Mol. Spectrosc. 355, 40 (2019).
  * Baba and Waki (2001) T. Baba and I. Waki, J. Appl. Phys. 89, 4592 (2001).
  * Honjou (2003) N. Honjou, Mol. Phys. 101, 3063 (2003).
  * Shi et al. (2012) D. Shi, W. Li, W. Xing, J. Sun, Z. Zhu, and Y. Liu, Comput. Theor. Chem. 980, 73 (2012).
  * Sci (manuscript in preparation) (manuscript in preparation).
  * Western (2017) C. M. Western, J. Quant. Spectrosc. Radiat. Transfer 186, 221 (2017).
  * Willits et al. (2012) J. T. Willits, A. M. Weiner, and S. T. Cundiff, Opt. Express 20, 3110 (2012).
  * Scholl et al. (1995) T. J. Scholl, R. Cameron, S. D. Rosner, and R. A. Holt, Can. J. Phys. 73, 101 (1995).
  * Stollenwerk et al. (2018) P. R. Stollenwerk, M. G. Kokish, A. G. S. de Oliveira-Filho, F. R. Ornellas, and B. C. Odom, Atoms 6, 53 (2018).