#  TableFormer: Robust Transformer Modeling for Table-Text Encoding

Jingfeng Yang Aditya Gupta† Shyam Upadhyay†
Luheng He† Rahul Goel† Shachi Paul †
⋆Georgia Institute of Technology
†Google Assistant
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Work done during an internship at Google.

###### Abstract

Understanding tables is an important aspect of natural language understanding.
Existing models for table understanding require linearization of the table
structure, where row or column order is encoded as an _unwanted_ bias. Such
spurious biases make the model vulnerable to row and column order
perturbations. Additionally, prior work has not thoroughly modeled the table
structures or table-text alignments, hindering the table-text understanding
ability. In this work, we propose a robust and structurally aware table-text
encoding architecture TableFormer, where tabular structural biases are
incorporated completely through learnable attention biases. TableFormer is (1)
strictly invariant to row and column orders, and, (2) could understand tables
better due to its tabular inductive biases. Our evaluations showed that
TableFormer outperforms strong baselines in all settings on SQA, WTQ and
TabFact table reasoning datasets, and achieves state-of-the-art performance on
SQA, especially when facing answer-invariant row and column order
perturbations (6% improvement over the best baseline), because previous SOTA
models’ performance drops by 4% - 6% when facing such perturbations while
TableFormer is not affected.111Code has been released at
https://github.com/google-research/tapas/blob/master/TABLEFORMER.md

##  1 Introduction

Recently, semi-structured data (e.g. variable length tables without a fixed
data schema) has attracted more attention because of its ubiquitous presence
on the web. On a wide range of various table reasoning tasks, Transformer
based architecture along with pretraining has shown to perform well
Eisenschlos et al. (2021); Liu et al. (2021).

Title | Length
---|---
Screwed Up | 5:02
Ghetto Queen | 5:00
(a) Tapas predicts incorrect answer based on the original table, while it
gives the correct answer if the first row is moved to the end of the table.

Question: Of all song lengths, which one is the longest?
Gold Answer: 5:02
TAPAS: 5:00
TAPAS after row order perturbation: 5:02
TableFormer: 5:02

Nation | Gold | Silver | Bronze
---|---|---|---
Great Britain | 2 | 1 | 2
Spain | 1 | 2 | 0
Ukraine | 0 | 2 | 0
(b) Tapas gives incomplete answer due to its limited cell grounding ability.

Question: Which nation received 2 silver medals?
Gold Answer: Spain, Ukraine
TAPAS: Spain
TableFormer: Spain, Ukraine
TableFormer w/o a proposed structural bias: Spain

Figure 1:  Examples showing the limitations of existing models (a) vulnerable
to perturbations, and (b) lacking structural biases. In contrast, our proposed
TableFormer predicts correct answers for both questions.

In a nutshell, prior work used the Transformer architecture in a BERT like
fashion by serializing tables or rows into word sequences Yu et al. (2020);
Liu et al. (2021), where original position ids are used as positional
information. Due to the usage of row/column ids and global position ids, prior
strategies to linearize table structures introduced spurious row and column
order biases Herzig et al. (2020); Eisenschlos et al. (2020, 2021); Zhang et
al. (2020); Yin et al. (2020). Therefore, those models are vulnerable to row
or column order perturbations. But, ideally, the model should make consistent
predictions regardless of the row or column ordering for all practical
purposes. For instance, in Figure 1, the predicted answer of Tapas model
Herzig et al. (2020) for Question (a) _“Of all song lengths, which one is the
longest?”_ based on the original table is _“5:00”_ , which is incorrect.
However, if the first row is adjusted to the end of the table during
inference, the model gives the correct length _“5:02”_ as answer. This probing
example shows that the model being aware of row order information is inclined
to select length values to the end of the table due to spurious training data
bias. In our experiments on the SQA dataset, Tapas models exhibit a 4% - 6%
(Section 5.2) absolute performance drop when facing such answer-invariant
perturbations.

Besides, most prior work Chen et al. (2020); Yin et al. (2020) did not
incorporate enough structural biases to models to address the limitation of
sequential Transformer architecture, while others inductive biases which are
either too strict Zhang et al. (2020); Eisenschlos et al. (2021) or
computationally expensive Yin et al. (2020).

To this end, we propose TableFormer, a Transformer architecture that is robust
to row and column order perturbations, by incorporating structural biases more
naturally. TableFormer relies on 13 types of task-independent
table$\leftrightarrow$text attention biases that respect the table structure
and table-text relations. For Question (a) in Figure 1, TableFormer could
predict the correct answer regardless of perturbation, because the model could
identify the same row information with our _“same row”_ bias, avoiding
spurious biases introduced by row and global positional embeddings. For
Question (b), Tapas predicted only partially correct answer, while TableFormer
could correctly predict _“Spain, Ukraine”_ as answers. That’s because our
_“cell to sentence”_ bias could help table cells ground to the paired
sentence. Detailed attention bias types are discussed in Section 5.2.

Experiments on 3 table reasoning datasets show that TableFormer consistently
outperforms original Tapas in all pretraining and intermediate pretraining
settings with fewer parameters. Also, TableFormer’s invariance to row and
column perturbations, leads to even larger improvement over those strong
baselines when tested on perturbations. Our contributions are as follows:

  * •

We identified the limitation of current table-text encoding models when facing
row or column perturbation.

  * •

We propose TableFormer, which is guaranteed to be invariant to row and column
order perturbations, unlike current models.

  * •

TableFormer encodes table-text structures better, leading to SoTA performance
on SQA dataset, and ablation studies show the effectiveness of the introduced
inductive biases.

##  2 Preliminaries: Tapas for Table Encoding

In this section, we discuss TAPAS which serves as the backbone of the recent
state-of-the-art table-text encoding architectures. Tapas Herzig et al. (2020)
uses Transformer architecture in a BERT like fashion to pretrain and finetune
on tabular data for table-text understanding tasks. This is achieved by using
linearized table and texts for masked language model pre-training. In the
fine-tuning stage, texts in the linearized table and text pairs are queries or
statements in table QA or table-text entailment tasks, respectively.

Specifically, Tapas uses the tokenized and flattened text and table as input,
separated by [SEP] token, and prefixed by [CLS]. Besides token, segment, and
global positional embedding introduced in BERT Devlin et al. (2019), it also
uses rank embedding for better numerical understanding. Moreover, it uses
column and row embedding to encode table structures.

Concretely, for any table-text linearized sequence
$S=\\{v_{1},v_{2},\cdots,v_{n}\\}$, where $n$ is the length of table-text
sequence, the input to Tapas is summation of embedding of the following:

$\begin{split}\text{token ids
}(W)&=\\{w_{v_{1}},w_{v_{2}},\cdots,w_{v_{n}}\\}\\\ \text{positional ids
}(B)&=\\{b_{1},b_{2},\cdots,b_{n}\\}\\\ \text{segment ids
}(G)&=\\{g_{seg_{1}},g_{seg_{2}},\cdots,g_{seg_{n}}\\}\\\ \text{column ids
}(C)&=\\{c_{col_{1}},c_{col_{2}},\cdots,c_{col_{n}}\\}\\\ \text{row ids
}(R)&=\\{r_{row_{1}},r_{row_{2}},\cdots,r_{row_{n}}\\}\\\ \text{rank ids
}(Z)&=\\{z_{rank_{1}},z_{rank_{2}},\cdots,z_{rank_{n}}\\}\end{split}$

where $seg_{i},~{}col_{i},~{}row_{i},~{}rank_{i}$ correspond to the segment,
column, row, and rank id for the $i$th token, respectively.

Figure 2:  TableFormer input and attention biases in the self attention
module. This example corresponds to table (a) in Figure 1 and its paired
question “query”. Different colors in the attention bias matrix denote
different types of task independent biases derived based on the table
structure and the associated text.

As for the model, Tapas uses BERT’s self-attention architecture Vaswani et al.
(2017) off-the-shelf. Each Transformer layer includes a multi-head self-
attention sub-layer, where each token attends to all the tokens. Let the layer
input $H=[h_{1},h_{2},\cdots,h_{n}]^{\top}\in\mathbb{R}^{n\times d}$
corresponding to $S$, where $d$ is the hidden dimension, and
$h_{i}\in\mathbb{R}^{d\times 1}$ is the hidden representation at position $i$.
For a single-head self-attention sub-layer, the input $H$ is projected by
three matrices $W^{Q}\in\mathbb{R}^{d\times d_{K}}$,
$W^{K}\in\mathbb{R}^{d\times d_{K}}$, and $W^{V}\in\mathbb{R}^{d\times d_{V}}$
to the corresponding representations $Q$, $K$, and $V$:

$\displaystyle Q=HW^{Q},\quad V=HW^{V},\quad K=HW^{K}$ (1)

Then, the output of this single-head self-attention sub-layer is calculated
as:

$\displaystyle\text{Attn}(H)=\text{softmax}(\frac{QK^{\top}}{\sqrt{d_{K}}})V$
(2)

##  3 TableFormer: Robust Structural Table Encoding

As shown in Figure 2, TableFormer encodes the general table structure along
with the associated text by introducing task-independent relative attention
biases for table-text encoding to facilitate the following: (a) structural
inductive bias for better table understanding and table-text alignment, (b)
robustness to table row/column perturbation.

#### Input of TableFormer.

TableFormer uses the same token embeddings $W$, segment embeddings $G$, and
rank embeddings $Z$ as Tapas. However, we make 2 major modifications:

#### 1) No row or column ids.

We do not use row embeddings $R$ or column embeddings $C$ to avoid any
potential spurious row and column order biases.

#### 2) Per cell positional ids.

To further remove any inter-cell order information, global positional
embeddings $B$ are replaced by per cell positional embeddings
$P=\\{p_{pos_{1}},p_{pos_{2}},\cdots,p_{pos_{n}}\\}$, where we follow
Eisenschlos et al. (2021) to reset the index of positional embeddings at the
beginning of each cell, and $pos_{i}$ correspond to the per cell positional id
for the $i$th token.

#### Positional Encoding in TableFormer.

Note that the Transformer model either needs to specify different positions in
the input (i.e. absolute positional encoding of Vaswani et al. (2017)) or
encode the positional dependency in the layers (i.e. relative positional
encoding of Shaw et al. (2018)).

TableFormer does not consume any sort of column and row order information in
the input. The main intuition is that, for cells in the table, the only useful
positional information is whether two cells are in the same row or column and
the column header of each cell, instead of the absolute order of the row and
column containing them. Thus, inspired by relative positional encoding Shaw et
al. (2018) and graph encoding Ying et al. (2021), we capture this with a same
column/row relation as one kind of relative position between two linearized
tokens. Similarly, we uses 12 such table-text structure relevant relations
(including same cell, cell to header and so on) and one extra type
representing all other relations not explicitly defined. All of them are
introduced in the form of learnable attention bias scalars.

Formally, we consider a function $\phi(v_{i},v_{j}):V\times
V\rightarrow\mathbb{N}$, which measures the relation between $v_{i}$ and
$v_{j}$ in the sequence ($v_{i},v_{j}\in S$). The function $\phi$ can be
defined by any relations between the tokens in the table-text pair.

#### Attention Biases in TableFormer.

In our work, $\phi(v_{i},v_{j})$ is chosen from 13 bias types, corresponding
to 13 table-text structural biases. The attention biases are applicable to any
table-text pair and can be used for any downstream task:

  * •

_“same row”_ identifies the same row information without ordered row id
embedding or global positional embedding, which help the model to be invariant
to row perturbations,

  * •

_“same column”_ , _“header to column cell”_ , and _“cell to column header”_
incorporates the same column information without ordered column id embedding,

  * •

_“cell to column header”_ makes each cell aware of its column header without
repeated column header as features,

  * •

_“header to sentence”_ and _“cell to sentence”_ help column grounding and cell
grounding of the paired text,

  * •

_“sentence to header”_ , _“sentence to cell”_ , and _“sentence to sentence”_
helps to understand the sentence with the table as context,

  * •

_“header to same header”_ and _“header to other header”_ for better
understanding of table schema, and _“same cell bias”_ for cell content
understanding.

Note that, each cell can still attend to other cells in the different columns
or rows through _“others”_ instead of masking them out strictly.

We assign each bias type a learnable scalar, which will serve as a bias term
in the self-attention module. Specifically, each self-attention head in each
layer have a set of learnable scalars $\\{b_{1},b_{2},\cdots,b_{13}\\}$
corresponding to all types of introduced biases. For one head in one self-
attention sub-layer of TableFormer, Equation 2 in the Transformer is replaced
by:

$\displaystyle\bar{A}=\frac{QK^{\top}}{\sqrt{d_{K}}},\quad A=\bar{A}+\hat{A}$
(3) $\displaystyle\text{Attn}(H)=\text{softmax}(A)V$ (4)

where $\bar{A}$ is a matrix capturing the similarity between queries and keys,
$\hat{A}$ is the Attention Bias Matrix, and
$\hat{A}_{i,j}=b_{\phi(v_{i},v_{j})}$.

#### Relation between TableFormer and ETC.

ETC Ainslie et al. (2020) uses vectors to represent relative position labels,
although not directly applied to table-text pairs due to its large
computational overhead Eisenschlos et al. (2021). TableFormer differs from ETC
in the following aspects (1) ETC uses relative positional embeddings while
TableFormer uses attention bias scalars. In practice, we observed that using
relative positional embeddings increases training time by more than 7x, (2)
ETC uses global memory and local attention, while TableFormer uses pairwise
attention without any global memory overhead, (3) ETC uses local sparse
attention with masking, limiting its ability to attend to all tokens, (4) ETC
did not explore table-text attention bias types exhaustively. Another table
encoding model MATE Eisenschlos et al. (2021) is vulnerable to row and column
perturbations, and shares limitation (3) and (4).

##  4 Experimental Setup

###  4.1 Datasets and Evaluation

We use the following datasets in our experiments.

#### Table Question Answering.

For the table QA task, we conducted experiments on WikiTableQuestions (WTQ)
Pasupat and Liang (2015) and Sequential QA (SQA) Iyyer et al. (2017) datasets.
WTQ was crowd-sourced based on complex questions on Wikipedia tables. SQA is
composed of $6,066$ question sequences (2.9 question per sequence on average),
constructed by decomposing a subset of highly compositional WTQ questions.

#### Table-Text Entailment.

For the table-text entailment task, we used TabFact dataset Chen et al.
(2020), where the tables were extracted from Wikipedia and the sentences were
written by crowd workers. Among total 118, 000 sentences, each one is a
positive (entailed) or negative sentence.

#### Perturbation Evaluation Set.

For SQA and TabFact, we also created new test sets to measure models’
robustness to answer-invariant row and column perturbations during inference.
Specifically, row and column orders are randomly perturbed for all tables in
the standard test sets.222We fixed perturbation random seeds to make our
results reproducible.

#### Pre-training

All the models are first tuned on the _Wikipidia_ text-table pretraining
dataset Herzig et al. (2020), optionally tuned on synthetic dataset at an
intermediate stage (“inter”) Eisenschlos et al. (2020), and finally fine-tuned
on the target dataset. To get better performance on WTQ, we follow Herzig et
al. (2020) to further pretrain on SQA dataset after the intermediate
pretraining stage in the “inter-sqa” setting.

#### Evaluation

For SQA, we report the cell selection accuracy for all questions (ALL) using
the official evaluation script, cell selection accuracy for all sequences
(SEQ), and the denotation accuracy for all questions
($\text{ALL}_{\text{d}}$). To evaluate the models’ robustness in the instance
level after perturbations, we also report a lower bound of example prediction
variation percentage:

$VP=\frac{(\text{t2f}+\text{f2t})}{(\text{t2t}+\text{t2f}+\text{f2t}+\text{f2f})}$
(5)

where t2t, t2f, f2t, and f2f represents how many example predictions turning
from correct to correct, from correct to incorrect, from incorrect to correct
and from incorrect to incorrect, respectively, after perturbation. We report
denotation accuracy on WTQ and binary classification accuracy on TabFact
respectively.

###  4.2 Baselines

We use $\textsc{Tapas}_{\text{BASE}}$ and $\textsc{Tapas}_{\text{LARGE}}$ as
baselines, where Transformer architectures are exactly same as
$\textsc{BERT}_{\text{BASE}}$ and $\textsc{BERT}_{\text{LARGE}}$ Devlin et al.
(2019), and parameters are initialized from $\textsc{BERT}_{\text{BASE}}$ and
$\textsc{BERT}_{\text{LARGE}}$ respectively. Correspondingly, we have our
$\textsc{TableFormer}_{\text{BASE}}$ and
$\textsc{TableFormer}_{\text{LARGE}}$, where attention bias scalars are
initialized to zero, and all other parameters are initialized from
$\textsc{BERT}_{\text{BASE}}$ and $\textsc{BERT}_{\text{LARGE}}$.

###  4.3 Perturbing Tables as Augmented Data

_Could we alleviate the spurious ordering biases by data augmentation alone,
without making any modeling changes?_ To answer this, we train another set of
models by augmenting the training data for TAPAS through random row and column
order perturbations.333By perturbation, we mean shuffling row and columns
instead of changing/swapping content blindly.

For each table in the training set, we randomly shuffle all rows and columns
(including corresponding column headers), creating a new table with the same
content but different orders of rows and columns. Multiple perturbed versions
of the same table were created by repeating this process $\\{1,2,4,8,16\\}$
times with different random seeds. For table QA tasks, selected cell positions
are also adjusted as final answers according to the perturbed table. The
perturbed table-text pairs are then used to augment the data used to train the
model. During training, the model takes data created by one specific random
seed in one epoch in a cyclic manner.

| Before Perturb | After Perturb
---|---|---
| ALL | SEQ | $\text{ALL}_{\text{d}}$ | ALL | $VP$
Herzig et al. (2020) | 67.2 | 40.4 | – | – | –
Eisenschlos et al. (2020) | 71.0 | 44.8 | – | – | –
Eisenschlos et al. (2021) | 71.7 | 46.1 | – | – | –
Liu et al. (2021) | – | – | 74.5 | – | –
$\textsc{Tapas}_{\text{BASE}}$ | 61.1 | 31.3 | – | 57.4 | 14.0%
$\textsc{TableFormer}_{\text{BASE}}$ | 66.7 | 39.7 | – | 66.7 | 0.2%
$\textsc{Tapas}_{\text{LARGE}}$ | 66.8 | 39.9 | – | 60.5 | 15.1%
$\textsc{TableFormer}_{\text{LARGE}}$ | 70.3 | 44.8 | – | 70.3 | 0.1%
$\textsc{Tapas}_{\text{BASE}}$ inter | 67.5 | 38.8 | – | 61.0 | 14.3%
$\textsc{TableFormer}_{\text{BASE}}$ inter | 69.4 | 43.5 | – | 69.3 | 0.1%
$\textsc{Tapas}_{\text{LARGE}}$ inter | 70.6 | 43.9 | – | 66.1 | 10.8%
$\textsc{TableFormer}_{\text{LARGE}}$ inter | 72.4 | 47.5 | 75.9 | 72.3 | 0.1%
Table 1:  Results on SQA test set before and after perturbation during
inference (median of 5 runs). ALL is cell selection accuracy, SEQ is cell
selection accuracy for all question sequences, $\text{ALL}_{\text{d}}$ is
denotation accuracy for all questions (reported to compare with Liu et al.
(2021)). $VP$ is model prediction variation percentage after perturbation.
Missing values are those not reported in the original paper.

##  5 Experiments and Results

| Before Perturb | After Perturb
---|---|---
| dev | test | $\text{test}_{\text{simple}}$ | $\text{test}_{\text{complex}}$ | $\text{test}_{\text{small}}$ | test | $\text{test}_{\text{simple}}$ | $\text{test}_{\text{complex}}$ | $\text{test}_{\text{small}}$
Eisenschlos et al. (2020) | 81.0 | 81.0 | 92.3 | 75.6 | 83.9 | – | – | – | –
Eisenschlos et al. (2021) | – | 81.4 | – | – | – | – | – | – | –
$\textsc{Tapas}_{\text{BASE}}$ | 72.8 | 72.3 | 84.8 | 66.2 | 74.4 | 71.2 | 83.4 | 65.2 | 72.5
$\textsc{TableFormer}_{\text{BASE}}$ | 75.1 | 75.0 | 88.2 | 68.5 | 77.1 | 75.0 | 88.2 | 68.5 | 77.1
$\textsc{Tapas}_{\text{LARGE}}$ | 74.7 | 74.5 | 86.6 | 68.6 | 76.8 | 73.7 | 86.0 | 67.7 | 76.1
$\textsc{TableFormer}_{\text{LARGE}}$ | 77.2 | 77.0 | 90.2 | 70.5 | 80.3 | 77.0 | 90.2 | 70.5 | 80.3
$\textsc{Tapas}_{\text{BASE}}$ inter | 78.4 | 77.9 | 90.1 | 71.9 | 80.5 | 76.8 | 89.5 | 70.5 | 79.7
$\textsc{TableFormer}_{\text{BASE}}$ inter | 79.7 | 79.2 | 91.6 | 73.1 | 81.7 | 79.2 | 91.6 | 73.1 | 81.7
$\textsc{Tapas}_{\text{LARGE}}$ inter | 80.6 | 80.6 | 92.0 | 74.9 | 83.1 | 79.2 | 91.7 | 73.0 | 83.0
$\textsc{TableFormer}_{\text{LARGE}}$ inter | 82.0 | 81.6 | 93.3 | 75.9 | 84.6 | 81.6 | 93.3 | 75.9 | 84.6
Table 2: Binary classification accuracy on TabFact development and 4 splits of test set, as well as performance on test sets with our perturbation evaluation. Median of 5 independent runs are reported. Missing values are those not reported in the original paper. Model | dev | test
---|---|---
Herzig et al. (2020) | – | 48.8
Eisenschlos et al. (2021) | – | 51.5
$\textsc{Tapas}_{\text{BASE}}$ | 23.6 | 24.1
$\textsc{TableFormer}_{\text{BASE}}$ | 34.4 | 34.8
$\textsc{Tapas}_{\text{LARGE}}$ | 40.8 | 41.7
$\textsc{TableFormer}_{\text{LARGE}}$ | 42.5 | 43.9
$\textsc{Tapas}_{\text{BASE}}$ inter-sqa | 44.8 | 45.1
$\textsc{TableFormer}_{\text{BASE}}$ inter-sqa | 46.7 | 46.5
$\textsc{Tapas}_{\text{LARGE}}$ inter-sqa | 49.9 | 50.4
$\textsc{TableFormer}_{\text{LARGE}}$ inter-sqa | 51.3 | 52.6
Table 3:  Denotation accuracy on WTQ development and test set. Median of 5
independent runs are reported.

Besides standard testing results to compare TableFormer and baselines, we also
answer the following questions through experiments:

  * •

How robust are existing (near) state-of-the-art table-text encoding models to
semantic preserving perturbations in the input?

  * •

How does TableFormer compare with existing table-text encoding models when
tested on similar perturbations, both in terms of performance and robustness?

  * •

Can we use perturbation based data augmentation to achieve robustness at test
time?

  * •

Which attention biases in TableFormer contribute the most to performance?

###  5.1 Main Results

Table 1, 2, and 3 shows TableFormer performance on SQA, TabFact, and WTQ,
respectively. As can be seen, TableFormer outperforms corresponding Tapas
baseline models in all settings on SQA and WTQ datasets, which shows the
general effectiveness of TableFormer’s structural biases in Table QA datasets.
Specifically, $\textsc{TableFormer}_{\text{LARGE}}$ combined with intermediate
pretraining achieves new state-of-the-art performance on SQA dataset.

Similarly, Table 2 shows that TableFormer also outperforms Tapas baseline
models in all settings, which shows the effectiveness of TableFormer in the
table entailment task. Note that, Liu et al. (2021) is not comparable to our
results, because they used different pretraining data, different pretraining
objectives, and BART NLG model instead of BERT NLU model. But TableFormer
attention bias is compatible with BART model.

###  5.2 Perturbation Results

One of our major contributions is to systematically evaluate models’
performance when facing row and column order perturbation in the testing
stage.

Ideally, model predictions should be consistent on table QA and entailment
tasks when facing such perturbation, because the table semantics remains the
same after perturbation.

However, in Table 1 and 2, we can see that in our perturbed test set,
performance of all Tapas models drops significantly in both tasks. Tapas
models drops by at least 3.7% and up to 6.5% in all settings on SQA dataset in
terms of ALL accuracy, while our TableFormer being strictly invariant to such
row and column order perturbation leads to no drop in performance.444In SQA
dataset, there is at most absolute 0.1% performance drop because of some bad
data point issues. Specifically, some columns in certain tables are exactly
the same, but the ground-truth selected cells are in only one of such columns.
TableFormer would select from one column randomly.  Thus, in the perturbation
setting, TableFormer outperforms all Tapas baselines even more significantly,
with at least 6.2% and 2.4% improvements on SQA and TabFact dataset,
respectively. In the instance level, we can see that, with Tapas, there are
many example predictions changed due to high $VP$, while there is nearly no
example predictions changed with TableFormer (around zero $VP$).

###  5.3 Model Size Comparison

We compare the model sizes of TableFormer and Tapas in Table 4. We added only
a few attention bias scalar parameters (13 parameters per head per layer) in
TableFormer, which is negligible compared with the BERT model size. Meanwhile,
we delete two large embedding metrics (512 row ids and 512 column ids). Thus,
TableFormer outperforms Tapas with fewer parameters.

Model | Number of parameters
---|---
$\textsc{Tapas}_{\text{BASE}}$ | 110 M
$\textsc{TableFormer}_{\text{BASE}}$ |  110 M - 2*512*768 \+ 12*12*13 = 110 M - 0.8 M + 0.002 M
$\textsc{Tapas}_{\text{LARGE}}$ | 340 M
$\textsc{TableFormer}_{\text{LARGE}}$ |  340 M - 2*512*1024 \+ 24*16*13 = 340 M - 1.0 M + 0.005M
Table 4: Model size comparison.

###  5.4 Analysis of TableFormer Submodules

In this section, we experiment with several variants of TableFormer to
understand the effectiveness of its submodules. The performance of all
variants of Tapas and TableFormer that we tried on the SQA development set is
shown in Table 5.

#### Learnable Attention Biases v/s Masking.

Instead of adding learnable bias scalars, we mask out some attention scores to
restrict attention to those tokens in the same columns and rows, as well as
the paired sentence, similar to Zhang et al. (2020) (SAT). We can see that
$\textsc{Tapas}_{\text{BASE-SAT}}$ performs worse than
$\textsc{Tapas}_{\text{BASE}}$, which means that restricting attention to only
same columns and rows by masking reduce the modeling capacity. This led to
choosing soft bias addition over hard masking.

#### Attention Bias Scaling.

Unlike TableFormer, we also tried to add attention biases before the scaling
operation in the self-attention module (SO). Specifically, we compute pair-
wise attention score by:

$\displaystyle
A_{ij}=\frac{(h_{i}^{\top}W^{Q})(h_{j}^{\top}W^{K})^{\top}+\hat{A}_{ij}}{\sqrt{d_{K}}}$
(6)

instead of using:

$\displaystyle
A_{ij}=\frac{(h_{i}^{\top}W^{Q})(h_{j}^{\top}W^{K})^{\top}}{\sqrt{d_{K}}}+\hat{A}_{ij},$
(7)

which is the element-wise version of Equation 1 and 3. However, Table 5 shows
that $\textsc{TableFormer}_{\text{BASE-SO}}$ performs worse than
$\textsc{TableFormer}_{\text{BASE}}$, showing the necessity of adding
attention biases after the scaling operation. We think the reason is that the
attention bias term does not require scaling, because attention bias scalar
magnitude is independent of $d_{K}$, while the dot products grow large in
magnitude for large values of $d_{K}$. Thus, such bias term could play an more
important role without scaling, which helps each attention head know clearly
what to pay more attention to according to stronger inductive biases.

| rc-gp | c-gp | gp | pcp
---|---|---|---|---
$\textsc{Tapas}_{\text{BASE}}$ | 57.6 | 47.4 | 46.4 | 29.1
$\textsc{Tapas}_{\text{BASE-SAT}}$ | 45.2 | - | - | -
$\textsc{TableFormer}_{\text{BASE-SO}}$ | 60.0 | 60.2 | 59.8 | 60.7
$\textsc{TableFormer}_{\text{BASE}}$ | 62.2 | 61.5 | 61.7 | 61.9
Table 5: ALL questions’ cell selection accuracy of TableFormer variants on SQA
development set. rc-gp represents the setting including row ids, column ids
and global positional ids, c-gp represents column ids and global positional
ids, gp represents global positional ids, and pcp represents per-cell
positional ids. “SAT” represents masking out some attention scores. “SO”
represents adding attention bias before scaling.

#### Row, Column, & Global Positional IDs.

With $\textsc{Tapas}_{\text{BASE}}$, $\textsc{TableFormer}_{\text{BASE-SO}}$,
and $\textsc{TableFormer}_{\text{BASE}}$, we first tried the full-version
where row ids, column ids, and global positional ids exist as input (rc-gp).
Then, we deleted row ids (c-gp), and column ids (gp) sequentially. Finally, we
changed global positional ids in gp to per-cell positional ids (pcp). Table 5
shows that $\textsc{Tapas}_{\text{BASE}}$ performs significantly worse from
rc-gp $\rightarrow$ c-gp $\rightarrow$ gp $\rightarrow$ pcp, because table
structure information are deleted sequentially during such process. However,
with $\textsc{TableFormer}_{\text{BASE}}$, there is no obvious performance
drop during the same process. That shows the structural inductive biases in
TableFormer can provide complete table structure information. Thus, row ids,
column ids and global positional ids are not necessary in TableFormer. We pick
TableFormer pcp setting as our final version to conduct all other experiments
in this paper. In this way, TableFormer is strictly invariant to row and
column order perturbation by avoiding spurious biases in those original ids.

###  5.5 Comparison of TableFormer and Perturbed Data Augmentation

| Befor Perturb | After Perturb
---|---|---
| ALL | SEQ | ALL | $VP$
$\textsc{Tapas}_{\text{BASE}}$ | 61.1 | 31.3 | 57.4 | 14.0%
$\textsc{Tapas}_{\text{BASE}}$ 1p | 63.4 | 34.6 | 63.4 | 9.9%
$\textsc{Tapas}_{\text{BASE}}$ 2p | 64.6 | 35.6 | 64.5 | 8.4%
$\textsc{Tapas}_{\text{BASE}}$ 4p | 65.1 | 37.0 | 65.0 | 8.1%
$\textsc{Tapas}_{\text{BASE}}$ 8p | 65.1 | 37.3 | 64.3 | 7.2%
$\textsc{Tapas}_{\text{BASE}}$ 16p | 62.4 | 33.6 | 62.2 | 7.0%
$\textsc{TableFormer}_{\text{BASE}}$ | 66.7 | 39.7 | 66.7 | 0.1%
Table 6: Comparison of TableFormer and perturbed data augmentation on SQA test
set, where $VP$ represents model prediction variation percentage after
perturbation. Median of 5 independent runs are reported.

As stated in Section 4.3, perturbing row and column orders as augmented data
during training can serve as another possible solution to alleviate the
spurious row/column ids bias. Table 6 shows the performance of
$\textsc{Tabpas}_{\text{BASE}}$ model trained with additional {1, 2, 4, 8, 16}
perturbed versions of each table as augmented data.

We can see that the performance of $\textsc{Tapas}_{\text{BASE}}$ on SQA
dataset improves with such augmentation. Also, as the number of perturbed
versions of each table increases, model performance first increases and then
decreases, reaching the best results with 8 perturbed versions. We suspect
that too many versions of the same table confuse the model about different row
and column ids for the same table, leading to decreased performance from 8p to
16p. Despite its usefulness, such data perturbation is still worse than
TableFormer, because it could not incorporate other relevant text-table
structural inductive biases like TableFormer.

Although, such data augmentation makes the model more robust to row and column
order perturbation with smaller $VP$ compared to standard
$\textsc{Tapas}_{\text{BASE}}$, there is still a significant prediction drift
after perturbation. As shown in Table 6, $VP$ decreases from 1p to 16p,
however, the best $VP$ (7.0%) is still much higher than (nearly) no variation
(0.1%) of TableFormer.

To sum up, TableFormer is superior to row and column order perturbation
augmentation, because of its additional structural biases and strictly
consistent predictions after perturbation.

| ALL | SEQ
---|---|---
$\textsc{TableFormer}_{\text{BASE}}$ | 62.1 | 38.4
\- Same Row | 32.1 | 2.8
\- Same Column | 62.1 | 37.7
\- Same Cell | 61.8 | 38.4
\- Cell to Column Header | 60.7 | 36.6
\- Cell to Sentence | 60.5 | 36.4
\- Header to Column Cell | 60.5 | 35.8
\- Header to Other Header | 60.6 | 35.8
\- Header to Same Header | 61.0 | 36.9
\- Header to Sentence | 61.1 | 36.3
\- Sentence to Cell | 60.8 | 36.2
\- Sentence to Header | 61.0 | 37.3
\- Sentence to Sentence | 60.0 | 35.3
\- All Column Related (# 2, 4, 6) | 54.5 | 29.3
Table 7: Ablation study of proposed attention biases.

###  5.6 Attention Bias Ablation Study

We conduct ablation study to demonstrate the utility of all 12 types of
defined attention biases. For each ablation, we set the corresponding
attention bias type id to _“others”_ bias id. Table 7 shows
$\textsc{Tapas}_{\text{BASE}}$’s performance SQA dev set. Overall, all types
of attention biases help the TableFormer performance to some extent, due to
certain performance drop after deleting each bias type.

Amongst all the attention biases, deleting _“same row”_ bias leads to most
significant performance drop, showing its crucial role for encoding table row
structures. There is little performance drop after deleting _“same column”_
bias, that’s because TableFormer could still infer the same column information
through _“cell to its column header”_ and _“header to its column cell”_
biases. After deleting all same column information (_“same column”_ , _“cell
to column header”_ and _“header to column cell”_ biases), TableFormer performs
significantly worse without encoding column structures. Similarly, there is
little performance drop after deleting _“same cell”_ bias, because TableFormer
can still infer same cell information through _“same row”_ and _“same column”_
biases.

###  5.7 Limitations of TableFormer

TableFormer increases the training time by around $20\%$, which might not be
ideal for very long tables and would require a scoped approach. Secondly, with
the strict row and column order invariant property, TableFormer cannot deal
with questions based on absolute orders of rows in tables. This however is not
a practical requirement based on the current dataset. Doing a manual study of
1800 questions in SQA dataset, we found that there are 4 questions555We find
such 4 questions by manually looking at all 125 questions where the model
predictions turn from correct to incorrect after replacing
$\textsc{Tapas}_{\text{LARGE}}$ with $\textsc{TableFormer}_{\text{LARGE}}$.
(0.2% percentage) whose answers depend on orders of rows. Three of them asked
_“which one is at the top of the table”_ , another asks _“which one is listed
first”_. However, these questions could be potentially answered by adding back
row and column order information based on TableFormer.

##  6 Other Related Work

#### Transformers for Tabular Data.

Yin et al. (2020) prepended corresponding column headers to cells contents,
and Chen et al. (2020) used corresponding column headers as features for
cells. However, such methods encode each table header multiple times, leading
to duplicated computing overhead. Also, tabular structures (e.g. same row
information) are not fully incorporated to such models. Meanwhile, Yin et al.
(2020) leveraged row encoder and column encoder sequentially, which introduced
much computational overhead, thus requiring retrieving some rows as a
preprocessing step. Finally, SAT Zhang et al. (2020), Deng et al. (2021) and
Wang et al. (2021) restricted attention to same row or columns with attention
mask, where such inductive bias is too strict that cells could not directly
attend to those cells in different row and columns, hindering the modeling
ability according to Table 5. Liu et al. (2021) used the seq2seq BART
generation model with a standard Transformer encoder-decoder architecture. In
all models mentioned above, spurious inter-cell order biases still exist due
to global positional ids of Transformer, leading to the vulnerability to row
or column order perturbations, while our TableFormer could avoid such problem.
Mueller et al. (2019) and Wang et al. (2020) also used relative positional
encoding to encode table structures, but they modeled the relations as
learnable relation vectors, whose large overhead prevented pretraining and led
to poor performance without pretraining, similarly to ETC Ainslie et al.
(2020) explained in Section 3.

#### Structural and Relative Attention.

Modified attention scores has been used to model relative positions Shaw et
al. (2018), long documents Dai et al. (2019); Beltagy et al. (2020); Ainslie
et al. (2020), and graphs Ying et al. (2021). But adding learnable attention
biases to model tabular structures has been under-explored.

##  7 Conclusion

In this paper, we identified the vulnerability of prior table encoding models
along two axes: (a) capturing the structural bias, and (b) robustness to row
and column perturbations. To tackle this, we propose TableFormer, where
learnable task-independent learnable structural attention biases are
introduced, while making it invariant to row/column order at the same time.
Experimental results showed that TableFormer outperforms strong baselines in 3
table reasoning tasks, achieving state-of-the-art performance on SQA dataset,
especially when facing row and column order perturbations, because of its
invariance to row and column orders.

## Acknowledgments

We thank Julian Eisenschlos, Ankur Parikh, and the anonymous reviewers for
their feedbacks in improving this paper.

## Ethical Considerations

The authors foresee no ethical concerns with the research presented in this
paper.

## References

  * Ainslie et al. (2020) Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020.  ETC: Encoding long and structured inputs in transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 268–284, Online. Association for Computational Linguistics.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv:2004.05150_.
  * Chen et al. (2020) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020.  Tabfact : A large-scale dataset for table-based fact verification.  In _International Conference on Learning Representations (ICLR)_ , Addis Ababa, Ethiopia.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019.  Transformer-XL: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2978–2988, Florence, Italy. Association for Computational Linguistics.
  * Deng et al. (2021) Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2021.  TURL: Table Understanding through Representation Learning.  In _VLDB_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Eisenschlos et al. (2021) Julian Eisenschlos, Maharshi Gor, Thomas Müller, and William Cohen. 2021.  MATE: Multi-view attention for table transformer efficiency.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7606–7619, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Eisenschlos et al. (2020) Julian Eisenschlos, Syrine Krichene, and Thomas Müller. 2020.  Understanding tables with intermediate pre-training.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 281–296, Online. Association for Computational Linguistics.
  * Herzig et al. (2020) Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020.  TaPas: Weakly supervised table parsing via pre-training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4320–4333, Online. Association for Computational Linguistics.
  * Iyyer et al. (2017) Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017.  Search-based neural structured learning for sequential question answering.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1821–1831, Vancouver, Canada. Association for Computational Linguistics.
  * Liu et al. (2021) Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, and Jian-guang Lou. 2021.  Tapex: Table pre-training via learning a neural sql executor.  _arXiv preprint arXiv:2107.07653_.
  * Mueller et al. (2019) Thomas Mueller, Francesco Piccinno, Peter Shaw, Massimo Nicosia, and Yasemin Altun. 2019.  Answering conversational questions on structured data without logical forms.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5902–5910, Hong Kong, China. Association for Computational Linguistics.
  * Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015.  Compositional semantic parsing on semi-structured tables.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1470–1480, Beijing, China. Association for Computational Linguistics.
  * Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018.  Self-attention with relative position representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 464–468, New Orleans, Louisiana. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wang et al. (2020) Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020.  RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7567–7578, Online. Association for Computational Linguistics.
  * Wang et al. (2021) Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. 2021\.  TUTA: Tree-based Transformers for Generally Structured Table Pre-training.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 1780–1790.
  * Yin et al. (2020) Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020.  TaBERT: Pretraining for joint understanding of textual and tabular data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8413–8426, Online. Association for Computational Linguistics.
  * Ying et al. (2021) Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. 2021.  Do Transformers Really Perform Bad for Graph Representation?  _arXiv preprint arXiv:2106.05234_.
  * Yu et al. (2020) Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020.  GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing.  _arXiv preprint arXiv:2009.13845_.
  * Zhang et al. (2020) Hongzhi Zhang, Yingyao Wang, Sirui Wang, Xuezhi Cao, Fuzheng Zhang, and Zhongyuan Wang. 2020.  Table fact verification with structure-aware transformer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1624–1629, Online. Association for Computational Linguistics.


# Application of Deep Learning Methods Combined with Physical Background in
Wide Field of View Imaging Atmospheric Cherenkov Telescopes

Ao-Yan Cheng  School of Physics and Technology, Wuhan University, Wuhan
430072, People’s Republic of China Hao Cai Corresponding author:
<EMAIL_ADDRESS>School of Physics and Technology, Wuhan University, Wuhan
430072, People’s Republic of China Shi Chen  School of Physics and Astronomy,
Yunnan University, Yunnan 650091, China  Tian-Lu Chen  The Key Laboratory of
Cosmic Rays (Tibet University), Ministry of Education, Lhasa 850000, Tibet,
People’s Republic of China  Xiang Dong  School of Physics and Technology,
Wuhan University, Wuhan 430072, People’s Republic of China You-Liang Feng  The
Key Laboratory of Cosmic Rays (Tibet University), Ministry of Education, Lhasa
850000, Tibet, People’s Republic of China  Qi Gao  The Key Laboratory of
Cosmic Rays (Tibet University), Ministry of Education, Lhasa 850000, Tibet,
People’s Republic of China  Quan-Bu Gou  Key Laboratory of Particle
Astrophysics, Institute of High Energy Physics, Chinese Academy of Sciences,
Beijing 100049, People’s Republic of China  Yi-Qing Guo  Key Laboratory of
Particle Astrophysics, Institute of High Energy Physics, Chinese Academy of
Sciences, Beijing 100049, People’s Republic of China  University of Chinese
Academy of Sciences, 19 A Yuquan Road, Shijingshan District, Beijing, 100049,
China  Hong-Bo Hu  Key Laboratory of Particle Astrophysics, Institute of High
Energy Physics, Chinese Academy of Sciences, Beijing 100049, People’s Republic
of China  University of Chinese Academy of Sciences, 19 A Yuquan Road,
Shijingshan District, Beijing, 100049, China  Ming-Ming Kang  College of
Physics, Sichuan University, Chengdu, 610064, China  Hai-Jin Li  The Key
Laboratory of Cosmic Rays (Tibet University), Ministry of Education, Lhasa
850000, Tibet, People’s Republic of China  Chen Liu  Key Laboratory of
Particle Astrophysics, Institute of High Energy Physics, Chinese Academy of
Sciences, Beijing 100049, People’s Republic of China  Mao-Yuan Liu  The Key
Laboratory of Cosmic Rays (Tibet University), Ministry of Education, Lhasa
850000, Tibet, People’s Republic of China  Wei Liu  Key Laboratory of Particle
Astrophysics, Institute of High Energy Physics, Chinese Academy of Sciences,
Beijing 100049, People’s Republic of China  University of Chinese Academy of
Sciences, 19 A Yuquan Road, Shijingshan District, Beijing, 100049, China
Fang-Sheng Min  Key Laboratory of Particle Astrophysics, Institute of High
Energy Physics, Chinese Academy of Sciences, Beijing 100049, People’s Republic
of China  Chu-Cheng Pan  School of Physics and Technology, Wuhan University,
Wuhan 430072, People’s Republic of China Bing-Qiang Qiao  Key Laboratory of
Particle Astrophysics, Institute of High Energy Physics, Chinese Academy of
Sciences, Beijing 100049, People’s Republic of China  Xiang-Li Qian  School of
Intelligent Engineering, Shandong Management University, Jinan, 250357, China
Hui-Ying Sun  School of Intelligent Engineering, Shandong Management
University, Jinan, 250357, China  Yu-Chang Sun  School of Physics and
Technology, Wuhan University, Wuhan 430072, People’s Republic of China Ao-Bo
Wang  School of Physics and Technology, Wuhan University, Wuhan 430072,
People’s Republic of China Xu Wang  School of Intelligent Engineering,
Shandong Management University, Jinan, 250357, China  Zhen Wang  Tsung-Dao Lee
Institute, Shanghai Jiao Tong University, 200240 Shanghai, China  Guang-Guang
Xin  Suzhou Aerospace Information Research Institute, Suzhou 215123, China
Yu-Hua Yao  Key Laboratory of Particle Astrophysics, Institute of High Energy
Physics, Chinese Academy of Sciences, Beijing 100049, People’s Republic of
China  College of Physics, Chongqing University, No.55 Daxuecheng South Road,
High-tech District, Chongqing, 401331, China  Qiang Yuan  Key Laboratory of
Dark Matter and Space Astronomy, Purple Mountain Observatory, Chinese Academy
of Sciences, Nanjing 210008, China  Yi Zhang  Key Laboratory of Dark Matter
and Space Astronomy, Purple Mountain Observatory, Chinese Academy of Sciences,
Nanjing 210008, China

###### Abstract

The HADAR experiment, which will be constructed in Tibet, China, combines the
wide-angle advantages of traditional EAS array detectors with the high
sensitivity advantages of focused Cherenkov detectors. Its physics objective
is to observe transient sources such as gamma-ray bursts and counterparts of
gravitational waves. The aim of this study is to utilize the latest AI
technology to enhance the sensitivity of the HADAR experiment. We have built
training datasets and models with distinctive creativity by incorporating
relevant physical theories for various applications. They are able to
determine the kind, energy, and direction of incident particles after careful
design. We have obtained a background identification accuracy of 98.6%, a
relative energy reconstruction error of 10.0%, and an angular resolution of
0.22-degrees in a test dataset at 10 TeV. These findings demonstrate the
enormous potential for enhancing the precision and dependability of detector
data analysis in astrophysical research. Thanks to deep learning techniques,
the HADAR experiment’s observational sensitivity to the Crab Nebula has
surpassed that of MAGIC and H.E.S.S. at energies below 0.5 TeV and remains
competitive with conventional narrow-field Cherenkov telescopes at higher
energies. Additionally, our experiment offers a fresh approach to dealing with
strongly connected scattered data.

Keywords:  VHE Gamma-Ray Astronomy, HADAR, Deep Learning, Convolutional Neural
Networks

##  1 Introduction

The investigation of ultra-high-energy gamma-ray astronomy [1] is pivotal for
understanding a range of extreme astrophysical phenomena. These ultra-high-
energy gamma rays predominantly originate from highly active galactic nuclei,
cataclysmic supernova explosions, neutron stars, and black holes. Importantly,
these observations form an empirical cornerstone for addressing some of the
most compelling scientific questions, including the detection of dark matter
and the identification of cosmic ray sources.

Space observatories like the Fermi Gamma-ray Space Telescope [2] need long
observation times to gather statistically relevant data because high-energy
gamma-ray flux decays quickly and has a power-law distribution. Consequently,
larger-area ground detectors are required for these high-energy gamma rays.

The Earth’s atmosphere becomes into a substantial interaction medium for high-
energy photons in the energy range above GeV [3], effectively preventing their
transmission. In this regime, high-energy $\gamma$-rays interact with Earth’s
upper atmospheric layers [4], leading to the generation of electron-positron
$e^{-}e^{+}$ pairs, which in turn instigate electromagnetic cascades. Within
these cascades, the relativistic electrons and positrons produce concentrated
beams of Cherenkov radiation. These beams serve as the principal observational
targets for our specialized ground-based detection systems, enabling nuanced
studies of the high-energy universe.

Gamma-ray astronomy has become one of the frontier disciplines for studying
the universe’s highest energy astrophysical phenomena over the last few
decades. The Cherenkov Telescope Array (CTA) is an international initiative
that seeks to better understand high-energy gamma rays in the universe by
identifying atmospheric Cherenkov radiation [5]. The 20 GeV to 300 TeV energy
range covered by CTA is intended to close the observational gap currently
present in this energy range. A group of five Cherenkov telescopes called the
High Energy Stereoscopic System (HESS) is now operational in Namibia and is
used to observe cosmic TeV energy rays. It offers crucial information for
comprehending gamma-ray sources [6] like pulsars and supernova remnants [7].
The Major Atmospheric Gamma Imaging Cherenkov (MAGIC) telescopes, two
telescopes situated on La Palma in the Canary Islands, are another notable
project. Through the detection of Cherenkov radiation from cosmic gamma-ray
sources [8], they investigate the physical characteristics and origins of
cosmic rays [9]. These telescopes with a small field of view are primarily
employed to make accurate observations of known high-energy astrophysical
objects, providing important knowledge and observational data for our
comprehensive understanding of these extreme cosmological occurrences.

The measurable signals from many extreme celestial occurrences, however, only
last for brief periods because of their intrinsically transitory character.
There is an urgent demand for wide-field detectors that can instantly capture
high-energy particle signals across a wide spatial range since large detector
arrays require time to recalibrate their directed reception. Wide-field
Cherenkov imaging telescopes are especially useful in this situation.

The HADAR experiment exemplifies this technology [10], comprising an array of
imaging atmospheric Cherenkov refractive telescopes anchored in atmospheric
Cherenkov principles. Positioned at an impressive altitude of 4300 meters in
Yangbajing, Tibet (N 30.0848, E 90.5522), the experiment incorporates four
water lenses, each spanning a diameter of 5 meters, strategically placed at
the vertices of a square with side lengths of 100 meters. These lenses,
ensconced within a hemispherical glass shell of 5 meters in diameter, are
housed atop an 8-meter diameter steel structure tank. This tank, rising to a
height of 7 meters, is brimming with high-purity water, while its base is
adorned with cameras outfitted with photomultiplier tubes. Upon the
interception of Cherenkov radiation, the quartet of detectors meticulously
logs the charge deposition on the photomultiplier tubes within a predefined
temporal frame. All collected data are subsequently archived in an external
storage apparatus.

To effectively detect very-high-energy (VHE) $\gamma$-ray sources, analytical
methodologies must execute several critical tasks:

  1. 1.

Background Suppression: The identification of distinct shape features within
detector images is pivotal. This allows for the isolation of target
$\gamma$-rays from the overwhelmingly abundant background of cosmic rays,
which are predominantly composed of protons.

  2. 2.

Energy Reconstruction: The accurate estimation of the original energy of
incident particles is achieved by correlating variables such as deposited
charge and the relative spatial coordinates within the detectors.

  3. 3.

Direction Reconstruction: Leveraging the stereoscopic images acquired from the
detectors, it is crucial to reconstruct the axis direction of the resultant
particle shower. This, in turn, facilitates the estimation of the original
direction from which the incident particles emanated.

Deep learning techniques are increasingly becoming integral to the data
analysis frameworks employed in Cherenkov telescope experiments. A variety of
computational algorithms, particularly those based on convolutional neural
networks (CNNs), have shown unparalleled success in addressing the
multifaceted analytical challenges endemic to this scientific domain.
Prominent initiatives such as H.E.S.S.[11] and CTA[12][13] have adeptly
harnessed the robust capabilities of CNNs to scrutinize their observational
datasets, yielding significant advancements. These technological strides
corroborate the findings delineated in the present study, reinforcing the
compelling case for the widespread utility and robustness of deep learning
methodologies within this specialized field of research.

The organization of this paper’s structure is as follows: The use of deep
learning is briefly introduced in Section 2 along with our data production
process. A more thorough overview will be provided in Sections 3 to 5, and the
pertinent data will be used as the basis for the training of background
suppression, energy reconstruction, and incident direction models. The
modeling calculations of the flux and observational sensitivity of the Crab
Nebula at various energies are covered in Section 6. Finally, Section 7 will
offer a thorough summary of the material presented.

##  2 Deep-Learning Approaches for HADAR Data

Deep Learning is an important branch of Artificial Intelligence (AI) and has
emerged as a growing field in recent years. With the significant increase in
computational power, especially with GPU chips, real-time computation of large
parallel data (e.g., high-dimensional matrices) has become possible. This has
led to the convergence of deep learning towards various data analysis
industries, simplifying tasks that were previously difficult for humans to
perform. As a type of artificial intelligence, deep learning methods simulate
the multi-layer neural network structure of the human brain to solve problems.
Essentially similar to human learning, deep learning methods continuously
update network parameters through a step-by-step understanding of data. This
process ultimately allows the network to extract as many useful features as
possible from the inputs and obtain model predictions through neural network
computations.

In our work, we have adopted Convolutional Neural Networks (CNNs). As an
excellent model for image recognition, CNNs can spontaneously extract local
relationships at different positions within the same dimension and across
different dimensions based on the input signals, ultimately yielding optimal
results. Fundamentally, a CNN is composed of multiple layers: specifically, it
consists of a series of convolutional-pooling layers connected to fully
connected layers. Due to this layered structure, errors from one layer get
propagated to the next, potentially leading to an exponential growth of
errors. To mitigate this, we use a specialized CNN, known as a Residual
Convolutional Neural Network. The advantage of this network is that each layer
receives not only the processed signals from the previous layer but also the
original signals from the previous layer. This significantly enhances the
model’s ability to fit the data.

By leveraging the power of CNNs, specifically Residual CNNs, we aim to tackle
the three main challenges in detecting VHE gamma-ray sources: background
suppression, energy reconstruction, and direction reconstruction. Our
experimental results, which we will detail in subsequent sections, demonstrate
the effectiveness of deep learning methods in these high-stakes, complex
analytical tasks.

Throughout the research process, all datasets used for training the neural
networks were generated using the widely-adopted CORSIKA Monte Carlo program
[14] [15]. In the simulations, the set altitude was 4300 $m$, corresponding to
an atmospheric depth of 606 g/cm2. The geomagnetic coordinates were set to the
location of Yangbajing in Tibet. The simulated primary cosmic ray particles
included both gamma rays (serving as signal) and protons (serving as
background), with energy ranges spanning from 20 GeV to 10 TeV. The incident
zenith angle was set to 20∘, and the azimuthal angle ranged from 0 to 360∘.
All events were uniformly scattered within a circle centered on the HADAR
array, with a radius of 400 $m$. After the shower simulation, the response of
the HADAR detector was simulated using appropriate software packages, and
corresponding telescope images were generated.

To implement the model incorporating the associated algorithms, we utilized
the deep learning framework based on PyTorch. Training and testing were
carried out on a machine equipped with 2 NVIDIA GeForce GTX 3090 GPUs and a
high-performance computing cluster furnished with 4 NVIDIA V100 Tensor Core
GPUs, achieving nearly identical results on both platforms.

For diverse research tasks, we have curated specialized datasets, adopted a
range of neural network architectures, and employed various loss functions.
Detailed descriptions of these components will be thoroughly discussed in the
subsequent three sections, where we will also articulate the underlying
rationale for these choices. Our codebase, named ’GPLearn,’ is publicly
available on GitHub. This package features modular functionalities, enabling
users to tailor module parameters to specific needs. With GPLearn, readers
have the capability not only to reproduce the results presented in this paper
using our supplied datasets but also to adapt the code for actual detector
data analysis. We anticipate that these contributions will find broad
applicability in the field of astrophysics.

To successfully execute GPLearn, ensure that you have both Python and the
compatible version of PyTorch installed on your system. To install GPLearn,
simply clone the repository by executing the following command in your
terminal:

git clone https://github.com/caihao/gpLearn.git

This will install the most up-to-date package in your current directory. For
detailed usage guidelines and instructions, kindly consult the appropriate
tutorials.

##  3 Background Suppression

The flux of cosmic rays typically exceeds that of high-energy gamma rays by
several orders of magnitude, posing a significant challenge for detection.
Consequently, the Cherenkov radiation generated by these high-energy cosmic
rays acts as a predominant source of background noise. Given that
approximately 90% of cosmic rays are proton constituents, this study primarily
focuses on evaluating the influence of protons on photon detection.
Particularly problematic are proton signals with energies threefold greater
than those of photon signals, as these are especially difficult to
distinguish. To tackle this challenge, we have conducted simulations involving
particles with a range of energies, aiming to effectively segregate the target
photon signals from the interfering proton background.

The prevailing methodology for discriminating between high-energy gamma rays
and background cosmic rays leverages the intrinsic differences between
electromagnetic and hadronic showers. When photons impinge upon the Earth’s
atmosphere, they predominantly induce electromagnetic showers, which in
contrast to the hadronic showers triggered by protons, yield more uniform
detector images. Specifically, in the data recorded by our detectors, these
images are represented as pixel arrays (as illustrated in Fig. 1). For a
fitted ellipse, conventional methods compute the image moments of these
pixels. If the shower is made up of photons or protons, it can be determined
by comparing the major axis and minor axis of the fitted ellipse. Since
photon-generated electromagnetic showers have a more concentrated energy
distribution, their pixel layouts follow linear trajectories. Contrarily,
hadronic showers brought on by protons exhibit more core hits, resulting in
pixel patterns that are significantly spread and uneven. The main goal of
current background suppression work is to distinguish between these two
categories of image structures with accuracy. CNNs, in particular, have
produced exceptional results in image identification using deep learning
approaches. Consequently, compared to conventional curve-fitting techniques,
they offer a more accurate discrimination capability.

In the task of background suppression, we have adapted the ResNet-18 residual
convolutional neural network model, as depicted in Fig. 2, to meet our
specific objectives. The tailored architecture of our model is illustrated in
Fig. 2(a). For a more comprehensive understanding of the Residual Block
component, we also offer an expanded view in Fig. 2(b) and Fig. 2(c). Unlike
the canonical ResNet-18 model, our customized version employs smaller
convolutional kernels with designated strides for the initial data
convolution. This choice is motivated by the fact that the region of interest
in our detector images is generally confined to a narrow central zone. The use
of smaller kernels proves advantageous for capturing critical shape and edge
features within this focal region, as substantiated by improved performance in
subsequent evaluations.

The output from our model is formulated as a two-dimensional vector that
quantifies the probabilities of a particle being categorized as either a
signal (photon) or background noise (proton). This vector undergoes automatic
normalization to ensure that its components sum to one, thus enhancing
interpretability. The normalized probabilities $s_{k}$ are computed according
to the equation:

$s_{k}=\frac{e^{z_{k}}}{\sum_{j=1}^{K}e^{z_{j}}}$

Here, $z_{k}$ denotes the raw output value corresponding to the $k^{th}$
class, and $K$ signifies the total number of classes. In our particular
application, $K=2$ , representing the categories of signal and noise. This
normalization ensures that the output probabilities are mutually exclusive and
exhaustive, summing to 1, which aids in yielding more interpretable outcomes.

To evaluate the divergence between the model’s predicted probability
distribution and the actual probability distribution, we employ cross-entropy
as our loss function, expressed as follows:

$CE(y,p)=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{C}y_{ij}\ln(p_{ij})$

Empirical results indicate that the cross-entropy loss function is highly
sensitive to deviations in the model’s predicted probabilities. Furthermore,
it facilitates more rapid convergence during the training phase of the model.

The physical context was taken into account during the data preparation.
Considering that electromagnetic cluster and hadron cluster will primarily
create differences in the degree of image standardization, we combined the
actual background of the current work: as a result, the absolute positions
between different detectors become less important. For this reason, we decided
to only retain a small part of the image (Fig.3(a)) centered on the actual
signal area (Fig.3(b)), and stitch the remaining image together (Fig.3(c)). In
the end, the model performed and processed the data above expectations
accurately and precisely, providing a satisfying answer to this physical
problem based just on intensity and relative positions.

To demonstrate the model’s effectiveness and performance, we introduce a
threshold factor, denoted by $\zeta_{0}$, and set its value to 0.5 as a
criterion for interpreting the classification of incoming particles. In this
context, $\zeta$ refers to the second component of the model’s two-dimensional
output vector. In an ideal scenario, the output for a photon signal should be
aligned at $\zeta_{\gamma}=0$, whereas the output for a proton signal should
converge to $\zeta_{proton}=1$. However, in practice, any intersection between
the distribution curves of $\zeta_{\gamma}$ and $\zeta_{proton}$ would signify
the model’s error rate in classification, as illustrated in Fig.4.

Figure 1: Left: Imaging of $\gamma$-ray Cherenkov light in a telescope (3TeV);
Right: Imaging of proton Cherenkov radiation in a telescope (9TeV)

(a) Modified ResNet-18 Model.

(b) Detailed Structure of Modified ResNet-18.

(c) Internal Structure of Residual Block.

Figure 2: The modified ResNet-18 Model used in the background suppression
work, and we rename it to ParticleNet. Through four Residual blocks,
ParticleNet increases the input data’s initial dimensions to 512, and then the
model processes it after a global pooling layer.

(a) Photon image captured by a single water lens; Most of the PMTs (in purple)
are not triggered, only a few PMTs within the white frame have captured
signals.

(b) Cropped pixel image centered on the location, which corresponds to the
part enclosed by the white frame in Fig.3(a).

(c) Stitched version of four processed detector pixel images, with the
processed image on the left situated in the white frame at the upper-left
corner.

Figure 3: Data preprocessing work involving the cropping and stitching of
pixel photos. Figure 4: Distribution of $\zeta$ values for photons and protons
in simulated signals. The distribution of photons is concentrated around
$\zeta=0$, while the distribution of protons is centered around $\zeta=1$.

We retain the events identified as photons, while excluding those identified
as protons. At the same time, we also record the correct identification rates
for both photon and proton events. The quality factor $Q$ is defined as
follows:

$Q=\frac{\varepsilon^{\gamma}_{s}}{\sqrt{\varepsilon^{p}_{s}}}$

Here, $\varepsilon^{\gamma}_{s}$ represents the survival rate of photons after
discrimination, and $\varepsilon^{p}_{s}$ represents the survival rate of
protons after discrimination. The higher the correct identification rate of
the model, the larger the quality factor, indicating better background
suppression.

Figure 6 illustrates the model’s accuracy in particle identification across
varying energy thresholds. The yellow curve, corresponding to protons,
demonstrates consistently high identification rates across the entire energy
spectrum. Conversely, the blue curve, representing photons, exhibits a more
variable identification rate, with a marked improvement at elevated energy
levels. Existing research indicates that the threshold energy for the
production of Cherenkov radiation is approximately 50 GeV. Near this energy
threshold, even conventional electromagnetic showers are prone to generating
anomalous results. As the energy level escalates, these showers exhibit
increasing regularity, leading to a consequent enhancement in the model’s
identification accuracy.

The results from the test set indicate that the identification rates for both
photons and protons exhibit a marked increase as particle energy rises. At
lower energy levels, particles generate smaller atmospheric showers, resulting
in a limited number of pixels captured by ground-based detectors. This
scarcity of data hampers the model’s ability to render accurate
identifications. Conversely, as the particle energy escalates, the quality of
images captured by the detectors improves substantially. This augmentation in
data quality, coupled with an adequate number of parameters, leads to a
significant uptick in the model’s identification accuracy.

The yellow curve in Figure 6 illustrates the performance gains achieved
through the utilization of deep learning techniques, standing in stark
contrast to the blue curve, which represents the efficacy of traditional
methods. In the domain of low-energy particles, the quality of identification
markedly ascends in tandem with rising energy levels, leading to a swift
incline in the quality factor curve. Conversely, at higher energy thresholds,
the model has ample raw parameters available for making accurate judgments,
resulting in a plateauing of the result curve; it is worth noting that the
model’s identification rate has already approached a robust 98% at this stage.

Figure 5: The identification accuracy for photons/protons at different energy
points in the test dataset. (The training process and results are from
GPLearn, and variations may occur with different datasets.)

Figure 6: Comparison of identification performance between deep learning and
traditional methods for particle discrimination. (The deep learning results
are provided by GPLearn; the traditional methods are calculated using the
maximum ellipse fitting technique, which may vary slightly from current
mainstream algorithms.)

##  4 Energy Reconstruction

The model cannot function as intended if the dataset and model created in
accordance with the background suppression work are utilized, and the related
labels are changed to particle energy. The reconstructed particle energy is
related to the incidence zenith angle, the distance from the cluster core, and
the quantity of charge deposited in the detector after accounting for the
pertinent physical background. Traditional approaches employ simulated data to
determine the energy distribution function at each place and actual
observational data to fit the following function [16]:

$E_{erc}=f(q,R,\theta)$

Because the absolute positional information of the relative detector array is
required to calculate the distance and direction information of the signal to
the actual incident center, and because this information was removed during
the dataset construction process for background suppression, data retaining
only relative positional information cannot reconstruct the final energy.

As a result, we added the relative position of the pixel centroid and the
absolute position of the detector array as new inputs to the model, based on
the data preprocessing work employed in background suppression work. The model
was created as depicted in Figure 7, taking into account that energy
resolution does not require particle classification work: four detector photos
are processed independently, combined after a series of fully connected
layers, and the final result is represented by a one-dimensional vector. This
model, which does not include convolutional layers, not only retains a high
level of reconstruction accuracy but also significantly speeds up calculation.

Expanding upon the data preprocessing techniques deployed in our background
suppression task, we have incorporated additional variables: the relative
positions of pixel centroids and the absolute coordinates of the detector
array. This is in recognition of the fact that energy is fundamentally
correlated with the amount of charge deposited. Accordingly, our model
architecture, as illustrated in Figure 7, processes images from four
individual detectors in parallel. These processed outputs are subsequently
fused through a sequence of fully connected layers, culminating in a one-
dimensional output vector.

In the test dataset, we employ both the model-predicted energy and the actual
particle energy to compute the associated energy resolution, denoted as
$\Delta E/E$. As illustrated in Figure 9, our findings reveal a trend of
improving energy resolution with increasing particle energy.

To provide an intuitive assessment of our model’s efficacy, we analyzed 2,500
samples from the test dataset, each representing a different energy level. We
generated a distribution curve to illustrate the model’s energy prediction
performance. On the x-axis, we plotted the predicted energy values, while the
y-axis represents the probability associated with each specific energy
prediction. To facilitate easier interpretation, we normalized the data for
each discrete energy level, ensuring that the integral of each curve across
all energy points sums to unity.

Figure 9 illustrates the distribution curves for energy prediction at 500 GeV
intervals. Areas of overlap between these curves signify regions where the
model’s estimations may deviate from actual conditions, constituting a
principal constraint on the accuracy of energy resolution. Importantly, our
model achieves a relative error of just 13.0% at an energy level of 1 TeV.
Even more remarkably, when we ascend to higher energy levels—such as 10
TeV—the relative error narrows further to an impressive 10.0%. This value
approaches the theoretical limits of energy resolution achievable with
Cherenkov telescopes, underscoring the significant strides our research has
made.

Figure 7: The GoogLeNet model used in the energy reconstruction work. The
model is further processed by combining the four detectors after processing
the charge pictures and relative location data from four distinct detectors
with fully linked layers independently.

Figure 8: Energy reconstruction’s relative error distribution. The relative
error rapidly reduces with particle energy until it reaches a value of 10.0%
at 10 TeV. By comparison, the relative error for the conventional technique is
around 20% at 1 TeV.

Figure 9: Energy prediction results using deep learning methods. Different
color curves barely overlap one another, showing that the model can almost
entirely differentiate gamma rays at the appropriate energy levels.

##  5 Direction Reconstruction

After atmospheric Cherenkov radiation reaches ground-level detectors, it is
refracted through our specialized water lenses before being captured on the
image plane. Calculating the path of this refracted light is complex. Our
initial effort focuses on estimating the shower core’s projected position on
this image plane. By calculating the fitting ellipse of the four detector
images and identifying the location where clustering occurs based on the
weighted intersection of the long axes of each ellipse, conventional methods
(such as Hillas reconstruction [17]) can determine where clustering occurs and
ultimately the direction of incidence of $\gamma$-rays [18] [19].

Unlike the energy reconstruction task, both the intensity and absolute
position of each pixel are intricately linked, serving as indirect indicators
of the gamma rays’ incident direction.

However, it is impractical to fully create a pixel matrix based on actual
positions because each pixel corresponds to a photomultiplier tube with a 1.25
cm diameter and a 100 m separation between detectors, making it impossible to
create a two-dimensional matrix with such a large input size. We continue to
use the cropping method in response to this. In contrast to the background
suppression work, each pixel after cropping is thought of as a three-
dimensional data set because it contains the deposited charge corresponding to
that pixel as well as its absolute coordinates prior to cropping, which have
been converted using the precise coordinates of the experimental site. Our
tests demonstrate that the model’s performance is much enhanced when the
absolute positioning information of pixels is retained.

To accommodate the specialized input dataset, we employ a CNN model featuring
three-dimensional convolutional layers, as illustrated in Figure 10. The raw
data undergoes a sequence of six up-sampling convolutional layers, each
followed by a pooling layer. This processed data is then fed into a linear
regression layer, culminating in a two-dimensional vector output. After
normalization, this vector serves as an azimuthal direction vector in the
Earth’s plane. The model subsequently computes the corresponding incident
angle based on this vector.

The error angle distance $\Omega$ is calculated from the azimuthal angle
through spherical coordinate transformation.

$cos\Omega=sin\theta sin\phi\cdot sin\theta sin\phi^{\prime}+sin\theta
cos\phi\cdot sin\theta cos\phi^{\prime}+cos^{2}\theta$

Here, $\phi$ represents the true incident azimuthal angle, while
$\phi^{\prime}$ represents the azimuthal angle reconstructed by the model.

All our simulations are based on an incident zenith angle of
$\theta=20^{\circ}$ . The results, plotted in Figure 11, reveal that the deep
learning method we employed yields approximately a 60% improvement in accuracy
over traditional methods as cited in [20]. Notably, the reconstruction error
decreases substantially with increasing energy levels. This enhanced accuracy
is attributable to the greater number of photons collected by the telescope at
higher energy levels, which in turn allows for a more precise reconstruction
of the event direction. In the energy range above 1 TeV, our model has already
reduced the angular error to below 0.2 degrees, providing a robust foundation
for subsequent research endeavors.

Figure 10: Model used for direction reconstruction: AngleNet Model. Through
six convolutions, AngleNet increases the input’s original dimension to 512,
and then processes it through a fully connected layer after a global pooling
layer. AngleNet makes use of Residual Blocks that are appropriate for three
dimensions, just like ParticleNet does. This helps to some extent with the
under-fitting issue with the model and considerably improves its ability to
forecast the future. Figure 11: Angular distance difference between
reconstructed and true incident directions at different energies, in degrees.
Deep learning techniques have a larger potential for the outcomes of angle
reconstruction than conventional elliptical fitting techniques since they
rebuild this physical process by looking at relationships among nearly all of
the original data. (Deep learning results are provided by GPLearn; traditional
method data is sourced from the literature [20])

##  6 Simulated measurements of the Crab Nebula

###  6.1 Sensitivity measurements

The Crab Nebula serves as a prototypical supernova remnant, emitting across a
wide range of wavelengths including radio, X-ray, and gamma-ray bands. The
study of this celestial object is pivotal for gaining insights into the
physical mechanisms underlying supernova explosions, the generation and
acceleration of cosmic rays, and addressing important scientific queries such
as the nature of dark matter. Among the numerous physical attributes
associated with the Crab Nebula that demand precise measurement, sensitivity
stands out as a crucial parameter. The level of sensitivity directly impacts
the quality—both in terms of accuracy and precision—of our estimates
concerning various physical phenomena related to the Crab Nebula.

In preceding sections, we delved into the deployment of deep learning
methodologies for the sophisticated analysis of detector data, achieving a
marked improvement in accuracy. In this subsequent segment, our focus shifts
to the specific application of deep learning in enhancing the sensitivity
measurements related to the Crab Nebula. We will illustrate how these advanced
computational techniques outperform traditional methods, offering a more
precise and reliable assessment of this crucial parameter. Additionally, we
will juxtapose our findings with those obtained from other Cherenkov imaging
telescope experiments to contextualize the efficacy and innovation of our
approach.

In order to calculate sensitivity, we conducted a simulation that accounts for
both gamma rays and scattered protons across an energy spectrum ranging from
100 GeV to 10 TeV. The simulated sampling area is set at $S_{sample}=800\times
800m^{2}$, and we used a fixed zenith angle of 20 degrees. The azimuthal angle
in the simulation varies between 0 and 180 degrees. Additionally, we
integrated geographical factors specific to the Yangbajing site and the
celestial trajectory of the Crab Nebula. Based on these considerations, we
estimated the annual observation time dedicated to the Crab Nebula to be
approximately 320 hours, which translates to $T_{crab,obs}=1.152\times
10^{6}s$.

For gamma rays, we can calculate the effective number of events coming from
the Crab Nebula received by the HADAR experiment in one year at energy $i$
using the equation

$N_{\gamma}^{1yr}[i]=T_{crab,obs}\cdot S_{sample}\cdot
F_{crab}\cdot\frac{N_{\gamma,sim}[i]}{N_{\gamma,sim,all}[i]}\cdot\varepsilon_{\gamma}$

Here, $F_{crab}=\int_{i}^{\infty}2.83\cdot
10^{-11}phcm^{-2}s^{-1}TeV^{-1}\cdot(E/TeV)^{-2.62}dE$represents the
integrated flux of the Crab Nebula for energies greater than $i$.
$N_{\gamma,sim}[i]$ is the number of effective simulated photon events in
energy range $i$, $N_{\gamma,sim,all}[i]$ is the total number of simulated
photon events, and $\varepsilon_{\gamma}$ represents the ratio of events
located within the angular resolution, set at 68%.

For scattered protons, the effective number of observed events for one year
can be calculated as

$N_{CR}^{1yr}[i]=T_{crab,obs}\cdot S_{sample}\cdot
F_{CR}\cdot\frac{N_{CR,sim}[i]}{N_{CR,sim,all}[i]}\cdot\Omega_{i}$

Here, $F_{CR}$ represents the integrated flux of protons in cosmic rays.
$N_{CR,sim}[i]$ is the number of effective simulated proton events in the
energy range $i$, $N_{CR,sim,all}[i]$ is the total number of simulated proton
events, and $\Omega_{i}$ represents the solid angle within the range of photon
angular resolution.

Subsequent to the implementation of background suppression techniques and the
exclusion of proton-like events, we can derive the integrated significance
across the designated energy spectrum via Equation [21]:

$S[i]=\frac{N_{\gamma}^{1yr}[i]}{\sqrt{N_{CR}^{1yr}[i]}}\cdot Q$

Utilizing the framework of Gaussian statistics, the sensitivity is defined as
the minimal flux from the Crab Nebula required for the detector to register a
signal at a 5-sigma significance level. Consequently, the specific sensitivity
pertaining to the Crab Nebula for the HADAR experiment can be calculated using
the formula:

$F_{sensitivity}[i]=F_{crab}\cdot\frac{5}{S[i]}\cdot i$

To offer a comprehensive perspective, we graphically delineate the calculated
sensitivity metrics, both prior to and following the incorporation of deep
learning algorithms. These are juxtaposed with corresponding data from other
pertinent experiments [22][23][24], as depicted in Figure 12.

Following the integration of advanced deep-learning algorithms, the HADAR
experiment has experienced a substantial enhancement in sensitivity within the
low-energy domain. Although it has yet to attain the level of sensitivity
exhibited by the CTA, the revamped HADAR experiment now stands as a formidable
contender in juxtaposition with established initiatives such as MAGIC and
H.E.S.S. Importantly, HADAR boasts a unique attribute: an expansive field of
view not typically afforded by conventional reflective Cherenkov telescopes.
This distinct advantage facilitates the real-time capture of transient
celestial sources within its observational purview, rendering the outcomes
particularly compelling.

Figure 12: Comparison of Sensitivity Between HADAR Experiment and Other
Experiments. The HADAR experiment’s sensitivity curve for the Crab Nebula with
a 5-fold significance over a year (320 hours of useful observation time) is
shown in the figure. The Crab Nebula sensitivity curves from experiments like
the Fermi Satellite (one year of effective observation time); MAGIC, H.E.S.S.
(50 hours of effective observation time); and ARGO-YBJ, HAWC (one year of
effective observation time) are also shown in the figure for comparison.
Figure 13: The energy spectra for the Crab Nebula’s main pulsar P1
(represented by black circles) and the interpulsar P2 (represented by blue
circles) in the 70GeV-1.5TeV energy range have been measured by MAGIC. The
energy spectrum of the Crab Nebula itself (depicted by hollow squares) is also
shown for comparison. (Data sourced from [26]).

###  6.2 Observation of the Pulsar in the Crab Nebula

The primary objective of this section is to undertake a rigorous analysis of
the observational data pertaining to the Crab Nebula pulsar in the very-high-
energy regime, as acquired through the MAGIC telescope [25]. The intent behind
calculating the significance of observations from the HADAR experiment lies in
laying the groundwork for future inquiries into pulsar emissions at very-high-
energy wavelengths.

We determined the spectral properties of the primary pulsar (P1) and
interpulsar (P2) of the Crab Nebula using data obtained from the MAGIC
telescope [26], as shown graphically in Figure 13. The formula for the
integral flux for each energy band is as follows:

$F_{P1}=1.1\times 10^{-11}\cdot(\frac{E/TeV}{0.15})^{-3.2}cm^{-2}s^{-1}$
$F_{P2}=2\times 10^{-11}\cdot(\frac{E/TeV}{0.15})^{-2.9}cm^{-2}s^{-1}$

Following the implementation of background filtering procedures, only events
characterized by gamma-like profiles are retained for subsequent analyses
concerning the pulse signals of the Crab Nebula’s primary pulsar (P1) and
interpulsar (P2). To quantify the observational significance $S$ for the HADAR
experiment, we employ the following equation:

$S=\frac{N_{on}-\alpha N_{off}}{\sqrt{\alpha(N_{on}+N_{off})}}$

Here, $N_{on}$ signifies the photon count emanating from the pulsar, while
$N_{off}$ represents the background photon count. Remarkably, at the 100 GeV
energy threshold, the dominant background source in observations of the Crab
Nebula pulsar transitions from protons to gamma rays emanating from the Crab
Nebula itself. Consequently, the term $N_{off}$ encapsulates both proton and
non-pulsar gamma-ray backgrounds. The observation time ratio $\alpha$ is
defined as $\alpha={t_{on}}/{t_{off}}$, where $t_{on}$, where $t_{on}$ and
$t_{off}$ denote the durations of HADAR detector observations for the source
and background, respectively.

Table 1 and Table 2 delineate the observational datasets acquired from the
MAGIC and HADAR experiments, respectively. These tables encapsulate the
derived significances corresponding to distinct energy bands for both the main
pulsar (P1) and the interpulsar (P2) of the Crab Nebula, as corroborated by
references [27] and [28].

Capitalizing on the robust data processing capabilities afforded by deep
learning methodologies, the HADAR experiment demonstrates a notably superior
level of significance within the low-energy domain as compared to the MAGIC
experiment. Nevertheless, as the energy spectrum ascends, certain constraints,
including equipment specifications and geographical considerations, result in
a discernible performance disparity in comparison to MAGIC. Moving forward, we
aim to augment the sensitivity of our apparatus through a multi-faceted
approach, encompassing the expansion of the detector’s effective observational
area as well as the refinement of computational models to enhance angular
resolution.

Table 1: Five years of observational (320 hours of effective observation time) data from the MAGIC experiment shows corresponding significances for different energy ranges for P1 and P2. (Data sourced from [26]). energy range | P1 | P2
---|---|---
[$GeV$] | $N_{ex}$ | Significance | $N_{ex}$ | Significance
100$-$400 | $1252\pm 442$ | $2.8\sigma$ | $2537\pm 454$ | $5.6\sigma$
$>$400 | $188\pm 88$ | $2.2\sigma$ | $544\pm 92$ | $6.0\sigma$
$>$680 | $130\pm 66$ | $2.0\sigma$ | $293\pm 69$ | $4.3\sigma$
$>$950 | $119\pm 54$ | $2.2\sigma$ | $190\pm 56$ | $3.5\sigma$
Table 2: Deep learning algorithms are used to determine the comparable significance in the same energy range estimated from the observation results of P1 and P2 pulsars in the HADAR experiment over a year (320 hours of effective observation time). energy range | P1 | P2
---|---|---
[$GeV$] | $N_{ex}$ | Significance | $N_{ex}$ | Significance
100$-$400 | $3698$ | $4.65\sigma$ | $6943$ | $7.31\sigma$
$>$400 | $500$ | $0.708\sigma$ | $1432$ | $2.046\sigma$
$>$680 | $175$ | $0.390\sigma$ | $584$ | $1.306\sigma$
$>$950 | $87$ | $0.261\sigma$ | $321$ | $0.963\sigma$

##  7 Outlook

With the emergence of large-scale AI language models, artificial intelligence
has been progressively incorporated into a myriad of industrial applications.
Within the realm of physics, data analytics stands as a burgeoning arena for
the expansion of AI technologies. Leveraging its computational prowess, AI has
the potential to uncover latent relationships within complex data sets,
accelerate data-fitting procedures, and simulate realistic models for
experimental measurements.

In high-energy physics, the synergy between AI and big data allows for the
precise reconstruction of energy, momentum, and mass metrics for particles, a
critical step towards elucidating the underlying properties and behaviors of
subatomic particles. Additionally, exact spatial reconstructions are
indispensable for tracking the decay products of heavy particles and for
capturing rare events that could signal novel particles or interaction
mechanisms. In the field of astronomy, AI technologies facilitate the real-
time processing of astronomical signals, enabling more efficient studies of
distant cosmic events and contributing to our understanding of the universe’s
origins and composition.

In this study, we effectively address complicated physical issues by
integrating real-world physical contexts, using appropriate data preparation,
and using appropriate models. Additionally, the excellent results from deep
learning also serve to confirm the accuracy of the corresponding physical
theories: the training process, which was built in accordance with real-world
physical theories, both theoretically supports and verifies the deep learning
results.

In the actual research process, we have demonstrated that:

  1. 1.

The relative position and shape information of Cherenkov radiation can be used
to discriminate particle types; at the same time, information contained in
data that is not sensitive to absolute position will not be lost after slicing
and cropping processes.

  2. 2.

The energy information of the Cherenkov radiation reaction is not sensitive to
the relative position information of pixels; moreover, energy can be
approximately represented by a linear function of charge amount, incidence
angle, and relative center distance.

  3. 3.

The directional information of Cherenkov radiation is sensitive to the
absolute position of pixels; therefore, retaining the absolute position
information of pixels is crucial for correctly inferring the initial direction
of particles when processing image data.

Deep learning techniques have simplified and improved some labor-intensive
physical reconstruction procedures. But because physical data features are
typically sensitive to a limited scale range, adopting small-scale
convolutional kernels aids the neural network in more acutely recognizing
features. The network can more efficiently gather data on global feature
information by expanding its dimensions. The cropping of data is made possible
for geographically distinct trials by adding the necessary positional
dimensions, which also suggests a new approach for data analysis work in big
detector arrays.

## References

  * [1] G. G. Fazio.  High-energy gamma-ray astronomy.  Nature, 225(5236):905–911, 1970.
  * [2] A. A. Abdo, M. Ackermann, M. Ajello, et al.  The first fermi large area telescope catalog of gamma-ray pulsars.  The Astrophysical Journal Supplement Series, 187:460–494, 2010\.
  * [3] Luis Anchordoqui, María Teresa Dova, Analisa Mariazzi, et al.  High energy physics in the atmosphere: phenomenology of cosmic ray air showers.  Annals of Physics, 314(1):145–207, 2004.
  * [4] Bernard Degrange and Gérard Fontaine.  Introduction to high-energy gamma-ray astronomy.  Comptes Rendus Physique, 16(6):587–599, 2015.
  * [5] M. Actis, G. Agnetta, F. Aharonian, et al.  Design concepts for the cherenkov telescope array cta: an advanced facility for ground-based high-energy gamma-ray astronomy.  Experimental Astronomy, 32(3):193–316, 2011.
  * [6] F. Aharonian, A. G. Akhperjanian, A. R. Bazer-Bachi, et al.  The h.e.s.s. survey of the inner galaxy in very high energy gamma rays.  The Astrophysical Journal, 636(2):777, 2006.
  * [7] F. Aharonian, A. G. Akhperjanian, A. R. Bazer-Bachi, et al.  Observations of the crab nebula with hess.  Astronomy and Astrophysics, 457:899–915, 2006.
  * [8] J. Albert, E. Aliu, H. Anderhub, et al.  Vhe gamma-ray observation of the crab nebula and its pulsar with the magic telescope.  The Astrophysical Journal, 674:1037–1055, 2008.
  * [9] J. Albert, E. Aliu, H. Anderhub, et al.  Discovery of very high energy gamma radiation from ic 443 with the magic telescope.  The Astrophysical Journal, 664(2):L87, 2007.
  * [10] Hui-Ying Sun, Tian-Lu Chen, Danzengluobu, et al.  Prospective study on observations of gamma-ray emission from active galactic nuclei using the hadar experiment.  Acta Phys. Sin., 72(4):049501–1–049501–12, 2023.
  * [11] I. Shilon, M. Kraus, M. Büchele, et al.  Application of deep learning methods to analysis of imaging atmospheric cherenkov telescopes data.  Astroparticle Physics, 105:44–53, 2019.
  * [12] Salvatore Mangano, Carlos Delgado, María Isabel Bernardos, et al.  Extracting gamma-ray information from images with convolutional neural network methods on simulated cherenkov telescope array data.  In Luca Pancioni, Friedhelm Schwenker, and Edmondo Trentin, editors, Artificial Neural Networks in Pattern Recognition, pages 243–254. Springer International Publishing.
  * [13] Thomas Vuillaume, Mikaël Jacquemont, Mathieu de Bony de Lavergne, et al.  Analysis of the cherenkov telescope array first large size telescope real data using convolutional neural networks.  Proceedings of 37th International Cosmic Ray Conference — PoS(ICRC2021), 2021.
  * [14] D. Heck, J. Knapp, J. N. Capdevielle, et al.  Corsika: A monte carlo code to simulate extensive air showers.  Report, 1998.  51.02.03; LK 01; Wissenschaftliche Berichte, FZKA-6019 (Februar 98).
  * [15] Tadeusz Wibig.  Testing the superposition model in small corsika shower simulations.  Journal of Physics G Nuclear Physics, 49:035201, 2022.
  * [16] V. A. Acciari, M. Beilicke, G. Blaylock, et al.  Veritas observations of the gamma-ray binary ls i +61 303.  The Astrophysical Journal, 679:1427–1432, 2008.
  * [17] A. M. Hillas.  The origin of ultra-high-energy cosmic rays.  Annual Review of Astronomy and Astrophysics, 22(1):425–444, 1984\.  doi: 10.1146/annurev.aa.22.090184.002233.
  * [18] H. Abdalla, R. Adam, F. Aharonian, et al.  Resolving acceleration to very high energies along the jet of centaurus a.  Nature, 582(7812):356–359, 2020.
  * [19] F. Aharonian, Q. An, Axikegu, et al.  Observation of the crab nebula with lhaaso-km2a - a performance study *.  Chinese Physics C, 45(2):025002, 2021.  (LHAASO Collaboration).
  * [20] Guang-Guang Xin, Yu-Hua Yao, Xiang-Li Qian, et al.  Prospects for the detection of the prompt very-high-energy emission from gamma-ray bursts with the high altitude detection of astronomical radiation experiment.  The Astrophysical Journal, 923:112, 2021.
  * [21] Ti-Pei Li and YuQian Ma.  Analysis methods for results in gamma-ray astronomy.  The Astrophysical Journal, 272:317–324, 1983.
  * [22] Collaboration Lhaaso, Zhen Cao, F. Aharonian, et al.  Peta-electron volt gamma-ray emission from the crab nebula.  Science, 373:425–430, 2021.
  * [23] Felix Aharonian, Sergey Bogovalov, and D. Khangulyan.  Abrupt acceleration of a ‘cold’ ultrarelativistic wind from the crab pulsar.  Nature, 482:507–509, 2012.
  * [24] Alice K. Harding and Constantinos Kalapotharakos.  Synchrotron self-compton emission from the crab and other pulsars.  The Astrophysical Journal, 811:63, 2015.
  * [25] J. Aleksić, E. A. Alvarez, L. A. Antonelli, et al.  Observations of the crab pulsar between 25 and 100 gev with the magic i telescope.  The Astrophysical Journal, 742:43, 2011.
  * [26] Magic Collaboration S. Ansoldi, Lucio Angelo Antonelli, P. Antoranz, et al.  Teraelectronvolt pulsed emission from the crab pulsar detected by magic.  Astronomy and Astrophysics, 585:1–6, 2015.
  * [27] J. Aleksić, E. A. Alvarez, L. A. Antonelli, et al.  Phase-resolved energy spectra of the crab pulsar in the range of 50-400 gev measured with the magic telescopes.  Astronomy and Astrophysics, 540:A69, 2012.
  * [28] J. Aleksić, E. A. Alvarez, L. A. Antonelli, et al.  Performance of the magic stereo system obtained with crab nebula data.  Astroparticle Physics, 35:435–448, 2012.