if all diagnostic tests (the centroid tests, odd & even test, secondary test,
and the shape of the transit in transit-view of flux data) are unremarkable,
their contributions to the disposition score are minimal. We are able to come
up with a simple rule for explaining how ExoMiner classifies most of the PCs:
if the full-orbit branch is the maximally contributing branch and its
contribution toward the PC category is greater than 0.1, ExoMiner likely
classifies that TCE as a PC. To be more precise, 90.83% of TCEs whose
maximally contributing ExoMiner branch is the full-orbit flux view, with a
contribution greater than $0.1$, are indeed PCs in the test set (similar
statistics in other sets). On the other hand, 99.12% of PCs in the test set
satisfy this rule, i.e., for 99.12% of PCs, the full-orbit flux view is the
maximally contributing branch with a contribution greater than 0.1 toward the
PC class. In other words, this simple rule for explaining how ExoMiner
classifies TCEs as PCs has a precision of 0.991 and a recall of 0.908.

Classification of AFPs: In this analysis, we were interested in understanding
whether ExoMiner utilizes the correct type of diagnostic test in order to vet
an AFP as non-PC. Based on domain knowledge of the diagnostic information
contained in each branchﬂs input data, we map the different false positive
classes to the branches whose inputs would be most crucial for manually
diagnosing the respective false positive type. Specifically, TCEs with their
greatest contribution from the transit-view secondary eclipse branch or the
transit-view odd & even flux branch are labeled as EB FPs, and TCEs with their
greatest contribution from the full-orbit centroid branch or transit-view
centroid branch are labeled as background object FPs. Given that we did not
have access to gold standard labels for these subcategories, we utilized the
flags, i.e., ‘Eclipsing Binary High Level Flag’ and ‘Offset High Level Flag’,
provided in the CFP list (Bryson et al., 2015) as gold standard labels. Using
these noisy labels, we found that the FP labels generated by applying the
branch-occlusion test to ExoMiner match those provided by the CFP list 78.92%
of the time. We believe that when we perform a more careful explainability
study of ExoMiner, we will improve our ability to flag the different TCE false
positive types.

Example TCEs: Table 8 shows a few TCEs for which our explainability analysis
provides further insights and demonstrates how useful explainability can be in
practice for understanding how the model works and finding ways to improve it.
The positive/negative contribution of a branch implies that removing that
branch increases/decreases the score of that TCE, which makes that TCE
more/less likely to be a PC.

Table 8: Contribution of different branches in classifying interesting TCEs.
Note that the contribution is calculated by subtracting ExoMiner’s original
score without occluding any branches from the score when a given branch is
occluded. Thus, a negative value implies that the branch is contributing
towards a PC label, and a positive value implies that the branch is
contributing towards a non-PC label.

| TCE KIC | Period (days) | Original Label |  ExoMiner Score | $B_{1}$ | $B_{2}$ | $B_{3}$ | $B_{4}$ | $B_{5}$ | $B_{6}$
---|---|---|---|---|---|---|---|---|---
6543893.11 | 10.30 | PC | 0.498 | -0.440 | -0.400 | -0.006 | 0.270 | 0.124 | 0.120
9141355.12 | 469.621 | PC | 0.425 | -0.188 | -0.325 | -0.034 | -0.027 | 0.101 | 0.016
11517719.13 | 2.50 | PC | 0.353 | -0.340 | -0.168 | 0.001 | 0.034 | 0.003 | 0.245
9996632.11 | 8.44 | AFP | 0.017 | -0.014 | -0.012 | 0.030 | 0.971 | 0.004 | 0.030
8937762.12 | 7.53 | AFP | 0.730 | -0.606 | -0.298 | -0.223 | 0.061 | 0.256 | -0.034
6500206.13 | 13.375 | AFP | 0.154 | -0.135 | -0.023 | -0.079 | -0.047 | 0.033 | 0.838
6666233.24 | 1.0248 | NTP | 0.066 | 0.578 | -0.027 | -0.001 | -0.010 | 0.014 | 0.023
10186945.24 | 0.794 | NTP | 0.095 | 0.549 | -0.019 | -0.019 | -0.008 | 0.007 | 0.005
8052016.45 | 33.1842 | NTP | 0.004 | -0.003 | 0.165 | -0.002 | -0.000 | 0.075 | 0.008
8043714.26 | 3.407 | NTP | 0.013 | -0.006 | -0.000 | 0.004 | 0.000 | 0.696 | -0.003
9705459.26 | 1.243 | NTP | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.635 | 0.000
5446285.27 | 416.653 | NTP | 0.003 | -0.003 | -0.003 | 0.000 | 0.000 | 0.002 | 0.803

  * $B_{1}$: Full-orbit flux, $B_{2}$: Transit-view flux, $B_{3}$: Full-orbit centroid, $B_{4}$: Transit-view centroid, $B_{5}$: Transit-view odd-even, $B_{6}$: Transit-view secondary.

  * 1

The centroid branch is one of the main contributors to the final score.

  * 2

The odd and even transit depth difference is one of the main contributors to
the final score.

  * 3

The secondary branch is one of the main contributors to the final score.

  * 4

The secondary transit of this TCE should refer to the first TCE. It is not
corrected because the phase information for the secondary of the first TCE is
inaccurate.

  * 5

The transit-view of flux reveals that this is not a real transit.

  * 6

After removing the primary, the pipeline folded the secondaries onto the data
gaps left behind by the primary event. Thus, the period for the second TCE in
each case was one half that of the primary that resulted in an odd and even
depth difference for the second TCE.

  * 7

Even though this is an NTP, there is a clear secondary event helping the model
to classify this as a non-PC.

For the PC category, the full-orbit flux branch is almost always (99.1% of the
time) the maximally contributing branch towards a PC classification (negative
values). However, when one or more other branches have a positive non-
negligible contribution, that leads to a misclassification. We provide three
examples in Table 8 for which other diagnostic test branches, besides the
full-orbit branch, are the maximally contributing branch. TCEs KIC 6543893.1,
KIC 9141355.1, and KIC 11517719.1 are misclassified due to centroid shift,
odd-even depth difference, and existence of a weak secondary event,
respectively. For each of these TCEs, the related branch is correctly used to
reduce the score of ExoMiner. Thus, ExoMiner clearly utilized the correct
diagnostic test for each classification even though it misclassified those
TCEs.

To correctly classify an AFP, however, the contribution of one of the
diagnostic tests should be large enough to compensate for the contribution of
the full-orbit flux branch. When that fails, ExoMiner is not able to classify
the TCE correctly. An example of an AFP that is misclassified by ExoMiner is
TCE KIC 8937762.1, for which the contribution of the transit-view odd & even
branch is not large enough to compensate for the negative contribution value
of the full-orbit and transit-view flux branches. TCEs KIC 9996632.1 and KIC
6500206, for which the transit-view centroid and weak secondary branches,
respectively, are the main contributing branches, are correctly classified by
ExoMiner.

(a) Secondary event for KIC TCE $6666233.1$

(b) Secondary event for KIC TCE $10186945.1$

Figure 18: The phase value pointing to the secondary event is inaccurate,
which prevents us from correcting the secondary of the second TCE of that EB
system (see Section 3.6). Figure 19: KIC TCE 8052016.4 is an NTP and is
correctly classified based on its unusual transit-view flux shape. Note that
the blue dots are the real flux data, and the red solid line is the transit
fit generated by the pipeline.

The most interesting insights provided by this explainability analysis are
those related to the NTP class. For two TCEs, i.e., KIC 6666233.2 and KIC
10186945.2, which are labeled as NTP, the branch contributing maximally toward
a non-PC classification (i.e., positive value) is the full-orbit view. Both of
these TCEs are the secondary eclipse of an EB; thus, their correct label is
AFP. Our preprocessing failed to correct the secondary event for these TCEs
because the phase generated by the pipeline for the secondary transit of the
primary event (i.e., TCEs KIC 6666233.1 and KIC 10186945.1) was inaccurate
(Figure 18). Thus, there was no transit in the secondary test for ExoMiner to
capture and utilize to generate a correct classification. However, because we
do not remove other TCEs when processing our flux data for each TCE, the
secondary transit (i.e., transit corresponding to TCEs KIC 6666233.1 and KIC
10186945.1) was visible in the full-orbit view of KIC 6666233.2 and KIC
10186945.2, and the model could use them in order to classify these TCEs
correctly.

Another interesting NTP is TCE KIC 8052016.4. The branch contributing the most
to ExoMiner’s score for this TCE is the transit-view flux branch. A close look
at the transit-view of this TCE (Figure 19) reveals that the model captured
the unusual transit shape of this TCE and labeled it correctly. Other
interesting scenarios are provided by TCEs KIC 8043714.2 and KIC 9705459.2,
which are actually part of an EB but are labeled as NTP. In both cases, the
primary events were completely removed by the pipeline and the second TCE was
triggered by the remaining secondaries. However, the pipeline folded the
secondaries onto the data gaps left behind by the primary event, so the period
for the second TCE in each case was one half that of the primary. Because of
this, the depth of the odd and even views are different for the subsequent TCE
since the transits in one view are residuals of the original TCE while
transits in the other view represent another transit. The last example is TCE
KIC 5446285.2, which is correctly classified using the secondary event branch.
The target star associated with this TCE is Kepler-88 which is a Transit
Timing Variation (TTV) system and hosts multiple exoplanets. The TTVs for
Kepler-88 b are sufficiently large ($\pm$0.5 days) that the pipeline was not
able to lock on to the nominal orbital period. At the detected period, other
transits of the planet are actually identified as the secondary for TCE KIC
5446285.2 leading to its correct classification.

The results of this preliminary study indicate that identifying the maximally
contributing branch to each TCEﬂs predicted score holds a high degree of
explainability power for an extremely simple explainability technique. To
develop higher fidelity explainability tests that can be integrated into the
ExoMiner classification pipeline, several improvements to this technique will
be explored in future work. The model architecture diagram shown in Figure 4
shows that the stellar parameter scalars branch and the DV diagnostic scalars
branch exist at a different hierarchical level than the convolutional
branches. Rather than setting the inputs of each convolutional branch to zero,
if we instead set each branchﬂs contribution to the fully connected block to
zero, the FP flags we obtain may be more accurate. In particular, this is
important for the transit-view odd & even flux branch where an input of all
zeroes has a different meaning due to the subtraction layer. Achieving more
accurate flags of eclipsing binaries through the transit-view odd & even flux
branch may be possible by implementing noise sensitivity mapping, as discussed
in Section 3.3. More efficient gradient-based methods like Grad-CAM (Selvaraju
et al., 2019) will also be explored as components of a more robust
explainability testing for ExoMiner. Refining these explainability tests will
not only provide more information about the model’s classifications, but will
also provide insight into how the model can be improved in future iterations.
As ExoMiner is applied to TESS data, these explainability tests will be
helpful in providing additional confidence in the dispositions and pointing
out the limitations of the model for classifying new signals.

##  8 ExoMiner for exoplanet validation

###  8.1 Classification with Rejection

For applications where the cost of misclassification is high (e.g., validating
new exoplanets), we need ways to reduce the error rate of the classifier. This
is a relatively old concept in statistics (Chow, 1970) and a standard one in
ML (Cortes et al., 2016; Nadeem et al., 2010). It is called classification
with rejection. One classification with rejection approach widely used in ML
is based on failing to classify examples when the model is not very confident
of its label. The idea is that if the classifier cannot classify a case with
high confidence, then it is not sure about its label. When the classifier
predicts the label of an example with a very high score, it means that the
model has a good understanding of the feature space in which that example lies
(think about it in terms of $p(y|X)$). As we reported in Section 6.1, by
reducing the size of the training set or adding label noise, ExoMiner becomes
less confident about its disposition. This verifies that by setting a
confidence-based rejection option, we control the risk of misclassification.
To study confidence-based rejection, we plot in Figure 20 the precision of
different classifiers when different threshold values for rejection are used
to reduce the rate of false positives. As can be seen, all methods perform
reasonably well for large thresholds. Note that the precision of ExoMiner at
$T=0.99$ is around 99.7%, much higher than the required precision value of 99%
for that threshold. The precision of ExoMiner even for $T=0.9$ is around 99%.

It is important that one uses a model-specific threshold $T$ for
classification with rejection because the precision does not necessarily
increase with larger thresholds, e.g., check AstroNet and ExoNet for $T$
$>0.97$. However, given that $T=0.99$ has been accepted to use for validating
new exoplanets (Morton et al., 2016; Armstrong et al., 2020) and ExoMiner
performs very well for that threshold, in the remainder of this paper, we
focus on using this threshold for different analyses (in addition to the
standard classification threshold of 0.5).

Figure 20: Precision of different classifiers when a threshold $T$ is used to
reduce the false positive rate. Figure 21: Reliability plots of ExoMiner, GPC,
RFC, and vespa.

###  8.2 Classifier Calibration

In order to make a decision based on the output of a classifier (e.g., to
validate new exoplanets), we should be able to interpret the scores generated
by that classifier. If a classifier generates probability scores, we can
estimate the error rate for different probability scores, i.e., for a given
probability score value of $s$, the classifier correctly classifies $100\times
s\%$ of examples. Such classifiers are called calibrated. Most discriminative
models for machine classification, however, do not generate calibrated scores
even when their generated scores are in the $[0,1]$ range. Such classifiers
require calibration before their scores can be used for decision making,
especially when a decision must be taken for any scenario (e.g., consider a
self driving car making a decision to stop or keep moving). Because of this,
Armstrong et al. (2020) calibrated their classifiers prior to validating new
exoplanets using isotonic regression (Niculescu-Mizil & Caruana, 2005a), one
of the two most common calibration methods (Platt, 1999; Niculescu-Mizil &
Caruana, 2005a).

Table 9: The level of calibration for different methods. Lower values are better. | Classifier | ECE | MCE
---|---|---
vespa(1-fpp) | 0.246 | 0.68
GPC | 0.013 | 0.249
RFC | 0.009 | 0.145
ExoMiner | 0.001 | 0.067

To study the calibration of a classifier, people use calibration plots
(reliability curves) that show the error rate of the classifier for different
probability score values. A classifier is perfectly-calibrated if the error
rate is equal to the score it generates. Figure 21 shows the calibration plots
of uncalibrated ExoMiner, vespa 1-fpp, and two calibrated
classifiers252525Note that GPC is calibrated by design and RFC was calibrated
in post-processing. from Armstrong et al. (2020), GPC and RFC, for the full
dataset (using CV). Note that vespa fpp is only available for KOIs, so its
reliability plot only uses a subset of the data used for the other three
classifiers. The results and conclusion, with regard to the level of
calibration, do not change if we use only KOIs for all models. As can be seen,
ExoMiner appears well calibrated already. To measure the level of calibration
of these classifiers, we use Expected Calibration Error (ECE) and Maximum
Calibration Error (MCE) (Guo et al., 2017). ECE and MCE are, respectively, the
weighted average and maximum of the absolute difference between the
classifier’s average accuracy and score for each score bin. Table 9 shows the
level of calibration for different classifiers used for validation of new
exoplanets. As can be seen, the out-of-box ExoMiner is well calibrated and can
be used without further calibration. It has been previously shown that the
scores generated by certain classifiers such as neural networks are well-
calibrated Niculescu-Mizil & Caruana (2005b). Nonetheless, we tried both
isotonic regression (Niculescu-Mizil & Caruana, 2005a) and Platt scaling
(Platt, 1999) to further improve the calibration of ExoMiner. These
calibration techniques did not work well because the bins in the middle of the
reliability plots do not have enough points per fold, leading to poor post-
processing calibration. To overcome this problem, Armstrong et al. (2020)
generated synthetic examples by interpolating between members of each class.
Generating such synthetic datasets for diagnostic test time series is not easy
and does not guarantee better results. As such and because the level of
calibration of ExoMiner is already high, we use the scores generated by the
classifier directly for planet validation.

We would like to emphasize that the calibration of methods for scores $>0.99$
is of particular interest for validating new exoplanets. As can be seen in the
zoomed area of Figure 21, only ExoMiner is an underestimator classifier. A
classifier is an underestimator when the fraction of positive examples for a
given probability score is higher than the score assignment; i.e., the mean
predicted value assigned by such a classifier is lower than its precision.
This is an useful property for validating exoplanets because a classifier that
is an underestimator in the high score range (close to 1.0) makes fewer errors
when validating new exoplanets. GPC, RFC, and vespa are all overestimator
classifiers in that range, meaning that there is more than 1% error when
validating exoplanets using a threshold value of $0.99$. Note that these
results are consistent with Figure 20 and the classification with rejection
discussion.

Table 10: Scores of different classifiers trained in this work after applying priors. This table describes the available columns. The full table is available online. | Column | Description
---|---
target_id | KIC ID
tce_plnt_num | TCE planet number
tce_period | TCE period
mes | TCE MES
original_label | NTP, AFP, or PC
binary_label | 0 for NTP or AFP, 1 for PC
astronet_prior |  AstroNet score with priors
exonet_prior |  ExoNet score with priors
exominer_prior |  ExoMiner score with priors
exominer_tce_prior |  ExoMiner-TCE score with priors
Table 11: The effect of prior on the precision & recall values of different models. The boost in performance due to prior diminishes as the model uses more complete set of diagnostic tests. |  | Threshold=0.5 | Threshold=0.99
---|---|---
Model/Metric | Original | Using priors | Original | Using priors
AstroNet | 0.861 & 0.885 | 0.920 & 0.920 | 0.958 & 0.010 | 1.000 & 0.102
ExoNet | 0.925 & 0.864 | 0.933 & 0.902 | 0.983 & 0.025 | 0.993 & 0.123
GPC | 0.921 & 0.964 | 0.910 & 0.977 | 0.991 & 0.605 | 0.993 & 0.663
RFC | 0.929 & 0.955 | 0.942 & 0.964 | 0.979 & 0.724 | 0.986 & 0.791
ExoMiner | 0.968 & 0.974 | 0.971 & 0.978 | 0.997 & 0.656 | 0.997 & 0.752
ExoMiner-TCE | 0.965 & 0.977 | 0.968 & 0.983 | 0.997 & 0.775 | 0.996 & 0.823

###  8.3 Utilizing prior information

As we mentioned in Section 6.2, ExoMiner does not perform well under some
specific data scenarios. In order to address that, we utilize prior
probabilities for different scenarios as used in Morton & Johnson (2011);
Morton (2012); Armstrong et al. (2020). More specifically, Armstrong et al.
(2020) utilized prior probabilities for different scenarios (EB, Hierachical
EB, BEB, hierarchical transiting planet, background transiting planet, known
other stellar source, non-astrophysical/systematic, planet) and combined them
with their calibrated classifier scores to improve the performance of the
classifier. We take the same approach to combine the prior probability
calculated in Armstrong et al. (2020) with the scores of ExoMiner, with a
somewhat different justification, as follows. There are two sets of
probability scores computed from two different sources of information: (1)
$p(y=1|M,X,x^{*})$ that learner $M$ (in our case ExoMiner) generates for
signal $x^{*}$ using the information from the training set $X$ and (2)
$P(y=1|I)$ where $I$ represents target star’s individual characteristics
(stellar type, local environment, spacecraft status at the transit time,
etc.). In order to combine these two scores, assuming that (1) I and (2)
$x^{*}$, $M$, $X$ are conditionally independent (i.e.,
$p(x*,M,X,I|y)=p(x*,M,X|y)\times p(I|y)$), we can use the Bayes theorem to get
the following equation that is the identical equation as the one reported in
Armstrong et al. (2020):

$p(y=1|M,X,x^{*},I)=\frac{p(y=1|M,X,x^{*})p(y=1|I)}{\sum_{y}p(y|M,X,x^{*})p(y|I)}$
(10)

The results of applying this technique to different classifiers are reported
in Table 10, with the performance metrics reported in Table 11. As can be
seen, the performance of all classifiers improved by applying the prior
information $I$. However, the use of this prior information is more helpful to
those classifiers, e.g., AstroNet and ExoNet, that do not use the
comprehensive diagnostic test data provided by Kepler pipeline. The
performance of the classifiers that use a more comprehensive set of diagnostic
tests, such as ExoMiner and GPC, is not improved as much. Interestingly, the
application of the prior information $I$ to these classifiers often makes the
classifiers more confident about the labels. This is because if either
$p(y=1|M,X,x^{*})$ or $p(y=1|I)$ are close to $1$, $p(y=1|M,X,x^{*},I)$
becomes close to 1262626This is because if either of these probabilities is
close to 1.0, numerator and denominator in Equation 10 become equal.. Thus,
there will be more TCEs with scores $>0.99$. This is why the recall values for
all classifiers at threshold $0.99$ increase. ExoMiner is a conservative
classifier and applying prior information $I$ leads to a large recall
improvement at the threshold $0.99$.

We also checked the performance of ExoMiner updated scores using the scenario
priors for changes in training set size and label noise, as we have done in
Section 6.1. Even though we are not providing the plots here for conciseness,
using the scenario priors improves the performance of ExoMiner, more
significantly when the size of the training set is small or there is more
label noise. Given that using the scenario priors improves the performance of
ExoMiner, we use these updated ExoMiner scores for the validation of new
exoplanets.

###  8.4 Dependence on candidate parameters

In this section, we study ExoMiner dependence on candidate parameters using
the Cumulative KOI catalog. Given that we use the updated ExoMiner scores
using scenario priors, as described in Section 8.3, for validation of new
exoplanets, we will focus here on these updated scores. Table 12 shows
ExoMiner scores for different values of KOI radius and multiplicity. As can be
seen, ExoMiner generally assigns higher scores to KOIs in multiple systems
compared to single systems even though it does not have access to the
information related to the number of detected transits for each target star.
Moreover, ExoMiner assigned lower scores to larger KOIs.

Table 12: ExoMiner Scores by KOI radius and multiplicity. | Selection | Count | Mean | Median
---|---|---|---
All | 6464 | 0.379 | 0.002
KOI in multiple systems | 1313 | 0.857 | 0.996
KOI singles | 5151 | 0.257 | 0.000
$R_{p}<2R_{\oplus}$ | 2044 | 0.402 | 0.000
$2R_{\oplus}<R_{p}<4R_{\oplus}$ | 1902 | 0.661 | 0.993
$4R_{\oplus}<R_{p}<10R_{\oplus}$ | 682 | 0.351 | 0.001
$10R_{\oplus}<R_{p}<15R_{\oplus}$ | 263 | 0.336 | 0.011
$R_{p}>15R_{\oplus}$ | 1487 | 0.028 | 0.000

To study the performance with regards to candidate parameters above the
validation threshold, we plotted the precision and recall of ExoMiner for
scores $>0.99$ for different pairs of candidate parameters in Figure 22.
ExoMiner is reliable in different areas of candidate parameters for planet
validation. At the validation threshold of 0.99, ExoMiner is very conservative
in labeling KOIs as planets so it has high precision and low recall. Also,
note that there are regions of parameter space for which there are not enough
KOIs. Because there are not enough examples in those area in the training set,
the model is not confident about the scores even if it assigns the cases
correctly to the class. This shows a similar effect to the training set size
experiment we have done in Section 6.1.

(a) Precision for orbital period versus planet radius

(b) Recall for orbital period versus planet radius.

(c) Precision for orbital period versus MES.

(d) Recall for orbital period versus MES.

(e) Precision for orbital period versus galactic latitude.

(f) Recall for orbital period versus galactic latitude.

Figure 22: Precision and recall for different parameters when threshold 0.99
is used for validation.

In the low-MES region (MES $<10.5$), there is only one FP, KIC TCE 10187017.6,
out of a total of thirteen TCEs that have scores $>0.99$. Interestingly, this
is the 6th TCE of a system for which the first five TCEs are all confirmed
exoplanets. Thus, we believe this could be planet number six with a period
that fits between planet e and planet f of KOI-82. This object is marked as a
centroid offset false positive in the certified false positive table, but the
TCERT report does not show clear evidence for this offset, and upon review
this object’s original vetter for the false positive working group agrees that
the background false positive classification is an error due to not accounting
for saturation (Steve Bryson, personal communication).

Figure 23 shows the distribution of scores for planet radius, MES, and
galactic latitude of KOI versus KOI orbital period. As can be seen, short- and
long-period KOIs, KOIs in the low-MES region, and KOIs at lower galactic
latitudes are all assigned low scores.

(a) Log orbital period versus log planet radius.

(b) Log orbital period versus log MES.

(c) Log orbital period versus galactic latitude.

Figure 23: Mean score of KOIs for different parameters. The number in each bin
shows the total number of KOIs in that bin. There are no KOIs in white bins.

##  9 Planet Validation

###  9.1 Setup

Similar to existing planet validation approaches, we use the rejection
threshold of $>0.99$ to validate new exoplanets. To make the validation
process more reliable, we need to identify where ExoMiner does not perform
well in order to design vetoing criteria to improve the validation process
even further.

As we detailed in Section 6.2, ExoMiner classifies some FPs into PC category
for different reasons: lack of enough information such as that given by the
contamination flag, or poor interpretation of diagnostic tests such as the
centroid test. As we discussed in Section 8.3, utilizing scenario priors
addressed some of these misclassifications and improved the performance.
Nonetheless, there are still cases that are not resolved by the application of
scenario priors, even though ExoMiner is making less than one error out of
every 100 TCEs that exceed the rejection threshold of 0.99. To be precise,
ExoMiner classifies 1727 TCEs with score $>0.99$, of which there are five
FPs272727These numbers are for the adjusted ExoMiner scores using scenario
priors. The total number of TCEs with original scores $>0.99$ is 1507, of
which there are four FPs, leading to a precision of 0.997, better than the
required precision of $0.99$ for the threshold of $0.99$. To further improve
the precision of ExoMiner in validating new exoplanets, we ensure that the KOI
is dispositioned as a ‘candidate’ in the Cumulative KOI catalog (similar to
Morton et al., 2016) and that no disposition flag is on in the Cumulative KOI
catalog. These vetoes remove those five FPs from the validated list. Regarding
low-MES region, even though the model performs well in this region282828There
was only one FP in the low-MES region whose correct disposition might be
planet. (see Figure 22(c)), we take the more conservative approach and do not
validate any KOI with MES $<10.5$.

Unlike Armstrong et al. (2020), we do not recommend using vespa fpp scores to
veto the validated exoplanets for multiple reasons:

  * •

vespa is poorly calibrated, as shown in Figure 21. As an overestimator
classifier, its scores cannot be used for validation because the error for any
rejection threshold is higher than that threshold, as seen in in Figures 21
and 24. As an example, vespa gives scores below $0.01$ to 2409 TCEs, of which
446 TCEs are FPs, leading to a precision of 0.815, which is much lower than
the required value of 0.99 for the applied threshold. As we mentioned in
Section 9, Morton et al. (2016) used other criteria in order to improve the
vetting. This led to a highly accurate approach for exoplanet validation in
terms of precision, but very low recall.

  * •

Inspired by Armstrong et al. (2020), we use the scenario priors $I$ as
complementary sources of information. These scenario priors are also used in
calculating vespa fpp scores. Thus, we are using part of the vespa model. By
only using scenario priors, we ignore the likelihood used to calculate vespa
fpp. We discussed in Section 2.2 the problems with the generative approach and
why a discriminative approach is preferred for classification problems.

  * •

The vespa model is highly dependent on the accuracy of stellar parameters.
Given the uncertainty about the values of stellar parameters, vespa fpp is
subject to continuous change.

  * •

As discussed in Section 5.4, when there is a disagreement between ExoMiner and
vespa scores for validating exoplanets at a threshold of 0.99, ExoMiner is
almost always correct. Thus, the advantage of vetoing based on vespa fpp is
not clear.

We also do not suggest using an outlier detection test to veto validated
exoplanets for two main reasons:

  * •

Unsupervised outlier detection is a subjective task where the results vary
from one algorithm to another or by changing the hyper-parameters of the same
algorithm; i.e., one can always come up with an outlier detection algorithm
such that some of these validated exoplanets are outliers.

  * •

By using the rejection threshold, we already identified TCEs for which the
model has not seen similar cases, as discussed in Section 8.1.

Nonetheless, as reported in Section 9.2, all validated exoplanets in this work
pass the outlier tests of Armstrong et al. (2020).

Figure 24: Distribution of CPs and CFPs as function of vespa fpp score. There are 596 CFPs out of a total of 2719 CPs+CFPs with vespa fpp $\leq 0.05$. This implies a misclassification rate of 21.9% if we use vespa fpp $\leq 0.05$ to validate new exoplanets. Not shown in this picture: there are 403 CFPs with vespa fpp $\leq 0.01$ out of 2372 CPs+CFPs (a misclassification rate of 17.0%). These results demonstrate that vespa fpp is not effective for screening against AFPs at the 0.01 confidence level in the absence of other constraints (see Section 9). Table 13: Original ExoMiner score and ExoMiner score after applying priors for the unlabeled KOIs. This table describes the available columns. The full table is available online. | Column | Description
---|---
target_id | KIC ID
tce_plnt_num | TCE planet number
tce_period | TCE period
mes | TCE MES
exominer | original ExoMiner score
exominer_prior |  ExoMiner score with priors

###  9.2 Validated Exoplanets Using ExoMiner

To validate new exoplanets, we apply ten ExoMiner models trained using 10-fold
CV to those KOIs that are not part of our working set, i.e., KOIs that are not
CPs, CFPs, or CFAs. We will designate this subset by unlabeled KOI set. There
are a total of 1922 such KOIs in the Cumulative KOI catalog from the Q1-Q17
DR25 run. We use the average score of ten models to classify these KOIs. Table
13 shows the scores provided by ExoMiner before and after applying priors for
the unlabeled KOI set. Ephemerides and other relevant parameters are also
presented. Given that a majority of TCEs in the training set are NTPs, and the
KOIs from which we are validating do not include any NTPs, the rate of FPs in
the training set is much higher than that of the unlabeled KOI set. Thus, the
probability scores generated by ExoMiner are more conservative in labeling
these KOIs as PC and validating them. ExoMiner labels 1245292929this number is
1186 for original ExoMiner scores without applying scenario priors. KOIs as PC
(scores $>0.5$) out of 1922 KOIs. Other existing classifiers also provide
similar results. For example, for posterior GPC (Armstrong et al., 2020), this
number is 1126 KOIs.

Out of these 1245 positively labeled KOIs by ExoMiner, eight KOIs are
dispositioned as ‘False Positive’ or have an FP flag on in the Cumulative KOI
catalog. Given that most of these positively labeled KOIs are already
dispositioned as ‘candidate’ in the Cumulative KOI catalog, this number should
not be surprising. One possible justification for why there are so many
candidates left in the unlabeled set could be because vetting false positives
is easier than confirming PCs. A TCE is certified as a false positive if one
of the diagnostic tests indicates a FP. However, it is not enough to confirm
an exoplanet if none of the diagnostic metrics is inconsistent with a true
transit signal; confirming new exoplanets requires follow-up studies or
statistical/ML validation.

Out of 1922 unlabeled KOIs, ExoMiner labels 368 KOIs as PC with validation
score $>0.99$, of which 46 are among the previously validated exoplanets by
Armstrong et al. (2020) and one among the previously validated by Jontof-
Hutter et al. (2021), leaving 321 new ones. Interestingly, out of a total of
943 unlabeled KOIs with MES $<10.5$, ExoMiner gives a score $>0.99$ to twenty
KOIs, even though 839 of them have a ‘candidate’ disposition in the Cumulative
KOI catalog. This implies that ExoMiner is more conservative in the low-MES
region, as discussed in Section 8.4. Removing these 20 KOIs with MES $<10.5$,
we validate 301 new exoplanets. Out of these 301 newly validated exoplanets,
42 KOIs are in multiplanet systems. We provide the list of newly validated
exoplanets in Table 14. None of these newly validated exoplanets are found to
be outliers based on the two algorithms in Armstrong et al. (2020). We also
report those KOIs with ExoMiner scores $>0.99$ in the low-MES region in Table
15, even though we are not validating them. Nine out of twenty such KOIs are
in multiplanet systems.

Table 14: List of newly validated exoplanets with ExoMiner score $>0.99$ and MES $>10.5$. The full list is available online. | Number | TCE KIC | KOI Name | Period (days) | Radius (Re) | MES |  ExoMiner score |  vespa Score | Kepler Name
---|---|---|---|---|---|---|---|---
1 | 11852982.1 | K00247.01 | 13.8150 | 2.2837 | 41.4900 | 0.9989 | 0.0076 | Kepler-1712b
2 | 7700622.1 | K00315.01 | 35.5810 | 2.4248 | 73.5900 | 0.9981 | 0.0000 | Kepler-1716b
3 | 7135852.1 | K00875.01 | 4.2210 | 3.8940 | 134.5000 | 0.9981 | 0.0009 | Kepler-1729b
4 | 6803202.1 | K00177.01 | 21.0608 | 1.9230 | 43.0500 | 0.9959 | 0.0005 | Kepler-1711b
5 | 2444412.1 | K00103.01 | 14.9111 | 3.1974 | 108.0000 | 0.9980 | 0.0001 | Kepler-1710b
6 | 7941200.1 | K00092.01 | 65.7046 | 3.1242 | 81.8900 | 0.9977 | 0.2000 | Kepler-1709b
7 | 8087812.1 | K04343.01 | 27.2108 | 1.2392 | 13.3900 | 0.9931 | 0.0005 | Kepler-1952b
8 | 7749773.1 | K02848.01 | 13.7873 | 1.4867 | 16.2100 | 0.9983 | 0.0006 | Kepler-1882b
9 | 11599038.1 | K01437.01 | 7.0173 | 1.8242 | 12.5500 | 0.9981 | 0.0001 | Kepler-1751b
10 | 10587105.3 | K00339.03 | 35.8662 | 2.7950 | 18.2200 | 0.9981 | 0.0006 | Kepler-529d
11 | 3645438.2 | K04385.01 | 7.2980 | 1.9421 | 12.2000 | 0.9988 | 0.0004 | Kepler-1600c
12 | 4665571.2 | K02393.01 | 4.6030 | 1.4594 | 14.9800 | 0.9980 | 0.0002 | Kepler-1834b
13 | 7364176.1 | K00373.01 | 135.1880 | 2.5685 | 62.9900 | 0.9980 | 0.8100 | Kepler-1718b
14 | 9823487.1 | K01489.01 | 16.0046 | 2.2268 | 27.3200 | 0.9986 | 0.0000 | Kepler-1753b
15 | 5374854.1 | K00645.02 | 23.7832 | 2.2960 | 28.8000 | 0.9957 | 0.0035 | Kepler-633c
16 | 8410415.1 | K02291.01 | 44.2992 | 2.2735 | 19.3700 | 0.9980 | 0.1200 | Kepler-1817b
17 | 3937519.2 | K00221.02 | 5.8966 | 1.2194 | 12.1100 | 0.9986 | 0.0017 | Kepler-495c
18 | 10134152.1 | K02056.01 | 39.3137 | 2.3190 | 25.3000 | 0.9984 | 0.0280 | Kepler-1787b
19 | 9150827.1 | K01408.01 | 14.5341 | 1.6130 | 25.6200 | 0.9986 | 0.0010 | Kepler-1749b
20 | 3641726.1 | K00804.01 | 9.0293 | 3.1732 | 47.3700 | 0.9975 | 0.0000 | Kepler-1727b
21 | 12121570.1 | K02290.01 | 91.5010 | 1.9679 | 18.6600 | 0.9925 | 0.0003 | Kepler-1816b
22 | 11046025.1 | K01646.01 | 3.7202 | 0.9038 | 14.0200 | 0.9972 | 0.0007 | Kepler-1759b
23 | 9837661.3 | K02715.03 | 5.7209 | 2.6284 | 15.0600 | 0.9974 | 0.1800 | Kepler-1321d
24 | 11821363.1 | K01494.01 | 8.1959 | 2.4400 | 17.2000 | 0.9980 | 0.0009 | Kepler-1754b
25 | 10074466.1 | K01917.01 | 35.0119 | 2.6116 | 27.8000 | 0.9977 | 0.0730 | Kepler-1776b
26 | 10386922.2 | K00289.01 | 26.6295 | 4.2096 | 76.4700 | 0.9974 | 0.0000 | Kepler-511c
27 | 7584650.1 | K02631.01 | 44.9976 | 2.2631 | 15.9600 | 0.9968 | 0.0008 | Kepler-1863b
28 | 7670943.1 | K00269.01 | 18.0116 | 1.7167 | 38.8800 | 0.9962 | 0.0005 | Kepler-1713b
29 | 5601258.1 | K02191.01 | 8.8479 | 1.5450 | 18.3500 | 0.9979 | 0.0650 | Kepler-1805b
30 | 7826659.1 | K02686.01 | 211.0340 | 3.1495 | 59.0000 | 0.9959 | 0.0001 | Kepler-1868b
31 | 7386827.1 | K01704.01 | 10.4189 | 2.4951 | 36.8400 | 0.9987 | 0.0250 | Kepler-1765b
32 | 9307509.1 | K02096.01 | 9.7630 | 1.7943 | 21.3900 | 0.9983 | 0.0260 | Kepler-1794b
33 | 6183511.1 | K02542.01 | 0.7273 | 1.2162 | 14.7400 | 0.9967 | 0.0054 | Kepler-1855b
34 | 7877978.1 | K02760.01 | 56.5732 | 2.2111 | 18.7700 | 0.9918 | 0.0029 | Kepler-1874b
35 | 5436013.1 | K02471.01 | 25.8860 | 1.4891 | 13.9800 | 0.9957 | 0.0053 | Kepler-1841b
36 | 11297236.1 | K01857.01 | 88.6440 | 4.0554 | 38.3600 | 0.9911 | 0.0000 | Kepler-1771b
37 | 12469800.1 | K02543.01 | 1.3020 | 1.7345 | 19.0400 | 0.9946 | 0.0044 | Kepler-1856b
38 | 9836959.1 | K01715.01 | 105.0150 | 2.2790 | 25.7600 | 0.9970 | 0.0045 | Kepler-1766b
39 | 3940418.1 | K00810.01 | 4.7830 | 2.9550 | 57.6400 | 0.9986 | 0.0370 | Kepler-1728b
40 | 5773121.1 | K04002.01 | 0.5242 | 1.5148 | 18.3500 | 0.9945 | 0.0160 | Kepler-1932b
41 | 5020319.1 | K00635.01 | 16.7199 | 2.8125 | 51.8700 | 0.9978 | 0.0170 | Kepler-1725b
42 | 5978361.1 | K00558.01 | 9.1786 | 2.8672 | 44.1000 | 0.9984 | 0.0280 | Kepler-1722b
43 | 3109930.1 | K01112.01 | 37.8103 | 3.8766 | 28.3800 | 0.9979 | 0.0130 | Kepler-1736b
44 | 9475552.2 | K02694.02 | 6.5661 | 1.8440 | 19.8300 | 0.9987 | 0.3000 | Kepler-1315c
45 | 10793172.2 | K02871.02 | 5.3637 | 0.9770 | 11.9800 | 0.9969 | 0.0190 | Kepler-1693c
. | . | . | . | . | . | . | . | .
. | . | . | . | . | . | . | . | .
. | . | . | . | . | . | . | . | .
Table 15: List of KOIs with ExoMiner score $>0.99$ and MES $<10.5$. | Number | TCE KIC | KOI Name | Period (days) | Radius (Re) | MES |  ExoMiner score |  vespa Score
---|---|---|---|---|---|---|---
1 | 6871071.4 | K02220.04 | 7.6648 | 1.6288 | 9.7590 | 0.9969 | 0.0005
2 | 10793576.1 | K03451.01 | 10.9076 | 1.6938 | 9.2950 | 0.9971 | 0.0005
3 | 12204137.1 | K04590.01 | 9.5406 | 2.0832 | 9.9220 | 0.9958 | 0.0850
4 | 9896018.3 | K02579.03 | 10.3015 | 1.6263 | 9.2720 | 0.9927 | 0.0002
5 | 7289317.2 | K02450.02 | 7.1929 | 1.0897 | 9.6700 | 0.9918 | 0.0008
6 | 9011877.1 | K05597.01 | 7.7771 | 0.9646 | 10.1700 | 0.9940 | 0.0003
7 | 9457948.1 | K04694.01 | 19.2527 | 1.6884 | 9.5330 | 0.9961 | 0.0015
8 | 4919550.1 | K04766.01 | 5.5668 | 1.1317 | 10.3100 | 0.9980 | 0.0380
9 | 4164922.2 | K03864.02 | 18.2574 | 1.0389 | 10.2100 | 0.9962 | 0.0007
10 | 2860656.1 | K04857.01 | 4.1150 | 1.8554 | 9.0710 | 0.9944 | 0.0001
11 | 9489524.4 | K02029.04 | 4.7885 | 0.7428 | 9.9180 | 0.9966 | 0.0110
12 | 7098255.1 | K04762.01 | 11.5673 | 1.4018 | 9.7880 | 0.9919 | 0.0790
13 | 7692093.1 | K03337.01 | 11.1662 | 1.5631 | 10.1600 | 0.9933 | 0.0012
14 | 7100673.5 | K04032.05 | 7.2352 | 0.9662 | 9.0370 | 0.9903 | 0.2000
15 | 6131130.1 | K03307.01 | 10.5071 | 1.4087 | 9.4610 | 0.9918 | 0.0250
16 | 5542466.3 | K01590.03 | 4.7467 | 1.5175 | 9.5630 | 0.9927 | 0.0071
17 | 5903749.1 | K03029.01 | 18.9763 | 2.4880 | 10.2100 | 0.9931 | 0.0150
18 | 6144039.1 | K04602.01 | 2.8586 | 0.8922 | 10.2000 | 0.9916 | 0.0170
19 | 9002538.2 | K03196.02 | 6.8830 | 0.7757 | 10.1100 | 0.9916 | 0.4700
20 | 8874090.2 | K01404.02 | 18.9063 | 1.7561 | 10.1300 | 0.9941 | 0.0014

Figure 25 displays a scatter plot of the planetary radius versus orbital
period for the previously confirmed and validated exoplanets from the Kepler,
K2 and TESS missions along with the 301 new Kepler exoplanets validated by
ExoMiner. The distribution of the ExoMiner-validated exoplanets is consistent
with that of the Kepler sample, with periods ranging from $\sim$0.5 days to
over 280 days, and with planetary radii as small as $0.6\,R_{\oplus}$ to as
large as $9.5\,R_{\oplus}$. Figure 26 shows a scatter plot of the planetary
radius versus the energy received by each planet in the Kepler, K2, TESS and
ExoMiner samples. As with the previous parametric plot, the distribution of
the radius versus received energy of the ExoMiner sample generally follows
that of the Kepler sample.

Figure 25: Planet radius vs. orbital period for confirmed transiting planets
and ExoMiner-validated planets with a score greater than 0.99. The CPs are
indicated by the discovery source with Kepler indicated by ‘+’, K2 by a
diamond ‘$\diamond$’, and TESS by ‘*’. ExoMiner-validated planets are
indicated by a magenta square. For reference, Earth is indicated by a black
circle. Figure 26: Planet radius vs. energy received by planet for confirmed
transiting planets and ExoMiner-validated planets with a score greater than
0.99. Symbols are the same as in Fig. 25.

##  10 Transfer Learning to TESS data

Training a machine classifier to vet TESS data is challenging because there is
a limited number of TCEs with gold standard dispositions available for the
TESS Mission. Existing works try to overcome the challenge of limited data by
using either synthetic data (Osborn et al., 2020) or self-dispositions based
on high level heuristics (Yu et al., 2019b) in order to train a machine
classifier to vet TESS TCEs. However, it is not easy to ensure that simulated
signals, including transiting planets, astrophysical signals and systematic
noise and artifacts, are sufficiently similar to the flight data; when used
for training, these synthetic data may harm the performance of the model. The
self-disposition approach is also noisy and could lead to an ineffective model
that is not able to vet TCEs accurately. Instead, we argue that if the
architecture of a machine classifier is general enough for the classification
of TCEs and the data are preprocessed appropriately, a trained classifier
should be able to vet TCEs from different missions comparably well. The
processes for manually vetting transit signals from TESS and Kepler are very
similar; i.e., independent of the source of the TCEs, vetting a TCE requires
evaluating the same diagnostic plots and metrics. If a TCE passes these tests,
it is vetted as a PC no matter where the data come from. Thus, we can transfer
(Ng, 2016) a model learned from Kepler data to vet TESS TCEs.

We conducted a preliminary analysis of ExoMiner’s performance at classifying
transit signals obtained from the ongoing TESS Mission. The Science Operations
Center (SOC) pipeline, which was created to process the data from the Kepler
Mission, was ported and tuned for TESS data in the Science Processing
Operations Center (SPOC, Jenkins et al., 2016). The SPOC pipeline is currently
used to generate TCEs that can be further designated as TESS Objects of
Interest (TOIs, Guerrero et al., 2021). Even though the population of target
stars observed by TESS303030TESS target stars are on average 30 to 100 times
brighter than those surveyed by the Kepler Mission. and the characteristics of
the systematic noise are different, the nature of the signals captured by the
two satellites is similar. In this first analysis, the same preprocessing
steps were applied to the TESS data to generate the input data to feed into
the model trained exclusively on data from the Kepler Mission.

One specific feature of TESS data is that the pipeline is run for each
individual sector, but there are also multi-sector runs that are conducted
using data from multiple sectors. The single-sector runs produce TCEs in the
approximately 28 day-long observation window, so the detected TCEs have
orbital periods shorter than this time interval313131A constraint is defined
such that TESS TCEs must have at least two observed transits.. Since there is
overlap between sectors, especially at the ecliptic poles, multi-sector runs
are able to produce TCEs with longer orbital periods when transits are
observed in multiple sectors. Currently, the data for the first 36 sectors are
publicly
available323232https://archive.stsci.edu/tess/bulk_downloads/bulk_downloads_ffi-
tp-lc-dv.html, last accessed on 3-9-2021 as well as the TCEs detected by the
SPOC
pipeline333333https://archive.stsci.edu/tess/bulk_downloads_bulk_downloads_tce.html,
last accessed on 3-9-2021. A total of approximately 42,500 TCEs were detected,
with an average of $\sim 1200$ TCEs per observed sector. Spanning from Sector
1 to Sector 26, nine multi-sector runs were conducted, yielding a total of
$\sim 29,500$ TCEs. From these runs, experts vetted the detected TCEs and
combine data from multiple sources to arrive at a catalog of
TOIs343434https://archive.stsci.edu/missions/tess/catalogs/toi/tois.csv
(Guerrero et al., 2021).

The TOI catalog made publicly available on 4 December 2020, which covered
Sectors 1-30, was used to generate the data on which we evaluated ExoMiner’s
performance. After preparing the TOI catalog (removing TOIs with missing
ephemerides or observation sectors), 1167 SPOC TOIs were preprocessed using
our pipeline to generate the data inputs to ExoMiner. Given that the TOI
catalog does not include the phase of the secondary event, ephemeris matching
between the TOIs and the detected TCEs was performed such that our
preprocessing pipeline could generate the transit-view secondary eclipse time
series. In this preliminary analysis of TESS data, we trained a new model on
Kepler Q1-Q17 DR25 TCE data using only the time series and stellar parameters
as features. We call this model ExoMiner-Basic. The classifications produced
by the model were compared against the TESS Follow-up Observing Program
Working Group (TFOPWG – Collins et al., 2018) dispositions, which are based on
additional observations and are the current gold standard dispositions. Our
analysis focuses on the TOIs dispositioned as Known Planets (KPs), CPs, FPs,
and FAs. We did not include the TOIs with PC disposition because this
disposition is not conclusive.

Table 16: Confusion matrix of TESS classification using a basic version of ExoMiner | Class | PC | Non-PC | Total
---|---|---|---
KPs | 185 | 21 | 206
CPs | 32 | 57 | 89
FPs | 27 | 67 | 94
FAs | 2 | 16 | 18
Total | 246 | 161 | 407

Table 16 shows the confusion matrix of applying ExoMiner-Basic to TESS data
for the standard classification threshold of 0.5. ExoMiner-Basic performs
reasonably well on the TESS data, resulting in a precision value of 0.88 and a
recall value of 0.73. We expect the performance of the model to increase if
the model has access to the DV diagnostics that the full ExoMiner model uses
for classifying Kepler data. However, an analysis of the misclassified KP and
CP TOIs gave insight into some issues that must be addressed in the future, as
shown in Figures 27, 28, and 29.

Figure 27: Odd & even transit-view flux phase folded time series generated for
TOI 822.01. The blue dots show the average bin value, and the black dots show
the cadences in the phase folded time series. Figure 28: Full-orbit flux time
series for TCE TIC 4897275.1 (associated with TOI 1774.01) obtained from the
DV summary generated by the SPOC pipeline for Sector 21.

There are cases of TOIs with transits located inside the downlink window for
TESS. Data are downlinked to Earth at the perigee, which happens approximately
every 13.7 days and lasts about a day. During that time, new data are not
collected, which prevents the observation of some transits. For example, TOI
822.01 (WASP-132b) was observed in Sector 11 and has an orbital period of
approximately 7.13 days, but the corresponding TCE (TIC 127530399.1) is
reported as having double the period. Given that one transit was located in
the downlink window, TPS did not fold the missing transit onto the observed
transits, incorrectly determining the period of the TOI, which is known from
previous observations. Since we are using the true orbital period from the TOI
catalog, the preprocessing pipeline generates one of the odd & even transit-
views to be flat (Figure 27), rendering this diagnostic test unreliable and
leading the model to incorrectly classify this TOI as a false positive, even
though it is a KP. As we discussed earlier, this problem can be addressed by
using additional input features that take into account the number of observed
transits and their variability for the odd and even views.

Figure 29: Inaccurate epoch information leads to off-centered transit-views.

Another issue with TOIs that only have a small number of observed transits is
that averaging the phase folded time series is not enough to filter the
stochastic noise in the time series; this becomes even more critical when S/N
is low. We expect to see more of these cases in TESS data considering that for
significant areas of the sky, the total observation time for many TCEs is much
shorter than for Kepler, which had a fixed field of view that was observed for
four years. For example, TOI 1774.01 was observed in Sector 21 and only has
two observed transits, as shown in Figure 28, which creates flux views with
significant levels of noise.

A further issue arises when incorrect epoch values are reported in the TOI
catalog because it causes the transit to be off-centered in the view. Since
ExoMiner expects the transit to be located in the center of the view, the
model classifies such TOIs as false positives. For example, the transit for
TOI 185.01 is shifted to the right in our view, as shown in Figure 29. A
possible approach that can be used to make the model less sensitive to the
location of the transit source in the views is to provide the model examples
in which the transit is slightly shifted from the center of the view during
training. Using such a data augmentation technique may make the model more
robust to inaccuracies in the orbital period and epoch estimates.

##  11 Discussion

We introduced ExoMiner, a new deep learning classifier that outperforms
existing machine classifiers in terms of various classification and ranking
metrics. ExoMiner is a robust and explainable deep learning model that allows
us to reliably and accurately build new catalogs of PCs. The key to ExoMiner’s
performance is its architecture that mimics the process by which domain
experts vet transit signals by examining multiple types of diagnostic tests in
forms of scalar values and time series data. This architecture also enabled us
to design a preliminary branch-occlusion explainability framework that
provides interpretability in terms of which diagnostic tests ExoMiner utilizes
in order to vet a signal. Moreover, since the general concept behind vetting
transit signals is the same for both Kepler and TESS data, and ExoMiner
utilizes the same diagnostic metrics as expert vetters do, we expect an
adapted version of this model to perform well on TESS data. Our preliminary
results on TESS data verify this hypothesis. Using ExoMiner, we also
demonstrate that there are hundreds of new exoplanets hidden in the 1922 KOIs
that require further follow-up analysis. Out of these, 301 new exoplanets are
validated with confidence using ExoMiner.

###  11.1 Caveats

There are caveats related to the ML approach taken in this work, which
include:

  * •

As we reported in Section 6.2, ExoMiner sometimes fails to adequately utilize
diagnostic tests, e.g., centroid test, or does not have the data required for
some FP scenarios, e.g., flux contamination. This leads to misclassification
of transit signals. To provide the extent to which this causes problems, some
statistics are in order. Out of a total of 8053 KOIs, there are 1779 KOIs that
have centroid test flag on. ExoMiner misclassifies 23 (0.29%) of these of
which only one had a score $>0.99$ (0.01%). After applying the priors, the
number of misclassified cases reduces to thirteen (0.16%) out of which two
(0.025%) have scores $>0.99$. Out of 1087 KOIs that have flux contamination,
ExoMiner misclassifies sixteen (0.2%) and give a score $>0.99$ to only one
(0.01%). After applying the priors, only one KOI with flux contamination is
misclassified by ExoMiner with a score of 0.84. Overall, applying the scenario
priors alleviates this problem; however as we have suggested in the validation
process discussed in Section 9.1, the classification performed by ExoMiner
should be vetoed using FP flags provided in the Cumulative KOI catalog in
order to reduce the risk of FPs being validated as exoplanets.

  * •

The assumption made in Equation 10 that $x$ and $I$ are independent might not
be valid. This is because the information used in $I$, e.g., crowded field,
might be present in the flux or centroid data ($x$). When this assumption does
not hold, we are counting evidences for or against supporting an exoplanet
classification twice (one from $I$ and another from $x$). Thus, the the
application of the priors could lead to underestimating or overestimating the
true probability. However, as we reported in Table 10, ExoMiner and other
existing classifiers benefit from the scenario priors overall in terms of
precision and recall. Therefore, the application of priors is beneficial.

  * •

The performance comparison between different classifiers should also be taken
with caution due to the existence of label noise. However, the relative
performance of different classifiers is reasonably sound, as we showed in
Section 5.2 and 5.3 using additional independent datasets.

  * •

In section 6.1, we only studied the effect of random label noise and
underrepresented scenarios in the training set. However, the true nature of
label noise or underrepresented scenarios could be systematic, limiting the
reliability of our analysis. The presence of systematic label noise or
underrepresented data scenarios could potentially lead to misclassification of
FP scenarios for which there is significant label noise or lack of enough
representative samples.

There are also astrophysical caveats, which include:

  * •

Given that there is no viable way to distinguish between BD FPs and real
exoplanets using only Kepler photometry, ExoMiner and all similar photometry
based data-driven models are not able to correctly classify BD FPs. However,
BDs are very rare in Kepler data: Santerne et al. (2016) found only three BDs
in a follow-up study of 129 giant planet candidates from Kepler. The authors
also calculated a BD occurrence rate of 0.29% for periods less than 400 days,
15 times lower than what they find for giant planets. Considering the
geometric probability of alignment, which we can assume is $\sim$5%
(conservatively), the chance of seeing a transiting BD is $\sim 0.05\times
0.29\approx 0.01\%$. Moreover, note that there is only a handful of validated
large planets in our list, i.e., 17/301 with
$4\,R_{\oplus}<R_{p}<10\,R_{\oplus}$ and only 7/301 with
$5\,R_{\oplus}<R_{p}<10\,R_{\oplus}$. Thus, given that transiting BDs can
often be identified via RV measurements, and the low occurrence rate of such
systems, the BD FPs should be of little concern.

  * •

There could exist blended EBs that do not exhibit secondary eclipses but might
be exposed by high resolution imaging and photometric observations with better
spatial resolution. There could still exist relevant patterns in the
photometry data for such objects that machine can utilize to correctly
classify them, even though those are not easy for human vetters to find and
use. However, we did not provide any specific studies of the performance of
deep learning on such objects in this work. Thus, we assume ExoMiner is not
able to detect such FPs.

  * •

Due to the different stellar samples, pixel scales and blending issues, TESS
will have a different distribution of false positives, and this will affect
classifier results. Thus, special care needs to be taken transferring to TESS
where validation is concerned.

###  11.2 Future direction

The analysis performed in this paper will allow us to pinpoint a number of
future directions for improving and building upon this work. First, we will
change the ExoMiner architecture to include new important features, including
1) the number of transits and the uncertainty about the depth of the odd and
even views to ensure that the model’s use of the odd & even branch is more
effective and 2) the Kepler magnitude to allow the model to recognize
saturated stars for which the centroid test is invalid. Second, we will
perform a full explainability study by utilizing more advanced explainability
techniques and design a framework that can provide experts with insight about
ExoMiner’s labels and scores. Third, we will be doing more depth analysis of
the effect of label noise and underrepresented scenariors and provide a list
of possible inaccurate labels in the Q1-Q17 DR25 dataset. Finally, we will
soon publish the results of utilizing an adapted version of ExoMiner to
classify TESS transit signals by extending the set of features in a similar
way to the one used for Kepler data, and addressing some of the issues
mentioned previously that affect TESS data.

## Acknowledgements

Hamed Valizadegan, Miguel Martinho, Laurent Wilkens, and Nikash Walia are
suported through NASA NAMS contract NNA16BD14C. Jeffrey Smith, Douglas
Caldwell, and Joseph Twicken are supported through NASA Cooperative Agreement
80NSSC21M0079. We would like to thank multiple people who directly or
indirectly contributed to this work. We are very grateful to Porsche M. Parker
and Haley E. Feck from USRA and Krisstina Wilmoth from NASA ARC who have
helped us tirelessly to recruit interns through NASA ARC $I^{2}$ intern
program and NASA ARC Office of STEM Engagement. Without these amazing interns,
this work would not have been possible. We are grateful to Patrick Maynard,
who has recently joined our team as an intern and generated new insights into
TESS data. We also appreciate David J. Armstrong in providing us with the
detailed results of their ML models (Armstrong et al., 2020), and Megan
Ansdell, who answered our questions regarding ExoNet code (Ansdell et al.,
2018). Our discussion with David J. Armstrong on the caveats of using ML for
this problem helped us tremendously in improving this work. This paper
includes data collected by the Kepler and TESS missions and obtained from the
MAST data archive at the Space Telescope Science Institute (STScI). Funding
for the Kepler mission is provided by the NASA Science Mission Directorate.
Funding for the TESS mission is provided by the NASA Explorer Program. STScI
is operated by the Association of Universities for Research in Astronomy,
Inc., under NASA contract NAS 5–26555. We acknowledge the use of public TESS
data from pipelines at the TESS Science Office and at the TESS Science
Processing Operations Center. Resources supporting this work were provided by
the NASA High-End Computing (HEC) Program through the NASA Advanced
Supercomputing (NAS) Division at Ames Research Center for the production of
the SOC Kepler and the SPOC TESS data products.

## References

  * Abadi et al. (2015) Abadi, M., Agarwal, A., Barham, P., et al. 2015, TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.  https://www.tensorflow.org/
  * Ansdell et al. (2018) Ansdell, M., Ioannou, Y., Osborn, H. P., et al. 2018, The Astrophysical Journal, 869, L7, doi: 10.3847/2041-8213/aaf23b
  * Armstrong et al. (2020) Armstrong, D. J., Gamper, J., & Damoulas, T. 2020, Exoplanet Validation with Machine Learning: 50 new validated Kepler planets.  https://arxiv.org/abs/2008.10516
  * Armstrong et al. (2016) Armstrong, D. J., Pollacco, D., & Santerne, A. 2016, Monthly Notices of the Royal Astronomical Society, 465, 2634, doi: 10.1093/mnras/stw2881
  * Baglin et al. (2006) Baglin, A., Auvergne, M., Barge, P., et al. 2006, in ESA Special Publication, Vol. 1306, The CoRoT Mission Pre-Launch Status - Stellar Seismology and Planet Finding, ed. M. Fridlund, A. Baglin, J. Lochard, & L. Conroy, 33
  * Bengio et al. (2012) Bengio, Y., Courville, A. C., & Vincent, P. 2012, CoRR, abs/1206.5538.  https://arxiv.org/abs/1206.5538
  * Bishop (2006) Bishop, C. M. 2006, Pattern Recognition and Machine Learning (Information Science and Statistics) (Berlin, Heidelberg: Springer-Verlag)
  * Borucki et al. (2010) Borucki, W. J., Koch, D., Basri, G., et al. 2010, Science, 327, 977
  * Borucki et al. (2011) Borucki, W. J., Koch, D. G., Basri, G., et al. 2011, ApJ, 736, 19, doi: 10.1088/0004-637X/736/1/19
  * Brown et al. (2018) Brown, A., Vallenari, A., Prusti, T., et al. 2018, Astronomy & astrophysics, 616, A1
  * Brown et al. (2011) Brown, T. M., Latham, D. W., Everett, M. E., & Esquerdo, G. A. 2011, AJ, 142, 112, doi: 10.1088/0004-6256/142/4/112
  * Bryson et al. (2013) Bryson, S. T., Jenkins, J. M., Gilliland, R. L., et al. 2013, Publications of the Astronomical Society of the Pacific, 125, 889–923, doi: 10.1086/671767
  * Bryson et al. (2015) Bryson, S. T., Abdul-Masih, M., Batalha, N., et al. 2015, KSCI-19093-003
  * Burke et al. (2019) Burke, C. J., Mullally, F., Thompson, S. E., Coughlin, J. L., & Rowe, J. F. 2019, The Astronomical Journal, 157, 143, doi: 10.3847/1538-3881/aafb79
  * Chaushev et al. (2019) Chaushev, A., Raynard, L., Goad, M. R., et al. 2019, Monthly Notices of the Royal Astronomical Society, 488, 5232–5250, doi: 10.1093/mnras/stz2058
  * Chollet (2015) Chollet, F. 2015, keras, https://github.com/fchollet/keras, GitHub
  * Chow (1970) Chow, C. 1970, IEEE Transactions on Information Theory, 16, 41, doi: 10.1109/TIT.1970.1054406
  * Collins et al. (2018) Collins, K., Quinn, S. N., Latham, D. W., et al. 2018, in American Astronomical Society Meeting Abstracts# 231, Vol. 231, 439–08
  * Cortes et al. (2016) Cortes, C., DeSalvo, G., & Mohri, M. 2016.  https://cs.nyu.edu/~mohri/pub/rej.pdf
  * Coughlin (2017) Coughlin, J. L. 2017, KSCI-19114-001
  * Coughlin et al. (2014) Coughlin, J. L., Thompson, S. E., Bryson, S. T., et al. 2014, The Astronomical Journal, 147, 119, doi: 10.1088/0004-6256/147/5/119
  * Coughlin et al. (2016) Coughlin, J. L., Mullally, F., Thompson, S. E., et al. 2016, The Astrophysical Journal Supplement Series, 224, 12, doi: 10.3847/0067-0049/224/1/12
  * Dattilo et al. (2019a) Dattilo, A., Vanderburg, A., Shallue, C. J., et al. 2019a, The Astronomical Journal, 157, 169, doi: 10.3847/1538-3881/ab0e12
  * Dattilo et al. (2019b) —. 2019b, AJ, 157, 169, doi: 10.3847/1538-3881/ab0e12
  * Deleuil et al. (2018) Deleuil, M., Aigrain, S., Moutou, C., et al. 2018, A&A, 619, A97, doi: 10.1051/0004-6361/201731068
  * Dosovitskiy et al. (2014) Dosovitskiy, A., Springenberg, J. T., Riedmiller, M. A., & Brox, T. 2014, CoRR, abs/1406.6909.  https://arxiv.org/abs/1406.6909
  * Eisner et al. (2020) Eisner, N. L., Barragán, O., Aigrain, S., et al. 2020, MNRAS, 494, 750, doi: 10.1093/mnras/staa138
  * Falkner et al. (2018) Falkner, S., Klein, A., & Hutter, F. 2018, CoRR, abs/1807.01774.  https://arxiv.org/abs/1807.01774
  * Girardi et al. (2005) Girardi, L., Groenewegen, M. A. T., Hatziminaoglou, E., & da Costa, L. 2005, A&A, 436, 895, doi: 10.1051/0004-6361:20042352
  * Greydanus et al. (2018) Greydanus, S., Koul, A., Dodge, J., & Fern, A. 2018, Visualizing and Understanding Atari Agents.  https://arxiv.org/abs/1711.00138
  * Guerrero et al. (2021) Guerrero, N. M., Seager, S., Huang, C. X., et al. 2021, ApJS, 254, 39, doi: 10.3847/1538-4365/abefe1
  * Guo et al. (2017) Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. 2017, CoRR, abs/1706.04599.  https://arxiv.org/abs/1706.04599
  * Howell et al. (2014) Howell, S. B., Sobeck, C., Haas, M., et al. 2014, Publications of the Astronomical Society of the Pacific, 126, 398–408, doi: 10.1086/676406
  * Jenkins (2020) Jenkins, J. M. 2020, KSCI-19081-003
  * Jenkins et al. (2010) Jenkins, J. M., Caldwell, D. A., Chandrasekaran, H., et al. 2010, The Astrophysical Journal Letters, 713, L87
  * Jenkins et al. (2010) Jenkins, J. M., Borucki, W. J., Koch, D. G., et al. 2010, ApJ, 724, 1108, doi: 10.1088/0004-637X/724/2/1108
  * Jenkins et al. (2014) Jenkins, J. M., McCauliff, S., Burke, C., et al. 2014, in Formation, Detection, and Characterization of Extrasolar Habitable Planets, ed. N. Haghighipour, Vol. 293, 94–99, doi: 10.1017/S1743921313012611
  * Jenkins et al. (2016) Jenkins, J. M., Twicken, J. D., McCauliff, S., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9913, Software and Cyberinfrastructure for Astronomy IV, ed. G. Chiozzi & J. C. Guzman, 99133E, doi: 10.1117/12.2233418
  * Jontof-Hutter et al. (2021) Jontof-Hutter, D., Wolfgang, A., Ford, E. B., et al. 2021, The Astronomical Journal, 161, 246, doi: 10.3847/1538-3881/abd93f
  * Kingma & Ba (2014) Kingma, D., & Ba, J. 2014, International Conference on Learning Representations
  * Kohonen et al. (2001) Kohonen, T., Schroeder, M. R., & Huang, T. S. 2001, Self-Organizing Maps, 3rd edn. (Berlin, Heidelberg: Springer-Verlag)
  * LeCun et al. (2015) LeCun, Y., Bengio, Y., & Hinton, G. 2015, nature, 521, 436
  * LeCun et al. (1989) LeCun, Y., Boser, B., Denker, J. S., et al. 1989, Neural Comput., 1, 541–551, doi: 10.1162/neco.1989.1.4.541
  * Li et al. (2019) Li, J., Tenenbaum, P., Twicken, J. D., et al. 2019, Publications of the Astronomical Society of the Pacific, 131, 024506, doi: 10.1088/1538-3873/aaf44d
  * Mathur & Huber (2016) Mathur, S., & Huber, D. 2016, Kepler Stellar Properties Catalog Update for Q1-Q17 DR25 Transit Search, Kepler Science Document KSCI-19097-004
  * McCauliff et al. (2015) McCauliff, S. D., Jenkins, J. M., Catanzarite, J., et al. 2015, The Astrophysical Journal, 806, 6, doi: 10.1088/0004-637x/806/1/6
  * Morton (2012) Morton, T. D. 2012, The Astrophysical Journal, 761, 6, doi: 10.1088/0004-637x/761/1/6
  * Morton et al. (2016) Morton, T. D., Bryson, S. T., Coughlin, J. L., et al. 2016, The Astrophysical Journal, 822, 86, doi: 10.3847/0004-637x/822/2/86
  * Morton & Johnson (2011) Morton, T. D., & Johnson, J. A. 2011, The Astrophysical Journal, 738, 170, doi: 10.1088/0004-637x/738/2/170
  * Nadeem et al. (2010) Nadeem, M., Zucker, J.-d., & Hanczar, B. 2010, Journal of Machine Learning Research - Proceedings Track, 8, 65.  http://proceedings.mlr.press/v8/nadeem10a/nadeem10a.pdf
  * Ng (2016) Ng, A. Y. 2016, in Advances in Neural Information Processing Systems (MIT Press)
  * Ng & Jordan (2002) Ng, A. Y., & Jordan, M. I. 2002, in Advances in Neural Information Processing Systems 14, ed. T. G. Dietterich, S. Becker, & Z. Ghahramani (MIT Press), 841–848
  * Niculescu-Mizil & Caruana (2005a) Niculescu-Mizil, A., & Caruana, R. 2005a, in Proceedings of the 22nd International Conference on Machine Learning, ICML ’05 (New York, NY, USA: Association for Computing Machinery), 625–632, doi: 10.1145/1102351.1102430
  * Niculescu-Mizil & Caruana (2005b) Niculescu-Mizil, A., & Caruana, R. 2005b, in Proceedings of the 22nd International Conference on Machine Learning, ICML ’05 (New York, NY, USA: Association for Computing Machinery), 625–632, doi: 10.1145/1102351.1102430
  * Osborn et al. (2020) Osborn, H. P., Ansdell, M., Ioannou, Y., et al. 2020, Astronomy & Astrophysics, 633, A53, doi: 10.1051/0004-6361/201935345
  * Platt (1999) Platt, J. C. 1999, in ADVANCES IN LARGE MARGIN CLASSIFIERS (MIT Press), 61–74
  * Ricker et al. (2015) Ricker, G. R., Winn, J. N., Vanderspek, R., et al. 2015, Journal of Astronomical Telescopes, Instruments, and Systems, 1, 014003, doi: 10.1117/1.JATIS.1.1.014003
  * Rolnick et al. (2017) Rolnick, D., Veit, A., Belongie, S. J., & Shavit, N. 2017, ICLR, abs/1705.10694.  https://arxiv.org/pdf/1705.10694.pdf
  * Sahu et al. (2006) Sahu, K. C., Casertano, S., Bond, H. E., et al. 2006, Nature, 443, 534–540, doi: 10.1038/nature05158
  * Santerne et al. (2016) Santerne, A., Moutou, C., Tsantaki, M., et al. 2016, A&A, 587, A64, doi: 10.1051/0004-6361/201527329
  * Schwamb et al. (2013) Schwamb, M. E., Orosz, J. A., Carter, J. A., et al. 2013, ApJ, 768, 127, doi: 10.1088/0004-637X/768/2/127
  * Seader et al. (2015) Seader, S., Jenkins, J. M., Tenenbaum, P., et al. 2015, The Astrophysical Journal Supplement Series, 217, 18, doi: 10.1088/0067-0049/217/1/18
  * Selvaraju et al. (2019) Selvaraju, R. R., Cogswell, M., Das, A., et al. 2019, International Journal of Computer Vision, 128, 336–359, doi: 10.1007/s11263-019-01228-7
  * Shallue & Vanderburg (2018) Shallue, C. J., & Vanderburg, A. 2018, The Astronomical Journal, 155, 94
  * Thompson et al. (2016a) Thompson, S. E., Fraquelli, D., van Cleve, J. E., & Caldwell, D. A. 2016a, KDMC-10008-006
  * Thompson et al. (2016b) Thompson, S. E., Caldwell, D. A., Jenkins, J. M., et al. 2016b, KSCI-19065-002
  * Thompson et al. (2018) Thompson, S. E., Coughlin, J. L., Hoffman, K., et al. 2018, The Astrophysical Journal Supplement Series, 235, 38, doi: 10.3847/1538-4365/aab4f9
  * Torres et al. (2004) Torres, G., Konacki, M., Sasselov, D., & Jha, S. 2004, The Astrophysical Journal, 614, doi: 10.1086/423734
  * Twicken et al. (2016) Twicken, J. D., Jenkins, J. M., Seader, S. E., et al. 2016, The Astronomical Journal, 152, 158, doi: 10.3847/0004-6256/152/6/158
  * Twicken et al. (2018) Twicken, J. D., Catanzarite, J. H., Clarke, B. D., et al. 2018, Publications of the Astronomical Society of the Pacific, 130, 064502, doi: 10.1088/1538-3873/aab694
  * Twicken et al. (2020) Twicken, J. D., Caldwell, D. A., Jenkins, J. M., et al. 2020, TESS Science Data Products Description Document EXP-TESS-ARC-ICD-0014 Rev F, NASA/TM 20205008729, NASA Ames Research Center.  https://archive.stsci.edu/missions/tess/doc/EXP-TESS-ARC-ICD-TM-0014-Rev-F.pdf
  * Vapnik (1998) Vapnik, V. N. 1998, Statistical Learning Theory (Wiley-Interscience).  https://www.wiley.com/en-us/Statistical+Learning+Theory-p-9780471030034
  * Yu et al. (2019a) Yu, L., Vanderburg, A., Huang, C., et al. 2019a, The Astronomical Journal, 158, 25, doi: 10.3847/1538-3881/ab21d6
  * Yu et al. (2019b) —. 2019b, The Astronomical Journal, 158, 25, doi: 10.3847/1538-3881/ab21d6
  * Zeiler & Fergus (2014) Zeiler, M. D., & Fergus, R. 2014, in Computer Vision – ECCV 2014, ed. D. Fleet, T. Pajdla, B. Schiele, & T. Tuytelaars (Cham: Springer International Publishing), 818–833