TransposedTabularDB Ttabular[1]

# Improving Robustness for Vision Transformer with a Simple Dynamic Scanning
Augmentation

Shashank Kotyan Danilo Vasconcellos Vargas Kyushu University
<EMAIL_ADDRESS>
###### Abstract

Vision Transformer (ViT) has demonstrated promising performance in computer
vision tasks, comparable to state-of-the-art neural networks. Yet, this new
type of deep neural network architecture is vulnerable to adversarial attacks
limiting its capabilities in terms of robustness. This article presents a
novel contribution aimed at further improving the accuracy and robustness of
ViT, particularly in the face of adversarial attacks. We propose an
augmentation technique called ‘Dynamic Scanning Augmentation’ that leverages
dynamic input sequences to adaptively focus on different patches, thereby
maintaining performance and robustness. Our detailed investigations reveal
that this adaptability to the input sequence induces significant changes in
the attention mechanism of ViT, even for the same image. We introduce four
variations of Dynamic Scanning Augmentation, outperforming ViT in terms of
both robustness to adversarial attacks and accuracy against natural images,
with one variant showing comparable results. By integrating our augmentation
technique, we observe a substantial increase in ViT’s robustness, improving it
from $17\%$ to $92\%$ measured across different types of adversarial attacks.
These findings, together with other comprehensive tests, indicate that Dynamic
Scanning Augmentation enhances accuracy and robustness by promoting a more
adaptive type of attention. In conclusion, this work contributes to the
ongoing research on Vision Transformers by introducing Dynamic Scanning
Augmentation as a technique for improving the accuracy and robustness of ViT.
The observed results highlight the potential of this approach in advancing
computer vision tasks and merit further exploration in future studies.

##  1 Introduction

Transformers Vaswani et al. (2017), a type of neural network based on self-
attention, has become the standard for many Text-based NLP tasks. Recently,
many neural network architectures based on transformers and self-attention
have been proposed for vision tasks that outperform other architectures.
Dosovitskiy et al. (2020); Ramachandran et al. (2019); Wu et al. (2020); Liu
et al. (2021); Tang & Ha (2021).

Although Vision Transformer (ViT) seems promising for the vision tasks in
terms of accuracy, it suffers from a few underlying problems like,

a) Robustness against adversarial attacks Shao et al. (2022); Mahmood et al.
(2021), and

b) Extraction of local features Han et al. (2021); Yuan et al. (2021).

Therefore, the potential of self-attention and transformers is yet to be fully
utilized, especially regarding solving vision problems. Until now,
augmentations have focused on modifying the image pixels based on
transformations since the whole image is processed at once. However, the
definition of augmentation can be extended to input sequences and how these
input sequences are created for the transformers. Transformers process images
as a sequence of patches, allowing for a new kind of augmentation based on the
dynamic scanning of images.

We propose a novel kind of augmentation, dynamic scanning augmentation relying
on non-systematic scanning of images to alleviate the limitations of ViT.
Specifically, we define four algorithms named Random Patches (RP), Random
Tracing (RT), Salient Patches (SP), and Salient Tracing (ST). Our
augmentation, as illustrated in Figure 1, scans the image and extract patches
either,

(a) from random scanning of image (low-biased scanning), i.e., Random Patches
(RP) or Random Tracing (RT), or

(b) derived with a saliency-guided mechanism (high-biased scanning), i.e.,
Salient Patches (SP) or Salient Tracing (ST).

Additionally, it is worth noting that our augmentation approach primarily
emphasizes scanning the image. As a result, it seamlessly integrates with any
existing image transformations or augmentations that may be employed before
implementing our dynamic scanning technique. This flexibility allows for
incorporating other preprocessing steps or enhancements before applying our
dynamic scanning algorithm.

Figure 1:  Overview of the different variations of proposed dynamic scanning
augmentation and their performance on the CIFAR-10 dataset. We report the
accuracy over non-attacked images and mean robust accuracy over three
different black-box adversarial attacks. All of the transformers employed with
dynamic scanning augmentation substantially improve the performance and
robustness of ViT.

Our Contributions:

$\bullet~{}$Novel augmentation based on dynamic input sequences:

Our dynamic scanning augmentation relies on non-systematic scanning of images,
creating dynamic input sequences for the transformers. Thus, when the
transformer processes a single image multiple times, it sees the image
differently due to the dynamic input sequence of patches. Our augmentation
allows us to shuffle the sequence of patches fed to the transformers. It also
allows the patches to be different between multiple scans of the same image.
This enables the transformers to focus on different parts of the image between
multiple scans, essentially learning more about the image and improving the
accuracy of the transformer.

$\bullet~{}$Adaptive attention:

Our investigations reveal that transformers employed with dynamic scanning
augmentation attend to different patches/regions depending on the input
sequence provided to the transformer. This can be seen as the adaptability of
attention to dynamic input sequences, suggesting that transformers employed
with dynamic scanning augmentation can attend to different regions in an image
to induce correct classification.

$\bullet~{}$Steep robustness against adversarial attacks:

We discover that by merely employing dynamic scanning augmentation in
transformers, we can significantly boost robustness (around $75\%$) against
various adversarial attacks. Moreover, this boost comes without degrading the
performance of transformers. Tests reveal that the crucial reason for improved
robustness in transformers is a more adaptive kind of attention enabled by
dynamic scanning augmentation.

##  2 Related Works

Transformers were first proposed in Vaswani et al. (2017) and showed
significant improvement in machine translation performance. Since the success
of transformers and self-attention modules in NLP, researchers have tried to
use the transformers in various vision tasks, like image classification
Dosovitskiy et al. (2020); Chen et al. (2020b), object-detection Carion et al.
(2020); Heng et al. (2020); Dai et al. (2021); Zhu et al. (2021); Sun et al.
(2021), segmentation Chen et al. (2021); Wang et al. (2021c, a); Zheng et al.
(2021), image enhancement Chen et al. (2021); Yang et al. (2020), image
generation Parmar et al. (2018), human behavior recognition Yang et al.
(2023), video in-painting Zeng et al. (2020), and video captioning Zhou et al.
(2018).

ViT, based on pure transformer architecture, achieved comparable performance
to the state-of-the-art on multiple image classification benchmarks
Dosovitskiy et al. (2020). ViT scans the images into a series of non-
overlapping patches systematically. This scanning of images makes learning
local features hard since every patch and pixel is processed only once in one
context.

However, the performance is still inferior to CNNs when trained from scratch
for small and mid-sized datasets. This drop in performance is attributed by
Dosovitskiy et al. (2020) to the lack of inductive biases, such as locality,
which are inherent to CNNs. However, studies from Lu et al. (2016, 2019);
Kannan et al. (2014) shows that the patch-matching algorithm can also be
invariant to rotation solving a particular inductive bias. This suggests that
performance in transformers can be further improved with better feature
matchings.

Research from Han et al. (2021); Yuan et al. (2021) suggests that ViT
inherently learns a poor representation of local features due to this lack of
inductive biases. Many approaches improve ViT by focusing on local features to
extract and learn more about the image patches. One example is Transformer-in-
Transformer (TNT) Han et al. (2021) architecture, in which an extra
transformer module is used to process pixel-level embeddings. The authors
suggest that ViT models the global relation among patch embeddings, while
pixel-level embeddings are required to extract local structures. Another
example is Token-to-Token ViT (T2T-ViT) Yuan et al. (2021), where it is
suggested that the complex splits and direct tokenizations hinder the learning
of local features such as edges. Research by Zhao et al. (2023) observes that
better generalization ability arises from the various distinct features
extracted from the input. Another research from Pang et al. (2023) suggests
that transformers have the capability to interweave fusion structures by
rethinking the transformer architecture to include global information about
alignment and transformation.

Interestingly, in both the Pyramid Vision Transformer (PVT) proposed by Wang
et al. (2021b) and the Swin Transformer proposed by Liu et al. (2021), the
researchers suggest that the Vision Transformer can benefit from a
hierarchical (pyramid) structure similar to CNNs to predict finer feature-maps
and perform better at pixel-level predictions. They suggest that ViT’s
columnar structure, which processes patches at a fixed scale, limits the
learning of the transformers. At the same time, in Swin Transformers, the
authors also propose to use the shifted windows approach for finer processing
of patches locally, which also allows learning cross-window relationships
between non-overlapping patches.

From a different perspective, permutation in input sequence was studied in
Attention Neuron proposed by Tang & Ha (2021) for reinforcement learning
tasks. In this approach, it was observed that when transformers are fed with a
permuted input sequence, they exhibit better robustness and generalization.
This article investigates dynamic input sequences created by dynamic scanning
of images from a different perspective. This includes different permutations
of input patches and extends the possibility of including a single patch
either more than once or not including it in the input sequence.

##  3 Overview of dynamic scanning augmentation

Figure 2:  Illustration of how the proposed dynamic scanning augmentation
differs from ViT. The figure also shows how the sequence of patches extracted
differs between different variations of dynamic scanning augmentation.

In the case of ViT, every patch of the image is only processed once
systematically by the transformer, as illustrated in Figure 2. Moreover, as
there are no overlapping pixels and patches, we can infer that transformers
only look at the patches and pixels once, limiting the information learned
about a patch. At the same time, the input sequence of patches also remains
consistent, i.e., there is no shuffling of patches within the input sequence.

However, transformers are most efficient for NLP Tasks, where every word
generally appears more than once in different contexts, making the
transformers learn more about the word and the contexts as a whole.
Theoretically, each pixel can also be viewed with eight different contexts
based on preceding and succeeding neighboring pixels. However, viewing every
pixel in every context is computationally heavy and often unnecessary.

Therefore, we propose dynamic scanning augmentation to learn more about the
image. This approach extracts the patches dynamically from the image, enabling
the transformers to perceive a single image differently when presented
multiple times. Moreover, there is no guarantee that a particular patch will
be fed to the transformers. Transformers must adapt to classify the image
based on the given input sequence of the patches.

Below, we give a brief description of the different variations of dynamic
scanning augmentation and also note the bias included in scanning using these
variants,

Figure 3:  Here, we show an example of regions scanned by various dynamic
scanning augmentation variants on an image. We also show the saliency map of
the original image, which is used as a guide for salient patches and salient
tracing. Further, we plot the frequency of scanned patches. This map is
homogeneous for ViT since every patch is extracted and processed only once.
However, for dynamic scanning augmentation, both possibilities exist: either a
patch is extracted multiple times in a single scan or not extracted.
Therefore, the darker color represents no extraction of patches, while the
lighter color represents patches extracted more than once. We also overlay
this frequency map onto the original image to visualize the regions of the
image extracted by our augmentation.  Algorithm 1 Algorithm for extraction of
patches using Random Patches

1:

2:image: The input image

3:num_patches: The number of patches to extract

4:patch_size: The size of each patch

5:

6:patches: The extracted patches

7:procedure RandomPatches

8: patches $\leftarrow$ empty list

9: for $i\leftarrow 1$ to num_patches do

10: coordinate $\leftarrow$ randomly selects a pixel location in the image

11: patch $\leftarrow$ crop the image using coordinate and patch_size

12: add patch to patches

13: end for

14: sequence $\leftarrow$ concatenate all patches in patches

15: return sequence

16:end procedure

Time Complexity: $O(\text{num\\_patches})$

Algorithm 2 Algorithm for extraction of patches using Random Tracing

1:

2:image: The input image

3:num_patches: The number of patches to extract

4:patch_size: The size of each patch

5:

6:patches: The extracted patches

7:procedure RandomTracing

8: patches $\leftarrow$ empty list

9: while length of patches $<$ num_patches do

10: selected_coordinates $\leftarrow$ randomly select two-pixel locations from
the image

11: ray_direction $\leftarrow$ direction from selected_coordinates[1] to
selected_coordinates[2]

12: current_coordinate $\leftarrow$ selected_coordinates[1]

13: patch $\leftarrow$ crop the image using current_coordinate and patch_size

14: add patch to patches

15: while current_coordinate $\neq$ selected_coordinate[2] do

16: next_coordinate $\leftarrow$ find next pixel location along ray_direction

17: current_coordinate $\leftarrow$ next_coordinate

18: patch $\leftarrow$ crop the image using current_coordinate and patch_size

19: add patch to patches

20: end while

21: end while

22: sequence $\leftarrow$ concatenate all patches in patches

23: return sequence

24:end procedure

Time Complexity: $O(\text{num\\_patches}\cdot\text{path\\_length})$, where
path_length represents the average length of the traced paths.

Algorithm 3 Algorithm for extraction of patches using Saliency Patches

1:

2:image: The input image

3:saliency_map: The saliency map of the image

4:num_patches: The number of patches to extract

5:patch_size: The size of each patch

6:

7:patches: The extracted patches

8:procedure SalientPatches

9: patches $\leftarrow$ empty list

10: pixel_order $\leftarrow$ sort pixel locations based on saliency_map in
descending order

11: for $i\leftarrow 1$ to num_patches do

12: patch $\leftarrow$ crop the image using pixel_order[i] and patch_size

13: add patch to patches

14: end for

15: sequence $\leftarrow$ concatenate all patches in patches

16: return sequence

17:end procedure

Time Complexity: $O(\text{num\\_patches})$

Algorithm 4 Algorithm for extraction of patches using Salient Tracing

1:

2:image: The input image

3:saliency_map: The saliency map of the image

4:num_patches: The number of patches to extract

5:patch_size: The size of each patch

6:

7:patches: The extracted patches

8:procedure SalientTracing

9: patches $\leftarrow$ empty list

10: pixel_order $\leftarrow$ sort pixel locations based on saliency_map in
descending order

11: count $\leftarrow 1$

12: while length of patches $<$ num_patches do

13: selected_coordinates $\leftarrow$ [pixel_order[count],
pixel_order[count+1]]

14: ray_direction $\leftarrow$ direction from selected_coordinates[1] to
selected_coordinates[2]

15: current_coordinate $\leftarrow$ selected_coordinates[1]

16: patch $\leftarrow$ crop the image using current_coordinate and patch_size

17: add patch to patches

18: while current_coordinate $\neq$ selected_coordinate[2] do

19: next_coordinate $\leftarrow$ find next pixel location along ray_direction

20: current_coordinate $\leftarrow$ next_coordinate

21: patch $\leftarrow$ crop the image using current_coordinate and patch_size

22: add patch to patches

23: end while

24: count $\leftarrow$ count $+1$

25: end while

26: sequence $\leftarrow$ concatenate all patches in patches

27: return sequence

28:end procedure

Time Complexity: $O(\text{num\\_patches}\cdot\text{path\\_length})$, where
path_length represents the average length of the traced paths.

Random Patches (RP) (Least Biased):

In this variant, we extract the patches randomly (uniform distribution) from
all over an image and feed them to transformers in a sequence as defined in
Algorithm 1. Since all the patches from an image are extracted randomly, we
include minimal bias in scanning the image.

Random Tracing (RT) (Low Biased):

In this variant, we choose two random patches in an image and trace all the
patches in the imaginary ray connecting these two chosen patches. We do this
by tracing through rays repeatedly till we have enough patches for the
transformers as defined in Algorithm 2. We include the bias of the sub-
sequence of patches linked to each other since we trace the patches between
two random patches.

Salient Patches (SP) (High Biased):

In this variant, we first start by creating a visual guidance map of the image
using a saliency map as proposed by Montabone & Soto (2010). We then feed the
patches to the transformer in order of the saliency of patches as defined in
Algorithm 3. Further, the saliency map by Montabone & Soto (2010) can be
referred to as an algorithm to separate the background from the foreground
without using any deep learning and relying solely on image processing. Hence,
this extraction of patches using a guide (saliency map) adds a built-in bias
in the scanning where the patches extracted depend on the saliency computed by
the algorithm.

Salient Tracing (ST) (High Biased):

In this variant, similar to Salient Patches, we first compute the saliency map
of the image as proposed by Montabone & Soto (2010) to generate our visual-
guidance map. Then, similar to Random Tracing, we then trace all the patches
that lie in the imaginary rays connecting in-between the salient patches
tracked in order of their saliency. We do this tracing repeatedly till we have
enough patches for the transformers as defined in Algorithm 4. Like Salient
Patches and Random Tracing, this extraction of patches using a guide also adds
an inherent bias to the scanning. It adds extra bias where the sub-sequence of
patches are linked to each other.

Figure 3 illustrates the different variations of dynamic scanning
augmentation. We show the scanned patches and plot the frequency map to
highlight where a patch is seen multiple times in a scan. The figure shows
that scanning for Random Patches results in extracting patches that are not
dependent on any factor within the image itself. However, as mentioned above,
other variations of dynamic scanning augmentation include some bias to extract
some patches.

Our dynamic scanning augmentation has the following characteristics;

(a) Non-Systematic scanning of images,

(b) Stochastic scanning of images,

(c) Possibility to re-look at the patches,

(d) Hold back some of the available visual information, and

(e) Acquiring information in different contexts.

We believe that because of these properties, it is possible to learn more
aggressively about robust features, which is verified by the experiments.

As we use non-systematic scanning, we modify the positional embedding used by
ViT. For ViT, the patches’ position refers to the position of the patch in the
sequence. Since ViT systematically scans the image, the position of the patch
in the sequence also corresponds to the position of the patch in the image,
and there exists a fixed number of patches for the transformer.

However, we use the index of the patch’s center pixel as the patch’s position
for our augmentations, as the position of the patch in the sequence is not
related to the position of the patch in the image. Also, the number of patches
fed to the dynamic scanning augmentation transformer is controlled with a
parameter. Therefore, if a patch center lies in $(x,y)$ pixel, we convert it
to one-dimensional using row-major form as $p=(x\times n)+y$, where $n$ is the
image size. Further, to accommodate the position of the [class] token, we
shift the raw position of the patches by $1$. It is to be noted that this
position encoding was also evaluated by Dosovitskiy et al. (2020) and showed
no change in performance.

In the realm of deep learning and natural language processing, the transformer
architecture has a complexity of $O(N^{2})$ where $N$ represents the sequence
input to the transformer Vaswani et al. (2017). ViT’s systematic scanning
utilizes a fixed value for $N$, often computed as $N=(\text{Image
Size}/\text{Patch Size})^{2}$ Dosovitskiy et al. (2020). In contrast, our
proposed dynamic scanning augmentation allows flexibility to the user for
choosing $N$ to adapt the algorithm’s computational resources to the specific
requirements of their tasks. Thus, converting $N$ as a hyperparameter, subject
to variation.

##  4 Experimental Results

###  4.1 Experimental Design and Settings

Table 1: Description of the datasets evaluated in our article. We evaluate a small-image size dataset, CIFAR-10, and one big image-size dataset, Imagenette. Imagenette is a subset of the bigger ImageNet dataset consisting of only $10$ classes of the original ImageNet dataset.  Dataset | CIFAR-10 | Imagenette
---|---|---
Image Size | $32\times 32\times 3$ | $220\times 220\times 3$
# Classes | 10 | 10
Training Data | 50,000 | 9,469
Testing Data | 10,000 | 3,925

Datasets: We experiment with a small-image size dataset CIFAR-10 Krizhevsky et
al. (2009), and one big-image size dataset Imagenette Howard  (A subset of
ILSVRC 2012 ImageNet Russakovsky et al. (2015) with $10$ classes), to evaluate
the performance and embedding of the transformers. Table 1 gives an overview
of the datasets

We use Auto-augment Cubuk et al. (2018) preprocessing to augment the images
before training. Auto-augment modifies the image with several transformations
depending on the dataset. Note that our dynamic scanning augmentation can be
complemented with other image manipulation-based augmentations, like auto-
augment and preprocessing-based adversarial defenses, to boost performance
further.

Transformer Variant Used: We use the ‘Base’ variant of the transformers, with
$12$ number of transformer layers and self-attention heads, latent vector size
$768$, and MLP size $3072$, similar to ViT Dosovitskiy et al. (2020). We train
all the models with AdamW optimizer with default hyper-parameters, with an
initial learning rate of $0.00008$. We also use a linear decay of learning
rate over $100$ epochs with a warmup of $2$ epochs. We use the HuggingFace
library Wolf et al. (2020) to implement the transformers.

Adversarial Attacks: To test the robustness of various transformers, we test
the transformers on different black-box adversarial attacks,

(a) Pixel Attack, an $L_{0}$ norm black-box attack Su et al. (2019),

(b) Threshold Attack, an $L_{\infty}$ norm black-box attack Kotyan & Vargas
(2022), and

(c) HopSkipJump Attack, a query efficient decision-based black box attack with
$L_{2}$ and $L_{\infty}$ norms variants Chen et al. (2020a).

We implement the attacks using the Adversarial Robustness Toolbox (ART)
Nicolae et al. (2018) library. We use the default hyper-parameters for the
attacks and set the threshold $th$ to $10$ for both Pixel and Threshold
attacks.

Computation Power Used: All the experiments were conducted on a single system
with Intel® Core™ i9-10900K CPU @ 3.70GHz with 10 cores, 62.7GiB of RAM and a
single NVIDIA® GeForce® RTX 3090.

Amount of Image Covered with Patch Size of 3

$\lceil~{}{ImageSize}/{PatchSize}~{}\rceil^{2}$ = 121

Amount of Image Covered with Patch Size of 9
$\lceil~{}{ImageSize}/{PatchSize}~{}\rceil^{2}$ = 16

Figure 4:  Plot of the amount of the image covered by $N$ patches using
different dynamic scanning augmentation for CIFAR-10.

Amount of Image Covered with Patch Size of 9

$\lceil~{}{ImageSize}/{PatchSize}~{}\rceil^{2}$ = 625

Amount of Image Covered with Patch Size of 15
$\lceil~{}{ImageSize}/{PatchSize}~{}\rceil^{2}$ = 225

Figure 5:  Plot of the amount of the image covered by $N$ patches using
different dynamic scanning augmentation for Imagenette.

Ablation Tests (Amount of Visual Information): An ablation study is conducted
to understand the amount of distinct pixels extracted from the image covered
by the different number of patches as shown in Figures 4.1 and 4.1. Ablation
study suggests that dynamic scanning augmentation extract around $55\%-60\%$
of the image on average when using the same number of patches as ViT, that is,
$\lceil~{}\text{Image Size}/\text{Patch Size}~{}\rceil^{2}$. The ablation
study also shows that the image covered by scans is highly variable in the
Salient Patches and Salient Tracing compared to the random counterparts. This
can be accounted for by the bias included in the scanning using a saliency
map. We also notice that the amount of distinct pixels for salient patches
remains lower for the Imagenette dataset, suggesting that the objects usually
do not cover the entire image, unlike CIFAR-10. This is expected since, in
Imagenette, the object is usually centered and usually does not occupy the
entire image. Further, as dynamic scanning augmentation is non-systematic and
stochastic, we report a mean of results over $5$ in different test runs for
all the experiments.

###  4.2 Performance of dynamic scanning augmentation on small-size dataset

Table 2:  Accuracy of transformers trained with the different number of
patches for the CIFAR-10 dataset. Bold results correspond to the model having
better accuracy than ViT (Table 2) for the experiment. (*) corresponds to the
number of patches, which are the same as ViT.

ViT | # of Patches | Random Patches | Random Tracing | Salient Patches | Salient Tracing
---|---|---|---|---|---
Patch Size = 3
| 25 | 70.73% $\pm$ 00.18% | 57.22% $\pm$ 00.36% | 64.38% $\pm$ 00.15% | 54.29% $\pm$ 00.12%
| 50 | 79.74% $\pm$ 00.12% | 67.96% $\pm$ 00.21% | 75.59% $\pm$ 00.12% | 65.67% $\pm$ 00.17%
| 100 | 85.08% $\pm$ 00.15% | 78.93% $\pm$ 00.27% | 79.59% $\pm$ 00.04% | 76.01% $\pm$ 00.18%
84.15% | *121 | 85.23% $\pm$ 00.24% | 82.99% $\pm$ 00.13% | 80.91% $\pm$ 00.06% | 80.01% $\pm$ 00.09%
| 150 | 85.88% $\pm$ 00.19% | 84.14% $\pm$ 00.09% | 80.58% $\pm$ 00.11% | 82.09% $\pm$ 00.11%
| 250 | 87.44% $\pm$ 00.24% | 86.56% $\pm$ 00.18% | 81.77% $\pm$ 00.14% | 85.72% $\pm$ 00.07%
| 400 | 88.63% $\pm$ 00.21% | 88.01% $\pm$ 00.12% | 81.71% $\pm$ 00.08% | 87.72% $\pm$ 00.11%
| 600 | 88.91% $\pm$ 00.09% | 88.32% $\pm$ 00.16% | 83.35% $\pm$ 00.09% | 87.87% $\pm$ 00.19%
| 900 | 89.05% $\pm$ 00.09% | 88.85% $\pm$ 00.10% | 83.37% $\pm$ 00.06% | 89.06% $\pm$ 00.04%
Patch Size = 9
73.90% | *16 | 78.17% $\pm$ 00.22% | 61.88% $\pm$ 00.33% | 73.34% $\pm$ 00.05% | 57.42% $\pm$ 00.35%
| 25 | 81.91% $\pm$ 00.15% | 66.03% $\pm$ 00.39% | 74.48% $\pm$ 00.05% | 64.90% $\pm$ 00.08%
| 50 | 83.95% $\pm$ 00.20% | 78.52% $\pm$ 00.17% | 76.49% $\pm$ 00.14% | 75.23% $\pm$ 00.12%
| 100 | 85.53% $\pm$ 00.21% | 82.08% $\pm$ 00.26% | 77.95% $\pm$ 00.07% | 82.05% $\pm$ 00.14%
| 150 | 85.31% $\pm$ 00.21% | 84.26% $\pm$ 00.25% | 78.58% $\pm$ 00.09% | 83.66% $\pm$ 00.10%
| 250 | 84.62% $\pm$ 00.22% | 84.42% $\pm$ 00.18% | 80.09% $\pm$ 00.08% | 84.83% $\pm$ 00.13%
| 400 | 86.10% $\pm$ 00.18% | 84.73% $\pm$ 00.23% | 79.60% $\pm$ 00.08% | 84.68% $\pm$ 00.10%
| 600 | 85.25% $\pm$ 00.09% | 83.90% $\pm$ 00.16% | 80.39% $\pm$ 00.08% | 84.73% $\pm$ 00.06%
| 900 | 84.98% $\pm$ 00.21% | 85.39% $\pm$ 00.08% | 81.07% $\pm$ 00.06% | 85.31% $\pm$ 00.08%

To analyze the performance of ViT and our dynamic scanning augmentation for
the small-sized dataset, we first report accuracy over the test dataset of
CIFAR-10, where the transformers were trained on the training dataset (Table
2). To the best of our knowledge, this is the first time CIFAR-10’s result
will be reported for the ViT-Base model trained on CIFAR-10 from scratch.
Results show that the performance of dynamic scanning augmentation using
Random Patches is superior to ViT and other variants of augmentation when the
number of patches of the image is the same as ViT.

The experimental results show that the variations of dynamic scanning
augmentation relying on saliency maps (Salient Patches and Salient Tracing)
perform poorly compared to the random variants. We suspect this happens due to
the bias in the extraction of patches, as the salient variants focus only on
the most salient parts (presumed foreground) of images. We also notice that
variations of dynamic augmentation relying on tracing (Random Tracing and
Salient Tracing) performed poorly for bigger patch sizes (for the same number
of patches as ViT), suggesting that bias introduced by tracing where the sub-
sequence of image patches depends on each other is not scalable to bigger
image sizes.

Further, to verify that visual information by providing distinct pixels in the
input sequence has an impact on performance, we experiment with transformers
trained with a different number of patches (Table 2). This ensures that we
feed the transformers either with more or less distinct patches, manipulating
the image’s visual information. We observe an increase in accuracy as the
number of patches increases for all the variations of dynamic scanning
augmentation, suggesting that more visual information benefits the
transformers. Here also, the results show the superiority of Random Patches
over other variations of dynamic scanning augmentation. We also notice an
increase in patch size for salient patches, and salient tracing results in
superior performance over smaller patch sizes compared to ViT, suggesting that
our dynamic scanning augmentation is more suitable for bigger patch sizes.

However, the performance increases substantially for dynamic scanning
augmentation relying on tracing when the number of patches increases,
suggesting that different context information is necessary to improve the
performance on the CIFAR-10 dataset where there are limited pixels.

###  4.3 Performance of dynamic scanning augmentation on large-size dataset

Table 3:  Accuracy of transformers trained with the different number of
patches for the Imagenette dataset. Bold results correspond to the model
having better accuracy than ViT (Table 3). (*) corresponds to the number of
patches that are the same as ViT.

ViT | # of Patches | Random Patches | Random Tracing | Salient Patches | Salient Tracing
---|---|---|---|---|---
Patch Size = 9
| 25 | 58.14% $\pm$ 00.14% | 27.56% $\pm$ 00.47% | 41.36% $\pm$ 00.22% | 20.58% $\pm$ 00.42%
| 50 | 66.36% $\pm$ 00.26% | 32.54% $\pm$ 00.37% | 47.85% $\pm$ 00.13% | 26.74% $\pm$ 00.23%
| 100 | 71.32% $\pm$ 00.20% | 44.93% $\pm$ 00.55% | 54.69% $\pm$ 00.13% | 40.32% $\pm$ 00.32%
| 150 | 72.39% $\pm$ 00.34% | 54.59% $\pm$ 00.69% | 59.59% $\pm$ 00.17% | 43.84% $\pm$ 00.34%
| 250 | 73.34% $\pm$ 00.63% | 63.72% $\pm$ 00.58% | 68.74% $\pm$ 00.15% | 48.66% $\pm$ 00.36%
| 400 | 74.94% $\pm$ 00.29% | 70.56% $\pm$ 00.20% | 74.98% $\pm$ 00.26% | 50.48% $\pm$ 00.15%
| 600 | 77.10% $\pm$ 00.29% | 74.06% $\pm$ 00.52% | 76.36% $\pm$ 00.31% | 53.68% $\pm$ 00.41%
75.97% | *625 | 77.40% $\pm$ 00.27% | 73.90% $\pm$ 00.47% | 75.73% $\pm$ 00.24% | 51.58% $\pm$ 00.36%
| 900 | 77.37% $\pm$ 00.28% | 77.15% $\pm$ 00.49% | 75.90% $\pm$ 00.19% | 56.38% $\pm$ 00.31%
Patch Size = 15
| 25 | 62.74% $\pm$ 00.39% | 29.21% $\pm$ 00.69% | 45.66% $\pm$ 00.12% | 17.91% $\pm$ 00.56%
| 50 | 68.98% $\pm$ 00.21% | 35.09% $\pm$ 00.52% | 52.48% $\pm$ 00.08% | 29.24% $\pm$ 00.29%
| 100 | 73.00% $\pm$ 00.61% | 49.68% $\pm$ 00.40% | 62.67% $\pm$ 00.22% | 43.08% $\pm$ 00.15%
| 150 | 75.07% $\pm$ 00.32% | 55.72% $\pm$ 00.54% | 68.34% $\pm$ 00.14% | 46.37% $\pm$ 00.25%
71.87% | *225 | 74.98% $\pm$ 00.23% | 63.44% $\pm$ 00.55% | 70.35% $\pm$ 00.17% | 49.93% $\pm$ 00.35%
| 250 | 73.71% $\pm$ 00.24% | 64.50% $\pm$ 00.41% | 70.61% $\pm$ 00.27% | 50.86% $\pm$ 00.53%
| 400 | 75.25% $\pm$ 00.37% | 71.07% $\pm$ 00.61% | 72.62% $\pm$ 00.28% | 53.14% $\pm$ 00.23%
| 600 | 77.54% $\pm$ 00.38% | 75.66% $\pm$ 00.22% | 73.62% $\pm$ 00.28% | 58.08% $\pm$ 00.29%
| 900 | 78.21% $\pm$ 00.22% | 75.33% $\pm$ 00.46% | 74.05% $\pm$ 00.07% | 59.67% $\pm$ 00.33%

To analyze the performance of ViT and our dynamic scanning augmentation for
the large-size dataset, we first report accuracy over the test dataset of
Imagenette, where the transformers were trained on the training dataset (Table
3). Results show that the performance of dynamic scanning augmentation using
Random Patches is again superior to ViT and other augmentation counterparts
when the number of patches is the same as ViT.

Interestingly, we observe some steep drop in performance for the salient
tracing variant of dynamic scanning augmentation. Since salient tracing
extracts extra information between the salient patches and good performance
for salient patches, the information available between the salient patches
hinders classification.

Next, we also experiment with transformers trained with the different number
of patches (Table 3). We again observe an increase in accuracy as the number
of patches increases for all the variants of dynamic scanning augmentation.
Here also, the results show the superiority of Random Patches over other
variants of dynamic scanning augmentation.

Results from both small-sized (Table 2) and large-sized datasets (Table 3)
suggest that our dynamic scanning augmentation can benefit from more distinct
visual information fed to the transformers trained. This performance of
dynamic scanning augmentation also suggests that not all information (patches)
present in the image is required for processing. Also, our dynamic scanning
augmentation performance does not degrade for a bigger patch size. At the same
time, it is reported that ViT suffers from performance degradation as the
patch size increases Dosovitskiy et al. (2020). The experiments support our
hypothesis of learning local features as there is no significant drop in
performance on increasing the patch size.

Figure 6:  Regions of images scanned by Salient Patches variant of dynamic
scanning augmentation. We plot, a) Saliency Map, b) Scanned Patches, c)
Frequency Map of Scanned Patches, and d) An Overlay of Frequency Map on the
original image.

###  4.4 Robustness of dynamic scanning augmentation against Black-Box
Adversarial Attacks

Table 4:  Robust Accuracy for different adversarial attacks over random $100$
images for models trained with CIFAR-10 with patch size $3$. We test Pixel
Attack (PA), Threshold Attack (TA), and two variants of Hop Skip Jump Attack
(HSJA). We also report a mean robustness across the four attacks. Value in
parentheses refers to the accuracy of transformers for the random $100$
images. Bold results in each column correspond to the model having the best
robustness against the adversarial attack.

Attacks | ViT | Random | Random | Salient | Salient
---|---|---|---|---|---
Patches | Tracing | Patches | Tracing
# of Patches = 121 (Same as ViT)
PA ($L_{0}$) | 26.74% (86%) | 90.24% (84%) | 83.53% (85%) | 24.42% (87%) | 48.78% (81%)
TA ($L_{\infty}$) | 00.00% (86%) | 92.86% (84%) | 86.05% (85%) | 35.23% (87%) | 56.25% (81%)
HSJA ($L_{\infty}$) | 32.56% (86%) | 90.48% (84%) | 80.00% (85%) | 21.84% (87%) | 48.78% (81%)
HSJA($L_{2}$) | 09.30% (86%) | 84.88% (84%) | 77.64% (85%) | 24.42% (87%) | 51.85% (81%)
Mean | 17.15% (86%) | 89.62% (84%) | 81.80% (85%) | 26.48% (87%) | 51.42% (81%)
# of Patches = 900
PA ($L_{0}$) | 26.74% (86%) | 84.62% (90%) | 86.90% (86%) | 28.41% (88%) | 63.33% (89%)
TA ($L_{\infty}$) | 00.00% (86%) | 96.63% (90%) | 97.70% (86%) | 48.28% (88%) | 79.31% (89%)
HSJA ($L_{\infty}$) | 32.56% (86%) | 92.31% (90%) | 86.05% (86%) | 19.32% (88%) | 61.62% (89%)
HSJA ($L_{2}$) | 09.30% (86%) | 94.44% (90%) | 92.94% (86%) | 18.18% (88%) | 73.03% (89%)
Mean | 17.15% (86%) | 92.00% (90%) | 90.89% (86%) | 32.00% (88%) | 69.32% (89%)

Figure 7:  Patches of different images attended by ViT and variations of
dynamic scanning augmentation. Due to the non-systematic scanning, we show
patches attended using two different extractions for our variations of dynamic
scanning augmentation. This shows that changing the input sequence of patches
for the same image corresponds to a change in attended patches for dynamic
scanning augmentationtransformers.  Table 5:  Robust accuracy of random
variants of dynamic scanning augmentation evaluated with stochastic
characteristic (when multiple scans of the same image are different) and non-
stochastic characteristic (when multiple scans of the same image are alike).
Value in parentheses refers to the accuracy of transformers for the random
$100$ images. Bold results in each column correspond to the model having the
best robustness against the adversarial attack.

Attacks | ViT | Stochastic | Non-Stochastic
---|---|---|---
Random | Random | Random | Random
Patches | Tracing | Patches | Tracing
PA | 26.74% (86%) | 90.24% (84%) | 83.53% (85%) | 82.09% (85%) | 83.86% (82%)
TA | 00.00% (86%) | 92.86% (84%) | 86.05% (85%) | 92.26% (85%) | 94.64% (82%)

To analyze the performance of ViT and our dynamic scanning augmentation
against different black-box adversarial attacks, we report the robustness of
the transformers on the test set of the CIFAR-10 dataset. Specifically, we
test against, (a) Pixel Attack Su et al. (2019), (b) Threshold Attack Kotyan &
Vargas (2022), and (c) Hop Skip Jump Attack Chen et al. (2020a). We attack the
random $100$ images correctly classified by the transformers.

Table 4 shows that evaluated adversarial attacks perform terribly over
transformer trained with random variants of dynamic scanning augmentation and
perform poorly over transformer trained with salient variants of dynamic
scanning augmentation. Concurrently, the same attacks are quite successful
against ViT. While the ViT can only correctly classify around $17\%$ of
attacked images, transformers trained with random patches can correctly
classify around $90\%$ of attacked images. This suggests that our dynamic
scanning augmentation has higher resiliency against black-box norm-based
adversarial attacks.

Further, as salient variants are more biased than random variants, the
experiment suggests that reducing the bias in the scanning of images leads to
higher resiliency against adversarial attacks. Interestingly, despite having
superior performance over non-attacked samples, Salient patches are more
vulnerable to adversarial attacks than Salient Tracing. This suggests that
when models focus on only salient regions of the images, it is easier for the
adversarial attacks to find adversarial perturbations to induce
misclassification.

##  5 Discussion and Analysis

###  5.1 On the Effect of Scanning Bias on Performance of Transformers

From Tables 2-3, we notice a substantial difference in the performance of
salient variants and random variants of dynamic scanning augmentation. On
closer inspection, we noticed that the saliency map generated by the algorithm
proposed by Montabone & Soto (2010) often failed to assign high saliency to
objects of interest in the image. We show three examples in Figure 6, in which
a prominent salient region is identified in the background in the first row.
At the same time, in the second row, we notice that the salient region is
diffused in both objects of interest (dog) and the background. In contrast,
the significant salient region consists of the object of interest (lion) in
the last row. This incorrect variance detection of object-of-interest limits
the performance of the salient variants of the dynamic scanning augmentationas
patches containing information about the object of interest are often not fed
to transformers.

Since the scanning is biased to extract salient patches from the image in the
salient variants, there is often a chance that the information about the
object of interest is insufficient to classify. Moreover, as we increase the
number of patches, we increase the chances of the patches containing
information about the object of interest being fed to the transformer.
Therefore, we notice an increase in performance. In contrast, the random
variants extract patches from all over the image and generally have better
performance.

###  5.2 Investigation of Adaptive Attention

Due to the systematic scanning of images, ViT can focus on the input sequence
of patches in only one way. This creates a one-to-one correspondence between
the input space (images) and the output space (classes). However, our
investigation shows that transformers can learn to focus on different patches
in multiple ways depending on different input sequences of patches by using
dynamic scanning augmentation. This creates a dynamic many-to-one
correspondence between the input space (images) and output space (classes).
Traditional augmentation practices modify images to have similar many-to-one
correspondence.

Our augmentation relies on scanning the image by transformers rather than
modifying the image to find multiple correspondences from input space to
output. Therefore, dynamic scanning augmentation networks have attention to
images that can adapt depending on the input sequences of patches. This
adaptability in attention can be visualized as shown in Figure 7, where for
dynamic scanning augmentation, there exist multiple attended patches for the
same image but different input sequences.

Since our augmentations enable the transformers to have adaptability in
attention, this induces adaptability in classification that naturally
increases the robustness of the network. Since adversarial perturbation
affects a patch or group of patches, the model can still try to focus on other
patches to give the correct classification. This is also supported by our
experiments using adversarial attacks (Table 4). Interestingly, we also found
that variations of dynamic scanning augmentation relying on saliency (biased
scanning) were less rigorous in finding multiple ways to classify, which have
poor robustness than the random variants that explored multiple ways more
rigorously.

In order to effectively evaluate the robustness and adaptability in
classification, we also attack the transformers trained by keeping the scan
for an image fixed for adversarial attacks, which usually processes the images
more than once to optimize perturbations (Table 5). Experimental robustness
reveals that stochastic patch extraction (different input sequences in
multiple scans of the same image) plays a minor role in elevating resiliency
against adversarial attacks. However, the crucial contributor to the
adversarial robustness of dynamic scanning augmentation transformers lies in
the adaptability in the image classification.

##  6 Conclusion

This article proposes various types of dynamic scanning augmentation related
to each other based on dynamic scanning of images and dynamic input sequences
to the transformers. We show that the transformers augmented with dynamic
scanning augmentation are more accurate and robust than the ViT. Transformers
trained with Random Patches outperform the ViT in all datasets tested with
over $5\%$ improvement in standard accuracy and over $75\%$ increase in
robustness across various adversarial attacks. Our investigations also show
that injecting bias into the transformers lead to a degradation of
performance. As Random Patches, the variant with the least bias outperforms
all other variants of dynamic scanning augmentation. In fact, our dynamic
scanning augmentation does not use all available pixels; on average, only
$60\%$ of the image pixels is enough to achieve superior results than ViT.
Moreover, increasing the distinct pixels in the input sequence also boosts the
performance of the transformers trained with our augmentation. A significant
contributing factor to robustness is adaptability in attention due to the
dynamic input sequence of patches. Our experiments reveal that transformers
employed with dynamic scanning augmentation attend to different
patches/regions depending on the input sequence provided to the transformer.
We define this adaptability in attention as the capability to attend to
different patches/regions depending on different images’ scans (input
sequences of patches), which plays a vital role in contributing to resiliency
against adversarial attacks. Thus, we show that non-systematic scanning of
images improves the performance and robustness of standard Vision
Transformers.

## Acknowledgments

This work was supported by JSPS Grant-in-Aid for Challenging Exploratory
Research - Grant Number JP22K19814, JST Strategic Basic Research Promotion
Program (AIP Accelerated Research) - Grant Number JP22584686, JSPS Research on
Academic Transformation Areas (A) - Grant Number JP22H05194.

## References

  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European Conference on Computer Vision_ , pp. 213–229. Springer, 2020.
  * Chen et al. (2021) Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao.  Pre-trained image processing transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12299–12310, 2021.
  * Chen et al. (2020a) Jianbo Chen, Michael I Jordan, and Martin J Wainwright.  Hopskipjumpattack: A query-efficient decision-based attack.  In _2020 ieee symposium on security and privacy (sp)_ , pp. 1277–1294. Ieee, 2020a.
  * Chen et al. (2020b) Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In _International Conference on Machine Learning_ , pp. 1691–1703. PMLR, 2020b.
  * Cubuk et al. (2018) Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation policies from data.  _arXiv preprint arXiv:1805.09501_ , 2018.
  * Dai et al. (2021) Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen.  Up-detr: Unsupervised pre-training for object detection with transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 1601–1610, 2021.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2020.
  * Han et al. (2021) Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang.  Transformer in transformer.  _Advances in Neural Information Processing Systems_ , 34:15908–15919, 2021.
  * Heng et al. (2020) Minghang Heng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong.  End-to-end object detection with adaptive clustering transformer.  In _Proceedings of the The British Machine Vision Conference (BMVC)_ , 2020.
  * (10) Jeremy Howard.  imagenette.  URL https://github.com/fastai/imagenette/.
  * Kannan et al. (2014) Rajkumar Kannan, Gheorghita Ghinea, and Sridhar Swaminathan.  Salient region detection using patch level and region level image abstractions.  _IEEE Signal Processing Letters_ , 22(6):686–690, 2014.
  * Kotyan & Vargas (2022) Shashank Kotyan and Danilo Vasconcellos Vargas.  Adversarial robustness assessment: Why in evaluation both l0 and l∞ attacks are necessary.  _Plos one_ , 17(4):e0265723, 2022.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 10012–10022, 2021.
  * Lu et al. (2016) Yan-Feng Lu, Hua-Zhen Zhang, Tae-Koo Kang, and Myo-Taeg Lim.  Dominant orientation patch matching for hmax.  _Neurocomputing_ , 193:155–166, 2016.
  * Lu et al. (2019) Yanfeng Lu, Lihao Jia, Hong Qiao, Yi Li, and Zongshuai Qi.  Enhanced biologically inspired model for image recognition based on a novel patch selection method with moment.  _International Journal of Wavelets, Multiresolution and Information Processing_ , 17(02):1940007, 2019.
  * Mahmood et al. (2021) Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk.  On the robustness of vision transformers to adversarial examples.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 7838–7847, 2021.
  * Montabone & Soto (2010) Sebastian Montabone and Alvaro Soto.  Human detection using a mobile platform and novel features derived from a visual saliency mechanism.  _Image and Vision Computing_ , 28(3):391–402, 2010.
  * Nicolae et al. (2018) Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards.  Adversarial robustness toolbox v1.1.0.  _CoRR_ , 1807.01069, 2018.  URL https://arxiv.org/pdf/1807.01069.
  * Pang et al. (2023) Youwei Pang, Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu.  Caver: Cross-modal view-mixed transformer for bi-modal salient object detection.  _IEEE Transactions on Image Processing_ , 32:892–904, 2023\.
  * Parmar et al. (2018) Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  In _International Conference on Machine Learning_ , pp. 4055–4064. Pmlr, 2018.
  * Ramachandran et al. (2019) Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens.  Stand-alone self-attention in vision models.  _Advances in neural information processing systems_ , 32, 2019.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Shao et al. (2022) Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, and Cho-jui Hsieh.  On the adversarial robustness of vision transformers.  In _Annual Conference on Neural Information Processing Systems_ , 2022\.
  * Su et al. (2019) Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai.  One pixel attack for fooling deep neural networks.  _IEEE Transactions on Evolutionary Computation_ , 23(5):828–841, 2019.
  * Sun et al. (2021) Zhiqing Sun, Shengcao Cao, Yiming Yang, and Kris M Kitani.  Rethinking transformer-based set prediction for object detection.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 3611–3620, 2021.
  * Tang & Ha (2021) Yujin Tang and David Ha.  The sensory neuron as a transformer: Permutation-invariant neural networks for reinforcement learning.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. (2021a) Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen.  Max-deeplab: End-to-end panoptic segmentation with mask transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 5463–5474, 2021a.
  * Wang et al. (2021b) Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 568–578, 2021b.
  * Wang et al. (2021c) Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia.  End-to-end video instance segmentation with transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8741–8750, 2021c.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pp. 38–45, Online, October 2020. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.emnlp-demos.6.
  * Wu et al. (2020) Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda.  Visual transformers: Token-based image representation and processing for computer vision.  _arXiv preprint arXiv:2006.03677_ , 2020.
  * Yang et al. (2020) Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo.  Learning texture transformer network for image super-resolution.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5791–5800, 2020.
  * Yang et al. (2023) Jiachen Yang, Zhuo Zhang, Shuai Xiao, Shukun Ma, Yang Li, Wen Lu, and Xinbo Gao.  Efficient data-driven behavior identification based on vision transformers for human activity understanding.  _Neurocomputing_ , 530:104–115, 2023.
  * Yuan et al. (2021) Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan.  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 558–567, 2021.
  * Zeng et al. (2020) Yanhong Zeng, Jianlong Fu, and Hongyang Chao.  Learning joint spatial-temporal transformations for video inpainting.  In _European Conference on Computer Vision_ , pp. 528–543. Springer, 2020.
  * Zhao et al. (2023) Yang Zhao, Shuai Xiao, Jiachen Yang, Wen Lu, and Xinbo Gao.  No-reference qquality index of tone-mapped images based on authenticity, preservation, and scene expressiveness.  _Signal Processing_ , 203:108782, 2023.
  * Zheng et al. (2021) Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 6881–6890, 2021.
  * Zhou et al. (2018) Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong.  End-to-end dense video captioning with masked transformer.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 8739–8748, 2018.
  * Zhu et al. (2021) Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In _International Conference on Learning Representations_ , 2021.


# GW170817 and GW190814: tension on the maximum mass

Antonios Nathanail  Institut für Theoretische Physik, Max-von-Laue-Strasse 1,
60438 Frankfurt, Germany  Elias R. Most Princeton Center for Theoretical
Science, Princeton University, Princeton, NJ 08544, USA  Princeton Gravity
Initiative, Princeton University, Princeton, NJ 08544, USA  School of Natural
Sciences, Institute for Advanced Study, Princeton, NJ 08540, USA  Luciano
Rezzolla Institut für Theoretische Physik, Max-von-Laue-Strasse 1, 60438
Frankfurt, Germany  Frankfurt Institute for Advanced Studies, Ruth-Moufang-
Strasse 1, 60438 Frankfurt, Germany  School of Mathematics, Trinity College,
Dublin 2, Ireland

###### Abstract

The detection of the binary events GW170817 and GW190814 has provided
invaluable constraints on the maximum mass of nonrotating configurations of
neutron stars, $M_{{}_{\rm TOV}}$. However, the large differences in the
neutron-star masses measured in GW170817 and GW190814 has also lead to a
significant tension between the predictions for such maximum masses, with
GW170817 suggesting that $M_{{}_{\rm TOV}}\lesssim 2.3\,M_{\odot}$, and
GW190814 requiring $M_{{}_{\rm TOV}}\gtrsim 2.5\,M_{\odot}$ if the secondary
was a (non- or slowly rotating) neutron star at merger. Using a genetic
algorithm, we sample the multidimensional space of parameters spanned by
gravitational-wave and astronomical observations associated with GW170817.
Consistent with previous estimates, we find that all of the physical
quantities are in agreement with the observations if the maximum mass is in
the range $M_{{}_{\rm TOV}}=2.210^{+0.116}_{-0.123}\,M_{\odot}$ within a
$2\textrm{-}\sigma$ confidence level. By contrast, maximum masses with
$M_{{}_{\rm TOV}}\gtrsim 2.5\,M_{\odot}$, not only require efficiencies in the
gravitational-wave emission that are well above the numerical-relativity
estimates, but they also lead to a significant under-production of the ejected
mass. Hence, the tension can be released by assuming that the secondary in
GW190814 was a black hole at merger, although it could have been a rotating
neutron star before.

equation of state — gravitational waves — methods: analytical — stars: neutron

††software: Scipy (Virtanen et al., 2020), Corner (Foreman-Mackey, 2016),

##  1 Introduction

The recent detection of the gravitational-wave (GW) event GW190814 has seen
involved the merger of black hole (BH), with a mass of $22.2-24.3\,M_{\odot}$,
with a compact object having a much smaller mass of $2.50-2.67\,M_{\odot}$
(The LIGO Scientific Collaboration et al., 2020). The unclear nature of the
secondary component has raised questions about the astrophysical evolutionary
paths that would yield objects with these masses in a binary system. When
assigning a NS nature to the secondary in GW190814, two scenarios are
possible. In the first one, the secondary was a nonrotating or slowly rotating
NS at merger, so that GW190814 should effectively be considered a BH-NS merger
(see, e.g., Zevin et al., 2020; Safarzadeh et al., 2020; Kinugawa et al.,
2020; Liu & Lai, 2020; Lu et al., 2020; Ertl et al., 2020, for some possible
formation scenarios). In this case, the maximum mass $M_{{}_{\rm TOV}}$ of
nonrotating NSs needs to reach values as large as $\gtrsim 2.5\,M_{\odot}$
(Fattoyev et al., 2020; Sedrakian et al., 2020; Tan et al., 2020; Tsokaros et
al., 2020; Godzieba et al., 2020; Biswas et al., 2020). In the second
scenario, the need for a large maximum mass can be replaced by the presence of
rapid rotation. In fact, it has been shown that uniformly rotating NS can
support about $20\%$ more mass than nonspinning ones (Breu & Rezzolla, 2016;
Shao et al., 2020). Note that in the case of NSs with a phase transition,
universal relations are still present, but depend on the properties of the
phase transition (Bozzola et al., 2019; Demircik et al., 2020).

Based on these universal relations, Most et al. (2020b) and Zhang & Li (2020)
have pointed out that a massive (rapidly) rotating NS with a mass
$>2.5\,M_{\odot}$ is perfectly consistent with a maximum mass $M_{{}_{\rm
TOV}}\simeq 2.3\,M_{\odot}$ inferred from the GW170817 event (see, e.g.,
Rezzolla et al., 2018; Shibata et al., 2019). Given the difficulty of
sustaining rapid rotation over the very long timescales associated with the
inspiral of the binary, the secondary must have collapsed at one point before
merger, so that in this second scenario GW190814 should effectively be
considered a BH-BH merger.

While a priori both scenarios are plausible, shedding light on which of them
is the most likely is important from several points of view. To this scope, we
here exploit the rich variety of GW and electromagnetic observables that have
been obtained with GW170817 to explore the two scenarios combining the
constraints set from the GW and electromagnetic signal from GW170817. In
particular, we employ a genetic algorithm to sample through the distributions
of maximum masses, ejected matter (Drout et al., 2017; Cowperthwaite et al.,
2017; Kasen et al., 2017; Villar et al., 2017; Coughlin et al., 2018), and GW
emission from numerical-relativity (NR) simulations (Zappa et al., 2018).
Consistent with previous results (Rezzolla et al., 2018; Shibata et al., 2019)
we find that GW170817’s observations clearly set an upper limit for the
maximum mass of $M_{{}_{\rm TOV}}\lesssim 2.33\,M_{\odot}$. When forcing the
algorithm to allow for maximum masses $M_{{}_{\rm TOV}}\gtrsim
2.4\,M_{\odot}$, we find that this requires unrealistically large GW
efficiencies from the merger remnant and a deficit in the ejected matter.

##  2 Framework for the genetic algorithm

The observations of a bright blue kilonova has provided convincing evidence
that the merger remnant in GW170817 could not have collapsed promptly to a BH.
Rather, it must have survived for a timescale of the order of one second (Gill
et al., 2019; Lazzati et al., 2020; Hamidani et al., 2020), and sufficiently
large so that the hypermassive NS (HMNS) produced by the merger has reached
uniform rotation at least in its core (Margalit & Metzger, 2017; Rezzolla et
al., 2018). Following Rezzolla et al. (2018), we recall that quasi-universal
relations exist between the masses of uniformly rotating stellar models along
the stability line to BH formation and the corresponding dimensionless angular
momentum $j_{\rm coll}$ normalised to the maximum (Keplerian) one $j_{\rm
Kep}$ (Breu & Rezzolla, 2016). We here express this relation as

$\displaystyle\chi(j_{\rm coll}/j_{\rm Kep}):=\frac{M_{\rm crit}}{M_{{}_{\rm
TOV}}}=1+\alpha_{2}\left(\frac{j_{\rm coll}}{j_{\rm
Kep}}\right)^{2}+\alpha_{4}\left(\frac{j_{\rm coll}}{j_{\rm
Kep}}\right)^{4}\,,$ (1)

where $\alpha_{2}=1.316\times 10^{-1}$ and $\alpha_{4}=7.111\times 10^{-2}$,
and the value of the Keplerian specific angular momentum is approximately
given by $j_{\rm Kep}\sim 0.68$ (see Eq. (4) in Most et al., 2020b, for more
accurate estimates). The function $\chi$ is defined between 0 and 1 and
describes all models with a mass that is critical for collapse to a BH. To fix
ideas, in the case of nonorotating models, $j_{\rm coll}=0$ and $\chi(0)=1$,
while for maximally rotating models $j_{\rm coll}=j_{\rm Kep}$ and (Breu &
Rezzolla, 2016) $\chi(1):={M_{\rm max}}/{M_{{}_{\rm TOV}}}\approx
1.20_{-0.05}^{+0.02}$, where $M_{\rm max}$ is the maximum mass that can be
sustained through uniform rotation (see Weih et al., 2018, for differentially
rotating stars). Note that range $\chi(1)$ is based on a specific set of
hadronic equations of state (EOSs) and that a different estimate suggests
$\chi(1)=1.17_{-0.05}^{+0.02}$ (Shao et al., 2020).

Because Eq. (1) expresses a relation between gravitational masses, while the
electromagnetic emission from GW170817 informs us about the ejected baryonic
mass, we need a relation between gravitational and baryonic mass $M_{b}$ for
uniformly rotating NSs at the mass-shedding limit (see, e.g., Timmes et al.,
1996; Gao et al., 2020, for a detailed discussion). Also in this case, this
relation obeys a quasi-universal relation near the values of the maximum mass
that, with a $2\textrm{-}\sigma$ uncertainty, is given by $\eta:={M_{b,{\rm
max}}}/{M_{\rm max}}\approx 1.171\pm 0.014$ at the maximum-mass limit
(Rezzolla et al., 2018). Note that $\eta$ is in principle a function of $M$
and that the value reported above is for $M=M_{\rm max}$. However, $\eta$ is
almost constant in the neighbourhood of $M_{\rm max}$ – where all of our
considerations are made – so that hereafter we simply write the conversion
between the two masses as $M_{b}=\eta M$.

The total gravitational mass of GW170817 as inferred from the GW signal is
$M_{g}=2.73_{-0.01}^{+0.04}$ (The LIGO Scientific Collaboration et al., 2019),
whose corresponding baryonic mass $M_{b}$ soon after the merger can be thought
of as being given by the combination of the baryonic mass in the HMNS
$M_{b,{{}_{\rm HMNS}}}$ – itself composed of the mass in the core and in a
Keplerian disk – and of the mass ejected dynamically, i.e.,

$M_{b}=M_{b,{\rm core}}+M_{b,{\rm disk}}+M_{\rm ej}^{\rm dyn}=\eta
M^{*}_{g}\,,$ (2)

where $M^{*}_{g}:=M_{g}-M_{\rm GW}^{\rm insp}$ and $M_{\rm GW}^{\rm insp}$ is
the energy lost to GWs in the inspiral. Here, the last equality relates the
baryonic and gravitational mass of the merger remnant. Defining now $\xi$ as
the fraction of the HMNS baryonic mass in the core, the two components of the
HMNS shortly after merger can be written as

$M_{b,{\rm core}}:=\xi\left(M_{b}-M_{\rm ej}^{\rm dyn}\right)=\xi\left(\eta
M^{*}_{g}-M_{\rm ej}^{\rm dyn}\right)$ (3)

The fraction $\xi$ is in principle unknown, but numerical simulations have
shown that this ratio is actually weakly dependent on the EOS and given by
$\xi\approx 0.95^{+0.04}_{-0.05}$ (Hanauske et al., 2017). As time goes by,
the merger remnant will loose part of its baryonic mass via the emission of
magnetically driven or viscous-driven winds, so that at collapse it will have
a baryonic mass

$M_{b}=M_{b,{\rm core}}^{\rm coll}+M_{b,{\rm disk}}^{\rm coll}+M_{\rm ej}^{\rm
dyn}+M_{\rm ej}^{\rm blue}+M_{\rm ej}^{\rm red}\,,$ (4)

where the last equality follows from rest mass conservation and $M_{b,{\rm
core}}^{\rm coll}$ and $M_{b,{\rm disk}}^{\rm coll}$ are the respective values
of the core and the disk at the time when BH formation of the core is
triggered, while $M_{\rm ej}^{\rm blue}$ ($M_{\rm ej}^{\rm red}$) is the part
of the ejected matter leading to the blue (red) emission in the kilonova and
differs from the dynamical ejecta from the timescale over which the material
is lost. The two components also differ in the typical velocities of the
matter, which is larger in the blue component ($v/c\lesssim 0.3$ for the blue
part and $v/c\ll 0.1$ for the red part), but also with in the electron
fraction $Y_{e}$, which is again larger in the blue component ($0.2\lesssim
Y_{e}\lesssim 0.3$ for the blue part and $Y_{e}\lesssim 0.2$ for the red
part). Numerical simulations of remnant disks indicate that most of the red
ejecta originate from the disk, whereas most of the blue ejecta will come from
the hot surface of the HMNS. Hence, for simplicity we will assume that the
blue ejecta originate from the HMNS only, while the red ejecta exclusively
represent unbound material of the disk. We classify the latter via a parameter

$\displaystyle f_{\rm disk}:=M_{\rm ej}^{\rm red}/M_{b,{\rm disk}}\simeq
0.2-0.5\,,$ (5)

representing the unbound fraction of the disk mass, which can be estimated
based on numerical simulations (Siegel & Metzger, 2017; Fernández et al.,
2019; Fujibayashi et al., 2018; Nathanail et al., 2020). In a merger scenario
as that of GW170817, where the remnant may have lived for about one second
(Gill et al., 2019), BH formation is triggered when the gravitational mass is
reduced by the emission of GWs and the remnant hits the stability line for
uniformly rotating models with a massive core $M_{b,\rm core}^{\rm coll}$

$\displaystyle M_{b,\rm core}^{\rm coll}$ $\displaystyle=\eta
M^{*}_{g}-M_{b,{\rm disk}}-M_{\rm ej}^{\rm dyn}-M_{\rm ej}^{\rm blue}=\eta\chi
M_{{}_{\rm TOV}}\,,$ (6)

where the last equality relates the baryon mass of the remnant core to the
maximum mass $M_{{}_{\rm TOV}}$ of nonrotating NS via Eq. (1). Indeed, when
expressed as as a constraint equation on the maximum mass, Eq. (6) can also be
written

$\displaystyle\eta\chi M_{{}_{\rm TOV}}=\xi\left(\eta M^{*}_{g}-M_{\rm
ej}^{\rm dyn}\right)-M_{\rm ej}^{\rm blue}\,.$ (7)

Two more equations can be used for consistency

$\displaystyle M_{\rm ej}^{\rm red}$ $\displaystyle=f_{\rm
disk}\left(1-\xi\right)\left(\eta M^{*}_{g}-M_{\rm ej}^{\rm dyn}\right)\,,$
(8) $\displaystyle\chi M_{{}_{\rm TOV}}$
$\displaystyle=M^{*}_{g}-\eta^{-1}\\!\left(M_{b,{\rm disk}}^{\rm
coll}\\!+\\!M_{\rm ej}^{\rm dyn}\\!+\\!M_{\rm ej}^{\rm blue}\\!+\\!M_{\rm
ej}^{\rm red}\right)-M^{\rm post}_{{}_{\rm GW}}\,,$ (9)

where the first one expresses the conservation of rest-mass leading to the
kilonova emission and the second one the conservation of gravitational mass
since $M^{\rm post}_{{}_{\rm GW}}$ is the mass lost to GWs after the merger.
Expression (9) does not constrain $M^{\rm post}_{{}_{\rm GW}}$, which thus
remains undetermined. As a way around (see also Fan et al. (2020)), we use an
approximate quasi-universal relation between the total mass lost to GWs
$M_{\rm GW}^{\rm tot}$ and the specific angular momentum of the remnant after
the merger (Zappa et al., 2018)

$\displaystyle m_{\rm GW}^{\rm tot}\sim c_{0}+c_{1}j_{\rm
rem,20}+c_{2}\left(j_{\rm rem,20}\right)^{2}\,,$ (10)

where $m_{\rm GW}^{\rm tot}:=M_{\rm GW}^{\rm tot}/({M_{g}\nu})$, $j_{\rm
rem,20}:=J_{\rm rem,20}/(M_{g}^{2}\nu)$ is the specific angular momentum of
the remnant within $\sim 20\,{\rm ms}$ from the merger, and
$\nu:=m_{1}m_{2}/(m_{1}+m_{2})^{2}$ is the symmetric mass ratio. Note that
$c_{0}=0.9,c_{1}=-0.4$ and $c_{2}=0.05$ (Zappa et al., 2018) and that the two
component masses $m_{1}$ and $m_{2}$ are chosen considering the low-spin prior
for GW170817, i.e., $\nu\in[0.243,0.25]$. By splitting total mass lost to GWs
into the two components relative to the inspiral and post-merger, i.e.,
$M_{\rm GW}^{\rm tot}=M^{\rm insp}_{{}_{\rm GW}}+M^{\rm post}_{{}_{\rm GW}}$,
Eq. (10) allows us to introduce an additional constraint between $j_{\rm
coll}$ – which we derive from Eq. (1) – and $M^{\rm post}_{{}_{\rm GW}}$.

Two more remarks before concluding the presentation of our methodology. First,
not all of the merger remnant’s angular momentum will end up in the collapsed
object. A number of physical processes will move part of the angular momentum
outwards, placing it on stable orbits relative to the newly formed BH. Because
the efficiency of this process depends on microphysics that is poorly
understood, we account for this by introducing a fudge factor $f_{B}$ defined
as $j_{\rm coll}=:(1-f_{B})j_{\rm rem,20}$, so that the specific angular
momentum of the disk is $j_{\rm disk}:=f_{B}j_{\rm rem,20}$. Second, since in
Eqs. (7), (8) and (9) the function $\chi$ always appears together with
$M_{{}_{\rm TOV}}$, it is difficult to set reasonable ranges for $\chi$.
However, numerical simulations have revealed that the dimensionless spin of
the BH produced by the merger $j_{{}_{\rm BH}}$ (and hence $j_{\rm
coll}\lesssim j_{{}_{\rm BH}}$) is actually constrained in a rather limited
range, i.e., $0.7\lesssim j_{{}_{\rm BH}}\lesssim 0.9$ (Kastaun et al., 2013;
Bernuzzi et al., 2014). Exploiting this information, and assuming
conservatively that $80\%$ of the specific angular momentum at collapse is
inherited by the BH, i.e., $j_{\rm coll}=0.8\,j_{{}_{\rm BH}}$, we can
effectively constrain $\chi$ to be in the range $1.11\leq\chi\lesssim 1.22$.
Very similar results are obtained when making the more drastic assumption that
only half of the BH spin comes from the remnant, i.e., $j_{\rm
coll}=0.5\,j_{{}_{\rm BH}}$, which further reduces the lower limit to be
$\chi=1.05$ (see Supplement Material for details).

In summary, we need to solve a multidimensional parametric problem as
expressed by Eqs. (7), (8) and (9) after varying in the appropriate ranges the
(ten) free parameters in the system: $\chi,\xi,\eta,M_{{}_{\rm GW}}^{\rm
insp},M_{\rm ej}^{\rm blue},M_{\rm ej}^{\rm dyn},M_{{}_{\rm TOV}},\nu,f_{\rm
disk}$ and $f_{B}$. While we treat all these parameters equally, some of them
($\eta,M_{\rm ej}^{\rm dyn},\nu$), vary in very narrow ranges and their
variations do not significantly affect the overall results. In practice, at
any iteration of the genetic algorithm we ensure that: (i) the total
gravitational mass of the system is $M_{g}=2.73_{-0.01}^{+0.04}$ (The LIGO
Scientific Collaboration & The Virgo Collaboration, 2017; The LIGO Scientific
Collaboration et al., 2019); (ii) the total ejected mass is $M_{\rm ej}^{\rm
tot}=0.0537\,\pm\,0.013\,M_{\odot}$ (Arcavi et al., 2017; Nicholl et al.,
2017; Chornock et al., 2017; Cowperthwaite et al., 2017; Villar et al., 2017;
Drout et al., 2017; Kasen et al., 2017; Tanaka et al., 2017; Waxman et al.,
2018; Coughlin et al., 2018); (iii) the dynamically ejected mass is $M_{\rm
ej}^{\rm dyn}\approx 10^{-3}\,M_{\odot}$ (Sekiguchi et al., 2015; Bovard et
al., 2017; Radice et al., 2018; Poudel et al., 2020); (iv) the blue/red
ejected components are respectively $0.005<M_{\rm ej}^{\rm
blue}/M_{\odot}<0.02$ and $0.03<M_{\rm ej}^{\rm red}/M_{\odot}<0.55$ (Drout et
al., 2017; Cowperthwaite et al., 2017; Smartt & Chen, 2017; Kasliwal et al.,
2017; Kasen et al., 2017; Villar et al., 2017; Tanaka et al., 2017; Waxman et
al., 2018; Coughlin et al., 2018) we have also explored a larger upper bound
on the blue ejecta, i.e., $M_{\rm ej}^{\rm blue}/M_{\odot}<0.05$, finding no
significant difference; see Supplemental Material; (v) the maximum mass is
taken to be in the range $1.70\,M_{\odot}<M_{{}_{\rm TOV}}<3\,M_{\odot}$, note
that the posterior lower bound is consistent with pulsar observations
(Antoniadis et al., 2013; Cromartie et al., 2020); (vi) the energy radiated in
GWs before the merger is constrained to be $0.035\lesssim M_{{}_{\rm GW}}^{\rm
insp}/M_{\odot}\lesssim 0.045$ (Zappa et al., 2018). Note that all of the
priors discussed in points (i)–(vi) are uniform.

##  3 Results

Figure 1: Uniform posterior from the analysis of Rezzolla et al. (2018)
(magenta) and the posterior obtained with the multidimensional genetic
algorithm (blue) discussed here. Indicated with vertical lines are the
$1\textrm{-}\sigma$ (dotted) and $2\textrm{-}\sigma$ (dashed) values.

Figure 2: Left panel: Posteriors for the mass radiated in GWs consistent with
the distribution in Fig. 1; the lavender-shaded area reports the upper limit
coming from NR simulations. Right panel: Posterior for the total ejected mass
consistent with the distribution in Fig. 1; the green-shaded area reports the
range estimated in the literature.

Figure 1 provides a first important impression of the results of the genetic-
algorithm. In particular, shown with a magenta-shaded area is the maximum-mass
estimate made by Rezzolla et al. (2018) and which is a simple uniform
posterior for $M_{{}_{\rm TOV}}=2.16_{-0.16}^{+0.17}\,M_{\odot}$. Shown
instead with a blue-shaded area is the posterior distribution obtained with
the genetic algorithm. The median of the distribution is $M_{{}_{\rm
TOV}}=2.210^{+0.117}_{-0.123}\,M_{\odot}$, where the errors reported here are
for $2\textrm{-}\sigma$ uncertainty. Overall, this yields a lower bound of
$M_{{}_{\rm TOV}}>2.087\,M_{\odot}$ and an upper bound of $M_{{}_{\rm
TOV}}<2.326\,M_{\odot}$ at $2\textrm{-}\sigma$ level (vertical dashed lines),
and thus in good agreement with massive-pulsar measurements (Antoniadis et
al., 2013; Cromartie et al., 2020) and previous estimates (Rezzolla et al.,
2018; Shibata et al., 2019). Interestingly, our results are in good agreement
with the conclusions reached by Shao et al. (2020) and Fan et al. (2020), who
have also considered the post-merger GW emission to deduce bounds on the
maximum mass.

As a consistency check, we can use set of parameters that yield the maximum-
mass distribution in Fig. 1, to estimate the GW energy lost both in the
inspiral and in the post-merger. This is shown in the left panel of Fig. 2,
where we report the posterior distributions for $M_{{}_{\rm GW}}^{\rm insp}$
(black dotted line) and $M_{\rm GW}^{\rm post}$ (black dashed line), as well
as their sum, $M_{{}_{\rm GW}}^{\rm tot}$ (black solid line). Also marked with
a vertical lavender-shaded is the upper limit estimated by Zappa et al.
(2018), $E^{\rm tot}_{\rm GW}/M_{g}\leq 0.045\,M_{\odot}c^{2}$, on the basis
of a large number of NR simulations, with an associated uncertainty of $20\%$.
A similar consistency is found in the right panel of Fig. 2, where we report
the posterior of the total ejected mass consistent with the maximum-mass
distribution in Fig. 1. Indicated with a green-shaded area are the constraints
obtained from the kilonova observations of GW170817. More specifically, the
width of the shaded area represents the standard deviation using various
estimates for the total ejected mass $M_{\rm ej}^{\rm tot}$ estimated for
GW170817 (Arcavi et al., 2017; Nicholl et al., 2017; Chornock et al., 2017;
Cowperthwaite et al., 2017; Villar et al., 2017; Drout et al., 2017; Kasen et
al., 2017; Tanaka et al., 2017; Waxman et al., 2018; Coughlin et al., 2018).
Clearly, also the ejected-mass distribution is in perfect agreement with
observational bounds when the maximum mass is below $2.326\,M_{\odot}$.

Figure 3: Left panel: The same as in the left panel of Fig. 2, but when
considering two fixed values for the maximum mass, i.e., $M_{{}_{\rm
TOV}}=2.4\,M_{\odot}$ (orange) and $M_{{}_{\rm TOV}}=2.5\,M_{\odot}$ (red).
Right panel: The same as in the right panel of Fig. 2, but for two fixed
maximum-mass values.

Given these results, it is natural to ask: does anything break down when
larger maximum masses are considered? The positive answer to this question is
contained in Fig. 3, whose panels are similar to those in Fig. 2, when however
the genetic algorithm is forced to consider two specific values of the maximum
mass, namely, $M_{{}_{\rm TOV}}=2.4\,M_{\odot}$ and $M_{{}_{\rm
TOV}}=2.5\,M_{\odot}$. Concentrating first on the left panel of Fig. 3, it is
clear that when allowing for large maximum masses, the mass radiated in GWs
after the merger, $M_{{}_{\rm GW}}^{\rm post}$ (red dashed line), is
significantly larger than what NR simulations predict; this is true both for
$M_{{}_{\rm TOV}}=2.4\,M_{\odot}$ and for $M_{{}_{\rm TOV}}=2.5\,M_{\odot}$.
This behaviour is due to the fact that remnants with a given $\chi$ will
radiate more GWs if they are more massive [cf., Eq. (9)]. Next, when
considering the right panel of Fig. 3 it is also easy to realize that large
maximum masses lead to a deficit in the ejected matter. This is simply due to
the fact that considering large-mass stars inevitably reduces the portion of
the budget available for the ejecta. We have confirmed that, even if
(unrealistically) large additional angular momentum transport was assumed,
these results remain unchanged for $M_{{}_{\rm TOV}}\gtrsim 2.5\,M_{\odot}$,
(see Supplemental Material).

In summary, while a value $M_{{}_{\rm TOV}}\lesssim 2.3\,M_{\odot}$ is fully
consistent with the GW emission from NR simulations and the observed ejected
mass, a value $M_{{}_{\rm TOV}}\gtrsim 2.5\,M_{\odot}$ requires efficiencies
in the GW emission that are well above the estimates from a large number of
accurate NR simulations and, overall, leads to an under-production of ejected
mass.

##  4 Conclusions

We have carried out a systematic investigation to ascertain whether the
tension on the maximum mass following the detections of GW170817 and GW190814
can in some way be resolved or at least attenuated. In particular, we have
employed a genetic algorithm to sample through the multidimensional space of
parameters that can be built on the basis of the astronomical observations
(i.e., ejected mass in the various components), GW observations (i.e.,
gravitational masses of the binary components), and of NR simulations (i.e.,
properties of the remnant and efficiency of GW emission).

The results of this investigation have allowed us to refine in a probabilistic
manner the previous estimates of the maximum mass (Rezzolla et al., 2018),
obtaining that $M_{{}_{\rm TOV}}=2.210^{+0.117}_{-0.123}\,M_{\odot}$ within a
$2\textrm{-}\sigma$ confidence level. In this range, all of the physical
quantities are in very good agreement with the estimates coming from the
observations. By contrast, we find that considering maximum masses with
$M_{{}_{\rm TOV}}\gtrsim 2.4-2.5\,M_{\odot}$ requires efficiencies in the GW
emission well above the NR estimates and leads to a significant under-
production of the ejected mass, well below the values expected from the
observations. Although robust, our results can be strengthened in a number of
ways. Improved post-merger modeling and long-term NR simulations (Fujibayashi
et al., 2018) can help to narrow the uncertainties in the parameter ranges for
$\chi$ and $j_{\rm rem}$. Refined universal relations of uniformly rotating
NSs including temperature dependence (Koliogiannis & Moustakidis, 2020), will
also help narrowing down the errors in $\eta$ and the upper limit of $\chi$.
Such improvements will be crucial to understand the viability of the maximum-
mass constraint for $M_{\rm TOV}\lesssim 2.3\,M_{\odot}$.

In light of these considerations, we conclude that the secondary in GW190814
was most likely a BH at merger, although it may well have been a rotating NS
at some stage during the evolution of the binary system.

Acknowledgements. It is a pleasure to thank C. Ecker, J. Papenfort, and L.
Weih for useful comments. Support comes in part also from “PHAROS”, COST
Action CA16214 and the LOEWE-Program in HIC for FAIR. ERM gratefully
acknowledges support from a joint fellowship at the Princeton Center for
Theoretical Science, the Princeton Gravity Initiative and the Institute for
Advanced Study.

## References

  * Antoniadis et al. (2013) Antoniadis, J., Freire, P. C. C., Wex, N., et al. 2013, Science, 340, 448
  * Arcavi et al. (2017) Arcavi, I., Hosseinzadeh, G., Howell, D. A., et al. 2017, Nature, 551, 64
  * Bauswein et al. (2017) Bauswein, A., Just, O., Janka, H.-T., & Stergioulas, N. 2017, Astrophys. J. Lett., 850, L34
  * Bernuzzi et al. (2014) Bernuzzi, S., Dietrich, T., Tichy, W., & Brügmann, B. 2014, Phys. Rev. D, 89, 104021
  * Biswas et al. (2020) Biswas, B., Nandi, R., Char, P., Bose, S., & Stergioulas, N. 2020, arXiv e-prints, arXiv:2010.02090
  * Bovard et al. (2017) Bovard, L., Martin, D., Guercilena, F., et al. 2017, Phys. Rev. D, 96, 124005
  * Bozzola et al. (2019) Bozzola, G., Espino, P. L., Lewin, C. D., & Paschalidis, V. 2019, European Physical Journal A, 55, 149
  * Breu & Rezzolla (2016) Breu, C., & Rezzolla, L. 2016, Mon. Not. R. Astron. Soc., 459, 646
  * Chornock et al. (2017) Chornock, R., Berger, E., Kasen, D., et al. 2017, Astrophys. J. Letters, 848, L19
  * Coughlin et al. (2018) Coughlin, M. W., Dietrich, T., Doctor, Z., et al. 2018, Mon. Not. R. Astron. Soc., 480, 3871
  * Cowperthwaite et al. (2017) Cowperthwaite, P. S., Berger, E., Villar, V. A., et al. 2017, Astrophys. J. Lett., 848, L17
  * Cromartie et al. (2020) Cromartie, H. T., Fonseca, E., Ransom, S. M., et al. 2020, Nature Astronomy, 4, 72
  * Demircik et al. (2020) Demircik, T., Ecker, C., & Järvinen, M. 2020, arXiv e-prints, arXiv:2009.10731
  * Drout et al. (2017) Drout, M. R., Piro, A. L., Shappee, B. J., et al. 2017, Science, 358, 1570
  * Ertl et al. (2020) Ertl, T., Woosley, S. E., Sukhbold, T., & Janka, H. T. 2020, Astrophys. J., 890, 51
  * Fan et al. (2020) Fan, Y.-Z., Jiang, J.-L., Tang, S.-P., Jin, Z.-P., & Wei, D.-M. 2020, Astrophys. J., 904, 119
  * Fattoyev et al. (2020) Fattoyev, F. J., Horowitz, C. J., Piekarewicz, J., & Reed, B. 2020, arXiv e-prints, arXiv:2007.03799
  * Fernández et al. (2019) Fernández, R., Tchekhovskoy, A., Quataert, E., Foucart, F., & Kasen, D. 2019, Mon. Not. R. Astron. Soc., 482, 3373
  * Foreman-Mackey (2016) Foreman-Mackey, D. 2016, Journal of Open Source Software, 1, 24
  * Fromm et al. (2019) Fromm, C. M., Younsi, Z., Baczko, A., et al. 2019, Astron. Astrophys., 629, A4
  * Fujibayashi et al. (2018) Fujibayashi, S., Kiuchi, K., Nishimura, N., Sekiguchi, Y., & Shibata, M. 2018, The Astrophysical Journal, 860, 64
  * Gao et al. (2020) Gao, H., Ai, S.-K., Cao, Z.-J., et al. 2020, Frontiers of Physics, 15, 24603
  * Gill et al. (2019) Gill, R., Nathanail, A., & Rezzolla, L. 2019, Astrophys. J., 876, 139
  * Godzieba et al. (2020) Godzieba, D. A., Radice, D., & Bernuzzi, S. 2020, arXiv e-prints, arXiv:2007.10999
  * Hamidani et al. (2020) Hamidani, H., Kiuchi, K., & Ioka, K. 2020, Mon. Not. R. Astron. Soc., 491, 3192
  * Hanauske et al. (2017) Hanauske, M., Takami, K., Bovard, L., et al. 2017, Phys. Rev. D, 96, 043004
  * Kasen et al. (2017) Kasen, D., Metzger, B., Barnes, J., Quataert, E., & Ramirez-Ruiz, E. 2017, Nature, 551, 80
  * Kasliwal et al. (2017) Kasliwal, M. M., Nakar, E., Singer, L. P., et al. 2017, Science, 358, 1559
  * Kastaun et al. (2013) Kastaun, W., Galeazzi, F., Alic, D., Rezzolla, L., & Font, J. A. 2013, Phys. Rev. D, 88, 021501
  * Kinugawa et al. (2020) Kinugawa, T., Nakamura, T., & Nakano, H. 2020, arXiv e-prints, arXiv:2007.13343
  * Koeppel et al. (2019) Koeppel, S., Bovard, L., & Rezzolla, L. 2019, Astrophys. J. Lett., 872, L16
  * Koliogiannis & Moustakidis (2020) Koliogiannis, P. S., & Moustakidis, C. C. 2020, arXiv e-prints, arXiv:2007.10424
  * Lazzati et al. (2020) Lazzati, D., Ciolfi, R., & Perna, R. 2020, Astrophys. J., 898, 59
  * Liu & Lai (2020) Liu, B., & Lai, D. 2020, arXiv e-prints, arXiv:2009.10068
  * Lu et al. (2020) Lu, W., Beniamini, P., & Bonnerot, C. 2020, arXiv e-prints, arXiv:2009.10082
  * Margalit & Metzger (2017) Margalit, B., & Metzger, B. D. 2017, Astrophys. J. Lett., 850, L19
  * Most et al. (2020a) Most, E. R., Papenfort, L. J., Tootle, S., & Rezzolla, L. 2020a, arXiv e-prints, arXiv:2012.03896
  * Most et al. (2020b) Most, E. R., Papenfort, L. J., Weih, L. R., & Rezzolla, L. 2020b, Mon. Not. R. Astron. Soc., 499, L82
  * Nathanail et al. (2020) Nathanail, A., Gill, R., Porth, O., Fromm, C. M., & Rezzolla, L. 2020, Mon. Not. R. Astron. Soc., 495, 3780
  * Nicholl et al. (2017) Nicholl, M., Berger, E., Kasen, D., et al. 2017, Astrophys. J. Letters, 848, L18
  * Poudel et al. (2020) Poudel, A., Tichy, W., Brügmann, B., & Dietrich, T. 2020, Phys. Rev. D, 102, 104014
  * Radice et al. (2018) Radice, D., Perego, A., Hotokezaka, K., et al. 2018, Astrophys. J., 869, 130
  * Rezzolla et al. (2018) Rezzolla, L., Most, E. R., & Weih, L. R. 2018, Astrophys. J. Lett., 852, L25
  * Safarzadeh et al. (2020) Safarzadeh, M., Ramirez-Ruiz, E., & Berger, E. 2020, arXiv e-prints, arXiv:2001.04502
  * Sedrakian et al. (2020) Sedrakian, A., Weber, F., & Li, J. J. 2020, Phys. Rev. D, 102, 041301
  * Sekiguchi et al. (2015) Sekiguchi, Y., Kiuchi, K., Kyutoku, K., & Shibata, M. 2015, Phys. Rev. D, 91, 064059
  * Shao et al. (2020) Shao, D.-S., Tang, S.-P., Sheng, X., et al. 2020, Phys. Rev. D, 101, 063029
  * Shibata et al. (2019) Shibata, M., Zhou, E., Kiuchi, K., & Fujibayashi, S. 2019, Phys. Rev. D, 100, 023015
  * Siegel & Metzger (2017) Siegel, D. M., & Metzger, B. D. 2017, Physical Review Letters, 119, 231102
  * Smartt & Chen (2017) Smartt, S., & Chen, T. e. a. 2017, Nature, 551, 75
  * Tan et al. (2020) Tan, H., Noronha-Hostler, J., & Yunes, N. 2020, arXiv e-prints, arXiv:2006.16296
  * Tanaka et al. (2017) Tanaka, M., Utsumi, Y., Mazzali, P. A., et al. 2017, Public. Astron. Soc. of Japan, 69, 102
  * The LIGO Scientific Collaboration & The Virgo Collaboration (2017) The LIGO Scientific Collaboration, & The Virgo Collaboration. 2017, Phys. Rev. Lett., 119, 161101
  * The LIGO Scientific Collaboration et al. (2019) The LIGO Scientific Collaboration, the Virgo Collaboration, Abbott, B. P., et al. 2019, Physical Review X, 9, 011001
  * The LIGO Scientific Collaboration et al. (2020) The LIGO Scientific Collaboration, the Virgo Collaboration, Abbott, R., et al. 2020, Astrophys. J. Lett., 896, L44
  * Timmes et al. (1996) Timmes, F. X., Woosley, S. E., & Weaver, T. A. 1996, Astrophys. J., 457, 834
  * Tsokaros et al. (2020) Tsokaros, A., Ruiz, M., & Shapiro, S. L. 2020, arXiv e-prints, arXiv:2007.05526
  * Villar et al. (2017) Villar, V. A., Guillochon, J., Berger, E., et al. 2017, Astrophys. J. Letters, 851, L21
  * Virtanen et al. (2020) Virtanen, P., Gommers, R., Oliphant, T. E., et al. 2020, Nature Methods, 17, 261
  * Waxman et al. (2018) Waxman, E., Ofek, E., Kushnir, D., & Gal-Yam, A. 2018, Mon. Not. R. Astron. Soc., 481, 3423
  * Weih et al. (2018) Weih, L. R., Most, E. R., & Rezzolla, L. 2018, Mon. Not. R. Astron. Soc., 473, L126
  * Zappa et al. (2018) Zappa, F., Bernuzzi, S., Radice, D., Perego, A., & Dietrich, T. 2018, Physical Review Letters, 120, 111101
  * Zevin et al. (2020) Zevin, M., Spera, M., Berry, C. P. L., & Kalogera, V. 2020, arXiv e-prints, arXiv:2006.14573
  * Zhang & Li (2020) Zhang, N.-B., & Li, B.-A. 2020, Astrophys. J., 902, 38

## Supplemental Material

In what follows we provide additional information that complements the one
provided in the main text. While the details illustrated below do not vary the
conclusions drawn in the main text, they provide additional technical details
on the genetic algorithm employed in our analysis. In addition, they help
investigate how the results change when the parameters are varied beyond the
(reasonable) ranges assumed so far.

For the solution of our multidimensional parametric problem the procedure we
adopt is as follows (see also Fromm et al., 2019; Nathanail et al., 2020, for
additional information). We start by recalling that genetic algorithms are
designed to generate high-quality solutions to problems of this type where a
searching optimization is sought. The name follows from the operators of
mutation, crossover and selection that are normally found in biological
systems. Our choice of a genetic algorithm in place of a more traditional
Bayesian analysis based on a Markov-Chain Monte Carlo approach is motivated
mostly by the overall simplicity of our problem and the reduced computational
costs that are associated with a genetic algorithm.

In practice, our algorithm samples through the parameter space of the ten free
parameters. From those it computes the $M_{\rm ej}^{\rm red}$ through Eqs. (3)
and (5), $M^{\rm post}_{{}_{\rm GW}}$ through Eq. (10) and the specific
angular momentum at collapse $j_{\rm coll}$ via Eq. (1), using the sampled
value of$\chi$. Subsequently, Eqs. (7), (8), and (9) are solved to match the
observed values of $M_{g}$ and $M_{\rm ej}^{\rm tot}$ within the errors,
finding the best-fit values. The genetic algorithm employed here makes use of
Python packages from the SciPY software library (Virtanen et al., 2020).

As a corollary to the discussion made in the main text and relative to Figs.
1–3, we provide with the corner plots in Fig. 4 information on the probability
distribution functions of the various quantities involved in our analysis.
More specifically, Fig. 4 shows the corner plot relative to maximum-mass
posterior shown in Fig. 1 and should therefore accompany the information
presented in Fig. 2. On the other hand, Fig. 5 refers to the case when the
genetic algorithm is forced to consider $M_{{}_{\rm TOV}}=2.5\,M_{\odot}$. In
this case, the maximum-mass is set to vary uniformly in the very small
interval around $M_{{}_{\rm TOV}}=2.5^{+0.0001}_{-0.0001}\,M_{\odot}$, leaving
all the other parameters free to be adjusted till a best-fit is found. In this
sense, the information in Fig. 5 complements what is reported in Fig. 3 and
shows that all the posterior distributions are pushed to be very narrow at the
edges of the allowed ranges. For instance, the dimensionless spin $\chi$ is
narrowly peaked around its minimum value $1.1$, the mass in the disk is much
smaller and of the order of $\simeq 0.035\,M_{\odot}$, while the blue and red
ejecta are comparable and equal to $\simeq 0.019\,M_{\odot}$.

Note that to avoid having a large number of small panels, we have limited
ourselves either to the most salient ones, omitting those quantities for which
the distributions are either almost constant or restricted to a very small
region. More specifically, in Fig. 4 the values found are:
$\xi=0.973_{-0.035}^{+0.011}$, $\eta=1.171_{-0.013}^{+0.014}$, $M_{\rm
ej}^{\rm dyn}=0.001_{-0.001}^{+0.001}\,M_{\odot}$, $f_{\rm
disk}=0.437_{-0.209}^{+0.061}$, and $\nu=0.243_{-0.000}^{+0.005}$, which
corresponds to $q=0.719_{-0.006}^{+0.136}$. We have also explored a modified
scenario in which the blue ejecta are larger than inferred from observations.
In particular, we have adjusted the upper bound on the blue ejecta from
$M_{\rm ej}^{\rm blue}/M_{\odot}<0.02$ to $M_{\rm ej}^{\rm
blue}/M_{\odot}<0.05$. In this case, we find that the blue ejecta converge to
a distribution with a median around $\sim 0.03\,M_{\odot}$, while the red
ejecta component decreases to $\sim 0.02\,M_{\odot}$. At the same time, the
changes in the posterior for the maximum mass are minute, i.e., $M_{{}_{\rm
TOV}}=2.192^{+0.125}_{-0.092}\,M_{\odot}$.

Figure 4: Corner plot reporting the posterior distributions of the most
important parameters in our analysis. Indicated with the two outermost
vertical dashed lines are the corresponding $2\textrm{-}\sigma$ values, while
the labels on the diagonal cells report the average values (central vertical
dashed line). Figure 5: Same as Fig. 4 but when the maximum mass is held fixed
at the value $M_{{}_{\rm TOV}}=2.5\,M_{\odot}$.

Finally, in Fig. 6 we provide information that is similar in content to that
in Fig. 3, but when we allow for the dimensionless spin to attain even smaller
values, i.e., $1.05\leq\chi\lesssim 1.22$. Note that in this case, the ejected
mass for $M_{{}_{\rm TOV}}=2.4\,M_{\odot}$ is within the observational bounds,
but the excess in radiated mass is more severe. The disagreement becomes even
stronger for $M_{{}_{\rm TOV}}=2.5\,M_{\odot}$.

Figure 6: The same as Fig. 3 but when we allow for the dimensionless spin to
attain even smaller values, i.e., $1.05\leq\chi\lesssim 1.22$. Note that in
this case disagreement in the radiated and ejected mass becomes even stronger
for $M_{{}_{\rm TOV}}=2.5\,M_{\odot}$.

As a concluding remark we note that the interpretation of the nature of
GW190425 is likely unaffected by our findings on the maximum masses of neutron
stars. While a BH-NS nature cannot be fully ruled out, the most plausible case
of a NS-NS nature of the system is perfectly compatible with our findings on
the maximum mass, as the initial masses in GW190425 are both well below the
maximum-mass limit we have presented here (see also Most et al., 2020a, for a
discussion on GW190425). On the other hand, an indirect impact that our
results have on GW190425 is on whether the merger led to a prompt collapse
(i.e., where the hypermassive neutron star collapses to a black hole either at
or shortly after merger), or to a stable long lived remnant. Using the results
of Koeppel et al. (2019) (but see also Bauswein et al., 2017), and given the
values for the maximum mass found here, a prompt or delayed collapse scenario
seems likely for GW190425.