11institutetext: Concordia University and North South University
11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>11email:
<EMAIL_ADDRESS>
# ViPTT-Net: Video pretraining of spatio-temporal model for tuberculosis type
classification from chest CT scans

Hasib Zunair 11 Aimon Rahman 22 Nabeel Mohammed 33112233112233

###### Abstract

Pretraining has sparked groundswell of interest in deep learning workflows to
learn from limited data and improve generalization. While this is common for
2D image classification tasks, its application to 3D medical imaging tasks
like chest CT interpretation is limited. We explore the idea of whether
pretraining a model on realistic videos could improve performance rather than
training the model from scratch, intended for tuberculosis type classification
from chest CT scans. To incorporate both spatial and temporal features, we
develop a hybrid convolutional neural network (CNN) and recurrent neural
network (RNN) model, where the features are extracted from each axial slice of
the CT scan by a CNN, these sequence of image features are input to a RNN for
classification of the CT scan. Our model termed as ViPTT-Net, was trained on
over 1300 video clips with labels of human activities, and then fine-tuned on
chest CT scans with labels of tuberculosis type. We find that pretraining the
model on videos lead to better representations and significantly improved
model validation performance from a kappa score of 0.17 to 0.35, especially
for under-represented class samples. Our best method achieved 2nd place in the
ImageCLEF 2021 Tuberculosis - TBT classification task with a kappa score of
0.20 on the final test set with only image information (without using clinical
meta-data). All codes and models are made available.
111https://github.com/hasibzunair/viptt-net

###### Keywords:

3D image classification Human Action Recognition Spatial-Temporal Information
Pretraining

††footnotetext: Copyright © 2021 for this paper by its authors. Use permitted
under Creative Commons License Attribution 4.0 International (CC BY 4.0). CLEF
2021 – Conference and Labs of the Evaluation Forum, September 21–24, 2021,
Bucharest, Romania

##  1 Introduction

Tuberculosis (TB) is a potentially fatal disease that generally affects the
lungs. The disease spreads through cough, spit or sneeze and can remain latent
within the human body. Although X-rays and microscopic analysis of bodily
fluid are generally used to diagnose the disease, Computed Tomography (CT)
provides detailed information about the infection. Deep learning models
demonstrate promising results in diagnosing TB from both X-rays and CT scans
[14, 19, 18, 16]. These methods are also proven to be effective for severity
scoring of the infection as well [28, 27, 17]. However, the types of TB can
vary and that may require a different course of treatment, making the
identification of TB type an important real-life problem. The deep learning
models are yet to provide high accuracy in this particular task.

The most challenging part of working with CT scan data is, it is three-
dimensional (3D), meaning along with height and width, each data point
contains depth information. Processing 3D data can be very computationally
expensive and may require pre-processing before training a model. These pre-
processing steps may include choosing selective slices or transforming slices
into uniform sizes. Both of these techniques may contribute to losing some
depth information. One of the most effective method of processing 3D data is
to resize it into fixed dimension using spline interpolation along x, y and z-
axis which was demonstrated in [28]. Although the method showed promising
result in TB severity scoring, might not perform well on other more
sophisticated tasks such as classifying TB types.

An alternative to directly processing 3D data is to decompose each scan into
individual slices and afterward feeding them 2D CNN [5, 8, 6, 9]. The problem
with this method is, as the slices are considered independent to each other,
the spatial information along the z-axis is missing during training. Moreover,
in this method, the label for the whole volume is assigned to each individual
slice, which might not be the case when it comes to CT scans. More
specifically, a CT scan of infected lungs may contain uninfected 2D slices.

In this work, we build a model to predict multiple tuberculosis types from
chest CT scans. We pretrain a hybrid convolutional neural network (CNN) and
recurrent neural network (RNN) model on human action recognition task, and the
fine-tune the model for the tuberculosis type classification task. Pretraining
significantly improves performance, especially for minority classes. The
method is evaluated on the Image-CLEF 2021 Tuberculosis - TBT classification
task which achieves 2nd place 222imageclef-2021-tuberculosis-tbt-
classification/leaderboards with a kappa score of 0.20 and accuracy of 0.42.

We summarize our contributions as follows:

  1. 1.

We pretrain a hybrid CNN-RNN model, termed ViPTT-Net, on human action
recognition task, and fine-tune the model on a small dataset of CT scans with
labels indicating tuberculosis types.

  2. 2.

We show pretraining ViPTT-Net on realistic videos improve performance for
tuberculosis type classification, especially for under-represented class
samples.

  3. 3.

We evaluate our best method on the Image-CLEF 2021 Tuberculosis - TBT
classification task which achieves 2nd place overall.

##  2 Methodology

We start by problem formulation followed by presenting the main building
blocks of our proposed method for multi-class CT image classification.

###  2.1 Problem Formulation

Given the labels of a set of CT scans, the objective is to predict the unknown
labels of the new CT scans
333https://www.imageclef.org/2021/medical/tuberculosis. More specifically, our
goal is to learn a discriminative function
$f(\textbf{X})\in\\{\textbf{1},\textbf{2},\textbf{3},\textbf{4},\textbf{5}\\}$,
where the numbers represent the tuberculosis type: Infiltrative, Focal,
Tuberculoma, Miliary and Fibro-cavernous repsectively. X represents a CT scan
volume of size $D\times W\times H$, where D, W, and H represent the depth,
width, and height of the volume respectively. The task is considered as a
multi-class volumetric image classification problem. A 3D volumetric scan X
can also be viewed as a time-series of 2D slices $\\{{X_{1},\dots,X_{D}\\}}$.
Therefore, we can also frame the task as a time-series sequence classification
problem.

###  2.2 ViPTT-Net Model

While convolutional neural networks (CNNs) have shown promising results at
processing image data [15, 13, 10], the same can be said for recurrent neural
networks (RNN) for sequential data [4, 22, 21]. We developed a hybrid
convolutional neural network (CNN) and recurrent neural network (RNN) model
termed ViPTT-Net which is capable of incorporating both spatial and temporal
features in the learning process. ViPTT-Net takes as input a CT scan and
outputs probability predictions which indicates the type of tuberculosis in
that CT scan. To deal with arbitrary volumes, we first resize the CT scan to a
fixed size $70\times 224\times 224$ using spline interpolated zoom (SIZ) [28]
which exploits the full geometry of the 3D volume by interpolating over the
z-axis.

Learning spatial features using CNN. To learn spatial features, ViPTT-Net
consists of a VGG-16 model as the feature extractor which is pretrained on
ImageNet [3]. The VGG-16 model has 16 layers with learnable weights: 13
convolutional layers, and 3 fully connected layers [23]. We extract features
from the last convolutional layer which results in a 512 dimensional feature
vector for a 2D input image. Since a CT scan consists of multiple 2D axial
slices (70 in our case), we wrap the VGG-16 model in a time-distributed layer
which is then applied to every axial slice (temporal frame) of the CT scan
independently. The final output is a sequence of image features where the
sequence length is 70 and each of the sequences are a 512 dimensional feature
vector.

As the VGG-16 feature extractor accepts inputs of 3-channels, we map the
1-channel axial slices of the CT scan slices to 3-channel using a
convolutional layer with 3 filters and kernel size of $1\times 1\times 1$
before input to the feature extractor.

Learning temporal features using RNN. Following the above, the sequence of
image features are aggregated using an RNN. RNNs are a type of neural network
which transforms a sequence of inputs into a sequence of outputs. Using an
RNN, the temporal order of the axial slices is preserved. We use the long
short-term memory (LSTM) [7] with 256 units and tanh activation. It is
important to mention that since we are interested in classifying the full
sequence, the output from the LSTM results in a 256 dimensional aggregated
feature vector from the last time step coming from the full sequence of image
features, rather than producing a sequence of outputs. We also experimented
with gated recurrent unit (GRU) but did not get good results on the validation
set.

Finally, the aggregated 256 dimensional feature vector is passed to a dense
layer with 1024 units with rectified linear hidden units (ReLUs) activation
and a dense layer with a softmax function (i.e. a dense softmax layer of 5
units for the multi-class classification case) that yields the probabilities
of predicted classes. While training on UCF50, instead of a softmax with 5
units, we use a softmax with 10 units for the 10 classes. An illustration of
ViPTT-Net is shown in Fig 1.

Figure 1: Schematic layout of the hybrid CNN-RNN model ViPTT-Net. Given a 3D
CT scan of arbitrary size, uniform resizing is performed across all dimensions
using SIZ [28]. Features are extracted from all the axial slices of the
processed CT scan to output a sequence of image features using a VGG-16 model.
These sequence of image features are input to an LSTM layer followed by dense
layers of 1024 neurons and finally 5 with softmax activation for the multi-
class classification problem.

###  2.3 Pretraining on Human Action Recognition Task

ViPTT-Net was first pretrained on a subset of the UCF50 dataset [20]. UCF50 is
human action recognition dataset with 50 action categories, consisting of
realistic videos taken from youtube. The classes of different action are
Baseball Pitch, Jumping Jack, Kayaking etc. Due to compute constraints, we use
1366 videos from 10 classes selected randomly: Mixing, Tennis Swing, Horse
Riding, Jump Rope, Jumping Jack, Baseball Pitch, Rowing, SkateBoarding,
Walking With Dog, Skijet. Video clips are resized to $70\times 224\times 224$,
where 70 is the number of temporal frames (image sequences), and 224 is width
and height of the temporal frame. It is also important to mention that each
RGB frame of the video is converted to grayscale (single channel) since the CT
scans also have single channel values. An instance of an action video is shown
in Fig 2(a).

Similar to pretraining for 2D image problems [25, 26], after training ViPTT-
Net on a subset of UCF50 dataset, the network was fine-tuned on the
tuberculosis type classification task by replacing the final fully connected
10-way softmax layer with a 5-way softmax. During training ViPTT-Net on UCF50,
the weights of the VGG-16 feature extractor was frozen. And while fine-tuning
on the tuberculosis type classification task, all the network layers were
trained.

(a) Sequence of images from a video of a man playing Jumping Jack.

(b) Sequence of axial images from a CT scan with the label Infiltrative.

Figure 2: Illustration of image sequences of a video and a CT scan from the
UCF50 and ImageCLEF 2021 - TBT datasets. Both samples are resized to depth of
70.

###  2.4 Weighted Loss and Data Augmentation

Due to heavy class imbalance, we assign weights in the loss function for each
tuberculosis type with the goal to reduce biasness towards the over-
represented class samples. Prior to training using weighted loss, the weights
are computed over the training set.

It is a standard practice to perform data augmentation to improve
generalization, especially when there are limited number of training data.
Data augmentation basically creates modified versions of the input data in a
dataset through random transformations such as horizontal and vertical flip,
zoom augmentation, horizontal and vertical shift, etc. While training, the 3D
CT scans are rotated with degree of rotations picked randomly from
$[-20,-10,-5,0,5,10,20]$ as a form of data augmentation. Notice that we added
0 in the range which means that the model looks at both augmented and non-
augmented data. We experimented without adding 0 in the range and led to poor
results on the validation set. We also tried blur and random shifts but did
not get good results.

##  3 Experiment Setup

We describe experimental details, i.e., dataset, implementation details,
evaluaton metrics, etc., and present quantitative results, comparing different
training strategies using ViPTT-Net.

###  3.1 Datasets and Preprocessing

The dataset is provided by ImageCLEF Tuberculosis Type 2021 Challenge [11,
12]. It contains CT scans of a total of 1338 TB patients, 917 of them have
been used for training and 421 for the test set. Each scan has a label that
indicates one of the TB types- Infiltrative, Focal, Tuberculoma, Miliary and
Fibro-cavernous. Each CT scan belongs to only one patient. The images have
dimensions of 512 x 512 pixels with varying depth sizes. In addition to
labels, some scans have additional meta-data. All the scans have auto-
generated lungs mask, although these masks are generally missing in largely
affected areas or have rough bounds. The original data is in NIFTI format,
storing the raw voxel intensity in Hounsfield units (HU). An instance of a CT
scan with Infiltrative tuberculosis type is shown in Fig 2(b).

###  3.2 Implementation Details

All experiments are performed on a Linux workstation running 4.8Hz and 64GB
RAM with and RTX 3080 GPU. Experiments are conducted using Python programming
language [24]. ViPTT-Net is implemented in Keras [2], with TensorFlow backend
[1].

ViPTT-Net is trained end-to-end using stochastic gradient descent (SGD)
optimization algorithm to minimize the categorical cross-entropy loss function
with an initial learning rate of 0.001 and batch size of 2. A factor of 0.1 is
used to reduce the learning rate once the loss stagnates. Training is
continued until the validation loss stagnates using an early stopping
mechanism, and then the best weights are retained. To keep consistent data
proportions same and ensure reproducibility, we perform a stratified train and
validation split with a ratio of 80/20 (732/184 CT scans) on the training data
provided by ImageCLEF 2021 - TBT.

Similar steps are followed while training on a subset of the UCF50 dataset,
except we do not use any data augmentation in this case.

###  3.3 Evaluation Metrics

According to challenge rules, the task is evaluated as a multi-class
classification problem. The main evaluation metric is the kappa score. Kappa
measures the inter-rater reliability for categorical items.

$\kappa\equiv\frac{p_{0}-p_{e}}{1-p_{e}}$ (1)

Here, $p_{0}$ indicates the relative observed agreement among raters and
$p_{e}$ is the probability of agreement by chance. If raters are in complete
agreement then $\kappa=1$ and no agreement other than by chance will result in
$\kappa=0$. The value would be negative if there is no effective agreement
among the raters or the agreement is worse than random. Additionally we also
show accuracy (ACC) and per-class F1 scores. For all metrics, the higher the
better.

###  3.4 Results

We study the effect of training ViPTT-Net for the task of tuberculosis type
classification using different training strategies:

No PT. ViPTT-Net is trained from scratch on the 732 CT scans annotated for
tuberculosis types

PT. ViPTT-Net is first pretrained on a subset of the UCF50 dataset which
around 1300 video clips annotated for human activities. Then the last layer is
replaced with a five unit softmax and fine-tuned on 732 CT scans annotated for
tuberculosis types.

PT+CW. Same as PT, but additionally weighted loss is used with weights
computed over the training set of 732 CT scans annotated for tuberculosis
types

PT+CW+AUG. Same as PT+CW, but additionally the 3D volumes are randomly rotated
during training.

Table 1 summarizes the results which show that pretraining ViPTT-Net on a
subset of the UCF50 dataset followed by fine-tuning for tuberculosis type
classification (PT) improves kappa score by 0.18 on the validation set
compared to training ViPTT-Net from scratch (No PT). In both configurations
(No PT and PT), F1 score is the lowest for Tuberculoma class with scores of 0
and 0.09 for No PT and PT respectively. This is improved by using weighted
loss (PT+CW), where the model achieves Tuberculoma F1 score 0.37. The F1 score
of PT+CW also improves for FibC class by a large margin compared to No PT,
although there is a slight drop in overall kappa score. PT+CW with data
augmentation (PT+CW+AUG) further improves performance, more specifically for
Miliary class by a large margin.

Table 1: Overall Kappa score and per-class F1 score achieved by different methods on the validation dataset of tuberculosis type classification. FibC denotes Fibro-cavernous. For all methods, the ViPTT-Net model is used. Method | Kappa | Infiltrative | Focal | Tuberculoma | Miliary | FibC
---|---|---|---|---|---|---
No PT | 0.17 | 0.61 | 0.46 | 0.0 | 0.2 | 0.14
PT | 0.35 | 0.68 | 0.56 | 0.09 | 0.4 | 0.48
PT+CW | 0.30 | 0.59 | 0.41 | 0.37 | 0.33 | 0.65
PT+CW+AUG | 0.33 | 0.59 | 0.47 | 0.27 | 0.54 | 0.61

We also report the final submission results of our best model on the test set
by ImageCLEF 2021 - TBT classification task in Table 2. We observe a similar
trend in performance improvements on the test set when using weighted loss.
Our best method ViPTT-Net which is trained with weighted cross entropy loss
and data augmentation performs the best on the test set.

Table 2: Overall Kappa and Accuracy score achieved by different methods on the test set by ImageCLEF 2021 Tuberculosis - TBT classification. Method | Kappa | Accuracy
---|---|---
PT | 0.13 | 42.30
PT+CW | 0.14 | 38.50
PT+CW+AUG | 0.20 | 42.30

##  4 Discussion and Conclusion

We address the problem of predicting tuberculosis types from 3D chest CT
scans. We develop a hybrid CNN-RNN model, termed ViPTT-Net, which is capable
of learning both spatial and temporal features of the CT scan. Our experiments
demonstrate that video pretraining on human action recognition task
significantly improves performance of tuberculosis type classification rather
than training the model from scratch. This is most significant for Miliary and
Fibro-cavernous types without specifically aiming to improve performance for
those classes. Interestingly, these classes along with Tuberculoma are the
tuberculosis types which have the least amount of samples in the dataset. To
further deal with class imbalance, we use a weighted loss function with
weights computed over the training set that improves performance of under-
represented classes significantly. Data augmentation also improved performance
for few tuberculosis types on the validation set. On the test set, the highest
kappa score was observed when pretraining ViPTT-Net on videos, using weighted
loss function and data augmentation. This method achieved 2nd place in the
ImageCLEF 2021 Tuberculosis - TBT classification task which operates on the CT
image alone without using the additional patient meta-data and the lung
segmentation masks.

## References

  * [1] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.: Tensorflow: A system for large-scale machine learning. In: 12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16). pp. 265–283 (2016)
  * [2] Chollet, F., et al.: Keras. https://keras.io (2015)
  * [3] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [4] Ebrahimi Kahou, S., Michalski, V., Konda, K., Memisevic, R., Pal, C.: Recurrent neural networks for emotion recognition in video. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. pp. 467–474 (2015)
  * [5] Gao, X.W., Hui, R., Tian, Z.: Classification of CT brain images based on deep learning networks. Computer methods and programs in biomedicine 138, 49–56 (2017)
  * [6] Gentili, A.: ImageCLEF2019: Tuberculosis-severity scoring and CT report with neural networks, transfer learning and ensembling. CLEF2019 Working Notes 2380, 9–12 (2019)
  * [7] Greff, K., Srivastava, R.K., Koutník, J., Steunebrink, B.R., Schmidhuber, J.: LSTM: A search space odyssey. IEEE transactions on neural networks and learning systems 28(10), 2222–2232 (2016)
  * [8] Grewal, M., Srivastava, M.M., Kumar, P., Varadarajan, S.: RADnet: Radiologist level accuracy using deep learning for hemorrhage detection in CT scans. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). pp. 281–284. IEEE (2018)
  * [9] Hamadi, A., Cheikh, N.B., Zouatine, Y., Menad, S.M.B., Djebbara, M.R.: ImageCLEF 2019: Deep learning for tuberculosis CT image analysis. CLEF2019 Working Notes 2380, 9–12 (2019)
  * [10] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [11] Ionescu, B., Müller, H., P’eteri, R., Ben Abacha, A., Sarrouti, M., Demner-Fushman, D., Hasan, S.A., Kozlovski, S., Liauchuk, V., Dicente, Y., Kovalev, V., Pelka, O., de Herrera, A.G.S., Jacutprakart, J., Friedrich, C.M., Berari, R., Tauteanu, A., Fichou, D., Brie, P., Dogariu, M., Ştefan, L.D., Constantin, M.G., Chamberlain, J., Campello, A., Clark, A., Oliver, T.A., Moustahfid, H., Popescu, A., Deshayes-Chossart, J.: Overview of the ImageCLEF 2021: Multimedia retrieval in medical, nature, internet and social media applications. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the 12th International Conference of the CLEF Association (CLEF 2021), LNCS Lecture Notes in Computer Science, Springer, Bucharest, Romania (September 21-24 2021)
  * [12] Kozlovski, S., Liauchuk, V., Dicente Cid, Y., Kovalev, V., Müller, H.: Overview of ImageCLEFtuberculosis 2021 - CT-based tuberculosis type classification. In: CLEF2021 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org $<$http://ceur-ws.org$>$, Bucharest, Romania (September 21-24 2021)
  * [13] Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems. pp. 1097–1105 (2012)
  * [14] Lakhani, P., Sundaram, B.: Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284(2), 574–582 (2017)
  * [15] LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)
  * [16] Li, X., Zhou, Y., Du, P., Lang, G., Xu, M., Wu, W.: A deep learning system that generates quantitative CT reports for diagnosing pulmonary tuberculosis. Applied Intelligence pp. 1–12 (2020)
  * [17] Liauchuk, V.: ImageCLEF 2019: Projection-based CT image analysis for TB severity scoring and CT report generation. CLEF2019 Working Notes 2380, 9–12 (2019)
  * [18] Nash, M., Kadavigere, R., Andrade, J., Sukumar, C.A., Chawla, K., Shenoy, V.P., Pande, T., Huddart, S., Pai, M., Saravu, K.: Deep learning, computer-aided radiography reading for tuberculosis: a diagnostic accuracy study from a tertiary hospital in india. Scientific reports 10(1), 1–10 (2020)
  * [19] Rajpurkar, P., O’Connell, C., Schechter, A., Asnani, N., Li, J., Kiani, A., Ball, R.L., Mendelson, M., Maartens, G., van Hoving, D.J., et al.: CheXaid: deep learning assistance for physician diagnosis of tuberculosis using chest x-rays in patients with hiv. NPJ digital medicine 3(1), 1–8 (2020)
  * [20] Reddy, K.K., Shah, M.: Recognizing 50 human action categories of web videos. Machine vision and applications 24(5), 971–981 (2013)
  * [21] Salehinejad, H., Ho, E., Lin, H.M., Crivellaro, P., Samorodova, O., Arciniegas, M.T., Merali, Z., Suthiphosuwan, S., Bharatha, A., Yeom, K., et al.: Deep sequential learning for cervical spine fracture detection on computed tomography imaging. arXiv preprint arXiv:2010.13336 (2020)
  * [22] Salehinejad, H., Sankar, S., Barfett, J., Colak, E., Valaee, S.: Recent advances in recurrent neural networks. arXiv preprint arXiv:1801.01078 (2017)
  * [23] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
  * [24] Van Rossum, G., et al.: Python programming language. In: USENIX annual technical conference. vol. 41, p. 36 (2007)
  * [25] Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., He, Q.: A comprehensive survey on transfer learning. Proceedings of the IEEE 109(1), 43–76 (2020)
  * [26] Zunair, H., Hamza, A.B.: Melanoma detection using adversarial training and deep transfer learning. Physics in Medicine & Biology 65(13), 135005 (2020)
  * [27] Zunair, H., Rahman, A., Mohammed, N.: Estimating severity from CT scans of tuberculosis patients using 3D convolutional nets and slice selection. In: CLEF (Working Notes) (2019)
  * [28] Zunair, H., Rahman, A., Mohammed, N., Cohen, J.P.: Uniformizing techniques to process CT scans with 3D CNNs for tuberculosis prediction. In: International Workshop on PRedictive Intelligence In MEdicine. pp. 156–168. Springer (2020)


# Ultrafast artificial intelligence: Machine learning with atomic-scale
quantum systems

Thomas Pfeifer1, Matthias Wollenhaupt2, Manfred Lein3

###### Abstract

We train a model atom to recognize hand-written digits between 0 and 9,
employing intense light–matter interaction as a computational resource. For
training, individual images of hand-written digits in the range 0-9 are
converted into shaped laser pulses (data input pulses). Simultaneously with an
input pulse, another shaped pulse (program pulse), polarized in the orthogonal
direction, is applied to the atom and the system evolves quantum mechanically
according to the time-dependent Schrödinger equation. The purpose of the
optimal program pulse is to direct the system into specific atomic final
states that correspond to the input digits. A success rate of about 40% is
demonstrated here for a basic optimization scheme, so far limited by the
computational power to find the optimal program pulse in a high-dimensional
search space. This atomic-intelligence image-recognition scheme is scalable
towards larger (e.g. molecular) systems, is readily reprogrammable towards
other learning/classification tasks and operates on time scales down to tens
of femtoseconds. It has the potential to outpace other currently implemented
machine-learning approaches, including the fastest optical on-chip
neuromorphic systems and optical accelerators, by orders of magnitude.

1Max-Planck-Institut für Kernphysik, 69117 Heidelberg, Germany
2Carl-von-Ossietzky Universität Oldenburg, Institut für Physik, Carl-von-
Ossietzky-Straße 9-11, 26129 Oldenburg, Germany
3Leibniz Universität Hannover, Institut für Theoretische Physik, Appelstraße
2, 30167 Hannover, Germany

corresponding authors<EMAIL_ADDRESS>matthias.wollenhaupt@uni-
oldenburg.de<EMAIL_ADDRESS>
Artificial intelligence is a research and application field of growing
interest, due to recent successes and breakthroughs in deep (multi-layer)
learning enabled by constant increases in (classical) computational power.
Within this field, image recognition, i.e., the classification of different
but conceptually equivalent (input) images into unique (output) categories,
has been one of the prime applications of artificial intelligence and machine
learning for many years [1]. With recent advances in optical science and
technology, in particular optical neuromorphic hardware [2], it has recently
been possible to accelerate image recognition to sub-nanosecond timescales
[3]. Here, the operation timescale is determined by the speed of light at
which information propagates in microscopic waveguides on the chip, and is
thus directly proportional to its size: The smaller the device and
computational units, the faster the clock rates for the recognition operations
will become. The fundamental limit of further minimization is the atomic
scale.

In atomic and molecular physics, extreme timescales down to femtosecond and
even attosecond duration are currently explored and controlled by measuring
and steering the motion of one, two, or several electrons [4, 5, 6, 7] with
lasers. Specific atomic states can be excited and laser coupled on ultrafast
(femtosecond) time scales [8], providing a quantum analog of neurons (quantum
states) and axons (laser coupling between states). Ground-state atoms and
molecules are fully quantum-correlated systems and therefore entanglement
naturally arises when these systems are fragmenting into two or more particles
[9, 10], e.g. due to photoionization or dissociation. Control of entangled
states using attosecond and femtosecond laser pulses has also recently been
addressed [11, 12, 13, 14]. The naturally arising question is whether atoms
and their interaction with intense laser light can be used as a high-speed
computational resource in applications such as machine learning for image
recognition. This question has recently been addressed for the case of two-
class recognition of hand-written digits and three-class recognition of iris-
flower types, using the process of high-order harmonic generation and thus an
optical output channel [15]. In the present work, we investigate whether an
atom could at the same time act as a quantum processor _and_ readout register
for machine learning, mapping two-dimensional images of digits directly to
atomic quantum states. The latter could then be either read out or serve as
input to subsequent (quantum) processing tasks.

Figure 1:  Conceptual representation of atomic artificial intelligence. A
quantum system (here: an atom) interacts with an input/data (id) pulse and a
program/code (pc) pulse (here: along two orthogonal polarization directions)
to deliver the data and the code, respectively. The quantum state populations
after the interaction are read out (e.g. by projecting them into the continuum
and employing an electron spectrometer) and the maximum population in a
certain state (after renormalization) is used to classify the result. The pc
pulse is found by training on a large number of hand-written digits.

In the following, we introduce our scheme and present the results for a model-
atom simulation towards a proof-of-principle demonstration. Both the input
data and the code that is necessary to process the data are fed into the atom
in the form of femtosecond light pulses (id pulse and pc pulse), see Fig. 1.
Due to the ultrafast time scale, incoherent coupling to the environment can be
neglected, i.e., the dynamics of the atom can be described as the time
evolution of a pure state evolving according to the time-dependent Schrödinger
equation (TDSE) $\mathrm{i}\partial_{t}\Psi=\mathcal{H}\Psi$ (atomic units are
used unless stated otherwise). We show that the atom can be successfully
trained, despite a huge parameter space for the optical program pulse that is
just obtained by evolutionary optimization. In future work, the training
approach can be vastly improved by using more suitable parametrizations for
the program pulse and advanced statistical methods based on Bayesian
inference.

Figure 2:  A sample of hand-written digits and their encoding into electric
fields (in arbitrary units) along one of the polarization axes. Figure 3:  (A)
Experimental implementation of the code and the number by shaped pulses in the
$x-$ and $y-$direction. (B) Atomic system with target states (0-9). (C) Scheme
for the iterative optimization of the code. Each of the 20 individuals
(represented by their 8x8 numerical codes) of one generation are put together
with all 100 hand-written digits of the training set and each individual’s
fitness $F$ is obtained by the sum over all the single fitness values,
$F=\sum_{n=1}^{100}f_{n}$.

We employ a multi-level Hamiltonian $\mathcal{H}_{0}$ to describe the atom,
which is dipole coupled to an external arbitrarily polarized time-dependent
laser field vector $\bm{E}(t)=E_{x}(t)\,\bm{e}_{x}+E_{y}(t)\,\bm{e}_{y}$ with
Cartesian unit vectors $\bm{e}_{x},\bm{e}_{y}$. The interacting Hamiltonian
thus reads

$\mathcal{H}=\mathcal{H}_{0}+\mathcal{V}_{x}\,E_{x}(t)+\mathcal{V}_{y}\,E_{y}(t).$
(1)

Using a 20-level model with basis states 1s, 2s, 2p-1,1, 3s, 3p-1,1, 3d-2,0,2,
4s, 4p-1,1, 4d-2,0,2, 4f-3,-1,1,3, we have

$\displaystyle\mathcal{H}_{0}=\begin{pmatrix}E_{\mathrm{1s}}&0&0&\\\
0&\\!\\!E_{\mathrm{2s}}&0&\\\ 0&0&\\!\\!E_{\mathrm{2p}}&\\\\[-6.544pt]
&&&{\mbox{\footnotesize{$\ddots$}}}\end{pmatrix}$ (2)

and

$\displaystyle\mathcal{V}_{x}=\begin{pmatrix}0&0&a&-a&\\\ 0&0&a&-a&\\\
a&a&0&0&\\\ -a&-a&0&0&\\\\[-7.11317pt]
&&&&{\mbox{\footnotesize{$\ddots$}}}\end{pmatrix},\hskip
5.69046pt\mathcal{V}_{y}=\begin{pmatrix}0&0&-\mathrm{i}a&-\mathrm{i}a\\\
0&0&-\mathrm{i}a&-\mathrm{i}a\\\ \mathrm{i}a&\mathrm{i}a&0&0\\\
\mathrm{i}a&\mathrm{i}a&0&0\\\\[-7.11317pt]
&&&&{\mbox{\footnotesize{$\ddots$}}}\end{pmatrix}.$ (3)

For simplicity, the energies are chosen as $E_{nl}=-1/(n+1)^{2}$. The coupling
matrix elements read $\mathcal{V}_{x}^{jk}=\,\langle
l,m|\sin\theta\cos\phi|l^{\prime},m^{\prime}\rangle$ and
$\mathcal{V}_{y}^{jk}=\,\langle
l,m|\sin\theta\sin\phi|l^{\prime},m^{\prime}\rangle$, where the index $j$
corresponds to the state $|n,l,m\rangle$, the index $k$ corresponds to the
state $|n^{\prime},l^{\prime},m^{\prime}\rangle$, and $|l,m\rangle$ are
angular momentum states. Here, the radial integrals have been set to unity to
emphasize the universality of the model. This means that, for example, the
number $a$ appearing in Eq. (3) is $a=\langle
0,0|\sin\theta\cos\phi|1,-1\rangle$. The 1s state $|1,0,0\rangle$ is taken as
the initial state for the time evolution.

In our first approach, the fields $E_{x}$ and $E_{y}$ are constructed as
follows. From a $\cos^{2}$-shaped spectral amplitude

$\tilde{E}(\omega)=\tilde{E}_{0}\,\cos^{2}{\left(\frac{\omega-\omega_{0}}{\Omega}{\pi}\right)},\quad\omega_{0}-\Omega/2\leq\omega\leq\omega_{0}+\Omega/2,$
(4)

where the total length of the spectrum is $\Omega=\pi/32\,\rm a.u.$ and the
central frequency $\omega_{0}$ corresponds to a laser wavelength of 800 nm,
the field is obtained as a Fourier-synthesis multiplied by an additional
$\cos^{2}$ temporal envelope to restrict the pulse to a finite total length
$T$, i.e.,

$E(t)=\sum_{j=0}^{63}\,\tilde{E}(\omega_{j})\,\cos(-\omega_{j}t+\varphi_{j})\,\cos^{2}(t\pi/T),\quad-T/2\leq
t\leq T/2$ (5)

with

$\omega_{j}=\omega_{0}+(j-31)\,\Delta\omega,\quad\Delta\omega=2\pi/T,\quad
T=4096\,\textrm{a.u.},\quad\varphi_{j}=-3v_{j}\textrm{rad}.$ (6)

Here, $v_{j}$ are the 64 values describing the input (hand-written digits from
the scikit-learn python package, 8x8 pixel representation, processed column-
wise from the lower right to the upper left corner, $0\leq v_{j}\leq 1$) or
program (numerical phase function of 64 different values), see Fig. 2 for
illustration.

The TDSE is then solved numerically by a split-step operator approach with a
time step of 1 a.u., and the state populations $p_{i}$ (“output”) are read out
for each pair of 100 input and 20 program fields. As not all states will be
equally populatable for an arbitrary set of pulses, we define normalized
populations $P_{i}=p_{i}/p_{0i}$, where $p_{0i}$ is obtained by applying
random pc pulses and reading out their final state populations. Among the
final states, we select 10 states (all dipole-accessible 4s, 4p, 4d, 4f
states) to encode the digits $i$ from “0” to “9”, and assign to each program
field a fitness $F=3N+P$ with $N$ the number of matches between the input
(hand-written digit) and classification output (state with highest normalized
population). To reward the pulses with particularly high population in the
correct classification state, $P$ represents the sum over all corresponding
normalized populations $P_{i}$ for each of the 100 input fields from the
training set.

The program fields (here: their 64-step phase functions) are then iteratively
optimized by an evolutionary algorithm to maximize their fitness. The aim is
to find an optimal program field that realizes correct classification of all
training digits. The basic evolutionary algorithm employed here for training
the model atom uses a population size of 20 individuals, each represented by
an array of 64 numbers (its “genes”), determining the spectral phase of the pc
pulse in the same way as for the input pulses. For each individual pc pulse
its fitness is calculated by applying the pulse along with 100 hand-written
digits from the training set. The best pc pulse is always kept for the next
generation (“cloning”) while the remaining 19 individuals are obtained by a
combination of cross-over and mutation (using random numbers) for its
64-number genetic array.

The results of three example optimization runs are shown in Fig. 4. While the
percentage of correctly classified digits generally rises throughout the
optimization (black line), it is particularly interesting to observe the
correlated increase of the success rate on the test set (grey line). As the
100-sample test set of digits is not shown to the algorithm during training
with the 100-sample training set, the increase of the test-set success rate
signifies that generalization is achieved in this approach, i.e., small
deviations between different hand-written versions of the same digit do not
prevent a correct classification.

Figure 4:  Results of three example runs of the evolutionary algorithm to
optimize the program fields for digit recognition. Black: Fittest individual
when applied to the training set. Grey: Fittest individual when applied to an
(unseen) test set for validation of the generalization of learning. Levels of
$>\,$40% and $\sim\,$30% recognition success rates ($N$/100 in %) for training
and test sets are routinely achieved, respectively.

In summary, we have introduced the concept of optically programmable learning
employing atomic quantum states for classification. For a proof of principle,
the concept was applied here to recognition of hand-written digits,
implemented with a few-level model of an atom and a straightforward encoding
of input/data and program/code as spectral phase functions of two orthogonally
polarized femtosecond optical laser fields.

Once the optimal program pulse is known, the digit-recognition code runs on a
femtosecond time scale, which is much faster than the processing time in any
classical or quantum computer. The ultrafast time evolution excludes
environment-induced decoherence and makes the proposed scheme robust. We note
that replacing the atom by a larger quantum system, such as a complex
molecule, will increase the size of the Hilbert space enormously, while the
ultrafast time scale is retained.

A key benefit of our approach is the following: The program field could be
modified towards other computational tasks, such as identifying letters, other
images, or prime numbers in future applications, always using the same quantum
system as computational core. The quantum system can thus be considered an
optically reprogrammable multi-purpose quantum processor (OpReMuQ). Moreover,
due to the intrinsic memory capability of atoms, they can be used as nodes in
larger networks of many atoms (or molecules), thus involving many entangled
electrons and vastly increasing the number of layers that can be used for
achieving more complex tasks. While in this first demonstration we essentially
only explored a one-particle few-level model, future experimental
implementations will involve few- or many-body dynamics to employ a much
larger state space and thus higher effective number of coupled layers of
states (neurons). A crucial point that sets this scheme apart from existing
quantum-information approaches and platforms is the fact that (entangling)
operations are not performed on spatially _separated_ entities but on
_compact_ systems of interacting particles. The implementation of traditional
quantum gates is therefore not possible; instead, suitably-shaped structured
light has to be found to perform the operations necessary for the envisaged
task.

Our scheme can be viewed as machine learning with a microscopic quantum
system. Similar to classical machine learning, there is not necessarily a
simple explanation as to how the trained machine takes its decisions. This
“lack of insight” has not prevented conventional artificial intelligence from
revolutionizing technology in recent years. We thus expect that ultrafast
artificial intelligence on the atomic scale has the potential to exploit
quantum mechanics for high-speed computational tasks in the future.

## References

  * [1] R. Plamondon and S.N. Srihari.  Online and off-line handwriting recognition: a comprehensive survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 22:63–84, 2000.
  * [2] J. Feldmann, N. Youngblood, C. D. Wright, H. Bhaskaran, and W. H. P. Pernice.  All-optical spiking neurosynaptic networks with self-learning capabilities.  Nature, 569:208–214, 2019.
  * [3] F. Ashtiani, A. J. Geers, and F. Aflatouni.  An on-chip photonic deep neural network for image classification.  Nature, 606:501–506, 2022.
  * [4] D. Pengel, S. Kerbstadt, D. Johannmeyer, L. Englert, T. Bayer, and M. Wollenhaupt.  Electron vortices in femtosecond multiphoton ionization.  Phys. Rev. Lett., 118:053003, 2017.
  * [5] W. Jiang, G. S. J. Armstrong, J. Tong, Y. Xu, Z. Zuo, J. Qiang, P. Lu, D. D. A. Clarke, J. Benda, A. Fleischer, H. Ni, K. Ueda, H. W. van der Hart, A. C. Brown, X. Gong, and J. Wu.  Atomic partial wave meter by attosecond coincidence metrology.  Nature Communications, 13(1):5072, 2022.
  * [6] M. Kretschmar, A. Hadjipittas, B. Major, J. Tümmler, I. Will, T. Nagy, M. J. J. Vrakking, A. Emmanouilidou, and B. Schütte.  Attosecond investigation of extreme-ultraviolet multi-photon multi-electron ionization.  Optica, 9(6):639–644, 2022.
  * [7] M. Yu, K. Liu, M. Li, J. Yan, C. Cao, J. Tan, J. Liang, K. Guo, W. Cao, P. Lan, Q. Zhang, Y. Zhou, and P. Lu.  Full experimental determination of tunneling time with attosecond-scale streaking method.  Light: Science & Applications, 11(1):215, 2022.
  * [8] S. Meister, A. Bondy, K. Schnorr, S. Augustin, H. Lindenblatt, F. Trost, X. Xie, M. Braune, B. Manschwetus, N. Schirmel, H. Redlin, N. Douguet, T. Pfeifer, K. Bartschat, and R. Moshammer.  Linear dichroism in few-photon ionization of laser-dressed helium.  The European Physical Journal D, 75(7):205, 2021.
  * [9] D. Akoury, K. Kreidi, T. Jahnke, T. Weber, A. Staudte, M. Schöffler, N. Neumann, J. Titze, L. P. H. Schmidt, A. Czasch, O. Jagutzki, R. A. C. Fraga, R. E. Grisenti, R. D. Muino, N. A. Cherepkov, S. K. Semenov, P. Ranitovic, C. L. Cocke, T. Osipov, H. Adaniya, J. C. Thompson, M. H. Prior, A. Belkacem, A. L. Landers, H. Schmidt-Böcking, and R. Dörner.  The simplest double slit: Interference and entanglement in double photoionization of H2.  Science, 318(5852):949–952, 2007.
  * [10] M. S. Schöffler, J. Titze, N. Petridis, T. Jahnke, K. Cole, L. P. Schmidt, A. Czasch, D. Akoury, O. Jagutzki, J. B. Williams, N. A. Cherepkov, S. K. Semenov, C. W. McCurdy, T. N. Rescigno, C. L. Cocke, T. Osipov, S. Lee, M. H. Prior, A. Belkacem, A. L. Landers, H. Schmidt-Böcking, T. Weber, and R. Dörner.  Ultrafast probing of core hole localization in N2.  Science, 320(5878):920–3, 2008.
  * [11] M. J. J. Vrakking.  Control of attosecond entanglement and coherence.  Physical Review Letters, 126(11):5, 2021.
  * [12] L. M. Koll, L. Maikowski, L. Drescher, T. Witting, and M. J. J. Vrakking.  Experimental control of quantum-mechanical entanglement in an attosecond pump-probe experiment.  Physical Review Letters, 128(4):6, 2022.
  * [13] H. Laurell, D. Finkelstein-Shapiro, C. Dittel, C. Guo, R. Demjaha, M. Ammitzböll, R. Weissenbilder, L. Neoričić, S. Luo, M. Gisselbrecht, C. L. Arnold, A. Buchleitner, T. Pullerits, A. L’Huillier, and D. Busto.  Continuous-variable quantum state tomography of photoelectrons.  Physical Review Research, 4:033220, 2022.
  * [14] F. Shobeiry, P. Fross, H. Srinivas, T. Pfeifer, R. Moshammer, and A. Harth.  Sub-femtosecond optical control of entangled states.  arXiv, 2110.06668, 2022.
  * [15] G. McCaul, K. Jacobs, and D. I. Bondar.  Towards single atom computing via high harmonic generation.  Eur. Phys. J. Plus, 138:123, 2023.


51 |  $\Delta$FWHM([N ii]) [km s-1] | Error on FWHM([N ii])
52 |  $z$(DR2) | The source redshift, as reported in the BASS/DR2 catalog (Koss et al., 2022b)
53 |  $z_{\rm corr}$(DR2, [S ii]) | Updated redshift, based on NLR measurements of the [S ii] emission lines from this paper
54 | DR2Type | DR2 Seyfert type according to Winkler (1992) classification (see Section 3.1 for details).

Note. — All errors are $1\sigma$ equivalent, and were obtained obtained
through our spectral bootstrapping procedure. When both lower and upper errors
are reported, these correspond to the 16th and 84th percentiles of the
corresponding distribution. When a single error is reported, it corresponds to
the standard deviation.

##

Table 6: Column description for the H$\beta$ measurements table.

Column | Name | Description
---|---|---
1 | BAT_ID | BASS identifier
2 | Subsample | The source of the optical spectrum (facility, instrument, etc.)
3 |  $f_{\rm Q}$(H$\beta$) | Quality flag for the fit: 1 for a good fit and 2 for an acceptable one
4 | FWHM(bH$\beta$) [km s-1] | FWHM of the broad H$\beta$ emission line
5 |  $\Delta^{-}$ FWHM(bH$\beta$) [km s-1] | Lower error on FWHM(bH$\beta$)
6 |  $\Delta^{+}$ FWHM(bH$\beta$) [km s-1] | Upper error on FWHM(bH$\beta$)
7 |  $\log L\left({\rm b}{\text{H}}\beta\right)$ [erg s-1] | Broad H$\beta$ line luminosity
8 |  $\log F\left({\rm b}{\text{H}}\beta\right)$ [erg s-1cm-2] | Broad H$\beta$ line integrated flux
9 |  $\Delta^{-}\log L\left({\rm b}{\text{H}}\beta\right)$ [erg s-1] | Lower error on $\log L\left({\rm b}{\text{H}}\beta\right)$
10 |  $\Delta^{+}\log L\left({\rm b}{\text{H}}\beta\right)$ [erg s-1] | Upper error on $\log L\left({\rm b}{\text{H}}\beta\right)$
11 |  $\log L_{5100}$ [erg s-1] | Monochromatic luminosity at rest-frame 5100 Å, $\lambda L_{\lambda}$(5100 Å)
12 |  $\log F_{5100}$ [erg s-1cm-2] | Monochromatic Flux at rest-frame 5100 Å, $\lambda F_{\lambda}$(5100 Å)
13 |  $\Delta^{-}\log L_{5100}$ [erg s-1] | Lower error on $\log L_{5100}$
14 |  $\Delta^{+}\log L_{5100}$ [erg s-1] | Upper error on $\log L_{5100}$
15 | EW(bH$\beta$) [Å] | Rest-frame equivalent width of the broad H$\beta$ line
16 |  $\Delta^{-}$EW(bH$\beta$) [Å] | Lower error on EW(bH$\beta$)
17 |  $\Delta^{+}$EW(bH$\beta$) [Å] | Upper error on EW(bH$\beta$)
18 |  $\log L_{\rm peak}$(bH$\beta$) [erg s-1 Å-1] | Monochromatic luminosity of the broad H$\beta$ line at peak
19 |  $\log F_{\rm peak}$(bH$\beta$) [erg cm-2 s-1 Å-1] | Monochromatic flux of the broad H$\beta$ line at peak
20 |  $\Delta^{-}\log L_{\rm peak}$(bH$\beta$) [erg s-1 Å-1] | Lower error on $\log L_{\rm peak}$(bH$\beta$)
21 |  $\Delta^{+}\log L_{\rm peak}$(bH$\beta$) [erg s-1 Å-1] | Upper error on $\log L_{\rm peak}$(bH$\beta$)
22 |  $\Delta v_{\rm peak}$(bH$\beta$) [km s-1] | Velocity shift of the broad H$\beta$ line peak
23 |  $\Delta^{-}\Delta v_{\rm peak}$(bH$\beta$) [km s-1] | Lower error on $\Delta v_{\rm peak}$(bH$\beta$)
24 |  $\Delta^{+}\Delta v_{\rm peak}$(bH$\beta$) [km s-1] | Upper error on $\Delta v_{\rm peak}$(bH$\beta$)
25 |  $\Delta v_{\rm cent}$(bH$\beta$) [km s-1] | Velocity shift of the broad H$\beta$ line centroid
26 |  $\Delta^{-}\Delta v_{\rm cent}$(bH$\beta$) [km s-1] | Lower error on $\Delta v_{\rm cent}$(bH$\beta$)
27 |  $\Delta^{+}\Delta v_{\rm cent}$(bH$\beta$) [km s-1] | Upper error on $\Delta v_{\rm cent}$(bH$\beta$)
28 |  $\log M_{\rm BH}$(bH$\beta$) [$M_{\odot}$, TN12] | Broad H$\beta$-based BH mass estimate, following Trakhtenbrot & Netzer (2012)
29 |  $\Delta^{-}\log M_{\rm BH}$(bH$\beta$) [$M_{\odot}$] | Lower error on $\log M_{\rm BH}$(bH$\beta$)
30 |  $\Delta^{+}\log M_{\rm BH}$(bH$\beta$) [$M_{\odot}$] | Lower error on $\log M_{\rm BH}$(bH$\beta$)
31 |  $\log L$([O iii]) [erg s-1] | Luminosity of the narrow [O iii] $\lambda 5007$ emission line
32 |  $\log F$([O iii]) [erg s-1cm-2] | Integrated flux of the narrow [O iii] $\lambda 5007$ emission line
33 |  $\Delta\log L$([O iii]) [erg s-1] | Error on $\log L$([O iii])
34 |  $\Delta v$([O iii]) [km s-1] | Velocity shift of the [O iii] $\lambda 5007$ line peak/centroid
35 |  $\Delta\Delta v$([O iii]) [km s-1] | Error on $\Delta v$([O iii])
36 | FWHM([O iii]) [km s-1] | FWHM of the narrow [O iii] $\lambda 5007$ emission line
37 |  $\Delta$FWHM([O iii]) [km s-1] | Error on FWHM([O iii])
38 |  $\log L$(nH$\beta$) [erg s-1] | Luminosity of the narrow H$\beta$ line
39 |  $\log F$(nH$\beta$) [erg s-1cm-2] | Integrated flux of the narrow H$\beta$ line
40 |  $\Delta\log L$(nH$\beta$) [erg s-1] | Error on $\log L$(nH$\beta$)
41 |  $\Delta v$(nH$\beta$) [km s-1] | Velocity shift of the narrow H$\beta$ line peak/centroid
42 |  $\Delta\Delta v$(nH$\beta$) [km s-1] | Error on $\Delta v$(nH$\beta$)
43 | FWHM(nH$\beta$) [km s-1] | FWHM of the narrow H$\beta$ emission line
44 |  $\Delta$FWHM(nH$\beta$) [km s-1] | Error on FWHM(nH$\beta$)
45 |  $z$(DR2) | The source redshift, as reported in the BASS/DR2 catalog (Koss et al., 2022b)
46 |  $z_{\rm corr}$(DR2, [O iii]) | Updated redshift, based on NLR measurements of the [O iii] emission lines from this paper
47 | DR2Type | DR2 Seyfert type according to Winkler (1992) classification (see subsection 3.1 for details)

Note. — All errors are $1\sigma$ equivalent, and were obtained obtained
through our spectral bootstrapping procedure. When both lower and upper errors
are reported, these correspond to the 16th and 84th percentiles of the
corresponding distribution. When a single error is reported, it corresponds to
the standard deviation.

Table 7: Column description for the Mg ii $\lambda 2798$ measurements table.

Column | Name | Description
---|---|---
1 | BAT_ID | BASS identifier
2 | Subsample | The source of the optical spectrum (facility, instrument, etc.)
3 |  $f_{\rm Q}$(Mg ii) | Quality flag for the fit: 1 for a good fit and 2 for an acceptable one
4 | FWHM(Mg ii) [km s-1] | FWHM of the broad Mg ii emission line
5 |  $\Delta^{-}$ FWHM(Mg ii) [km s-1] | Lower error on FWHM(Mg ii)
6 |  $\Delta^{+}$ FWHM(Mg ii) [km s-1] | Upper error on FWHM(Mg ii)
7 |  $\log L$(Mg ii) [erg s-1] | Broad Mg ii line luminosity
8 |  $\log F$(Mg ii) [erg s-1cm-2] | Broad Mg ii line integrated flux
9 |  $\Delta^{-}\log L$(Mg ii) [erg s-1] | Lower error on $\log L$(Mg ii)
10 |  $\Delta^{+}\log L$(Mg ii) [erg s-1] | Upper error on $\log L$(Mg ii)
11 |  $\log L_{3000}$ [erg s-1] | Monochromatic luminosity at rest-frame 3000 Å, $\lambda L_{\lambda}$(3000 Å)
12 |  $\log F_{3000}$ [erg s-1cm-2] | Monochromatic flux at rest-frame 3000 Å, $\lambda F_{\lambda}$(3000 Å)
12 |  $\Delta^{-}\log L_{3000}$ [erg s-1] | Lower error on $\log L_{3000}$
13 |  $\Delta^{+}\log L_{3000}$ [erg s-1] | Upper error on $\log L_{3000}$
14 | EW(Mg ii) [Å] | Rest-frame equivalent width of the broad Mg ii line
15 |  $\Delta^{-}$EW(Mg ii) [Å] | Lower error on EW(Mg ii)
16 |  $\Delta^{+}$EW(Mg ii) [Å] | Upper error on EW(Mg ii)
17 |  $\log L_{\rm peak}$(Mg ii) [erg s-1 Å-1] | Monochromatic luminosity of the broad Mg ii line at peak
18 |  $\log F_{\rm peak}$(Mg ii) [erg s-1 Å-1] | Monochromatic flux of the broad Mg ii line at peak
19 |  $\Delta^{-}\log L_{\rm peak}$(Mg ii) [erg cm-2 s-1 Å-1] | Lower error on $\log L_{\rm peak}$(Mg ii)
20 |  $\Delta^{+}\log L_{\rm peak}$(Mg ii) [erg s-1 Å-1] | Upper error on $\log L_{\rm peak}$(Mg ii)
21 |  $\Delta v_{\rm peak}$(bH$\beta$) [km s-1] | Velocity shift of the broad Mg ii line peak
22 |  $\Delta^{-}\Delta v_{\rm peak}$(Mg ii) [km s-1] | Lower error on $\Delta v_{\rm peak}$(Mg ii)
23 |  $\Delta^{+}\Delta v_{\rm peak}$(Mg ii) [km s-1] | Upper error on $\Delta v_{\rm peak}$(Mg ii)
24 |  $\Delta v_{\rm cent}$(Mg ii) [km s-1] | Velocity shift of the broad Mg ii line centroid
25 |  $\Delta^{-}\Delta v_{\rm cent}$(Mg ii) [km s-1] | Lower error on $\Delta v_{\rm cent}$(Mg ii)
26 |  $\Delta^{+}\Delta v_{\rm cent}$(Mg ii) [km s-1] | Upper error on $\Delta v_{\rm cent}$(Mg ii)
28 |  $\log M_{\rm BH}$(Mg ii) [$M_{\odot}$, MR16] | Broad Mg ii-based BH mass estimate, following Mejía-Restrepo et al. (2016)
29 |  $\Delta^{-}\log M_{\rm BH}$(Mg ii) [$M_{\odot}$] | Lower error on $\log M_{\rm BH}$(Mg ii)
30 |  $\Delta^{+}\log M_{\rm BH}$(Mg ii) [$M_{\odot}$] | Lower error on $\log M_{\rm BH}$(Mg ii)
31 |  $z$(DR2) | The source redshift, as reported in the BASS/DR2 catalog (Koss et al., 2022b)
32 |  $z_{\rm corr}$(DR2, Mg ii) | Updated redshift, based on BLR measurements of the Mg ii emission line from this paper

Note. — All errors are $1\sigma$ equivalent, and were obtained obtained
through our spectral bootstrapping procedure. When both lower and upper errors
are reported, these correspond to the 16th and 84th percentiles of the
corresponding distribution. When a single error is reported, it corresponds to
the standard deviation.

Table 8: Column description for the C iv $\lambda 1549$ measurements table.

Column | Name | Description
---|---|---
1 | BAT_ID | BASS identifier
2 | Subsample | The source of the optical spectrum (facility, instrument, etc.).
3 |  $f_{\rm Q}$(C iv) | Quality flag for the fit: 1 for a good fit and 2 for an acceptable one
4 | FWHM(C iv) [km s-1] | FWHM of the broad C iv emission line
5 |  $\Delta^{-}$ FWHM(C iv) [km s-1] | Lower error on FWHM(C iv)
6 |  $\Delta^{+}$ FWHM(C iv) [km s-1] | Upper error on FWHM(C iv)
7 |  $\log L$(C iv) [erg s-1] | Broad C iv line luminosity
8 |  $\log F$(C iv) [erg s-1cm-2] | Broad C iv line integrated flux
9 |  $\Delta^{-}\log L$(C iv) [erg s-1] | Lower error on $\log L$(C iv)
10 |  $\Delta^{+}\log L$(C iv) [erg s-1] | Upper error on $\log L$(C iv)
11 |  $\log L_{1450}$ [erg s-1] | Monochromatic luminosity at rest-frame 1450 Å, $\lambda L_{\lambda}$(1450 Å)
12 |  $\log F_{1450}$ [erg s-1cm-2] | Monochromatic flux at rest-frame 1450 Å, $\lambda L_{\lambda}$(1450 Å)
13 |  $\Delta^{-}\log L_{1450}$ [erg s-1] | Lower error on $\log L_{1450}$
14 |  $\Delta^{+}\log L_{1450}$ [erg s-1] | Upper error on $\log L_{1450}$
15 | EW(C iv) [Å] | Rest-frame equivalent width of the broad C iv line
16 |  $\Delta^{-}$EW(C iv) [Å] | Lower error on EW(C iv)
17 |  $\Delta^{+}$EW(C iv) [Å] | Upper error on EW(C iv)
18 |  $\log L_{\rm peak}$(C iv) [erg s-1 Å-1] | Monochromatic luminosity of the broad C iv line at peak
19 |  $\log F_{\rm peak}$(C iv) [erg cm-2 s-1 Å-1] | Monochromatic flux of the broad C iv line at peak
20 |  $\Delta^{-}\log L_{\rm peak}$(C iv) [erg s-1 Å-1] | Lower error on $\log L_{\rm peak}$(C iv)
21 |  $\Delta^{+}\log L_{\rm peak}$(C iv) [erg s-1 Å-1] | Upper error on $\log L_{\rm peak}$(C iv)
22 |  $\Delta v_{\rm peak}$(bH$\beta$) [km s-1] | Velocity shift of the broad C iv line peak
23 |  $\Delta^{-}\Delta v_{\rm peak}$(C iv) [km s-1] | Lower error on $\Delta v_{\rm peak}$(C iv)
24 |  $\Delta^{+}\Delta v_{\rm peak}$(C iv) [km s-1] | Upper error on $\Delta v_{\rm peak}$(C iv)
25 |  $\Delta v_{\rm cent}$(C iv) [km s-1] | Velocity shift of the broad C iv line centroid
26 |  $\Delta^{-}\Delta v_{\rm cent}$(C iv) [km s-1] | Lower error on $\Delta v_{\rm cent}$(C iv)
27 |  $\Delta^{+}\Delta v_{\rm cent}$(C iv) [km s-1] | Upper error on $\Delta v_{\rm cent}$(C iv)
28 |  $\log M_{\rm BH}$(C iv) [$M_{\odot}$, MR16] | Broad C iv-based BH mass estimate, following Mejía-Restrepo et al. (2016)
29 |  $\Delta^{-}\log M_{\rm BH}$(C iv) [$M_{\odot}$] | Lower error on $\log M_{\rm BH}$(C iv)
30 |  $\Delta^{+}\log M_{\rm BH}$(C iv) [$M_{\odot}$] | Lower error on $\log M_{\rm BH}$(C iv)
31 |  $z$(ref) | The source redshift, as reported in the BASS/DR2 catalog (Koss et al., 2022b)
32 |  $z_{\rm corr}$(DR2, C iv) | Updated redshift, based on BLR measurements of the C iv emission line from this paper

Note. — All errors are $1\sigma$ equivalent, and were obtained obtained
through our spectral bootstrapping procedure. When both lower and upper errors
are reported, these correspond to the 16th and 84th percentiles of the
corresponding distribution. When a single error is reported, it corresponds to
the standard deviation.

##  Appendix E Broad H$\alpha$ vs. Mid-IR Emission

Figure 15 shows the broad H$\alpha$to mid-IR ratios for our sample,
$L\left({\rm b}{\text{H}}\alpha\right)/L_{12\,{\rm\mu m}}$ , vs.
FWHM(H$\alpha$)and $\log N_{\rm H}$. These serve to demonstrate that the mid-
IR emission can substitute the ultra-hard X-ray emission when deriving (or
using) the corrections presented in Section 4.5.

Figure 15: Left: $L\left({\rm b}{\text{H}}\alpha\right)/L_{12\,{\rm\mu m}}$ vs
FWHM(H$\alpha$). Right: $L\left({\rm b}{\text{H}}\alpha\right)/L_{12\,{\rm\mu
m}}$ vs $\log N_{\rm H}$ color-coded by FWHM(H$\alpha$).  Large crosses
represent the median values of $L\left({\rm
b}{\text{H}}\alpha\right)/L_{12\,{\rm\mu m}}$ after binning in FWHM(H$\alpha$)
(left panel) and $\log N_{\rm H}$ (right panel) in equally spaced quantiles.
Horizontal error-bars represent the bin edges and vertical error bars the
errors in the median $L\left({\rm b}{\text{H}}\alpha\right)/L_{12\,{\rm\mu
m}}$ estimated from bootstrapping.

##  Appendix F The broad H$\beta$ line versus column densities

In Figure 16 we show the width of the broad H$\beta$ emission line,
FWHM(H$\beta$), vs. the line-of-sight column density, $N_{\rm H}$. This figure
complements Fig. 9.

Figure 16: H$\beta$ FWHMs versus $N_{\rm H}$ and the projected distribution of
these quantities for Sy1 (light-blue), Sy1.2 (orange), Sy1.5 (light-green) and
Sy1.9 (red). Horizontal error-bars represent the bin edges and vertical error-
bars represent the errors in the median FWHM from each bin estimated from
bootstrapping. This figures complements Fig. 9, however in contrast to
FWHM(H$\alpha$), here the median FWHM(H$\beta$) does not vary with $\log
N_{\rm H}$ (within the error bars).

## References

  * Ahumada et al. (2020) Ahumada, R., Prieto, C. A., Almeida, A., et al. 2020, ApJS, 249, 3, doi: 10.3847/1538-4365/ab929e
  * Akritas & Bershady (1996) Akritas, M. G., & Bershady, M. A. 1996, ApJ, 470, 706, doi: 10.1086/177901
  * Ananna et al. (2022) Ananna, T. T., Weigel, A. K., Trakhtenbrot, B., et al. 2022, ApJS, 261, 9, doi: 10.3847/1538-4365/ac5b64
  * Anglés-Alcázar et al. (2017) Anglés-Alcázar, D., Faucher-Giguère, C.-A., Quataert, E., et al. 2017, MNRAS, 472, L109, doi: 10.1093/mnrasl/slx161
  * Asmus et al. (2015) Asmus, D., Gandhi, P., Hönig, S. F., Smette, A., & Duschl, W. J. 2015, MNRAS, 454, 766, doi: 10.1093/mnras/stv1950
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33, doi: 10.1051/0004-6361/201322068
  * Bär et al. (2019) Bär, R. E., Trakhtenbrot, B., Oh, K., et al. 2019, MNRAS, 489, 3073, doi: 10.1093/mnras/stz2309
  * Baron & Ménard (2019) Baron, D., & Ménard, B. 2019, MNRAS, 487, 3404, doi: 10.1093/mnras/stz1546
  * Baron et al. (2016) Baron, D., Stern, J., Poznanski, D., & Netzer, H. 2016, ApJ, 832, 8, doi: 10.3847/0004-637X/832/1/8
  * Batiste et al. (2017) Batiste, M., Bentz, M. C., Raimundo, S. I., Vestergaard, M., & Onken, C. A. 2017, ApJ, 838, L10, doi: 10.3847/2041-8213/aa6571
  * Baumgartner et al. (2013) Baumgartner, W. H., Tueller, J., Markwardt, C. B., et al. 2013, ApJS, 207, 19, doi: 10.1088/0067-0049/207/2/19
  * Bentz & Katz (2015) Bentz, M. C., & Katz, S. 2015, PASP, 127, 67, doi: 10.1086/679601
  * Bentz et al. (2009) Bentz, M. C., Peterson, B. M., Netzer, H., Pogge, R. W., & Vestergaard, M. 2009, ApJ, 697, 160, doi: 10.1088/0004-637X/697/1/160
  * Bentz et al. (2013) Bentz, M. C., Denney, K. D., Grier, C. J., et al. 2013, ApJ, 767, 149, doi: 10.1088/0004-637X/767/2/149
  * Bohlin et al. (1978) Bohlin, R. C., Savage, B. D., & Drake, J. F. 1978, ApJ, 224, 132, doi: 10.1086/156357
  * Bongiorno et al. (2014) Bongiorno, A., Maiolino, R., Brusa, M., et al. 2014, MNRAS, 443, 2077, doi: 10.1093/mnras/stu1248
  * Bower et al. (2017) Bower, R. G., Schaye, J., Frenk, C. S., et al. 2017, MNRAS, 465, 32, doi: 10.1093/mnras/stw2735
  * Brok et al. (2022) Brok, J. S. d., Koss, M. J., Trakhtenbrot, B., et al. 2022, ApJS, 261, 7, doi: 10.3847/1538-4365/ac5b66
  * Buchner & Bauer (2017) Buchner, J., & Bauer, F. E. 2017, MNRAS, 465, 4348, doi: 10.1093/mnras/stw2955
  * Burtscher et al. (2016) Burtscher, L., Davies, R. I., Graciá-Carpio, J., et al. 2016, A&A, 586, A28, doi: 10.1051/0004-6361/201527575
  * Cackett et al. (2015) Cackett, E. M., Gültekin, K., Bentz, M. C., et al. 2015, ApJ, 810, 86, doi: 10.1088/0004-637X/810/2/86
  * Caglar (in prep.) Caglar, T., e. a. in prep., ApJS
  * Caglar et al. (2020) Caglar, T., Burtscher, L., Brandl, B., et al. 2020, A&A, 634, A114, doi: 10.1051/0004-6361/201936321
  * Caplar et al. (2017) Caplar, N., Lilly, S. J., & Trakhtenbrot, B. 2017, ApJ, 834, 111, doi: 10.3847/1538-4357/834/2/111
  * Caplar et al. (2018) —. 2018, ApJ, 867, 148, doi: 10.3847/1538-4357/aae691
  * Cappellari et al. (2006) Cappellari, M., Bacon, R., Bureau, M., et al. 2006, in EAS Publications Series, Vol. 20, EAS Publications Series, ed. G. A. Mamon, F. Combes, C. Deffayet, & B. Fort, 127–130, doi: 10.1051/eas:2006058
  * Cardelli et al. (1989) Cardelli, J. A., Clayton, G. C., & Mathis, J. S. 1989, ApJ, 345, 245, doi: 10.1086/167900
  * Clemens et al. (2004) Clemens, J. C., Crain, J. A., & Anderson, R. 2004, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 5492, Ground-based Instrumentation for Astronomy, ed. A. F. M. Moorwood & M. Iye, 331–340, doi: 10.1117/12.550069
  * Coatman et al. (2016) Coatman, L., Hewett, P. C., Banerji, M., & Richards, G. T. 2016, MNRAS, 461, 647, doi: 10.1093/mnras/stw1360
  * Coatman et al. (2017) Coatman, L., Hewett, P. C., Banerji, M., et al. 2017, MNRAS, 465, 2120, doi: 10.1093/mnras/stw2797
  * Collin et al. (2006) Collin, S., Kawaguchi, T., Peterson, B. M., & Vestergaard, M. 2006, A&A, 456, 75, doi: 10.1051/0004-6361:20064878
  * Davies et al. (2015) Davies, R. I., Burtscher, L., Rosario, D., et al. 2015, ApJ, 806, 127, doi: 10.1088/0004-637X/806/1/127
  * Decarli et al. (2010) Decarli, R., Falomo, R., Treves, A., et al. 2010, MNRAS, 402, 2453, doi: 10.1111/j.1365-2966.2009.16049.x
  * Dong et al. (2008) Dong, X., Wang, T., Wang, J., et al. 2008, MNRAS, 383, 581, doi: 10.1111/j.1365-2966.2007.12560.x
  * Duras et al. (2020) Duras, F., Bongiorno, A., Ricci, F., et al. 2020, A&A, 636, A73, doi: 10.1051/0004-6361/201936817
  * Eracleous & Halpern (1994) Eracleous, M., & Halpern, J. P. 1994, ApJS, 90, 1, doi: 10.1086/191856
  * Fabian et al. (2009) Fabian, A. C., Vasudevan, R. V., Mushotzky, R. F., Winter, L. M., & Reynolds, C. S. 2009, MNRAS, 394, L89, doi: 10.1111/j.1745-3933.2009.00617.x
  * Falcón-Barroso et al. (2017) Falcón-Barroso, J., Lyubenova, M., van de Ven, G., et al. 2017, A&A, 597, A48, doi: 10.1051/0004-6361/201628625
  * Ferrarese & Merritt (2000) Ferrarese, L., & Merritt, D. 2000, ApJ, 539, L9, doi: 10.1086/312838
  * Fiore et al. (2009) Fiore, F., Puccetti, S., Brusa, M., et al. 2009, ApJ, 693, 447, doi: 10.1088/0004-637X/693/1/447
  * Foreman-Mackey et al. (2013) Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. 2013, PASP, 125, 306, doi: 10.1086/670067
  * Gandhi et al. (2009) Gandhi, P., Horst, H., Smette, A., et al. 2009, A&A, 502, 457, doi: 10.1051/0004-6361/200811368
  * Gaskell & Harrington (2018) Gaskell, C. M., & Harrington, P. Z. 2018, MNRAS, 478, 1660, doi: 10.1093/mnras/sty848
  * Ginsburg & Mirocha (2011) Ginsburg, A., & Mirocha, J. 2011, PySpecKit: Python Spectroscopic Toolkit, Astrophysics Source Code Library.  http://ascl.net/1109.001
  * Goad et al. (1999) Goad, M. R., Koratkar, A. P., Axon, D. J., Korista, K. T., & O’Brien, P. T. 1999, ApJ, 512, L95, doi: 10.1086/311884
  * Goad et al. (2016) Goad, M. R., Korista, K. T., De Rosa, G., et al. 2016, ApJ, 824, 11, doi: 10.3847/0004-637X/824/1/11
  * Goodrich (1989) Goodrich, R. W. 1989, ApJ, 340, 190, doi: 10.1086/167384
  * Goodrich (1990) —. 1990, ApJ, 355, 88, doi: 10.1086/168743
  * Goodrich (1995) —. 1995, ApJ, 440, 141, doi: 10.1086/175256
  * Graham (2016) Graham, A. W. 2016, in Astrophysics and Space Science Library, Vol. 418, Galactic Bulges, ed. E. Laurikainen, R. Peletier, & D. Gadotti, 263, doi: 10.1007/978-3-319-19378-6_11
  * Greene & Ho (2005) Greene, J. E., & Ho, L. C. 2005, ApJ, 630, 122, doi: 10.1086/431897
  * Greene & Ho (2007) —. 2007, ApJ, 667, 131, doi: 10.1086/520497
  * Grier et al. (2013) Grier, C. J., Peterson, B. M., Horne, K., et al. 2013, ApJ, 764, 47, doi: 10.1088/0004-637X/764/1/47
  * Grupe et al. (1999) Grupe, D., Beuermann, K., Mannheim, K., & Thomas, H. C. 1999, A&A, 350, 805.  https://arxiv.org/abs/astro-ph/9908347
  * Grupe et al. (1998) Grupe, D., Beuermann, K., Thomas, H. C., Mannheim, K., & Fink, H. H. 1998, A&A, 330, 25.  https://arxiv.org/abs/astro-ph/9710298
  * Harris et al. (2020) Harris, C. R., Millman, K. J., van der Walt, S. J., et al. 2020, Nature, 585, 357, doi: 10.1038/s41586-020-2649-2
  * Hunter (2007) Hunter, J. D. 2007, Computing in Science & Engineering, 9, 90, doi: 10.1109/MCSE.2007.55
  * Ichikawa et al. (2017) Ichikawa, K., Ricci, C., Ueda, Y., et al. 2017, ApJ, 835, 74, doi: 10.3847/1538-4357/835/1/74
  * Ichikawa et al. (2019) —. 2019, ApJ, 870, 31, doi: 10.3847/1538-4357/aaef8f
  * Ishibashi et al. (2018) Ishibashi, W., Fabian, A. C., Ricci, C., & Celotti, A. 2018, MNRAS, 479, 3335, doi: 10.1093/mnras/sty1620
  * Jaffarian & Gaskell (2020) Jaffarian, G. W., & Gaskell, C. M. 2020, MNRAS, 493, 930, doi: 10.1093/mnras/staa262
  * Jahnke et al. (2009) Jahnke, K., Bongiorno, A., Brusa, M., et al. 2009, ApJ, 706, L215, doi: 10.1088/0004-637X/706/2/L215
  * Jones et al. (2009) Jones, D. H., Read, M. A., Saunders, W., et al. 2009, MNRAS, 399, 683, doi: 10.1111/j.1365-2966.2009.15338.x
  * Jovanović et al. (2010) Jovanović, P., Popović, L. Č., Stalevski, M., & Shapovalova, A. I. 2010, ApJ, 718, 168, doi: 10.1088/0004-637X/718/1/168
  * Kaspi et al. (2005) Kaspi, S., Maoz, D., Netzer, H., et al. 2005, ApJ, 629, 61, doi: 10.1086/431275
  * Kaspi et al. (2000) Kaspi, S., Smith, P. S., Netzer, H., et al. 2000, ApJ, 533, 631, doi: 10.1086/308704
  * Kelly & Shen (2013) Kelly, B. C., & Shen, Y. 2013, ApJ, 764, 45, doi: 10.1088/0004-637X/764/1/45
  * Kim et al. (2018) Kim, D., Im, M., Canalizo, G., et al. 2018, ApJS, 238, 37, doi: 10.3847/1538-4365/aadfd5
  * King (2003) King, A. 2003, ApJ, 596, L27, doi: 10.1086/379143
  * Kormendy & Ho (2013) Kormendy, J., & Ho, L. C. 2013, ARA&A, 51, 511, doi: 10.1146/annurev-astro-082708-101811
  * Koss et al. (2017) Koss, M., Trakhtenbrot, B., Ricci, C., et al. 2017, ApJ, 850, 74, doi: 10.3847/1538-4357/aa8ec9
  * Koss et al. (2020) Koss, M. J., Strittmatter, B., Lamperti, I., et al. 2020, arXiv e-prints, arXiv:2010.15849.  https://arxiv.org/abs/2010.15849
  * Koss et al. (2022a) Koss, M. J., Trakhtenbrot, B., Ricci, C., et al. 2022a, ApJS, 261, 1, doi: 10.3847/1538-4365/ac6c8f
  * Koss et al. (2022b) Koss, M. J., Ricci, C., Trakhtenbrot, B., et al. 2022b, ApJS, 261, 2, doi: 10.3847/1538-4365/ac6c05
  * Koss et al. (2022c) Koss, M. J., Trakhtenbrot, B., Ricci, C., et al. 2022c, ApJS, 261, 6, doi: 10.3847/1538-4365/ac650b
  * Kozłowski (2017) Kozłowski, S. 2017, ApJS, 228, 9, doi: 10.3847/1538-4365/228/1/9
  * Lamperti et al. (2017) Lamperti, I., Koss, M., Trakhtenbrot, B., et al. 2017, MNRAS, 467, 540, doi: 10.1093/mnras/stx055
  * Lansbury et al. (2017) Lansbury, G. B., Alexander, D. M., Aird, J., et al. 2017, ApJ, 846, 20, doi: 10.3847/1538-4357/aa8176
  * Lapiner et al. (2021) Lapiner, S., Dekel, A., & Dubois, Y. 2021, MNRAS, 505, 172, doi: 10.1093/mnras/stab1205
  * Lawrence (1991) Lawrence, A. 1991, MNRAS, 252, 586, doi: 10.1093/mnras/252.4.586
  * Lutz et al. (2004) Lutz, D., Maiolino, R., Spoon, H. W. W., & Moorwood, A. F. M. 2004, A&A, 418, 465, doi: 10.1051/0004-6361:20035838
  * Maiolino et al. (2001a) Maiolino, R., Marconi, A., & Oliva, E. 2001a, A&A, 365, 37, doi: 10.1051/0004-6361:20000012
  * Maiolino et al. (2001b) Maiolino, R., Marconi, A., Salvati, M., et al. 2001b, A&A, 365, 28, doi: 10.1051/0004-6361:20000177
  * Maiolino & Rieke (1995) Maiolino, R., & Rieke, G. H. 1995, ApJ, 454, 95, doi: 10.1086/176468
  * Maiolino et al. (2007) Maiolino, R., Shemmer, O., Imanishi, M., et al. 2007, A&A, 468, 979, doi: 10.1051/0004-6361:20077252
  * Marconi et al. (2004) Marconi, A., Risaliti, G., Gilli, R., et al. 2004, MNRAS, 351, 169, doi: 10.1111/j.1365-2966.2004.07765.x
  * Marcotulli et al., (in prep) Marcotulli et al.,, L. in prep, ApJ
  * Marziani et al. (2015) Marziani, P., Sulentic, J. W., Negrete, C. A., et al. 2015, Ap&SS, 356, 339, doi: 10.1007/s10509-014-2136-z
  * Marziani et al. (2013) Marziani, P., Sulentic, J. W., Plauchu-Frayn, I., & del Olmo, A. 2013, A&A, 555, A89, doi: 10.1051/0004-6361/201321374
  * Mazzucchelli et al. (2017) Mazzucchelli, C., Bañados, E., Venemans, B. P., et al. 2017, ApJ, 849, 91, doi: 10.3847/1538-4357/aa9185
  * McLure & Jarvis (2002) McLure, R. J., & Jarvis, M. J. 2002, MNRAS, 337, 109, doi: 10.1046/j.1365-8711.2002.05871.x
  * Mejía-Restrepo et al. (2018a) Mejía-Restrepo, J. E., Lira, P., Netzer, H., Trakhtenbrot, B., & Capellupo, D. M. 2018a, Nature Astronomy, 2, 63, doi: 10.1038/s41550-017-0305-z
  * Mejía-Restrepo et al. (2018b) Mejía-Restrepo, J. E., Trakhtenbrot, B., Lira, P., & Netzer, H. 2018b, MNRAS, 478, 1929, doi: 10.1093/mnras/sty1086
  * Mejía-Restrepo et al. (2016) Mejía-Restrepo, J. E., Trakhtenbrot, B., Lira, P., Netzer, H., & Capellupo, D. M. 2016, MNRAS, 460, 187, doi: 10.1093/mnras/stw568
  * Merloni et al. (2010) Merloni, A., Bongiorno, A., Bolzonella, M., et al. 2010, ApJ, 708, 137, doi: 10.1088/0004-637X/708/1/137
  * Merloni et al. (2014) Merloni, A., Bongiorno, A., Brusa, M., et al. 2014, MNRAS, 437, 3550, doi: 10.1093/mnras/stt2149
  * Netzer (2019) Netzer, H. 2019, MNRAS, 488, 5185, doi: 10.1093/mnras/stz2016
  * Oh et al. (2015) Oh, K., Yi, S. K., Schawinski, K., et al. 2015, ApJS, 219, 1, doi: 10.1088/0067-0049/219/1/1
  * Oh et al. (2018) Oh, K., Koss, M., Markwardt, C. B., et al. 2018, ApJS, 235, 4, doi: 10.3847/1538-4365/aaa7fd
  * Oh et al. (2022) Oh, K., Koss, M. J., Ueda, Y., et al. 2022, ApJS, 261, 4, doi: 10.3847/1538-4365/ac5b68
  * Oke & Gunn (1982) Oke, J. B., & Gunn, J. E. 1982, PASP, 94, 586, doi: 10.1086/131027
  * Onken et al. (2004) Onken, C. A., Ferrarese, L., Merritt, D., et al. 2004, ApJ, 615, 645, doi: 10.1086/424655
  * Onori et al. (2017) Onori, F., La Franca, F., Ricci, F., et al. 2017, MNRAS, 464, 1783, doi: 10.1093/mnras/stw2368
  * Osterbrock (1981) Osterbrock, D. E. 1981, ApJ, 249, 462, doi: 10.1086/159306
  * Paliya et al. (2019) Paliya, V. S., Koss, M., Trakhtenbrot, B., et al. 2019, ApJ, 881, 154, doi: 10.3847/1538-4357/ab2f8b
  * Pancoast et al. (2014a) Pancoast, A., Brewer, B. J., & Treu, T. 2014a, MNRAS, 445, 3055, doi: 10.1093/mnras/stu1809
  * Pancoast et al. (2014b) Pancoast, A., Brewer, B. J., Treu, T., et al. 2014b, MNRAS, 445, 3073, doi: 10.1093/mnras/stu1419
  * Park et al. (2012) Park, D., Woo, J.-H., Treu, T., et al. 2012, ApJ, 747, 30, doi: 10.1088/0004-637X/747/1/30
  * Peterson (2014) Peterson, B. M. 2014, Space Sci. Rev., 183, 253, doi: 10.1007/s11214-013-9987-4
  * Popović et al. (2019) Popović, L. Č., Kovačević-Dojčinović, J., & Marčeta-Mandić, S. 2019, MNRAS, 484, 3180, doi: 10.1093/mnras/stz157
  * Price-Whelan et al. (2018) Price-Whelan, A. M., Sipőcz, B. M., Günther, H. M., et al. 2018, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Ramos Almeida & Ricci (2017) Ramos Almeida, C., & Ricci, C. 2017, Nature Astronomy, 1, 679, doi: 10.1038/s41550-017-0232-z
  * Ricci et al. (2017a) Ricci, C., Trakhtenbrot, B., Koss, M. J., et al. 2017a, ApJS, 233, 17, doi: 10.3847/1538-4365/aa96ad
  * Ricci et al. (2017b) —. 2017b, Nature, 549, 488, doi: 10.1038/nature23906
  * Ricci et al. (2017c) Ricci, F., La Franca, F., Onori, F., & Bianchi, S. 2017c, A&A, 598, A51, doi: 10.1051/0004-6361/201629380
  * Ricci et al. (2017d) Ricci, F., La Franca, F., Marconi, A., et al. 2017d, MNRAS, 471, L41, doi: 10.1093/mnrasl/slx103
  * Ricci et al. (2022) Ricci, F., Treister, E., Bauer, F. E., et al. 2022, ApJS, 261, 8, doi: 10.3847/1538-4365/ac5b67
  * Ricci & Steiner (2019) Ricci, T. V., & Steiner, J. E. 2019, MNRAS, 486, 1138, doi: 10.1093/mnras/stz919
  * Richards et al. (2002) Richards, G. T., Fan, X., Newberg, H. J., et al. 2002, AJ, 123, 2945, doi: 10.1086/340187
  * Richards et al. (2006) Richards, G. T., Lacy, M., Storrie-Lombardi, L. J., et al. 2006, ApJS, 166, 470, doi: 10.1086/506525
  * Richards et al. (2011) Richards, G. T., Kruczek, N. E., Gallagher, S. C., et al. 2011, AJ, 141, 167, doi: 10.1088/0004-6256/141/5/167
  * Riffel et al. (2006) Riffel, R., Rodríguez-Ardila, A., & Pastoriza, M. G. 2006, A&A, 457, 61, doi: 10.1051/0004-6361:20065291
  * Rojas et al. (2017) Rojas, A. F., Masetti, N., Minniti, D., et al. 2017, A&A, 602, A124, doi: 10.1051/0004-6361/201629463
  * Rojas et al. (2020) Rojas, A. F., Sani, E., Gavignaud, I., et al. 2020, MNRAS, 491, 5867, doi: 10.1093/mnras/stz3386
  * Runnoe et al. (2014) Runnoe, J. C., Brotherton, M. S., DiPompeo, M. A., & Shang, Z. 2014, MNRAS, 438, 3263, doi: 10.1093/mnras/stt2429
  * Runnoe et al. (2012) Runnoe, J. C., Brotherton, M. S., & Shang, Z. 2012, MNRAS, 422, 478, doi: 10.1111/j.1365-2966.2012.20620.x
  * Schlegel et al. (1998) Schlegel, D. J., Finkbeiner, D. P., & Davis, M. 1998, ApJ, 500, 525, doi: 10.1086/305772
  * Schulze & Wisotzki (2010) Schulze, A., & Wisotzki, L. 2010, A&A, 516, A87, doi: 10.1051/0004-6361/201014193
  * Schulze et al. (2015) Schulze, A., Bongiorno, A., Gavignaud, I., et al. 2015, MNRAS, 447, 2085, doi: 10.1093/mnras/stu2549
  * Shang et al. (2007) Shang, Z., Wills, B. J., Wills, D., & Brotherton, M. S. 2007, AJ, 134, 294, doi: 10.1086/518505
  * Shankar et al. (2016) Shankar, F., Bernardi, M., Sheth, R. K., et al. 2016, MNRAS, 460, 3119, doi: 10.1093/mnras/stw678
  * Shankar et al. (2019) Shankar, F., Bernardi, M., Richardson, K., et al. 2019, MNRAS, 485, 1278, doi: 10.1093/mnras/stz376
  * Shen (2013) Shen, Y. 2013, Bulletin of the Astronomical Society of India, 41, 61.  https://arxiv.org/abs/1302.2643
  * Shen et al. (2008) Shen, Y., Greene, J. E., Strauss, M. A., Richards, G. T., & Schneider, D. P. 2008, ApJ, 680, 169, doi: 10.1086/587475
  * Shen & Ho (2014) Shen, Y., & Ho, L. C. 2014, Nature, 513, 210, doi: 10.1038/nature13712
  * Shen & Liu (2012) Shen, Y., & Liu, X. 2012, ApJ, 753, 125, doi: 10.1088/0004-637X/753/2/125
  * Shen et al. (2019) Shen, Y., Wu, J., Jiang, L., et al. 2019, ApJ, 873, 35, doi: 10.3847/1538-4357/ab03d9
  * Shimizu et al. (2018) Shimizu, T. T., Davies, R. I., Koss, M., et al. 2018, ApJ, 856, 154, doi: 10.3847/1538-4357/aab09e
  * Silk & Rees (1998) Silk, J., & Rees, M. J. 1998, A&A, 331, L1.  https://arxiv.org/abs/astro-ph/9801013
  * Smette et al. (2015) Smette, A., Sana, H., Noll, S., et al. 2015, A&A, 576, A77, doi: 10.1051/0004-6361/201423932
  * Stern (2015) Stern, D. 2015, ApJ, 807, 129, doi: 10.1088/0004-637X/807/2/129
  * Stern & Laor (2012) Stern, J., & Laor, A. 2012, MNRAS, 426, 2703, doi: 10.1111/j.1365-2966.2012.21772.x
  * Storchi-Bergmann et al. (2017) Storchi-Bergmann, T., Schimoia, J. S., Peterson, B. M., et al. 2017, ApJ, 835, 236, doi: 10.3847/1538-4357/835/2/236
  * Suh et al. (2020) Suh, H., Civano, F., Trakhtenbrot, B., et al. 2020, ApJ, 889, 32, doi: 10.3847/1538-4357/ab5f5f
  * Trakhtenbrot & Netzer (2012) Trakhtenbrot, B., & Netzer, H. 2012, MNRAS, 427, 3081, doi: 10.1111/j.1365-2966.2012.22056.x
  * Trippe et al. (2010) Trippe, M. L., Crenshaw, D. M., Deo, R. P., et al. 2010, ApJ, 725, 1749, doi: 10.1088/0004-637X/725/2/1749
  * Vasudevan et al. (2009) Vasudevan, R. V., Mushotzky, R. F., Winter, L. M., & Fabian, A. C. 2009, MNRAS, 399, 1553, doi: 10.1111/j.1365-2966.2009.15371.x
  * Veilleux et al. (1997a) Veilleux, S., Goodrich, R. W., & Hill, G. J. 1997a, ApJ, 477, 631, doi: 10.1086/303735
  * Veilleux et al. (1997b) Veilleux, S., Sanders, D. B., & Kim, D.-C. 1997b, ApJ, 484, 92, doi: 10.1086/304337
  * Veilleux et al. (1999) Veilleux, S., Sanders, D. B., & Kim, D. C. 1999, ApJ, 522, 139, doi: 10.1086/307635
  * Veilleux & Zheng (1991) Veilleux, S., & Zheng, W. 1991, ApJ, 377, 89, doi: 10.1086/170338
  * Vernet et al. (2011) Vernet, J., Dekker, H., D’Odorico, S., et al. 2011, A&A, 536, A105, doi: 10.1051/0004-6361/201117752
  * Vestergaard & Osmer (2009a) Vestergaard, M., & Osmer, P. S. 2009a, ApJ, 699, 800, doi: 10.1088/0004-637X/699/1/800
  * Vestergaard & Osmer (2009b) —. 2009b, ApJ, 699, 800, doi: 10.1088/0004-637X/699/1/800
  * Vestergaard & Peterson (2006) Vestergaard, M., & Peterson, B. M. 2006, ApJ, 641, 689, doi: 10.1086/500572
  * Vietri et al. (2020) Vietri, G., Mainieri, V., Kakkad, D., et al. 2020, A&A, 644, A175, doi: 10.1051/0004-6361/202039136
  * Wang et al. (2009) Wang, J.-G., Dong, X.-B., Wang, T.-G., et al. 2009, ApJ, 707, 1334, doi: 10.1088/0004-637X/707/2/1334
  * Wenger et al. (2000) Wenger, M., Ochsenbein, F., Egret, D., et al. 2000, A&AS, 143, 9, doi: 10.1051/aas:2000332
  * Winkler (1992) Winkler, H. 1992, MNRAS, 257, 677, doi: 10.1093/mnras/257.4.677
  * Woo et al. (2013) Woo, J.-H., Schulze, A., Park, D., et al. 2013, ApJ, 772, 49, doi: 10.1088/0004-637X/772/1/49
  * Woo et al. (2015) Woo, J.-H., Yoon, Y., Park, S., Park, D., & Kim, S. C. 2015, ApJ, 801, 38, doi: 10.1088/0004-637X/801/1/38
  * Zheng et al. (1991) Zheng, W., Veilleux, S., & Grandi, S. A. 1991, ApJ, 381, 418, doi: 10.1086/170664
  * Ziegler & Bender (1997) Ziegler, B. L., & Bender, R. 1997, MNRAS, 291, 527, doi: 10.1093/mnras/291.3.527