# FROM SHALLOW TO DEEP: COMPOSITIONAL REASONING OVER GRAPHS FOR VISUAL
QUESTION ANSWERING

###### Abstract

In order to achieve a general visual question answering (VQA) system, it is
essential to learn to answer deeper questions that require compositional
reasoning on the image and external knowledge. Meanwhile, the reasoning
process should be explicit and explainable to understand the working mechanism
of the model. It is effortless for human but challenging for machines. In this
paper, we propose a Hierarchical Graph Neural Module Network (HGNMN) that
reasons over multi-layer graphs with neural modules to address the above
issues. Specifically, we first encode the image by multi-layer graphs from the
visual, semantic and commonsense views since the clues that support the answer
may exist in different modalities. Our model consists of several well-designed
neural modules that perform specific functions over graphs, which can be used
to conduct multi-step reasoning within and between different graphs. Compared
to existing modular networks, we extend visual reasoning from one graph to
more graphs. We can explicitly trace the reasoning process according to module
weights and graph attentions. Experiments show that our model not only
achieves state-of-the-art performance on the CRIC dataset but also obtains
explicit and explainable reasoning procedures.

Index Terms—  visual question answering, graph neural modules, compositional
reasoning, multi-layer graphs

##  1 Introduction

One of the goals of AI is to learn to “see” and “talk”, which is consistent
with Visual Question Answering (VQA) task — aiming to answer natural language
questions about an image. Most existing works [1, 2, 3] focused on “shallow”
questions which are answerable by solely referring to the visible content of
the image. For example, question Q1 in Fig.1 only requires recognizing the
color of the helmet to answer, without multi-step reasoning incorporating
external knowledge.

However, the “deeper” questions (Q2 in Fig.1) are still obstacles for these
methods. To answer this question, a desirable agent should be able to
understand the semantic of the question, perceive the visual content (e.g.
helmet,boy), incorporate commonsense knowledge (e.g. <helmet,Used For,protect
head>) and finally compositionally reasons over these clues to predict the
correct answer. Besides, it should present the decision-making process to
better understand the model’s underlying working mechanism. Therefore, how to
extend shallow visual understanding to deeper compositional reasoning and
meanwhile provide an explainable diagnosis are essential to achieve a long-
standing general VQA goal.

Most of existing VQA models [4, 5, 6, 7, 8, 9] that exhibit reasoning
capabilities can be divided into three categories. Firstly, attention-based
methods [4, 5] stack attention layer that focuses on image regions relevant to
the question. The reasoning process can be post-hoc extracted by observing
attention weights. However, it is implicit reasoning process because the
intermediate heat map cannot clarify what the current decision step is.
Secondly, memory-based methods [6, 7] perform read and write operations to
external memory module iteratively. It implements reasoning by modeling
interaction between multiple parts of data over several passes with attention
mechanism, which still faces shortcomings of implicit reasoning. Thirdly,
module-based methods [8, 9] implement reasoning procedure by parsing the
question into a layout of sub-tasks that are carried out by separate neural
sub-networks. The reasoning steps can be explicitly defined by each module,
but relying on strong layout supervision. Moreover, most existing methods
rarely utilize commonsense knowledge that is essential for deeper reasoning.

Fig. 1: Examples of shallow and deep visual questions. Fig. 2: An overview of
our model at time-step $t$. It consists of graph representations, graph
modules and module controller

In this work, we propose a Hierarchical Graph Neural Module Network (HGNMN) to
address the above problems by compositional reasoning over multi-layer graphs.
Compared with holistic features, graph can model the relationships between
objects and makes it easier for a model to reasons from one node to another
along the edges [10]. Therefore, we first encode an image by multi-layer
graphs from the visual, semantic and commonsense views, which contain the
clues supporting the answer from different modalities. Second, we decompose
the inference process into several sub-tasks based on a set of well-designed
neural modules, which perform specific functions over graphs under the
guidance of the semantic information of the question. Third, we propose a
module controller that supplies soft module weights and input query at each
reasoning step, which makes our model fully differentiable and trainable using
gradient descent without resorting to expert layouts. Experimental results on
the CRIC dataset [11] demonstrate that HGNMN has comparable performance to
existing modular networks and monolithic networks. Meanwhile, the reasoning
process can be explicitly traced via module weights and graph attention
mechanism.

##  2 Proposed Approach

In this section, we elaborate on the proposed Hierarchical Graph Neural Module
Network (HGNMN) for answering deeper visual questions. Fig. 2 shows the
overall architecture. It consists of three components: (1) Graph
Representations. Since the clues supporting answer the question may exist in
different modalities, we encode the image by constructing multi-layer graphs
from the visual, semantic and commonsense views respectively; (2) Graph
Modules, a set of well-designed neural modules, that can conduct reasoning
over hierarchical graphs; (3) Module Controller guides the execution of each
module at each reasoning step.

###  2.1 Graph Representations

Taking the inspiration from recent graph-based methods [6, 12, 13] in visual
and language tasks, we first encode the image as multi-layer graphs for
unifying the structure representation of different modalities, including
visual graph, semantic graph and commonsense graph.

Visual Graph represents visual objects and their relationships of the image
since most of the questions in VQA are visually related. We construct a fully-
connected visual graph over a set of objects identified by Faster-RCNN [14].
Each node corresponds to a detected object and encoded by visual feature. Each
edge denotes the relationships between objects, encoded by relative spatial
feature.

Semantic Graph keeps high-level abstraction of the objects and relationships
within natural language also provides essential semantic information. We first
generate captions of the image with DenseCap [15]. Then we construct a
semantic graph using SPICE [16], where node represents the name or attribute
of an object and edge represents the relationship between them. We use the
averaged GloVe [17] embeddings to represent nodes and edges.

Commonsense Graph contains knowledge related to the image and question that is
used for answering open-domain visual questions. We use ConceptNet [18] as
original knowledge base, which can be seen as a large set of triples of the
form $(h,r,t)$, where $h$ and $t$ represent head and tail entities, $r$
represents relationship between them. We select 10 types of relations to
reduce the computation cost following [11]. A desirable knowledge retrieval
should include most of the useful information while ignore the irrelevant
ones. To achieve this goal, we first use labels of visual objects as keys to
extract corresponding entities of ConceptNet. Then we retrieve the first-order
subgraph using these selected nodes from ConceptNet, which includes all edges
connecting with at least one candidate node, i.e. the label is either $h$ or
$t$. However, the subgraph contains much irrelevant information. Each object
and extracted triplet is associated with probability scores, denoted as
$S_{l}$ and $S_{t}$. We assign $a\cdot S_{l}+b\cdot S_{t}$ as the final score
of the edge, where $a$ and $b$ are hyper-parameters that used to adjust the
importance of the $S_{l}$ and $S_{t}$. Then we obtain the commonsense graph by
ranking and selecting top-K edges along with the nodes according to scores.
All nodes and edges are encoded by averaged GloVe embeddings[17].

###  2.2 Graph Modules

We first define the Find, And, Filter modules to attend objects that are
relevant to some subjects, attributes or logic information of the question
following [9, 19]. However, deeper questions commonly cover relationships
(e.g. geometric positions or semantic interactions between objects) beyond
mere object detection, so we propose the Relate module to transfer node
attention maps along the edge in one graph guided by relation information of
the question. Moreover, deeper questions require collecting evidence from
different modalities, so it is necessary to associate multi-layer graphs. We
additionally propose the CrossGraph module to extend the reasoning in one
graph to more graphs. The Describe module is proposed to obtain the graph
features because the attentive node maps need to be transformed to an
embedding to predict the answer. Due to some questions may not require
complete $T$-step reasoning, we also introduce a NoOp module proposed by [9],
which can be used to pad reasoning steps to maximum length $T$. We will
describe each kind of module in detail below. Modules of the same type only
differ in inputs and parameters.

Find [$\mathbf{X},\mathbf{c}$] is proposed to attend the relevant objects
given the input query and outputs an attention map
$\mathbf{a}\in\mathbb{R}^{n}$ over $n$ graph nodes. We first transform the
input query $\mathbf{c}$ and node features $\mathbf{X}$ into the same
dimensions and then fuse them together to pass a MLP:

$\mathbf{a}=\text{softmax}(f_{mlp}(F(W_{1}\mathbf{X},W_{2}\mathbf{c}))),$ (1)

where $W_{1},W_{2}$ are weight matrices (as well as $W_{3},\dots,W_{10}$
mentioned below),
$F(\mathbf{x},\mathbf{y})=\text{ReLU}(\mathbf{x}+\mathbf{y})-(\mathbf{x}-\mathbf{y})^{2}$
is multimodal fusion function proposed in [19].

And [$\mathbf{a}_{1},\mathbf{a}_{2}$] aims to combine attention maps
$\mathbf{a}_{1},\mathbf{a}_{2}$ generated from previous reasoning steps. We
implement it by adding two input attention weights, i.e.
$\mathbf{a}=\mathbf{a_{1}}\oplus\mathbf{a_{2}}$, where $\oplus$ is element-
wise addition.

Filter [$\mathbf{a},\mathbf{c}$] is proposed to find nodes relevant to the
input query on the basis of the output from the previous step, which is used
to further locate objects according to new input information. We implement it
based on Find module, i.e.
$\mathbf{a}=\mathbf{a}\oplus\text{Find}(\mathbf{c})$, where $\oplus$ is
element-wise addition.

Relate [$\mathbf{a},\mathbf{E},\mathbf{c}$]. The deeper questions mostly
involve the interaction between objects. Therefore it is essential to transfer
the current node to adjacent nodes via attended edge. We first find the
relevant edges given the input query. The attention weight
$W_{ij}\in\mathbf{W}^{n\times n}$ of edge $e_{ij}$ is computed by:

$W_{ij}=\text{ReLU}(f_{mlp}(W_{3}\mathbf{c}\odot W_{4}\mathbf{e}_{ij})),$ (2)

where $\mathbf{e}_{ij}\in\mathbf{E}$ is edge embedding. Thanks to graph
structure, we can transfer the node weights along the attentive relations to
update attention weights by matrix multiplication:
$\mathbf{a}=\text{norm}(\mathbf{W}^{T}\mathbf{a})$, where $\text{norm}(\cdot)$
is normalization operation.

$\text{CrossGraph}_{G_{m}\to
G_{n}}$[$\mathbf{a}_{m},\mathbf{a}_{n},\mathbf{X}_{m},\mathbf{X}_{n},\mathbf{c}$].
Reasoning within one graph is not enough to answer deep questions. It is
essential to associate nodes of different graphs to extend the reasoning
process from one graph to multi-layer graphs. We achieve this goal by
computing attention map over nodes of graph $G_{n}$ guided by the node
features of graph $G_{m}$:

$\displaystyle\mathbf{a^{\prime}}_{n}(i)=\text{softmax}($ $\displaystyle
W_{7}(\tanh(W_{5}\mathbf{X}^{T}_{m}\mathbf{a}_{m}+W_{6}\mathbf{X}_{n}(i))),$
(3) $\displaystyle\mathbf{a}_{n}=\mathbf{a^{\prime}}_{n}+\mathbf{a}_{n},$ (4)

where $\mathbf{X}_{n}(i)$ is node feature of the $i$-th node of $G_{n}$.

Describe [$\mathbf{a},\mathbf{X}$] aims to transform the attentive node
features to an embedding that summarize the entire graph by computing a
weighted sum of node features guided by the input attention, i.e.
$\mathbf{y}=\mathbf{X}^{T}\mathbf{a}$.

NoOp [a] just outputs the input $\mathbf{a}$ without transforming, which is
used to pad reasoning steps to a maximum length $T$.

###  2.3 Module Controller

To make the network fully differentiable, we draw inspire from [9] to propose
a module controller to make soft layout selection via a soft module weights
distribution and supplies input query at each reasoning step $t$. Then we
execute all modules of each graph and average the outputs according to the
module weights as the input for the next step. We take one of the graphs as an
example to describe the procedure in detail below.

Specifically, we first encode the question by LSTM to get all word embeddings
$\\{\mathbf{h}_{l}\\}_{l=1}^{L}$ and question embedding $\mathbf{q}$. At each
time-step $t$, we generate intermediate embedding $\mathbf{u}$ based on
question embedding and input query of previous step, i.e
$\mathbf{u}=f_{mlp}([\mathbf{q};\mathbf{c}_{t-1}])$. The input query should
include question information that is more relevant to the current reasoning
step. Therefore we generate input query by textual attention over words guided
by intermediate embedding:

$\alpha_{l}=\text{softmax}(f_{mlp}(\mathbf{u}\odot\mathbf{h}_{l}))$ (5)
$\mathbf{c}_{t}=\sum_{l=1}^{L}\alpha_{l}\cdot\mathbf{h}_{l},$ (6)

where $\odot$ is element-wise multiplication. To select which module is more
important at the current step, we predict soft module weights which resemble a
probability distribution over all the modules using MLP:

$\mathbf{w}^{(t)}=\text{softmax}(f_{mlp}(\mathbf{u}))$ (7)

We execute all the modules and perform a weighted average of their outputs
with respect to the module weights:

$\mathbf{a}^{(t)}=\sum_{m\in M}w_{m}^{(t)}\cdot\mathbf{a}^{(t)}_{m},$ (8)

where $w_{m}^{(t)}\in\mathbf{w}^{(t)}$, $M$ is module list of each graph and
$\mathbf{a}^{(t)}_{m}$ is the output attention map of $m$-th module. At the
final step, we extract graph features from Describe modules of different
graphs and fuse them to predict the answer using MLP:

$Ans=f_{mlp}([W_{8}\mathbf{y}_{G_{1}},W_{9}\mathbf{y}_{G_{2}},W_{10}\mathbf{y}_{G_{3}},W_{11}\mathbf{q}])$
(9)

Model | Expert layout | Acc
---|---|---
Q-Type | No | 25.96
Q-Only | No | 39.40
I-Only | No | 14.18
Q+I | No | 48.47
ButtomUp [20] | No | 52.26
RVC-w/o-KG [11] | Yes | 54.68
RVC-$l_{ans}$ [11] | No | 51.20
RVC [11] | Yes | 58.38
HGNMN (full) | No | 60.32

Table 1: Results on test set of CRIC dataset.

##  3 Experiments

###  3.1 Experimental Settings

Dataset. We evaluated our model on the CRIC [11], a large VQA dataset contains
1,303,271 deeper questions than other common datasets, which are randomly
split into train (60%), validation (20%) and test (20%). It also provides
expert layout annotations for each question.

Implementation Details. For each image, we extract 36 objects along with
features, labels and probability scores from Faster-RCNN [14]. We select the
top-10 captions and set $a=0.7,b=0.3$ in the graph construction. We truncate
or pad the length of question to 20 words. The max reasoning steps $T$ is set
to 12. We adopt cross entropy loss to train the model by Adam optimizer with
$0.001$ learning rate.

###  3.2 Experimental Results

Comparison with SOTA Methods. In this section, we evaluate the performance of
following methods on the CRIC: (1) Q-Type, Q-Only, I-Only and Q+I are four
basic baselines proposed in [11]. (2) BottomUp [20] implements soft attention
on object regions and combines the attended image features and question
features to predict the answer. (3) RVC [11] builds upon neural module
networks with supervision of expert layout. (4) RVC-w/o-KG [11] is a variation
of RVC that doesn’t use the knowledge graph to answer the question. (5)
RVC-$l_{ans}$ [11] is a variation of RVC that can be trained without
supervision of expert layout.

The results are summarized in Table 1. Our model achieves a new SOTA result.
In particular, although RVC is trained with expert layout, our model still
outperforms RVC by 1.94%. Compared with RVC-$l_{ans}$ that also does not
require expert layout, our model improves by 9.12%, which proves the
effectiveness of reasoning on multi-layer graphs.

Ablation Study. We conduct ablation studies to further investigate the key
components of HGNMN. (1) We remove the visual, semantic and commonsense graph
from full model respectively to analyze the contributions of each layer of
graphs. The results shown in Table 3 all decrease. Thereinto, the commonsense
graph is most important since most of the questions rely on external
knowledge. (2) We further evaluate the effectiveness of different neural
modules in Table 3. We remove the And, Filter, Relate and CrossGraph modules
respectively. The Find, Describe and NoOp modules are kept because answering
any questions at least requires these operations. We find that the Relate and
CrossGraph modules are more important than others, since they can extend
reasoning along edges and associate multiple graphs which are essential for
multi-step reasoning.

Model | Acc
---|---
w/o VG | 57.63
w/o SG | 59.11
w/o KG | 55.25

Table 2: Ablation study of hierarchical graphs

Model | Acc
---|---
w/o And | 59.12
w/o Filter | 58.47
w/o Relate | 57.39
w/o CrossGraph | 57.14

Table 3: Ablation study of different modules

Fig. 3: Visualization of intermediate reasoning steps.

Qualitative Evaluation Fig. 3 shows the qualitative evaluation of one example.
We visualize the results of the first 7 steps and omit the rest NoOp
operations. At each reasoning step, we mark the module with the largest weight
in red and the most attentive word of the question in blue and present the
attention map of corresponding graph. It shows that our model can extract
explicit and explainable reasoning process. For example, it locates the “black
object” in VG via Filter at the 2nd step; finds the most relevant object in SG
via CrossGraph at the 3rd step; finds the nodes related to “car” in KG via
Relate at the 6th step.

##  4 Conclusion

In this paper, we propose a Hierarchical Graph Neural Module Network (HGNMN)
for answering deeper questions of VQA. We encode the image by multi-layer
graphs from different views and define a set of graph-based neural modules to
extend reasoning from single graph to more graphs. Moreover, it can be trained
end-to-end without expert layout supervision. Experimental results on CRIC
dataset show that HGNMN outperforms state-of-the-art approaches. Our model is
more interpretable by observing its intermediate outputs.

## References

  * [1] Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller, and Xinlei Chen,  “In defense of grid features for visual question answering,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 10267–10276.
  * [2] Hedi Ben-Younes, Remi Cadene, Nicolas Thome, and Matthieu Cord,  “Block: Bilinear superdiagonal fusion for visual question answering and visual relationship detection,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2019, vol. 33, pp. 8102–8109.
  * [3] Tuong Do, Thanh-Toan Do, Huy Tran, Erman Tjiputra, and Quang D Tran,  “Compact trilinear interaction for visual question answering,”  in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 392–401.
  * [4] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola,  “Stacked attention networks for image question answering,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 21–29.
  * [5] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian,  “Deep modular co-attention networks for visual question answering,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2019, pp. 6281–6290.
  * [6] Jing Yu, Zihao Zhu, Yujing Wang, Weifeng Zhang, Yue Hu, and Jianlong Tan,  “Cross-modal knowledge reasoning for knowledge-based visual question answering,”  Pattern Recognition, vol. 108, pp. 107563, 2020.
  * [7] Chao Ma, Chunhua Shen, Anthony Dick, Qi Wu, Peng Wang, Anton van den Hengel, and Ian Reid,  “Visual question answering with memory-augmented networks,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 6975–6984.
  * [8] Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng, William Wang, and Jingjing Liu,  “Meta module network for compositional visual reasoning,”  arXiv preprint arXiv:1910.03230, 2019.
  * [9] Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko,  “Explainable neural computation via stack neural module networks,”  in Proceedings of the European conference on computer vision, 2018, pp. 53–69.
  * [10] Sibei Yang, Guanbin Li, and Yizhou Yu,  “Graph-structured referring expression reasoning in the wild,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 9952–9961.
  * [11] Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen,  “From two graphs to n questions: A vqa dataset for compositional reasoning on vision and commonsense,”  arXiv preprint arXiv:1908.02962, 2019.
  * [12] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu,  “Relation-aware graph attention network for visual question answering,”  in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 10313–10322.
  * [13] Zihao Zhu, Jing Yu, Yujing Wang, Yajing Sun, Yue Hu, and Qi Wu,  “Mucko: Multi-layer cross-modal knowledge reasoning for fact-based visual question answering,”  in Proceedings of the International Joint Conference on Artificial Intelligence, 2020, pp. 1097–1103.
  * [14] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun,  “Faster r-cnn: Towards real-time object detection with region proposal networks,”  in Advances in neural information processing systems, 2015, pp. 91–99.
  * [15] Justin Johnson, Andrej Karpathy, and Li Fei-Fei,  “Densecap: Fully convolutional localization networks for dense captioning,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4565–4574.
  * [16] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould,  “Spice: Semantic propositional image caption evaluation,”  in European Conference on Computer Vision. Springer, 2016, pp. 382–398.
  * [17] Jeffrey Pennington, Richard Socher, and Christopher D Manning,  “Glove: Global vectors for word representation,”  in Proceedings of the 2014 conference on empirical methods in natural language processing, 2014, pp. 1532–1543.
  * [18] Robyn Speer, Joshua Chin, and Catherine Havasi,  “Conceptnet 5.5: an open multilingual graph of general knowledge,”  in Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017, pp. 4444–4451.
  * [19] Jiaxin Shi, Hanwang Zhang, and Juanzi Li,  “Explainable and explicit visual reasoning over scene graphs,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 8376–8384.
  * [20] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang,  “Bottom-up and top-down attention for image captioning and visual question answering,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 6077–6086.


# Template AASTeXArticle with Examples: v6.3111Released on June, 10th, 2019

Joshua Y.-Y. Lin Department of Physics, University of Illinois, 1110 West
Green Street, Urbana, IL 61801, USA  Illinois Center for Advanced Study of the
Universe, 1110 West Green Street, Urbana, IL 61801, USA  National Center for
Supercomputing Applications, 605 East Springfield Avenue, Champaign, IL 61820,
USA  Center for Computational Astrophysics, Flatiron Institute, Simons
Foundation, 162 5th Ave, New York, NY 10010, USA  Dominic W. Pesce Center for
Astrophysics $\;|\;$ Harvard $\,\&\,$ Smithsonian, 60 Garden Street,
Cambridge, MA 02138, USA  Black Hole Initiative at Harvard University, 20
Garden Street, Cambridge, MA 02138, USA  George N. Wong Department of Physics,
University of Illinois, 1110 West Green Street, Urbana, IL 61801, USA
Illinois Center for Advanced Study of the Universe, 1110 West Green Street,
Urbana, IL 61801, USA  School of Natural Sciences, Institute for Advanced
Study, 1 Einstein Drive, Princeton, NJ 08540, USA  Princeton Gravity
Initiative, Princeton University, Princeton, New Jersey 08544, USA  Ajay
Uppili Arasanipalai Department of Physics, University of Illinois, 1110 West
Green Street, Urbana, IL 61801, USA  Ben S. Prather Department of Physics,
University of Illinois, 1110 West Green Street, Urbana, IL 61801, USA
Illinois Center for Advanced Study of the Universe, 1110 West Green Street,
Urbana, IL 61801, USA  Charles F. Gammie Department of Astronomy, University
of Illinois, 1002 West Green Street, Urbana, IL 61801, USA  Illinois Center
for Advanced Study of the Universe, 1110 West Green Street, Urbana, IL 61801,
USA  National Center for Supercomputing Applications, 605 East Springfield
Avenue, Champaign, IL 61820, USA  Department of Physics, University of
Illinois, 1110 West Green Street, Urbana, IL 61801, USA

# VLBInet: Radio Interferometry Data Classification for EHT with Neural
Networks

Joshua Y.-Y. Lin Department of Physics, University of Illinois, 1110 West
Green Street, Urbana, IL 61801, USA  Illinois Center for Advanced Study of the
Universe, 1110 West Green Street, Urbana, IL 61801, USA  National Center for
Supercomputing Applications, 605 East Springfield Avenue, Champaign, IL 61820,
USA  Center for Computational Astrophysics, Flatiron Institute, Simons
Foundation, 162 5th Ave, New York, NY 10010, USA  Dominic W. Pesce Center for
Astrophysics $\;|\;$ Harvard $\,\&\,$ Smithsonian, 60 Garden Street,
Cambridge, MA 02138, USA  Black Hole Initiative at Harvard University, 20
Garden Street, Cambridge, MA 02138, USA  George N. Wong Department of Physics,
University of Illinois, 1110 West Green Street, Urbana, IL 61801, USA
Illinois Center for Advanced Study of the Universe, 1110 West Green Street,
Urbana, IL 61801, USA  School of Natural Sciences, Institute for Advanced
Study, 1 Einstein Drive, Princeton, NJ 08540, USA  Princeton Gravity
Initiative, Princeton University, Princeton, New Jersey 08544, USA  Ajay
Uppili Arasanipalai Department of Physics, University of Illinois, 1110 West
Green Street, Urbana, IL 61801, USA  Ben S. Prather Department of Physics,
University of Illinois, 1110 West Green Street, Urbana, IL 61801, USA
Illinois Center for Advanced Study of the Universe, 1110 West Green Street,
Urbana, IL 61801, USA  Charles F. Gammie Department of Astronomy, University
of Illinois, 1002 West Green Street, Urbana, IL 61801, USA  Illinois Center
for Advanced Study of the Universe, 1110 West Green Street, Urbana, IL 61801,
USA  National Center for Supercomputing Applications, 605 East Springfield
Avenue, Champaign, IL 61820, USA  Department of Physics, University of
Illinois, 1110 West Green Street, Urbana, IL 61801, USA

###### Abstract

The Event Horizon Telescope (EHT) recently released the first horizon-scale
images of the black hole in M87. Combined with other astronomical data, these
images constrain the mass and spin of the hole as well as the accretion rate
and magnetic flux trapped on the hole. An important question for the EHT is
how well key parameters, such as trapped magnetic flux and the associated disk
models, can be extracted from present and future EHT VLBI data products. The
process of modeling visibilities and analyzing them is complicated by the fact
that the data are sparsely sampled in the Fourier domain while most of the
theory/simulation is constructed in the image domain. Here we propose a data-
driven approach to analyze complex visibilities and closure quantities for
radio interferometric data with neural networks. Using mock interferometric
data, we show that our neural networks are able to infer the accretion state
as either high magnetic flux (MAD) or low magnetic flux (SANE), suggesting
that it is possible to perform parameter extraction directly in the visibility
domain without image reconstruction. We have applied VLBInet to real M87 EHT
data taken on four different days in 2017 (April 5, 6, 10, 11), and our neural
networks give a score prediction $0.52,0.4,0.43,0.76$ for each day, with an
average score $0.53$, which shows no significant indication for the data to
lean toward either the MAD or SANE state.

††journal: ApJ

##  1 Introduction

The Event Horizon Telescope (EHT) is a globe-spanning network of millimeter
wavelength observatories (; ). Data from individual observatories are combined
to measure the Fourier components of a source intensity on the sky (EHTC III).
The sparse set of Fourier components together with a regularization procedure
can then be used to reconstruct an image of the source (EHTC IV). The
resulting images of the black hole at the center of M87 (hereafter M87*) have
a ringlike structure—attributed to emission from hot plasma surrounding the
black hole—with an asymmetry that contains information about the motion of
plasma around the hole (; ). Combined with data from other sources, the EHT
images constrain the black hole spin and mass as well as properties of the
surrounding magnetic field structure and strength (; ). More recently, the EHT
has produced polarized emission maps of M87* (EHTC VII), and these imply a
highly organized magnetic field structure in the source (EHTC VIII).

Black hole accretion flows can be divided into two qualitatively different
states, called MAD and SANE, depending on the strength of the magnetic field
near the event horizon. In the magnetically arrested disk (MAD) state,
magnetic fields near the horizon are dynamically important enough to limit the
flow of plasma onto the hole (Bisnovatyi-Kogan & Ruzmaikin, 1974; Ichimaru,
1977; Igumenshchev et al., 2003; Narayan et al., 2003). When MAD, the magnetic
flux $\Phi_{B}$ through the black hole horizon is at the maximum value
sustainable by the accretion flow; additional flux escapes outward through the
inflowing plasma. The flux can be characterized by the dimensionless flux
$\phi=\Phi_{B}/\sqrt{\dot{M}R_{g}^{2}\,c}$ ($\dot{M}\equiv$ accretion rate;
$R_{g}\equiv GM/c^{2}$). In the MAD state $\phi\simeq 15$.222We use
Lorentz–Heaviside units rather than Gaussian units to be consistent with Porth
2019. In the latter system, $\phi_{c}\approx 50$. In the standard and normal
evolution (SANE) state, by contrast, $\phi<\phi_{c}$ and the flow is organized
in a conventional, geometrically thick, centrifugally supported disk. For a
fixed accretion rate and black hole spin, MAD flows have stronger jets driven
by the Blandford–Znajek effect. Is it possible to identify whether a source is
in the MAD or SANE state directly from VLBI data?

Various methods have been proposed for using black hole images or VLBI data to
constrain system parameters—see (Johnson et al., 2020; Gralla & Lupsasca,
2020; Lupsasca et al., 2018) for measuring black hole parameters using long-
baseline visibilities and (Wong, 2021; Hadar et al., 2021; Chesler et al.,
2021) through autocorrelations of position-dependent light echoes—however, the
general parameter extraction problem is challenging because simulated images
contain an abundance of information about not only the black hole spacetime
and gross accretion flow properties, but also about fluctuating, turbulent
structure in the accreting plasma. Past efforts to compare simulated GRMHD
images with VLBI data have typically either fit to the visibilities
constructed from simulated images (e.g., the “average image scoring” procedure
from EHTC V, which runs into issues of attempting to match or marginalize over
specific instantiations of turbulence between simulation and observation) or
else have focused on observationally accessible proxies for physical
quantities (e.g., the metric derived in Palumbo et al. 2020 and used in EHTC
VIII). Here we consider a neural network (NN)-based approach to parameter
extraction from VLBI data. This approach has the benefit of using all the data
(as opposed to observationally accessible proxies) and can ideally distill
from the data all information that constrains a model parameter.

Earlier work by van der Gucht et al. (2020) found evidence that neural
networks could be used to infer values for a limited set of parameters in a
library of synthetic black hole images of SANE flows. Work by Lin et al.
(2020a) explored both MAD and SANE models and investigated neural network
interpretability. Both these earlier efforts worked in the image domain,
though the real data correspond to sparse sampling in the Fourier domain. In
recent years, work by Sun & Bouman (2020); Sun et al. (2020); Morningstar et
al. (2018) have shown that deep neural networks can be used for visibility-to-
image reconstruction, as well as tracking their underlying distribution in a
probabilistic way using certain evaluation loss metrics. Work by Popov et al.
(2021) used convolutional neural networks to estimate an inclination angle
from complex visibility with synthetic disk-based model. Here we explore the
possibility of using neural networks to perform an end-to-end analysis of
synthetic M87* data in the visibility domain.

Our paper is organized as follows. In Section 2, we describe our procedure for
generation of synthetic observations. In Section 3 we describe our neural
network, and we present results in Section 4 and provide a discussion in
Section 5.

##  2 Synthetic data generation

Figure 1: Example synthetic images based on numerical models of black hole
accretion flows with M87-like parameters. Top row: sample images of MAD
accretion flows. Bottom row: sample images of SANE flows.

###  2.1 Simulating black hole images with GRMHD/GRRT

The training data for our supervised machine learning pipeline consist of
synthetic images based on general relativistic magnetohydrodynamic (GRMHD)
simulations of black hole accretion. The image dataset used in this work is
the same as the one used in (Lin et al., 2020a). We obtained $40000$ images
from the image library, and split them into half for training and testing.

The GRMHD simulations were generated with the iharm3D code (Prather et al.
JOSS, submitted) for five black hole spins ($a_{*}=-0.94,-0.5,0,0.5,0.94$) in
both MAD and SANE states. Notice that there is a continuum of possible SANE
states; our SANE models have $\phi\simeq 1$. For selected spins and magnetic
states we consider multiple GRMHD simulations initialized with different seed
perturbations to assess the likelihood that our neural networks are over-
fitting the simulations.

We then imaged the GRMHD simulation output using ipole (Mościbrodzka & Gammie,
2018). To do this we must specify additional physical parameters: the
approximate size of the black hole and its distance from the observer; the
accretion rate (set so that the intensity integrated over the images matches
the mm flux density of M87*); and a parameter $R_{\mathrm{high}}$ that
regulates assignment of the electron temperature from fluid variables (see
EHTC V). $R_{\mathrm{high}}$ is drawn uniformly from a discrete set of allowed
values $[1,10,40,160]$. In addition we must specify parameters related to the
imaging process: the position angle (PA) of the source, the angular resolution
of the image, and the coordinates of the black hole center on the image. To
prevent the neural network from correlating imaging parameters with source
parameters we introduce image-by-image variations: the PA is drawn uniformly
from $[0,2\pi)$; the angular scale is drawn uniformly from a small interval
corresponding to a $\pm 10\%$ change in angular scale of the source
(equivalently a 10% range in black hole mass or distance); the image center is
drawn uniformly from a square region corresponding to a $[-10,10]$ $\mu$as
displacement in each coordinate.

Figure 2: Visibility amplitude versus baseline length for a randomly-selected
single training image with different noise components applied. Panel (a) shows
the noise-free Fourier samples, panel (b) shows these samples after the
addition of complex Gaussian noise, and panel (c) shows the same as (b) after
additionally modulating each visibility by station-based complex gain noise
(see subsection 2.2)

###  2.2 Synthetic Data Generation

For each image in the training and testing sets, we generated synthetic VLBI
observations using eht-imaging (Chael et al., 2016, 2018) in a manner similar
to that described in EHTC IV. The baseline coverage and observing cadence
match the 2017 April 11 observations of M87 carried out by the EHT after
coherently time-averaging each baseline’s complex visibilities on a per-scan
basis (; ). The source image in each case is assumed to be located at the sky
position of the M87 black hole, and the baseline thermal noise values have
been taken from the original calibrated observations provided by the EHT
collaboration (Event Horizon Telescope Collaboration, 2019f).

In the absence of noise the complex visibilities are, according to the van
Cittert–Zernike theorem (Thompson et al., 2017), samples of the Fourier
transform of the sky emission. Real-world telescope arrays suffer from a
number of a priori unknown systematics, however, the most severe of which are
variations in the phase and amplitude of the signal received at each station
caused by atmospheric turbulence and other signal path effects.

We have thus generated three classes of mock observations for each of the
training and testing images: (1) a “noise free” observation that contains only
the Fourier transform of the input image sampled on the EHT baselines, (2) a
“thermal noise” observation in which each complex visibility has been
modulated according to its measurement uncertainty, and (3) a “full noise”
observation that includes station-based complex gain fluctuations. The
synthetic gain amplitudes are modeled as Gaussian-distributed with unit mean
and a standard deviation of 10% for all stations, while the synthetic gain
phases are drawn uniformly from $[0,2\pi)$; both gain amplitudes and phases
are sampled independently for each station at each timestamp.

Gain variations are particularly severe at the short ($\sim$1.3 mm) observing
wavelength of the EHT, motivating the construction and use of so-called
“closure” data products that are immune to station-based corruption. The two
standard sets of closure quantities are closure phases and closure amplitudes,
which are constructed using triangles and quadrangles of baselines,
respectively. Denoting a visibility measurement on the baseline between
stations $i$ and $j$ as $V_{ij}$, the closure phase on triangle $ijk$ is given
by the argument of the directed triple product,

$\psi_{ijk}={\rm arg}\left(V_{ij}V_{jk}V_{ki}\right).$ (1)

The closure amplitude on a quadrangle $ijk\ell$ is given by

$A_{ijk\ell}=\left|\frac{V_{ij}V_{kl}}{V_{i\ell}V_{jk}}\right|.$ (2)

Both closure phases and closure amplitudes have the property that station-
based noise cancels, making them robust observables. In this paper we
construct two classes of neural networks, one to treat complex visibilities
and the other to treat closure quantities.

Figure 3: VLBInet Pipeline.

##  3 Neural networks

|  |
---|---|---
Figure 4: Two neural network classes used in this paper. The first accepts
complex visibilities as input, and the second accepts closure quantities as
input.

We decided to use fully connected neural networks, also known as multilayer
perceptrons (MLPs), for the MAD/SANE classification task. Previous work used
convolutional NNs (CNNs) and image domain data (van der Gucht et al., 2020;
Lin et al., 2020a). We use sparsely sampled visibility domain data—where the
input is not translation invariant—and therefore selected the MLP architecture
as one of the simplest architectures that avoids possible inductive biases
associated with the translational invariance embedded in, for example, CNNs
(Battaglia et al., 2018). We will refer to this arrangement as VLBInet.

When working with complex visibilities, we decompose the complex numbers into
amplitudes and phases and then project the phases into sines and cosines to
enable phase wrapping. For both complex visibilities and closure quantities
the data are organized in a consistent way so that each neuron in the input
layer is associated with a particular point in the uv domain (complex
visibilities) or triangle or quadrangle (closure quantities).

We trained 6 different versions of VLBInet, ranging over 2 different
visibility inputs (complex/closure) and 3 different noise scenarios. All
versions consist of 7 layers. For complex visibilities, the numbers of neurons
at each layer are $648,256,64,64,64,64,1$ and the full model has $195,137$
trainable parameters. For closure quantities the numbers of neurons at each
layer are $381,256,64,64,64,64,1$ and the model has $126,785$ trainable
parameters. In MLPs each connection contains a weight and a bias. We use
$\tanh$ as the activation function. We use binary cross-entropy (BCE) loss for
the MAD/SANE since the output is expected to be in one of these two
categories: $\text{BCELoss}=t\log(p)+(1-t)\log(1-p)$, where $t=0$ for MAD and
$t=1$ for SANE. Here p is the score prediction with range from $0-1$. VLBInet
is built using the Python 3 deep learning library _pytorch_ (Paszke et al.,
2019).

##  4 Results

###  4.1 Results on synthetic data

|  |
---|---|---
Figure 5:  Histogram of scores for MAD and SANE models. If the neural network
could identify the model perfectly then all MADs would score 0 and all SANEs
would score 1. The grey dashed line is the decision boundary: data with score
$>$ 0.5 are classified as SANE model, and $<$ 0.5 are classified as MAD model.

Figure 5 shows the distribution of MAD/SANE scores for 20,000 samples drawn
with uniform probability from the test set. Notice that the vertical axis is
logarithmic. Evidently the data contain information about whether the source
is MAD or SANE, and this information can be detected by the NN. Figure 6 shows
the performance of our VLBInet with different noise models and the form of
VLBI data (complex visibilities/ closure quantities) are given, and we
calculate their area under curve (AUC) (Bradley, 1997).

|  |
---|---|---
Figure 6:  We calculate the area under Receiver operating characteristic (ROC)
curve as a performance measure for different noise models along with complex
visibilities/ closure quantities. Here we treat SANE model as positive.

To quantify how well the NN performs in the classification task, we split the
data into 20 subsets and calculate the mean and standard deviation of the
prediction across the subsets. The resulting uncertainty estimates are shown
in Table 1.

For complex visibilities it is evident that almost no information is lost even
when thermal noise is added; the real loss of information is due to gain
corruption. By contrast, the closure quantities lose information via thermal
noise but lose nothing through gain corruption. This is unsurprising since
closure quantities are unaffected by gain errors.

For complex visibility, the MAD/SANE classifier network is able to correctly
identify MAD/SANE with $79.67\%\pm 1.13\%$ accuracy without noise, $79.13\%\pm
1.25\%$ accuracy with thermal noise presence and achieve $70.99\%\pm 1.59\%$
accuracy with complex visibility, thermal noise + amplitude gain + phase
error. For closure visibility, the MAD/SANE classifier network is able to
correctly identify MAD/SANE with $76.57\%\pm 1.31\%$ accuracy without noise,
$73.55\%\pm 1.14\%$ accuracy with thermal noise presence and achieve
$73.24\%\pm 1.57\%$ accuracy with complex visibility, thermal noise +
amplitude gain + phase error.

###  4.2 Results on real M87* data

We ran the trained VLBInet (trained on models with added noise and using
closure quantities) on real M87 VLBI data taken on April 5th, 6th, 10th, and
11th, 2017. VLBInet output scores of $0.52$, $0.4$, $0.43$, $0.76$,
respectively. Among the 20,000 data in our test set, there are $140$ instances
that are similar to the prediction scores (within $0.142$, one sigma from the
score distribution for 4 days). Among the $140$ that are similar, $47\%$ of
them are “MAD”.

Figure 7: Visibility amplitude of M87* taken at 2017 April 5th, 6th, 10th and
11th. The corresponding MAD/SANE scores are $0.52$, $0.4$, $0.43$, $0.76$ for
each day’s observation.

It is interesting that the results are similar across days, suggesting that
our data are consistent with no strong indication (e.g., over $2\sigma$) with
the MAD/SANE models. If the models faithfully represent the physical situation
in M87*, improvements in data quality (e.g., better baseline coverage) are
likely to enable a confident classification of the source as MAD or SANE.

Classification accuracy
---
Visibility form |  Noise free |  Thermal noise |  Thermal noise + complex gains
Complex |  $79.7\%\pm 1.1\%$ |  $79.1\%\pm 1.3\%$ |  $71.0\%\pm 1.6\%$
Closure |  $76.6\%\pm 1.3\%$ |  $73.6\%\pm 1.1\%$ |  $73.2\%\pm 1.6\%$
Table 1: Summary Statistics  Neural Networks MAD/SANE prediction score with
closure quantities
---
Observational date |  Score |  Prediction
2017 April 5 |  $0.52$ |  SANE
2017 April 6 |  $0.40$ |  MAD
2017 April 10 |  $0.43$ |  MAD
2017 April 11 |  $0.76$ |  SANE
Table 2: Neural Networks MAD/SANE prediction score for M87 on all four
observational dates

##  5 Discussion

We have presented a neural network pipeline that can perform data
classification to identify the accretion state of M87* as either MAD or SANE
using VLBI observations, after training on a library of $2\times 10^{4}$
simulations. Our pipeline operates directly on interferometric data
products—either complex visibilities or closure quantities—and thus does not
rely on an image reconstruction procedure. We find that the networks are able
to achieve a ${>}70$% classification accuracy across the board, even in the
presence of realistic levels of both thermal and complex gain data
corruptions.

Our network performs its most accurate classification ($\sim$80%) when the
data are provided as noise-free complex visibilities and its least accurate
classification ($\sim$71%) when the data are provided as complex visibilities
containing both thermal and complex gain uncertainties. The network achieves
intermediate performance ($\sim$76%) when it is provided with noise-free
closure quantities and when the closure quantities contain realistic levels of
noise the performance is more similar ($\sim$73%) to that of the corrupted
complex visibilities. These performance differences are qualitatively
consistent with the expected relative amounts of source information retained
in visibilities versus closure quantities (Blackburn et al., 2020). We also
note that the classification accuracy using any variant of interferometric
data product is considerably worse than that expected when performing
classification using the original images, which can achieve essentially
perfect ($\sim$99%; Lin et al. 2020a) classification accuracy; again, this
difference reflects the loss of information due to sparse Fourier sampling and
the reduced angular resolution of the data.

Although we did not provide the neural network with explicit priors, our input
training data set covered a limited set of parameter values and thus produced
an implicit, data-driven prior through the machine learning process.

###  5.1 Comparison with traditional analysis pipelines

The initial theoretical analysis performed by the EHT included a comparison
between observational data products and simulated images of black hole
accretion flows. In a comparison such as this a $\chi_{\nu}^{2}$ (reduced chi
squared) distance comparison of real and synthetic data is not useful because
intra-model variations—due to changes in image features caused by turbulent
fluctuations in the source, for example—are large compared to measurement
errors.

In contrast to snapshot-by-snapshot comparisons, the average image scoring
(AIS) procedure described in EHTC V was used to compute the likelihood the
data would be drawn from a given model. The AIS procedure ruled out a few
retrograde-spin accretion flow models based on variability, but it left much
of the parameter space unconstrained. Compared to our data-driven approach,
the AIS procedure is less constraining.

###  5.2 Future work

We have trained VLBInet only to classify M87 source models as MAD or SANE
based on the 2017 EHT configuration. In future work we plan to extend VLBInet
to estimate black hole mass and spin as well as parameters describing the
state of the plasma, and we plan to consider other arrays such as the EHT 2021
configuration and prospective ngEHT configurations (Blackburn et al., 2019).
We will also introduce more realistic data corruptions using a more
sophisticated end-to-end VLBI synthetic data generation pipeline (e.g.,
Roelofs et al. 2020).

In this work, we applied the neural network to M87 VLBI data. There are few
limitations, however: 1) if our GRMHD and GRRT simulations are flawed then the
neural network, which has been trained on a simulation-based synthetic
dataset, may not able to generalize to realistic data. 2) Similarly if the
noise models used in the generation of synthetic VLBI observables are flawed
then the neural network, which has been trained on the flawed noise model in
the synthetic dataset, may not be able not able to generalize to realistic
noise (e.g., realistic atmospheric, instrumental, and calibration effects) and
hence could predict spurious result. 3) Our training set covers the model
parameter space sparsely—for example, we use only five values for black hole
spin—and the prediction could therefore be biased.

Although we have demonstrated the utility of our framework only for the EHT,
we believe that it should in principle work for general classification
analyses using interferometric data. For instance, a similar network trained
on an appropriate set of simulated protoplanetary disk observations using ALMA
could plausibly constrain properties such as disk inclination, thickness, and
mass. In practice, using the network in such a fashion is likely to be limited
by the availability of suitable simulations, perhaps motivating the
development of simulation capabilities. This could have broad impact on the
community, as interferometric techniques have been crucial for many
astrophysical observational cases, including studies of dark matter
substructure in strong gravitational lensing (Hezaveh et al., 2016; Lin et
al., 2020b) and starburst galaxies (Vieira et al., 2013).

Since the EHT captures information about source polarization and time
dependence, we also plan to study whether these additional data can further
constrain physical parameters of the source. Synchrotron emission is naturally
polarized and the local properties of the magnetic field in the plasma alter
the electric vector position angle as radiation propagates through the plasma.
Thus, the structure of the magnetic field in the plasma influences the
orientation of linear polarization observed in black hole images. Palumbo et
al. (2020) and EHTC VIII showed that the qualitative and quantitative
differences between the near-horizon magnetic field strength and structure
suggest that linear polarization data might be particular useful in
differentiating between MAD and SANE models. We will apply interpretability
methods (Ghorbani et al., 2019; Lin et al., 2020a) to gain a more quantitative
understanding of the features in the visibility domain. It is critical to
understand the origin of accurate neural network classifications/regressions
so that we can be confident that we are not overfitting.

As pointed out by Sun et al. (2020), the VLBI observing proccess can be viewed
as a physics-constrained autoencoder in which the encoder is the sampling
strategy through the radio telescope. Our work could potentially be helpful
for evaluating telescope site candidates in a fast and automatic way (Raymond
et al., 2021). Visibility in some sense is similar to the hidden
representation/bottleneck in the bottleneck of variational autoencoder.
Traditionally, one would need to reconstruct the image and do data analysis on
top of the reconstructed images. Since the whole VLBI process looks like an
auto-encoder, we could in principle not only optimize the decoder (from
measurement to parameters) but also optimize the encoder (from source to
encoded representation that preserve the most information through sparse
sampling) (Sun & Bouman, 2020; Bakker et al., 2020).

The authors thank He Sun, Katie Bouman, Michael Janssen, Alexander Raymond,
Bart Ripperda, Shep Doeleman, Gil Holder, Max Welling, Abhishek Joshi, Vedant
Dhruv, David Ruhe, Michael Eickenberg, Shirley Ho, and David Spergel for
useful discussion. The authors thank John Wardle, Shiro Ikeda for useful
feedback. JL, GNW, BP, and CFG were supported by the National Science
Foundation under grants AST 17-16327, OISE 17-43747, and AST 20-34306. GNW was
supported in part by a Donald C. and F. Shirley Jones Fellowship and the
Institute for Advanced Study. DWP acknowledges support provided by the NSF
through grants AST-1952099, AST-1935980, AST-1828513, and AST-1440254, and by
the Gordon and Betty Moore Foundation through grant GBMF-5278. This work has
been supported in part by the Black Hole Initiative at Harvard University,
which is funded by grants from the John Templeton Foundation and the Gordon
and Betty Moore Foundation to Harvard University. This work used the Extreme
Science and Engineering Discovery Environment (XSEDE) resource stampede2 at
TACC through allocation TG-AST170024. This work utilizes resources supported
by the National Science Foundation’s Major Research Instrumentation program,
grant 1725729, as well as the University of Illinois at Urbana–Champaign. JL
thanks the AWS Cloud Credits for Research program. JL and CFG thank the GCP
Research Credits program.

## References

  * Bakker et al. (2020) Bakker, T., van Hoof, H., & Welling, M. 2020, Advances in Neural Information Processing Systems, 33
  * Battaglia et al. (2018) Battaglia, P. W., Hamrick, J. B., Bapst, V., et al. 2018, arXiv preprint arXiv:1806.01261
  * Bisnovatyi-Kogan & Ruzmaikin (1974) Bisnovatyi-Kogan, G. S., & Ruzmaikin, A. A. 1974, Ap&SS, 28, 45, doi: 10.1007/BF00642237
  * Blackburn et al. (2020) Blackburn, L., Pesce, D. W., Johnson, M. D., et al. 2020, The Astrophysical Journal, 894, 31
  * Blackburn et al. (2019) Blackburn, L., Doeleman, S., Dexter, J., et al. 2019, arXiv preprint arXiv:1909.01411
  * Bradley (1997) Bradley, A. P. 1997, Pattern recognition, 30, 1145
  * Chael et al. (2018) Chael, A. A., Johnson, M. D., Bouman, K. L., et al. 2018, ApJ, 857, 23, doi: 10.3847/1538-4357/aab6a8
  * Chael et al. (2016) Chael, A. A., Johnson, M. D., Narayan, R., et al. 2016, ApJ, 829, 11, doi: 10.3847/0004-637X/829/1/11
  * Chesler et al. (2021) Chesler, P., Blackburn, L., Doeleman, S., et al. 2021, Classical and Quantum Gravity
  * Collaboration (2019) Collaboration, E. H. T. 2019, ApJ, 875, L1, doi: 10.3847/2041-8213/ab0ec7
  * Event Horizon Telescope Collaboration (2019a) Event Horizon Telescope Collaboration. 2019a, ApJ, 875, L2, doi: 10.3847/2041-8213/ab0c96
  * Event Horizon Telescope Collaboration (2019b) —. 2019b, ApJ, 875, L3, doi: 10.3847/2041-8213/ab0c57
  * Event Horizon Telescope Collaboration (2019c) —. 2019c, ApJ, 875, L4, doi: 10.3847/2041-8213/ab0e85
  * Event Horizon Telescope Collaboration (2019d) —. 2019d, ApJ, 875, L5, doi: 10.3847/2041-8213/ab0f43
  * Event Horizon Telescope Collaboration (2019e) —. 2019e, ApJ, 875, L6, doi: 10.3847/2041-8213/ab1141
  * Event Horizon Telescope Collaboration (2019f) —. 2019f, First M87 EHT Results: Calibrated Data, CyVerse Data Commons, doi: 10.25739/g85n-f134
  * Event Horizon Telescope Collaboration (2021a) —. 2021a, The Astrophysical Journal Letters, 910, L12
  * Event Horizon Telescope Collaboration (2021b) —. 2021b, ApJ, 910, L13, doi: 10.3847/2041-8213/abe4de
  * Ghorbani et al. (2019) Ghorbani, A., Wexler, J., Zou, J. Y., & Kim, B. 2019, in Advances in Neural Information Processing Systems, 9273–9282
  * Gralla & Lupsasca (2020) Gralla, S. E., & Lupsasca, A. 2020, Physical Review D, 102, 124003
  * Hadar et al. (2021) Hadar, S., Johnson, M. D., Lupsasca, A., & Wong, G. N. 2021, Phys. Rev. D, 103, 104038, doi: 10.1103/PhysRevD.103.104038
  * Hezaveh et al. (2016) Hezaveh, Y. D., Dalal, N., Marrone, D. P., et al. 2016, The Astrophysical Journal, 823, 37
  * Ichimaru (1977) Ichimaru, S. 1977, ApJ, 214, 840, doi: 10.1086/155314
  * Igumenshchev et al. (2003) Igumenshchev, I. V., Narayan, R., & Abramowicz, M. A. 2003, ApJ, 592, 1042, doi: 10.1086/375769
  * Johnson et al. (2020) Johnson, M. D., Lupsasca, A., Strominger, A., et al. 2020, Science advances, 6, eaaz1310
  * Lin et al. (2020a) Lin, J. Y.-Y., Wong, G. N., Prather, B. S., & Gammie, C. F. 2020a, arXiv preprint arXiv:2007.00794
  * Lin et al. (2020b) Lin, J. Y.-Y., Yu, H., Morningstar, W., Peng, J., & Holder, G. 2020b, arXiv preprint arXiv:2010.12960
  * Lupsasca et al. (2018) Lupsasca, A., Porfyriadis, A. P., & Shi, Y. 2018, Physical Review D, 97, 064017
  * Morningstar et al. (2018) Morningstar, W. R., Hezaveh, Y. D., Levasseur, L. P., et al. 2018, arXiv preprint arXiv:1808.00011
  * Mościbrodzka & Gammie (2018) Mościbrodzka, M., & Gammie, C. F. 2018, MNRAS, 475, 43, doi: 10.1093/mnras/stx3162
  * Narayan et al. (2003) Narayan, R., Igumenshchev, I. V., & Abramowicz, M. A. 2003, PASJ, 55, L69, doi: 10.1093/pasj/55.6.L69
  * Palumbo et al. (2020) Palumbo, D. C. M., Wong, G. N., & Prather, B. S. 2020, ApJ, 894, 156, doi: 10.3847/1538-4357/ab86ac
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., et al. 2019, in Advances in Neural Information Processing Systems 32, ed. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, & R. Garnett (Curran Associates, Inc.), 8024–8035
  * Popov et al. (2021) Popov, A., Strokov, V., & Surdyaev, A. 2021, Astronomy and Computing, 100467
  * Porth (2019) Porth, O. e. a. 2019, ApJS, 243, 26, doi: 10.3847/1538-4365/ab29fd
  * Raymond et al. (2021) Raymond, A. W., Palumbo, D., Paine, S. N., et al. 2021, The Astrophysical Journal Supplement Series, 253, 5
  * Roelofs et al. (2020) Roelofs, F., Janssen, M., Natarajan, I., et al. 2020, Astronomy & Astrophysics, 636, A5
  * Sun & Bouman (2020) Sun, H., & Bouman, K. L. 2020, arXiv preprint arXiv:2010.14462
  * Sun et al. (2020) Sun, H., Dalca, A. V., & Bouman, K. L. 2020, in 2020 IEEE International Conference on Computational Photography (ICCP), IEEE, 1–12
  * Thompson et al. (2017) Thompson, A. R., Moran, J. M., & Swenson, George W., J. 2017, Interferometry and Synthesis in Radio Astronomy, 3rd Edition, doi: 10.1007/978-3-319-44431-4
  * van der Gucht et al. (2020) van der Gucht, J., Davelaar, J., Hendriks, L., et al. 2020, Astronomy & Astrophysics, 636, A94
  * Vieira et al. (2013) Vieira, J., Marrone, D. P., Chapman, S., et al. 2013, Nature, 495, 344
  * Wong (2021) Wong, G. N. 2021, ApJ, 909, 217, doi: 10.3847/1538-4357/abdd2d