# Enhance Multimodal Transformer With External Label And In-Domain Pretrain:
Hateful Meme Challenge Winning Solution

Ron Zhu
_ALFRED SYSTEM_

(December 2020)

###### Abstract

Hateful meme detection is a new research area recently brought out that
requires both visual, linguistic understanding of the meme and some background
knowledge to performing well on the task. This technical report summarises the
first place solution of the Hateful Meme Detection Challenge 2020, which
extending state-of-the-art visual-linguistic transformers to tackle this
problem. At the end of the report, we also point out the shortcomings and
possible directions for improving the current methodology.

##  1 Introduction

The Hateful Memes Challenge[8] introduces the dataset designed to be truly
multimodal by including confounders that prevent the model from exploiting
unimodal prior. Despite the recent advances in multimodal reasoning, we only
get 0.71 AUROC when applying state of the art multimodal models to the
challenge, which is far behind from non-expert human performance. In this
paper, we discuss the difference between the commonly seen multimodal
reasoning task and the meme classification. And the approaches we build on top
of the existing visual-linguistic models to improve the performance and
achieving 0.845 AUROC on the hateful memes detection dataset.

Figure 1: Complete data processing pipeline

##  2 Problem Description

In the typical visual-linguistic multimodal dataset, there is a direct
relationship between the image and textual input. Hateful memes have their
unique characteristic that is much different from the usual multimodal
dataset. Compare to the multimodal dataset for a task like VCR[16] and VQA[1],
only some of them have a direct relationship that appears in common multimodal
down-stream tasks and pre-train datasets. Most of the time, the meme’s hateful
part is linked by subtle wording or external knowledge about the real-world
event. In that sense, hateful memes challenge is more of a visual-linguistic-
knowledge multimodal classification problem.

Moreover, hateful memes are built on diverse topics and often contain novel
objects that no off-the-shelf classifier will recognize, which is much
different from the everyday object presented in typical multimodal datasets.
Visual hints like injury, traditional costume wear by the subject, and
historical scenes are hard to recognize and low on the sample number. Those
factors make visual understanding part of the challenge tough to solve even
without the lingual part. The context of the visual modality context has an
enormous impact on the meme’s polarity. Because visual hint is hard to
recognize even for the state of the art image classifier or object detector,
it becomes essential to incorporate information from a different source and
format.

In some cases, the position of the meme caption will also affect the meaning
of the meme. For example, intentionally place the caption of “wishing machine”
on top of the female subject make the meme hateful, but if we place the
caption on anything else instead of human will not. And this not commonly see
and accounted for in the general visual-linguistic model.

##  3 Approach

The solution comprises four different VL transformer architecture ensemble,
namely: VL-BERT[13], UNITER[3], VILLA[6], and ERNIE-Vil[15]. All the models
besides ERNIE-Vil were modified for better performance of hateful meme
detection. Furthermore, we conduct extra steps to extract more information
from the hateful memes dataset for both training and inference.

###  3.1 Background

In the recent trend of applying the pre-trained transformer model to VL
multimodal task have greatly improved the SOTA of multiple benchmarks. State
of the art VL transformers like Oscar[10], VILLA, ERNIE-Vil are utilize large
scale dataset with paired image and text for pretraining. Although they
achieve enormous improvement on the common sense VL tasks, pretrained VL
transformer still performs poorly on hateful meme detection. The task requires
the model to have a deep understanding of image, text, real-world events, and
the ability to recognize inter-modal interaction between three modalities.

###  3.2 Additional Data Source

As forward-mentioned, we will need more than the image and text description of
memes to do well on hateful meme detection. So at the first step of the data
preprocessing, we will try to extract a few critical information from the data
source using Web Entity Detection and FairFace[7] classifier.

Memes often contain a reference to famous historical events, news,
celebrities. And memes reference to news topic and different sub-culture trend
are changing rapidly. It not uncommon to see new memes popping up every few
days or weeks. So using static knowledge sources(ex: Wikipedia) alone may not
be adequate for this problem. To deal with this challenge, we use Google
Vision Web Entity Detection to capture the image’s context. Despite this,
without taking the entity’s background knowledge and the relation between
entities into account, we only solve half of the problem.

According to the Hateful Meme Challenge paper, amount all types of hateful
memes, hateful meme related to race or ethnicity is the most prominent. Almost
half of the hateful memes can be categorized to it. Furthermore, it is
unrealistic to expect a pretrained VL transformer to learn to identify race
with just a few thousand samples from the hateful memes dataset. Thus the
inclusion of race and gender labels created from the FairFace classifier. We
apply the classifier by first detecting and classifying every face or head in
the image, then mapping the label back to the person’s bounding box with the
largest overlapped area with the face. In the end, we should get the race and
gender of most of the person to appear in the image unless the person’s head
is obscured.

###  3.3 Extend VL-BERT Visual-Linguistic Framework

To jointly train multiple types of labels with the original meme image and
text. We need a way to combine information from different sources and formats
seamlessly. Also, the ability to explicitly link images and text to the
external labels will help deal with the unique characteristic of memes and
remove the need for learning how to use external labels from scratch.

Inspire by Oscar, we represented all external labels as a special type of text
token and linked it to the general area in the image or a special image region
using visual feature embedding. By doing so, we also create the implicit link
between text tokens that have related image features. For example, text token
“bowler hat” with the image region of a hat and “man dressed in a three-piece
suit” with the image region covers the entire person wearing the hat/suit.
With a versertail framework like this, we can easily add more types of labels
to handle different types of memes in the future.

Figure 2: Architecture of extened VL-BERT

###  3.4 UNITER-ITM

Many pretrained VL transformer inspires by BERT[4] also include the VL
adaptation of the NSP(next sentence prediction) task - ITM(image text
matching). Although some research shows that the inclusion of ITM tasks gives
no benefit or even reduces the performance on the downstream task, we can
still find it helpful depending on the task on hand. In the case of hateful
memes detection, there is an apparent relationship between it and ITM; namely,
the meme with the description(text) aligning the image’s content is most
probably benign, but meme may not necessarily being hateful when they are not
aligning. It may just the meme’s description and image being unrecognizable or
hard to understant by the model, or actually being hateful and comparing
people to object or animals.

With the UNITER pre-trained transformer, we keep the ITM head pre-trained on
ITM and WRA(word region alignment) tasks instead of randomly initializing the
classifier head weight. Then swap the weights of class 0 of classifier head
with class 1, so now class “image-text-match” will become “benign” and “image-
text-not-match” become “hateful”.

Suggest by recent research[2] both single-stream or dual-stream VL
transformers are biasing toward linguistic modal. By reusing ITM head as the
starting point, we should get an extra benefit of reducing linguistic bias due
to the face that ITM and WRA require both visual and linguistic feature to
work, which is not always true for another pre-train task like MRC(masked
region classification) or MLM(masked language modeling).

Figure 3: UNITER-ITM

###  3.5 ERNIE-Vil

ERNIE-Vil is the VL model pretrained on information extracted from the scene
graph to teach the model about fine-grained object attribute and relationship
model. Being the STOA model on the VCR leaderboard, ERNIE-Vil can achieve
competitive performance to extended VL-BERT and UNITER-ITM with some hyper-
parameter tuning. Including ERNIE-Vil into the final ensemble add diversity to
the prediction and improve generalization.

##  4 Experiment

###  4.1 Apply entity/race/gender tags

By applying the new labels obtained from the external model to extended VL-
BERT, our accuracy get a significant boost on the dev-seen set of hateful
memes dataset. Nevertheless, when we apply the same entity tags to other
single stream models like UNITER and Oscar, they give worse or almost no
performance difference. Then apply the same trick on ERNIE-Vil give a
noticeable jump in performance. While the difference in architecture and
training target is small between single-stream models we tested, entity tags’
effect differs. There are a few potential reasons that may cause this
phenomenon:

1) Lack of explicitly visual-linguistic fusion mechanism (VL-BERT’s visual
embedding for every token, and ERNIE-Vil’s cross-modality layers) make it hard
to learn how to utilize a new type of input format with a small dataset like
the one we are using.

2) Lower lingual representation quality(VL-BERT has been pre-trained with text
only MLM task, ERNIE-Vil have the independent text modality stream) let the
entity tags that are represented in the text become less effective.

Figure 4: dev-seen AUROC of extended VL-BERT with entity/race/gender tags

(a) UNITER-ITM with and w/o entity tags

(b) ERNIE-Vil with and w/o entity tags

Figure 5: Hateful Memes dev-seen set AUROC

###  4.2 Reuse of UNITER ITM head

As shown in the figure.6, UNITER-ITM compare to randomly initialized
classification head:

1) seen not as easy to overfit.

2) getting better performance. (0.778 vs. 0.765 AUROC).

3) may suffer less bias toward text modality, thanks to ITM and WRA pre-train
task require both modal to work.

The high accuracy of UNITER-ITM at the very start of the training may also
indicate a correlation between ITM pre-train task and hateful memes
classification. This leaves an interesting question of if there is another
pre-train task for the VL model that also has the same high correlation to
hateful memes classification.

Figure 6: dev-seen AUROC of UNITER w/ and w/o reuse ITM head

##  5 Feature Direction

Hateful meme detection is currently one of the few tasks that require the
model to identify entities and relationships from image, text, and external
knowledge base. In the Hateful Memes dataset, hateful metaphor hidden between
relation visual, linguistic, knowledge base entities are common but extremely
hard to spot for the machine. Most of the pre-trained models currently apply
to hateful meme detection are only take image and text into account, and the
pre-train dataset only contained everyday object and simple description that
gives a shallow explanation of the image. Although the pre-trained language
model used in some VL models can be treated as a knowledge base and have
decent performance in the task like open-domain QA. However, this kind of
implicit knowledge base is hard to update and apply to a different domain.
Several research attempts to resolve this problem by letting the model learn
to use the external knowledge base instead of remembering everything inside
the model’s parameters, for example, RAG[9].

A better and more flexible source of knowledge that provides in-depth
information about the entity that appears in the meme should help reduce the
performance gap between machine and human. One of the promising directions
also comes from the field of open-domain QA: knowledge graph enhanced pre-
train transformers(language model). There are many ways we can inject external
knowledge into pre-trained transformers like K-Adapter[14], K-BERT[11]. One of
them, MHGRN[5], is especially interesting to me. Due to its plug-and-play
nature, MHGRN can place after any pre-trained transformer; in our case, any
one of the VL transformers will work. This gave us many options and
flexibility on how we want to handle the visual-linguistic part of the
problem. Moreover, the attention GNN used in MHGRN makes it easier to scale
the model to a larger graph, which may be a critical part of the problem when
dealing with the subtle relationship between key entities inside a meme.
Despite the advantage mentioned above, using a knowledge-graph-based method to
solve hateful meme detection still come with a few challenges. To cover the
diverse topic meme is referencing from, we will also need a sizeable knowledge
graph like wiki-data or ConceptNet[12] to compensate for it. For every meme we
are dealing with, we need to extract a subgraph that contains the entity that
appears in the meme’s image or text and an entity that is potentially related
to the context of the meme. Entities in a large scale knowledge like wiki-data
usually have dozens of type of relation per entity. Thus building a subgraph
that both includes enough information and has a reasonable size will be
challenging.

##  6 Conclusion

In this paper, we propose two methods to improve pre-trained visual-linguistic
transformers’ performance on the task of hateful meme detection. The first
approach focuses on building a visual-linguistic framework on top of VL-BERT
to incorporate pieces of information that are almost impossible to learn from
the hateful meme challenge dataset directly. The second approach tries to
utilize the entire pre-trained model to get a better score on meme
classification. In spite of the improvement brought by those two approaches,
there is still much space to improve. Furthermore, we also point out some of
the promising research directions that may give the model a better
understanding of the memes.

## References

  * [1] Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh.  Vqa: Visual question answering, 2016.
  * [2] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu.  Behind the scene: Revealing the secrets of pre-trained vision-and-language models, 2020.
  * [3] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In ECCV, 2020.
  * [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [5] Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, and Xiang Ren.  Scalable multi-hop relational reasoning for knowledge-aware question answering, 2020.
  * [6] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu.  Large-scale adversarial training for vision-and-language representation learning, 2020.
  * [7] Kimmo K”arkk”ainen and Jungseock Joo.  Fairface: Face attribute dataset for balanced race, gender, and age.  arXiv preprint arXiv:1908.04913, 2019.
  * [8] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine.  The hateful memes challenge: Detecting hate speech in multimodal memes, 2020.
  * [9] Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.  Retrieval-augmented generation for knowledge-intensive nlp tasks, 2020\.
  * [10] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao.  Oscar: Object-semantics aligned pre-training for vision-language tasks, 2020.
  * [11] Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang.  K-bert: Enabling language representation with knowledge graph, 2019.
  * [12] Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge, 2018\.
  * [13] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations, 2020\.
  * [14] Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu ji, Guihong Cao, Daxin Jiang, and Ming Zhou.  K-adapter: Infusing knowledge into pre-trained models with adapters, 2020\.
  * [15] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang.  Ernie-vil: Knowledge enhanced vision-language representations through scene graph, 2020.
  * [16] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning, 2019.


# Vision Aided Environment Semantics Extraction and Its Application in mmWave
Beam Selection

Feiyang Wen, Weihua Xu, Feifei Gao, Chengkang Pan, and Guangyi Liu F. Wen, W.
Xu, and F. Gao are with Institute for Artificial Intelligence, Tsinghua
University (THUAI), Beijing National Research Center for Information Science
and Technology (BNRist), Department of Automation, Tsinghua University,
Beijing, P.R. China, 100084 (email<EMAIL_ADDRESS><EMAIL_ADDRESS>feifeigao@ieee.org).C. Pan and G. Liu are with
China Mobile Research Institute, Beijing, China. (e-mail:
<EMAIL_ADDRESS>liuguangyi@chinamobile.com).

###### Abstract

In this letter, we propose a novel mmWave beam selection method based on the
environment semantics extracted from user-side camera images. Specifically, we
first define the environment semantics as the spatial distribution of the
scatterers that affect the wireless propagation channels and utilize the
keypoint detection technique to extract them from the input images. Then, we
design a deep neural network with the environment semantics as the input that
can output the optimal beam pairs at the mobile station (MS) and the base
station (BS). Compared with the existing beam selection approaches that
directly use images as the input, the proposed semantic-based method can
explicitly obtain the environmental features that account for the propagation
of wireless signals, thus reducing the storage and computational burden.
Simulation results show that the proposed method can precisely estimate the
location of the scatterers and outperform the existing works based on computer
vision or light detection and ranging (LIDAR).

###### Index Terms:

Beam selection, mmWave, environment semantics, deep learning, computer vision

††publicationid: pubid:

##  I Introduction

MILLIMETER wave (mmWave) with beamforming is a critical technology for next-
generation wireless communications and can achieve a higher transmission rate.
However, the traditional beamforming approaches, e.g., pilot-based strategies
[1] or beam sweeping [2], are bottlenecked by a high spectrum overhead,
especially in time-varying scenarios such as vehicle-to-infrastructure (V2I)
communications. Recently, it has been indicated that with the aid of
environment sensing information, such as point clouds [3]-[5] and images
[6]-[9], one can implement beamforming with lower latency and fewer spectrum
resources.

In [3], the authors utilize a deep neural network (DNN) classifier to predict
the optimal beam pairs from the point clouds scanned by light detection and
ranging (LIDAR). The authors of [4] and [5] optimize the LIDAR-based method
with federated learning strategies and various deep learning techniques. With
the aid of images taken at the base station (BS) side, [6] presents a multi-
modal beam prediction scheme, and [9] proposes a strategy for proactive
blockage prediction and user hand-off. Based on images from the camera view of
the mobile station (MS), the authors of [7] perform channel covariance matrix
estimation, and the authors of [8] propose a beam alignment method supported
by the object detection techniques.

However, the existing vision-based approaches are implemented in a
straightforward manner, i.e., the optimal beam pairs are predicted directly
from the input images, which may lead to certain disadvantages. According to
the discrete physical model for wireless propagation [10], among all the
environmental characteristics, only a few scatterers can significantly affect
the wireless channel. Hence, there is a large amount of redundancy in the
images, which reduces the training efficiency, causes the overfitting problem,
and jeopardizes the accuracy of the beam selection. Moreover, without
explicitly extracting the effective scatterers, the existing methods have to
use more complicated DNN models, resulting in a larger computational and
storage burden. To accurately represent the propagation environment, the
semantic information of the images, i.e., the characteristics that account for
the propagation of the wireless signal, should be extracted. Therefore, we
define environment semantics by considering the spatial distribution of the
effective scatterers.

In this letter, we propose a vision-aided environment semantics extraction
method and apply it to beam selection for V2I communication scenarios. In
order to eliminate the redundant information, we represent the environment
semantics as semantic heatmaps and extract the heatmaps from the input images.
Then, the optimal beam pair is predicted by another neural network from the
heatmaps. Simulation results indicate that the proposed method can accurately
capture the environment semantics and outperform the existing vision-based or
LIDAR-based methods.

##  II System Model

###  II-A Channel Model

Consider a downlink V2I communication system where a stationary BS serves an
MS on a vehicle in the mmWave band. Both BS and MS are equipped with a uniform
planar array (UPA). The numbers of antennas are $N_{B}=N_{B}^{a}\times
N_{B}^{b}$ and $N_{M}=N_{M}^{a}\times N_{M}^{b}$, respectively. The channel
matrix $\mathbf{H}$ can be represented by the widely-adopted geometric channel
model:

$\mathbf{H}=\sum_{p=1}^{P}\alpha_{p}\mathbf{a}_{r}(\theta_{p}^{A},\varphi_{p}^{A})\mathbf{a}_{t}^{\mathrm{H}}(\theta_{p}^{D},\varphi_{p}^{D}),$
(1)

where $P$ is the number of multipath components (MPC), $\alpha_{p}$ denotes
the complex gain of the $p$th propagation path, and
$\theta_{p}^{A},\varphi_{p}^{A},\theta_{p}^{D},\varphi_{p}^{D}$ are the
elevation and azimuth of the angle of arrival and departure of the $p$th path,
respectively. Moreover, the steering vectors
$\mathbf{a}_{r}(\theta_{p}^{A},\varphi_{p}^{A})\in\mathbb{C}^{N_{M}\times
1},\mathbf{a}_{t}(\theta_{p}^{D},\varphi_{p}^{D})\in\mathbb{C}^{N_{B}\times
1}$ are defined as

$\displaystyle\mathbf{a}(N^{a},N^{b};\theta,\varphi)=$
$\displaystyle\frac{1}{\sqrt{N^{a}N^{b}}}[1,...,e^{j\pi(a\cos{\theta}+b\sin{\theta}\sin{\varphi})},$
(2)
$\displaystyle...,e^{j\pi((N^{a}-1)\cos{\theta}+(N^{b}-1)\sin{\theta}\sin{\varphi})}].$

Then, we have $\mathbf{a}_{r}=\mathbf{a}(N_{M}^{a},N_{M}^{b})$ and
$\mathbf{a}_{t}=\mathbf{a}(N_{B}^{a},N_{B}^{b})$.

Assume that BS and MS are equipped with fixed beam codebooks. The codebook at
the transmitter is
$\mathcal{W}_{B}=\\{\mathbf{w}_{B}^{1},\mathbf{w}_{B}^{2},...,\mathbf{w}_{B}^{C_{B}}\\}$,
while the codebook at the receiver is
$\mathcal{W}_{M}=\\{\mathbf{w}_{M}^{1},\mathbf{w}_{M}^{2},...,\mathbf{w}_{M}^{C_{M}}\\}$,
where $\mathbf{w}_{B}^{i}\in\mathbb{C}^{N_{B}\times
1},\mathbf{w}_{M}^{j}\in\mathbb{C}^{N_{M}\times 1}$ denote the transmitting
and receiving vector. Under the noise-free assumption, the received power gain
of beam pair $(\mathbf{w}_{B}^{i},\mathbf{w}_{M}^{j})$ is

$y_{ij}=|(\mathbf{w}_{M}^{j})^{\mathrm{H}}\mathbf{H}\mathbf{w}_{B}^{i}|^{2}.$
(3)

The goal is to select the optimal beam pair from the beam codebooks that can
maximize the power gain, i.e.,

$(\mathbf{w}_{B}^{i*},\mathbf{w}_{M}^{j*})=\mathop{\arg\max}_{\mathbf{w}_{B}^{i}\in\mathcal{W}_{B},\mathbf{w}_{M}^{j}\in\mathcal{W}_{M}}y_{ij}.$
(4)

###  II-B Definition of the Semantic Heatmaps

To represent the environment semantics, we define the effective scatterers and
generate the semantic heatmaps by projecting the effective scatterers into the
camera view of MS.

Note that the weak paths have less impact on the channel, and considering them
would increase the computational and storage burden. Therefore, we neglect the
paths with $|\alpha_{p}|^{2}/|\alpha_{max}|^{2}<P_{th}$, where
$|\alpha_{max}|^{2}$ denotes the power gain of the strongest path in a scene
and $P_{th}$ is a tunable threshold to trade off the accuracy against the
computational resources. As shown in Fig. 1(a), for each none-line-of-sight
(NLOS) path, we define the scatterer that reflects the last hop along the
signal propagation as the effective scatterer; while for the line-of-sight
(LOS) path, BS serves as the effective scatterer. In this way, we ensure that
all the effective scatterers are in the camera view of MS and can be extracted
from the images.

(a) The definition of the effective scatterers.

(b) The generation of the semantic heatmaps.

Figure 1: An illustration to the semantic heatmaps.

We assume that MS is equipped with $N_{C}$ monocular cameras, which are
installed at different azimuths to provide multi-view images. Meanwhile, all
the cameras are equipped at the same location with the antenna.111In practice,
if the cameras are fixed far away from the antenna, then one can utilize novel
view synthesis techniques [16] to generate the antenna-view images from other
camera views. We define the elevation and the azimuth of the $i$th camera
relative to MS as $(\theta_{i}^{C},\varphi_{i}^{C})$, the size of the taken
image as $H^{C}\times W^{C}$, and the horizontal field of view of the camera
as $2\beta_{i}$. If the $p$th effective scatterer is in the $i$th camera’s
field of view, then it is projected onto the coordinates
$(x_{p}^{C},y_{p}^{C})$ of the image plane as

$\begin{bmatrix}x_{p}^{C}\\\ y_{p}^{C}\\\
1\end{bmatrix}=\begin{bmatrix}\frac{W^{C}}{2\tan\beta_{i}}&0&H^{C}/2\\\
0&\frac{W^{C}}{2\tan\beta_{i}}&W^{C}/2\\\
0&0&1\end{bmatrix}\begin{bmatrix}\frac{\tan\theta_{p,i}^{C}}{\cos\varphi_{p,i}^{C}}\\\
\tan\varphi_{p,i}^{C}\\\ 1\end{bmatrix},$ (5)

where $\theta_{p,i}^{C}=\theta_{p}^{A}-\theta_{i}^{C}$ and
$\varphi_{p,i}^{C}=\varphi_{p}^{A}-\varphi_{i}^{C}$ denote the elevation and
the azimuth of the $p$th path in the $i$th camera’s field of view.

Based on the projection process, we generate two types of heatmaps for each
camera view: the spatial distribution heatmap $D_{i}$ and the strength heatmap
$S_{i}$. Both heatmaps coincide with the image plane at a lower resolution,
i.e., $D_{i},S_{i}\in[0,1]^{\frac{H^{C}}{R}\times\frac{W^{C}}{R}}$, and the
coordinates of the $p$th scatterer are denoted as
$(x_{p}^{H},y_{p}^{H})=(\lfloor\frac{x_{p}^{C}}{R}\rfloor,\lfloor\frac{y_{p}^{C}}{R}\rfloor)$.
The corresponding elements of the heatmaps are set to
$D_{i}(x_{p}^{H},y_{p}^{H})=1$ and
$S_{i}(x_{p}^{H},y_{p}^{H})=|\alpha_{p}|^{2}/|\alpha_{max}|^{2}$,
respectively. Then, as a measure to improve the prediction performance of DNN,
we spread all the projected points on $D_{i}$ and $S_{i}$ using a Gaussian
kernel $K=\exp{(\frac{(x-x_{p,i}^{H})^{2}+(y-y_{p,i}^{H})^{2}}{2\sigma^{2}})}$
[14], where $\sigma$ is an empirical parameter determined by the size of the
heatmaps.

##  III Vision Based Environment Semantics Extraction and Beam Selection

Figure 2: The proposed deep learning based method for environment semantics
extraction and beam selection.

To obtain the optimal beam pairs in (4), we propose a two-stage beam selection
approach, as shown in Fig. 2. We assume that the location and the orientation
of the MS can be obtained by vehicular sensors such as inertial measurement
units and GPS receivers, and we assume that the location of the BS is known.
Therefore, the location $(x_{M},y_{M},z_{M})$ and the orientation $\theta_{M}$
of the MS relative to the BS are available.

Different from [6], [9], during the communication process, the optimal beam
pairs are predicted by MS. In the first stage, MS obtains its location
information and multi-view images and sends them to a well-designed DNN for
semantics extraction. In the second stage, based on the semantic heatmaps and
the location information, another DNN predicts the optimal beam pairs and
feeds them back to the BS. In the following subsections, we will present the
two stages in detail.

###  III-A Environment Semantics Extraction

####  III-A1 Preprocessing

At each time step, MS obtains the RGB images $I_{1},I_{2},...,I_{N_{C}}$,
where $I_{i}\in\mathbb{R}^{H^{C}\times W^{C}\times 3}$, and the location
information that satisfies $(x_{M},y_{M},z_{M},\theta_{M})\in\mathbb{R}^{4}$.
We simply concatenate the images in the channel dimension to obtain the input
image feature $I_{in}\in\mathbb{R}^{H^{C}\times W^{C}\times 3N_{C}}$. However,
as shown in [15], a DNN can poorly represent high-frequency features from the
raw coordinate form of the location information, leading to limited
performance. Hence, we encode $x_{M},y_{M}$ using a high-frequency function
[15]:

$\gamma(p)\\!\\!=\\!\\!(\sin(2^{0}\pi p),\cos(2^{0}\pi p),...,\sin(2^{L-1}\pi
p),\cos(2^{L-1}\pi p)),$ (6)

where $L$ is set to 5. The input location vector is concatenated as
$L_{in}=(\gamma(x_{M}),\gamma(y_{M}),z_{M},\cos\theta_{M},\sin\theta_{M})\in\mathbb{R}^{23}$.

####  III-A2 Semantics Extraction via Keypoint Detection

The semantics extraction network takes $I_{in},L_{in}$ as input and predicts
the semantic heatmap
$\hat{H}\in[0,1]^{\frac{H^{C}}{R}\times\frac{W^{C}}{R}\times 2N_{C}}$, where
the first $N_{C}$ channels and the last $N_{C}$ channels correspond to $D_{i}$
and $S_{i}$ for each camera view, respectively.

As shown in Fig. 2, we use a convolutional layer and several residual blocks
[17] to extract the image feature map from $I_{in}$. Meanwhile, we use fully
connected layers, 2-D reshape layer, and residual blocks to obtain the
location feature map from $L_{in}$. The image feature map can represent the
interplay of environment scatterers, especially for dynamic scatterers like
vehicles, while the location feature map contains the exact geometric relation
between BS and MS. We fuse the two feature maps using the multiplication
operation to take full advantage of the contained information. Next, we adopt
the stacked hourglass networks [11], [12] to predict $\hat{H}$. Generally, the
hourglass networks capture multi-scale spatial relationships through a
repeated compressing-upsampling architecture.

Denote $H$ as the groundtruth of $\hat{H}$ 111During the training phase, the
angle of arrival (AOA) and the power gains of the main propagation paths are
required to generate $H$. In practice, they can be obtained via channel
estimation or AOA estimation algorithms.. We use the mean squared error (MSE)
loss $L_{S}=\sum_{x,y;C=N_{C}+1}^{2N_{C}}(\hat{H}_{xyC}-H_{xyC})^{2}$ for
$S_{i}$ and a variation of focal loss [13], [14]

$L_{D}\\!=\\!-\\!\\!\\!\\!\\!\sum_{x,y;C=1}^{N_{C}}\\!\\!\\!\left\\{\begin{aligned}
&(1-\hat{H}_{xyC})^{2}\log(\hat{H}_{xyC}),\text{if}\ H_{xyC}=1\\\
&(1-H_{xyC})^{4}\hat{H}_{xyC}^{2}\log(1-\hat{H}_{xyC}),\text{otherwise}\end{aligned}\right.$
(7)

for $D_{i}$, respectively. The heatmap prediction loss function is then
defined as $L_{hm}=L_{D}+L_{S}$.

###  III-B Beam Selection

The beam selection network takes the predicted semantic heatmap $\hat{H}$ and
the location vector $L_{in}$ as inputs. We process $\hat{H}$ by several
convolutional layers and process $L_{in}$ by fully connected layers. Next, two
fully connected layers followed by a softmax layer are adopted to yield the
prediction vector $\hat{y}_{ij}\in[0,1]^{N_{B}\times N_{M}}$. The predicted
optimal beam pair is then $(\hat{i},\hat{j})=\mathop{\arg\max}_{i,j}y_{ij}$.

The loss function for beam prediction is defined as [5]

$L_{pd}=-(1-\beta)\sum_{i,j}y_{ij}^{*}\log{(\hat{y}_{ij})}-\beta\sum_{i,j}\overline{y}_{ij}\log{(\hat{y}_{ij})},$
(8)

where $\overline{y}_{ij}$ is the normalized form of the power gain $y_{ij}$;
$y_{ij}^{*}=1$ for $(i^{*},j^{*})$ and $y_{ij}^{*}=0$ on other components;
$\beta\in[0,1]$ is a tunable hyperparameter. The overall loss function is then
defined as $L_{all}=L_{pd}+L_{hm}$.

##  IV Simulation Results

###  IV-A Simulation Setup

As shown in Fig. 3, we delineate a rectangle with width $W_{A}$ and length
$L_{A}$ as the coverage area of BS in the autonomous driving simulation
software CARLA [18]. Then, we utilize the traffic simulation software SUMO
[19] to generate a continuous traffic flow, which contains 5 types of vehicles
as listed in TABLE I. The vehicles move according to the routes and velocities
planned by SUMO, and the locations of the vehicles are synchronized to CARLA.
Once a vehicle moves within the coverage area of BS, the cameras on the
vehicle will take images at a regular time interval $T_{C}$, generating a
sequence of images as the vehicle passes through the coverage area. Meanwhile,
the entire simulation scene is recorded and synchronized to the ray tracing
software Wireless Insite [20] for wireless channel simulation, in which all
the vehicles are converted to cubes with the same sizes, locations, and
orientations as those in CARLA. In this way, we generate the dataset
containing the sequences of MS locations, images, and corresponding channel
information. The dataset is made available to the public [21].

Figure 3: The simulation scenario in CARLA. TABLE I: Simulated Vehicle Types. Type | Car #1 | Car #2 | Car #3 | Van | Bus
---|---|---|---|---|---
Length/m | 4.81 | 4.90 | 4.15 | 5.20 | 11.08
Width/m | 2.17 | 2.06 | 2.00 | 2.62 | 3.25
Height/m | 1.52 | 1.48 | 1.38 | 2.48 | 3.33
TABLE II: Ray Tracing Parameters for Wireless Insite. Parameter | Value
---|---
Carrier Frequency | 60 GHz
Propagation Model | X3D
Building Material | Concrete
Vehicle Material | Metal
Maximum Number of Reflections | 6
Maximum Number of Diffractions | 1
Maximum Paths Per Receiver Point | 25

We set $W_{A}=$ 48 m, $L_{A}=$ 192 m, and $T_{C}=$ 0.5 s; the UPA of BS is
fixed at 3 m above the street; the UPA of MS is fixed at 0.1 m above the roof
center of the vehicle; the cameras are installed horizontally with $N_{C}=$ 4,
$\varphi_{i}^{C}=$ 0∘, 90∘, 180∘, 270∘ respectively, $2\beta_{i}=$ 90∘, and
$H_{C}\times W_{C}=$ 192$\times$512 in pixels. For the semantic heatmaps, we
set $\frac{H_{C}}{R}\times\frac{W_{C}}{R}=$ 48$\times$128, $P_{th}=$ -10 dB,
and $\sigma=$ 1.5. The ray tracing parameters are shown in TABLE II;
$N_{B}^{a}=N_{M}^{a}=$ 8, $N_{B}^{b}=N_{M}^{b}=$ 64, $C_{B}=C_{M}=$ 64;
$\mathbf{w}_{B}^{i}=\mathbf{a}_{t}(\theta_{t},\frac{2i-C_{B}-1}{2C_{B}}\pi)$,
$i=1,2,...,C_{B}$ and
$\mathbf{w}_{M}^{j}=\mathbf{a}_{r}(\theta_{r},\frac{2j-C_{M}-1}{2C_{M}}\pi)$,
$i=1,2,...,C_{M}$, where $\theta_{t}=92^{\circ}$ and $\theta_{r}=88^{\circ}$
according to the heights of BS and MS. Similar to [3], we select 343 candidate
beam pairs, which have become optimal more than three times in the whole
dataset. We then divide the dataset based on sequences, i.e., we obtain the
training samples and the test samples from different sequences. The training
set contains 801 sequences with 13163 LOS samples and 4759 NLOS samples, while
the test set contains 492 sequences with 3725 LOS samples and 3245 NLOS
samples.

The number of channels, layers, and stacks of the hourglass networks are 64,
4, and 2, respectively; $\beta$ is set to 0.8. We utilize batch normalization
for each convolutional layer or residual block. The details are presented in
Fig. 2.

###  IV-B Numerical Results

TABLE III: Performance Comparision Between Different Methods. Model | Proposed | LIDAR | VBALA | Location
---|---|---|---|---
A(1) | 57.82% | 44.73% | 47.68% | 37.06%
T(1) | 80.14% | 67.69% | 72.16% | 57.47%
A(5) | 75.12% | 69.74% | 72.04% | 65.81%
T(5) | 90.14% | 86.94% | 88.61% | 83.97%
Figure 4: Top-$K$ beam selection accuracy and throughput ratio of the three
methods for LOS and NLOS test samples.

We compare the proposed method with the LIDAR-based method [5] and the vision-
based beam alignment when the MS location is available (VBALA) [8]. The LIDAR-
based method performs beam selection from the point cloud and the MS location,
while VBALA predicts the beam pairs from the MS location and the vehicle
distribution feature captured by the object detection. For the LIDAR-based
method, we equip MS with a LIDAR at 1 m above the roof center of the vehicle,
which has a coverage radius of 100 m. For VBALA, we set the grid size $L_{G}=$
12 m and $W_{G}=$ 2 m. Furthermore, we adopt a location-based beam selection
method [6], which only uses the location of the MS relative to the BS, as a
baseline. For fairness, all the methods are trained for 30 epochs, using the
same beam prediction loss function $L_{pd}$ in (8).

Similar to [5], we evaluate the top-$K$ accuracy $A(K)$ with the throughput
ratio $T(K)$. As shown in Fig. 4 and TABLE III, the accuracy and the
throughput ratio of the proposed method significantly outperform the
baselines. Meanwhile, we compare the methods for LOS and NLOS test samples.
Though the proposed method can outperform the baseline methods in LOS cases,
its major advantage is exhibited in NLOS cases, where the top-1 accuracy is
approximately 13.3% higher than VBALA. Since the environment semantics can be
extracted in an explicit manner, the proposed method has a deeper perception
of multipath effects and blockages, leading to a better NLOS performance.

(a) The precision-recall curves. The distance threshold is 3 pixels, which
corresponds to an angular misalignment of approximately 2.1∘.

(b) Visualization of spatial distribution heatmaps.

(c) Visualization of strength heatmaps.

Figure 5: The performance of semantics extraction. Assume the number of
groundtruth, predicted, and successfully detected scatterers are
$N_{g},N_{p},N_{d}$. The precision and the recall are defined as $N_{d}/N_{p}$
and $N_{d}/N_{g}$ respectively.

Moreover, we note that the threshold $P_{th}$ can affect the accuracy and the
storage overhead of the proposed method. Specifically, we define the averaged
number of effective scatterers per camera view as $N_{E}$, which reflects the
storage requirement of the training dataset. As shown in TABLE IV, the
proposed method achieves the highest accuracy at $P_{th}=-$10 dB. In addition,
when $P_{th}<-$5 dB, the consumption of storage resources is significantly
reduced without a major loss of accuracy.

TABLE IV: Performance Comparision Between Different Thresholds. $P_{th}$ | -1 dB | -5 dB | -10 dB | -15 dB
---|---|---|---|---
A(1) | 56.84% | 57.55% | 57.82% | 57.76%
T(1) | 79.27% | 80.06% | 80.14% | 79.71%
$N_{E}$ | 0.328 | 0.495 | 0.793 | 1.231

Furthermore, we evaluate the effectiveness of the environment semantics
extraction in Fig. 5. We obtain the maximum points on the predicted spatial
distribution heatmaps by non-maximum suppression. If the distance between the
predicted maximum point and the groundtruth is within a threshold, then the
corresponding scatterer will be perceived as detected. Besides, since the
stronger paths can more significantly influence the channel, the corresponding
scatterers should be extracted more precisely. Therefore, we adjust the
threshold $P_{th}$, train the corresponding semantics extraction model, and
evaluate the precision-recall curves. The proposed method is shown to
successfully extract the effective scatterers from the input images,
especially for the stronger ones.

##  V Conclusion

We have proposed a vision-aided environment semantics extraction method and
applied it to mmWave beam selection. We define the environment semantics as
the spatial distribution of the effective scatterers, which are represented as
the semantic heatmaps, and we extract them via keypoint detection techniques.
Compared with the existing vision-based or LIDAR-based methods, the proposed
method shows deeper insight into the wireless propagation environment.
Simulation results show that the proposed method significantly outperforms the
existing methods in accuracy and leads to lower overhead for beam selection.

## References

  * [1] V. Venkateswaran and A.-J. van der Veen, “Analog beamforming in MIMO communications with phase shift networks and online channel estimation,” IEEE Trans. Signal Process., vol. 58, no. 8, pp. 4131-4143, Aug. 2010.
  * [2] J. Song, J. Choi, and D. J. Love, “Common codebook millimeter wave beam design: Designing beams for both sounding and communication with uniform planar arrays,” IEEE Trans. Commun., vol. 65, no. 4, pp. 1859-1872, Apr. 2017.
  * [3] A. Klautau, N. González-Prelcic, and R. W. Heath, “LIDAR data for deep learning-based mmWave beam-selection,” IEEE Wireless Commun. Lett., vol. 8, no. 3, pp. 909-912, Jun. 2019.
  * [4] M. B. Mashhadi, M. Jankowski, T.-Y. Tung, S. Kobus, and D. Gündüz, “Federated mmWave beam selection utilizing LIDAR data,” IEEE Wireless Commun. Lett., vol. 10, no. 10, pp. 2269-2273, Oct. 2021.
  * [5] M. Zecchin, M. B. Mashhadi, M. Jankowski, D. Gündüz, M. Kountouris, and D. Gesbert, “LIDAR and position-aided mmWave beam selection with non-local CNNs and curriculum training,” IEEE Trans. Veh. Technol., vol. 71, no. 3, pp. 2979-2990, Mar. 2022.
  * [6] G. Charan, T. Osman, A. Hredzak, N. Thawdar, and A. Alkhateeb, “Vision-position multi-modal beam prediction using real millimeter wave datasets,” 2022 IEEE Wireless Commun. Netw. Conf. (WCNC), 2022, pp. 2727-2731.
  * [7] W. Xu, F. Gao, J. Zhang, X. Tao, and A. Alkhateeb, “Deep learning based channel covariance matrix estimation with user location and scene images,” IEEE Trans. Commun., vol. 69, no. 12, pp. 8145-8158, Dec. 2021.
  * [8] W. Xu, F. Gao, X. Tao, J. Zhang, and A. Alkhateeb, “Computer vision aided mmWave beam alignment in V2X communications,” IEEE Trans. Wireless Commun., 2022
  * [9] G. Charan, M. Alrabeiah, and A. Alkhateeb, “Vision-aided 6G wireless communications: Blockage prediction and proactive handoff,” IEEE Trans. Veh. Technol., vol. 70, no. 10, pp. 10193-10208, Oct. 2021.
  * [10] A. M. Sayeed, “Deconstructing multiantenna fading channels,” IEEE Trans. Signal Process., vol. 50, no. 10, pp. 2563-2579, Oct. 2002.
  * [11] A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks for human pose estimation,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2016, pp. 483-499.
  * [12] A. Newell, Z. Huang, and J. Deng, “Associative embedding: End-to-end learning for joint detection and grouping,” in Proc. Adv. Neural Inf. Process. Syst., 2017, pp. 2274-2284.
  * [13] H. Law and J. Deng, “CornerNet: Detecting objects as paired keypoints,” in Proc. Eur. Conf. Comput. Vis. (ECCV), Sep. 2018, pp. 734-750.
  * [14] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, “CenterNet: Keypoint triplets for object detection,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2019, pp. 6569-6578.
  * [15] B. Mildenhall, P.P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NERF: Representing scenes as neural radiance fields for view synthesis,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2020, pp. 405-421.
  * [16] R. Tucker and N. Snavely, “Single-View view synthesis with multiplane images,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 548-557.
  * [17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016, pp. 770-778.
  * [18] https://carla.org
  * [19] https://www.eclipse.org/sumo
  * [20] https://www.remcom.com/wireless-insite-em-propagation-software
  * [21] https://github.com/wfy3042/Environment-Semantics-Extraction-Dataset


# Ultraluminous X–ray sources are beamed

Jean–Pierre Lasota1,2 & Andrew King3,4,5
1 Institut d’Astrophysique de Paris, CNRS et Sorbonne Université, UMR 7095,
98bis Bd Arago, 75014 Paris, France
2 Nicolaus Copernicus Astronomical Center, Polish Academy of Sciences, ul.
Bartycka 18, 00-716 Warsaw, Poland
3 Astrophysics Group, School of Physics & Astronomy, University of Leicester,
Leicester LE1 7RH, UK
4 Astronomical Institute Anton Pannekoek, University of Amsterdam, Science
Park 904, 1098 XH Amsterdam, Netherlands
5 Leiden Observatory, Leiden University, Niels Bohrweg 2, NL-2333 CA Leiden,
Netherlands

###### Abstract

We show that magnetar models for ultraluminous X–ray sources (ULXs) have
serious internal inconsistencies. The magnetic fields required to increase the
limiting luminosity for radiation pressure above the observed (assumed
isotropic) luminosities are completely incompatible with the spin–up rates
observed for pulsing ULXs. We note that at least one normal Be–star + neutron
star system, with a standard (non–magnetar) field, is observed to become a ULX
during a large outburst and return to its previous Be–star binary state
afterwards. We note further that recent polarimetric observations of the
well–studied binary Cyg X–3 reveal that it produces strong emission directed
away from the observer, in line with theoretical predictions of its total
accretion luminosity from evolutionary arguments. We conclude that the most
likely explanation for ULX behaviour involves radiation beaming by accretion
disc winds. A large fraction of X–ray binaries must pass through a ULX state
in the course of their evolution.

###### keywords:

accretion, accretion discs – black hole physics – binaries: close – pulsars:
general – X-rays: binaries.

##  1 Introduction

Ultraluminous X-ray sources (ULXs) are defined by the two conditions

(i) apparent luminosities (assumed isotropic)
$L_{X}>10^{39}\,\mathrm{erg}\,\mathrm{s}^{-1}$, and

(ii) locations away from galaxy centres.

These restrictions select a group of objects not straightforwardly explained
either as accreting stellar–mass binaries, or as more massive accretors.
Condition (i) requires $L_{X}$ to exceed the Eddington luminosity for a
$10{\rm M_{\odot}}$ black hole, i.e.

${L_{\rm Edd}}=1.3\times 10^{38}m\,{\rm erg\,s^{-1}},$ (1)

with $m={M}/{{\rm M_{\odot}}}=10$, which implies a corresponding Eddington
accretion rate

$\displaystyle{\dot{M}_{\rm Edd}}\equiv\frac{{L_{\rm Edd}}}{\eta c^{2}}$
$\displaystyle=1.4\times 10^{18}\eta_{0.1}^{-1}m\,{\rm g\,s^{-1}}$ (2)
$\displaystyle=2.2\times 10^{-8}\,\eta_{0.1}^{-1}m\,{\rm M_{\odot}}\rm
yr^{-1},$ (3)

where $\eta=0.1\eta_{0.1}$ is the radiative efficiency of accretion. Condition
(ii) rules out the central massive black holes in galaxies.

ULXs were identified as a separate class of objects at the end of the previous
millenium (Colbert & Mushotzky, 1999). By now, only two models of ULX
behaviour remain under serious consideration.

The older of these two current models for ULX behaviour is disc–wind beaming
(King et al., 2001). This asserts that the assumption of isotropic emission
made in computing $L_{X}$ from observations is not valid for binary systems
transferring mass at rates $\dot{m}{\dot{M}_{\rm Edd}}$, with $\dot{m}\gg 1$,
because in this case radiation pressure expels most of the transferred mass in
quasispherical winds which are opaque except along narrow channels along the
accretion disc axis (Shakura & Sunyaev, 1973). This means that most of the
emitted accretion luminosity $\sim{L_{\rm Edd}}$ is beamed along these
channels. ULXs are sources where the observer lies in one of the beams: the
effect is that the apparent (assumed isotropic) luminosity inferred is

$L_{\rm sph}\sim\frac{1}{b}{L_{\rm Edd}}\gg{L_{\rm Edd}},$ (4)

where the total solid angle of the two channnels is $4\pi b$.

The more recent model for ULX behaviour, which we shall refer to as the
‘magnetar model’, was inspired by the discovery by Bachetti et al. (2014) that
the source ULX–2 in the galaxy M82 is pulsed. This implies that the accretor
is a magnetized neutron star111This discovery had the effect of making an
early model for ULXs invoking accretion on to more massive
(‘intermediate–mass’, or ‘IMBH’) black holes relatively unattractive.. The
magnetar model asserts that unusually strong surface fields of neutron–star
accretors reduce the electron scattering opacity defining the Eddington
luminosity, making ${L_{\rm Edd}}$ numerically larger: the high luminosities
of ULXs are actually sub–critical in this picture. For magnetar–strength
fields ($\gtrsim 10^{14}$ G) the modified ${L_{\rm Edd}}$ exceeds the
assumed–isotropic luminosity $L_{X}$.

We note that the two models imply fundamentally different significances for
the ULX phenomenon. Disc–wind beaming asserts that the ULX state is one that a
large fraction of otherwise standard X–ray binaries pass through during a
particular phase of their evolution, whereas the magnetar hypothesis reduces
the ULX class to a relatively small subset of these systems defined by very
strong magnetic fields.

The aim of this paper is to evaluate recent evidence allowing a clear decision
between beaming and strong magnetic fields as the basic cause of ULX
behaviour.

##  2 Beaming

The suggestion by King et al. (2001) of beaming as the explanation for the
high apparent luminosities of ULXs was motivated by study of the X–ray binary
Cyg X–2 (King & Ritter, 1999). The neutron star in this system has evidently
survived the companion star attempting to transfer $\sim 3{\rm M_{\odot}}$ to
it at a highly super–Eddington rate, without retaining more than a small
fraction of it.

This corresponds closely to the picture of how a disc deals with a
super–Eddington mass rate suggested by Shakura & Sunyaev (1973). A
radiation–pressure powered wind from the disc surface keeps the disc accretion
rate at the local Eddington limit corresponding to each disc radius. This
raises the true total emitted accretion luminosity only by a logarithmic
factor, to

$L_{\rm acc}\simeq{L_{\rm Edd}}[1+\ln\dot{m}],$ (5)

so that even a huge (by X–ray binary standards) accretion rate of $\sim
10^{4}{\dot{M}_{\rm Edd}}$ would give a total accretion luminosity of only
$10{L_{\rm Edd}}$, i.e $\sim 2\times 10^{39}\,\mathrm{erg}\,\mathrm{s}^{-1}$
for a $1.4{\rm M_{\odot}}$ neutron star.

But importantly the emission is now highly anisotropic: the outflowing wind is
densest near the radius at which the full Eddington luminosity is attained,
and has a large optical depth both along the disc plane and in the vertical
direction. Thus most of the disc radiation emitted within the wind region
diffuses by scattering until it is able to escape through the central open
funnels parallel to the disc axis. Since the funnel is tall and thin and has
scattering walls, the escaping radiation is beamed by a factor $b\ll 1$, so
that the apparent luminosity deduced by an observer in the beam, who assumes
the luminosity to be isotropic, is

$L_{\rm app}=\frac{1}{b}{L_{\rm Edd}}[1+\ln\dot{m}]\gg{L_{\rm Edd}}.$ (6)

King (2009) showed that for $\dot{m}\gg 1$, the observed correlation
$L_{bb}\propto T_{bb}^{-4}$ between ULX soft X–ray blackbody luminosity and
temperature implies that

$b\simeq\frac{73}{\dot{m}^{2}}.$ (7)

This agrees with deductions from simple accretion disc theory, as conditions
far from the disc centre are set by the mass supply rate, while those near the
disc centre all converge to what is set by a near–Eddington central accretion
rate.

King & Lasota (2020) noted that when the accretor is a magnetized neutron star
its magnetic axis is not necessarily aligned with the disc (i.e. funnel) axis,
and it is very common for the neutron star spin to be misaligned from the
binary orbit defining the accretion disc plane. When these three axes are not
aligned the system appears as a pulsing ULX, or PULX. For a neutron–star spin
axis strongly misaligned from the central disc axis at the spherization
radius, large polar caps produce the sinusoidal pulse light curves observed in
PULXs since a significant part of the pulsed emission can escape without
scattering, giving a large pulse fraction. Using this disc–wind–beaming model
King et al. (2017); King & Lasota (2019, 2020) (see also King et al. 2023)
were able to obtain self–consistent sets of parameters for the 10 known PULXs,
finding magnetic fields in the range of $\sim 2\times 10^{10}-10^{13}$G,
mass–transfer rates $\dot{m}$ between $\sim 10$ and $\sim 100$, and beaming
factors from $\sim 0.01$ to $\sim 0.5$.

##  3 Magnetar models

But as we noted above, soon after the discovery of the first PULX a different
explanation of the apparent super–Eddington luminosities observed in these
X–ray sources became possible. Dall’Osso et al. (2015); Eksi et al. (2015))
assumed that the PULX magnetic fields had magnetar ($\gtrsim 10^{14}$ G)
fieldstrengths. These substantially reduce the scattering cross-sections and
so increase the critical luminosity at which the radiation pressure force
equals the pull of gravity. In this scenario PULX luminosities are above the
usual Eddington luminosity, but actually sub–critical, so that accretion
proceeds in the same way as in other X–ray pulsars.

Indeed, very strong magnetic fields lower the Thomson and Compton scattering
opacity (Canuto et al., 1971; Herold, 1979) for photons with energies
$E_{\gamma}$ lower than the cyclotron frequency $E_{\rm cyc}$:

$\displaystyle\frac{\sigma_{B1}}{\sigma_{T}}\approx$
$\displaystyle\sin^{2}\theta+\left(\frac{E_{\gamma}}{E_{\rm
cyc}}\right)^{2}\cos^{2}\theta$ (8)
$\displaystyle\frac{\sigma_{B2}}{\sigma_{T}}\approx$
$\displaystyle\left(\frac{E_{\gamma}}{E_{\rm cyc}}\right)^{2},\,\rm
for\,\frac{E_{\gamma}}{E_{\rm cyc}}\ll 1,$ (9)

where indices 1 and 2 correspond to the two linear photon polarizations,
$\sigma_{T}$ is the Thomson cross-section and $\theta$ is the angle between
the directions of the magnetic field and light propagation. The opacities
depend on the photon polarization, but as shown by Paczynski (1992) their
Rosseland means differ at most by a factor 2, depending on the angle between
the direction of the photon propagation and the field lines. Therefore in the
presence of a very strong magnetic field, the critical luminosity
corresponding to the equality of the radiation pressure and gravitational
forces can be written as

${L_{\rm crit}}\approx 2B_{12}^{4/3}\left(\frac{g}{2\times 10^{14}\rm
cm\,s^{-2}}\right)^{-1/3}{{L_{\rm Edd}}},$ (10)

where $g=GM/R^{2}$ (Paczynski, 1992). Thus in this picture the apparent
(assumed isotropic) PULX luminosities $\gtrsim
10^{40}\,\mathrm{erg}\,\mathrm{s}^{-1}$ must be emitted by a plasma permeated
by magnetar–strength fields $>10^{14}\rm G$.

Although at first sight attractive, the idea of magnetars in PULXs faces the
difficulty that these very strongly magnetized neutron stars have never been
observed in binary systems (see King & Lasota 2019; King et al. 2023 and
references therein). Accepting it requires belief in a cosmic conspiracy
making them detectable in binaries only when these have high mass transfer
rates. We shall see in the next Section that this idea disagrees with
observations in any case.

We now know that out of the $\sim$ 1800 observed ULXs (see King et al. 2023
and references therein) at least 10 contain magnetized neutron stars, detected
through their periodic pulses (PULXs). Four of them are transient: they are
members of Be–X binary systems, which become X–ray sources when the eccentric
orbit of the compact companion (in most if not all cases a neutron star) of
the massive Be star crosses its circumstellar disc. In most cases this
disc–crossing produces sub–Eddington–luminosity outbursts (called “Type I”),
but from time to time, most probably because of von Zeipel-Kozai-Lidov
oscillations of the circumstellar disc (Martin et al., 2014), it results in a
giant (super–Eddington; ‘Type II’) outburst.

Swift/XRT observations of galaxies NGC 4945, NGC 7793 and M81 suggest that
although persistent ULXs dominate the high end of galaxy luminosity functions,
the number of systems emitting ULX luminosities are probably dominated by
transient sources. These transients are most probably not Be–X systems
(Brightman et al., 2023).

##  4 Magnetic fields in ULXs cannot have magnetar strengths

There is a simple physical argument that rules out the presence of magnetars
in observed PULXs. The argument is based on the value of their spin-up rate
$\dot{\nu}$ ($\nu$ is the pulsar’s spin frequency).

After the discovery of the first PULX M82 ULX-2, Kluźniak & Lasota (2015)
pointed out that it differs from other X-ray pulsars (XRPs) not only through
its higher luminosity but also in its extremely high spin–up rate.

Figure 1: The $L_{39}$ – $\dot{\nu}_{-10}$ diagram for XRPs and PULXs. Red
dots: the ten PULXs with known spin-up rates. Blue diamonds: selected (for
comparison) sub–Eddington–luminosity X-ray pulsars (For details see King et
al. 2023)

.

It is immediately obvious that both ‘normal’ X–ray systems and PULXs lie on
exactly the same strong correlation between spin–up rate $\dot{\nu}$ and X–ray
luminosity $L_{X}$ – see Fig 1. This correlation extends more than seven
orders of magnitude in luminosity, and arises because the spin–up results from
the accretion torque on the neutron star:

$\dot{\nu}=\frac{\dot{J}(R_{M})}{2\pi I}=\frac{\dot{M}(GMR_{M})^{1/2}}{2\pi
I}\propto\dot{M}^{6/7}\mu^{2/7},$ (11)

where $R_{\rm M}\propto\dot{M}^{-2/7}\mu^{4/7}$ (from Eq. 12) is the
magnetospheric radius, $\mu=BR^{3}$ the neutron star’s magnetic moment (with
$B$ the field and $R$ the neutron–star radius) and $I$ is the neutron star’s
moment of inertia.

The magnetospheric radius is defined by the equation (Frank et al., 2002)

$R_{\rm M}=2.6\times
10^{8}q\,\left(\frac{\dot{M}}{10^{17}\,\mathrm{g}\,\mathrm{s}^{-1}}\right)^{-2/7}\left(\frac{M}{{\rm
M_{\odot}}}\right)^{-3/7}\mu_{30}^{4/7}\,\rm cm,$ (12)

where the factor $q\sim 1$ takes into account the geometry of the accretion
flow at the magnetosphere and $\mu=10^{30}\mu_{30}\rm Gcm^{3}$.

Assuming $M\approx 1{\rm M_{\odot}}$, $q\approx 1$, $I=10^{45}\rm g\,cm^{2}$
and using Eq. (12), Eq. (11) gives

$\dot{M}\approx 5.7\times
10^{18}\dot{\nu}^{7/6}_{-10}\mu_{30}^{-1/3}\,\mathrm{g}\,\mathrm{s}^{-1}$ (13)

as the accretion rate required to spin up a magnetised neutron star at the
rate $\dot{\nu}$.

Now we can calculate the luminosity produced by this accretion rate.
Supercritical luminosities are not proportional to the accretion rate (see Eq.
5). But very strong magnetic fields make the critical luminosity much larger
than the Eddington value, i.e. $L_{\rm crit}\gg{L_{\rm Edd}}$ (see Eq. 10). So
for $L_{\rm crit}>L_{X}\gtrsim{L_{\rm Edd}}$ the standard formula
$L_{X}=0.1\dot{M}c^{2}$ applies, even though $L_{X}$ exceeds the usual
Eddington value.

Then for magnetar PULXs ($\mu\gtrsim 10^{31}{\rm Gcm^{3}}$) we get from Eq.
(13) the luminosity

$L_{X}\approx 2\times
10^{38}\dot{\nu}^{7/6}_{-10}\mu_{31}^{-1/3}\,\mathrm{erg}\,\mathrm{s}^{-1}\approx{L_{\rm
Edd}}.$ (14)

But in deriving this equation we assumed $L\gtrsim L_{\rm crit}\gg{L_{\rm
Edd}}$, which would require a much smaller field (i.e. $\mu_{31}\ll 1$).222Eq.
(14) explains why sub–Eddington–luminosity XRPs ($\mu_{31}\sim 0.001-1$) have
$\dot{\nu}<10^{-10}\rm s^{-2}$. This contradiction shows that magnetars cannot
be present in systems with both $L_{X}>10^{39}\,\mathrm{erg}\,\mathrm{s}^{-1}$
and $\dot{\nu}\gtrsim 10^{-10}\rm s^{-2}$.

In other words, PULXs cannot contain magnetars.

This in turn means that the super–Eddington luminosity observed in PULXs is
not intrinsic, and must presumably be anisotropic, i.e. beamed.

Importantly, since $L_{\rm crit}\sim B^{4/3}$, in a dipole field the critical
luminosity decreases radially outwards as $R^{-4}$. So at radius $\sim 100$
stellar radii all of the cross–section suppression is lost, well inside the
magnetosphere. Then a hyper–Eddington luminosity emitted near the neutron–star
surface would blow away all of the gas in the upper part of the accretion
column, thus cutting off the mass supply supposedly producing the posited
hyper–Eddington emission. This rules out the interpretion of the CRSF observed
in the magnetized, non–pulsing ULX-8 in M51 as an effect of protons orbiting a
$9\times 10^{14}$G magnetic field, as envisaged by Brightman et al. (2018),
and provides another strong argument against the presence of magnetars in
ULXs333We are grateful to the anonymous referee of the present paper for
suggesting this line of argument.

##  5 Magnetic Fields in PULXs

We conclude from the last Section that neutron stars in PULXs have magnetic
fields spanning the same range as the usual XRPs – from $10^{8}$ G to several
$10^{13}$ G (Revnivtsev & Mereghetti, 2018). They are evidently normal XRPs
observed in a special phase of the evolution of their parent binary systems,
as is implicit in the original suggestion by King et al. (2001). We can see
examples of this in real time in observations of Be–star PULXs. These are
normal XRPs for most of their lifetimes, and become PULXs only during their
occasional giant outbursts. This allows one to follow the transformation of an
XRB into a PULX and its return to ‘normal’ again.

Figure 2: X-ray derived pulsed period history of SMC X-3. Black diamonds and
red stars denote RXTE period detections above the 99 and 99.99 per cent
confidence levels respectively. Blue triangles denote Swift detections of the
pulse period during the current outburst. A single XMM–Newton detection at MJD
54274 was found in the literature and is denoted by a green square. (From
Townsend et al. 2017.)

The best studied case is that of the binary SMC X-3. It shows that as the
system enters the ULX phase, the neutron–star spin evolution becomes dominated
by the accretion torque, as assumed in Eq. (11). Between giant outbursts this
sources is an XRP which spins down. In Fig. 2 (Townsend et al., 2017) this
corresponds to the time right up to the beginning of a giant outburst, on MJD
57599; then a significant spin–up is observed. From SMC X-3’s long–term spin
history Townsend et al. (2017) deduce that the angular momentum transferred by
accretion during the 5–month giant outburst was larger than the total angular
momentum lost by magnetic braking over the previous 18 years of the spin–down
phase. The long–term spin–down rate of SMC X-3 is about 500 times lower than
the rate of spin–up observed during the giant outburst, showing that the
torques acting during this outburst are far larger than during the
out–of–outburst phases. During weaker (Type I) outbursts, the spin period
continues to increase, but during the giant outburst the spin–up rate is
tightly correlated with the X–ray luminosity through the super–Eddington phase
(Weng et al., 2017), in agreement with Eq. (11). This means that in PULXs the
spin–up rate is strongly correlated with the X–ray luminosity both in time and
over the population. There are no magnetars in PULXs.

##  6 Cyg X-3: the second hidden ULX in the Galaxy

Quite recently, Veledina et al. (2023) performed X–ray polarimetry indicating
‘unambiguously’ that the Wolf–Rayet X-ray binary Cyg X–3, consisting of a
helium star transferring mass to a black hole on its thermal timescale, is a
ULX with a beaming factor444We use here the symbol $b$ as defined in King
(2009); by contrast Veledina et al. 2023 use $b$ to denote his $1/b$.
$b\approx 1/65$, but seen from the side. This system is assumed to contain a
black hole. Earlier inferred examples of ‘sideways’ ULXs notably include the
extreme source SS433 (cf Begelman et al. 2006; King & Muldrew 2016).

From Eq. (7) we find that this requires an Eddington factor $\dot{m}\simeq
69$. This is consistent with the estimates of the mass transfer rate found by
Lommen et al. (2005) on evolutionary grounds. Together with the similar
estimates for SS433 (Begelman et al. 2006; King & Muldrew 2016) this appears
to be explicit confirmation that compact binaries with mass transfer rates
exceeding the Eddington rate produce beamed emission, as first suggested by
King et al. (2001). Moreover, the estimate (7) appears to be in reasonable
agreement with observation.

The good match between the luminosities of some ULX nebulae and the luminosity
of their ULX irradiators has been used as an argument against strong
geometrical beaming in these ultraluminous sources since in these cases the
nebula would see an isotropic emission. However, the sources in question are
spectrally soft and it is not surprising that the irradiating luminosity
inferred from photoionisation modelling is consistent with the observed
luminosity (see King et al. 2023 for a detailed discussion).

##  7 Conclusion

We have shown that magnetar models for ULX behaviour have serious internal
inconsistencies. In particular the fieldstrengths required to increase the
radiation pressure luminosity limit above the observed (assumed isotropic)
luminosities are completely incompatible with the spinup rates observed for
PULXs. In addition we note that at least one normal Be–star system, with a
standard (non–magnetar) field, is observed to become a ULX during a large
outburst.

In contrast, recent polarimetric observations of the well–studied binary Cyg
X–3 reveal that it produces strong emission beamed away from the observer.

We conclude that ULXs are beamed.

##  8 Data Availability

No new data were generated or analysed in support of this research.

## References

  * Bachetti et al. (2014) Bachetti M., et al., 2014, Nature, 514, 202
  * Begelman et al. (2006) Begelman M. C., King A. R., Pringle J. E., 2006, MNRAS, 370, 399
  * Brightman et al. (2018) Brightman M., et al., 2018, Nature Astronomy, 2, 312
  * Brightman et al. (2023) Brightman M., et al., 2023, ApJ, 951, 51
  * Canuto et al. (1971) Canuto V., Lodenquai J., Ruderman M., 1971, Phys. Rev. D, 3, 2303
  * Colbert & Mushotzky (1999) Colbert E. J. M., Mushotzky R. F., 1999, ApJ, 519, 89
  * Dall’Osso et al. (2015) Dall’Osso S., Perna R., Stella L., 2015, MNRAS, 449, 2144
  * Eksi et al. (2015) Eksi K. Y., Andac I. C., Cikintoglu S., Gencali A. A., Gungor C., Oztekin F., 2015, MNRAS, 448, L40
  * Frank et al. (2002) Frank J., King A., Raine D. J., 2002, Accretion Power in Astrophysics: Third Edition. CUP
  * Herold (1979) Herold H., 1979, Phys. Rev. D, 19, 2868
  * King (2009) King A. R., 2009, MNRAS, 393, L41
  * King & Lasota (2019) King A., Lasota J.-P., 2019, MNRAS, 485, 3588
  * King & Lasota (2020) King A., Lasota J.-P., 2020, MNRAS, 494, 3611
  * King & Muldrew (2016) King A., Muldrew S. I., 2016, MNRAS, 455, 1211
  * King & Ritter (1999) King A. R., Ritter H., 1999, MNRAS, 309, 253
  * King et al. (2001) King A. R., Davies M. B., Ward M. J., Fabbiano G., Elvis M., 2001, ApJ, 552, L109
  * King et al. (2017) King A., Lasota J.-P., Kluźniak W., 2017, MNRAS, 468, L59
  * King et al. (2023) King A., Lasota J.-P., Middleton M., 2023, New Astron. Rev., 96, 101672
  * Kluźniak & Lasota (2015) Kluźniak W., Lasota J.-P., 2015, MNRAS, 448, L43
  * Lommen et al. (2005) Lommen D., Yungelson L., van den Heuvel E., Nelemans G., Portegies Zwart S., 2005, A&A, 443, 231
  * Martin et al. (2014) Martin R. G., Nixon C., Lubow S. H., Armitage P. J., Price D. J., Doğan S., King A., 2014, ApJ, 792, L33
  * Paczynski (1992) Paczynski B., 1992, Acta Astron., 42, 145
  * Revnivtsev & Mereghetti (2018) Revnivtsev M., Mereghetti S., 2018, in Beskin V. S., Balogh A., Falanga M., Lyutikov M., Mereghetti S., Piran T., Treumann R. A., eds, , Vol. 54, The Strongest Magnetic Fields in the Universe. Springer, pp 299–320, doi:10.1007/978-1-4939-3550-5˙9
  * Shakura & Sunyaev (1973) Shakura N. I., Sunyaev R. A., 1973, A&A, 24, 337
  * Townsend et al. (2017) Townsend L. J., Kennea J. A., Coe M. J., McBride V. A., Buckley D. A. H., Evans P. A., Udalski A., 2017, MNRAS, 471, 3878
  * Veledina et al. (2023) Veledina A., et al., 2023, arXiv e-prints, p. arXiv:2303.01174
  * Weng et al. (2017) Weng S.-S., Ge M.-Y., Zhao H.-H., Wang W., Zhang S.-N., Bian W.-H., Yuan Q.-R., 2017, ApJ, 843, 69