# Dialog-to-Actions: Building Task-Oriented Dialogue System via Action-Level
Generation

Yuncheng Hua  ,  Xiangyu Xi  0000-0002-4238-5071 Meituan Group, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>,  Zheng Jiang  Meituan Group, China
<EMAIL_ADDRESS>,  Guanwei Zhang  Meituan Group, China
<EMAIL_ADDRESS>,  Chaobo Sun  Meituan Group, China
<EMAIL_ADDRESS>,  Guanglu Wan  Meituan Group, China
<EMAIL_ADDRESS>and  Wei Ye  Peking University, ChinaChina
<EMAIL_ADDRESS>
###### Abstract.

End-to-end generation-based approaches have been investigated and applied in
task-oriented dialogue systems. However, in industrial scenarios, existing
methods face the bottlenecks of reliability (e.g., domain-inconsistent
responses, repetition problem, etc) and efficiency (e.g., long computation
time, etc). In this paper, we propose a task-oriented dialogue system via
action-level generation. Specifically, we first construct dialogue actions
from large-scale dialogues and represent each natural language (NL) response
as a sequence of dialogue actions. Further, we train a Sequence-to-Sequence
model which takes the dialogue history as the input and outputs a sequence of
dialogue actions. The generated dialogue actions are transformed into verbal
responses. Experimental results show that our light-weighted method achieves
competitive performance, and has the advantage of reliability and efficiency.

Task-Oriented Dialogue System, Action-Level Generation, Dialog-to-Actions

††ccs: Computing methodologies Discourse, dialogue and pragmatics

##  1\. Introduction

Recently, the end-to-end generation-based methods that directly output
appropriate NL responses or API calls have been deeply investigated in task-
oriented chatbots (Byrne et al., 2021; Yang et al., 2021; Lin et al., 2021;
Jang et al., 2021; Thoppilan et al., 2022), and have been proven valuable for
real-world business, especially after-sale customer services (Li et al., 2017;
Yan et al., 2017; Zhu, 2019; Lee et al., 2019; Zhu et al., 2020; Song et al.,
2021; Acharya et al., 2021; Sun et al., 2021; Xi et al., 2022). Based on the
large-scale pre-trained language models (Radford et al., 2019; Raffel et al.,
2020), generation-based methods have the advantage of simpler architecture and
anthropomorphic interaction. Despite the significant progress, we find these
token-level generation methods suffer from the following two limitations in
practical scenarios.

1\. The token-level generation methods have limited reliability, which is
essential for industrial task-oriented dialogue systems. Due to the pre-
trained language models’ characteristics, the models may generate responses
that are learned from the pre-training corpus. In certain cases, such
responses are meaningless and not semantically incoherent with the current
business domain, interrupting online interaction. Worse still, the models
occasionally get stuck in generating repetitive responses across multiple
turns (e.g., repeatedly enquiring the users for the same information). Above
issues are also widely observed by other researchers (See et al., 2019; Fu et
al., 2021) and
practitioners.111https://github.com/microsoft/DialoGPT/issues/45

2\. The token-level generation methods may fail to meet the efficiency
requirement of the industrial systems, especially with large decoding steps.
The long computation time of the token-level generation models leads to
unacceptable response latency of online dialogue systems, especially when the
model generates a sentence of length that exceeds a threshold (e.g., 1,544 ms
of T5 for a sentence of 30 words, as Figure 3 shows). Owing to the latency
problem, a large number of service requests may be suspended or blocked during
the peak period. Also, the computation resources (e.g., GPUs) required by the
aforementioned systems might be unaffordable for small companies.

Figure 1. The system architecture and dialogue sample. In (b), the dialogue
action and corresponding utterance segment are marked by the same color (e.g.,
“A226” and “I am really sorry”).

To address the above two problems, in this paper, we propose a task-oriented
dialogue system based on the action-level generation method. Inspired by Xi et
al. (2022), we represent responses with Dialogue Actions, i.e., a class of the
responses with unique and identical semantic meaning that can be automatically
obtained by clustering. While Xi et al. (2022) directly treats a whole
response as a specific dialogue action, we split one response into multiple
segments (Jin et al., 2004) and each segment can be mapped to a dialogue
action. In this way, each response is represented as a sequence of dialogue
actions. Given the dialogue context, a Seq2Seq model with an action-level
recurrent decoder is used to generate the sequence of dialogue actions.
Further, a frequency-based sampling method is used to compose the final
response, based on the generated sequence of dialogue actions. Since the core
component of our approach is the generation model which takes the _dialogue
context_ as the inputs and outputs _actions_ , our method is named as Dialog-
To-Actions (abbr. DTA). Compared with existing token-level generation-based
systems, our DTA has the advantage of 1) reliability, since the generated
natural language responses derive from the predefined dialogue actions; 2)
efficiency, since the decoding space (i.e., dialogue actions) and the decoding
steps are much smaller.

##  2\. Framework Description

###  2.1. Overview

We follow the workflow employed in the previous end-to-end task-oriented
dialogue systems (Byrne et al., 2021; Lin et al., 2021), where the system
takes the dialogue history as input, and generates a text string that either
serves as a verbal staff response to the user or API calls (e.g., information
inquiry, action execution, etc). When an API is invoked, the information
returned from the API will be incorporated into the system’s next response. A
dialogue sample following such system interaction life cycle can be found in
Figure 1 (b).

The key idea of our work is to generate dialogue actions and then compose a
verbal response. To do so, we first construct dialogue actions from large-
scale dialogues (Step 1) and represent each response as a sequence of dialogue
actions (Step 2), as Figure 1 (a) shows. A Seq2Seq model with an action-level
recurrent decoder is utilized to generate dialogue actions (Step 3), and the
generated actions are further used to compose the verbal response (Step 4). We
exemplify using the after-sale customer service of electric bike rental
business, where users and staffs communicate online through text messages. The
technical details are introduced as follows.

###  2.2. Step 1: Dialogue Action Construction

A dialogue action refers to a cluster of utterances or utterance fragments
that share identical semantic meaning and represent a common communicative
intention, for instance making a request or querying information. Xi et al.
(2022) views a group of utterances with identical semantic information as
dialogue action and selects a response corresponding to a specific staff
action. However, the oversimplified setting, i.e., abstracting a whole
utterance into an action, leads to relatively limited expressiveness and
scalability. To make the responses more targeted and flexible, we construct
dialogue actions based on utterance segments (of staff) rather than
utterances. Specifically, each utterance is divided into multiple segments by
a rule-based approach (Jin et al., 2004). Further, following Xi et al. (2022),
we exploit a two-stage method to cluster the segments. Specifically, ConSERT
(Yan et al., 2021) is utilized to generate representations for each utterance
segment, and K-means is then applied to cluster the segments. We choose the
number of clusters $K$ empirically to balance the purity and the number of the
clusters, and treat each cluster of segments as a dialogue action (e.g.,
$A_{1}$ and $A_{2}$ in Figure 1 (a)).

###  2.3. Step 2: Response Standardization

Response standardization aims to standardize the responses (from the large-
scale dialogues) by mapping each response to a sequence of dialogue actions.
Following Yu et al. (2021), we exploit a retrieval-based method, which
retrieves clustered segments that are most similar to the given input
utterance segment and label the input based on the corresponding clusters. As
Figure 2 shows, given an input segment $x$, we use BM25 to recall top $k$
segments $\\{u_{1},...,u_{k}\\}$ from all clustered segments. Further, we
exploit a BERT-based text similarity computation model $S$ to rerank the $k$
segments and select the segment $\hat{u}$ with the highest similarity to $x$,
denoting as:

(1)
$\hat{u}=\underset{u_{i}\in\\{u_{1},...,u_{k}\\}}{\mathrm{argmax}}S(x,u_{i})$

where $S(x,u_{i})$ refers to the similarity between $x$ and $u_{i}$. $x$ is
then annotated with the dialogue action $A_{i}$ that $\hat{u}$ belongs to.

Figure 2. The workflow of response standardization.

Furthermore, we record the correspondence between the dialogue actions and
utterance segments, as well as the frequencies of utterance segments in the
dialogues. Specifically, we employ a key-value dictionary $\mathfrak{D}$, in
which a key refers to a dialogue action $A_{i}$ while its value is a nested
dictionary where the mapping relationship between the unique segments
$\\{x_{1},...,x_{n}\\}$ and $A_{i}$ and the segments’ occurrence frequencies
$\\{f_{1},...,f_{n}\\}$ are recorded. The dictionary $\mathfrak{D}$ is used
for composing the verbal response (Step 4).

###  2.4. Step 3: Action Sequence Prediction

Given a dialogue $\mathcal{D}=\\{U_{1},S_{1},...,U_{T},S_{T}\\}$ as a set of
utterances exchanged between user ($U_{i}$) and staff ($S_{i}$) alternatively,
through above carefully-designed steps, each staff utterance $S_{i}$ is
represented as an action sequence $\mathcal{A}_{S_{i}}$. At the $m$-th turn,
given a dialogue history
$\mathcal{H}_{m}=\\{U_{m-w},S_{m-w},...,S_{m-1},U_{m}\\}$, we propose a
Seq2Seq model to produce staff responses $S_{m}$. We first resort to the
Seq2Seq model to output an action sequence of
$\mathcal{A}_{S_{m}}=(A_{1}^{S_{m}},A_{2}^{S_{m}},...,A_{k}^{S_{m}})$, where
$k$ denotes the length of the action sequence, and then use the action
sequence to form verbal response (in § 2.5).
Encoder Given the dialogue history $\mathcal{H}_{m}$, we sequentially
concatenate all the utterances in $\mathcal{H}_{m}$ and each staff utterance’s
corresponding action sequence $\mathcal{A}_{S_{i}}$, forming a token sequence
$(w_{1},...,w_{n})$. The Bi-LSTM model is used to encode the token sequence
into a sequence of continuous representations $\boldsymbol{H}$:

(2) $\boldsymbol{H}=(\boldsymbol{h}_{1},...,\boldsymbol{h}_{n})={\rm
BiLSTM}(w_{1},...,w_{n})$

Decoder Considering the efficiency requirement and small decoding space, we
use the Luong attention method and employ LSTM as the decoder to calculate
hidden state $\boldsymbol{s_{t}}$ at time-step $t$ as follows:

(3) $\boldsymbol{s_{t}}={\rm
LSTM}(\boldsymbol{s_{t-1}},g(\boldsymbol{y_{t-1}}),\boldsymbol{c_{t-1}})$

where $\boldsymbol{y_{t-1}}$ denotes the probability distribution over
dialogue action space at step $t$-1 and $g(\boldsymbol{y_{t-1}})$ denotes the
action has the highest probability. After obtaining hidden state
$\boldsymbol{s_{t}}$ and context vector $\boldsymbol{c_{t}}$, we generate
probability distribution at time-step $t$ as follows:

(4) $\boldsymbol{y_{t}}={\rm
Softmax}(\boldsymbol{W_{d}}[\boldsymbol{s_{t}};\boldsymbol{c_{t}}])$

where $\boldsymbol{W_{d}}$ is weight parameter. Given the ground-truth label
$y_{t}$ at time-step $t$, we use $p(y_{t}|y_{<t},\mathcal{H}_{m})$ to denote
the cross-entropy loss at step $t$ where $y_{<t}$ denotes the previously-
generated actions.

The optimization objective is defined as:

(5)
$\begin{split}L_{Gen}&=-\sum_{\mathcal{D}\in\mathcal{C}}\sum_{\mathcal{H}_{m}\in\mathcal{D}}\sum_{t=1}^{l_{m}}p(y_{t}|y_{<t},\mathcal{H}_{m})\end{split}$

where $\mathcal{C}$ denotes the set of dialogues and $l_{m}$ denotes the
length of the dialogue action sequence at the $m$-th turn of dialogue
$\mathcal{D}$.

###  2.5. Step 4: Response Generation

Based on the action sequence generated in § 2.4, we compose the verbal
response by selecting an utterance segment for each action and combining the
segments sequentially. Considering the segments with higher frequencies are
more likely to be the formal utterance that staff commonly use, we sample the
segments from $\mathfrak{D}$ (built in § 2.3) following the principle that the
higher the frequency, the more likely the segment to be selected. By doing
this, we ensure the quality as well as the diversity of the verbal responses.

##  3\. Experiments

###  3.1. Experimental Settings

####  3.1.1. Dataset

We perform an experiment with a Chinese online after-sale customer service of
electric bike rental business. In this scenario, the users may finish riding
but forgot to lock the bike, and thus request the staff to remotely lock the
bike and reduce the fees. The staff is required to judge whether the fee can
be reduced by checking the status of the order via the back-end APIs. We
collect the user-staff dialogues from the logs of online services for a week.
The data statistics are shown in Table 1. The dialogues are randomly split
into train, dev, and test sets with a ratio of 8:1:1. We construct 1,420
dialogue actions (§ 2.2), and each API call is treated as a dialogue action.
The dataset will be released online.

STAT TYPE | VALUE
---|---
Dialogs | 8,363
Total turns | 55,576
Avg. turns per dialog | 6.65
Dialogue Actions | 1,420
Table 1. Data statistics.

####  3.1.2. Baselines

To evaluate the effectiveness and efficiency of our method, we compare it with
the following state-of-the-art baselines: (1) LSTM which exploits a classical
LSTM-based sequence-to-sequence architecture (Sutskever et al., 2014); (2)
Transformer which uses Transformer (Vaswani et al., 2017) as encoder and
decoder; (3) CDiaGPT which is a GPT model pre-trained on a large-scale Chinese
conversation dataset (Wang et al., 2020); (4) T5 which is a Text-to-Text
Transfer Transformer (T5) model (Raffel et al., 2020) pretrained with the CLUE
Corpu.

####  3.1.3. Evaluation Metrics

To comprehensively evaluate the effectiveness of different models, we perform
both offline evaluation (i.e., on the dataset) and online evaluation (i.e.,
online A/B testing).

Offline Evaluation The models take a specific ground-truth conversation
history (i.e., context) as input and generate a response. Following Byrne et
al. (2021), we report the BLEU-4 score of each model in the test set.
Considering the API calls are highly important, we observe the generated API
calls in each turn and report the macro Precision (P), Recall (R), and
F1-Score (F1) of API calls.

Online Evaluation Following Xi et al. (2022), we deploy the models online and
perform A/B testing. For each model, 120 dialogues are randomly sampled. The
annotators who possess domain knowledge are required to perform the
satisfaction assessment by grading each dialogue as “Low”, “Medium”, or “High”
satisfaction degree. 222The grading criteria can be summarized as follows: (i)
Score “Low” denotes that Chatbot can not handle user requirements correctly.
(ii) Score “Middle” denotes that Chatbot can handle user requirements
correctly, but may generate a disfluent or incomplete response. (iii) Score
“High” denotes that Chatbot can handle user requirements correctly and
complete the conversation perfectly.

###  3.2. Main Results

The offline evaluation and online evaluation are shown in Table 3 and 2
respectively, from which we have the following observations: (1) Large-scale
pre-trained language models significantly improve the performance of token-
level generation models. For example, compared with the plain LSTM model, T5
achieves an absolute improvement of 26.20% for the BLEU-4 score and 4.98% for
F1. (2) Compared with the CDiaGPT and the T5 models, our light-weighted DTA
achieves competitive performance in the offline evaluation, and earns the
highest satisfaction rating in the online evaluation, verifying the
effectiveness of our proposed method.

Model | Low | Medium | High
---|---|---|---
LSTM | 30.00 | 29.17 | 40.83
Transformer | 27.50 | 28.33 | 44.17
CDiaGPT | 12.50 | 13.33 | 74.17
T5 | 5.83 | 15.83 | 78.33
DTA | 8.33 | 12.50 | 79.17
Table 2. Statistical Results of Online Evaluation (%) Model | BLEU-4 | P | R | F1
---|---|---|---|---
LSTM | 20.62 | 66.16 | 78.84 | 71.72
Transformer | 26.21 | 62.59 | 75.90 | 64.74
CDiaGPT | 42.54 | 71.18 | 86.43 | 77.11
T5 | 46.83 | 70.91 | 87.25 | 76.70
DTA | 44.82 | 68.03 | 90.80 | 77.74
Table 3. Statistical Results of Offline Evaluation (%).

###  3.3. In-Depth Analysis

####  3.3.1. Effect of Efficiency Issue

To investigate the effect of efficiency issue, we collect each model’s
computation time for processing the test samples under the same infrastructure
(i.e., Tesla v100, 32GB RAM size, etc). Considering the computation time is
highly correlated with the decoding steps, we first divide the generated
responses into 10 subsets based on the response length, and then calculate the
average computation time of each subset, as Figure 3 shows. We can observe
that: (1) Despite the comparable performance, DTA has a significant advantage
in computation efficiency over other models (e.g., 3.37 ms of DTA v.s. 1265.56
ms of CDiaGPT v.s. 2470.69 ms of T5 for subset “[50, 59]”). (2) DTA
outperforms other models more significantly with longer responses. The reason
is that the decoding steps of the token-level generation models are identical
to the response length, while DTA performs action-level decoding. The above
observation verifies that our system provides an effective solution to build
online dialogue services with limited computation resources.

Figure 3. The computation time of different models.

####  3.3.2. Effect of Reliability Issue

We investigate the effect of reliability issue by quantitatively inspecting
the repetition problem. Specifically, we calculate the Jaccard index of
responses of each turn and the previous turns (Costa, 2021). The average
Jaccard index of each model is shown in Table 4, from which we can observe
that: (1) The Jaccard index of human response is the smallest, indicating that
existing models have room for improvement in terms of the repetition problem.
(2) Our method has a much smaller Jaccard index than CDiaGPT and T5. The
action sequence generation, together with the sampling strategy, can
effectively alleviate the repetition problem.

Model | Jaccard Index
---|---
Human Response | 0.129
CDiaGPT | 0.214
T5 | 0.207
DTA | 0.142
Table 4. Jaccard Index of different models.

A concrete online dialogue is shown in Figure 1 (b), where the CDiaGPT
generates exactly the same responses. Though DTA generates similar action
sequences (e.g., combinations of Actions A226, A372, A249 and A109), there are
much fewer cases where the action sequence exactly repeats previous turns. The
small differences in action sequences can lead to large changes in verbal
responses. Besides, the sampling mechanism ensures that the same action in
different turns corresponds to different segments (e.g., A226 in three turns),
which further enables DTA with better diversity.

##  4\. Conclusion

In this paper, we propose a task-oriented dialogue system via action-level
generation. An effective framework is proposed to build the generation model
from the large-scale dialogues with minimum manual effort. The experimental
analyses demonstrate our system’s capability of tackling the reliability and
the efficiency problems encountered with the existing end-to-end generation
methods. In the future, we are interested in exploring an integrated system
that unifies the discrete modules in DTA in an end-to-end architecture.

##  5\. Presenter BIOGRAPHY

Presenter: Yuncheng Hua. He is an algorithm engineer at Meituan, focusing on
researching and building dialogue systems.

##  6\. Company Portrait

Meituan is China’s leading shopping platform for locally found consumer
products and retail services including entertainment, dining, delivery, travel
and other services.

## References

  * (1)
  * Acharya et al. (2021) Anish Acharya, Suranjit Adhikari, Sanchit Agarwal, Vincent Auvray, Nehal Belgamwar, Arijit Biswas, Shubhra Chandra, Tagyoung Chung, Maryam Fazel-Zarandi, Raefer Gabriel, Shuyang Gao, Rahul Goel, Dilek Hakkani-Tur, Jan Jezabek, Abhay Jha, Jiun-Yu Kao, Prakash Krishnan, Peter Ku, Anuj Goyal, Chien-Wei Lin, Qing Liu, Arindam Mandal, Angeliki Metallinou, Vishal Naik, Yi Pan, Shachi Paul, Vittorio Perera, Abhishek Sethi, Minmin Shen, Nikko Strom, and Eddie Wang. 2021.  Alexa Conversations: An Extensible Data-driven Approach for Building Task-oriented Dialogue Systems. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations_. Association for Computational Linguistics, Online, 125–132.  https://doi.org/10.18653/v1/2021.naacl-demos.15
  * Byrne et al. (2021) Bill Byrne, Karthik Krishnamoorthi, Saravanan Ganesh, and Mihir Kale. 2021.  TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 671–680.  https://doi.org/10.18653/v1/2021.acl-long.55
  * Costa (2021) Luciano da F Costa. 2021\.  Further generalizations of the Jaccard index.  _arXiv preprint arXiv:2110.09619_ (2021).
  * Fu et al. (2021) Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. 2021\.  A theoretical analysis of the repetition problem in text generation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 12848–12856.
  * Jang et al. (2021) Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. 2021.  GPT-Critic: Offline Reinforcement Learning for End-to-End Task-Oriented Dialogue Systems. In _International Conference on Learning Representations_.
  * Jin et al. (2004) Meixun Jin, Mi-Young Kim, Dongil Kim, and Jong-Hyeok Lee. 2004\.  Segmentation of Chinese long sentences using commas. In _Proceedings of the Third SIGHAN Workshop on Chinese Language Processing_. 1–8.
  * Lee et al. (2019) Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Zheng Zhang, Yaoqin Zhang, Xiang Li, Jinchao Li, Baolin Peng, Xiujun Li, Minlie Huang, and Jianfeng Gao. 2019.  ConvLab: Multi-Domain End-to-End Dialog System Platform. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics, Florence, Italy, 64–69.  https://doi.org/10.18653/v1/P19-3011
  * Li et al. (2017) Feng-Lin Li, Minghui Qiu, Haiqing Chen, Xiongwei Wang, Xing Gao, Jun Huang, Juwei Ren, Zhongzhou Zhao, Weipeng Zhao, Lei Wang, et al. 2017\.  Alime assist: An intelligent assistant for creating an innovative e-commerce experience. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 2495–2498.
  * Lin et al. (2021) Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, Peng Xu, Feijun Jiang, Yuxiang Hu, Chen Shi, and Pascale Fung. 2021\.  Bitod: A bilingual multi-domain dataset for task-oriented dialogue modeling.  _arXiv preprint arXiv:2106.02787_ (2021).
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020\.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ 21, 140 (2020), 1–67.
  * See et al. (2019) Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019.  What makes a good conversation? how controllable attributes affect human judgments.  _arXiv preprint arXiv:1902.08654_ (2019).
  * Song et al. (2021) Shuangyong Song, Chao Wang, Haiqing Chen, and Huan Chen. 2021\.  An Emotional Comfort Framework for Improving User Satisfaction in E-Commerce Customer Service Chatbots. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers_. Association for Computational Linguistics, Online, 130–137.  https://doi.org/10.18653/v1/2021.naacl-industry.17
  * Sun et al. (2021) Kai Sun, Seungwhan Moon, Paul Crook, Stephen Roller, Becka Silvert, Bing Liu, Zhiguang Wang, Honglei Liu, Eunjoon Cho, and Claire Cardie. 2021\.  Adding Chit-Chat to Enhance Task-Oriented Dialogues. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 1570–1583.  https://doi.org/10.18653/v1/2021.naacl-main.124
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014\.  Sequence to sequence learning with neural networks.  _Advances in neural information processing systems_ 27 (2014).
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022\.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ (2022).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2020) Yida Wang, Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, and Minlie Huang. 2020.  A Large-Scale Chinese Short-Text Conversation Dataset. In _NLPCC_.  https://arxiv.org/abs/2008.03946
  * Xi et al. (2022) Xiangyu Xi, Chenxu Lv, Yuncheng Hua, Wei Ye, Chaobo Sun, Shuaipeng Liu, Fan Yang, and Guanglu Wan. 2022\.  A Low-Cost, Controllable and Interpretable Task-Oriented Chatbot: With Real-World After-Sale Services as Example. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Madrid, Spain) _(SIGIR ’22)_. Association for Computing Machinery, New York, NY, USA, 3398–3402.  https://doi.org/10.1145/3477495.3536331
  * Yan et al. (2021) Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021\.  ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 5065–5075.  https://doi.org/10.18653/v1/2021.acl-long.393
  * Yan et al. (2017) Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. 2017\.  Building task-oriented dialogue systems for online shopping. In _Thirty-First AAAI Conference on Artificial Intelligence_.
  * Yang et al. (2021) Yunyi Yang, Yunhao Li, and Xiaojun Quan. 2021.  Ubar: Towards fully end-to-end task-oriented dialog system with gpt-2. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 14230–14238.
  * Yu et al. (2021) Dian Yu, Luheng He, Yuan Zhang, Xinya Du, Panupong Pasupat, and Qi Li. 2021\.  Few-shot Intent Classification and Slot Filling with Retrieved Examples. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 734–749.  https://doi.org/10.18653/v1/2021.naacl-main.59
  * Zhu et al. (2020) Qi Zhu, Zheng Zhang, Yan Fang, Xiang Li, Ryuichi Takanobu, Jinchao Li, Baolin Peng, Jianfeng Gao, Xiaoyan Zhu, and Minlie Huang. 2020\.  ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics, Online, 142–149.  https://doi.org/10.18653/v1/2020.acl-demos.19
  * Zhu (2019) Xiaoming Zhu. 2019\.  Case ii (part a): Jimi’s growth path: Artificial intelligence has redefined the customer service of jd. com.  In _Emerging Champions in the Digital Economy_. Springer, 91–103.


# VLA Observations of the AE Aqr-type Cataclysmic Variable LAMOST
J024048.51+195226.9

Paul E. Barrett Department of Physics, The George Washington University,
Washington, DC 20052 USA

###### Abstract

AE Aqr was until recently the only known magnetic cataclysmic variable (MCV)
containing a rapidly spinning (33.08 s) white dwarf (WD). Its radio emission
is believed to be a superposition of synchrotron emitting plasmoids, because
it has a positive spectral index spanning three orders of magnitude ($\approx
2-2000$ GHz) and is unpolarized. Both characteristics are unusual for MCVs.
Recently, Thorstensen has suggested that the cataclysmic variable LAMOST
J024048.51+195226.9 (henceforth, J0240+19) is a twin of AE Aqr based on its
optical spectra. Optical photometry shows the star to be a high-inclination,
eclipsing binary with a spin period of 24.93 s, making it the fastest spinning
WD. This paper presents three hours of Very Large Array radio observations of
J0240+19. These observations show that the persistent radio emission from
J0240+19 is dissimilar to that of AE Aqr in that it shows high circular
polarization and a negative spectral index. The emission is most similar to
the nova-like CV V603 Aql. We argue that the radio emission is caused by a
superposition of plasmoids emitting plasma radiation or electron cyclotron
maser emission from the lower corona of the donor star and not from the
magnetosphere near the WD, because the latter site is expected to be modulated
at the orbital period of the binary and to show eclipses of which there is no
evidence. The radio source J0240+19, although weak ($\lesssim 1$ mJy), is a
persistent source in a high-inclination eclipsing binary, making it a good
laboratory for studying radio emission from CVs.

Cataclysmic variable stars: Intermediate Polars; White dwarf stars; Magnetic
stars; AE Aquarii; Astronomy: Radio

††facilities: VLA††software: CASA (6.1.2), Julia (1.6.2), Visfit

##  1 Introduction

Magnetic cataclysmic variables (MCVs) showing two photometric periods are
called intermediate polars (also known as DQ Herculis stars). The longer
period (of order a few hours) is associated with the orbital period $P_{orb}$
of the binary and the shorter period (of order tens of minutes) with the
rotation period $P_{rot}$ of the white dwarf (WD). The ratio
$P_{rot}/P_{orb}\approx 0.1$. The exception to the spin-orbit rule is AE
Aquarii (AE Aqr), which until recently, contained the fastest known rotating
WD with a period of 33.08 s ($P_{rot}/P_{orb}\approx 0.001$; Patterson 1979).
The rapid rotation along with the moderately strong magnetic field of the WD
(1-10 MG) causes almost all of the accreted material ($\sim 99.9$%; Meintjes
et al. 2015) to be expelled from the binary in a so called “magnetic
propeller”. The WD’s rapidly spinning magnetosphere acts as a centrifugal
barrier to the accreted diamagnetic plasmoids and propels them out of the
system (Wynn & King, 1995; Wynn et al., 1997). This accretion scenario is
similar to that of accreting millisecond pulsars.

AE Aqr also has several other unique observational characteristics. de Jager
et al. (1994) using 14 years of X-ray data found that the WD is spinning down
at a rate of $\dot{P}_{\star}\approx 5.64\times 10^{-14}$ s s-1. This rate
equates to a spin-down luminosity $L_{s-d}=I\Omega\dot{\Omega}\sim 10^{34}$
erg s-1. This is roughly 500 times larger than the accretion luminosity of
$2\times 10^{31}$ erg s-1 (Eracleous & Horne, 1996).

Bookbinder & Lamb (1987) and Bastian et al. (1988) using the National Radio
Astronomy Observatory (NRAO) Very Large Array found that AE Aqr is a
persistent radio emitter showing rapid variablity and large flares.
Observations at 1.5, 4.9, 15, and 22.5 GHz show a long-term averaged spectral
index $\alpha\propto 0.5$ ($S_{\nu}\propto\nu^{\alpha}$; Bastian et al. 1988).
The flux density at 15 GHz showed variations of a few mJy on minute timescales
during quiescence and peaked at 12 mJy during flares, which occur about every
five hours. They also note that in those cases where the flares are temporally
resolved, the rise time is shorter than the decay time, which suggests that an
external medium limits the flare’s rate of expansion. Ground-based millimeter
and submillimeter, and spaced-based infrared observations show that the
positive spectral index extends to terahertz frequencies (Abada-Simon et al.,
2005; Dubus et al., 2007; Torkelsson, 2013). Above 2 THz the spectrum turns
over and has a spectral index of $\alpha\propto-0.7$. This nonthermal radio
emission, spanning more than three orders of magnitude in frequency, is
attributed to a superposition of synchrotron emitting plasmoids that expand
and cool radiatively through synchrotron radiation (van der Laan, 1963, 1966).

Thorstensen (2020) recently argued that the CV LAMOST J024048.51+195226.9,
also known as CRTS J024048.5+19227, (hereafter J0240+19) may be a possible
twin of AE Aqr based on its unusual optical spectrum, which shows no He II
emission and weak He I lines. In addition, the light curves show large
irregular flaring on the timescale of minutes. However, the time resolution of
the photometry was insufficient to search for pulsations on the order of tens
of seconds. Using publically available Catalina Real-Time Transient Survey
(CRTS) and All-Sky Automated Survey for Supernovae (ASAS-SN) data, Littlefield
& Garnavich (2020) identify a shallow dip in the light curves at the
secondary’s inferior conjunction and identify this feature with an eclipse by
the secondary star. This means that J0240+19 is a high inclination system and
explains why the He I $\lambda 6678$ Å emission line briefly disappears at
orbital phase 0. Additional photometric observations by Garnavich et al.
(2021) confirmed the high inclination of J0240+19, but did not detect the WD’s
spin period. The spin period was finally found by Pelisoli et al. (2021) to be
24.9328(38) s with an amplitude of 0.2% in the g band, which is below the
detection limits of the previous searches. J0240+19 is now the fastest known
rotating WD. A search of the NRAO Very Large Array Sky Survey (VLASS) version
1.2 (tile T15t04, 2019 Jun 07; Lacy et al. 2020) showed the presence of a
$508\pm 120\ \mu$Jy ($4\sigma$) radio source at 3 GHz within 1 arcsec of the
J0240+19 position. It was also detected in a MeerKAT pointed observation on
2020 Aug 12 with a flux density of $600\pm 20\ \mu$Jy at 1.284 GHz and
spectral index $\alpha\approx-0.6$ (Pretorius et al., 2021). Neither
observation reported any polarization measurements.

In this paper we present three hours of NRAO Karl G. Jansky Very Large Array
(VLA) observations of J0240+19. The observations are split between one hour of
spectroscopy in order to compare the spectrum of J0240+19 to that of AE Aqr
and two hours of photometry in order to detect the spin period of the WD. If
J0240+19 is a twin of AE Aqr, then its radio flux density is expected to
increase with frequency and to be $\approx~{}1.4$ mJy at 22 GHz, assuming a
flux density of $508\ \mu$Jy at 3 GHz and a spectral index of 0.5, the same as
AE Aqr. At this flux density, it should be possible to detect a large
amplitude pulsation with a period as short as 30 s.

##  2 Observations and Data Reduction

J0240+19 was observed by the VLA for three hours on two seperate dates. The
first observation measured the spectrum of J0240+19 across five radio bands
from 2–26 GHz during a one hour scheduling block. The exposures in each band
range between 600–628 s duration. The results of this observation determined
which frequency band to use for the second photometric observation of two
hours. This second observation was done using the C band (4–8 GHz) with an on-
source time of about 1.6 hours. During the second observation, the VLA cycled
between the target J0240+19 and the phase calibrator J0238+1636 every 570 s
with about 525 s being on target. Each observation used the 3-bit samplers for
wideband coverage. The radio sources J0137+3309 (=3C48) and J0238+1636 were
used as the flux and polarization, and the phase and gain calibrators for each
observation, respectively. No polarization leakage calibrators were observed,
because the cross-polarization is $<1$% and varies slowly over sevaral months
(Perley & Sault, 2014). This accuracy is sufficient for our needs. Note that
the flux calibrator 3C48 has been undergoing a flare since about January 2018,
which will affect the absolute flux scale. The effect is smaller at low
frequencies ($\sim 5$% at S band) and greater at high frequencies ($\sim 10$%
at Ku band; VLA SUS 2021). Table 1 gives a log of the observations.

Table 1: Log of the observations of J0240+19. Start Date | Start Time | Band | Frequency | Exposure
---|---|---|---|---
| (UTC) |  | (GHz) | (h:m:s)
2021 Jun 11 | 16:52:36 | K | 18–26 | 00:05:28
| 16:59:54 | Ku | 12–18 | 00:05:22
| 17:07:09 | X | 8–10 | 00:05:12
| 17:14:15 | C | 4–8 | 00:05:10
| 17:21:20 | S | 2–4 | 00:05:00
2021 Jul 20 | 15:22:50 | C | 4–8 | 01:35:55

The data were calibrated using version 6.1.2 of the CASA (Common Astronomy
Applications Software) calibration pipeline (McMullin et al., 2007). The
imaging application tclean is used to generate IQUV and RRLL images of each
target scan to check for any source confusion and radio frequency
interference. None were found. The flux densities are measured by fitting a
point source to the UV data using the Julia programming language package
Visfit (Barrett, 2021). The package uses a box-constrained Levenberg-Marquardt
algorithm to minimize the model residuals. The position of the point source is
constrained to be within $\pm$2 arcsec of the known source position. When
measuring the left and right circular polarizations, the flux densities are
measured simulaneously using the same source position. Although the phase
centers for the left and right polarizations may have slightly different
positions, this approach is recommended for faint sources.

##  3 Analysis

###  3.1 Spectroscopy

The spectrum of J0240+19 obtained during the first observation is shown in
Figure 1. This observation occured between orbital phases 0.70–0.78 using the
ephemeris of Garnavich et al. (2021). The average flux density and standard
deviations for each frequency band are shown in black. It varies from $195\pm
12$ $\mu$Jy at 3 GHz to $23\pm 8$ $\mu$Jy at 22 GHZ. The spectral index of
these measurements is roughly consistent with the negative spectral index
measured by MeerKAT of $\alpha\propto-0.6$ (Pretorius et al., 2021), where
$S_{\nu}\propto\nu^{\alpha}$. Figure 1 also shows the left (LL) and right (RR)
circular polarizations for a narrower bandwidth (512 MHz) across the entire
frequency range. These data show that the circular polarization varies
randomly from one frequency to the next. In particular, the measurement at
4.231 GHz shows strong ($>50$%) circularly polarized emission. Note that the Q
and U images show no significant linear polarization ($\lesssim 1$%).

Figure 1: 2-26 GHz spectrum of LAMOST J0240+19. The average flux densities
(black points) are shown for each of the five spectral bands (S, C, X, Ku and
K). The left (blue) and right (red) circular polarizations are also shown for
a narrower bandwidth (512 MHz). Note the high polarization at 4.231 GHz.

###  3.2 Photometry

Because the spectroscopy showed that the S and C bands have similar flux
densities ($\approx~{}200$ $\mu$Jy), the photometry was done using the C band,
because its wider bandwidth provides greater sensitivity. Unfortunately,
J0240+19 was even weaker during the second observation with an average flux
density of $\approx~{}100\mu$Jy. Such a low flux density precludes any attempt
at searching for, or detecting, a spin period as short as 30 s. We found that
the best compromise between time resolution and signal-to-noise ratio is an
integration time of 265 s. The observation occurred just after inferior
conjunction between orbital phases 0.08–0.32 using the ephemeris of Garnavich
et al. (2021). The resulting left and right circular polarization light curves
are shown in Figure 2. Like the spectrum shown in Figure 1, the light curve
also shows that the flux density and circular polarization vary randomly on a
timescale of at most a few minutes. The light curve is too short and variable
to show any evidence of an orbital period.

Figure 2: 1.6 hours of C band photometry (4–8 GHz) of LAMOST J0240+19. The
left (blue) and right (red) circular polarization are shown. The integration
time for each flux measurement is 265 s.

##  4 Discussion

###  4.1 Coherent radio emission

The brightness temperature, $T_{B}\approx 10^{14}\
S_{-3}D_{2}\nu_{9}^{-2}r_{10}^{-2}$ K, where $S_{-3}$ is the observed flux
density in mJy, $D_{2}$ is the distance in units of 100 pc, $\nu_{9}$ is the
frequency in GHz, and $r_{10}$ is the radius of the projected source area in
$10^{10}$ cm. For J0240+19, $T_{B}\approx 10^{13}$ K for $S_{-3}=0.2$ at 3
GHz, $D=6.2$ (Gaia Collab., 2020), and $r_{10}=7$. The value of $r_{10}$
assumes that the area of the radio emission has a radius twice that of the
donor star. This high $T_{B}$ implies a coherent emission process.

Following the discussion of Bastian & Bookbinder (1987), there are two
coherent mechanisms that can produce highly circularly polarized emission as
seen in the spectrum and light curve of J0240+19: magnetic plasma radiation
and electron cyclotron maser emission (ECME). Synchrotron emission as
suggested by (Pretorius et al., 2021) is unlikely. These emissions can be
produced by the ordinary (O), the extraordinary (X), or the low frequency
branch of the extraordinary (Z) modes depending on the emission mechanism. In
the case of plasma emission, the radiation is produced at either the
fundamental or the second harmonic of the electron plasma frequency,
$\omega_{pe}=(4\pi n_{e}e^{2}/m_{e})^{1/2}$, where $n_{e}$ is the electron
density, $e$ is the electron charge, and $m_{e}$ is the electron mass.
Emission at the fundamental frequency can be completely polarized and is
ascribed to the O mode. Whereas, emission at the second harmonic is moderately
polarized and is related to either the O or X modes. For the frequency range
of 2–18 GHz, $\omega_{pe}$ gives a density of the source region of $\approx
10^{9}-10^{11}$ cm-3.

In the case of ECME, the polarized emission is related to the O, X, or Z
modes. Emission at the fundamental gyrofrequency, $\omega_{Be}=eB/m_{e}c$, is
unlikely to escape the source region, because of the strong gyroresonant
absorption at the second harmonic. However, emisson at the second harmonic of
the O or X modes may escape the source region and be highly polarized. The
polarized emission at $\approx 2$ GHz places an upper limit of $\lesssim
4\times 10^{10}$ cm-3 on the electron density, since the ECME requires the
plasma frequency to be less than the gyrofrequency. It also implies that the
ambient magnetic field of the source region $B$ ($\approx 360\
(\frac{\omega_{Be}}{1\ \small{\textrm{GHz}}})$ G) is in the range of 360 –
3200 G.

###  4.2 The radio emission compared to AE Aqr

The two physical constraints of a low density plasma and kiloGauss magnetic
field restricts the location of the radio emission to a low density plasma in
the magnetosphere of the WD or the lower corona of the donor star. Emission
from the accretion stream or disk is therefore unlikely because the densities
in those regions are too high ($\gtrsim 10^{14}$ cm-3 Barrett et al. 2020). We
first consider emission from the WD magnetosphere. Assuming a 1 MG dipolar
surface field for the WD, The strength of the magnetic field with radius is:
$B=10^{6}r_{9}^{-3}$ G, where $r_{9}$ is the WD radius in $10^{9}$ cm.
Therefore, the radio emission comes from a region between 7–14 WD radii for
magnetic fields of 360–3200 G. These radii are much smaller than the radius of
the donor star and are therefore likely to be modulated by the orbital period
and eclipsed by the donor star. Although the radio observations are sparse and
do not cover the orbital phase of the eclipse (0.95–0.05), neither the MeerKAT
L band (Pretorius et al., 2021) nor VLA C band observations show any evidence
of an orbital modulation. The MeerKAT observations occur during orbital phases
0.25–0.44 and our VLA observation during phases 0.08–0.32. Therefore, emission
from the WD magnetosphere is problematic.

Based on these arguments, the donor star’s corona is the most likely location
of the radio emission. Roche tomography of the donor star of AE Aqr (Dunford &
Smith, 2005; Watson et al., 2006; Smith et al., 2012) shows that $\sim 20$% of
its surface is affected by high-latitude starspots similar to that seen in
Doppler images of rapidly rotating isolated stars. This implies that the donor
star in AE Aqr is highly active, which is expected for a rapidly rotating star
($P_{rot}<10$ days), because the magnetic dynamo saturates at about this spin
period. We propose that the donor star in J0240+19 is also magnetically active
and is the source of the radio emission. This means that the surface field of
the donor star is $\sim$ few kG in strength. It also provides an explanation
of the random nature of the circular polarization in time and frequency due to
a superposition of plasma radiation or ECME from several emitting plasmoids.

Figure 3: The 4-24 GHz spectrum of AE Aqr showing a positive spectral index
betwen the C and K bands.

In order to compare the spectra of AE Aqr and J0240+19, we have reanalyzed one
of the observations of AE Aqr by Barrett et al. (2017) from 2015\. Figure 3
shows a positive spectral index between the C and K bands (6 GHz and 22 GHz,
resp.). Because each band was observed several minutes apart and the radio
emission varies rapidly ($\sim$minutes), the X band (8 GHz) was caught during
a period of negative spectral index. As noted previously, AE Aqr has a
positive spectral index spanning over three orders of magnitude and only
begins to turnover at $\approx 2$ THz. Also note that radio emission in all
three bands is unpolarized ($P_{V}\lesssim 0.1$%). This spectrum is unlike
that of the spectrum of J0240+19 with its negative spectral index and strong
circular polarization. Instead, the spectrum of J0240+19 is similar to that of
the nova-like V603 Aql, which on average has a negative spectral index between
2–12 GHz and shows strong, randomly varying, circular polarization (Coppejans
et al., 2015; Barrett et al., 2017, 2020, 2021)111Although, V603 Aql has been
suspected of being an IP, to date no optical or X-ray spin period has been
detected (see e.g., Haefner & Metz 1985; Udalski & Schwarzenberg-Czerny 1989;
Gnedin et al. 1990). In addition, the specific radio luminosities at 3 GHz are
of similiar magnitude. That of J0240+19 is 7.3–18.3 $\times 10^{15}$ ergs s-1
Hz-1 and that of V603 Aql is 2.0–3.5 $\times 10^{15}$ ergs s-1 Hz-1. These
luminosities are in the range of 1–50 $\times 10^{15}$ ergs s-1 Hz-1, which is
typical of most CVs (See Figure 2 of Pretorius et al. 2021).

##  5 Conclusions

The three hours of VLA observations of J0240+19 show a spectrum and light
curve that are dissimilar to those of AE Aqr. The spectrum has a negative
spectral index between 2–26 GHz and the emission shows periods of high
circular polarization, while those of AE Aqr show a positive spectral index
and unpolarized emission. The radio emission from J0240+19 is characteristic
of magnetic plasma radiation or electron cyclotron maser emission, while that
of AE Aqr and AR Sco is characteristic of synchrotron emission. Therefore, the
radio emission from J0240+19 is unlike that of AE Aqr and AR Sco, although the
characteristics of their optical emission are very similar. The radio emission
from J0240+19 is similar to most radio-emitting magnetic CVs and is most
similar to the nova-like CV V603 Aql, which is suspected of containing a
rapidly spinning WD.

We argue that the source of the radio emission in J0240+19 is the magnetically
active donor star and not a low density region of the magnetosphere near the
WD, because the radio light curves show no evidence of an orbital modulation.
If the radio emission is from the WD magnetosphere, then it should be eclipsed
during inferior conjunction, because of its proximity to the WD. We therefore
encourage additional radio observations to refute this assertion by looking
for radio eclipses. Although J0240+19 is a weak radio source ($\lesssim 1$
mJy), it is a persistent source in a high-inclination, eclipsing CV, making it
a good laboratory for studying radio emission from CVs.

##  6 Acknowledgements

We thank Paul Mason for a critical reading of the manuscript and several
useful comments. The National Radio Astronomy Observatory is a facility of the
National Science Foundation operated under cooperative agreement by Associated
Universities, Inc.

## References

  * Abada-Simon et al. (2005) Abada-Simon, M., et al. 2005, A&A, 433, 1063.
  * Barrett et al. (2017) Barrett, P. E., Dieck, C., Beasley, A. J., Singh, K. P., & Mason, P. A. 2017, AJ, 154, 252–260.
  * Barrett et al. (2020) Barrett, P., Dieck, C., Beasley, A. J., Mason, P. A., & Singh, K. P. 2020, AdSpR, 66, 1226\.
  * Barrett (2021) Barrett, P. 2021, in preparation.
  * Barrett et al. (2021) Barrett, P., Dieck, C., Beasley, A. J., Mason, P. A., & Singh, K. P. 2021, in preparation.
  * Bastian & Bookbinder (1987) Bastian, T. S. & Bookbinder, J. A. 1987, Nature, 326, 678.
  * Bastian et al. (1988) Bastian, T. S., Dulk, G. A., & Chanmugam, G. 1988, ApJ, 324, 431.
  * Bookbinder & Lamb (1987) Bookbinder, J. A. & Lamb, D. Q. 1987, ApJ, 323, L131.
  * Coppejans et al. (2015) Coppejans, D. L., Körding, E. G., Miller-Jones, J. C. A., Rupen, M. P., Knigge, C., Sivakoff, G. R., & Groot, P. J. 2015, MNRAS, 451, 3801.
  * de Jager et al. (1994) de Jager, O. C., Meintjes, P. J., O’Donoghue, D., & Robinson, E. L. 1994, MNRAS, 267, 577.
  * Dunford & Smith (2005) Dunford, A. & Smith, R. C. 2005, ASPConf (eds. Hameury & Lasota), 330, 399.
  * Dubus et al. (2007) Dubus, G., Taam, R. E., Hull, C., Watson, D. M., & Mauerhan, J. C. 2007, ApJ, 663, 516.
  * Eracleous & Horne (1996) Eracleous, M. & Horne, K. 1996, ApJ, 471, 427.
  * Gaia Collab. (2020) Gaia Collaboration, Brown, A. G. A., Vallenari, Prusti, A. T., de Bruijne, J. H. J., Babusiaux, C., & Biermann, M. (2020), arXiv 2012.01533.
  * Garnavich et al. (2021) Garnavich, P., Littlefield, C., Wagner, R. M., van Roestel, J., Jaodand, A. D., Szkody, P., & Thorstensen, J. R. 2021, ApJ, 917, 22.
  * Gnedin et al. (1990) Gnedin, Y. N., Borisov, N. V., & Natsvlishvili, T. M., 1990, Soviet Astro Let, 16, 272\.
  * Haefner & Metz (1985) Haefner, R. & Metz, K. 1985, A&A, 145, 311.
  * Lacy et al. (2020) Lacy, M., Baum, S. A., Chandler, C. J., Clarke, T. E., Deustua, S., et al., 2020, PASP, 132, 1.
  * Littlefield & Garnavich (2020) Littlefield, C. & Garnavich, P. 2020, RNAAS, 4, 171.
  * McMullin et al. (2007) McMullin, J. P., Waters, B., Schiebel, D., Young, W., & Golap, K. 2007, Astronomical Data Analysis Software and Systems XVI (ASP Conf. Ser. 376), ed. R. A. Shaw, F. Hill, & D. J. Bell (San Francisco, CA: ASP), 127\.
  * Meintjes et al. (2015) Meintjes, P., Odendaal, A., & van Heerden, H. 2015, AcPPP, 2, 86.
  * Patterson (1979) Patterson, J. 1979, ApJ, 234, 978.
  * Pelisoli et al. (2021) Pelisoli, I., Marsh, T. R., Dhillon, V. S., Breedt, E., Brown, A. J., Dyer, M. J., Green, M. J., Kerry, P., Littlefair, S. P., Parsons, S. G., Sahman, D. I., & Wild, J. F. 2021, MNRAS, in press.
  * Perley & Sault (2014) Perley, R. & Sault, B. 2014, EVLA Memo 178.
  * Pretorius et al. (2021) Pretorius, M. L., Hewitt, D. M., Woudt, P. A., Fender, R. P., Heywood, I., Knigge, C., Miller-Jones, J. C. A., Buckley, D. A. H., Worters, H. L., Potter, S. B., & Williams, D. R. A. 2021, MNRAS, 503, 3692.
  * Smith et al. (2012) Smith, R. C., Dunford, A., & Watson, C. A. 2012, SAIt (eds. Giovanelli & Sabau-Graziati), 83, 708.
  * Thorstensen (2020) Thorstensen, J. R. 2020, AJ, 160, 151.
  * Torkelsson (2013) Torkelsson, U. 2013, IAUS, 290, 145\.
  * Udalski & Schwarzenberg-Czerny (1989) Udalski, A. & Schwarzenberg-Czerny, A. 1989, Act Astron, 39 125.
  * van der Laan (1963) van der Laan, H. 1963, MNRAS, 126, 535.
  * van der Laan (1966) van der Laan, H. 1966, Nat, 211, 1131\.
  * VLA SUS (2021) Very Large Array Science User Support 2021, science.nrao.edu/facilities/vla/docs/manuals/oss/performance/fdscale
  * Watson et al. (2006) Watson, C. A., Dhillon, V. S., & Shahbaz, T. 2006, MNRAS, 368, 637.
  * Wynn & King (1995) Wynn, G. A. & King, A. R. 1995, MNRAS, 275, 9.
  * Wynn et al. (1997) Wynn, G. A., King, A. R., & Horne, K. 1997, MNRAS, 286, 436.