# Domain-agnostic Question-Answering with Adversarial Training

Seanie Lee111Equal contribution Donggyu Kim00footnotemark: 0 Equal
contribution  Jangwon Park111Equal contribution

###### Abstract

Adapting models to new domain without fine-tuning is a challenging problem in
deep learning. In this paper, we utilize an adversarial training framework for
domain generalization in Question Answering (QA) task. Our model consists of a
conventional QA model and a discriminator. The training is performed in the
adversarial manner, where the two models constantly compete, so that QA model
can learn domain-invariant features. We apply this approach in MRQA Shared
Task 2019 and show better performance compared to the baseline model.

##  1 Introduction

Followed by the success of deep learning in various tasks, it becomes
important to build a single model covering various domains without further
fine-tuning to out-of-domain distribution. Because for real world application,
a model is required to generalize to unseen sources of data.

In case of Question Answering (QA) task which is one of the promising areas in
NLP, however, models outperforming human on SQuAD Rajpurkar et al. (2016)
cannot generalize well to other datasets. Models rather overfit to a specific
dataset and require additional training on other dataset to adapt to new
domain Yogatama et al. (2019).

Thus, in order to build a domain-agnostic QA model which is capable of
handling out-of-domain data, it is necessary for model to learn domain-
invariant features rather than specific ones. In this paper, we apply
adversarial training framework to train a QA model with domain-agnostic
representation. As shown in Figure 1, the model is divided into two
components, which are the QA model and the domain discriminator. The
discriminator predicts domain label of hidden representation from QA model.
During the training, the QA model tries to fool the discriminator so that the
hidden representation becomes indistinguishable to the discriminator.
Meanwhile the discriminator is trained to identify the domain label correctly.
As a result, QA model can learn domain-invariant features. Our framework can
be applied to any existing QA model because the architecture of QA model stays
unchanged.

We train and validate our method on 12 datasets (6 datasets for training and 6
datasets for validation) which are provided by MRQA Shared Task. Each training
dataset is considered different domain for adversarial learning in which QA
model learns domain-invariant feature representation by competing with
discriminator. Our experimental result shows that the proposed method improves
performance compared to baseline.

Figure 1: Overall training procedure for learning domain-invariant feature
representation. Model learns to predict start and end position in the passage
and fool discriminator for domain-invariant representation.

##  2 Related Works

Pre-trained Language Model Recently, there have been several applications for
using pre-trained language models, such as ELMo Peters et al. (2018), GPT
Radford et al. (2018), or BERT Devlin et al. (2018) to transfer the knowledge
from pre-training to various downstream NLP tasks.

BERT is pretrained with bidirectional encoder Vaswani et al. (2017) on large
corpora. Unlike other auto-regressive language models (unidirectional or
concatenation of forward and backward language model), BERT randomly masks
some input tokens and predicts the masked tokens based on its context. The
masked language model enables bidirectional representation, which leads to
significant improvements on a number of NLP tasks, such as sentence
classification, POS tagging or question answering.

Domain Generalization Even though many deep learning models surpass human-
level performance on various task, they perform poorly on out-of-domain
dataset. To address this problem, domain adaptation and domain generalization
are proposed, making models more robust to out-of-domain data. The difference
between domain adaptation and domain generalization is that for domain
generalization, data from the target domain is not available during training.

Several methods for domain generalization exist. One of them is to train a
model for each in-domain dataset. When testing on out-of-domain, select the
most correlated in-domain dataset and use that model for inference Xu et al.
(2014). Other works such as Ghifary et al. (2015); Muandet et al. (2013),
model is trained to learn a domain-invariant feature by using multi-view
autoencoders and mean map embedding-based techniques.

Other approaches Khosla et al. (2012); Li et al. (2017) break down parameters
of a model into domain-specific and domain-agnostic components during training
with in-domain dataset, and use the domain invariant parameters for predicting
data from unseen target domain.

Recently, meta-learning has been proposed for domain generalization. Some
methods Li et al. (2018a); Balaji et al. (2018); Li et al. (2019) leverage
meta-learning framework for domain generalization.

Adversarial Training The idea of adversarial training is originally proposed
in the field of image generation Goodfellow et al. (2014), known as Generative
Adversarial Network (GAN). GAN is also adopted in text generation Yu et al.
(2017) with policy gradient for bypassing non-differentiable operation. The
concept of adversarial training is not limited to the task of generation. It
can be extended to text classification Chen et al. (2016); Liu et al. (2017);
Chen and Cardie (2018), and relation extraction Wu et al. (2017). Likewise,
attempts are made to get language-invariant features with adversarial training
Chen et al. (2016); Zhang et al. (2017).

Adversarial training has been used for domain adaptation or domain
generalization as well. In Domain-Adversarial Neural Network (DANN) Ganin et
al. (2016), it has two classifiers: one classifies task-specific class labels,
and the other classifies whether the data belong to source or target domain.
Recently, One approach Li et al. (2018b) extends adversarial autoencoder by
minimizing maximum mean discrepancy among different domains for domain-
invariant feature representation.

Datasets | Samples | Avg.Q.len | Avg.P.len |  Source
---|---|---|---|---
BioASQ (BA) | 1,504 | 16.4 | 353.9 |  Bio-medical literature
DROP (DP) | 1,503 | 12.0 | 268.4 |  Wiki + National Football League (NFL) game summaries and history articles
DuoRC (DR) | 1,501 |  9.8 | 798.9 |  Wiki + IMDb
RACE (RA) |  674 | 12.4 | 381.0 |  English exams for Chinese middle and high school
RelationExtraction (RE) | 2,948 | 11.6 |  38.0 |  Wiki (WikiReading dataset)
TextbookQA (TQ) | 1,503 | 12.1 | 751.0 |  1k lessons and 26k multi-modal questions, from middle school science curriculum
Table 1: Statistics of out-of-domain validation dataset. Q and P stands for
question and passage, respectively. Length is calculated based on word-level
token.

##  3 Proposed Methodology

We assume that there exists domain invariant feature representation such that
QA model generalize well to predict answer on unseen out-of-domain. In order
to adapt to out-of-domain, adversarial learning procedure is leveraged for
learning domain-invariant representation. We present our proposed method in
detail in the following sections.

###  3.1 Problem Definition

We formulate the task as follows: given the $K$ in-domain datasets
$\mathcal{D}_{i}$ , consisting of triplets of passage $\mathbf{c}$, question
$\mathbf{q}$, and answer $\mathbf{y}$, where
$\mathcal{D}_{i}=\\{\mathbf{c}^{(k)}_{i},\mathbf{q}^{(k)}_{i},\mathbf{y}^{(k)}_{i}\\}^{N_{k}}_{i=1}$.
The model learned from $\\{\mathcal{D}_{i}\\}_{i=1}^{K}$ predicts answer
$\mathbf{y}^{l}_{j}$ from $\mathbf{c}^{l}_{j},\mathbf{q}^{l}_{j}$ for each $L$
out-of-domain datasets $\\{\mathcal{D}_{j}\\}^{L}_{j=1}$.

Model | BA | DP | DR | RA | RE | TQ | Avg
---|---|---|---|---|---|---|---
EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM |  F1
Bert-base | 46.44 | 60.81 | 28.31 | 37.88 | 42.78 | 53.32 | 28.23 | 39.51 | 73.33 | 83.89 | 44.30 | 52.03 | 43.90 | 54.57
Bert-base-adv | 43.35 | 60.04 | 30.51 | 40.01 | 45.97 | 57.89 | 26.50 | 39.73 | 72.67 | 83.53 | 45.62 | 55.67 | 44.10 |  56.15
Model | BP | CQ | MC | MR | ST | TR | Avg
EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM |  F1
Bert-base | 38.36 | 57.38 | 47.40 | 55.29 | 54.16 | 66.12 | 47.83 | 64.81 | 58.64 | 77.02 | 36.73 | 53.96 | 47.19 | 62.43
Bert-base-adv | 42.92 | 61.09 | 48.13 | 56.50 | 55.83 | 69.30 | 52.82 | 68.78 | 52.73 | 75.63 | 39.08 | 56.79 | 48.59 |  64.68
Table 2: Model performance on validation and test set. Above is the validation
set and below is the test set.

###  3.2 Prediction Model

Our method can be applied to any QA models which learn representation in the
joint embedding space of passage and question. In this paper, we use BERT for
QA because it is pre-trained on a large corpus and known to be generalized on
several different tasks. As for standard QA task, the model is trained to
minimize negative log-likelihood of answer $\mathbf{y}$ for all the given in-
domain datasets, where $N,\mathbf{y}_{i,s}$, and $\mathbf{y}_{i,e}$ are
respectively the total number of in-domain data, the start position and the
end position of answer in the passage.

$\displaystyle\begin{split}\mathcal{L}_{QA}=-\frac{1}{N}\sum\limits_{k=1}^{K}\sum\limits_{i=1}^{N_{k}}&\big{[}\log
P_{\theta}(\mathbf{y}^{(k)}_{i,s}|\mathbf{x}^{(k)}_{i},\mathbf{q}^{(k)}_{i})+\\\
&\log
P_{\theta}(\mathbf{y}^{(k)}_{i,e}|\mathbf{x}^{(k)}_{i},\mathbf{q}^{(k)}_{i})\big{]}\end{split}$
(1)

###  3.3 Adversarial Training

Minimizing the cross-entropy as in equation (1) does not ensure that the model
will generalize on unseen domain. Rather it tends to overfit to certain
datasets. Inspired by GAN Goodfellow et al. (2014), we propose a simple yet
effective method to regularize the model such that it learns domain-invariant
features.

In the adversarial training procedure, QA model learns to make the
discriminator to be uncertain about its prediction. On the other hand, the
discriminator is trained to classify the joint embedding of question and
passage from QA model into the given $K$ domains. If the QA model can project
question and passage into an embedding space where the discriminator cannot
tell the difference between embeddings from different $K$ domains, we assume
the QA model learns domain-invariant feature representation.

We formulate the adversarial training as follows. A discriminator $\bm{D}$ is
trained to minimize the cross-entropy loss as of equation (2), where $l$ is
domain category and $\mathbf{h}\in\mathbb{R}^{d}$ is the hidden representation
of both question and passage. In our experiment, we use [CLS] token
representation from BERT for $\mathbf{h}$.

$\displaystyle\mathcal{L}_{\bm{D}}=-\frac{1}{N}\sum\limits_{k=1}^{K}\sum\limits_{i=1}^{N_{k}}\log
P_{\phi}(l^{(k)}_{i}|\mathbf{h}^{(k)}_{i})$ (2)

For the QA model, it tries to maximize the entropy of
$P_{\phi}(l^{(k)}_{i}|h^{(k)}_{i})$. In other words, it minimizes Kullback-
Leibler (KL) divergence between uniform distribution over $K$ classes denoted
as $\mathcal{U}(l)$ and the discriminator’s prediction as in equation (3).
Then the final loss for QA model is
$\mathcal{L}_{QA}+\lambda\mathcal{L}_{adv}$ where $\lambda$ is a hyper-
parameter for controlling the importance of the adversarial loss. In our
experiments, we alternate between optimizing QA model and discriminator.

$\displaystyle\begin{split}\mathcal{L}_{adv}=\frac{1}{N}\sum\limits_{k=1}^{K}\sum\limits_{i=1}^{N_{k}}{KL}(\mathcal{U}(l)\;\|\;P_{\phi}(l^{(k)}_{i}|\mathbf{h}^{(k)}_{i}))\end{split}$
(3)

##  4 Experiments and Result

###  4.1 Dataset

We validate our adversarial model for MRQA Shared Task with 6 different out-
of-domain datasets, which are BioASQ (BA) Tsatsaronis et al. (2012), DROP (DP)
Dua et al. (2019), DuoRC (DR) Saha et al. (2018), RACE (RA) Lai et al. (2017),
RelationExtraction (RE) Levy et al. (2017), and TextbookQA (TQ) Kembhavi et
al. (2017). Table 1 shows the statistics and description of these datasets.
Each dataset has about 1k samples. However, the number of samples from each
dataset varies. Thus, we use stratified sampling in order to make class-
balanced stochastic mini-batch having certain amount of samples from all
domains. We use maximum sequence length of 64 and 384 for question and passage
respectively. But some examples are longer than 384. Therefore each passage is
split into several chunks with a window size of 128. We discard samples
without answers because all questions are considered to be answerable from
given context in MRQA shared task.

Note that the final evaluation shown in the Table 2 is conducted by MRQA
organizers with additional 6 out-of-domain undisclosed private test datasets,
which are BioProcess (BP) Scaria et al. (2013), ComplexWebQuestion (CQ) Talmor
and Berant (2018), MCTest (MC) Richardson et al. (2013), QAMR (MR) Michael et
al. (2017), QAST (ST) Jitkrittum et al. (2009) and TREC (TR) Voorhees (2001).

###  4.2 Implementation Details

We implement our model based on the HuggingFace’s open-source BERT
implementation111https://github.com/huggingface/transformers in Pytorch Paszke
et al. (2017). The performance of the baseline in our experiment differs from
the official baseline of MRQA, which is based on AllenNLP Gardner et al.
(2018). We follow the hyperparameters as BERT for our model. In detail, we use
"bert-base-uncased" with a learning rate 3e-5 and a batch size of 64.
Additionally, our model requires one more hyperparameter $\lambda$, which
indicates the importance of adversarial loss as described in the equation (3).
We find out that the value of 1e-2 for $\lambda$ gives the best result in our
experiments. The baseline and adversarial model are trained on V100 GPU for
about 5 GPU hours. For training, we use 6 in-domain datasets, which are SQuAD,
TriviaQA Joshi et al. (2017), Natural Questions Kwiatkowski et al. (2019),
HotpotQA Yang et al. (2018), SearchQA Dunn et al. (2017), and NewsQA Trischler
et al. (2016) provided by MRQA. We select the best performing model on
validation set, where models are trained for 1 or 2 epochs. The codes for our
model are available at https://github.com/seanie12/mrqa.

###  4.3 Performance Comparison

Table 2 shows the performance evaluation results of models on out-of-domain
datasets. In the table, the model trained with our adversarial learning is
named with ’-adv’. The top of the table is the result of validation datasets
while the bottom is the result of test datasets. As shown in the table,
overall, the model with adversarial learning has better performance compared
to the baseline in terms of both EM and F1 measures.

For validation datasets, the average F1 score of our model is about $1.5$
point higher than the baseline. In detail, our model outperforms the baseline
in DP, DR, RC, and RA dataset by large margin. But the adversarial learning
degrades performance in BA and RE. We can see the same aspect in terms of EM
score. Similar to the result of validation datasets, our model shows better
performance in terms of EM (Exact Match) and F1 on the most of test datasets
except for ST. Overall, our model has superior performance with considerable
margin of over $2$ point in F1.

##  5 Discussion

In this section, we discuss some trials that have failed to improve the
performance but might be helpful for future works.

###  5.1 Span Refinement

QA sample consists of a question, a passage, and an answer span. There could
exist multiple answer spans because more than one phrase in the passage can be
matched with the answer text. For simplicity, only the first occurrence of
answer text is used for training in most of the baseline codes. However,
considering context and semantic of the given question and answer, a certain
phrase in the passage is more likely to be plausible answer span relevant to
the question. In order to find the most plausible answer span, a question and
sentences in the passage are encoded into fixed-size vectors with universal
sentence encoder Cer et al. (2018). We choose the span in a sentence, which is
the most similar to the question in terms of cosine similarity, as golden
span. In our experiment, this approach boosts up the performance of some
datasets but degrades the performance a lot in the other datasets.

###  5.2 Meta Learning

We apply meta learning to domain generalization Li et al. (2018a, 2019);
Balaji et al. (2018) to simulate train/test domain shift. For every epoch, one
dataset is randomly selected as virtual test domain. As described in Finn et
al. (2017), QA model is trained to maximize meta objective, which leads to
improve the performance in train domain, but also in test domain. But this
requires to compute Hessian-vector products, which slows down the training.
This is even worse for BERT because there are 110M parameters to fine-tune.
Moreover, contrary to the previous works, the meta learning for domain
generalization does not help improve the performance.

##  6 Conclusion

We leverage adversarial learning to learn domain-invariant features. In our
experiments, the proposed method consistently improves the performance of
baseline and it is applicable to any QA model. In future work, we will try
adversarial learning for pre-training model with diverse set of domains.

## Acknowledgments

We would like to thank Seonghan Ryu, Donghyeon Lee and Nicolas Remond for
their valuable feedback, as well as the anonymous reviewers for their
insightful comments.

## References

  * Balaji et al. (2018) Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. 2018.  Metareg: Towards domain generalization using meta-regularization.  In _Advances in Neural Information Processing Systems_ , pages 998--1008.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018\.  Universal sentence encoder.  _arXiv preprint arXiv:1803.11175_.
  * Chen et al. (2016) Xilun Chen, Ben Athiwaratkun, Yu Sun, Kilian Q. Weinberger, and Claire Cardie. 2016\.  Adversarial deep averaging networks for cross-lingual sentiment classification.  _CoRR_ , abs/1606.01614.
  * Chen and Cardie (2018) Xilun Chen and Claire Cardie. 2018.  Multinomial adversarial networks for multi-domain text classification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1226--1240, New Orleans, Louisiana. Association for Computational Linguistics.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  _arXiv preprint arXiv:1903.00161_.
  * Dunn et al. (2017) Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017.  Searchqa: A new q&a dataset augmented with context from a search engine.  _arXiv preprint arXiv:1704.05179_.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 1126--1135. JMLR. org.
  * Ganin et al. (2016) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016\.  Domain-adversarial training of neural networks.  _J. Mach. Learn. Res._ , 17(1):2096--2030.
  * Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew E. Peters, Michael Schmitz, and Luke Zettlemoyer. 2018\.  Allennlp: A deep semantic natural language processing platform.  _arXiv preprint arXiv:1803.07640_.
  * Ghifary et al. (2015) Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. 2015.  Domain generalization for object recognition with multi-task autoencoders.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2551--2559.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.  Generative adversarial nets.  In _Advances in neural information processing systems_ , pages 2672--2680.
  * Jitkrittum et al. (2009) Wittawat Jitkrittum, Choochart Haruechaiyasak, and Thanaruk Theeramunkong. 2009\.  QAST: Question answering system for ThaiWikipedia.  In _Proceedings of the 2009 Workshop on Knowledge and Reasoning for Answering Questions (KRAQ 2009)_ , pages 11--14, Suntec, Singapore. Association for Computational Linguistics.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  _arXiv preprint arXiv:1705.03551_.
  * Kembhavi et al. (2017) Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017.  Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4999--5007.
  * Khosla et al. (2012) Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A Efros, and Antonio Torralba. 2012.  Undoing the damage of dataset bias.  In _European Conference on Computer Vision_ , pages 158--171. Springer.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453--466.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017.  Race: Large-scale reading comprehension dataset from examinations.  _arXiv preprint arXiv:1704.04683_.
  * Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017.  Zero-shot relation extraction via reading comprehension.  _arXiv preprint arXiv:1706.04115_.
  * Li et al. (2017) Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. 2017.  Deeper, broader and artier domain generalization.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 5542--5550.
  * Li et al. (2018a) Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. 2018a.  Learning to generalize: Meta-learning for domain generalization.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Li et al. (2018b) Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. 2018b.  Domain generalization with adversarial feature learning.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Li et al. (2019) Yiying Li, Yongxin Yang, Wei Zhou, and Timothy M Hospedales. 2019.  Feature-critic networks for heterogeneous domain generalization.  _arXiv preprint arXiv:1901.11448_.
  * Liu et al. (2017) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017.  Adversarial multi-task learning for text classification.  _CoRR_ , abs/1704.05742.
  * Michael et al. (2017) Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2017\.  Crowdsourcing question-answer meaning representations.  _arXiv preprint arXiv:1711.05885_.
  * Muandet et al. (2013) Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. 2013.  Domain generalization via invariant feature representation.  In _International Conference on Machine Learning_ , pages 10--18.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017.  Automatic differentiation in pytorch.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  _arXiv preprint arXiv:1802.05365_.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  _URL https://s3-us-west-2.amazonaws.com/openai-assets/researchcovers/languageunsupervised/language understanding paper.pdf_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Richardson et al. (2013) Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013.  Mctest: A challenge dataset for the open-domain machine comprehension of text.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 193--203.
  * Saha et al. (2018) Amrita Saha, Rahul Aralikatte, Mitesh M Khapra, and Karthik Sankaranarayanan. 2018\.  Duorc: Towards complex language understanding with paraphrased reading comprehension.  _arXiv preprint arXiv:1804.07927_.
  * Scaria et al. (2013) Aju Thalappillil Scaria, Jonathan Berant, Mengqiu Wang, Peter Clark, Justin Lewis, Brittany Harding, and Christopher D Manning. 2013.  Learning biological processes with global constraints.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1710--1720.
  * Talmor and Berant (2018) Alon Talmor and Jonathan Berant. 2018.  The web as a knowledge-base for answering complex questions.  _arXiv preprint arXiv:1803.06643_.
  * Trischler et al. (2016) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016.  Newsqa: A machine comprehension dataset.  _arXiv preprint arXiv:1611.09830_.
  * Tsatsaronis et al. (2012) George Tsatsaronis, Michael Schroeder, Georgios Paliouras, Yannis Almirantis, Ion Androutsopoulos, Eric Gaussier, Patrick Gallinari, Thierry Artieres, Michael R Alvers, Matthias Zschunke, et al. 2012.  Bioasq: A challenge on large-scale biomedical semantic indexing and question answering.  In _2012 AAAI Fall Symposium Series_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998--6008.
  * Voorhees (2001) Ellen M Voorhees. 2001.  The trec question answering track.  _Natural Language Engineering_ , 7(4):361--378.
  * Wu et al. (2017) Yi Wu, David Bamman, and Stuart Russell. 2017.  Adversarial training for relation extraction.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1778--1783, Copenhagen, Denmark. Association for Computational Linguistics.
  * Xu et al. (2014) Zheng Xu, Wen Li, Li Niu, and Dong Xu. 2014.  Exploiting low-rank structure from latent domains for domain generalization.  In _European Conference on Computer Vision_ , pages 628--643. Springer.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  _arXiv preprint arXiv:1809.09600_.
  * Yogatama et al. (2019) Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. 2019.  Learning and evaluating general linguistic intelligence.  _arXiv preprint arXiv:1901.11373_.
  * Yu et al. (2017) Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017.  Seqgan: Sequence generative adversarial nets with policy gradient.  In _Thirty-First AAAI Conference on Artificial Intelligence_.
  * Zhang et al. (2017) Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017.  Adversarial training for unsupervised bilingual lexicon induction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1959--1970, Vancouver, Canada. Association for Computational Linguistics.


# A socio-demographic study of the exoplanet direct imaging community

Lucie Leboulleux  LESIA, Observatoire de Paris, Université PSL, CNRS,
Université de Paris, Sorbonne Université,
5 place Jules Janssen, 92195 Meudon, France  Élodie Choquet  Aix Marseille
Univ, CNRS, CNES, LAM, Marseille, France  Elsa Huby  LESIA, Observatoire de
Paris, Université PSL, CNRS, Université de Paris, Sorbonne Université,
5 place Jules Janssen, 92195 Meudon, France  Garima Singh  LESIA, Observatoire
de Paris, Université PSL, CNRS, Université de Paris, Sorbonne Université,
5 place Jules Janssen, 92195 Meudon, France  Faustine Cantalloube  Max Planck
Institute for Astronomy, Königstuhl 17, 69117 Heidelberg, Germany

###### Abstract

Astronomy and science are fields in which specific groups remain under-
represented despite multiple studies that investigate this issue and propose
solutions. In this article, we analyze the demographics and social behavior of
the exoplanet direct imaging community. Our focus is on identifying possible
under-representation among this group, and quantifying inappropriate social
behaviors. During the Spirit of Lyot conference 2019 (Tokyo, Japan), we
conducted a survey that gathered a participation rate of 53%. We analyzed the
data collected under the prisms of gender balance and seniority
representation. The proportions of women and of non-binary persons reveal a
more diverse community in comparison to the other scientific groups (e.g. the
IAU members), but still far from a balanced representation of all genders.
Early-career scientists appear to have a lower visibility in the field than
permanent researchers, with PhD students being under-represented at
international conferences, and postdocs being excluded from conference Science
Organizing Committees. Regarding social relations, the results are alarming,
in particular when it comes to self-censoring of women or to unprofessional
behavior, which was experienced by 54% of this community (gender-biased
behavior: 29%; oral interruption: 33%; inappropriate behavior: 33%), and in
particular by women. We recommend the community to become pro-active to build
a safe environment and to continue its inclusion efforts. One aspect could be
to systematically include socio-demographic surveys in conference registration
forms to monitor the evolution of the community, in particular at larger
scales. To do so, the survey questions available on GitHub.

minorities — gender — demographics — high-contrast imaging — exoplanets

††journal: BAAS

##  1 Introduction

Science, Technology, Engineering and Mathematics (STEM) are fields
traditionally affected by a low workforce diversity, despite inclusion being
repeatedly pointed out as necessary to increase the performance and the
quality of a workplace (Slater et al., 2008). In particular, the gender gap
has been studied from various angles: access to permanent positions (Berné &
Hilaire, 2020), responsibilities (Rathbun et al., 2015; Rathbun, 2017;
Piccialli & Rathbun, 2019), proposal acceptance rate (Reid, 2014; Lonsdale et
al., 2016; Patat, 2016; Spekkens et al., 2018), conference attendance
(Davenport et al., 2014; Prichard et al., 2019), citations (Caplar et al.,
2017), etc. – and on various scales: in general science and technology
(Casadevall & Handelsman, 2014), in astronomy (Cesarsky & Walker, 2010;
Norman, 2019), at the scale of a country (Spekkens et al., 2018; Berné &
Hilaire, 2020), of an instrument (Reid, 2014; Lonsdale et al., 2016), of an
institute (De Rosa et al., 2019; Hibon et al., 2018), of a sub-field of
astronomy (D’Orgeville et al., 2014), etc. Despite these multiple studies,
gender imbalance is still significant and an ongoing issue. In order to
stimulate changes of behavior and improve the situation, under-representation
and biased behaviors need to be recognized as an issue, outspoken, and
monitored within the STEM community rather than minimized or disregarded
comparatively to scientific questions. To do so, more studies are needed to
monitor the evolution of the demographics and of the behaviors, and to extend
the awareness to other minorities under-represented in science.

In astronomy, several recent studies focused on factors impacting the
visibility of women and having a direct effect on their professional evolution
and/or recognition. In particular, D’Orgeville et al. (2014) provided a
detailed overview of the reasons causing women to leave astronomy and more
specifically the field of adaptive optics. The authors pointed out numerous
factors including social misconduct towards women and the impostor syndrome of
which women are in majority subjects in this field. They also formulated a
number of propositions to counterbalance this effect in the future. Other
studies have focused on gender-based biases within selection committees that
affect the visibility of women in astronomy, such as conference speakers
selection, for instance at the American Astronomical Society (AAS) meeting 223
(Davenport et al., 2014) or at the 2014 to 2016 American Geophysical Union
(AGU) Fall Meetings (Ford et al., 2018), or observing program selection, for
example the Hubble Space Telescope (Reid, 2014) and the ESO (Patat, 2016) time
allocations. These biases observed in astronomy are similar to what is seen in
other fields in Science: in microbiology for instance, a study by Casadevall &
Handelsman (2014) also suggests that the gender balance within Science
Organizing Committees (SOC) and conference conveners directly impacts the
distribution of talks per gender at conferences, which has also been observed
by Nittrouer et al. (2018) in six disciplines (biology, bio-engineering,
political science, history, psychology, and sociology).

Other studies specifically addressed the impact of seniority on the gender
balance in astronomy and pointed out important dependencies between these two
aspects on demographic and behavior questions. For instance, Cesarsky & Walker
(2010) showed that the gender ratio evolves with age or career status,
suggesting that some gender-based results are degenerate with the career level
of the probed population. A few studies focused on disentangling the career
stage with the gender. For instance, Spekkens et al. (2018) showed that during
the Canadian time allocation process, gender is the only significant
discrimination parameter.

These numerous studies illustrate the importance of staying alert about
inequalities towards women and minorities in Science in general and in
Astronomy in particular. Capturing the demographics of a community is the
first step to: 1) identify under-representation issues, 2) develop solutions
to improve the inclusion of all groups and, 3) set a reference point to
monitor the (hopefully positive) evolution of this community. For these
reasons and to complement similar studies in other sub-fields of astronomy, we
probed the attendees of the Spirit of Lyot 2019 conference held in Tokyo,
Japan, and obtained the first demographic snapshot of the Exoplanet Direct
Imaging community.

The Spirit of Lyot (SOL) conference is a major international conference
gathering the community studying extrasolar systems with high-contrast imaging
instruments. Its main motivation is to bring together researchers with
different expertise ranging from observation to instrumental research and
development working towards the same scientific objective: the detection and
characterization of exoplanetary systems. This conference brings together a
large fraction of this community on a 4-year basis. The fourth edition of the
SOL conference was held in Tokyo in October 2019 where around 200 researchers
participated. Here, we report the outcome of the survey that was shared with
all the attendees of the conference, thus providing a large and representative
sample of the field of Direct Imaging of Exoplanets.

In section 2, we describe the survey proposed at the SOL conference and the
methodology used to analyze the data. Section 3 presents the general
demographic overview of the participants. Sections 4 and 5 describe and
analyze the results of the survey as a function of the gender and of the
career position of the participants, respectively. The conclusion section
summarizes the outcomes of our study, identifies its main limitations and
proposes solutions to overcome them.

##  2 Methodology

The results presented in this report were obtained from a survey conducted in
October 2019 at the Spirit of Lyot conference in Tokyo, Japan. The survey was
initiated during an unofficial splinter meeting which included around 12
female and non binary participants interested in gender studies. Discussions
during this meeting enabled to identify broad categories as well as specific
points interesting to be probed.

The complete list of questions asked in the survey is presented in appendix A.
An improved version is available on open source on GitHub:
https://github.com/lleboulleux/socio-demographic-community-survey-in-STEM.
Four main topics were addressed: the general demographics in the field, the
visibility and the ability of the participants to self-promote during the SOL
conference, their visibility and their recognition in the field in general,
and the occurrence of unprofessional behaviors in this community. While most
of the questions were objective and factual queries, a few others (questions
12 to 15, 17, 19, and 20) focused on the subjectivity or on the perception of
the participants. We should point that none of the authors has a background in
social science and the formulation and topics addressed by the questions may
benefit from researchers in social science to guaranty additional neutrality.
We encourage the community and in particular demographic studies experts to
either commit edits to the form in the Github repository and/or directly
contact the authors to improve the survey. We further discuss the implications
in Sec. 4 and 6, respectively.

The survey was opened on October 24th 2019 on the fourth day of conference,
and sent to all of the conference participants. It was advertised by the SOC
between conference sessions and by the Local Organizing Committe (LOC) on the
conference email list. The survey was open for three weeks and was closed on
November 14th. A reminder email was sent to the conference participants on
November 4th, at midpoint of survey open period. All the answers were
collected on a voluntary basis, which may bias the accuracy of the results as
some groups may be more responsive than others (e.g D’Orgeville et al., 2014).
These limitations are discussed specifically in sections 4 and 5.

In the following sections, we analyze the answer ratios of each question,
first for the participants of the survey as a whole, then for specific groups
among the participants. We computed uncertainties on these ratios at $66\%$
confidence level following the study by D’Orgeville et al. (2014), with
uncertainties given by $\sigma=\sqrt{\frac{M(N-M)}{N}}$, where $N$ is the
total number of people in the study group and $M$ the number of people of the
specific category analyzed in this study group. For instance, $N$ could be the
number of women while $M$ could be the number of people having answered ”yes”
to a specific question in the women group.

##  3 General demographics

We collected precisely 100 answers to the survey out of a total of 190
conference participants. This gives a participation rate of 53%, a value high
enough to provide statistical results representing the SOL 2019 conference
participants.

In Fig. 1 we present the main demographic characteristics obtained from these
100 answers.

|
---|---
|
Figure 1:  Overall demographics of the participants of the Spirit of Lyot 2019
conference: these graphics show the ratios of the participant to the survey as
a function of their gender, career level, field of expertise, and affiliation
country.

The gender balance in the survey participants is as follows: 69% defined
themselves as men, 29% as women, and 2% as non-binary people. These
proportions confirm the under-representation of women in this field and are
comparable to the ones in the other fields presented in the introduction (e.g.
27% of women individuals in the field of Adaptive Optics, D’Orgeville et al.
(2014)). Compared to the whole field of Astronomy, this proportion of women in
the field of Exoplanet Direct Imaging is lower than the one in AAS (34%,
Davenport et al., 2014), but higher than the fraction of women in the IAU
(18%, data from January 2018) and equivalent to the one attending the AGU Fall
Meetings (28%, Ford et al., 2018). The presence of non-binary people is also
encouraging, but can hardly be compared to any reference percentage since the
studies including them are unfortunately too rare and should be developed,
which Rasmussen et al. (2019) and Strauss et al. (2020) propose
recommendations for. Although encouraging, the representation of women and non
binary people at the SOL conference shows that significant efforts need to be
performed to reach an acceptable gender balance in this community.

In terms of career level distribution, 24% of the participants were PhD
students, 27% post-doctorate researchers, 47% faculty researchers, and 2
people did not answer this question. The participation to the conference was
thus at 51% for young scientists and 47% for permanent researchers. If we
assume an average ratio of two non-permanent positions per permanent position
in the community as deduced from observations in different institutes and
accounting for geographical disparities (higher ratio in the USA than in
European countries), we notice a significant under-representation of young-
career scientists at the 2019 SOL conference. In comparison, the
representation of PhD students at the AGU Fall meetings was 29% between 2014
and 2016, a ratio 5 points higher than at the Spirit of Lyot 2019 conference
(Ford et al., 2018). This indicates a community which favors self-promotion
(faculties are in charge of managing travel budgets) over the promotion of
young scientists in the team. This is perceived as a negative point for this
community given that PhD students and postdocs strongly rely on international
conferences to progress in their career and gain visibility.

The expertise of the conference attendees was fairly spread over three main
domains: instrumentation (39%), observations (33%), and a combined
observational and instrumental expertise (23%). This reflects the rational of
the Spirit of Lyot conference series, which specifically aims at bringing
together these fields of expertise related to the direct imaging of
exoplanetary systems, a field where astrophysical results depend on complex
instruments. In addition, a low but noticeable fraction of the attendees
claimed expertise in theory (5%), indicating that the field also requires
theoretical work to interpret their observations. Monitoring the progress of
the fraction of theoreticians over time may indicate an interesting evolution
of the field toward more fundamental research about the formation and
evolution of exoplanets and circumstellar disks.

Finally, the participants of the 2019 SOL conference came in majority from the
USA (45%) and from Europe (36%, mainly France and the Netherlands at 15% and
10%, respectively). The other continents were much less represented, with only
9% of the participants coming from South America (exclusively Chile), 8% from
Asia (mainly Japan, 7%), and no participant from Africa or Oceania. This is a
negative sign in terms of the geographic representation in this field, in
particular considering that several major instruments used by this community
are installed in South America and the conference was held in Asia. Among the
possible reasons for these geographic disparities are financial reasons
(different travel budgets), the importance of the field in each country (also
linked to the financial reasons through hiring resources), and interest in the
conference within each country.

In Tables 1 and 2, we show the local distributions of gender and career level,
respectively, among the participants as a function of their affiliation
country. We excluded the countries with only one answer to the survey from
this analysis (Belgium, China, Sweden). Although some ratios are affected by
small sample statistics and should be used carefully, significant differences
appear between some countries in terms of gender career category
representation. In particular, the three most represented countries, with more
than 10% of the participants, were the USA, France, and the Netherlands. We
see important differences between them: French participants were more balanced
in gender than the other two countries with 40% female scientists, but more
strongly favored the participation of the permanent researchers over young
scientists (40%). Conversely, Dutch participants had a large majority of male
participants (only 20% of women), but promoted their students and postdocs
more strongly (80% of young scientists) over the faculties. The US
participants were more balanced on these categories, and included the only non
binary scientists who answered the survey, and thus showed a more diverse
environment. However, their female and young-career scientists remain under-
represented, with only 24% of women, and 47% of PhD students and postdocs.

Table 1: Gender distribution among the participants for each country with more than one respondent to the survey. Country | Female | Male | Non binary
---|---|---|---
| (%) | (%) | (%)
USA | $24\pm 6$ | $71\pm 7$ | $4\pm 3$
France | $40\pm 13$ | $60\pm 13$ | $0\pm 0$
Netherlands | $20\pm 13$ | $80\pm 13$ | $0\pm 0$
Chile | $33\pm 16$ | $67\pm 16$ | $0\pm 0$
Japan | $43\pm 19$ | $57\pm 19$ | $0\pm 0$
Switzerland | $50\pm 25$ | $50\pm 25$ | $0\pm 0$
Germany | $33\pm 27$ | $67\pm 27$ | $0\pm 0$
Canada | $0\pm 0$ | $100\pm 0$ | $0\pm 0$
United Kingdom | $50\pm 35$ | $50\pm 35$ | $0\pm 0$
Table 2: Career level distribution among the participants for each country with more than one respondent to the survey. Country | Faculty | Postdoc | PhD Student
---|---|---|---
| (%) | (%) | (%)
USA | $49\pm 7$ | $29\pm 7$ | $18\pm 6$
France | $60\pm 13$ | $20\pm 10$ | $20\pm 10$
Netherlands | $20\pm 13$ | $30\pm 14$ | $50\pm 16$
Chile | $67\pm 16$ | $11\pm 10$ | $22\pm 14$
Japan | $57\pm 19$ | $14\pm 13$ | $29\pm 17$
Switzerland | $0\pm 0$ | $100\pm 0$ | $0\pm 0$
Germany | $33\pm 27$ | $33\pm 27$ | $33\pm 27$
Canada | $50\pm 35$ | $0\pm 0$ | $50\pm 35$
United Kingdom | $50\pm 35$ | $0\pm 0$ | $50\pm 35$

##  4 Gender-based analysis

###  4.1 Involvement of the participants

In order to analyze the interest of the conference participants to the survey
and estimate the accuracy of the results per gender category, we monitored the
evolution of the male and female respondents over three weeks during which the
survey was open. The results are plotted in Fig. 2, with the number of
participants per gender with time on the left and the fraction of each gender
category with time on the right.

In Fig. 2, left, we observe that the number of female and non-binary
participants approach their final values very early on during the survey
period (respectively within a few days and within a week of the opening date)
compared to the male participation, then show a near-flat slope. Conversely,
the slope of the male participation near the end of the survey period
(excluding the last 5 days when no new answer was received) is steeper. This
different slope indicates that the survey is more likely missing answers from
male participants than from female and non-binary participants and suggests a
different level of interest to gender studies between these categories.

Similarly, assuming the three gender groups had a similar interest in the
survey, they would have answered at the same rate and their fractions would
have quickly stabilized around their final value in Fig. 2, right, ie. $69\%$
for men, $29\%$ for women, and $2\%$ for non-binary people. If we exclude the
last 5 days when no answer was received, we see that this expected
stabilization never occurs and that the final ratios evolve until the last
answer. Later, this trend is even clearer when compared to the same analysis
performed between the career level groups in Fig. 9, right.

First, we conclude that we may have an over-representation of women and under-
representation of men in this study compared to the total participants of the
2019 SOL conference. Second, it shows that women answered on average earlier
than men, indicating a higher interest and involvement in the problems
addressed in this survey. Finally, we notice that the reminder email sent on
November 4th (vertical grey line of Fig. 2) had a significant impact on the
number of answers (+23), and mainly on the male participants (+20 answers).

Because of the low number of non-binary participants in the survey, the
results for this sub-group suffer from large uncertainties and does not
guaranty their anonymity in the next sections of the gender analysis. We thus
chose to limit the rest of the analysis to the male and female genders only.
We hope that conducting a similar study on a larger sample in the future will
enable to report the responses of marginalized genders while preserving the
anonymity of the people.

|
---|---
Figure 2:  Evolution of the number of answers per gender with time (left) and
of the proportion of each gender group with time (right). The vertical grey
lines indicate the date of the reminder email. We can observe that women tend
to answer faster than men to the survey, and the slight non-zero slopes on the
right plot at the closure of the survey indicates an under-representation of
male attendees to the survey and an over-representation of female attendees.
The participation of non-binary attendees reaches a plateau well before the
closure of the survey.

###  4.2 Distribution of career levels per gender

Differences in the distribution of men and women between the main career
status have been observed in many scientific communities, showing that women
are more often graduate students than permanent researchers in comparison to
male scientists (e.g. in the field of Adaptive optics, D’Orgeville et al.
(2014) or at the AGU Fall Meeting attenders, Ford et al. (2018)). Numerous
factors have been studied in D’Orgeville et al. (2014), all leading to the
conclusion that there is a leakage of women at each career step in the field
of astronomy, ultimately leading to fewer women at higher positions. For the
specific case of France, inequalities in access to the permanent positions is
evident (Berné & Hilaire, 2020), with a success rate to permanent positions
twice higher for men than for women. More generally, female scientists are
less promoted and less funded than their male colleagues, generating a gap in
the career distribution between the two genders (Shen, 2013). In this context,
studying the career distribution per gender in our survey is of particular
interest.

In Fig. 3, we compare the professional positions of the male and female
participants to the 2019 SOL conference survey. We see that the proportion of
women with non-permanent positions is slightly higher than for men, with
$31\%\pm 9\%$ of female postdocs versus $25\%\pm 5\%$ of male postdocs, and
$45\%\pm 9\%$ of female faculties versus $51\%\pm 6\%$ of male faculties. It
thus seems that women have more difficulties to access permanent positions
compared to male associates. However, the large uncertainties does not allow
us to draw a solid conclusion here. This is a phenomenon commonly observed in
other scientific communities that the fraction of women leaving academia
accumulates throughout the classical research path, from graduation to full
permanent position, in contrast to men associates (the _leaky pipeline_
phenomenon) (D’Orgeville et al., 2014; Ford et al., 2018). In Fig. 3, we also
observe that both gender groups have the same ratio of PhD students ($24\%$).
It indicates that the general issue of women under-representation starts at
the beginning of their career, with very few female students admitted in a PhD
program in this field.

As discussed in Sec. 4.1, the extrapolation of the gender-based results to the
whole Exoplanet Direct Imaging community may be limited by the likely under-
representation of men in the survey answers. Yet, we show in Sec. 5 that the
three career groups participated to the survey at similar rate, suggesting
that the career level distributions are accurate and representative of this
community.

Figure 3:  Career level distribution in both female and male gender groups.

###  4.3 Exposure and visibility at the Spirit of Lyot 2019 conference

|
---|---
Figure 4:  Proportions of women and of men who asked questions at the end of a
talk (left) and proportions of women and of men who asked for a poster pop
talk (right) at the SOL conference.

To analyse the exposure of each gender group at the 2019 SOL conference,
several questions in the survey specifically asked if the participants
requested a talk when they registered at the conference (question 7), and if
they were actually attributed a talk by the SOC (question 8).

From the answers, we derive that within the speakers, $67\%$ were men, $33\%$
were women, and none were non-binary people. The ratios for men and women are
close to the fraction of female and male participants at the conference and
indicate that the conference program was well representative of the binary
attendees. However, we can regret that no non-binary person obtained an oral
presentation.

The results show that men and women equally asked for contributing talks
($78\%\pm 5\%$ for men vs. $76\%\pm 8\%$ for women) and that women were
slightly more successful at obtaining one than men ($41\%\pm 6\%$ for men vs
$48\%\pm 9\%$ for women), although the difference is not significant given the
uncertainties. As a comparison, Ford et al. (2018) reports that at larger
scales (AGU Fall Meetings from 2014 to 2016), women are given fewer
opportunities than men to give oral presentations. As a reason for this
observation, they explain that women are predominantly PhD students compared
to men, and that students represent the least invited career group. In
addition, they notice that conveners are mostly men, who seem less likely to
give speaking opportunities to women.

The following results about self-confidence and self-promotion at the SOL
conference are also derived from the specific questions inquired in the
survey. Figure 4 (left) indicates that male attendees asked questions
significantly more frequently than women following the talks ($50\%$ of the
male participants vs $31\%$ of the female participants). This suggests that a
larger fraction of women in this community are subject to self-censorship than
men and did not consider the conference environment comfortable or benevolent
enough to bring themselves forward. This behavior has also been observed and
studied in other conferences such as the AAS meeting 223 (Davenport et al.,
2014) and the 2014 UK National Astronomy Meeting (Pritchard et al., 2014) and
on larger samples (Schmidt & Davenport, 2017). In particular, the survey of
the AAS meeting 223 (Davenport et al., 2014) showed that the sessions chaired
by women had a higher number of questions asked by women.

Finally, a similar trend is observed for the poster-pop presenters: $31\%$ of
the male participants vs $14\%$ of the female participants asked to advertise
a poster on stage (Fig. 4, right). Poster presentations were however open to
all participants without selection by the SOC. This second point thus
questions the appreciation of women for their own work and their confidence in
front of an audience. We received explanations from 8 women for not
volunteering to present their poster on stage, and sorted them in four
categories: lack of information about the opportunity (50%), lack of
confidence (25%), lack of interest in poster pops (12.5%), and lack of time to
prepare the poster pop (12.5%). The similar analysis of the 13 explanations
provided by male attendees also shows an important lack of information (31%),
but much more frequently a lack of interest (31%) or a lack of time (23%) for
the exercise. Only one man suggested a lack of confidence, and one mentioned
difficulties to have the poster presentation approved at the institutional
level.

###  4.4 Visibility and recognition by peers

|
---|---
Figure 5:  Proportions of women and of men who attended an international
conference in 2018 (left) and proportions of women and men having been invited
in the SOC of an international conference in 2018 (right).

In this section, we now extend the question of visibility and recognition to
the daily professional environment instead of just the 2019 SOL conference. To
address this topic, we focused on the general access to conferences, the
participation to SOCs, and the inclusion in peer-reviewed publications as co-
authors.

In terms of visibility and exposure to the community, $72\%$ of the male
participants and $62\%$ of the female participants attended an international
conference in 2018 (Fig. 5, left). This suggests that women may have higher
difficulties to access conferences than men, although the sample was too small
to guaranty the significance of this result. Given the importance of exposure
at conferences to be recognized by peers, to advertise projects, and to
promote one’s career, the trend observed here in the Exoplanet Direct Imaging
community should not be neglected and would need to be monitored and confirmed
with a larger sample (Prichard et al., 2019).

In terms of recognition within the field, the survey showed that in 2018,
slightly more women than men have been invited in the SOC of an international
conference ($21\%$ of the female participants vs. $14\%$ of the male
participants, see Fig. 5 right). The uncertainties are however also too large
to confirm this trend at a significant level. If confirmed on a larger sample,
it may indicate an effort in this community towards a better representation of
women at conferences. However, $50\%$ of the women having participated in a
SOC in 2018 perceived that they were invited to fulfill a gender quota, and
one of them specified that it was explicitly communicated to her. We note that
the formulation of the question in the survey asking about the impression of
filling a gender quota in a SOC may have been ambiguous about the time period
considered and about the type of SOC (e.g. international conference vs.
institutional committees, etc.), because two additional women answered
affirmatively while indicating that they have not been invited in the SOC of
an international conference in 2018. This ambiguity can be removed in future
surveys by specifying that the question 17 refers to the same SOC as question
16. In any case, these numbers question the real motivation of increasing the
representation of women in SOCs and on the recognition of their scientific
expertise within this community.

|
---|---
Figure 6:  Proportions of women and of men who felt being unfairly absent
(left) and unfairly present (right) in a publication co-author list.

Finally, the other aspect related to the recognition by peers probed in the
survey concerned the inclusion in publications as co-author. Figure 6 shows no
significant difference between the fractions of men and of women who have
sensed being unfairly excluded from a publication. We note that, independently
of the gender considerations, a large fraction of people considers having been
excluded from author lists ($\sim 30\%$ in total), which may indicate a larger
issue. Likewise, we see no significant difference between the fractions of men
and of women who felt unfairly present in the lists of co-authors ($\sim
18\%$). These two results indicate that there is no discrimination on gender
in the inclusion in publication author lists within the Exoplanet Direct
Imaging community. However, we note that, unlike the other questions of the
survey, these two questions called on a subjective feeling and the responses
were dependent on the sensitivity of the participant. This will be more
discussed in the conclusion.

###  4.5 Unprofessional behaviors per gender

In this section, we analyzed the feedback to the survey questions probing the
occurrence of and sensitivity to unprofessional behaviors in the Exoplanet
Direct Imaging community. Similar to the previous section, some of these
questions called on subjective interpretations of the respondents.

Figure 7:  Proportions of women and of men who have already been interrupted
or prevented from talking in a professional environment. A significant
difference can be observed between the perception of both gender groups.

First of all, the survey inquired if the participants had already been
interrupted while talking or prevented from talking at a professional event.
Independently of the gender considerations, the total number of people who
suffered from this behavior is 33%. It reflects poor attitudes, indicating a
lack of awareness in respecting boundaries among peers in the Exoplanet Direct
Imaging community. In addition, the gender-based results revealed that $23\pm
5\%$ of men versus $59\pm 9\%$ of women consider they have faced this issue
(Fig. 7). This shows a significant gender bias on this problem, with women
being 2.6 times more frequently interrupted than men. This result is
particularly appalling knowing that such conduct can impact one’s self-
confidence, leadership, and credibility, all indirectly linked to one’s
visibility, recognition, and thus career evolution. We recommend the community
to be more respectful in this aspect and to be watchful against disrespectful
interruptions, in particular of women.

|
---|---
|
Figure 8:  Proportions of women and of men who experienced an inappropriate
behavior (top left). Fractions of women and men who noticed a situation of
inappropriate behavior (top right). Fractions of women and men who experienced
a difference of behavior towards themselves due to their gender (bottom left).
Fractions of women and men who noticed a gender-based difference of behavior
towards somebody (bottom right).

Furthermore, $33\%$ of the participants have already been victims of an
inappropriate behavior in this community. This rate is alarming as it shows a
generalized and frequent behavior, revealing an unsafe work environment. It is
also significantly unbalanced between men ($25\%$ of male victims) and women
($52\%$ of female victims), with 5.1 times more women experiencing
inappropriate behaviors than men, as shown in Fig. 8 (top left). It implies
that significant changes of conducts are urgent in the field of high-contrast
imaging to make it a safe zone for everybody and for women in particular.

Complementary to these distressing results, we also observe that 48% of the
participants have already noticed a situation of inappropriate behavior, women
slightly more frequently than men ($55\%$ vs. $45\%$, respectively, see Fig.
8, top right). These values are somewhat encouraging as they indicate that
people of both genders are aware of inappropriate behaviors and are capable of
identifying such situations.

Finally, we obtain from the survey that a majority of the female participants
(69%) have experienced being treated differently because of their gender (see
Fig. 8, bottom left). In comparison, our data shows that this experience has
happened to a negligible fraction of the male participants ($13\%$). Such a
gender-based difference in behavior is not welcomed in a professional
environment because it affects the credibility, leadership, and confidence of
a person and influences his·her·their professional abilities. Thus we call for
a significant change of conduct within the Exoplanet Direct Imaging community.
Knowing that such gender-based changes of behavior may be performed
unintentionally, we encourage the community to become further aware of its own
prejudices, unintentional biases and be attentive and observing about one’s
own actions.

It is encouraging to note that 57% of the participants have already noticed
such gender-based differences of behavior. Although, Fig. 8 (bottom right)
shows that women are significantly more vigilant about such behaviors than men
($76\%$ vs. $51\%$, respectively).

Overall, the results from this particular analysis draw a rather unsafe
picture of the High-Contrast Imaging community. Combining the three types of
unprofessional behaviors probed in the survey (questions 12, 14, 18), 54% of
the community have experienced at least one of these situations (gender-biased
behavior: 29%; oral interruption: 33%; inappropriate behavior: 33%). For
women, the proportion of victims is 80% (gender-biased behavior: 69%; oral
interruption: 59%; inappropriate behavior: 52%). Such events occur twice more
frequently for women than for men, who are affected at 43% by unprofessional
behaviors.

These rates are alarmingly high and reveal an unsafe environment.
Inappropriate behaving needs to be considered as a general issue and should be
resolved urgently in our field. We observe that a large fraction of the
community (around 50%) generally notices inappropriate behaviors or gender-
based differences. According to D’Orgeville et al. (2014), inappropriate
behavior is one of the major problems causing women to leave academic
research. The results described for the Exoplanet Direct Imaging community
should thus be monitored and improved both to build a workplace safer for
everybody and to push for a better gender representation.

##  5 Results based on the career level

In this section, we analyze the results of the survey as a function of the
professional position of the participants: Faculty member (or similar types of
permanent position), postdoctoral researcher, or PhD student. About half of
the participants were faculty members (47%), and the other participants almost
equally split between postdoctoral researchers (27%) and PhD students (24%),
see Fig. 1. Two individuals in our data do not fit into these three predefined
categories and answered ”Other” in the career level options.

###  5.1 Involvement in the survey

Similar to the gender-based analysis, we analyzed the rate at which
participants from the different career groups responded to the survey (Fig.
9). First, we observe that the relative fractions between the three main
career categories have all stabilized around their final values within the
first two days of the survey period. Unlike the gender analysis, in which a
bias against the survey was identified for the male group, the present
analysis shows that all three career groups had a similar interest in the
survey as their answers were received at the same rate. It also indicates that
the survey accurately captured the actual ratios for each career group
attending the Spirit of Lyot conference, despite the 53% participation rate to
the survey.

Figure 9:  Evolution of the number of answers to the survey with time per
professional category, both in absolute values (left) and relative to the
total number of answers per day (right). The ratios of answers reach a stable
value for the three main professional categories as early as the second day of
the survey, unlike the gender balance rations. This shows a similar interest
to the survey by all three major career categories.

###  5.2 Gender balance per professional category

Figure 10:  Gender distribution per career group.

The current and the following sections focus only on the three major
categories (Faculties, Postdocs, PhD students) and do not include the two
”Other” entries as they are not statistically significant in this sample.

Figure 10 shows the gender balance in each professional category. It shows the
same results as Fig. 3 but with a different perspective. Here, we see that
gender imbalance exists in the three main career steps in the Exoplanet Direct
Imaging community. Furthermore, the figure also aims at illustrating the so-
called _leaky pipeline_ process in the community. At first glance, we observe
a rather constant proportion of women at each career step (29% of female PhD
student, 33% of female postdocs, 28% of female faculties). This tentatively
indicates that there is no obvious inequalities in the recruiting process
within this community. However, the large error bars (7-9%) are comparable to
the ratio fluctuations between the career groups, which prevents us from
drawing a firm conclusion for this field.

In addition, the low 29% ratio of female PhD student is not encouraging to
progress towards a better gender balance in the foreseeable future. Finally,
we note that non-binary individuals equally spread in the younger PhD student
and postdoc populations. This indicates that the Exoplanet Direct Imaging
community is slowly starting to be more inclusive and diverse. Monitoring the
evolution of these demographics in the next 5 years is necessary to assess
whether these individuals are offered equal opportunities to obtain faculty
positions.

###  5.3 Expertise per professional level

Figure 11:  Expertise distribution per career group.

Figure 11 shows the spread in expertise in each career group. We notice that
the expertise is equally spread between ”Instrumentation”, ”Observation”, and
”Observation and instrumentation” within the Faculty and Postdoc populations.
PhD Students tend to have a single expertise (Instrumentation 50%,
Observations 33%, Theory 8%), but rarely a combination of several (8% total).
It reveals an (implicit) policy in this community to develop a strong
expertise in a specific field at the PhD level and to diversify one’s skills
with additional expertise at the postdoc level.

We notice that there are significantly more PhD students working in
instrumentation (50%) compared to the senior groups (30-36%). It suggests that
instrumentation is very attractive to the undergraduate students and is a good
entry point for starting a career in astronomy. It also indicates that young
doctors who specialized in instrumentation are more likely to broaden their
expertise to observational skills than the ones with other expertise. This
trend may be explained by several reasons: 1) there is a disinterest for
instrumentation at the postdoc level, possibly explained by limited job
opportunities in astronomical instrumentation or ample opportunities in
private companies as compared to the other expertises, or 2) young
instrumentalist doctors have more opportunities to develop observational
skills, for instance after having commissioned an instrument or an instrument
sub-system.

###  5.4 Visibility and recognition by peers per career level

Figure 12:  Proportions of each career group who attended an international
conferences in 2018 (left) and who were invited into the SOC of an
international conference in 2018 (right).

Figure 12 (left) shows the exposure given to each career category in
international conferences. Figure 12 (right) presents the recognition level
received by each career category within the community through invitations to
join the SOC of an international conference.

Our data shows that postdocs receive the most exposure and visibility at
international conferences, with 89% of them who attended at least one
conference in 2018. Counting the participation to the 2019 SOL conference, it
shows that the large majority of postdocs attends an international conference
at least once per year. This is a positive result, as the postdoctoral
positions are based on short-term contracts, hence the postdocs are required
to actively advertise their work and expand their network in search for a new
position. Thus, this high visibility ratio demonstrates a healthy community
which encourages and supports the postdocs in their career developments.

However, only 46% of the PhD student participants attended an international
conference in 2018. We note that some of the PhD students attending the SOL
conference may not have yet started their PhD program in 2018, so this
percentage has to be taken carefully. Nevertheless, combined with the low
attendance of PhD students (24%) as compared to the permanent researchers
(47%), it strengthens the previous analysis that this community does not
promote its PhD students enough and prevents a fraction of them from attending
conferences. This is yet another downside of the community, knowing that these
events are critical for young researchers to promote their work and develop
their professional network. This behavior is likely to make it more difficult
for PhD students to find a postdoc position in the field.

In comparison, faculties receive ample amount of opportunities to regularly
attend international conferences. In addition to attending the 2019 SOL
conference, 70% of the faculty participants attended an international
conference in 2018. In addition to being the largest population of the SOL
conference (47% of the participants), this also strengthens the analysis that
faculties in this field expose and promote themselves much more often than
their PhD students.

Finally, Fig. 12, right, shows that only permanent researchers have been
invited to join the SOC of an international conference in 2018. Similarly, the
SOC of the 2019 Spirit of Lyot conference was exclusively composed of faculty
members. Although it can be argued that managing the scientific organization
of conferences requires some level of professional experience, the exclusion
of postdocs from SOCs demonstrates again a significant bias against young-
career researchers in this community. We recommend the Exoplanet Imaging
community to be more inclusive of young scientists in the future by supporting
PhD students to attend international conferences and include postdocs in their
scientific organization.

###  5.5 Inappropriate behavior

Figure 13:  Proportions of each career group who experienced (left) or noticed
(right) situations of inappropriate behaviors.

Figure 13 shows how the different professional categories are exposed to
inappropriate behaviors. On the left, we show the fraction of each category
that has experienced a situation of inappropriate behavior, and on the right
the fraction that has noticed such behaviors.

In the left panel, we notice a significant difference between the professional
categories in terms of experiencing an inappropriate behavior: the majority of
postdoc participants have been victims of inappropriate behavior (56%), about
twice more than the permanent astronomers (28%) and PhD students (21%).
Although all these ratios are high and concerning, this particularly high rate
at the postdoc level is notably alarming given the vulnerability of postdocs
to the professional insecurity, which is already a source of anxiety.

In the right panel, we notice that about half of the faculty community and a
majority of postdocs have noticed a situation of inappropriate behavior at
work or at a conference. On the one hand, this indicates that such behaviors
happen relatively often in this community, which is a concern. On the other
hand, it also indicates that the community is aware and attentive to such
situations, with the postdoc community being the most vigilant, with 63% of
them having noticed inappropriate behaviors in the past. The PhD student
community, however, seems relatively protected from exposure to the
inappropriate behaviors, with 29% of them having noticed such a situation.

It is particularly interesting to cross-correlate the occurrence of
inappropriate behaviors between the gender and career groups, to identify more
precisely the social groups which are most prone to such behaviors. In table
3, we show the fractions of people having experienced inappropriate behaviors
as a function of both their gender and career level. The non-binary and female
postdoc group appear to be the most exposed to inappropriate behaviors, with
80% of them who have gone through such conduct. In comparison, male PhD
students and male faculties are the most protected groups (19% and 21%,
respectively). This is concerning and immediate steps should be taken to make
this community safer, diverse and inclusive given how this well-known problem
forces under-represented scientists (women and non-binary persons) to quit the
field (D’Orgeville et al., 2014). The fact that it happens predominantly at
the postdoctoral stage, which is the most insecure in one’s career, is
particularly worrying as it discourages them to continue in the field and thus
contributes to a vicious circle working against a representative gender-
balanced community.

Table 3: Proportions of victims of inappropriate behaviors within each gender and career group. To preserve the anonymity of the non-binary participants despite their small sample, their answers are combined with those of women.  | Female & Non binary | Male
---|---|---
PhD students | 25 % | 19 %
Postdocs | 80 % | 41 %
Faculty members | 46 % | 21 %

###  5.6 Inclusion in publications

In Fig. 14, we show the fraction of people who mentioned being unfairly absent
in the author list of a publication within the different professional
categories. Here too we notice that the postdoc community most often declares
that they have been unfairly left out from the publication authorship (41% of
them), compared to the faculty (28% of them) and PhD student communities (25%
of them). This shared anxiety among the postdoc community may be enhanced by
the insecurity of their short-term position and a need for a high publication
rate to obtain a faculty position.

We do not report significant differences between these communities regarding
the feeling of having been unfairly _present_ in author lists (17% over all
the participants regardless of their professional category).

Figure 14:  Proportions of each career groups who have already felt being
unfairly absent from a publication author list.

##  6 Discussion and conclusions

In this paper, we presented an overview of the community working on the direct
imaging of exoplanetary systems based on a survey conducted at the Spirit of
Lyot 2019 conference in Tokyo, Japan. The questions of the survey focused on
several aspects about the general demographics, representation and self-
confidence markers during this conference, the equity of exposure and
recognition in the field through authorship, SOC invitation and access to
conferences, and the occurrence of inappropriate behaviors at the workplace.
The survey collected 100 answers, providing a 53% participation rate from the
conference attendees. In addition to the overall study, the results were
analyzed within two main categories: as a function of the gender and of the
career level of the participants.

From the global demographics analysis, we extract three main results:

  * •

Women are under-represented in the community, with a 29% representation rate.

  * •

Young-career scientists (PhD students and postdocs) were under-represented at
this conference, with a participation rate of 51% compared to a representation
in the field estimated between 66% and 75%.

  * •

Significant representation disparities exist between the different countries
present at the conference.

From the gender-based analysis, we gathered several key results. The principal
positive results are the following:

  * •

The proportions of PhD students, postdocs, and permanent researchers do not
significantly vary between men and women. Owing to the large uncertainties, we
cannot draw any conclusion about recruiting discrimination in the field.

  * •

At the Spirit of Lyot conference, the fractions of female contributing
speakers (33%) was comparable to the participation of women to the conference
(29%), showing an unbiased representation effort from the SOC.

  * •

Generally, in this field, a slightly higher fraction of women are invited in
the SOC of international conferences compared to men ($21\%$ of the female
participants vs $14\%$ of the male participants), which shows an effort
towards a better gender representation. However, a large fraction of these
women (50% of them) felt they were included in SOCs in order to fill a gender
quota which questions the ability of the community to recognize them for their
expertise.

However, several negative results are also reported:

  * •

Women were more subject to self-censorship than men at the conference, which
was deduced from two markers: they were significantly fewer to ask questions
after the talks ($31\%$ of them vs $50\%$ of the male participants), and they
were significantly fewer to advertise their poster on stage ($14\%$ of them vs
$31\%$ of the male participants). The former behavior was also observed in
other studies (Davenport et al., 2014; Pritchard et al., 2014; Schmidt &
Davenport, 2017).

  * •

The most striking and alarming result is that 80% of women reported having
experienced unprofessional behaviors, ranging from gender-biased behaviors
(69% of them), to interruptions while they were speaking (59% of them), to
inappropriate behaviors (52% of them). These rates are twice higher than for
men (43% of them experienced a type of unprofessional behavior). These
behaviors contribute to creating an unsafe working environment and to
undermining women’s confidence, credibility, and leadership.

On an encouraging note, a majority of the participants, of all genders
(although more often women), have already noticed such unwanted behaviors,
which shows that the community is overall aware of these issues and can be in
a position to intervene and stop such situations from happening.

From our career-based analysis, we mainly found that:

  * •

PhD students from this field are under-represented at conferences. This is
seen both with their low participation to the Spirit of Lyot 2019 conference
(24%, 5 points lower than for instance the AGU meetings) and with the low
fraction of them attending an international conference in 2018 (46%). In
comparison, faculties seem to more easily support themselves to conferences,
with a 47% fraction at the Spirit of Lyot conference and with 70% of them
attending an international conference in 2018. This may create difficulties
for students in this community to find a postdoctoral position at the end of
their thesis.

  * •

Postdocs in this field are excluded from the SOC of international conferences.
This strengthens the analysis that this community has a bias against young-
career scientists.

  * •

A majority of postdocs has already experienced inappropriate behaviors (56%),
significantly more than the PhD students and the faculties (around 25% for
each). In particular women and non-binary postdocs are predominantly victims
of such situations (78% of them). In comparison, 41% of the male postdocs have
encountered inappropriate situations. This is particularly alarming given that
postdoc is the most insecure type of position: it may discourage early-career
researchers especially female and non-binary scientists to continue working in
this field.

This survey can be used as a reference point for future similar studies in
order to monitor the evolution of the demographics and social behaviors in the
field of direct imaging of exoplanetary systems. We also point out the
systematic behaviors which need to be addressed so that the quality of the
work environment can be improved. In the prospect of developing a safer and a
well-represented community, we formulate a number of recommendations:

  1. 1.

Be proactive to prevent all sorts of professionally inappropriate behaviors,
ranging from micro-aggression (interruptions, biased comments…) to serious
aggression (harassment, intimidation,…).

  2. 2.

Be more inclusive to under-represented genders, in particular at the PhD
student recruiting level.

  3. 3.

In order to better promote young-career scientists, provide more support and
encouragement to the PhD students to attend international conferences, and be
more inclusive of postdocs when forming the SOC of conferences.

  4. 4.

Systematically implement a similar socio-demographic survey in the
registration form of future conferences and workshops.

The later point has the capability of enabling the future studies to further
monitor the evolution of gender representation and different biases over time,
and allow to refine the solutions to counterbalance these issues. Furthermore,
it would enable to increase the sample size and obtain more precise results.
It would also provide awareness and more visibility to the experience of
minorities including non-binary people. In particular, it was noted that many
of the gender studies in science remain biased towards non-binary people and
coming studies should follow the recommendations of Rasmussen et al. (2019)
and and Strauss et al. (2020). In order to help increasing the number of such
studies, an improved version of the proposed survey is now available in open
source on GitHub using this link: https://github.com/lleboulleux/socio-
demographic-community-survey-in-STEM. We encourage different communities to
use it as a template to homogenize socio-demographic studies and allow long-
term monitoring. We also invite the users of this survey to provide comments
and feedbacks to improve its completeness. We are particularly interested in
inputs and suggestions from researchers with a social science expertise.

This study adds up to an increasing number of publications studying the
community of researchers itself, instead of their research. While analyzing
the results, we also listed improvements that could make this survey more
precise on some aspects: 1) the questions about the inappropriate behaviors
could be more specific about the different types of behaviors targeted in the
study. These questions could even be repeated to probe the occurrence of a
range of behaviors, from unwanted (e.g. gender-biased comments in a
professional discussion) to inappropriate (e.g. sexual comments in a
professional environment) to serious offences (e.g. aggression, harassment).
2) Some questions could be more specific on the period considered (e.g.
questions 12–15, 17, 19, 20), which would remove ambiguities and make some
career-based results more accurate. 3) Additional topics could be probed with
this survey, such as the number of citations (see a very interesting study of
gender-based differences on this topic by Caplar et al. (2017)), the
occurrence of solicitations by scientific journals to review manuscripts
(Lerback & Hanson, 2017), and the responsibility in large projects (Piccialli
& Rathbun, 2019). More generally, social scientists are the experts for such
studies and should be involved to guaranty that the survey questions are not
phrased in a partial and suggestive way, but are expressed explicitly and
neutrally. In addition, studies also acknowledge the experience of other
under-represented groups in astronomy, such as LGBT (Kay, 2009; Richey et al.,
2019) or African American and Hispanic researchers (Nota et al., 2009).

##  Appendix A Survey sent to all participants to the Spirit of Lyot 2019
conference

The survey contained the following questions:

1) Choose the option that best describes your gender (Female / Male / Non
binary / Other / Prefer not to disclose)

2) Choose the option that best describes your gender identity (Cisgender /
Transgender / Other / Prefer not to disclose)

3) What country do you currently live in?

4) What kind of position do you occupy? (Intern / PhD student / Postdoc /
Faculty)

5) Did you attend an international conference in 2018?

6) What is your expertise? (Instrumentation / Observation / Theory)

7) At this conference (Lyot 2019): Did you ask for a talk?

8) At this conference (Lyot 2019): Did you get a talk?

9) At this conference (Lyot 2019): Did you ask for a poster pop talk?

10) At this conference (Lyot 2019): If not, why?

11) At this conference (Lyot 2019): Did you ask questions to a speaker at the
end of their talk?

12) HCI in general: Have you ever experienced, at work or at a conference, a
situation of inappropriate behavior?

13) HCI in general: Have you ever noticed, at work or at a conference, a
situation of inappropriate behavior?

14) HCI in general: Have you ever noticed, at work or at a conference, that a
person was behaving differently to you because of your gender?

15) HCI in general: Have you ever noticed, at work or at a conference, that a
person was behaving differently to somebody because of their gender?

16) HCI in general: Have you been invited to join a SOC of an international
conference in 2018?

17) HCI in general: Have you ever been invited to a SOC in order to fulfill a
ratio of minority?

18) HCI in general: Have you ever been cut off while talking or prevented from
talking at a professional event?

19) HCI in general: Have you ever been unfairly absent from a list of co-
authors?

20) HCI in general: Have you ever been unfairly present in a list of co-
authors?

21) Are there any comments you would like to share with us?

This work was co-authored by women and non-binary people having met at the
Spirit of Lyot 2019 conference in Tokyo, Japan. It received full help and
support from the LOC and the SOC of the conference and the participation rate
of $53\%$ indicates an implication of the conference audience. We sincerely
thank these three entities and particularly the SOC and LOC, who supported
this initiative and helped advertise it among the conference attendees. The
authors are sincerely grateful to the referee for their constructive comments,
which improved the quality of the paper and strengthened its conclusions. LL
has received support of IRIS Origines et Conditions d’Apparition de la Vie
(OCAV) of PSL Idex under the program Investissements d’Avenir with the
reference ANR-10-IDEX-0001-02 PSL. GS would like to acknowledge the funding
received from the European Union’s Horizon 2020 research and innovation
programme under the Marie Skłodowska-Curie grant agreement No 798909.

## References

  * Berné & Hilaire (2020) Berné, O., & Hilaire, A. 2020, Nature Astronomy, 4, 296, doi: 10.1038/s41550-020-1068-5
  * Caplar et al. (2017) Caplar, N., Tacchella, S., & Birrer, S. 2017, Nature Astronomy, 1, 0141, doi: 10.1038/s41550-017-0141
  * Casadevall & Handelsman (2014) Casadevall, A., & Handelsman, J. 2014, mBio, 5, doi: 10.1128/mBio.00846-13
  * Cesarsky & Walker (2010) Cesarsky, C., & Walker, H. 2010, Astronomy and Geophysics, 51, 2.33, doi: 10.1111/j.1468-4004.2010.51233.x
  * Davenport et al. (2014) Davenport, J. R. A., Fouesneau, M., Grand, E., et al. 2014, arXiv e-prints, arXiv:1403.3091.  https://arxiv.org/abs/1403.3091
  * De Rosa et al. (2019) De Rosa, G., Oliveira, C., Pacifici, C., et al. 2019, in BAAS, Vol. 51, 25.  https://arxiv.org/abs/1907.04880
  * D’Orgeville et al. (2014) D’Orgeville, C., Rigaut, F., Maddison, S., & Masciadri, E. 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9148, Proc. SPIE, 91481V, doi: 10.1117/12.2059088
  * Ford et al. (2018) Ford, H. L., Brick, C., Blaufuss, K., & Dekens, P. S. 2018, Nature Communications, 9, 1358, doi: 10.1038/s41467-018-03809-5
  * Hibon et al. (2018) Hibon, P., Péron, M., Primas, F., et al. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10704, Observatory Operations: Strategies, Processes, and Systems VII, 107040K, doi: 10.1117/12.2315165
  * Kay (2009) Kay, L. 2009, in Women in astronomy and space science: Meeting the Challenges of an Increasingly Diverse Workforce, Vol. 2009, P42
  * Lerback & Hanson (2017) Lerback, J., & Hanson, B. 2017, Nature, 541, 455, doi: 10.1038/541455a
  * Lonsdale et al. (2016) Lonsdale, C. J., Schwab, F. R., & Hunt, G. 2016, arXiv e-prints, arXiv:1611.04795.  https://arxiv.org/abs/1611.04795
  * Nittrouer et al. (2018) Nittrouer, C. L., Hebl, M. R., Ashburn-Nardo, L., et al. 2018, Proceedings of the National Academy of Sciences, 115, 104, doi: 10.1073/pnas.1708414115
  * Norman (2019) Norman, D. 2019, in WG Annual Report (2019) - Women in Astronomy, IAU WG Reports
  * Nota et al. (2009) Nota, A., Bruff, S., Flanagan, K., Mountain, M., & Durand, B. 2009, in astro2010: The Astronomy and Astrophysics Decadal Survey, Vol. 2010, P42
  * Patat (2016) Patat, F. 2016, The Messenger, 165, 2.  https://arxiv.org/abs/1610.00920
  * Piccialli & Rathbun (2019) Piccialli, A., & Rathbun, J. A. 2019, in EPSC-DPS Joint Meeting 2019, Vol. 2019, EPSC–DPS2019–1312
  * Prichard et al. (2019) Prichard, L., Oliveira, C., Aloisi, A., et al. 2019, in BAAS, Vol. 51, 22\.  https://arxiv.org/abs/1909.10996
  * Pritchard et al. (2014) Pritchard, J., Masters, K., Allen, J., et al. 2014, Astronomy and Geophysics, 55, 6.8, doi: 10.1093/astrogeo/atu245
  * Rasmussen et al. (2019) Rasmussen, K., Maier, E., Strauss, B. E., et al. 2019, in Bulletin of the American Astronomical Society, Vol. 51, 75.  https://arxiv.org/abs/1907.04893
  * Rathbun (2017) Rathbun, J. 2017, Nature Astronomy, 1, 0148, doi: 10.1038/s41550-017-0148
  * Rathbun et al. (2015) Rathbun, J. A., Dones, L., Gay, P., et al. 2015, in AAS/Division for Planetary Sciences Meeting Abstracts #47, AAS/Division for Planetary Sciences Meeting Abstracts, 312.01
  * Reid (2014) Reid, I. N. 2014, PASP, 126, 923, doi: 10.1086/678964
  * Richey et al. (2019) Richey, C. R., Lee, K. M. N., Rodgers, E., & Clancy, K. B. H. 2019, in Bulletin of the American Astronomical Society, Vol. 51, 0206
  * Schmidt & Davenport (2017) Schmidt, S. J., & Davenport, J. R. A. 2017, Nature Astronomy, 1, 0153, doi: 10.1038/s41550-017-0153
  * Shen (2013) Shen, H. 2013, Nature, 495, 22, doi: 10.1038/495022a
  * Slater et al. (2008) Slater, S., Weigand, R., & Zwirlein, T. 2008, Business Horizons, 51, 201, doi: 10.1016/j.bushor.2008.01.003
  * Spekkens et al. (2018) Spekkens, K., Cofie, N., & Crabtree, D. 2018, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10704, Proc. SPIE, 107040L, doi: 10.1117/12.2314973
  * Strauss et al. (2020) Strauss, B. E., Borges, S. R., Faridani, T., et al. 2020, arXiv e-prints, arXiv:2009.08247.  https://arxiv.org/abs/2009.08247