0 Research please specify

# How Visualization PhD Students Cope with Paper Rejections

Shivam Agarwal†<EMAIL_ADDRESS>Shahid Latif†
<EMAIL_ADDRESS>and Fabian Beck<EMAIL_ADDRESS>†
Equal contribution
paluno, The Ruhr Institute of Software Technology
University of Duisburg-Essen, Germany

###### Abstract

We conducted a questionnaire study aimed towards PhD students in the field of
visualization research to understand how they cope with paper rejections. We
collected responses from 24 participants and performed a qualitative analysis
of the data in relation to the provided support by collaborators, resubmission
strategies, handling multiple rejects, and personal impression of the reviews.
The results indicate that the PhD students in the visualization community
generally cope well with the negative reviews and, with experience, learn how
to act accordingly to improve and resubmit their work. Our results reveal the
main coping strategies that can be applied for constructively handling
rejected visualization papers. The most prominent strategies include:
discussing reviews with collaborators and making a resubmission plan, doing a
major revision to improve the work, shortening the work, and seeing rejection
as a positive learning experience.

Introduction

To appear in Celebrating the Scientific Value of Failure (FailFest) Workshop
at IEEE VIS 2020.

Facing paper rejections in academia is common and sometimes has adverse
effects on individuals. Existing research studies the effect of paper
rejections on scholars [7, 4] and provide some guidelines on how to cope with
such experiences (e.g., [3, 9]). However, the studies were conducted for
management and statistical sciences and the results might not necessarily
generalize to the visualization community due to the diversity in reviewing
models and culture. Additionally, these studies did not focus on PhD students
specifically. We are interested in this group because we want to understand,
among other things, if and how PhD students are negatively affected by paper
rejections as well as what are their strategies to cope with it. PhD students,
as new and inexperienced members of the community, might suffer more from
negative reviews and paper rejections. If badly timed, a single rejected paper
can lead to significant delays in completing the PhD dissertation or even
threaten its successful completion.

Within visualization research, there exists literature that outlines common
reasons of a paper rejection and provides guidelines for preparing a
manuscript for submission [8]. Additionally, senior researchers of the
visualization community have made suggestions on how to deal with paper
rejections. Elmqvist [5, 6] described some strategies on how to handle paper
rejections while acknowledging that young PhD students are most vulnerable.
Taking a wider perspective, Shneiderman [10] laid his thoughts on a new
paradigm of research providing guidance to the students, junior and senior
researchers, and policymakers. However, we lack research that studies how PhD
students in the visualization research community perceive and handle paper
rejections.

In this paper, we aim at understanding and revealing the strategies of
visualization PhD students in coping with paper rejections. We conducted an
online study to gather responses from our target user group, i.e., PhD
students working in visualization research community. Based on the responses,
we performed a qualitative data analysis and report our findings. We provide
our questionnaire, raw data, and coded responses as a supplemental material
[1].

##  1 Study Design

We decided to use a short online questionnaire for the study to reach
visualization PhD students internationally. To not bias the participants
towards certain answers and to also discover unexpected strategies, the
questionnaire used open-ended questions for addressing the main research
questions. Hence, a mostly qualitative analysis is required to identify common
themes and noteworthy exceptions. To capture the full breadth of the
community, we strove towards diversity of participants with respect to the
country of affiliation and level of experience (among PhD students).

###  1.1 Questionnaire

The online questionnaire consists of three sections: general information,
publication history, and handling paper rejections. The questions in the first
section focus on knowing the stage of participants’ doctoral studies,
countries of their universities, and field of study during their bachelor and
master studies. The second section has two questions and aims at gauging the
research experience of participants in terms of submitted and rejected number
of peer-reviewed research articles. The third and final section captures the
experience of participants with the papers that were rejected and re-
submitted. All questions in the third section were open-ended questions and
participants could write as much as they wanted. In particular, we asked the
following four questions:

  * •

Q1: How did your lab-mates/colleagues/supervisors help to handle paper
rejections?

  * •

Q2: What strategic decisions did you take to re-submit the rejected papers?

  * •

Q3: How did you handle the papers that got rejected more than once? (if any)

  * •

Q4: How did your impression of paper reviews change over time?

These questions focus on capturing the possible ways or steps that students
may have taken to cope when they receive a paper rejection. The questionnaire
was designed to be completed in approximately 25–30 minutes.

###  1.2 Target Participants and Recruiting

The target participants were the PhD students from the visualization community
who have submitted research articles to peer-review venues and have already
experienced one or more paper rejections. This target group ranges from first
year PhD students to the ones who have recently (in the past 6 months)
graduated or left the PhD program. The questionnaire was posted to the IEEE
VIS and Infovis.org mailing lists as well as to the Discord channel used by
EuroVis 2020 and Euro Graphics 2020. We also forwarded the questionnaire
within our professional network by email and advertised it on Twitter.

In total, we received 43 responses until the submission date. In this paper,
as responses have been still coming in during data analysis, we focus on the
first 24 responses and plan on including the later responses in an extended
version of this paper. The following analysis and descriptions are purely
based on the first 24 responses.

###  1.3 Analysis Methodology

To analyze the responses, we followed an open coding process. The process
began with two of the authors independently coding the first five responses,
followed by a discussion on a merged set of authoritative codes. This resulted
in a list of 19 codes. In the next stage, we divided the remaining responses
into two equal parts (9 and 10 responses in these groups) and each coder used
the list of already established 19 codes to do the coding. During this second
stage of coding, both coders still came across new types of comments and
occasionally had to define new codes. These codes were then immediately added
to the shared list of codes. Notifying the other coder allowed him to use the
newly defined code as well. This process, once again, resulted in some partly
redundant codes that were again merged and resolved in another follow-up
meeting. This refinement left us with a set of 28 different codes. Now, both
coders used this set of codes to validate each other’s coding. The resulting
conflicts during the validation phase were then resolved in subsequent
meetings, which also led to a further consolidation to 25 codes (cf. Table 1).

Figure 1: Stage of PhD studies of the participants.  Figure 2: Participants by
country of affiliation.  Figure 3: Number of submitted and rejected articles
per participant.

##  2 Results

Based on the assigned codes (Table 1), we report the results per question
(Q1–Q4), each discussing the most frequently mentioned types of answers and
noteworthy individual examples. We begin with summarizing the general
information and publication history of the participants. We numbered the
participants with identifiers P1–P24. Individual participants are listed when
referring to groups of five participants or fewer; else, we just provide the
count of participants, but details can be found in the supplemental material
[1].

Table 1: List of codes along with their frequency of occurrence per question and total (sorted by total). Codes | Frequency
---|---
Q1 |  Q2 |  Q3 |  Q4 |  Total
improvement |  4 |  13 |  11 |  4 |  32
downgrade work |  – |  16 |  5 |  – |  21
discuss reviews |  19 |  – |  1 |  – |  20
impression gets better |  – |  – |  – |  12 |  12
see rejection as positive |  – |  1 |  4 |  5 |  10
resubmission plan |  9 |  – |  – |  – |  9
similar venue scope |  – |  9 |  – |  – |  9
painful reviews at first |  – |  – |  – |  9 |  9
informal discussion |  6 |  – |  – |  2 |  8
impression remains the same |  – |  – |  – |  7 |  7
submit to journal |  – |  6 |  – |  – |  6
fair reviews |  1 |  – |  – |  5 |  6
high-level reflection of reviews |  3 |  – |  – |  2 |  5
discuss reviews quickly |  3 |  – |  – |  – |  3
contradictory reviews |  – |  – |  – |  3 |  3
different venue scope |  – |  2 |  – |  – |  2
disagree with reviews |  – |  – |  – |  2 |  2
no special discussion |  1 |  – |  – |  – |  1
lack of support from coauthors |  1 |  – |  – |  – |  1
encouragement from supervisor |  1 |  – |  – |  – |  1
dealing with rejections meeting |  1 |  – |  – |  – |  1
supervisor’s decision |  – |  1 |  – |  – |  1
post on arxiv |  – |  1 |  – |  – |  1
unclear reviews |  – |  – |  – |  1 |  1
get early feedback |  – |  1 |  – |  – |  1
Total |  49 |  50 |  21 |  52 |  172

###  2.1 Participants

Among the 24 participants, about 84 percent of the participants had done a
master degree in computer science, and almost 63 percent had completed a
bachelors in computer science. As shown in Figure 1, a majority of the
participants (22) were PhD students ranging from first year to sixth year and
the ones who graduated in the past six months. One respondent (P2) had left
the PhD program and one participant (P22) had graduated two years ago.
Participants were enrolled at universities in 10 different countries, with
half of the responses coming from Germany and USA ( Figure 2). None of the
participant claimed of being enrolled with multiple universities. Among the
participants, the number of submitted and rejected peer-reviewed articles
ranged from 0 to 26 and 0 to 12 respectively (Figure 3). One participant (P8,
a second year student) did not submit any article to date. Another participant
(P21, a first year student) had not experienced any rejection. But since both
of them provided valuable comments in their responses, we decided to still
include them in our analysis.

###  2.2 Support of Collaborators (Q1)

As seen from Table 1, the most mentioned support strategies within the
collaboration groups of participants were to discuss reviews (19 participants)
and make a concrete resubmission plan (9 participants) with collaborators of
the paper. Six participants reported to have had informal discussions (e.g.,
vis-à-vis during a coffee or lunch break, or via instant messaging) with their
colleagues. They either discussed the reviews quickly (P3, P9, and P14) or
celebrated the success of their peers and commiserated with the others who
faced a rejection (P15 and P24). For getting support within the work group,
the similarity of research topics seems to play a role as P14 commented: “I
just talked briefly with my supervisor about it. Since I’m the only person in
my lab doing visualization research, the other ones don’t know about the
problems so much in detail.”

Towards improving the submission, participant P7 complained about the lack of
support from coauthors in revising or reworking the paper and mentioned that
they did not have background in visualization research. Four participants (P2,
P7, P10, and P12) discussed the exact improvements and feasibility of these
improvements for their paper with coauthors. Three experienced students (P10,
P13, and P19) stated that they did a high-level reflection of reviews to
understand what went wrong and to find areas of improvement.

###  2.3 Strategies for Resubmission (Q2)

When it comes to resubmitting rejected papers, prominent strategies are either
downgrading work to resubmit in a lower-ranked venue (16 participants) or
making improvements in the work (13 participants). The improvement might
involve considerably extending the work with additional contributions (P12) or
by including a user study (P1, P3, and P6). Both P1 and P12 stated that they
converted short papers to full papers. The participants who decided to go for
an improvement either resubmit their papers to a similar scoped venue (9
participants) or to a different scoped venue (P7 and P15). It could be better
to switch to a different venue for better aligning contributions with the
target audience as P15 stated: “In the case of one paper the reviewers
commented that the novelty/contribution to the visualization community was
limited, so here we submitted the application-based paper to a workshop with a
better audience fit […].”

Being afraid that somebody else might publish similar research first,
participant P24 mentioned that they had posted the paper on arXiv and
described it as: “[…] if we were scared of being ‘scooped’ we would put it on
arXiv and go for the next venue. If no external pressure, we hold on to next
year’s visualization-related conference.”

The responses also indicated few submission strategies of the papers.
Participant P7 reported his/her tendency of submitting a poster to get early
feedback: “I tend to like to do posters, because I can try to think about how
to present the work visually, and in verbal discussions with others. This is
generally helpful for me.” Another strategy is to submit to a journal (6
participants) rather than waiting for the next conference cycle. P4 responded
that he/she prefers to directly submit to journals as it provides more review
cycles (major and/or minor revisions) in contrast to conferences where a major
revision is considered as a reject. Participant P14 also favored this strategy
as some journals in visualization community (e.g., TVCG) offers the
presentation of accepted work at any major visualization conference.

###  2.4 Handling Multiple Rejects (Q3)

Eleven participants mentioned that they had made further improvements before
resubmitting the papers that were rejected more than once. Similar to handling
first rejections of their papers, the participants reported to downgrade work
by resubmitting to lower-rated venues (P7, P15, P19, P20, and P23). None of
the participants mentioned to have abandoned their work after getting it
rejected multiple times. However, participant P7 highlighted that shortening
the work could be a good strategy to avoid spending more time on a project.
Four participants (P7, P14, P15, and P20) mentioned that they had seen
rejection as a positive learning experience while handling papers that got
multiple rejects. The response of participant P7 summarizes this positive
outlook: “My attitude is ‘this will get in somewhere, eventually’. I have seen
my papers improve dramatically after revising and resubmitting, so I tend to
think of rejections as a good thing, because the paper wasn’t ready and will
only get better each time I try again. If it were to be on a topic where I
didn’t feel like making substantial changes, I might stop at submitting a
short paper or a poster, but I haven’t encountered this yet.” – P7 However,
participant P14 also highlighted the difficulty while handling papers that got
multiple rejections as: “It was really hard to not give up. There were many
tears. But in the end, you have to stand up again and improve your work.”.
Eleven participants either did not provide any answer to the question or
mentioned that, until now, they had not faced multiple rejections of the same
paper.

###  2.5 Impression of Rejected Paper Reviews (Q4)

Nine participants highlighted that it was painful to read the reviews at
first, especially, on the notification day. Considering how the subjective
impression of the reviews changes over time, half of the participants (12)
mentioned that their impression got better, while seven participants reported
that their impression remained the same. Four participants (P5, P6, P7, and
P15) said that they saw rejection as positive. A response from participant P6
summarizes a strategy to avoid taking reviews personally: “[The impression of
the reviews] fairly remained the same. I always took it constructively and
know that rejections are part of the science process. It’s not [a] reflection
on me personally or my skills.”.

Five participants (P3, P12, P14, P20, and P22) reported that overall they had
gotten fair reviews while two of them (P3 and P22) disagreed with reviews on
some points. Three participants (P1, P7, and P14) reported that sometimes
reviewers had provided contradictory reviews, while participant P18 mentioned
receiving unclear reviews. Participant P13 mentioned a strategy of not
considering each point in a review; instead he/she had learned to do a high-
level reflection of reviews with experience: “My perception of specific
reviews did not change much with time, but my general view on reviews over the
years changed a lot, in such a way that I am not any longer simply considering
each point in the review, but am asking myself more often, what went wrong in
the communication of the content.” – P13 Two participants (P15 and P24)
reported doing informal discussion of the reviews. Elaborating on their
responses, participant P24 said that “[…] Talking with peers who also
submitted papers on review day also adds to stress if your paper wasn’t
accepted.” On the other hand, participant P15 mentioned that going through the
reviews with friends as a social event helps him/her to remain positive,
especially when some comments in the reviews are disheartening.

##  3 Study Limitations

The geographical diversity of the participants is limited to 10 countries from
Europe and North America only. Moreover, 12 out of 24 (50%) responses are from
Germany and USA. We believe this might be the result of using our personal
networks to circulate the questionnaire. Therefore, the analysis results may
be dominated by the work culture of visualization PhD students in Europe and
North America.

Since the questions in the third section of our questionnaire were open-ended
and broad, we provided some examples for each question to specify along which
lines participants can think of while answering the questions. On the one
hand, those examples were good to encourage participants to share their
thoughts in case they felt lost. But this might have influenced the responses
to some extent on the other hand.

Along similar lines, we might have biased the participants by advertising the
study in a way that implied that they might feel frustrated about rejected
papers—this was done to attract the attention of more participants. In
general, just focusing on rejected papers takes a perspective that cuts short
examples where papers were improved in a short revision cycle and then
directly accepted. Since we had to compromise on the extensiveness of the
questionnaire, we cannot make statements about this or other aspects of the
submission process and strategies.

##  4 Discussion

Based on the responses, we can infer that PhD students in the field of
visualization research take the reviews seriously. The students seem to
discuss reviews and formulate a resubmission plan and continue to make further
improvements in their work. The students frequently use the strategy of
downgrading work while resubmitting, either by shortening the length or opting
for a lower-rated venue. It seems that the strategies work well as they
eventually get their papers published. This also indicates that visualization
research community has many venues and tracks where the improved work
eventually gets published. The strategies reported by visualization PhD
students—discussing reviews, improving the work, and resubmitting it to
another venue—align well with the suggestions made by Shneiderman [10, Section
6.6] and Elmqvist [5].

Although the responses of PhD students indicate that reviews are painful at
first, they try to see rejection as a positive event. Some students mentioned
that they try not to take the reviews personally and highlighted that
rejections do not mean incompetency, which resonates with the advice by
Elmqvist [6]. Other students also mentioned that engaging in informal
discussions with colleagues and friends helps them cope with paper rejection.
Knowing about these strategies can be helpful especially for young PhD
students who have limited knowledge about the normalcy of receiving rejections
in a peer-review research process. It maybe a good idea to talk about
rejections, both formally and informally, and discuss strategies on how to
cope with them. The discussions potentially creates a social support system
that helps in coping with rejections and matches well with the recommendation
by Day [3].

There were few responses which highlighted the occurrences of unclear and
contradictory reviews. Even though these factors contribute to the painful
experience on the day of notification, PhD students in visualization research
usually handle them positively. As a result, many students highlighted that
their impression of reviews gets better with time. Some students reported that
they had used their judgment and sometimes had disagreed with few comments in
the reviews. Additionally, some experienced students mentioned doing a high-
level reflection of reviews trying to understand the issues that might have
misdirected the reviewers. These strategies can also be helpful for finding
areas of improvement and formulating a plan of action, especially when the
reviews are unclear, have contradictory views, or contain insufficient
information to back the review rating. The high-level reflection of reviews
can be done in different ways. Elmqvist suggested summarizing the reviews [5]
while Shneiderman shared personal experiences of handling difficult reviews
and highlighted using one’s own judgment to continue improving the work [10,
Section 6.6].

Since nine respondents (37.5%) indicated to have experienced emotional
distress on the day of rejection notification after reading the reviews, it
may be indicative of a harsh textual connotation being used in the reviews. We
argue that conscious efforts can be made by reviewers to proactively minimize
the painful rejection experience. A more polite text tone over a harsh comment
may be used to communicate the same review. Although not always possible,
concrete suggestions for improving the work can be clearly highlighted.
Moreover, alternative reviewing models can be explored and adopted that have
helped in reducing the harsh tone; Besançon et al. [2] concluded in a recent
survey that reviewers use a more polite tone in open and non-anonymized peer
reviewing model.

##  5 Conclusion and Future Work

Based on the responses we can infer that most PhD students in the
visualization research have a positive outlook to cope with paper rejections.
However, sometimes the positive outlook does not correlate with the absence of
a painful experience while facing rejections (e.g., P14). We hope that knowing
about the strategies of others can help the students who are facing paper
rejections. Also, we hope that initiating discussions on the normalcy of paper
rejections in academia and talking about known strategies in coping with
rejections can be helpful especially to the young PhD students joining a
research team.

As part of future work, we would be interested in discerning the strategies of
students having different levels of experience. Another aspect would be to
validate the results of this analysis by getting expert feedback of
experienced reviewers from the community. Also, we plan to extend the analysis
to the complete set of responses.

## Acknowledgments

We wish to thank all the researchers who participated in our study and shared
their thoughts and experiences. We are indebted to Ben Shneiderman and Niklas
Elmqvist who pointed us to valuable resources on the subject matter.

## References

  * [1] S. Agarwal.  Supplementary material: How visualization PhD students cope with paper rejections, 2020. doi: 10 . 17605/osf . io/vgj36
  * [2] L. Besançon, N. Rönnberg, J. Löwgren, J. P. Tennant, and M. Cooper.  Open up: a survey on open and non-anonymized peer reviewing.  Research Integrity and Peer Review, 5(1):1–11, 2020. doi: 10 . 1186/s41073-020-00094-z
  * [3] N. E. Day.  The silent majority: Manuscript rejection and its impact on scholars.  Academy of Management Learning & Education, 10(4):704–718, 2011. doi: 10 . 5465/amle . 2010 . 0027
  * [4] M. S. Edwards and N. M. Ashkanasy.  Emotions and failure in academic life: Normalising the experience and building resilience.  Journal of Management & Organization, 24(2):167–188, 2018. doi: 10 . 1017/jmo . 2018 . 20
  * [5] N. Elmqvist.  Dealing with rejection.  https://sites.umiacs.umd.edu/elm/2016/10/25/dealing-with-rejection/, 2016\.  accessed: July 2020.
  * [6] N. Elmqvist.  Above all, persistence.  https://sites.umiacs.umd.edu/elm/2018/08/16/above-all-persistence/, 2018\.  accessed: July 2020.
  * [7] S. A. Horn.  The social and psychological costs of peer review: Stress and coping with manuscript rejection.  Journal of Management Inquiry, 25(1):11–26, 2016. doi: 10 . 1177/1056492615586597
  * [8] T. Munzner.  Process and pitfalls in writing information visualization research papers.  In Information Visualization: Human-Centered Issues and Perspectives, pp. 134–153. 2008. doi: 10 . 1007/978-3-540-70956-5_6
  * [9] D. B. Rubin.  Converting rejections into positive stimuli.  In Past, Present, and Future of Statistical Science, pp. 593–603. 2014. doi: 10 . 1201/b16720-55
  * [10] B. Shneiderman.  The new ABCs of research: Achieving breakthrough collaborations.  1st ed., 2016.