30 min technical talks in parallel sessions. Themes are two or three cross-
cutting sessions identified for each year. Minisymposia are topical sessions
organized by different scientific communities.

Birds of a Feather (BoF) Sessions

self-organized sessions where communities of interest can meet, talk, and
plan.

Lightning Talks

5 min talks (no going over!) for an hour every afternoon on late-breaking news
and whimsical projects.

Posters

traditional static and semi-interactive presentation _en masse_.

Sprints

self-organized, intense development of Python packages in the SciPy ecosystem.

In a typical year, tutorials run in two or three parallel sessions on Monday
and Tuesday; the conference is held Wednesday through Friday, and sprints run
through the night from Saturday to Sunday.

####  3.10.2 Virtual SciPy 2020

After briefly debating delaying the meeting, SciPy 2020 was held, as
scheduled, from July 6 to 12, 2020. In the transition to a virtual format, we
kept the key elements of a traditional meeting, but rearranged the schedule,
shown in Fig. 15,

Figure 15: Schedule for SciPy 2020, showing virtual platforms used for
different portions.

to hold live tutorials in the mornings, followed by live keynotes, and then
live question & answer sessions, lightning talks, and BoFs in the afternoons,
all over the course of five days. Sprints were held after the conference, on
Saturday and Sunday, as usual. For this first, purely virtual meeting, we
reduced registration fees, inclusive of tutorials, to US$75 standard/US$25
student (fees for in-person SciPy 2019 were US$500 standard/US$225 student,
with the same charges again for tutorial attendees). As SciPy traditionally
draws a North American audience, meeting times were kept in Central Daylight
Time (there is a long history of other regional meetings, e.g., EuroScipy,
SciPy India, SciPy Latin America, and SciPy Japan).

SciPy talks have been recorded for many years and distributed via YouTube
since 2013 [21]; the difference for 2020 being that Theme and Minisymposia
talks were pre-recorded and made available via the SciPy YouTube channel
several days in advance; speakers were given a few minutes to recapitulate
their talk at the beginning of their respective question & answer session.
Tutorials and Birds of a Feather were held in two parallel sessions; all other
elements were held sequentially. Poster authors were responsible for hosting
their own content asynchronously. After an initial kickoff using the Crowdcast
conference platform, individual sprint leaders set up their preferred mode of
communicating and streaming with their team.

Tutorials and live sessions were hosted on the Crowdcast platform. This
platform afforded four "hosts" to accommodate speakers and session chairs. For
conventional sessions, this worked well, but for non-traditional elements like
BoFs and lightning talks, with the potential for many speakers in quick
succession, more hosts would have been beneficial.

SciPy meetings have had dedicated Slack workspaces for many years, with both
official and self-organized channels. The Crowdcast platform included a chat
feature that was used heavily, but it did not offer any sort of direct
messaging capability and it was challenging to correlate participants between
Crowdcast and Slack.

Due to the limited number of channels and hosts per channel allowed by
Crowdcast, informal themed sessions were planned for Zoom breakout rooms; due
to technical difficulties, these were switched to Google Meet at the last
minute.

More than 1400 attended at least one live session and more than 1200 attended
at least one tutorial. This substantial increase can likely be attributed to
attendees not needing to travel and to the significantly reduced registration
fees. Many attendees from overseas emphasized that the virtual format was what
enabled them to attend and they expressed the hope that SciPy would retain a
virtual component in the future.

####  3.10.3 Plans for Virtual SciPy 2021

With more time to plan and a better idea what to expect than in 2020, we
strove to recapture as much of the feeling of the in-person meetings as
possible. To achieve this,

  * •

we returned to a conventional schedule, illustrated in Fig. 16, of 2 days of
tutorials, 3 days of conference, and 2 days of sprints.

Figure 16: Schedule for SciPy 2021, showing virtual platforms used for
different portions.

  * •

all talks will be live. Talks will still be recorded and made available
afterwards on YouTube, as they have been for more than a decade, but we will
not be posting pre-recorded talks.

  * •

we are changing the primary meeting platform to Airmeet. This change was
motivated by scaling considerations, as the new platform allows up to 10
concurrent sessions with 10 presenters, each. A "green room" for presenters to
prepare before going live and live technical support were additional
attractions of this platform.

  * •

we will host the more interactive/social elements, e.g., BoFs, posters, and
general networking on the gather.town avatar-based virtual meeting space. This
will enable us to have both large and small meeting spaces, as well as a
nostalgic recreation of SciPy’s original home at CalTech.

  * •

sprints will be hosted on Discord. This platform was favored for collaborative
coding due to its combination of video streaming, screen sharing, and
chatting. Sprint organizers are free to choose a different platform if it
works better for their team.

  * •

we will again have a Slack workspace for SciPy 2021, in addition to chat
features in Airmeet, gather.town, and Discord.

  * •

we increased registration to US$125 standard/US$50 student in order to support
the improved attendee experience via a wider range of platforms this year.
These rates are still well below the in-person rates, as well as the rates
charged by many domain-specific society virtual conferences. Some sponsor
funds are available to support attendees for whom these rates are a hardship.

SciPy 2021 is scheduled for July 12 to 18, 2021.

####  3.10.4 Lessons learned

  * •

Many more people could attend a virtual meeting who would otherwise be limited
by time, cost, travel policies, or family constraints. Several appreciated the
reduced environmental impact of virtual attendance. Overseas attendance was up
significantly.

  * •

Conversely, work and family tend to be less forgiving for those who are not
really “away” at a meeting.

  * •

Many people miss the informal and social interactions of the traditional in-
person meeting.

  * •

Pre-recorded talks need to be “dropped” far enough in advance for people to
view them prior to any live discussion/Q&A session. For attendees already
experiencing "Zoom-fatigue", the prospect of spending all night watching
videos as soon as the live meeting is over can be exhausting.

  * •

Interactive, dynamic, or social elements of a meeting either require more
people with "hosting" privileges or a platform that allows all participants to
engage equally.

  * •

Quasi-anonymity makes managing conduct issues a challenge. Lack of direct-
messaging capability on some platforms and correlating identities between
different platforms can make it difficult to address concerns in private.

###  3.11 US ATLAS / Canada ATLAS Computing Bootcamp

####  3.11.1 Introduction

The US-ATLAS Computing Bootcamp is an ongoing annual bootcamp designed to
educate newcomers to the ATLAS Collaboration — in particular graduate students
— on the core technical computing concepts and tools that are used throughout
ATLAS and the broader HEP community. The goal of the bootcamp is to have all
students have basic proficiency in the computing tool-chain for a typical
ATLAS analysis and an understanding of how to use them. The scope and format
of the bootcamp is modeled off of the widespread software workshops by The
Carpentries and the training formats of the High Energy Physics Software
Foundation (HSF). A similar workshop was planned by Canada ATLAS in 2020, but
given the pandemic was canceled. The Canada ATLAS instructor team contained
guest instructors from US-ATLAS, and so the Canada ATLAS instructors joined
with the US-ATLAS instructor team to form a joint bootcamp that allowed for a
total of 44 students from both countries to participate.

The 2020 Bootcamp covered:

  * •

Version control with Git

  * •

Use of CERN’s Enterprise Edition version of GitLab

  * •

Building software with CMake and ATLAS CMake

  * •

Containerization with Docker

  * •

Continuous integration of code and delivery of analysis software with GitLab
Pipelines

  * •

Analysis preservation and reinterpretation with RECAST

  * •

Fundamentals of machine learning

  * •

Sustainable software development

and was held entirely online with all the resources being publicly available
from the bootcamp website [22, 23].

####  3.11.2 Bootcamp Format

In addition to having all of the bootcamp materials be publicly available
online, the 2020 bootcamp had all instruction and interactions occur through a
Discord server that was setup for the bootcamp, seen in Figure 17. Discord
offered a unified platform for instruction, discussion, and problem solving
across text, audio, and video communication. This makes for a very effective
platform for a teaching environment like a bootcamp, where the video feed of
the instructor team and technical discussion are able to coexist on a single
platform. Additionally, unlike Zoom, the text support for Discord supports a
flavor of Markdown that provides syntax highlighting, which is useful tool
when trying to communicate subtle differences in code that could otherwise be
difficult to see. An example of this is also seen in Figure 17 where a student
has posted code with no syntax highlighting at the top of the screen and an
instructor has replied with syntax highlighting enabled. Syntax highlighting
exists across many communication platforms, like Mattermost and Slack, but the
advantage of it in Discord is the ability to have it integrated into live
discussions with video.

Figure 17: View of the US-ATLAS Computing Bootcamp Discord server where
students and instructors are interacting on questions related to ATLAS CMake
and RECAST.

Another strong advantage of Discord is the ability to move seamlessly between
existing video and audio channels or “rooms“. This is critically useful when
teaching, as it allows for a student in the main instructional video stream to
signal an instructor that they need assistance, have an instructor and the
student move to a breakout room with audio/video and text support to iterate
on the student problem, and then move back to the main stream without needing
to request permissions or support from an administrator. Lowering the barrier
for students to interact with an instructor and get help without disrupting
the main discussion improves the learning experience, and is similar to the
kind of interaction an instructor and student would have at an in-person
bootcamp.

Discord is designed to be extensible and has a rich plugin and “bot” ecosystem
for adding support and services to the platform. For the 2020 bootcamp, the
instructional team enabled the "Dyno bot" which is provides user customizable
dashboarding, moderation, and interaction tools. For example, as seen in
Figure 18, an instructor is able to quickly generate an interactive poll in
the text channel for the main video room by using Dyno’s markup syntax. The
poll dynamically updates and allows for users to change their response through
time, allowing for instructors to gauge real time progress of the entire class
and be able to direct instructor time to students that have questions or are
behind without having to single individuals out. While they were not needed or
used for the bootcamp, Discord has a robust selection of extensible moderation
tools that have been well vetted and improved upon given Discord’s use as a
community interaction platform.

Figure 18: A member of the instructor team uses Dyno’s markup syntax to
generate an interactive poll to gauge student progress.

Areas of potential concern or consideration regarding Discord do exist.
Discord is publicly available, but has additional features as paid services.
The free tier of the service has limited video streaming quality, though for a
trivial price the bootcamp was able to purchase a bandwidth extension that
delivered video quality that was as good, if not superior, to Zoom’s.
Similarly, the free tier has a limited number of simultaneous video streams
per room, which mean that only one speaker in the general “auditorium” room
could share their video at a time, or in a student and instructor breakout
room. Additionally, in-platform video recording is not supported unless users
are using Open Broadcaster Software (OBS), which can be problematic if users
are not experienced with setting up video recording software. Self hosting of
the servers for Discord is also currently not supported, which could be
problematic for use in some countries given the Mozilla Foundation rates
Discord’s data privacy practices as not being particularly robust [24].

####  3.11.3 Summary

The use of a Discord server for the 2020 US-ATLAS and Canada ATLAS Computing
Bootcamp had a large positive impact on the success of the bootcamp. In
addition to creating a centralized virtual location for all discussion and
instruction across text, audio, and video, the lower barrier for rapid
interaction between students and instructors when students had questions was
critical to students learning. Having a centralized location for all
interactions also kept the bootcamp focused and avoided overwhelming
participants with application fatigue. The server is persistent as well, so
the discussions and resources added to the Discord server are still useful
references at the time of writing in 2021.

It is worth noting though that the use of Discord for bootcamps or workshops
is not contingent of the events being fully remote. It is possible to still
have student peer interactions and discussions that leverage the Discord
server’s strengths, and possibly serve as a bridge in hybrid in-person and
remote workshops between the remote and local participants. While the
experience of debugging code with a student while physically next to them is
preferable and not replicated in a fully remote environment, the shared rooms
and tools on the Discord server made the experience significantly better than
attempting to have the interactions be over a text chat only.

###  3.12 Neutrino 2020

####  3.12.1 Neutrino 2020 Virtual Conference

The Neutrino conference series is the largest conference in neutrino physics.
It is an international conference which has been hosted around the world every
other year since 1972. The Neutrino 2020 [25] conference was slated to be held
in the U.S. in June 2020 and planning had begun for a 5-day in-person
conference in downtown Chicago, Illinois. In March of that year, given the
realities of the COVID-19 pandemic, the conference was completely retooled to
an online conference. The Neutrino 2020 team, including 3 conference co-
chairs, 11 local organizing committee members, 5 session chairs, and
additional support from Fermilab staff had less than 3 months to prepare for a
virtual conference. The conference was hosted online by Fermilab and the
University of Minnesota on eight 1/2 days over the course of two weeks from
June 22 - July 2, 2020. Conference registration was handled via Fermilab
Indico. Conference content was archived both on the Neutrino 2020 website [25]
and using Zenodo [26].

Neutrino 2020 was an unexpectedly large success. It brought a very large
number of neutrino physicists together during a challenging time. While recent
in-person Neutrino conferences have typically had roughly 600 to 900
participants in attendance, the Neutrino 2020 conference had record attendance
with 4 350 people from every continent, including Antarctica, participate in
the online conference. The conference reached participants from 67 countries,
roughly 60 % of whom were students or post-docs (Figure 19), thus allowing
access to a much broader and more diverse set of conference participants.

Figure 19: Demographics of Neutrino 2020 conference participants by career
stage. More than half of the participants were students and postdocs. The
"other" category includes the general public, members of the press, etc.

####  3.12.2 Technical Aspects

The Neutrino 2020 conference included four main components:

  * •

Plenary talks were broadcast live via a Zoom webinar Monday through Thursday
from 7:00 to 11:30 CDT across the two weeks of the online conference.
Conference participants were given Fridays and the weekends off during the
event. During the conference, 79 plenary talks were delivered by neutrino
physicists around the globe. These talks were recorded live and posted daily
[27] so that conference participants who were not able to attend the live
presentation sessions could view the talks after they were presented and still
be a part of online discussions of their content over Slack. There were over
62 000 visits to the Neutrino 2020 web page to view recorded talks during the
conference. Increased support provided by Fermilab was needed to handle the
web load.

  * •

A web portal [28] was created to host the conference poster session. This web
portal was custom built for the conference to host the 532 conference posters
(a Neutrino conference series record) as well as an optional 2 minute pre-
recorded summary video that was uploaded by poster presenters. This too was
popular. There were over 5 800 views of the posters on YouTube throughout the
duration of the conference. The web portal also offered an alternative to the
virtual reality in case participants were not able to attend the live poster
sessions in the virtual reality platform.

  * •

Conference interactions took place via Slack. During the conference, over 23
000 communications were posted on Slack. Separate Slack channels for each of
the conference plenary sessions were created in advance as well as a channel
for interacting directly with the conference organizers. Conference
participants were also allowed to create their own channels. An especially
popular channel, for example, was a Jobs Board on Slack advertising job
openings in neutrino physics.

  * •

Perhaps the most popular aspect of Neutrino 2020 was an innovative Virtual
Reality platform [29] that was used to host the three live poster sessions
during the conference. Over 3 400 participants visited the virtual reality
platform that included $>500$ conference posters in themed rooms. See section
3.12.3 for more on the Neutrino 2020 Virtual Reality portal.

The Neutrino 2020 conference was promoted daily on social media via both
Facebook and Twitter and conference organizers also hosted a public event on
the last day of the conference in the form of an online live Physics Slam
[30].

####  3.12.3 Virtual Reality at Neutrino 2020

To give conference participants something new to try and to increase personal
interaction during the entirely online event, Neutrino 2020 hosted an
innovative virtual reality platform [29] using open source software from
Mozilla Mozilla Hubs. Conference registrants had access to this virtual
reality platform that gave poster presenters an opportunity to share their
work and conference participants a chance to interact. The virtual reality
platform was available for the duration of the conference, although due to
popular demand it was kept live for a few days even after the conference had
ended. Using the virtual reality platform, participants had a chance to get
immersed in a virtual poster session where they were able to see the posters
and interact live with poster presenters during dedicated poster sessions.
Visitors were able to create custom avatars and able to move within and
between the virtual reality rooms. When a person approached another person’s
avatar, people could speak and hear each others voices. Participants also had
the chance to explore virtual sight-seeing of Fermilab and downtown Chicago as
well as visit dedicated virtual social rooms to meet up with colleagues.
During the conference, 3,409 conference attendees visited the virtual reality
platform, including the 532 early career scientists who presented posters of
their research. The virtual reality platform received overwhelmingly positive
feedback by those who attended and on social media [31]. An example of the
space can be seen in Figure 20. For more views of the Neutrino 2020 Virtual
Reality platform, please also see [32].

Figure 20: Example view inside the Neutrino 2020 Virtual Reality platform that
hosted the 532 neutrino physics posters created by conference participants.

####  3.12.4 What Worked Well

In the end, the response to the first-ever virtual Neutrino conference was
very positive [31, 32, 33]. There are several aspects that helped make this
large online conference a success:

  * •

Dress rehearsals for the local organizing committee were held weekly in the
two months leading up to the conference to give conference organizers
experience running a Zoom webinar and in working together as a team.

  * •

Written job descriptions were created for each task needed to run the
conference. These assignments included hosting the webinar, speaker
assistance, talk recording and posting, session chairing, Q$\&$A handling in
the webinar, poster assistance, Slack monitoring, virtual reality platform
assistance, email list monitoring, social media content posting, and Zoom
webinar technical support.

  * •

An attempt was made to anticipate possible problems in advance. A written
trouble-shooting guide was created by the local organizing committee that
included a list of possible things that could go wrong during the conference,
who would respond, and how one should respond.

  * •

At the end of each day during the conference, a regrouping session was held
for the local organizing committee so that the group could discuss and solve
any issues that arose during the conference that day.

  * •

A Code of Conduct [34] set expectations for behavior during the conference.
Conference participants could not register for the conference or receive
connection details without agreeing to the Code of Conduct. An online
reporting system was also made available during the conference. Reports of
violations of the Code of Conduct could be made anonymously.

  * •

Conference organizers asked plenary speakers to provide recordings of each of
them pronouncing their own names. These audio recordings were made available
to session chairs to ensure correct pronunciation of presenters’ names.

  * •

Pulling together advice from a variety of sources, detailed instructions were
provided to speakers including a checklist and tips for giving a talk over
Zoom. Other conferences may also find these speaker tips helpful [35].

  * •

Rehearsal sessions for plenary talk presenters were arranged during the weeks
leading up to the conference so that speakers could practice sharing their
slides and test their audio and video. Practice sessions were also set up
during the first 1/2 hour before the conference on each day of the event in
case speakers wanted to do a last minute check of their setup.

  * •

Conference organizers set up multiple ways for conference participants to
communicate with the organizers during the conference including over email,
Slack, and in the virtual reality platform so as to be able to ask direct
questions of the conference team. A Frequently Asked Questions (FAQ) page [36]
was also posted online for conference participants and updated daily.

####  3.12.5 Lessons Learned

Lessons learned from Neutrino 2020 were twofold. First, it would, of course,
had been better had the conference organizers had more than 2.5 months to pull
together this large scale online event. Unfortunately, due to the timing and
uncertainty related to COVID-19, more planning time was not possible. Second,
the conference would have benefited from a larger support staff to handle
real-time conference registrations, as registration was allowed during the
event and many participants expected to be able to attend the conference on
the same day as having registered. The latter was an unanticipated large task
with upwards of 100 people per day registering for the conference as word got
out about the event on social media and in the community.

####  3.12.6 Summary

Neutrino 2020 was hosted as an entirely online event from June 22 to July 2,
2020. This virtual conference was put together by a small team of $\approx 15$
people in 2.5 months and included plenary talks, a web portal for posters, a
popular virtual reality platform, and an online chat forum via Slack. The
conference had record attendance: 4 350 attendees from 67 countries in all 7
continents, $60\leavevmode\nobreak\ \%$ of whom were students and postdocs.
This reach included many scientists who might not have otherwise been able to
attend an in-person conference because of funding, visas, family
responsibilities, or other issues. Given this extensive and diverse
participation, some aspects of Neutrino 2020 may well affect the planning and
organization of future in-person and online conferences.

###  3.13 Common Themes and Key Findings

Several themes from the experience talks and subsequent discussion emerged:

  * •

Virtual meetings come with several benefits such as increased participation by
early-career scientists and international collaborators, smaller overall
budgets are required to run an event, decreased cost of attendance and reduced
carbon footprint associated with the event (due to a lack of required travel).

  * •

Roughly 25%-50% of registered participants are connected at a given time and
this decreases over the event, while the vast majority of registrants attend
the event at some point over its duration (see Figures 13 and 14, for
example).

  * •

Challenges include a lack of personal interaction opportunities and diminished
quality of those interactions, “Zoom fatigue”, attention capture and
retention, and synchronous participation over wide range of time zones.

  * •

Zoom is an adequate current tool for “traditional” presentation-style, but
other tools are needed to be able to approach in-person style interactivity
(many great ideas shown at the workshop!).

##  4 Tools and Techniques

For virtual conferences at a minimum we need a way for all participants to
hear each other. This could be as simple as a conference call. More subtle
communications can occur if the voice call is supplemented with video feeds of
the speaker and other participants. This is provided by many of the different
platforms. Slides and online demonstrations of software can add additional
context and information. Most of the video conferencing systems can allow for
someone to share their screen.

In this way, the simple video conferencing systems can provide the basics of
what people experience in an in-person conference. There is additional tooling
that can be brought to bear that can extend this conference experience and, in
some ways, take participants further than has been possible in an in-person
environment.

###  4.1 Virtual Whiteboards

Visual communication is a very powerful medium. It is common for workshop
presenters to include diagrams on their slides and for organizers to use
Google Docs for collaborative note taking during the meeting which often also
includes diagrams. These diagrams are static and cannot reflect new knowledge
gained during the session or the collective wisdom of the gathered
participants.

A useful tool is Miro, which is an online whiteboard built for collaboration.
It has a number of features that make it easy to share a link to the
whiteboard and to manage even a large number of contributors who may wish to
view the whiteboard, follow a user around as they discuss or modify the board.
Anyone can add items to the board or cleanup diagrams as a discussion
proceeds. The Miro whiteboard offers infinite space so groups can break up and
even work on their own ideas an a separate area of the board. Current
alternatives to Miro include Mural and Google’s Jamboard.

###  4.2 Q and A Management

In traditional in-person meetings, managing questions from the audience can be
difficult and often favor the boldest. For virtual or hybrid meetings this
issue is can be harder unless additional tools are brought in, given the need
for participants to unmute, and the larger audience sizes. Several tools offer
the chance to add more sophistication to the process of submitting and
selecting questions. In particular, Slido, which is a tool that allows
participants to pose questions. The whole audience can review and up-vote
questions that interest them. Using this tool makes is easy for people to
submit questions even if they are uncomfortable speaking to the assembled
group and insures that precious conference time is used for questions that are
of general interest.

Live polling is a way to increase participant engagement and expose additional
background on the audience as a whole. Results are often presented in
interactive ways that evolve in real time as results come in, often taking the
form of word clouds or bar and pie charts. While Zoom offers this built-in,
tools like Slido and Mentimeter have a wider variety in question types,
outcome displays, and interactivity.

Although particularly pertinent for online and hybrid events, there is nothing
to stop these tools being used at fully in-person conferences as well.

###  4.3 Increased Social Interactions

As has been described in this report, a common challenge for virtual events
over the last year has been giving participants the same opportunity to meet
new people and reconnect with old acquaintances. Several tools exist which can
help in this regard with many being trialed in the different events described
in Section 3. One such approach is to give participants a common map which
can, for example, represent the conference “venue”. By moving around this map
participants can interact with each other based on their proximity. In
particular, gather.town uses a 2D map reminiscent of old 8-bit computer games
while Mozilla Hubs offers a 3D world that can be used with virtual reality
headsets. Both of these have been used successfully for poster sessions and
other more interactive pieces of typical conferences. An alternative approach
is to offer participants small rooms which allow for a handful of people to
join at a time. Platforms like Airmeet and Remo allow audience members to
choose a table to sit down at and begin discussions, similar to finding a
table in a lunch canteen or classroom. An alternative to this is guided
networking, where participants are less directly involved in choosing who they
meet, such as offered by RemotelyGreen.

##  5 Diversity, Inclusion and Accessibility Considerations

The shift to fully virtual meetings, conferences, and workshops during the
pandemic, and subsequent discussions on the likelihood of being able to
maintain such a strong virtual component in the future has highlighted many
areas of inequality in our current model of in-person meetings. In this
section, we highlight the benefits as well as potential pitfalls in terms of
diversity, inclusion, and accessibility that virtual meetings have over in-
person ones. Given the lessons learned over the past year it is imperative
that we continue to take these considerations into account post-pandemic, and
ensure we do not simply return to what we were doing before.

One of the largest impacts of online meetings is in the reduced travel costs
for attendees. This directly allows for participation by a larger and more
diverse group of people around the world, and especially benefits people in
less well-funded groups, newly-established research groups, and those located
far away from the meeting location. This also has a strong benefit for early
career researchers which often have challenging economic circumstances, given
that many institutions require one to cover all the bills in advance and only
reimburse after the meeting, sometimes many weeks later. Finally, many groups
have a funding limit of one in-person event per year, but researchers would
participate in many more events remotely. This is again especially useful for
early career researchers (or those who are new to the field).

On the other hand, network reliability in many parts of the world - perhaps
most especially those who would benefit most from the reduced travel costs -
may be a hindrance to being able to fully participate in and get the most out
of the meeting. In such a situation, a balance between live and pre-recorded
talks may be helpful, as well as having the meeting recorded and available to
watch at a later time.

Travel itself also has challenges for many people who may benefit from online
meetings. Travel time is a luxury that many, such as those with family or
other personal commitments, cannot afford. Some individuals may be unable to
travel due to disabilities, health limitations, or, especially in near-term
post-COVID times, those who have been unable to be vaccinated. Many U.S.
National Laboratories have travel caps or bans in place, limiting researchers’
travel to in-person meetings and certain countries. And while visa
restrictions are not generally a concern for most Northern American or
European passport holders, the costs, administration load, and time involved
in visa applications is significant for many researchers around the world.
Additionally, while all effort should be made to ensure the safety of
attendees at all times, meetings may be held in locations that are potentially
life-threatening to participants, given their race, sexual or gender
orientation, or religion.

A particular feature of online meetings that has accessibility benefits is the
relative ease of availability of captioning for presentations. While this does
have additional cost implications for the organisers, it should be considered
up-front as default rather than an after-thought. As automatic captioning in
discussions with many technical terms can be complicated and render
inaccuracies, it is highly preferable to accurately caption the content in as
close to real-time as possible through professional captioning services or
other similarly effective approaches. Captioning benefits not only those with
hearing-impairment, but also non-native speakers, and even new students and
researchers who may be unfamiliar with all the technical jargon in the field.

Organising social activities for fully online meetings over the past year has
required some innovation, but in the cases where it was done successfully has
contributed positively to the inclusivity of the event. The social component
is a very important part of in-person meetings, but can be alienating if one
does not already know a number of people attending. One of the things to think
about going into the future will be how to maintain this inclusivity and avoid
a fabricated "class divide" in meetings with both online and in-person
components.

A Code of Conduct must exist and be agreed upon by all parties in advance to
explicitly outline the rules for participation and the values associated with
the event. A carefully crafted Code of Conduct can go a long way towards
fostering diversity and inclusion by establishing acceptable behavior and
promoting inclusive thinking. However, having an agreed upon Code of Conduct
is not enough – clearly stated protocols for reporting and handling of
violations, including expectations for anonymity, data privacy and
repercussions for violators must be associated with any Code of Conduct policy
advertisement otherwise it risks being ineffectual. Virtual events bring
additional challenges regarding conduct issues as compared with in-person
events given the impersonal and sometimes quasi-anonymous nature of remote
interactions.

Finally, it is important for event organizers to be proactive about
establishing a diverse set of speakers, panelists and participants (and
organizers!). There is much more to say on this important topic, especially
regarding effective and ineffective approaches to improving diversity in
meetings, workshops and conferences (and our research itself!). However, most
points are not specific to the virtual event format which is the topic of this
report. Still, we would be remiss in not mentioning this critically important
aspect of event organization, so we conclude this section with a reminder on
this point.

##  6 Best Practices

With participants spread out around the globe, the most important goal for any
virtual meeting is to most efficiently and effectively use the time that is
available. Synchronous time \- where all attendees are available and able to
be attentive - is to be considered as precious, and careful and thorough
planning is necessary to make optimal use of this commodity. Drawing from the
experience gained from the conferences and workshops described in Section 3,
we list some non-exhaustive suggestions for virtual meeting organizers and
attendees to help with smooth running, and to make the meeting as productive
as possible.

###  6.1 Timetable Planning

  * •

The most important initial step is to consider the aim and goals of the
meeting or conference, as this will dictate the way the timetable is planned.

  * •

For large conferences that aim to show latest results, organizers should
carefully consider the fraction of synchronous time dedicated to plenary talks
(whether live or pre-recorded), compared to the time spent on questions and
discussions, and ensure these are in line with the conference goals.

  * •

For workshops where the goal is to have discussions or brainstorming of
solutions, the majority of synchronous time should be dedicated to these
activities. Any necessary material should be distributed and reviewed
beforehand, though highlights or focal points could be summarised at the start
of each session.

  * •

A well planned agenda that includes links to any tools that may be used during
the meeting will go a long way in ensuring smooth running and optimal use of
time.

  * •

Realistic planning of the timetable is important; meeting chairs should
contact speakers in advance to agree on the amount of time allocated for
presentations, questions, or any other activities.

###  6.2 Before the meeting

  * •

For large conferences, dress rehearsals for the local organizing committee in
the weeks leading up to the conference can give the organizers experience
running the various tools used in the conference, and in working together as a
team.

  * •

Written job descriptions should be created for each task needed to run the
conference and assigned to individuals. These may include hosting the webinar,
speaker assistance, talk recording and posting, session chairing, Q$\&$A
handling in the webinar, poster assistance, chat program monitoring, virtual
reality platform assistance, email list monitoring, social media content
posting, and webinar technical support.

  * •

It is helpful to try to anticipate possible problems in advance. A written
trouble-shooting guide should be created that includes a list of possible
things that could go wrong during the conference, how one should respond, and
the person responsible for handling the problem.

  * •

A Code of Conduct should be a non-negotiable aspect of every conference,
workshop, or meeting, to set expectations for behavior during the event. Where
registration is required, participants should not be able to register for the
conference or receive connection details without agreeing to the Code of
Conduct.

  * •

Clearly stated protocols for the reporting and handling of conduct violations
should be associated with the Code of Conduct itself. An online reporting
system should be made available during the conference. Reports of violations
of the Code of Conduct must be able to be made anonymously. Organizers should
consider assigning someone outside of the organizing committee to serve as an
at-the-ready ombudsperson to investigate complaints and strive for a
satisfactory resolution.

  * •

Conference organizers should ask all speakers for their preferred pronouns
(provided voluntarily, of course), and to provide audio recordings of each of
them pronouncing their own names. The organizers should provide the means (a
specific service or more likely instructions for a common computing platforms)
for speakers to easily produce these audio recordings. These audio recordings
should be made available to session chairs to ensure correct pronunciation of
presenters’ names. Presenter pronouns should always be respected.

  * •

Conference organizers should consider asking participants to provide, on a
voluntary basis, information beyond the usual registration content to
facilitate interactions, such as research interests. For example, I might be
interested in knowing if there are other participants interested in machine
learning approaches to neutrino event reconstruction. A searchable database of
keywords from participants research interests could prove useful to increase
interactions. Something similar could be achieved by allowing participants to
create topical channels on the event’s Slack workspace.

  * •

Detailed instructions should be provided to speakers including a checklist and
tips for giving a talk over the conference platform of choice. As an example,
see the speaker tips provided by Neutrino2020 [35].

  * •

Rehearsal sessions for plenary talk presenters should be arranged during the
weeks leading up to the conference, so that speakers can practice sharing
their slides and test their audio and video. Practice sessions could also be
set up during the first half hour before the conference on each day of the
event, in case speakers want to do a last-minute check of their setup.

  * •

Conference organizers can set up multiple ways for conference participants to
communicate with the organizers during the conference including over email,
collaboration workspaces such as Slack, and in a virtual reality platform (if
available). Be sure to allocate someone responsible for overseeing each.

  * •

A Frequently Asked Questions (FAQ) page (see for example [36]) should be
posted online for conference participants and updated daily.

###  6.3 During the Meeting

  * •

Staying on time is possibly the biggest challenge for any meeting or
conference - virtual or not. Chairs should have the authority to stop a talk,
for example by muting the speaker, if they run overtime. They should also be
able to show the speaker how much time they have remaining at any time.

  * •

Have a back channel, instant communication mechanism between key organizers
during the meeting to troubleshoot on the fly.

  * •

At the end of each day during the conference, a regrouping session should be
held for the local organizing committee to discuss and solve any issues that
arose during the conference that day.

###  6.4 Discussion sessions

  * •

Chairs should communicate their preferred method for taking questions, whether
by “raising one’s hand”, or by comments in the chat panel or a dedicated
workspace. Each of these should, of course, be monitored.

  * •

Dedicated workspaces are a useful tool for extending discussions after the
allocated time is up, or for additional questions there may not have been time
for.

###  6.5 For Presenters

  * •

Speakers, whether live or pre-recorded, must do their best to stick to their
allocated time. To run overtime in your presentation shows a lack of
preparation at best and can be construed as disrespect for everyone else
involved.

  * •

Speakers should make sure they attend any rehearsal sessions held for them to
familiarise themselves with the conference platform.

###  6.6 Socialising

  * •

Section 4 lists a number of tools available to improve the social aspect of
fully virtual meetings. While this is no substitute for in-person
interactions, use of one or more of these tools is encouraged.

  * •

In some of the smaller meetings, a nice touch by the organizing committee has
been sending out small “care packages” which may include things like local
snacks, branded merchandise (such as stickers), etc. Whilst this is an
additional financial and administration load, it is well received by
participants and helps to foster a feeling of community and can also give a
little bit of local flair.

##  7 Looking Forward

It has become exceedingly clear over the past year that online meetings have a
variety of strong advantages, and as we move to planning post-pandemic
meetings we should strive to maintain as many of these advantages as possible.
However, the in-person interactions, informal chats and networking have been
greatly missed over the past year, and this is perhaps the strongest motivator
for researchers to want to return to in-person meetings as soon as it is
allowed.

A possible approach to this is to avoid the binary choice between in-person
and online, and work toward creating a new, hybrid event format that pulls
from the benefits of each. Such an event would host a sufficient number of in-
person attendees to be able to cover most of the costs of the conference,
whilst also providing the ability to attend online at a reduced cost if the
attendee so wishes. The additional logistical requirements for successfully
including this virtual component are certainly not negligible, but have
already been demonstrated a number of times in events such as the large LHC
experiments’ collaboration weeks where remote participants are able to attend
talks and participate in post-talk discussions via a teleconferencing tool
such as Zoom.

In such meetings one has previously been required to be physically present if
one is giving a talk, a requirement that will need to be relaxed to maintain a
more inclusive approach to the meeting, and to avoid a “class divide” that
favours those with privileged access to in-person events. Technological
challenges such as unreliable internet connection is of course a concern, but
one possibility to overcome this has already been seen in virtual conferences
over the past year where talks are pre-recorded and uploaded ahead of time.

On its own this version of hybrid event still lacks the social and networking
opportunities for the virtual participants, however, and a way to overcome
this could be to host local conference hubs at strategic locations around the
world, where the virtual attendees could get together for discussions amongst
themselves. The hubs could be located at a local university with potentially
low overhead costs, and could also help to solve some of the technological
challenges for more remote participants.

To maintain some level of equity between the meeting host location and the
hubs, it should be encouraged that a fraction of the speakers broadcast their
talks from the hubs themselves. For those with the means to travel, this could
be considered an outreach opportunity, or an chance for a researcher to return
to and meet with local researchers from their home country.

A large amount of work goes into organising any conference, and a hybrid
conference will certainly prove an even bigger challenge to conference
organisers than those that are either fully in-person or fully virtual.
Additional costs will be involved, to cover the technological and room
requirements necessary for a good online event, while the risk of lower in-
person attendance also has financial considerations, as many conference
budgets are developed based on a minimum number of (in-person) attendees. An
initial approach here could be to charge the usual fee for in-person
attendees, and a smaller fee (up to US$50 per person seems to be considered
reasonable based on discussions during this workshop) for the online
attendees, to cover the additional costs involved with taking the conference
online. Local hubs should be discouraged from charging additional fees for
attendance at the hub.

Ultimately, conference organisers want their attendees - especially those
attending in-person - to have the best experience possible. A particular
challenge for the “hybrid with hubs” approach is with time zones, trying to
find time slots for the talks and discussions that work for most attendees
worldwide. Whilst this is impossible to solve completely, many conferences
over the past year have already done a good job at managing their schedule for
maximum participation across the globe, providing session recordings for
others to be able to catch up on at their convenience, or with pre-recorded
talks available shortly before the conference combined with a live discussion
session between the speakers during the event. In most of these cases, the
actual time spent in live talks has been reduced to half a day instead of a
full day, to account for the variation in time zones. This should be
considered a positive development, with the other half of the day available
for discussions, poster sessions, local outreach activities, or local tours.
These supplementary events could be organised in the local hubs as well as the
main conference venue, providing part of the social aspect of the conference
that virtual attendees would otherwise be missing. Finally, while the
conference organisers should be encouraged to make the timetable as accessible
as possible for all, in the “hybrid with hubs” approach perhaps a good
incentive to attend in-person would be to be able to attend the live talks in
the most convenient timezone.

For workshops and conferences that are organized through funded projects or
projects yet to be proposed, creative thinking about how to make better use of
funds nominally slated for participant costs and project personnel travel to
create and curate workshop content would be prudent in the post-pandemic "new
normal". For example, these funds could be used to improve accessibility to
event content through professional captioning or to pay for services that can
produce high-quality training materials to provide more in-depth understanding
and amplify the impact after completion of the event.

It would be really encouraging if within the next two years one or more major
conferences takes a lead on trialling the “hybrid with hubs” approach and
implements many of the suggestions outlined in this report.

Given what our community has experienced with virtual meetings over the last
year due to this terrible pandemic, it is hard to imagine the HEP community
returning to a pre-pandemic approach of "fully" in-person meetings. Through
this ongoing journey, we have learned the value of a substantial virtual
component to meetings organization as well as many associated challenges. This
is a genie that will not easily go back into the bottle. New approaches such
as the "hybrid with hubs" approach explored in our workshop and described in
this report as well as technical innovations that continue to blur the lines
of experience between virtual presence and physical presence will shape the
future of meetings in HEP and beyond.

## Acknowledgments

We thank the attendees for their active participation in the workshop to
discuss these issues and suggest paths forward. This workshop was partially
supported through the U.S. National Science Foundation (NSF) under Cooperative
Agreement OAC-1836650 (IRIS-HEP).

## Disclaimer

Certain commercial software platforms or services are identified in this paper
in order to specify the conference formats adequately. This does not imply a
recommendation or endorsement by IRIS-HEP, the individual conference
organizers or sponsors, the National Institute of Standards and Technology or
the National Science Foundation that the software platforms or services
identified are necessarily the best available for the purpose.

## Glossary

Ai-Media

    An AI/ML-based real-time captioning service. https://www.ai-media.tv/
Airmeet

    An online meeting platform. https://www.airmeet.com/
BinderHub

    A cloud service to share reproducible interactive computing environments from code repositories. https://binderhub.readthedocs.io/
Blackboard Collaborate

    Virtual classroom tool. https://www.blackboard.com/teaching-learning/collaboration-web-conferencing/blackboard-collaborate
Crowdcast

    Event hosting platform. https://www.crowdcast.io/
Discord

    A VoIP, instant messaging, and digital distribution platform. https://discord.com/
Facebook

    A social media and social networking service. https://www.facebook.com/
gather.town

    An avatar-based meeting platform. https://gather.town/
GitHub

    Version control and software development host. https://github.com/
Google Doc

    Collaborative document editing service. https://docs.google.com/
Google Meet

    A video communication service. https://meet.google.com/
Indico

    An event management platform developed by CERN. https://indico.cern.ch/
Jamboard

    A virtual whiteboard. https://jamboard.google.com/
Jitsi

    A collection of free and Open Source multiplatform VoIP, video conferencing, and instant messaging applications. https://jitsi.org/
Jupyter

    Interactive data science and scientific computing application. https://jupyter.org/
Mattermost

    An Open Source collaboration platform. https://mattermost.com/
Mentimeter

    Interactive presentations with live polls, quizzes, word clouds, and Q&As. https://www.mentimeter.com
Mibo

    An avatar-based meeting platform. https://getmibo.com/
Microsoft Teams

    A business communications platform featuring workspace chat and videoconferencing, file storage, and application integration. https://teams.microsoft.com/
Miro

    An online collaborative whiteboard platform. https://miro.com/about
Mozilla Hubs

    An avatar-based meeting platform. https://hubs.mozilla.com
Mural

    A digital workspace for visual collaboration. https://mural.co
Otter.ai

    An AI/ML-based real-time speech to text transcription service. https://otter.ai/
Remo

    An interactive virtual event platform. https://remo.co
RemotelyGreen

    AI driven online networking events. https://remotely.green
Skype

    A video and voice calling platform. https://www.skype.com/
Slack

    A virtual workspace, featuring chat, teams, document sharing, and app integration. https://slack.com/
Slido

    An interactive app for hybrid meetings. https://www.sli.do/
Twitter

    A micro-blogging and social networking service. https://www.twitter.com/
Vidyo

    A real-time video communication platform. https://www.vidyo.com/
Vimeo

    A video sharing platform. https://vimeo.com/
Webcast

    A CERN-hosted webcast server. https://webcast.web.cern.ch/
Whova

    Event management software. https://whova.com/
Wonder

    A virtual meeting space. https://www.wonder.me/
YouTube

    A video sharing platform. https://www.youtube.com/
Zenodo

    A general-purpose open-access repository for deposition of research papers, data sets, research software, reports, and any other research related digital artifacts. https://zenodo.org/
Zoom

    A video conferencing platform. https://zoom.us/

## References

  * [1] Institute for Research and Innovation in Software for High-Energy Physics. https://iris-hep.org.
  * [2] The HEP Software Foundation et al., “A roadmap for HEP software and computing R&D for the 2020s”,  Computing and Software for Big Science 3 (Mar, 2019) 7, 10.1007/s41781-018-0018-8. https://doi.org/10.1007/s41781-018-0018-8.
  * [3] P. Elmer, M. Neubauer, and M. D. Sokoloff, “Strategic Plan for a Scientific Software Innovation Institute (S2I2) for High Energy Physics”, arXiv:1712.06592.
  * [4] HSF WLCG Virtual Workshop on New Architectures, Portability and Sustainability. https://indico.cern.ch/event/908146.
  * [5] HSF WLCG Virtual Workshop. https://indico.cern.ch/event/941278.
  * [6] HSF-WLCG Workshop Organisers, “Feedback from May 2020 HSF-WLCG Workshop”. https://indico.cern.ch/event/925974/.
  * [7] HSF Workshop Organisers, “Feedback from November 2020 HSF-WLCG Workshop”. https://indico.cern.ch/event/1000087/.
  * [8] 25th International Conference on Computing in High-Energy and Nuclear Physics. https://indico.cern.ch/event/948465.
  * [9] LHCP 2020. https://indico.cern.ch/event/856696/attachments/1964474/3531885/LHCP2020_stats.pdf.
  * [10] The LLVM Project. http://llvm.org.
  * [11] 2020 Virtual LLVM Developers’ Meeting. https://llvm.org/devmtg/2020-09.
  * [12] B Line Events. https://blineevents.com/.
  * [13] Snowmass 2021 (virtual) conference. https://snowmass21.org.
  * [14] Snowmas Community Planning Meeting. https://indico.fnal.gov/event/44870.
  * [15] OSG All-Hands Meeting 2020. https://opensciencegrid.org/all-hands/2020.
  * [16] OSG All-Hands Meeting 2021. https://opensciencegrid.org/all-hands/2021.
  * [17] OSG Virtual School Pilot 2020. https://opensciencegrid.org/virtual-school-pilot-2020.
  * [18] OSG Virtual School 2021. https://opensciencegrid.org/virtual-school-2021.
  * [19] HEP Software Foundation PyHEP Working Group. https://hepsoftwarefoundation.org/workinggroups/pyhep.html.
  * [20] PyHEP 2021 (virtual) workshop. https://indico.cern.ch/e/PyHEP2021.
  * [21] SciPy: Annual Scientific Python Conference Talks & Tutorials. https://www.youtube.com/c/enthought/playlists.
  * [22] “US-ATLAS Computing Bootcamp 2020 Indico page”. https://indico.cern.ch/event/933434/.  Accessed on 2021-06-11.
  * [23] “US-ATLAS Computing Bootcamp 2020 Website”. https://matthewfeickert.github.io/usatlas-computing-bootcamp-2020/.  Accessed on 2021-06-11.
  * [24] “Mozilla Foundation’s _*Privacy Not Included_ Guide: Discord”. https://foundation.mozilla.org/en/privacynotincluded/discord/.  Accessed on 2021-06-11.
  * [25] Neutrino 2020 (virtual) conference. http://nu2020.fnal.gov.
  * [26] Neutrino 2020 conference archive on Zenodo. https://zenodo.org/communities/neutrino2020-talks/?page=1&size=20.
  * [27] Neutrino 2020 plenary talks. https://indico.fnal.gov/event/43209/timetable/#20200622.detailed.
  * [28] Neutrino 2020 posters. https://nusoft.fnal.gov/nova/nu2020postersession/.
  * [29] Neutrino 2020 Virtual Reality. https://conferences.fnal.gov/nu2020/virtual-reality/.
  * [30] Neutrino 2020 Physics Slam. https://www.c2st.org/event/neutrino-2020-physics-slam/.
  * [31] Neutrino 2020 Virtual Reality on Twitter. https://twitter.com/search?q=%40nu2020_chicago%20%22VR%22&src=typed_query.
  * [32] "More than 3000 Scientists Gather Online for Neutrino 2020", Symmetry magazine article, July 13, 2020 . https://www.symmetrymagazine.org/article/more-than-3000-scientists-gather-online-for-neutrino-2020.
  * [33] "Neutrino 2020 Zooms Into Virtual Reality", Cern Courier article, July 23, 2020. https://cerncourier.com/a/neutrino-2020-zooms-into-virtual-reality/.
  * [34] Neutrino 2020 Code of Conduct. https://conferences.fnal.gov/nu2020/conduct/.
  * [35] Neutrino 2020 speaker tips. https://conferences.fnal.gov/nu2020/wp-content/uploads/2020/08/Nu2020-speaker-instructions.pdf.
  * [36] Neutrino 2020 Frequently Asked Questions (FAQ) page. https://conferences.fnal.gov/nu2020/faqs/.