#  IRIS-HEP Blueprint Workshop Summary
Learning from the Pandemic: the Future of Meetings in HEP and Beyond

Mark S. Neubauer  University of Illinois at Urbana-Champaign, Champaign, IL,
USA  Todd Adams  Florida State University, Tallahassee, FL, USA  Jennifer
Adelman-McCarthy  Fermi National Accelerator Laboratory, Batavia, IL, USA
Gabriele Benelli  Brown University, Providence, RI, USA  Tulika Bose
University of Wisconsin–Madison, Madison, WI, USA  David Britton  University
of Glasgow, Glasgow, UK  Pat Burchat  Stanford University, Stanford, CA, USA
Joel Butler  Fermi National Accelerator Laboratory, Batavia, IL, USA  Timothy
A. Cartwright  University of Wisconsin–Madison, Madison, WI, USA  Tomáš
Davídek  Charles University, Prague, Czech Republic  Jacques Dumarchez  LPNHE
- Sorbonne Université - Paris, France  Peter Elmer  Princeton University,
Princeton, NJ, USA  Matthew Feickert  University of Illinois at Urbana-
Champaign, Champaign, IL, USA  Ben Galewsky  University of Illinois at Urbana-
Champaign, Champaign, IL, USA  Mandeep Gill  Stanford University, Stanford,
CA, USA  Maciej Gladki  European Organization for Nuclear Research (CERN),
Meyrin, Switzerland  Aman Goel  University of Illinois at Urbana-Champaign,
Champaign, IL, USA  Jonathan E. Guyer  National Institute of Standards and
Technology, Gaithersburg, MD, USA  Bo Jayatilaka  Fermi National Accelerator
Laboratory, Batavia, IL, USA  Brendan Kiburg  Fermi National Accelerator
Laboratory, Batavia, IL, USA  Benjamin Krikler  University of Bristol,
Bristol, UK  David Lange  Princeton University, Princeton, NJ, USA  Claire Lee
Fermi National Accelerator Laboratory, Batavia, IL, USA  Nick Manganelli
University of California at Riverside, Riverside, CA, USA  Giovanni Marchiori
APC Paris, France  Meenakshi Narain  Brown University, Providence, RI, USA
Ianna Osborne  Princeton University, Princeton, NJ, USA  Jim Pivarski
Princeton University, Princeton, NJ, USA  Harrison Prosper  Florida State
University, Tallahassee, FL, USA  Graeme A Stewart  European Organization for
Nuclear Research (CERN), Meyrin, Switzerland  Eduardo Rodrigues  Oliver Lodge
Laboratory, University of Liverpool, Liverpool, UK  Roberto Salerno  LLR -
Ecole Polytechnique, Palaiseau, France  Marguerite Tonjes  University of
Illinois at Chicago, Chicago, IL, USA  Jaroslav Trnka  University of
California at Davis, Davis, CA, USA  Vera Varanda  ARISF / LPNHE, Paris,
France  Vassil Vassilev  Princeton University, Princeton, NJ, USA  Gordon T.
Watts  University of Washington, Seattle, WA, USA  Sam Zeller  Fermi National
Accelerator Laboratory, Batavia, IL, USA  Yuanyuan Zhang  Fermi National
Accelerator Laboratory, Batavia, IL, USA

###### Abstract

The COVID-19 pandemic has by-and-large prevented in-person meetings since
March 2020. While the increasing deployment of effective vaccines around the
world is a very positive development, the timeline and pathway to “normality”
is uncertain and the “new normal” we will settle into is anyone’s guess.
Particle physics, like many other scientific fields, has more than a year of
experience in holding virtual meetings, workshops, and conferences. A great
deal of experimentation and innovation to explore how to execute these
meetings effectively has occurred. Therefore, it is an appropriate time to
take stock of what we as a community learned from running virtual meetings and
discuss possible strategies for the future. Continuing to develop effective
strategies for meetings with a virtual component is likely to be important for
reducing the carbon footprint of our research activities, while also enabling
greater diversity and inclusion for participation.

This report summarizes a virtual two-day workshop on Virtual Meetings held May
5–6, 2021 which brought together experts from both inside and outside of high-
energy physics to share their experiences and practices with organizing and
executing virtual workshops, and to develop possible strategies for future
meetings as we begin to emerge from the COVID-19 pandemic. This report
outlines some of the practices and tools that have worked well which we hope
will serve as a valuable resource for future virtual meeting organizers in all
scientific fields.

###### keywords:

high energy physics, virtual meetings

###### Contents

  1. 1 Introduction
  2. 2 Overview
    1. 2.1 IRIS-HEP and the Blueprint Process
    2. 2.2 Virtual Meetings Workshop
    3. 2.3 Conferences and Workshops Before COVID-19
    4. 2.4 COVID-19 Effects on Meetings
  3. 3 Experiences with Virtual Meetings
    1. 3.1 HSF/WLCG Virtual Workshops
    2. 3.2 LHCP 2020
    3. 3.3 ICHEP 2020
    4. 3.4 Moriond 2021
    5. 3.5 Connecting the Dots 2020
    6. 3.6 LLVM Developers Meeting
    7. 3.7 Snowmass Community Planning Meeting
    8. 3.8 OSG Virtual Meetings
    9. 3.9 PyHEP 2020 Experience and 2021 Plans
    10. 3.10 SciPy 2020 Experience and 2021 Plans
    11. 3.11 US ATLAS / Canada ATLAS Computing Bootcamp
    12. 3.12 Neutrino 2020
    13. 3.13 Common Themes and Key Findings
  4. 4 Tools and Techniques
    1. 4.1 Virtual Whiteboards
    2. 4.2 Q and A Management
    3. 4.3 Increased Social Interactions
  5. 5 Diversity, Inclusion and Accessibility Considerations
  6. 6 Best Practices
    1. 6.1 Timetable Planning
    2. 6.2 Before the meeting
    3. 6.3 During the Meeting
    4. 6.4 Discussion sessions
    5. 6.5 For Presenters
    6. 6.6 Socialising
  7. 7 Looking Forward

##  1 Introduction

The COVID-19 pandemic caused by the SARS-CoV-2 virus has had a devastating
effect on human health and well-being, and on the global economy. First
identified in Wuhan, China in December 2019, SARS-CoV-2 spread rapidly among
human population leading to the World Heath Organization (WHO) declaring
COVID-19 a global health emergency on January 31, 2020. The continued spread
around the world led to global travel restrictions in February 2020 and the
WHO officially declaring COVID-19 as a pandemic on March 11, 2020.

As the alarming scale of the global health crisis became apparent in March
2020, workshop organizers began to cancel or postpone planned in-person
events. Many organizers scrambled to re-purpose their workshops for a virtual
format where attendees participated remotely through videoconferencing
technologies such as Zoom. In some cases this re-purposing was done on short
order rather than cancelling the event. The Connecting-the-Dots Workshop held
in April 2020 is one such an example and is described in Section 3.5.

This report summarizes a two-day workshop on the topic of Virtual Meetings
held over the period of May 5 to 6, 2021. The workshop is part of the
"Blueprint" process of the NSF-funded Institute for Research and Innovation in
Software for High-Energy Physics (IRIS-HEP) [1].

The report is organized similarly to flow of the workshop. In Section 2, the
IRIS-HEP Blueprint process and this workshop is introduced followed by an
overview of pre-COVID-19 meeting organization and the ways that COVID-19
rapidly changed the _status quo_. Section 3 provides a summary of the talks
given during the workshop by organizers of virtual workshops since April 2020.
At the end of Section 3, common themes and key findings from these workshop
experiences are presented. In Section 4 we outline some of the practices and
tools that have worked well (or have not but seem like they should have) for
virtual workshops. In Section 5 we present important considerations for
diversity, inclusion and accessibility for virtual workshops. Best practices
for virtual workshops at each of the stages of preparation and execution are
presented in Section 6. The final section (§7) synthesizes what we have
learned about virtual workshops over the last year and presents some ideas for
the organization of future meetings in HEP and beyond, such as the "hybrid
with hubs" approach.

##  2 Overview

###  2.1 IRIS-HEP and the Blueprint Process

The goal of the IRIS-HEP is to address key computational and data science
challenges of the High-Luminosity Large Hadron Collider (HL-LHC) experiments
and other HEP experiments in the 2020s. IRIS-HEP resulted from a 2-year
community-wide effort involving 18 workshops and 8 position papers, most
notably a Community White Paper [2] and a Strategic Plan [3]. The institute is
an active center for software R&D, functions as an intellectual hub for the
larger community-wide software R&D efforts, and aims to transform the
operational services required to ensure the success of the HL-LHC scientific
program.

The IRIS-HEP Blueprint activity is designed to inform development and
evolution of the IRIS-HEP strategic vision and build (or strengthen)
partnerships among communities driven by innovation in software and computing.
The blueprint process includes a series of workshops that bring together IRIS-
HEP team members, key stakeholders, and domain experts from disciplines of
importance to the Institute’s mission. This Blueprint meeting on the topic of
_Virtual Meetings_ is one of a series of workshops that have also included

  * •

Analysis Systems R&D on Scalable Platforms (2019)

  * •

Fast Machine Learning and Inference (2019 & 2020)

  * •

A Coordinated Ecosystem for HL-LHC Computing R&D (2019)

  * •

Software Training (2020)

  * •

Sustainable Software in HEP (2020)

  * •

Future Analysis Systems & Facilities (2020)

  * •

Portable Inference (2020)

The Blueprint workshop discussions are captured and inform key outcomes which
are summarized in a short report made publicly available, such as this report.

###  2.2 Virtual Meetings Workshop

While the increasing deployment of effective vaccines around the world is a
very positive development, the timeline and pathway to “normality” is
uncertain and the “new normal” we will settle into is anyone’s guess. Particle
physics, like many other scientific fields, has more than a year of experience
in holding virtual meetings, workshops, and conferences. A great deal of
experimentation and innovation to explore how to execute these meetings
effectively has occurred. The IRIS-HEP team has substantial experience with
(co-)organizing and sponsoring workshops, conferences, and meetings over the
years through the process of establishing the institute and its role as an
intellectual hub for software R&D.

For these reasons, it was viewed as an opportune time to take stock of what we
as a community have learned from running virtual meetings and discuss
effective strategies for the future through this virtual meetings workshop.
Continuing to develop effective strategies for meetings with a virtual
component is likely to be important for reducing the carbon footprint of our
research activities, while also enabling greater diversity and inclusion for
participation.

The workshop on Virtual Meetings was held May 5 to 6, 2021. The aim for the
workshop was to bring together experts from both inside and outside of
particle physics to share their experiences and practices with organizing and
executing virtual workshops, and to develop possible strategies for future
meetings as we begin to emerge from the challenging conditions of the COVID-19
pandemic.

Attendees participated remotely using a variety of videoconferencing and
collaborative tools. Eighty-nine people registered for the workshop, though as
a virtual workshop, attendees came and went at various times. We estimate that
there were about 70 to 80 participants in the workshop at any one time.

The timeline of the workshop presentations and activities are summarized in
Table 1.

Day 1 (via Zoom)
---
08:30 | Welcome, Blueprint Activity and Workshop Overview | M. Neubauer
08:50 | A Virtual Hitchhiker’s Guide to Virtual Conferencing | B. Krikler
09:20 | HSF/WLCG Virtual Workshop Experience | G. Stewart
09:40 | LHCP 2020 | G. Marchiori, R. Salerno
10:00 | ICHEP 2020 | T. Davidek
10:20 | Moriond 2021 | V. Varanda
10:40 | Connecting the Dots 2020 | D. Lange
11:00 | LLVM Developers Meeting | V. Vassilev
11:35 | Snowmass Community Planning Meeting | B. Jayatilaka
11:55 | OSG Virtual Meetings | T. Cartwright
12:15 | PyHEP 2020 Experience and 2021 Plans | E. Rodrigues
12:35 | SciPy 2020 Experience and 2021 Plans | J. Guyer
12:55 | US ATLAS / Canada ATLAS Computing Bootcamp | M. Feickert
13:15 | Neutrino 2020 | S. Zeller
13:35 | Discussion Session |
Day 2 (via gather.town)
08:30 | Day 2 Topics and Goals | M. Neubauer
08:50 | Summary of Community Input from DPF Townhall | M. Narain
09:20 | Tools and Techniques for Virtual Workshops Session | B. Galewsky
11:00 | Taking Stock of Experiences and Discussion on Future Events |
12:00 | Summary Report Discussion and Writing |
Table 1: Workshop Agenda (times are CDT) Presentations are available from the
workshop website*.
*https://indico.cern.ch/event/1026363/timetable

The first day of the workshop was held using the Zoom videoconferencing and
chat platform. It was primarily focused on experiences with virtual workshops
via talks by its organizational representatives. The workshops were chosen to
span a wide range of scales and purpose, from training events with tens of
attendees such as the US ATLAS / Canada ATLAS Computing Bootcamp (Section
3.11) to international conferences with thousands of attendees such as the
International Conference on High Energy Physics (Section 3.3) and Neutrino
2020 (Section 3.12). At the end of the first day, a discussion session was
used to culled out common themes and key finds from the experiences talks.

The second day was held using the gather.town platform. gather.town is
videoconferencing software like Zoom, but with the added component of seeing
the virtual “room” that you are occupying with other participants represented
as avatars, with the ability to move around and interact with others. To
demonstrate the gather.town functionality for the workshop, the Fermilab
Wilson Hall floorplan was imported and various areas were defined including a
poster area and a private space where participants could arrange for one-on-
one or group discussions. The presentations and discussion sessions were held
virtually in the Wilson Hall 1 West (WH1W) conference room, as shown in Figure
1.

Figure 1: Day 2 of the workshop was held in a virtual Wilson Hall 1 West space
within the gather.town application.

The Day 2 focus was on community input from an American Physics Society
Division of Particles and Fields Townhall event, demonstration of tools &
techniques for virtual meetings and forward-looking discussion of possible
meeting strategies as we begin to emerge from the challenging conditions of
the COVID-19 pandemic. This discussion was facilitated by the use of a
collaborative whiteboard tool called a "Miro board" where participants
answered questions in the form of five "exercises":

  1. 1.

Conference Essence: What are elements that are core to the conference? What
are important elements that also happen at a conference?

  2. 2.

Online/In-Person Format Comparison: What are the pros and cons of an online
meeting format? Of an in-person format? Of a hybrid format?

  3. 3.

Attendance Composition: Who is most likely to attend in-person conferences?

  4. 4.

Attractors/Repellers: What might attract someone to move from one mode of
conference to another? What might dissuade them?

  5. 5.

Tools and Techniques: Which are important to achieving the goals of a given
conference? Which might attract people to our attend our conference?

The input provided via this collaborative whiteboard along with the notes and
Zoom recording of the discussions are represented in this summary in the
appropriate sections. An example of the Miro board in action from the workshop
is shown Figure 2.

Figure 2: Example use of the Miro collaborative whiteboard to work through a
set of questions during Day 2 of the workshop. A set of virtual sticky-notes
were annotated and placed in the appropriate square.

###  2.3 Conferences and Workshops Before COVID-19

Most readers of this document will have experience with attending in-person
conferences, workshops, and/or meetings as part of their research or other
professional pursuits. Therefore, we do not attempt to discuss all elements of
in-person meetings in this report. Rather, we use this section to touch on
some of the key aspects from the virtual meetings workshop as a baseline for
the discussion that follows.

The status quo in HEP before COVID-19 was, especially for the larger workshop
and conferences, was to travel to the venue to either deliver a talk on one’s
research of research on behalf of a collaboration or attend in-person to hear
talks and participate in discussions. Of course, videoconferencing has been
around for a long time and remote participation has been component to events,
but these were more likely than not due to specific constraints by individuals
rather than prominent element generally embraced by the organizers.

Travel to and from in-person events most often involves a substantial
commitment of time and funds (most likely from research grants for which
principle investigators typically budget as part of their research program).
Depending on distance from the researcher’s home location, its common to have
one or two days on each side of the event consumed with traveling through a
network of airports, train stations, etc. This level of time commitment away
from home can be challenging for those with children, take care of others, or
have other personal constraints. The transportation itself also involves a
carbon load on the environment which can be substantial in the aggregate when
one considers the scale of global research travel and the distances involved
to participate in-person at international conferences and collaboration
events.

Outside of the presentations themselves, in-person meetings provide
opportunities for chance (or planned) encounters with other participants (e.g.
during coffee breaks for conference dinners) to discuss and develop new
research ideas or directions, foster new collaborations and engage in deep
technical discussions about talks that were presented. These encounters and
informal interactions can be career-changing for young scientists, both in
terms of networking and access to leading scientists in their respective
fields. These types of interactions are difficult to replicate in a remote
format given the current state of technology for virtual presence.

Participants in the virtual meetings workshop noted other benefits of the in-
person format, including an opportunity to focus on a specific topic for a
period of time that is often difficult given responsibilities and distractions
in their home/work environment, meet collaborators in-person (sometimes for
the first time!), and obtain an expanded viewpoint on science, society and
cultures that comes with traveling to somewhere new.

###  2.4 COVID-19 Effects on Meetings

As mentioned in Section 1, the move from largely in-person meetings to almost
exclusively virtual events due to the COVID-19 pandemic was abrupt and
unprecedented. Of course, it was not only the organizers and participants of
conferences and workshops that had to pivot – schools and universities had to
move quickly to remote or "hybrid" learning environments and businesses and
people alike had to adapt to the challenges of COVID-related lock-downs and
restrictions. Such a rapid transition to a virtual presence in personal and
professional lives at scale was only possible due to the general availability
of personal computers (including laptops, tablets and smart phones) coupled to
high-speed networks and software tools like Zoom.

One clear effect that COVID-19 has had on our community is to expose
inefficiencies in the _status quo_ model for conferences, workshops and
meetings and bring to the surface concerns around equity, access and
environmental aspects of this model. A constructive rethinking of this model
informed by experiences since April 2020 and emerging technologies is the
first step to addressing some of the pre-existing deficiencies toward a more
inclusive, environmentally-conscious and effective approach to future
meetings.

##  3 Experiences with Virtual Meetings

This section summarizes the experiences drawn from a dozen virtual scientific
workshops with a variety of sizes and purposes since April 2020. Not
surprisingly, given that this report is a summary of a Blueprint workshop for
a HEP software institute, the virtual events presented here are either
software-focused, HEP-focused or both.

###  3.1 HSF/WLCG Virtual Workshops

Since the HSF Community White Paper [2] the HEP Software Foundation (HSF) and
the Worldwide LHC Computing Grid (WLCG) have held joint workshops to advance
R&D in software and computing for HEP.

A workshop had been planned for May 2020, in Lund, Sweden, but with the
degenerating COVID-19 situation in March it became clear that an in-person
workshop was impossible to hold and we started to plan a virtual event [4]
that could replace it. This was one of the first reasonably large meetings in
the community that moved to a virtual format, but the lessons learned here and
at the follow up workshop in November 2020 [5] helped refine and inform the
community on how to hold successful online events [6, 7]. This section is a
summary of the most important lessons learned from these workshops.

####  3.1.1 Focused Topic and Scheduling

It was immediately obvious to the organisers that simply moving the planned
five full-day face-to-face meeting to a video conference setup would not work.
Instead, the May workshop [5] was shortened and refocused to tackle a specific
topic: New Architectures, Portability, and Sustainability.

Narrowing the scope meant also that scheduling the event would be easier.
Given the relative centres of gravity in high-energy physics of Europe and the
Americas, the ‘golden hour’ in which to schedule meetings is later afternoon
in Europe and morning in the Americas. e.g., 16:00 CERN time is a comfortable
9:00 at Fermilab and 7:00 in the Pacific timezone, which is early, but
possible. In May we went for fairly short sessions of two hours only, 16:00 to
18:00 CERN. Finshing at 18:00 avoided being too disruptive for European
participants. However, it does mean that only a limited amount of time is
available – again emphasising the need to focus on key topics during each
session.

Unfortunately these times make it really difficult for Asia-Pacific colleagues
to participate. In larger events, a mixture of sessions that start in the
European mornings (thus afternoon in Asia) work well, e.g., starting at 9:00
CERN time, as was chosen for the recent conference [8]. Unfortunately, it is
very difficult to have an event during working hours for North America and
Asia: 17:00 in New York is still 5:00 in Tokyo as the Pacific Ocean is really
large.

Bear in mind that concentrating on online events is fatiguing and many
participants will be experiencing multiple distractions (e.g. emails arriving,
phone calls, children needing attention, etc.). It is imperative to schedule
regular breaks to allow people to have a mental rest, stretch out, make a cup
of tea, etc. Practical experience suggests that one break every two hours is a
minimum. If one insists on running an activity during the break, light-hearted
polls can keep people amused.

####  3.1.2 Material Upload and Videos

Given the difficulties of scheduling a convenient time for presentations that
can suit all participants, one compensating factor is to make sure that
material is uploaded in advance. For the first workshop we attempted to get
material a week in advance. However, this was hard for speakers to manage and,
in fact, feedback from participants was that one day in advance is sufficient,
Figure 3.

Figure 3: Participant survey responses from the May HSF-WLCG workshop on their
preference for when presentations should be uploaded.

The usefulness of this advanced availability of the material was borne out by
the fact that 2/3 of workshop attendees reported viewing the material before
the sessions, as shown in Figure 4.

Figure 4: Survey response on if participants thought that material should be
available in advance and if they viewed it.

If the session is really at an inconvenient time for some people, the best
solution is to record the video of the talk and make it available _post-
facto_. If Zoom is being used for the presentations, then its built-in
recording facilities are ideal for this purpose. Depending on the organisers
familiarity with video editing tools, session videos can be posted unedited,
or split by talk or even given fancy introductions and epilogues. Splitting by
talk is relatively easily managed using open source tools like ffmpeg.

Be aware, especially at CERN and in Europe, that data privacy laws apply. It
is strongly advised, in fact even mandatory, that you gather specific consent
from all participants to not only record the meeting, but also to make it
available publicly on platforms such as the CERN Document Server or YouTube.

####  3.1.3 Hosting the Event

At a minimum, a video conferencing service will be needed to host the event.
Almost every university and laboratory now has a solution for this. More often
than not this is the Zoom platform, which at the moment is a _de facto_
standard in HEP. This is for good reason: the clients are stable, the
infrastructure scales well and features such as recording work well. It also
has the advantage of being extremely familiar to most participants, thus
technical hurdles are minimised. Back in May 2020, quite a number of
participants had problems screen sharing, "hand" raising, muting/un-muting,
but by now most people have worked these out. Still, it is a wise idea,
especially for larger events to:

  * •

Run some practice sessions in advance for speakers (and session chairs) to
test their setup

  * •

Schedule one or two minutes between speakers to change sharing

  * •

Have a copy of the slides downloaded locally by the chairs, in case the
speaker has a slide sharing issue

During presentations, the usual recommendation is that the speaker shares
slides and that they have their camera enabled to give a slightly more
intimate feel. Everyone else should be muted and have their camera off, unless
they are speaking. Having a participants’ guide for the event makes it clear
to people how they should contribute to the meeting discussion. E.g., in
larger meetings using Zoom’s _raise hand_ feature is a very good discipline to
adopt.

In addition to these preparatory steps for speakers the hosts of the event
need to be well prepared. At the very least there should be a chair and a co-
chair for each session. The co-chair can monitor the meeting chat (§3.1.4)
while the chair is managing the speaker and contributors.

The hosts of the meeting should have an out-of-band communication mechanism
that they can use to discuss issues as the meeting is ongoing – private chat
channels on instant messaging tools like Mattermost, Slack, or Skype work well
for this. The co-chair can warn if a speaker looks badly set to go over time,
for example, or perhaps the organisers feel there is an important question
that should be prioritised.

Good timekeeping in online meetings is very important. Like it or not,
participants will multiplex with other meetings and events and knowing that a
presentation will happen at the scheduled time is vital for people to join for
the sessions they desire. Family or other personal commitments are a reality
for many people and it is not acceptable to exclude people by throwing the
schedule out the window when the meeting starts. Make sure that when
organising the meeting there is sufficient discussion time scheduled and that
it is stressed to the speakers how much time they have for their presentation.
Do not be afraid to remind them during their talk if it looks like they will
overrun, or even insist they finish up if they exceed their time budget.

Finally, although "Zoom-bombing" (unwanted remote attendance by uninvited
individuals) is quite rare in our community, it can happen. Make sure that
Zoom rooms are password protected and only distribute the passwords to people
registered for the event (CERN’s Indico allows Zoom links to be nicely
protected, or email them to the list of registered participants). Knowing what
to do if a Zoom-bombing occurs is really important to minimise disruption if
there is an incident – Zoom has good tools for managing this now and even a
‘big red button’ that will allow the hosts to suspend all participant
activities in the case of a concerted attack. Know how you will proceed with
the meeting in case something happens: e.g., keep the waiting room active, the
(co-)chair shares slides, keep chat and rename disabled, don’t allow
participants to unmute except by invitation, etc.

All of this preparation will help the event to run smoothly.

####  3.1.4 Chat Tools

Having a channel by which discussion can happen on presentations is a very
useful thing for online meetings. It allows participants to ask questions in a
non-disruptive way during the talk (and some people even just feel more
comfortable asking a question in text chat, so it also improves inclusivity).
It also allows for discussions to continue after the time slot for the
presentation has finished; it can also allow people to ask questions in
advance (as we noted, slides should go up well before the meeting starts).

The chat channel of the video system itself is rather disfavoured for this.
Usually it is ephemeral and lacks many useful features, so save that only for
technical issues during the meeting (it can be the co-chair’s job to redirect
questions if they do get asked there).

A number of options are used in the community, each with pros and cons:

  * •

Publicly writable Google Docs work well at a small to medium scale – make sure
that a skeleton layout is in place in advance so that comments and questions
go to the correct place. A few of the drawbacks here are that contributors
need some discipline to identify themselves in the comment they make
(anonymous tapirs are cute, but not helpful) and that comments are easily
misplaced as people can write anywhere in the document at any time.

  * •

Slack and Mattermost are chat tools that impose some restrictions, in the
sense that everything is a serial stream. That discipline can be useful to
prevent chaos and more easily manage a discussion. However, it can also be
confusing if multiple discussions are happening at the same time interleaved.
This is mitigated if people use ‘Reply’ functionality that threads a
discussion; success depends on how familiar people are with these kind of
tools. If multiple sessions are running, use a different channel for each one
to cut down on cross-talk.

  * •

Discord is a popular platform (albeit less well known in HEP) that allows for
multiple chats, breakouts and even video. It is very popular in the gaming
community and seems to work particularly well for training and tutorial
events. This platform was used successfully in the US ATLAS / Canada ATLAS
Bootcamp described in Section 3.11.

For the HSF-WLCG workshops, the Google Doc solution was adopted and people
thought that it did help the discussion, Figure 5. Likewise, Mattermost, as
used at the vCHEP2021 conference was view positively by 54 % of participants,
Figure 6 (this was despite a definite tension between security and convenience
that caused some participants technical problems).

Figure 5: Participant survey responses towards asking questions in the Google
Doc live notes. Figure 6: Participant survey responses at vCHEP about the
Mattermost chat tool.

####  3.1.5 Virtual Social Platforms

One aspect of workshops and conferences that is much harder to replicate
online is the social and informal human contact that we have during coffee
breaks, lunches and social events. The opportunity to have less structured
discussions, to branch out to other topics or just to enjoy the company of
colleagues is not yet easily replicated through any virtual platform. There
are no lack of platforms here (gather.town, Wonder, Mibo, to name a few), but
it is clear that barriers to normal human interaction do remain. Perhaps in
the future this will change, with technical advances (e.g., virtual reality)
or with us becoming more accustomed to such platforms. For now, though not
without their merits, do not expect them to be a substitute for face-to-face
interactions.

####  3.1.6 Conclusions

The HSF and WLCG experience of running virtual events demonstrates that for
the presentation of scientific content a virtual event functions rather well.
The same is true of direct discussions, which can be supported in a variety of
ways through live notes or text chat as well as live chat. Running a
successful virtual event (just like an in-person one) requires careful
planning and a lot of work from the organisers, from scheduling the event at
the best times, running the event properly and then posting videos of the
event after the fact. Virtual social platforms can supplement the
presentations and discussion, but still fall far short of the in-person
experience.

One meta-consideration when organising events is that as the barrier to entry
for organising a virtual event is rather low, there was a tendency to have
more events during 2020 than we would have had if it had been a normal year.
Zoom-fatigue arose for more reasons that just what was happening in the HEP
universe, but it affected many of our colleagues in a real way during
difficult times. So we should endeavour to make any virtual events focused,
meaningful and useful and, as ever, be sympathetic and understanding to the
pressures that people have found themselves under during these extraordinary
times.

###  3.2 LHCP 2020

####  3.2.1 Origin of the LHCP 2020 online conference

The LHCP conference series started in 2013 after a successful fusion of two
international conferences, "Physics at Large Hadron Collider Conference" and
"Hadron Collider Physics Symposium". It consists of a series of yearly
conferences where the latest experimental and theoretical results on the Large
Hadron Collider (LHC) physics are presented. They include research areas such
as the Standard Model Physics and Beyond, the Higgs boson, Supersymmetry,
Heavy Quark Physics, and Heavy Ion Physics as well as recent progress in the
high luminosity upgrades of the LHC and future colliders developments. The
LHCP conference typically attracts between 300 and 400 participants every year
from all over the world, discussing together in parallel, plenary and poster
sessions spanning a full week.

The 8th episode of the series, LHCP 2020, was due to take place in Paris,
France, in the International Conference Centre of Sorbonne Université, the
week of May 25 to 30, 2020. At the beginning of March 2020 due to the outbreak
of COVID-19 it became clear that an in-person conference would not be
feasible, and it was decided by the conference Steering Committee (SC), in
consultation with the conference International Advisory Committee (IAC), to
postpone the Paris conference by one full year, to become LHCP 2021. However,
the LHC experimental collaborations strongly suggested that an online "LHCP-
like" event would be still held, in a similar period of the year and with a
similar format to the planned LHCP 2020 conference, so that physicists could
still join together – though virtually – to discuss the new results and keep
the community together. The LHCP 2020 organisers with the assistance of the
CERN IT department for technical details embarked in the project of setting up
such online event with only two months to go before the start of the
conference.

The LHCP 2020 online conference took place the week of May 25 to 30, 2020 as
originally planned for the in-person conference. It was the first online HEP
conference with a large (>1000) audience and as such the organisers had to
take decisions based on limited previous experience. The success of the LHCP
2020 online conference and the lessons learned from it were useful for the
setup of later HEP conferences.

####  3.2.2 LHCP 2020 online: the preparation

The organisation of LHCP 2020 online revolved around two main points: the
programme and the technical organisation.

The programme had already been setup for the in-person conference, and the
main issue here concerned the modification of the schedule in order to be
accessible to a community that would be scattered over a large number of
continents and timezones rather than being all sitting together in the same
site. Two alternative scenarios were considered:

  * •

keep the structure proposed for the in-person conference, with at most four
parallel sessions taking place at the same time, and similar duration of the
pauses (30′ breaks), but with fewer blocks every day (typically one parallel
and one plenary session with a pause in between), for about four hours/day but
spanning two full weeks (Monday to Friday);

  * •

increment the maximum number of concurrent parallel sessions, shorten the
breaks, and keep three sessions/day, for a total of about six hours/day from
Monday-Friday plus a shorter concluding 4-hour session on Tuesday.

It was not considered to shorten the programme due to the large amount of
material that the experimental collaborations and the theory community had
already prepared to release and discuss at the conference.

The second scenario (see Figure 7) was chosen, with sessions between 12:30 and
18:30 CEST (18:30 to 00:30 CST, 5:30 to 11:30 PDT). This would make a little
fraction of the session less accessible to our colleagues in the countries on
timezones far away from CEST, but would have the advantage of keeping the
event within the same week as initially planned and advertised, so that
participants would have less difficulties to find the time to interrupt their
teaching and research activities and join the virtual event.

Figure 7:  The final LHCP 2020 timeline (CEST time zone) and block structure.

Concerning the technical and practical organisation, the main decisions taken
were the following:

  * •

to keep the poster sessions, which would be held in virtual meeting rooms and
to keep the awards for the best poster;

  * •

to publish the conference proceedings as initially planned, giving to the
speakers and poster presenters the possibility to have a short citeable write-
up on their presentation after the conference;

  * •

to waive completely the fees for all participants making the conference
accessible to the widest possible option and avoiding the need to handle
processing small amounts of money from a large number of participants;

  * •

to extend the deadlines for registration as well as poster abstract submission
until a few days before the start of the conference to maximize the potential
audience.

####  3.2.3 Technical setup

When the online LHCP 2020 conference was announced, participants started
enthusiastically to register, at a pace of order of 50 new participants per
day, increasing with the approach of the starting day of the conference. It
soon became clear that the number of participants would reach several hundreds
and that it would require an online videoconferencing tool that would be able
to handle well such a large number of simultaneous connections and that would
be intuitive to use for the participants, the speakers, and the session
conveners. At the same time, when the world locked down and the meetings of
the large LHC collaborations moved online, the videoconferencing solution
adopted until then by CERN, Vidyo, started to show saturation issues with
several persons unable to join meetings. The recommendation from CERN IT was
therefore to move to Zoom, which proved to be extremely scalable and intuitive
to use. CERN established, in collaboration with Zoom, a pilot program that
provided professional licences of the tool to CERN users during an evaluation
period. Some Zoom features such as recording of the meetings on the cloud were
disabled for privacy concerns.

In order to control access to the online meetings and restrict it to
authorised people, to avoid "gatecrashing", and even to avoid that
participants by accident would unmute their microphone or start broadcasting
their video during the presentations and disrupt the normal flow of the
conference (not an unlikely event with more than 1000 participants), we
decided to choose the "Webinar"-style format for the plenary and parallel
sessions, rather than the regular Zoom meeting style. The webinar format would
give more control to the conveners ("Hosts" in Zoom’s terminology) on who
could (or could not) turn on their microphone, after having virtually raised
their hand; it would provide a practice session for speakers ("Panelists")
where they could join the meeting before its actual start, to test their
connection and the sharing of their screen, and to avoid the problem of not
being able to connect to the meeting in case they by accident tried to connect
only after 500 participants (the limit of the Zoom CERN pilot) had already
connected. On the negative side of this choice, the participants could not see
who the other persons attending the same event (apart from the speakers and
conveners) were, reducing the sense of community that is an important,
integral part of in-person conferences.

A few days before the beginning of the conference, when the deadline for the
registration was reached, all registrations were reviewed and those from
participants not from the HEP community nor with an institutional e-mail
address (for instance, an address provided by a University to a student) were
rejected (only a handful of registrations would not be accepted). The
confirmed registered participants then received by e-mail the connection
details and the passwords for the Zoom webinars.

The webinars for the plenary sessions were created by the conference
organisers. On the other hand, due to the impossibility for a single Zoom user
to create several meetings running in parallel in the same time slot, the
webinars for the parallel sessions were created by the conveners of the
session, who were instructed on how to setup and password-protect the meeting
by the conference organisers. To allow for off-line access to the conference
presentations and Q&A, the option to record the meetings (on the local PC of
the host) was enabled. The consent of each speaker to have their talk recorded
was collected through the Indico webpage of the conference beforehand (in case
of a speaker not accepting to be recorded, recording would have been paused).
The participants connecting to the Zoom webinar would have to accept by
clicking on the "Accept" button of a Zoom pop-up window or otherwise
disconnect.

For the "online" poster presentations, it was decided to have two sessions of
one hour each, in different times of the day to allow poster presenters from
each part of the world to find at least one suitable time slot for them to
join it. Poster sessions took place in Zoom regular meeting rooms, one for
each poster presenter. The speakers were asked to create their own Zoom
meeting rooms and upload the link to their meeting room on the conference
website, while passwords were sent separately to the participants by e-mail.
For presenters without a Zoom license, the CERN IT department created a
lightweight CERN account for the duration of the conference, giving them
access to the CERN Zoom pilot program. Poster presenters were also asked to
pre-record a 3 min short presentation of their poster and upload it, together
with the poster itself and the connection details, to the conference agenda
before the poster sessions.

Finally, since the number of registered participants in the end (1300)
exceeded the maximum capacity of the CERN Zoom license (1000 participants
connected at the same time), for the plenary sessions it was put in place by
the CERN IT department a streaming of the Zoom webinars to the CERN Webcast.
In order to allow participants connected to the plenary sessions through
Webcast to ask their question, a plenary-qa channel was created on a dedicated
Discord online server created for the LHCP 2020 conference.

####  3.2.4 Accessibility and inclusion

An effort to make the conference as widely accessible as possible was done by
the organisers by compressing the schedule into an agenda that would allow
participants from all over the world to connect to most of the sessions at a
reasonable time of the day.

Arrangements had also been made by the local organisation, before the
conference was moved online, to support the stay of young participants from
developing countries with the creation of a dedicated budget for waiving fees
and booking of low-cost accommodation solution in Paris, as well as - for
instance - to provide dedicated quiet rooms for mothers needing to breastfeed.

When the conference was announced in its online version, four weeks before the
starting date of the conference, a request was brought forward by one
participant to provided full real-time captioning for the whole conference
(plenary and parallel sessions). The organisation looked around for potential
solutions that would suit the user requesting them as well as fit in the very
limited budget that the online conference had, from the CERN and EPJC
sponsorships (that were also to cover for other expenses such as the poster
awards, the publication of the conference proceedings, and communication
activities). As the option to caption only the Q&A sessions was deemed not
acceptable by the participant, it was proposed to caption in real time only a
few, selected sessions chosen by the participant. For the other sessions, we
decided to propose off-line captioning of all the talks, as obtained by
running the recordings through an automatic AI-powered tool, Otter.ai.
However, the agreement on this option came only a few working days before the
starting of the contract; meanwhile, the company that had already been chosen
for the captioning was not available any longer, and an alternative one had to
be found quickly (including finding extra budget to cover for the price
increase compared to the other one). This introduced some unfortunate delays,
that we regret, in the setting up of the contract and we were only able to
arrange live captioning for the sessions attended by the participant during
the second half of the conference.

For the future conferences we recommend, based on this experience, that users
with special requests like this one get in contact with the organisers well in
advance with respect to the start of the conference, possibly at least two
months before, and that they share with the conference organisers any
knowledge they might have of captioning companies providing sufficiently
accurate transcripts. For LHCP2020, the transcripts from the company that was
hired can be found on the conference website; the quality of the transcript is
not close to 100 %, due to the jargon/communication ubiquitous in our field.
Another possibility would be that the speakers record their presentations and
prepare a transcript of the talk in advance, before the start of the
conference, and upload it to the conference website when their talk is
scheduled. In this case only the question time would require to be captioned.

On a longer term, the CERN IT department is launching a project on an
automated speech recognition (ASR) system for high-energy physics in
collaboration with Universita Politecnica de Valencia, which has a very
advanced system for automated transcription with a very efficient training
system. The goal is to train an ASR on HEP talks with captions validated by
HEP physicists, so that such an ASR could be used in the future to caption
conferences as well as other online meetings (at CERN during most of the
working days there are hundreds of meetings running in parallel on Vidyo or
Zoom, which would translate into tens of millions of euros per year if all
meetings were captioned). Some of the LHCP2020 participants kindly volunteered
to edit the transcripts (automatic or from the 3rd party company) of some
LHCP2020 presentations, and thus provide a high-quality training dataset for
this tool, consisting of 40 presentations with transcripts validated by high-
energy physicists. Hopefully this will become an accurate, and at the same
time cost effective, solution, that could have large application (transcribed
talks can also be searched for and are more accessible not just to hearing-
impaired physicists but also to those with difficulties in understanding the
speaker).

####  3.2.5 Running of the conference

Each day the conference would alternate plenary and parallel sessions, with
short breaks between each session. Each session of the conference was chaired
by two or more conveners.

The Zoom webinars for each session were started by the "host" (one of the
session conveners or the conference organisers) typically 30 minutes before
the starting time of the session, in practice mode, to allow the speakers to
join the webinar and test their audio and video setup as well as the slide
sharing. The employee of the captioning company providing real-time subtitles
would also join at this time and test the captioning functionality.

Typically 10 minutes before the start of the session, the webinar were
broadcast to all the participants, who would then be able to connect. At this
point, streaming of the plenary sessions to Webcast would also be started.

People joining the webinar would have their microphone muted and their video
turned off by default. Video was reserved for speakers and session
chairpersons.

The session would then begin with the chairpersons starting the recording of
the session, turning on their video, saying a few introductory words, with a
brief welcome statement on the scope of the session, and a reminder to the
speakers to mute themselves unless they are speaking, and to the participants
to use the "raise hands" feature of Zoom to ask questions at the end of each
presentation (or, alternatively, to post their questions in the Discord chat
if connected via Webcast).

The chairpersons would then introduce the first speaker, who would turn on
their video and microphone, share their screen, and start their presentation,
while the chairpersons would mute their own microphone. During the
presentation, from time to time, the chairpersons would unmute temporarily to
remind the speaker of the time left. At the end of the presentation, the
chairpersons would unmute and handle the question time, unmuting the
participants who had "raised their hand" in Zoom and would thus be able to ask
their own question, or directly forwarding to the speaker the question posted
in the Discord chat. Then, the chairpersons would repeat the previous steps
for each of the following presentations, until the end of the session, when
the host would close the meeting.

Overall this setup worked very well, with no particular hiccups, with the
exception of one afternoon in which, due to a broader outage of various IT
services at CERN, it was not possible to start any Zoom webinar. The parallel
session meetings of the afternoon were thus reverted to Vidyo meeting rooms
that had been created on purpose as backup solutions, while the CERN IT
department worked on a fix of the outage, that arrived in time for the
following plenary session. The sessions went on as planned, the only problem
being that some of the recordings of the parallel sessions of that day were
not saved due to a glitch in the Vidyo server and are thus lost forever.

####  3.2.6 (Virtual) coffee breaks

In an in-person conference, coffee (and other meal) breaks serve two important
purposes (in addition of course to provide the participants the time for a
much needed pause or to get their favourite beverage): they allow the
participants to socialise, as well as to discuss with the speakers of the
session that just finished and asked them more questions on their
presentations. To recreate these possibilities in the online conference
experience, two solutions were adopted:

  * •

various virtual rooms, on the same Discord online server used to allow plenary
session attendees on Webcast to ask their questions, were created, one for
each parallel topic as well as a more general "coffee-and-tea" room, where
people could, if they wish, interact with each other, on physics or more
general topics. The same channel were also provided to allow the participants
connected to the same parallel session, if they wished, to say "hello" to
their fellow participants, and thus enforce the feeling of community that
might be partially lost with the Zoom webinars, in which the regular
participants cannot see who the other participants are.

  * •

It was suggested to the speakers, on a voluntary basis, to be available, after
the end of their session, in a Zoom meeting room ("breakout room") where they
could meet directly the participants that were interested in their
presentations and wanted to ask more questions about them.

While the Discord channels were barely used, the breakout room after the
sessions had a reasonable success. Not all the speakers provided them nor all
the participants took advantage of them, but those who did were largely
satisfied.

####  3.2.7 Communication

A small, but dedicated communication team created content that highlighted
events of the day which they posted on the Twitter channel of the conference,
@LHCPConference, re-posting on this social medium slides - with a short
associated comment - on the hot topics being shown in the plenary sessions.

####  3.2.8 Statistical information about the participants and the organising
committees

The final number of registered participants to the conference was 1301, coming
from institutions based in 56 countries and spanning 17 time zones. The
geographical distribution of the participants is shown figure 8.

Figure 8:  The geographical distribution of the participants of the LHCP2020
conference, based on the institute they work for (or university they study
at), with timezones overlaid.

The distributions of the gender of the conference participants and of the
members of various organising committees, as defined by the participants
themselves at registration time when they were given the choice between
’Female’, ’Male’ and ’Rather not say’, are shown in figure 9. Among the
organising committees (PC, SC, IAC) the male:female ratio was varying between
63:37 and 65:35, never exceeding 2:1. Among the session conveners, who were
appointed by the SC for the plenary sessions and by the PC for the parallel
ones, a similar ratio is found (64:36). Among the participants a higher
male:female ratio of 70:30 was observed, and even slightly larger among the
speakers (74:26), that were partially invited, partially selected by the
experimental collaborations, or otherwise volunteered for a poster
presentation by submitting an abstract to the conference website.

Figure 9:  The distribution of the participants and organisers of the LHCP2020
conference according to their self-declared gender.

The logs of the videoconferencing sessions indicate that there were up to 520
unique participants in the plenary sessions connected on Zoom as well as up to
377 on Webcast. In the parallel sessions there were typically several tens of
connections, with some sessions reaching up to 150 participants.

####  3.2.9 Participant satisfaction survey

After the conference, an online survey was circulated to the conference
participants to collect their feedback about the logistics, organisation,
format of the conference, and potential ideas on how to improve it. The
questionnaire was left open for about two weeks, and two reminders were sent
to the conference participants before the deadline. We collected a large
sample size ($N$) of 354 people. The survey participants are well distributed
in current position, age, continent in which their institution is located, and
between experimental and theoretical physicists. The sample does not differ
from the true population average attending a high energy conference as one of
the LHCP conference series. The information about the respondents are
collected in an online PDF file [9].

The sample is composed 44.4 % active conference participants (namely
participants that had at least one of the following roles: speakers, poster
presenters, conveners, members of the organising committee) and 55.6 % simple
attendees. Most of the respondents attended either the plenary or the parallel
or both the sessions while the poster sessions were less attended by 7.1 % of
the respondents.

##### Time format

The first set of questions that was asked in the survey concerned the choice
of the time format of the conference. Most of the participants were satisfied
with the choice of concentrating the programme in the hours between 12:30 and
18:30 Central European Summer Time (CEST), with two short 15 minutes breaks
and several parallel sessions taking place at the same time. On a 0–10 scale,
the average satisfaction was 8.7, with a drop to 7.6 for participants in North
America. The choice of scheduling the parallel sessions in the "prime time" of
the day, typically between 14:30 and 16:00 CEST, in order to make them as much
accessible as possible to participants from all over the world, was also
appreciated, with an average satisfaction of 8.6/10.

The participants were also asked about possible alternative formats:

  * •

a 2 week long conference, with less parallel sessions running in parallel,
longer breaks, and about 4 hours of conference per day.

  * •

a 1-week long conference, with "premiere" sessions whereby talks are live
followed by 1 or 2 "replay" sessions to cover all time zones and during which
recorded talks would be re-transmitted.

Only 23 % to 24 % of the respondents would have preferred these alternative
options to the format that was chosen for LHCP2020.

As an optional request, the respondents were asked to provide additional
feedback on the time format. The few received suggestions were on reducing the
numbers of sessions in parallel, on having less compressed sessions, and on
having longer time for discussions. These would unavoidably affect the
conference time schedule forcing to have either longer conference days or
going to a two-week long conference, solutions that as shown before are
disfavoured by the majority of the respondents.

##### Technical setup

The choice of Zoom as the videoconferencing platform was highly appreciated,
with an average satisfaction of 8.8/10. A few respondents also proposed
alternative platforms such as Vidyo or Microsoft Teams. It was also suggested
that CERN develops its own videoconferencing system based on Jitsi. Among the
participants who filled the survey, only 17 % connected to the plenary
sessions through Webcast, either because they could not use Zoom (10.5 %) or
they preferred Webcast over Zoom (6.2 %). The average satisfaction of the
Webcast users is 8.5/10. No alternative solution was proposed.

Concerning the format of the sessions, we asked the participants which one
would be more appropriate between a Webinar and a Meeting for plenary and
parallel sessions, poster sessions and breakout rooms. The webinar format is
clearly preferred for the plenary sessions, while the meeting format is
favoured for the poster and breakout rooms among the respondents who have an
opinion on this topic. For the parallel sessions the respondents with an
opinion are split almost evenly between the webinar and meeting formats.

The satisfaction of the participants concerning the way the question time was
technically handled can be summarised as follows. The use of the raise-hand
feature of Zoom in order to be able to turn on the microphone and ask the
question, as would happen in in-person conferences, was appreciated by a large
majority or participants, with an average satisfaction of 8.4/10. About a
quarter of the respondents of the survey would prefer to use a chat-based
system in which questions can be entered in a text window and can be up or
down voted. The Discord chat that was provided to users following the plenary
sessions on Webcast in order to ask questions was essentially not used. The
respondents that used Webcast, however, found it more useful than not (average
vote 5.6/10) to have the option available.

The satisfaction of the participants concerning how the feeling of community,
lost in a online conference, was recreated, can be summarised as follows. Most
participants did not actually use the Discord chat rooms created for the
parallel sessions and for the virtual coffee breaks; the minority who did,
found them of limited utility (average vote 5.3/10). Concerning the breakout
rooms where to meet the speakers and discuss with them during the virtual
coffee breaks, a bit more than one thirds of the respondents answered that
they used them and the idea was deemed quite satisfactory (average vote
6.7/10). On average, all the respondents said that providing such rooms is
rather important for a virtual conference (average vote 7.0/10), and future
conferences could thus consider to request it to all the speakers.

The satisfaction of the participants concerning the recordings and live
transcriptions can be summarised as follows. The respondents found the
recordings of great utility (average vote 8.5/10), few of them already watched
the recordings (17.8 %) and about half of the participants will watch them in
the future.

When asked (optionally) to provide further feedback about the technical
organisation of the conference, the respondents provided a few comments such
as:

  * •

find a better way for the participants to see who else is attending the same
event. A possible solution for this, at least for parallel sessions, could be
moving from webinar-style to meeting-style Zoom meeting rooms

  * •

having a clock running during a speaker’s presentation, visible to the
speakers themselves, rather than the chair interrupting to tell the speaker
how much time is left. Of course the speakers themselves are encouraged to use
a clock on their desk or on the computer that they use during the presentation
to keep track of the time, but solutions integrated into the technical tool
used for the videoconference would be a welcome addition.

  * •

breakout rooms are an interesting idea for virtual conferences and should be
further encouraged

  * •

having a system to upvote questions (but no downvoting)

  * •

releasing the recordings of the talks the same day as the presentation,
otherwise people could lose interest afterwards. It should be noted that post-
processing the recordings of the whole sessions (splitting them into single
contributions, adding captions) takes time and person-power and even with a
dedicated IT team it took 2 to 3 weeks after the end of the conference to have
all the recordings online on the LHCP2020 conference website. A potential
alternative would be that speakers themselves provide pre-recorded, self-
transcribed talks. This would also solve the problem of the quality of the
transcripts, which was deemed terrible by some respondents and acceptable
(though not perfect) by others.

##### Funding

The third set of questions of the survey concerned questions related to the
conference funding. Considering that the budget is very severely constrained
for an online conference without a fee, the participants assigned to the items
to be funded the following importance, ordered from higher to lower:

  * •

communication, average vote 7.8/10;

  * •

publication of conference material (recordings, post production, …), average
vote 7.5/10;

  * •

outreach, average vote 6.6/10;

  * •

poster awards, average vote 5.9/10;

  * •

proceedings, average vote 5.9/10;

  * •

others, average vote 1.3/10.

We had explicitly asked any other possible items to spend money and we have
received few suggestions. Three suggestions were related to assuring the
accessibility of the conference material to (visually, hearing, etc.) impaired
participants and one suggestions was to have a solid IT infrastructure.

The majority of the respondents would prefer to maintain a "fee free"
registration (average vote 7.4/10). However, when asked about a reasonable fee
for the the (major) online conferences, 45 % of the participants would find it
acceptable to pay a fee of between 20 and 50 euros, with 43 % favoring lower
(26 %) or no fees (17 %) at all. The respondents found that their
institutes/funding agencies should contribute financially to support the
organisation of a fully online conference with an average vote 6.4/10.

##### Outreach

While outreach activities had been foreseen for the in-person conference in
Paris, no outreach programme was included in the online conference, since the
organising committee focused, in the limited amount of time that was left for
preparing the online conference, on the scientific programme and the technical
setup. However, we asked the conference participants how important would it be
to propose a public online outreach program during an online conference and,
optionally, who could be the target public and to propose some ideas for the
future. The average importance of proposing outreach activities in an online
event is 6.6/10.

Such outreach activities should mainly target students of various levels
(high-school, undergraduates) as well as general public (ranging from kids to
retirees) with an interest in science. Attention should be paid also to
propose activities that can reach out to students and post docs from nations
without much budget for travelling around the world. Such activities could be
useful to remind the general public why fundamental science is important and
how the public money that supports it is used. Plenty of ideas are already
being developed on potential online outreach activities, by the outreach teams
of the LHC Collaborations as well as by the International Particle Physics
Outreach Group (IPPOG). They include webinars, virtual tours of the detectors,
virtual labs, hackathons, physics-inspired quizzes, and summary talks of the
main highlights of the conference explained in a way that is accessible to
everybody.

##### Overall satisfaction and feedback

The overall satisfaction of the conference participants is shown in figure 10.
The average satisfaction was found to be high at 8.6 out of 10.

Figure 10:  LHCP2020 online survey: overall satisfaction of the conference
participants.

####  3.2.10 Conclusion

Because of the extraordinary conditions related to the COVID-19 pandemic, the
LHCP 2020 conference, the eighth in the series of annual conferences on the
physics of the Large Hadron Collider, was exceptionally organised online, in
the week of May 25 to 30, 2020. It was the first fully online HEP conference
with more than 1000 participants. The conference was a success with a very
high level of satisfaction based on participant surveys. We believe the
experience gained in the preparation of the LHCP2020 online conference, as
well as the feedback received by the participants, will be useful not only in
the organisation of online events, but can also be beneficial for low-cost,
inclusive, and delocalized conferences.

####  3.2.11 Acknowledgments

We would like to thank all the people involved in the organisation of the
conference, the members of the international advisory for their suggestions
and guidance, the programme committee for setting up such an exciting program
and the speakers for delivering high quality talks and adapting to the new
schedule, the local organising committees for the preparatory work for the
Paris conference that was partially reused for LHCP2020.

We would also like to thank the CERN IT department, and in particular Thomas
Baron and Jonathan Coloigner, for all the technical support in the setup of
Zoom, Webcast, Discord, the recordings of the presentations and their
captioning. Many thanks to Connie Potter and Dawn Hudson who helped with the
preparation of the conference Indico webpage, the conference advertisement,
and took care of reimbursing the fees already paid when the in-person
conference was cancelled. An outstanding work of communication on the social
media was done during the conference by our Twitter team: kudos to Yasmine
Amhis, Zaida Conesa del Valle, and Laure Marie Massacrier.

Finally, we would like to thank all the participants of the conference, who
all together made its success, and in particular those that volunteered to
edit the transcripts of the recordings – hoping that future online events
might benefit from this work – and those who filled the survey and gave us the
precious feedback presented in this document.

###  3.3 ICHEP 2020

####  3.3.1 Introduction

The International Conference on High Energy Physics (ICHEP) is organized every
two years. Since its first edition in Rochester 1950 it has grown to the
leading conference in particle physics worldwide, where the latest
experimental results and theory achievements are presented. The usual format
is three full days of parallel sessions and another three days of plenaries.
The typical attendance exceeds 1000 participants.

The Local Organizing Committee (LOC) in Prague started the preparations in
2016, foreseeing a standard conference format. In early spring of 2020 it
became clear that the conference could not be held as originally planned due
to the COVID-19 pandemic. Several discussions within LOC and with C11 IUPAP
Commission took place, the final decision was to move the conference wholly to
a virtual format.

####  3.3.2 Conference format

Since the conference participants are scattered over the whole world, the
sessions were organized in shorter time slots, alternating between morning
(8:00 to 13:00 CEST) and afternoon (15:30 to 20:30 CEST) slots that reasonably
match people’s working hours in Asia and America, respectively. Speakers were
asked during the registration about their preferred time slots.

In order to accommodate the large number of accepted contributions (total 800
talks in 17 parallel sessions, 44 plenary contributions, 150 posters), the
conference was extended to four days of parallel sessions and four days of
plenary sessions. In addition to these so-called "premiere" sessions involving
presentations, specific “replay” sessions were organized to further ease the
attendance to participants from East/West time zones. During the replay
sessions, the recordings on YouTube were streamed. The timeline and sessions
are summarized in Figure 11. All sessions were recorded and the videos were
available on YouTube and later also in CERN Document Server (CDS).

Figure 11: The layout of ICHEP2020 parallel (left) and plenary (right)
sessions. “P” stands for the premiere session, “R” indicates the replay
session.

####  3.3.3 Technical solutions

The Zoom platform was chosen as the technical solution for the ICHEP2020
virtual conference. All talks were given live during the premiere sessions
rather than being pre-recorded. A dedicated Zoom room with a technical
assistant from LOC was available few days before the conference to allow
participants to test their audio/video. This proactive approach minimized
problems during the premiere sessions. As already mentioned above, the
premiere sessions were recorded in Zoom platform by technical assistants from
LOC as well as by the corresponding session convenor. This solution was
adopted as precaution of potential internet connection problems on the
organizer’s side. Nevertheless, these backup recordings were never needed, as
everything wound up running smoothly.

A Zoom Webinar up to 3000 participants was used for all plenary sessions and
Neutrino parallel session (where the largest audience was expected). For the
remaining parallel sessions Zoom large meeting (maximum 500 participants) was
used. All sessions were simultaneously streamed to YouTube, which provided a
backup solution for those who could not eventually join Zoom directly.

Apart from talks, young researchers and students were also invited to present
their work as posters. Poster sessions were organized during the parallel
session week. Posters were turned into mini-talks (maximum 5 slides) and
presenters were offered to upload into the agenda additional material (e.g. a
standard poster) with further details. Plenary sessions were close-captioned
by Whitecoat Captioning company that was hired specifically for this purpose.

####  3.3.4 Interactions

Opportunity for discussion – both within each session as well as informal
discussion afterwards – is a key ingredient of every conference. The
discussion between participants was stimulated by several means:

  * •

Usual questions and answers at the end of each talk or poster presentation.
Additionally, Zoom rooms remained open for typically one hour after the
respective session finished in order to enable further discussion

  * •

A dedicated Mattermost channel was setup for continuous discussion in each
parallel session

  * •

Topical discussion sessions were organized during the plenary session week.
Senior scientists were asked to lead and stimulate discussion on individual
topics

  * •

A virtual tour through Prague via drawings by one of our colleagues was
offered

####  3.3.5 Other program elements

Other events originally planned to accompany the conference were also moved to
a virtual format:

  * •

Outreach and public relations activities were performed on Facebook (Czech)
and Twitter (English)

  * •

An art competition called "BeInspired" was organized

  * •

A public lecture was given online by Barry Barish, the Nobel Prize winner
2017, in the evening of the first day of plenary session week

  * •

A European Research Council (ERC) workshop was organized during the weekend
between the parallel and plenary sessions

####  3.3.6 Statistics & feedback

Table 2 summarizes the attendance for the last three ICHEP conferences. The
virtual format of ICHEP2020 allowed for a much larger number of registrants as
compared to the previous in-person conferences. Since the ICHEP2020 conference
fee was waived, the number of registrants does not necessarily reflect the
number of active participants. Nevertheless, we encountered 2835 unique
participants (connected more than 15 min) on Zoom and approximately 200 on
YouTube. Parallel sessions were typically attended by 100 to 200 people and
plenary sessions had typically 500 to 1200 participants. The discussion panels
were attended by 20 to 100 people.

| ICHEP2020 | ICHEP2018 | ICHEP2016
---|---|---|---
Gender | Regs. | Fraction | Regs. | Fraction | Regs. | Fraction
Male | 2178 | 72 % | 893 | 77 % | 1144 | 80 %
Female | 772 | 26 % | 267 | 23 % | 286 | 20 %
Rather not say | 54 | 2 % |  |  |  |
Other | 6 | 0 % |  |  |  |
Total | 3010 | 100 % | 1160 | 100 % | 1430 | 100 %
Table 2: Number of registrants over the last three ICHEP conferences

The effort involved in organization of the virtual conference was very
appreciated and we received many congratulations. The positive feedback is
illustrated in Fig. 12, which shows the response to the post-conference
survey.

Figure 12: The results the satisfaction survey, 300 participants responded.

####  3.3.7 Concluding remarks

The ICHEP2020 was a great success for a virtual conference as evidenced by the
high-level of satisfaction from the surveys. Many more people participated in
the conference and the lack of travel burden helped make the conference more
inclusive. Nevertheless, more people were needed to run the conference (Zoom
technical assistants, management of recordings, etc.) compared to a standard
in-person event. The limitation on attendees’ interactions were at least
partially offset by topical discussion sessions and live chats on Mattermost.
Strong institutional support for ICHEP2020 is acknowledged, which allowed for
waiving of the conference fee, for example.

###  3.4 Moriond 2021

####  3.4.1 The spirit of Moriond

Since 1966, physicists have been meeting in the snowy mountains of the Alps
for a week of discussions. It was thought to be a place where all persons were
sharing a time and place, with no hierarchy. Everybody has the same speech
duration, whether they be theorists or experimentalists, graduate students or
Nobel Laureates. People from all over the world live for one week in a family-
like environment, presenting and discovering the latest results in the field,
but more importantly, _interacting_. Full attendance is required and specific
grants are proposed for those with difficulties to pay for the living
expenses.

This concept has been working for 54 years, never failing. Until the one we do
not name arrived and got us all stuck at our homes. Again, no hierarchy
applied. But the “meeting” was seriously compromised and with a knife in our
hearts we had to cancel the 2020 edition only two weeks before its launch.

One year later, responding to a certain demand, but also to our own need to
pursue, we had to face that the 2021 edition of the Rencontres de Moriond
would be virtual or would again not be at all, at the risk of falling into a
definitive breach where results would be presented elsewhere, finances would
fall into a serious situation and inactivity would lead us to different paths.
After all, there was a calendar to maintain, results kept being obtained and
released, students kept working on their theses and needed to be presented to
society in view of postdocs. The show must go on, as Freddy would say.

####  3.4.2 Three sessions, Three experiences

Over the years, the Rencontres de Moriond have split into half a dozen
different sessions, dedicated to different subjects and communities. Some are
annual, others biennial, others even quadrennial. In 2021, for this
Moriond@home we decided to organise three sessions

  * •

Electroweak Interactions (annual)

  * •

QCD and High Energy Interactions (annual)

  * •

Gravitation (bi-annual)

Each session had a very different structure with different tools – we were
definitively trying different things!

##### Gravitation

The Gravitation session taking place every odd year would have to jump into
2023 if we skipped 2021. This would have made it impossible for a large number
of graduate students to present results during their thesis time, which
motivated us to go forward with a virtual event dedicated to them – a full
poster session.

It was a short three half-day format (Tuesday to Thursday, from 13:00 to 17:00
local time), divided into two daily poster sessions of sixty minutes each
preceded by a thirty minute plenary introduction to the theme of the session.
The existence of the gather.town platform was discovered by a scientific
organizing committee member and we thought that, as it was addressed mainly to
a young public, it was worth trying.

The result was better than the organizers could have expected. It required, on
the organising side, a thorough learning on how to use and customise the
gather.town platform for our purposes. We ended up having one conference room,
six poster rooms, one meeting room, a bar and a garden where people could meet
informally.

We had 124 participants, mostly graduate students and postdocs, but also quite
a few “regulars” that were happy to meet the community, even if virtually. The
possibility to present in small groups and navigate freely from group to group
made it really close to an on-site conference. The selective activation of
microphones and cameras inside a well-defined area without interfering with
neighbouring areas is a real bonus. Also, the fact that the venue was always
available and that participants could stay and meet at any time outside the
official hours made the attendance quite lively. From a technical point of
view, we created a short User’s Guide with basic hints and proposed daily ten
minute meetings during the prior week so that people could test the platform.
Overall, there were no major difficulties except for really senior users, who
needed to be individually guided for the first five minutes or so.

At the end it was almost frustrating not to continue using the venue, as it
appeared so full of possibilities remaining unexplored.

##### Electroweak Interactions and Unified Theories

The EW session was a hybrid between what we usually do and the possibilities
that virtual meetings offer. It consisted of short half-day live sessions
(14:00 to 17:00 from Sunday to Saturday local time) over Zoom. Regular talks
were pre-recorded using also Zoom and stored using Vimeo. Links to the talks
were accessible from the program on the website and released 24 hours before
the dedicated live session. Each of these sessions started with a Young
Scientists Forum (five minute talks on the session’s theme) followed by the
discussion on the recorded talks and questions. In parallel, there was a
conference Slack account, with one channel per talk and one channel per live
session. Participants could then watch the talks at their convenience and take
the time to write questions on the corresponding Slack channel. These
questions could be answered by the speaker but would also be considered by the
session Chairperson, along with live questions that were included in the
discussion. Speakers could also follow-up with a discussion on Slack after the
live session.

This organisation was intended to limit the problem of people in significantly
different time zones, making shorter live sessions and allowing each person to
watch and comment on talks at their own pace and availability.

As questions were expected on Slack, we chose to disable the chat on Zoom in
order to avoid multiple sources of input for the session Chairpersons. Indeed
exchanges worked rather well summing up with a peak of 758 messages in a
single day for an attendance of 216 persons. Of course, things do not always
go well by themselves – we had to make a real effort for these exchanges to
work properly. Each session was moderated by a dedicated member of the
Scientific Organizing Committee, who appointed two dedicated Chairpersons (one
theorist and one experimentalist) who in turn chose secret agitators. A whole
system!

##### QCD and High Energy Interactions

The QCD session was the closest one to the classical format – seven days, from
14:00 to 20:00 local time with live talks and discussion, split by a 30 min
break. It was held fully over Zoom, each 15 min talk was followed by five
minutes of questions and discussion. For some themes, a general discussion
time was reserved at the end of the program.

Virtual coffee breaks and after-meeting discussions were expected to happen on
gather.town, but participants did not seem to find an interest in using this
extra space. The truth is that sessions were quite long and intense and breaks
were used as … well, real breaks! You can’t stretch your real legs in
gather.town.

There was an attendance of 120 persons, which is quite similar to usual in-
person participation.

####  3.4.3 Technical Aspects

##### Costs

We thought there would be an important investment in subscriptions to all the
different tools and platforms we wanted to use. But in the end it was
surprisingly cheaper than expected, as all tools used have a special plan for
education that we benefited from after a few email exchanges and documentation
provided. The reduced price was about 20 % of the regular one, which was very
affordable. Furthermore, there was no special need for hardware or supplies.

##### Human Resources

Regarding the secretariat, we had more or less the same need as for a in-
person event – dealing with the website, registrations, participant requests,
accounting, etc. was a little less demanding than for an on-site event. On the
other hand, we did not need any specific computer support. The managing of the
various platforms did require a large amount of preparation time, but could be
dealt with by the Organising Committee – creating the necessary spaces and
channels, checking and uploading each poster and video in due time, creating
all necessary links and reacting to other various details that one could not
foresee. Also, a constant human presence at-the-ready in the background was
required, making sure everything was working from the technical point of view
and being prepared to jump into a back-up solution if anything failed.
Practically speaking, this really meant an enormous amount of time during the
week.

Also, as already mentioned, an increased participation from the members of the
scientific committee was required as compared to a traditional in-person event
where building the program is their main concern. Things needed to run like
clockwork - checking speakers sound, screen sharing and image, preparing
questions, avoiding silences, handling the discussion threads, etc.

##### Fees

We chose to charge a limited fee of 50 euros, which was an amount that was
thought to be accessible to all. This was not meant to fully cover organising
expenses since we have a permanent secretariat, but it served as a kind of
engagement, avoiding inactive registrations and limiting paperwork and mail
exchanges for a null participation. We also hoped it would maximise the
presence of participants, since they did pay for the thing after all. In the
very few cases of participants explaining that they could not pay for this
fee, for budgetary or administrative reasons, we did waive it, although we did
not open a webpage for grant applications in these cases.

####  3.4.4 Participation

We had a similar number of participants in the virtual conference to what we
usually have. This relates both to the amount of talks (that was the same) and
to the fee (that might have discouraged a few visitors). We were satisfied
with this result, as we were not aiming for a very large or different event,
but rather a substitute to what we usually do. It stayed between the limits of
what is comfortably handled by our team.

We could see different trends of the daily participation amongst the three
sessions. While Gravitation was quite stable around 80 persons for 124
registrants, the EW and QCD instances had a general tendency to decrease along
the week. Also, on a daily analysis, we could see that in QCD there were fewer
participants per day relative to registrants, and clearly decreasing towards
the end of the day, while in EW it was a bit more stable. Only 1/4 of the
participants in EW and 1/6 in QCD attended the full week, as measured in terms
of connection time, and it does not mean people were actually listening all
the time. This result is quite understandable – it is very difficult to stay
connected during a full week while at work or at home. One may stay focused on
a few subjects but hardly fully participating throughout.

####  3.4.5 Lessons Learned

A very detailed preparation is needed, otherwise interactions fall into a
background noise. To achieve this, more human presence is needed before and
during the events. We also noticed that interactions are often based on prior
acquaintances. This was quite obvious in gather.town and Slack, where people
seemed happy to “see” each other. The discussion relies greatly on the fact
that people know each other and hence dare to talk, but it is difficult to
create new connections under these circumstances.

Sessions need to be short since participants cannot make themselves available
for the whole day. Sessions of three hours seem to be a good compromise. As
mentioned before, there is no real cut-off from the home/work environment as
when one is isolated in a hotel in the mountains, sometimes with a poor
internet connection (those were the days…). Also it is difficult to include
all participant time zones comfortably, so for someone having to wake up very
early or go to bed very late, making things short is generally appreciated.

Regarding the platforms used, Zoom is the easiest to use, but by far not the
most interactive. gather.town can do it all, but might not be intuitive for
some users and can be limiting in resources available – not all web browsers
supported it equally well at the time of the conference and smartphone
features are reduced. Slack seems to be a must have for communication in the
background, allowing an organised and long-lasting exchange. Combining the two
functionalities into a single application would seem to be good practice, as
too many tools or spaces can get people confused and/or information can get
lost. But this requires taking the time to structure and adapt these tools to
specific needs, so participants will find a time and place for everything.

Proposing a dedicated time slot for testing is a really useful activity. This
could happen either the week before, or simply 1/2 hour before each daily
session where organizers make themselves available for individual testing of
the various features, navigation, document sharing, sound and image,
resources, etc. This really helped to make people comfortable and confident
that everything would work smoothly when the time comes to start the
conference.

####  3.4.6 Conclusion

Virtual meetings can work, but cannot efficiently replace an in-person meeting
where the purpose is to discuss and where as much happens outside the
conference room as inside, if not more. Some things cannot happen online and
in the Moriond case, the virtual is not at all the preferred option. But in
many other situations, it is clearly a field where things are improving and
many intelligent solutions are being found.

Remains the difficulty of setting the world to rights while having a beer
virtually…

###  3.5 Connecting the Dots 2020

The Connecting The Dots (CTD) workshop series brings together experts on track
reconstruction and other problems involving pattern recognition in sparsely
sampled data. The CTD scientific program is nominally 2.5 days with a fully
plenary format, including three types of presentations: Plenary talks; young-
scientist talks; and posters. An in person workshop had been scheduled for
April of 2020 with about 80 people expected to attend.

CTD2020 was moved to a virtual conference in mid-March once it was clear that
the local and international restrictions due to the COVID-19 pandemic would
prevent an in-person event. By this time, the workshop had accepted abstracts
for contributions and had published an initial timetable for its event.
Essentially six weeks remained before the workshop, so only a limited time was
available to determine the format and overall details of a virtual event. Both
the local organizers and international committee were strongly in favor of a
virtual conference rather than no conference.

A virtual format was formulated in less than two weeks. The organizers wanted
to preserve the full scientific program as planned for the in-person meeting –
talks and posters, the planned length of oral presentations, and the planned
opportunities for interacting with all presenters (oral and poster). We judged
it to be not possible to schedule the entire scientific program during the
overlap between the EU extended-working day and the US extended-working day.
Asia-pacific participation was not prioritized as there were no abstracts from
that region contributed to the workshop. Finally, the organizers felt that we
had limited flexibility in rescheduling as speakers needed time to get results
approved (e.g, could not move presentations forward).

In the end, the organizers settled on a two-phase workshop: “recording
sessions” followed by dedicated question & answer sessions with about one week
in between for viewing recording contributions. Posters were allocated as
short talks as we had insufficient time to converge on a more creative
solution appropriate for a poster presentation.

The recording sessions happened outside of the EU/US overlap time period to
make scheduling easier, had an audience, and had a short question & answer
period for each time. There were six sessions over three days. Recordings were
done via Zoom, which was new to the CERN community at the time, and used Zoom
’s cloud recording service. Speakers were asked to share their video during
their presentation if they could (most but not all could). Sessions had twenty
participants on average and went essentially without technical issues.

The cloud recordings were available to the organizers about one hour after
each recording session. The organizers automated the process of splicing each
session into talks and managed to upload recordings to both the Indico
workshop agenda and YouTube before the start of the next recording session.
Some aspects were done by hand, including finding the time break for each talk
and uploading to both Indico and YouTube. YouTube does have an API to upload
videos but its free API quota is sufficient to upload only a few videos per
day.

A Mattermost channel was set up to host discussion for each contribution as
videos were viewed. In the end, this went essentially unused, suggesting that
something more integrated with other infrastructure was needed instead of a
standalone discussion platform.

The question/answer sessions happened about one week after the recording
sessions. There were three sessions, each two hours. Sessions were held from
10:00 to 12:00 Eastern time and were recorded. Each speaker had a one-minute
per one-slide introduction to their talk to get the discussion started. More
than 200 people had registered for these sessions. In general, the question &
answer sessions had good (50+) attendance, however the discussion was largely
driven by a few “experts”. The short introduction to each talk worked well.
Speakers followed the format and generally successfully conveyed the important
conclusions of their work.

Conclusions that the organizers drew from this workshop included:

  * •

There was a clear audience preference towards real-time interactivity rather
than recordings.

  * •

Free/open discussion was clearly more difficult in a virtual setting.
Perceived higher barrier to speaking up and coffee/lunch/evening discussion
time was lost

  * •

Indico timetables did not handle events without events every day very well.
Days without contributions were shown in the agenda making it more difficult
to find active days.

  * •

It is important to engage, remember and thank the audience. Workshop
photographs are one good approach (enabled by Zoom).

  * •

Often conferences and workshops are organized by a small team having limited
experience as event organizers. For CTD, the organizers did not consider the
potential impact of up-front cost commitments when planning CTD2020. As HEP
starts to organize in-person or hybrid events, it is also a good opportunity
to think about best-practices for in-person event organization.

###  3.6 LLVM Developers Meeting

####  3.6.1 Introduction

The LLVM Project [10] is a collection of modular and reusable compiler and
toolchain technologies. The LLVM Developers’ Meeting is a bi-annual gathering
of the entire LLVM Project community. The conference is organized by the LLVM
Foundation and many volunteers within the LLVM community. Developers and users
of LLVM, Clang, and related subprojects will enjoy attending interesting
talks, impromptu discussions, and networking with the many members of our
community.

One of the event’s main goals is to provide a "venue" where the geographically
distributed developers community can interact and exchange ideas. The
canonical event format includes:

  * •

Technical Talks – These 20 to 30 min talks cover all topics from core
infrastructure talks, to project’s using LLVM’s infrastructure. Attendees will
take away technical information that could be pertinent to their project or
general interest.

  * •

Tutorials – Tutorials are 50 min sessions that dive down deep into a technical
topic. Expect in-depth examples and explanations.

  * •

Lightning Talks – These are fast 5 min talks that provide a "taste" of a
project or topic. Attendees will hear a wide range of topics.

  * •

Panels – Panel sessions are guided discussions about a specific topic. The
panel consists of $\approx 3$ developers who discuss a topic through prepared
questions from a moderator. The audience is also given the opportunity to ask
questions of the panel.

  * •

Birds-of-a-Feather – Large round table discussions with a more formal directed
discussion.

  * •

Student Research Competition – Students present their research using LLVM or
related subprojects. These are usually 20 min technical presentations with
Q&A. The audience will vote at the end for the winning presentation and paper.

  * •

Poster Session – An hour long session where selected posters are on display
for attendees to ask questions and discuss.

  * •

Round Table Discussions – Informal and impromptu discussions on a specific
topic. During the conference there are set time slots where groups can
organize to discuss a problem or topic.

  * •

Evening Reception – After a full day of technical talks and discussions,
attendees gather for an evening reception to continue conversations and meet
other attendees.

The type of attendance is active project developers, novice and advanced
users, students and researchers, programming language enthusiasts, and anybody
interested in compilers. The usual in-person attendance is around 400 people.

####  3.6.2 Event Format

In 2020 due to the COVID-19 pandemic the event was fully virtual [11]. During
its duration of three days there were keynote, technical talks, lightning
talks, tutorials, poster sessions, bird-of-a-feather, round tables, student
research competition (SRC), breaks, social event. The keynote, technical
talks, lightning talks, tutorials and SRC were pre-recorded with a moderated
live Q&A session. The SRC also included voting for the three best talks. The
talks relied on the Whova app for content delivery and Zoom for the live Q&A.
The rest used the Remo platform which allowed attendees to virtually walk
around and network.

The event was aided by a professional event organization team [12].

The registration deadline was one week before the event and then extended
until the end of the event. There were two types of tickets: a free
registration subsidized by sponsors; and a US$50 supporter ticket which
supports the LLVM Foundation activities. The total amount of registrants
exceeded 800, but the real attendance was around 400.

The event had a code-of-conduct and special program encouraging diversity and
inclusivity. The event organized branded merchandise such as T-shirts and
other apparel.

####  3.6.3 Interactions

The conference materials were made available at the registration deadline – a
week before the event. The discussions happened in the Whova platform as well
as the live session questions. There were channels for public discussions. The
talks were made available in YouTube.

####  3.6.4 Conclusion

The LLVM Developers’ Meeting spans at least at three continents which makes
picking a convenient time for speakers and attendees very challenging. It also
seemed difficult for many people to dedicate the necessary amount of time
while being in their usual work/home environment. The virtual platforms served
their purpose well on a technical level, however did not achieve the usual
attendance at breaks and the social event.

The event happened relatively early in the pandemic when the experience
organizing virtual events was not substantial. Virtual events pose specific
set of challenges in terms of time zones, networking and technical solutions.
Overall, the event achieved its mission due to the extra efforts by the
organizers and the community.

###  3.7 Snowmass Community Planning Meeting

####  3.7.1 Origin of the Snowmass Community Planning Meeting

The Snowmass Planning process [13] gathers the U.S. particle physics community
to begin defining the most important questions in our field. Over the course
of $\approx$1 year, various frontier working groups identify opportunities to
address those questions and produce a report that is used as input for the
Particle Physics Project Prioritization Panel (P5). The output of P5 defines
the roadmap for the field over the subsequent $\approx$10 years. This exercise
is repeated approximately every 7 to 8 years. A Snowmass Community Planning
Meeting (CPM) [14] was scheduled to bring the community together and commence
this cycle.

The nature of the meeting involves a significant amount of planning
discussions, debate, and interactions with physicists from a wide set of
working groups. The CPM represents the beginning of a process, rather than a
summary or conclusion of work that has already been performed. There are
nominally ten frontiers that reflect physics drivers as well as technology and
infrastructure. Naturally, many proposed topics span multiple frontiers. Thus
there are needs for both plenary lectures as well as parallel discussions.

The goals of the CPM were stated on the event page at [14], and highlighted
communication:

  * •

Inspire the community about the field, and encourage them to engage broadly in
the Snowmass process

  * •

Inform the community about plans from other regions and from related fields
and planned Snowmass activities

  * •

Listen to the community

  * •

Provide space for members across the field to talk to each other and to
discuss, promote, and develop new ideas

  * •

Establish cross working-group connections and identify gaps

The 2013 Snowmass cycle began with a similar CPM meeting at Fermilab in the
fall of 2012. This was a fully in-person meeting, and approximately 400
community members participated in the meeting. The 2020 CPM initially
attempted to replicate that successful kickoff meeting, and aimed for a 2.5
day meeting in November of 2020. In March 2020, a local organizing committee
(LOC) was formed to begin planning for the meeting. Throughout the spring of
2020, it became clear that the hopes of an in-person meeting were inconsistent
with the reality imposed by COVID-19, and by June of 2020, the decision was
made to move to a fully virtual meeting format. The LOC chosen for the in-
person meeting continued and began the process of adapting for a fully virtual
meeting.

####  3.7.2 Meeting Organization/Technical Setup

As the CPM focus was on U.S. planning, the meeting times were prioritized to
fall during the traditional 9 am to 5 pm workday for all continental U.S. time
zones. Thus, the meetings started at 11 am US Central time and concluded by 4
pm US Central. To accommodate the broad program the community needed to
discuss, the meeting was extended to four days. New dates were selected to
avoid major conferences, holidays, and the US general election. The virtual
Snowmass CPM was scheduled for October of 2020, and the meeting organization
benefited from the conferences, meetings, and workshops that were forced into
a virtual format on short notice during the spring and summer of 2020.
Registration fees were eliminated to encourage participation and improve
accessibility.

The meeting structure consisted of two days of plenary meetings and two days
of parallel sessions. The plenary agenda focused on interactions with the
representatives of other global HEP programs, the views from funding agencies,
and a community town hall to introduce new initiatives. On the final day, the
plenary sessions focused on summaries from the frontier working groups and
motivational visionary talks from experienced physicists.

The plenary sessions utilized a Zoom webinar with a capacity of 5000
participants. A decision was made to keep the security settings fairly strict
to avoid unwanted incidents. Chat was disabled in the public-facing portion of
the webinar, and the question & answer feature was enabled for the
participants to communicate with the speakers and moderators. To improve
accessibility during the meeting, captioning services were procured from Ai-
Media to provide real-time description of the presentations via the Zoom
application. To encourage participant engagement, multiple Slack channels were
created within the Snowmass Slack space. These channels were communicated to
the participants and provided means to contact the organizers, discuss the
content of the plenary program and ask questions of the speakers.

Plenary sessions speakers were provided with unique "panelist" links to access
the private side of the Zoom webinar. In the weeks leading up to the meeting,
the LOC held multiple rehearsals to practice the mechanics for the plenary
sessions, such as webinar content projection, recording and the question &
answer interface. These rehearsals also provided speakers with opportunities
to become familiar with the interface. Additional technical and code-of-
conduct training were required for all session chairs.

During the parallel sessions, up to 18 Zoom "rooms" were provided in the
format of a traditional Zoom meeting. Since these were smaller groups and
focused on discussion and interaction, the security settings enabled chat and
participants could unmute themselves and share their video. Presenters were
promoted to co-host to share their slides. Typically the program was scheduled
such that each room contained similarly themed topics. This structure was
chosen to prevent excessive fragmentation of the meeting, and to ensure
participants could locate the relevant topics. Between sessions, the rooms
were held open to provide space for carryover conversations and to encourage
the development of new ideas and relationships. We note these rooms were often
underutilized. Each parallel session room had a designated "host" that focused
on the Zoom meeting logistics and security, while the physics content of the
meeting was handled by the session conveners.

The overall structure of the meeting and instructions on how to navigate the
meeting was collected in a guide for participants, which was modeled on a
similar conference guide provided for ICHEP 2020 (§3.3). This included
specific instructions for Zoom hosts, conveners, moderators, and speakers, and
was posted to the Indico agenda.

####  3.7.3 Statistics and lessons learned

About 2000 people registered to attend the meeting prior to the start of the
CPM. Registration was required to receive the link to the Zoom webinar. By the
final day of the meeting over 3000 people had registered. Statistics were
collected on meeting participants from within the Zoom software. Unique
connections (identified by Zoom display name) were tracked as a function of
time, and the total integrated unique connections were recorded (see here for
more information).

Figure 13: Snowmass CPM participation statistics as determined by self-
identified Zoom user display name. (a) Unique connections vs time (b)
Integrated unique participations vs time

.

Since individuals have the freedom to enter different names into the Zoom
software, it is estimated that these statistics over-count the actual
attendance by up to 10 %.

An examination of Figure 13 reveals several features associated with this
meeting. First, more than 2500 unique individuals connected to the meeting at
some point during the week. This represents about 85 % of registrants. This
attendance represents a factor of $\approx$6 times the participation of the
in-person meeting in 2012. Removing the barriers of cost and ensuring the
content was only a click away greatly enhanced the accessibility of the
meeting.

On the flip side, engagement was a challenge each day and over the course of
the week. Attendance tended to peak early in each day and dropped off anywhere
from 20 % to 50 % from the peak. Attendance also dwindled from a peak of
nearly 1200 participants early on the first day to 400 at the conclusion of
the meeting

Because security was prioritized during the plenary webinar sessions to avoid
Zoom "bombing" or other disruptive experiences, participants reported a lack
of sense of community. They were unable to clearly see other participants and
directly chat through the Zoom interface. The Slack channels were used at some
level, but the traffic was relatively low. Many of the in-person brainstorming
interactions were not replicated by this setup. An investigation into tools
such as Slido which allows an integrated, moderated wrapper to the Zoom
webinar is warranted for future efforts.

###  3.8 OSG Virtual Meetings

####  3.8.1 Introduction

The OSG is a distributed organization that advances the state of the art of
distributed High-Throughput Computing in the United States and worldwide. With
staff and users scattered around the globe, the OSG has relied on virtual
meetings for a long time, especially for smaller groups and events. However
two longstanding and larger in-person kinds of events—the OSG All-Hands
Meeting and the OSG User School—were significantly affected by the COVID-19
pandemic, and they are the focus of this section.

####  3.8.2 OSG All-Hands Meeting

The OSG All-Hands Meeting (AHM) is the premier annual meeting for the OSG,
bringing together campuses, science users, site administrators, developers,
and anyone else interested in the latest news and technologies. Historically,
the format has been a mix of conference presentations, workshop segments for
discussion, training, and time for the OSG Council to meet. In 2020, the usual
in-person AHM had been scheduled for March. In early spring, the then-nascent
pandemic caused OSG management to postpone the meeting until September 2020,
to see how things played out. Of course, by autumn, travel and mass gatherings
in the United States were largely not an option so the meeting was made
virtual [15]. And the regularly scheduled AHM in March 2021 was never even
considered as an in-person event [16]. Because the focus of the AHM had always
been to bring the community together in person, using a virtual format was a
significant change.

For the switch to an all-virtual format, the program changed in a few critical
ways. First, the schedule was extended to a whole week (as opposed to 3 and a
half days) but with daily content reduced to a single track totalling 3 hours.
The net result of these changes was significantly fewer hours of
presentations. Further, workshop-like and training elements were dropped in
favor of other, more focused meetings. However, open discussion time was added
to the end of each day in hopes of recapturing a bit of the feeling of being
in a shared space. Thus, the resulting daily schedule was 1.5 hours of talks,
1 hour of break, 1.5 hours of talks, and then 1 hour of discussion time, for a
total of 5 hours of allocated time each day. Especially for the AHM 2021, the
talks for each day were organized around a topic or two with a specific
audience in mind; that way, attendees with specific interests could focus on
certain days. There were usually 3 to 5 talks per session, so about 10 min to
20 min apiece with a bit of time for questions during the transition times.

The main technologies used for the AHMs were the Fermi National Accelerator
Laboratory instance of the Indico scheduling system and Zoom for the meetings
themselves. Indico has been used for a long time in the OSG community, so that
was nothing new. Zoom was chosen due to its already ubiquitous nature in March
2020—most people in the community already had and used it, plus several of the
institutions containing OSG staff had institutional access which made planning
easy. Each day, one main Zoom meeting was home to the talk sessions, the break
between, and one of the discussion areas afterward (dubbed “the hallway”);
separate Zoom meetings existed for the one or two parallel, topical discussion
areas. Registration in Indico was required and the Zoom links with embedded
passwords were emailed to all registered participants daily. OSG staff had
roles throughout the event, with a session moderator on duty at all times,
helpers during sessions to deal with Zoom issues and chat questions, and so
on. There was also a Slack workspace channel for the meeting, but it was used
lightly. Looking at participants for the AHM 2021, a total of 336 people
registered. By analyzing detailed Zoom logs, it seems that just under 300
unique attendees joined at least once during the week, which is easily double
the participation of any prior in-person OSG AHM. However, peak attendance
during any day or session was about half of the total registration count.
Thus, it can be misleading to look just at peak attendance statistics, which
are easy to see while the meeting is in progress. From the peaks, one might
conclude that a significant fraction of registrants did not attend, when in
fact over 85 % of registrants attended at some point. Attendance varied
considerably by topic, as might be expected, and discussion times were
attended lightly. Figure 14 shows an example of OSG AHM 2021 live attendance
numbers.

Figure 14: Sample of OSG All-Hands Meeting 2021 live attendance.

In sum, both virtual OSG AHMs went smoothly and participants who provided
feedback (there was no post-event survey) seemed happy. Having fewer total
hours of content meant that the program committee had to focus carefully on
invited talks and the structure of sessions and days, and this extra focus
seemed to result in a high-quality program that appealed to many. A real
benefit of having virtual meetings was people from around the world who could
not attend in person, due to travel constraints, were able to participate.
Attendees came and went as schedules allowed and topics appealed, and it
seemed that the hour-long break between sessions helped some people fit the
AHM into their busy schedules. Discussion times were lively and mostly on
topic for those few who joined. Of course, there are always ways to improve!
By far the biggest request was to record and post videos of the talks, which
will be done for future virtual events. And the staff identified a number of
smaller ways in which to try to engage the community more and keep
participants interested in the content longer.

####  3.8.3 OSG User School

The OSG User School is by far the largest and most important training event
that OSG offers to its community of researchers around the world. For 10 years
prior to the pandemic, the School was run at the University of
Wisconsin–Madison as a week-long in-person synchronous training program. In
many respects, this event was like the Moriond conference described in Section
3.4: The focus was on camaraderie and the shared, intensive learning
experience, with everyone from advanced undergraduates to graduate students,
post-docs, staff, and faculty all treated the same and working together. In
2020, the pandemic began just as the application period for the School was
ending, throwing all plans for the year into disarray.

After much consideration, the plan for 2020 evolved into an offering with a
curriculum similar to but pared back from previous in-person events. Then to
take advantage of some benefits of remote learning, the OSG Virtual School
Pilot 2020, as it became known, added a strong focus on personalized learning
and one-on-one and small-group learning and consulting opportunities. The
primary goal for participants morphed into a very personal one: Learn enough
about distributed High-Throughput Computing to apply it to at least one
research project, and have that project running by the end of the event. Of
the accepted applicants, twenty chose to participate in the re-imagined Pilot
[17].

For technology, the venue for lectures was Blackboard Collaborate and then
Zoom, Slack, and regular email were used heavily for individual and small-
group interactions. For each lecture day—and there were only five such
days—live lectures occurred twice each day to provide flexibility in
participant scheduling; lectures were recorded and posted after the event.
Exercises were posted on the public website, and all participants were given
logins to Access Points at the University of Wisconsin–Madison and OSG
Connect.

Overall, the Pilot School went very well and most participants reached their
individual goals for the event. The staff learned a great deal, too, and so
the OSG Virtual School 2021 will follow a similar pattern bit with
improvements throughout [18]. Nonetheless, the OSG User School will return to
its in-person format as soon as possible.

####  3.8.4 General Observations

Based on OSG’s experience with virtual events in 2020 and 2021, there are a
few observations to make about such events and about what the future might
hold.

As others in this report have noted, virtual conferences and open meetings
have a number of advantages and disadvantages. The primary advantage is the
opportunity to reach more members of the community who would otherwise be
unable to attend due to time constraints, travel costs (financial and other),
and so forth. Both OSG All-Hands Meetings had far more total unique
participants—double or more—than previous in-person events. Accordingly,
without travel and other in-person costs, OSG was able to offer its events
without charge, expanding access and simplifying logistics.

There are many disadvantages, all fairly obvious at this point: lack of in-
person connections and interactions, time zone disparities that make
attendance difficult for some, a greater sense of fatigue that necessarily
shortens events, and seemingly reduced engagement due in part to still being
“in the office” and the ease of leaving and rejoining at will. Nonetheless,
perhaps because of the pan- in pandemic, everyone seemed to understand the
limitations and did their best to engage and enjoy.

One might be tempted to say that an all-virtual event is easier to organize
than an in-person one. However, the OSG experience was that the effort savings
from having no travel, accommodation, food, and venue logistics were largely
offset by effort expenditures on preparing virtual venues, documenting and
practicing key procedures and back-up plans, preparing staff, and so forth.
Simply put, it takes a great deal of effort to run a successful virtual event.
That being said, preparations for a virtual event need not start as far in
advance (by calendar days) as for an in-person one.

Another lesson learned was that it can be useful to completely rethink the
approach to and goals of an event when it must become virtual. For OSG, this
was especially true of the User School, where the in-person approach simply
did not translate into a virtual one. But even for the All-Hands Meetings, it
was expected that interactions among participants would be severely reduced
and hence the common AHM elements that depended on it—such as training
events—were best omitted and handled via other, focused events.

Looking forward, the next great challenge will be the hybrid event. While no
specific plans are in place yet, it is generally accepted within OSG that
future AHMs will have to include a significant online component. What does
such an event look like and how will it work? The key will be to identify the
elements of virtual events that worked the best and preserve those. For
example, it may be good to keep the condensed, single-track schedule of the
virtual AHMs, even for the in-person participants, thereby acknowledging the
fatiguing, come-and-go nature of virtual events. The extra time made available
by a condensed schedule could be used for in-person activities that emphasize
the benefits of being together for those who choose to do so—that is, more
workshops, training events, discussions, and loosely structured collaborative
work time. Much work remains to be done in this space. Given the uncertainty
of the future, it might even be helpful to have another workshop in a couple
years to discuss hybrid meetings!

###  3.9 PyHEP 2020 Experience and 2021 Plans

####  3.9.1 Origin and format of the PyHEP series of workshops

The PyHEP, "Python in HEP", workshop series started in 2018, recognizing the
increasing importance of Python in Particle Physics. The workshops are
organised as part of the activities of the PyHEP Working Group of the HSF, the
HEP Software Foundation [19]. Under the support of the HSF, the aim from the
onset has always been to provide an informal environment to discuss and
promote the usage of Python in the Particle Physics community at large.
Furthermore, diversity and inclusion aspects have also been taken seriously
into consideration, in terms of the set of participating communities, and
cultural backgrounds, gender, ethnicity, disability, sexual orientation of
participants.

The first workshop, PyHEP 2018, was run as a pre-CHEP 2018 conference event,
profiting from the presence of a large community in Sofia, Bulgaria. Slightly
over 10 % of the CHEP 2018 attendees participated in PyHEP 2018.

The workshop format remained unchanged until the time of the COVID-19
pandemic, that is with an in-person format, given the spirit and goals of
PyHEP:

  * •

Only plenary, topical, sessions

  * •

Bring together users and developers

  * •

Very informal, with significant time for (lively) discussions

  * •

Educative, not just informative

  * •

A mix of keynote presentations, tutorials and "standard" 20 + 10 minute
presentations

The COVID-19 pandemic constrained the workshop organization into a decision to
either cancel the event or run it fully virtually. With a few months ahead of
us we opted for a virtual event and we learned a great deal from this
experience.

####  3.9.2 PyHEP 2020 organisation and running

To adapt to the requirements and constraints of a virtual event, the duration
of the workshop was extended from 2.5 days as in the 2019 in-person format to
5 days, with shorter sessions. Because of the global nature of the event we
organized the sessions in two different times zones – a "Europe-friendly"
session and an "Americas-friendly" session. The former was by default 3-hour
long whereas the latter only lasted 1 to 1.5 hours. We had approximately a
third of the time devoted to tutorials and two thirds devoted to standard
presentations.

PyHEP 2020 became a truly global event with participants from all over the
world. Without travel and/or budget constraints, and no registration fees, the
level of interest increased significantly, from about 50 to 70 participants in
in-person events to an incredible 1000 registrations, which we had to limit to
for technical reasons related to the video conferencing system.
Understandably, we observed a significant increase in the fraction of students
registered, including undergraduates. In fact, almost 40 % of the participants
were not members of an experiment or a collaboration (e.g. students and
theory, simulation or instrumentation colleagues).

The workshops try to timely address topics in the spotlight as well as
important topics specific to Particle Physics. PyHEP 2020 had sessions on the
following topics: analysis fundamentals, analysis platforms and systems,
automatic differentiation, performance, fitting and statistics, and the HEP
analysis ecosystem.

We strongly encouraged all presentations to be prepared as (Jupyter) notebook
presentations, with all material made publicly available on GitHub. To enhance
the interactive experience we also encouraged the preparation of a "Binder
launch button" so that any participant could follow and experiment in real
time the notebooks being presented by simply launching on the browser the
material. To ensure a smooth run we used both the Binder Federation and the
CERN BinderHub resources (for those with CERN accounts), and made sure that
resources on Binder were (kindly) allocated by the Binder Team for the
relevant repositories at the time of the presentations. All material got
posted onto the workshop agenda, including slides, GitHub repository links,
and the links to the recordings. The latter were captioned – thanks to our
sponsors – and uploaded on to a dedicated playlist of the HSF channel on
YouTube.

As mentioned above, PyHEP workshops allow for a fair fraction of time for
discussion, which is paramount. The online platform Slido was used to
crowdsource questions from the audience: via a web page each participant has
the opportunity to post a question, even anonymously, and also "upvote" or
"downvote" any question so that, effectively, by the beginning of the Q&A
session, the session chair sees a prioritising set of questions that got
selected as the most popular and interesting ones. At the end of the Q&A
sessions all questions got copied to Slack in the appropriate topical channel,
where speaker and participants could continue to discuss and exchange. A few
polls were also run via Slido as a fun means to socialize. The use of Slido
turned out to work extremely well; about 40 % of the participants joined and
used the platform at least once.

####  3.9.3 PyHEP 2021 planning

The PyHEP 2021 workshop [20] will again be held as a virtual event, in early
July. At the time of writing we have approximately 950 registrations, which
shows the continuous interest from a large fraction of the community in these
series of workshops.

Unlike last year’s edition, this year’s will not be run in two different time
zones but nominally between 14:00 and 17:00 CEST, a time slot that has proven
appropriate, or at least acceptable, to a large fraction of the attendees. The
format is otherwise largely kept the same, though live streaming to YouTube
will be experimented for the first time atop the set-up of the usual video
conferencing room (with Zoom).

####  3.9.4 Lessons learned

As expected, virtual events are far more inclusive and truly global, with
participation from all over the world, even if time zone constraints imply
that some participants are either attending (very) early in the day or rather
late in the day. Workshops such as PyHEP, which are meant to foster exchanges
among experts and learners, with plenty of time devoted to discussions, are
nevertheless made more challenging when running virtually, though the
organisation burden is less. In the future we may consider alternating the
workshops as virtual/hybrid and in-person.

###  3.10 SciPy 2020 Experience and 2021 Plans

SciPy, the Scientific Computing with Python Conference, is a community
conference dedicated to the advancement of scientific computing through open
source Python software for mathematics, science, and engineering. The annual
SciPy Conference allows participants from all types of organizations to
showcase their latest projects, learn from skilled users and developers, and
collaborate on code development.

The first SciPy meeting was held at CalTech in 2001 for a few dozen attendees.
By 2009, attendance had reached 150 and, exceeding the capacity of the CalTech
facilities, the meeting moved to the campus of UT-Austin in 2010. The most
recent in-person meeting had roughly 800 registrants.

####  3.10.1 Typical in-person SciPy meeting

A SciPy meeting is organized around three main elements:

Tutorials

half- and full-day, interactive classroom setting. Instructors propose topics,
and the Tutorials Committee chooses those that make the most compelling case
for their pedagocial approach and appeal to the widest audience of SciPy
attendees.

Conference

single- and multi-track sessions for audiences ranging from small groups to
all attendees.

Keynotes

45 min to 1 h plenary talks, typically one focused on a significant scientific
advance enabled by Python, one focusing on the ecosystem of scientific python
packages, and one addressing one of the broad themes of that year’s meeting,
not necessarily from a Python perspective. In recent years, we have also held
a keynote on diversity, equity, and inclusion.

Themes and Minisymposia