Digital Information Seeking and Sharing Behavior During the First Wave of the
COVID-19 Pandemic

Mehk Fatima1, Aimal Rextin2, Mehwish Nasim* 3, Osman Yusuf 4

1 Asthma and Allergy Institute Pakistan and Department of Computer Science &
Information Technology, University of Lahore, Gujrat Campus, Gujrat, Pakistan

2 Asthma and Allergy Institute Pakistan and Department of Computer Science,
COMSATS University Islamabad, Islamabad, Pakistan

3 Flinders University, Australia.

4 Asthma and Allergy Institute Pakistan

<EMAIL_ADDRESS>
## Abstract

People turn to search engines and social media to seek information during
population-level events, such as during civil unrest, disease outbreaks,
fires, or flood. They also tend to participate in discussions and disseminate
information and opinions via social media forums, and smartphone messaging
applications. COVID-19 pandemic was not any different. However, the proper
medical awareness and correct information dissemination is critical during a
pandemic. An unprecedented amount of internet traffic related to this topic
was generated during the start of the pandemic all over the world.

In this work, we have analysed the electronic data generated by users from
Pakistan on Google Search Engine and WhatsApp to understand their information-
seeking behavior during the first wave of the pandemic. The paper aims at
analysing how the Pakistani public developed their understanding about the
disease, (its origin, cures, and preventive measures to name a few) by
analysing digital data. We found that the public actively searched and
discussed information at the start of the pandemic. However, their interest
waned with time and was reinvigorated only when something novel or shocking
seemed to have occurred. Understanding this _information seeking behavior_
will allow corrective actions to be taken by health policymakers to better
inform the public for possible future waves of a pandemic through electronic
media, as well as and the social media companies and search engines to address
misinformation among the users in the emergent markets.

## Introduction

first wave of COVID-19 pandemic tested the health care systems and economies
around the world. A key point noted in Pakistan was the lack of understanding
about the causes, symptoms, and preventive measures of COVID-19. That
frequently led to lax attitude towards social distancing protocols[1] or
widespread adoption of pseudo-medicinal remedies that are known be ineffective
or to have dangerous side effects [2].

The World Health Organization [3], after the H1N1 pandemic considered reliable
communication as one of the most pivotal components to fight a pandemic. In
the current pandemic, social media played a key role in the spread of both
information and misinformation about the pandemic. For example, in Spain,
WhatsApp was the second most used channel to acquire information about the
pandemic [4]. Since the information on social media is not always reliable
hence it is not surprising that in extreme cases this has led to people using
hazardous chemicals to wash their vegetables or clean their throats even in
developed countries like the US[5]. Two malarial prophylaxis drugs:
Chloroquine and Hydroxychloroquine, got massive public attention in the last
week of March 2020 after the endorsement of multiple high-profile figures
including President Donald Trump, resulting in online searching of chloroquine
increasing by 442% and hydroxychloroquine by 1389% in America [6].

In October 2020, Gallup Pakistan released a survey report that described the
public’s behavior on COVID-19, just before the increase in the number of new
infections during second wave in Pakistan. According to that report, almost
75% of the population thought that COVID-19 was under control and the need for
continued precautions was no longer necessary. Whereas, nearly 70% of the
public thought that the threat of the SARS-CoV-2 virus was exaggerated, 46%
consider COVID-19 as a foreign conspiracy, and 45% public thought that it was
a laboratory-made virus. Hence, according to the survey, the public
underestimated the threat posed by this disease and many even considered it
unreal [7].

Conveying correct medical information to the public is extremely important in
a country like Pakistan with limited medical resources, and the high
prevalence of pseudo-medicine and quackery in order to ensure that the
information seekers have been provided with correct and reliable information.
The aim of this paper is to understand how the Pakistani public developed its
understanding of this disease, its origin, cures, and preventive measures.
Understanding this _information seeking behavior_ will allow corrective
actions to be taken by health policymakers to better inform the public for
possible future waves of this pandemic, especially through electronic
channels.

We now review how the first wave of the pandemic progressed as it is likely to
influence how people seek information about it online. Figure 1 summarises
major events during the first wave from a local perspective.

Fig 1: Timeline of major events during first wave of COVID-19 pandemic in
Pakistan.

### Organization

The paper is organized as follows: Section $2$ summarizes studies relevant to
this research. In Section $3$ we will give an overview of our method including
which sources of digital information we will use and our strategy to analyse
this data. In Section $4$ we will present the results from our first source of
digital information, Google Trends, while in Section $5$ we present the
results from analysing the $2{nd}$ digital data source, WhatsApp. Section $6$
concludes the paper.

## Related Work

Due to the sheer volume of available online information, it is very difficult
for a common person to differentiate between information and misinformation
[8]. Ideally, a rational and careful selection of their information diet [9]
is needed which is frequently not easy. Hence it is no surprise that according
to a WHO report published in 2011, effective communication is one of the
biggest challenges that need to be handled in a global pandemic [10].

The public acquired the needed information about the SARS-CoV-2 virus from
various formal and informal sources and then applied various strategies to
make sense of information [11] [12]. These information sense-making techniques
included the expertise of the source, the tone and style of language, the
sponsor of the site and whether there is consensus between multiple sources
[13]. There are various instruments available to assess the reliability of
online health information such as Health on Net [14], the JAMA benchmark [15]
and DISCERN [16]. However, surprisingly most health information websites lack
these instruments and hence making it difficult for a layperson to assess the
reliability of available health information [17].

Still, the internet is used extensively for health-related information, for
example, in one study 56.1% of endocrinology patients said that they seek
health information online, with 77.1% believing the information to be of high-
quality [18]. In another study, senior adults (mean age 77 years) internet was
trusted more than television or radio for health-related information but less
than their social relations especially those connected to healthcare [19],
whereas younger adults and adolescents [20] are still more likely to trust the
health information available on the internet more than older people [21, 22].
A study conducted in Australia reported that myths about COVID-19 were more
believed by lesser young educated men and those with non-English speaking
background [23].

Online search trends can be very helpful in digital surveillance and
prediction of an infectious disease. Relative search volume regarding COVID-19
increased during the early period of the pandemic and there was a positive
correlation between daily new cases and relative search volume [24][25]. A
study found a strong correlation between COVID-19 related Google trends and
daily new cases in the US, with $R$ value around $0.80$ [26]. Similarly, a
sharp hike in Google trends happened in searches related to COVID-19 after the
detection of the first case in Taiwan [27]. This strong correlation between
the daily confirmed cases and related Google search trends worldwide can be
used to predict the trends of outbreak [26] [28]. The rapid increase in web
searches on COVID-19 and related topics also created an infodemic like
situation and caused the worldwide spread of misinformation on disease [29,
30, 31]. Effective strategies are needed by governments and public health
organizations to better manage such infodemic and strengthen the public
awareness on the outbreak [24]. Educating the public to use websites of
official public health forums can be helpful in this regard [32].

Social media played a key role in propagating health-related misconceptions
and poses a big challenge to practitioners and policymakers [33]. The main
reason for this challenge is that many people are not clear about the
relationship between science, policy-making and media [34], and they tend to
rely more on nonscientific but more definitive advice. For example,
misinformation circulating during the 2014 Ebola outbreak challenged the
efforts of health workers to control the epidemic [35]. Even in countries like
Germany, Italy, US and UK, social media movements incited people to resist
getting measles vaccination [36],[37]. Similarly, another study analyzed
$2691$ tweets about the treatments and preventive measures of gynecologic
cancer and found $30\%$ of them to contain misinformation [38]. While another
study found that $40\%$ of links shared on health-related forums contained
fake news and were shared more than $450,000$ times between 2012-2017 [39].

Psychological and cognitive biases greatly influence how we react in a
pandemic. People anticipated that the SARS-CoV2 virus cases would grow
linearly, and they underestimated the possibility of an exponential growth
[40]. This lack of understanding and failure of public health messaging had
disastrous results even in developed countries [41]. Conspiracy theories and
myths pose another serious threat to public health and affect the behavioral
responses of people [42] that can dangerously affect the situation in a
pandemic. Pseudo-medicinal information and conspiracy theories about COVID-19
circulated through social media traveled faster than the virus itself [43].
Many websites containing unproven claims about COVID-19 are widely visited by
the public. They also shared on social media sometimes due to naivety and
surprisingly sometimes intentionally to share inaccurate information [8]. This
lack of accurate health information on COVID-19 also severely affected the
psychological condition of the general public during the pandemic [44].

Several attempts are made to track the online health-related pseudo-medicinal
information. Ghenai and Mejova built a tool to track misinformation about
health concerns on Twitter. They collected more than $13$ million tweets
during the Zika outbreak in 2016 and applied automated machine learning
techniques to identify rumor-carrying tweets [45]. In a similar study,
researchers tried to track rumors about vaccines on social networks [46].
Another study presented a customizable interactive web-based framework that
integrates various heterogeneous health related data streams including tweets,
news, reports etc. They specifically focus on HPV vaccine scare-related data
and visualize the results in the form of several graphs using this framework
[47]. Another study based on user-centric modeling presented a classifier to
identify users who are more prone to spread pseudo-medicinal information about
fake cures of cancer on Twitter. They characterized the users based on several
user attributes, linguistic styles, sentiments, and post timings [48].

Google search trends, search queries, and social media debates can be used to
analyze the public interests on a specific topic. This data can be
specifically helpful during international crises and epidemics. A few studies
tried to predict the spread of epidemics in a specific geographical location
by performing analysis on search engine queries and Google trends in that
region [49, 50, 51]. Search query analysis also showed that the public
immediately started searching about the pandemic but they started searching
for prevention and protection e.g. social distancing in the initial stages of
the COVID-19 pandemic [52]. Similarly, studies indicated that social media can
also be a useful way for early detection of epidemics [53] [54].

## Method

In this section we will present our methodology to understand the digital
information seeking behaviour of Pakistani users during the first wave
pandemic of COVID-19. Although digital information can take many forms like
text, images, and videos, we focused on analysing text data due to ease of
availability and simplicity of analysis. It is popular to study the spread
information or misinformation in text based digital data on online social
networking sites such as Twitter[55][56]. However, we took an alternative
route and chose the following two other sources of digital information:

  1. 1.

We studied how information was searched on Google search engine for COVID-19
related searches through Google Trends. We focused more on search queries that
are known to be pseudo-medicinal information on COVID-19. We then compared
these search trends with real world circumstances like changing government
SOPS, the number of cases, and number of deaths etc. This will help us
understand the information trajectory in Pakistan during the current COVID-19
crises

  2. 2.

WhatsApp is a popular way of communication in Pakistan. According to a mobile
ranking forum, WhatsApp is the second most used mobile application in the
country. Owing to the popularity of this communication medium, we identified a
public WhatsApp group focused on COVID-19. We then exported the conversations
in the form of textual data from that group. In an automated way, we replaced
all names and phone numbers with unique identifiers and we then stored the
data as a .CSV file with the following columns: Time and Date, User
Identifier, Text Message. We then analysed it to better understand the mood
and the content of these messages.

To better understand the vast volume of digital text data, we first needed to
define keywords so the data can be mined more easily. It was obvious to use
corona, covid etc as these keywords but it was not clear which pseudo-
medicinal treatment we should use as keywords. Next we will discuss how we
made our list of pseudo-medicinal treatments.

We shared with the participant’s important information about the study. There
were three stages in which this study handled data about humans. The first
stage was when we aimed to identify keywords for the Google Trends and public
WhatsApp group data. A written consent form and important information about
the study were shared with all participants who filled our online
questionnaire. Google Trends is publicly available and is aggregated trends of
millions of search queries hence no verbal or written consent was taken.
Similarly, the WhatsApp group was a public group, and data only anonymized
data was downloaded for analysis, so no consent was needed. The Departmental
Ethics Committee of COMSATS University, Islamabad reviewed the study protocol
and confirmed that this study adheres to required ethical standards. The
approval letter was numbered on 19-05-2020/CS/ETHICS.

Data of public WhatsApp groups has been used in social science research [57],
[58]. Bursztyn et. al suggest that users must be aware that their data will be
shared with other members once they participate in a group, public or not
[57]. When we started this work, we informed the ethics committee in detail
about the methodology and they agreed that only informing the group members
will suffice, as there is no objection from the platform’s end on saving
WhatsApp messages by the admin of the group. We communicated with the group
admin for permission to use their data, the admin then communicated with the
group members, and received no objection from any group member. The privacy
policy for WhatsApp groups states that a user share their messages and profile
information with other members of the group (https://www.whatsapp.com/legal/)
and they can also save and email upto 10,000 messages to anyone
(https://faq.whatsapp.com/en/android/23756533/). The data was then anonymized
using a script written in the language R.

### Pseudo-Medicinal Treatments

We observed that misinformation about the treatment of COVID-19 were
circulating on social media during the early months of COVID-19 in Pakistan.
The authors of this papers observed the following non-scientific treatments of
COVID-19 were popular on social media platforms during the first wave of the
pandemic in Pakistan:

  1. 1.

Herbs and Spices like garlic, ginger tea, lemon tea, olive leaf and senna
makki (a laxative). Albeit some of these are harmless but senna makki, being a
laxative, can cause dehydration and in extreme cases deaths in patients.

  2. 2.

Homeopathic drugs like Arsenicum album received publicity as a powerful
immunity booster and were thought to help in preventing COVID-19. While the
drug by itself may not have any serious side effects but people consider
themselves immune to COVID-19 and as consequence took lesser preventive
measures.

  3. 3.

Medicines like Chloroquine (also marketed with brand name Resochin) and
Hydroxychloroquine are antimalarial drugs that gained a lot of attention on
social and traditional media without any solid scientific backing and
sometimes can be dangerous. Further this medicine is also used for rheumatoid
arthritis patients and over-the counter availability of this drug created a
shortage in the market for the patient in need.

  4. 4.

Convalescent Plasma therapy was a popular treatment in Pakistan [59] and
worldwide including U.S. despite many concerns on its effectiveness and
possible adverse effects in some cases [60].

The above list of unproven treatments of COVID-19 can act as keywords for our
textual analysis. However, it is not clear if the general Pakistani public was
aware of these terms and hence used it for in their online search queries and
conversations. So we tried to confirm the suitability of using these keywords
through a small non-representative sample of users. We electronically sent our
consent form and questionnaire to potential participants and then proceeded
with only those who indicated their consent electronically. The questionnaire
lists misinformation on social media (as listed above) as keywords and asked
the participant to tick the boxes which they recognized. The targeted
population of this survey was the general Pakistani public and we tried to get
the representation from different areas of the country. A total of 40
participants (18 Males and 22 Females) filled this online survey. The
participants were geographically distributed in 19 locations around Pakistan
from residents of small towns to those living in big cities.

All home remedies identified by our research team were validated through this
survey. As shown in Figure 2 (a), the most popular home treatment was Senna
Makki, followed by Lemon Tea, Ginger Tea, and Garlic. Figure 2 (b) shows the
popularity of unproven medical treatments for COVID-19 in Pakistan, which
shows that a lot of interest was shown by internet users in plasma therapy as
a possible treatment for COVID-19 plasma therapy followed by
Hydroxychloroquine and Chloroquine.

Fig 2: Popularity of Unproven treatments identified on different social and
electronic media forums among Pakistani internet users. We can see that the
most popular home treatment was Senna Makki, followed by Lemon Tea, Ginger
Tea, and Garlic. On the other hand, plasma therapy was a very popular
medicinal treatment followed by Hydroxychloroquine and Chloroquine

## Analysis $1$: Search Trends During COVID-19 Pandemic

We performed a systematic analysis to evaluate the relationship between Google
search trends on various non-scientific treatments on pseudo-medicinal
information and the changing situation of COVID-19 pandemic in Pakistan. We
also tried to map the government initiatives to deal with the pandemic and
major events such as the first COVID-19 case and the first death. We also
evaluated their effects on Google trends on popular pseudo-medicinal
information topics in the country.

### Dataset

_Google Trends_ represents the popularity of a specific search term on Google
during a specific duration. For a specific search term, Google Trends shows
the daily relative popularity of the search term during this duration. It
returns a number $n$ ranging between $0$ and $100$, where $n=100$ on the day
when the search term was most popular and $n=0$ when it was least popular.
This allows the analysis of search interests of users in specific regions as
well as around the globe. It also provides the comparison of search trends on
multiple search terms by similarly normalizing them between $0$ and $100$.
This allows us to compare the relative popularity of multiple search terms,
giving us insight to public interests and concerns at a specific time.

We first performed a comparative analysis of Google Trends on different terms
that can be alternatively used for searching details about COVID-19 in
Pakistan. We choose $4$ generic terms that are commonly used in Pakistan:
COVID-19, Coronavirus, Covid, and Corona. As shown in Figure 3, all these
terms started to appear in Google Trends during the 3rd week of January 2020
and they all reached their peak in 3rd Week of March 2020. A sudden rise
appeared in search trends during the second Week of March as the government
implemented various spread control measures including nationwide lockdown. We
also calculated their mean popularity on Google Trends and in the remainder of
this paper, we will only use the mean of the various terms used to search for
Coronavirus as shown in Figure 3.

Fig 3: Comparison of Google Trends on Coronavirus search terms during pandemic
in Pakistan. The dashed line shows the mean popularity of COVID-19 related
searches.

We also checked whether the quick spread of COVID-19 in the region and the
rising number of infections and deaths resulted in increased searches in
COVID-19 by the public. Here our assumption is that the increased number of
searches indicate increased public concern about the pandemic. We explored
this by comparing the day to day statistical data of COVID-19 cases in
Pakistan with the popularity of COVID-19 searches using data from Google
Trends. We computed the correlation of search popularity of COVID-19 searches
with daily new infections and found that it has a value of just $0.27$.
Similarly, the correlation of popularity of COVID-19 searches with daily
deaths came out to be $0.23$. We can see that these are very small values,
indicating that the search popularity was insignificantly influenced by the
spread of the pandemic. Figure 4 (a) shows how search popularity change with
rising number of new infections and similarly Figure 4 (b) shows how search
popularity change with daily deaths. It is surprising as well concerning that
people seem to be searching very little even at the peak of the pandemic and
it seems that Google searches about COVID-19 was fuelled more because of the
initial total lack of knowledge about the virus and the resulting disease.

Fig 4: A comparison of how internet search statistics about COVID-19 changed
with changing number of infections and deaths.

### Search Interest Regarding Treatment and Prevention of COVID-19

We also performed a comparative analysis of Google trends in Pakistan on
different preventive measures suggested by WHO for COVID-19. We choose 5
popular terms that were commonly used in Pakistan: N95 mask, Sanitizer, Social
distancing, stay at home, disinfectant. Figure 5 shows the comparison and it
is clear that “Sanitizer” is clearly the most popular of these terms during
the first wave of the pandemic in Pakistan. We used the top trending term
“Sanitizer” for further analysis to evaluate the relationship between these
trends and peaks of trends on different pseudo-medicinal information search
terms.

Fig 5: Comparison of Google Trends on Preventive Measures during Pandemic in
Pakistan.

Figure 6 shows the pattern of how the popularity of the search term sanitizer
varies over time. We can see that this search term experienced a sudden rise
in popularity immediately after the first patient was detected, this was when
the whole country was experiencing previously unknown levels of fear and
uncertainty. At this time, many items essential for the pandemic like
sanitizers, masks, and other hygiene related items experienced a sudden
increase in demand. This was probably when the public was searching if
sanitizers were available online or if they are available at cheaper price.
However, very soon private businesses started to fill this newly created
demand and as a result the search term ”sanitizer” quickly dropped in
popularity However, it still remained more popular than pre-pandemic time.

Fig 6: Comparison of Google Trends on Sanitizer during March-August 2020.

We also investigated the Google Trends to check the popularity of
“Chloroquine” and “Hydroxychloroquine” in Pakistan after Trump’s endorsement
of these drugs and found a pattern very similar to that in the US. As
presented in Figure 7, there was a sudden hike in Google searches on both
medicinal terms in the last week of March, 2020 and they remain popular till
3rd week of April, 2020. This indicates that the backing of a popular
personality drives the public interests and increases the trustworthiness of a
piece of information.

Fig 7: Comparison of Google Trends on Chloroquine and Hydroxychloroquine in
Pakistan.

In summary, the results presented suggests that people in Pakistan actively
search COVID-19 related information during the early stages of the first wave
of the outbreak but later the public interest seemed to have waned.

## Analysis $2$: WhatsApp Public Group Data

Now, we present an analysis of data from a public WhatsApp group and also
explore the popularity of the pseudo-medicinal information in the discussions
during the first wave of COVID-19.

At the start of the pandemic, the general public used various social media
forums to share information about the newly emerging disease. WhatsApp was
among the most popular forums used during the pandemic to spread the
information and pseudo-medicinal information [4]. Hence, we thought it would
be beneficial to try to get a broad understanding of the discussions on
WhatsApp about COVID-19. We explored many public WhatsApp groups for
discussions about it COVID-19 and its pseudo-medicinal keywords mentioned in
previous sections. But, we found that the keywords were mentioned
insignificant number of times. Therefore, we decided to focus on only one
public WhatsApp group specifically created in response to the COVID-19
pandemic called Understanding COVID-19. The group was created to understand
the causes and potential cures of this novel disease and its members actively
discussed various issues about the pandemic, especially in the start of the
pandemic when limited information about this disease was available.

This particular group was created in the middle of March 2020, when the number
of daily infections and deaths started to grow. This group has $53$ members,
all of whom are Pakistani residents and most of them live in different major
cities of Pakistan. Moreover, almost all members were literate and with basic
healthcare knowledge and some of them were even medical doctors. We reached
this conclusion as many members of the group introduced themselves when the
group was created. Even those who didn’t introduce themselves had very good
English and their tone and language indicated that they are well educated.

The complete record of this WhatsApp group was exported till the 3rd week of
August as a CSV file for text analysis in R. Since most of the discussion on
this group was in English with Urdu being used only occasionally, so the text
analysis was done only on text messages in English. As a prepossessing step,
we performed the following data cleaning steps, so the data could be be more
meaningful.

  1. 1.

All characters in Urdu or any other language were removed and we only
considered ASCII characters for our text analysis. Hence all emojis were also
ignored. Note if a message was written in Roman Urdu, then it was included in
our analysis. The national language of Pakistan when transliterated in Roman
script is called as _Roman Urdu_ [61]. This is a common practice in Pakistan
due to difficulties in entering text on Urdu keyboards. .

  2. 2.

Any multimedia files shared on the group were ignored.

  3. 3.

Web links shared on the group were also ignored, however, if a user had copy
pasted text from a website than those were part of the analysis.

The time series visualized in Figure 8 represents the frequency of messages
per day and the points indicate important events during the first wave of the
COVID-19 pandemic in Pakistan. We can see in Figure 8, the group remains
mostly active between March and July 2020, the period when COVID-19 cases were
at its peak. An increase in the number of messages in a day can be seen after
two important events, the day the first patient died and the day when the
national lockdown was initiated.

Fig 8: A representation of message frequency per day in COVID-19 public group
with important events in Pakistan. An increase in the number of messages in a
day can be seen after two important events, the day the first patient died and
the day with the national lock.

We then wanted to get an idea of the discussion in this WhatsApp group. We
plotted a word cloud as it gives a good idea of the most frequent words used
and hence the discussions between the group members. We performed the below
pre-processing steps so our plot can be more meaningful.

  * •

We converted all text to lower case.

  * •

We removed all punctuation marks.

  * •

We then performed stemming i.e. reduced all words to their root form.

These pre-processing steps were completed through the stm package of R. We
then plotted the word cloud using the same package as shown in Figure 9. The
word cloud generated from the text from the WhatsApp group shows the most
commonly used words in their discussions. We can observe by looking at the
most common words that there is a lot of concern in its participants including
discussions on patients, clinics, hospitals, medics, and also various possible
treatments.

Fig 9: The word cloud generated from the text from the WhatsApp group showing
the most commonly used words in their discussions. We can observe that there
is a lot of concern in its participants including discussions on patients,
clinics, hospitals, medics, and also various possible treatments.

We also wanted to check whether the increasing number of COVID-19 infections
and deaths also resulted in increased discussion on the WhatsApp group,
indicating increased public interest and concern about the pandemic. For this
purpose, we first computed the correlation, which came out to be $0.0006$ for
daily new infections and $-0.0135$ for the daily new deaths respectively. It
indicates that the discussion was not correlated at all with how fast the
pandemic is spreading, in fact in case of new deaths it seems that in some
cases higher number of deaths seem to have resulted in lower number of
messages on the WhatsApp group. We hence compared how the time series of daily
number of messages exchanged on the group changes the time series for the
daily number of infections and daily number of deaths respectively. Figure10
(a) compares the number of messages per day with the number of new COVID-19
number of cases per day in Pakistan, while Figure 10 (b)compares it with daily
deaths due to COVID-19. This patterns shows the concern of the public at the
very start of the pandemic and as the number of infections and deaths
increased. However, very soon with the increasing number of infections and
deaths in Pakistan. It can be seen from both figures that that the group was
more active in earlier phases of the COVID-19 (i.e., March-July 2020) and
later the public seemed to be less concerned about the pandemic and possible
dangers from it. The $4$ peaks in the WhatsApp messages count seem to
influenced more by coverage in traditional news media and initial fear among
the public. More specifically, the first peak was around the time soon after
the government implement country wide complete lockdown, the second peak came
because there was a lot discussion about possible treatments and sharing
individual experiences; the 3rd peak came because of there was a lot of
discussion about psychological issues resulting from the pandemic and sharing
views on how to cope with this new reality; while the 4th peak in mid June was
because this was when the total number of infections in Pakistan crossed
$100,000$.

Fig 10: Comparison of WhatsApp group activity with daily reported COVID-19
numbers in Pakistan. Please note that the peaks in (b) look larger because the
number of deaths are significantly less than the number of infections.The $4$
peaks in the WhatsApp messages count seem to influenced more by coverage in
traditional news media and initial fear among the public.

We also explored communication related to the two anti-malarial drugs
Chloroquine and Hydroxychloroquine, and found that both started to be
discussed immediately after the endorsement of President Trump and Elon Musk.
They remain in regular discussion till July 2020 as can be seen in Figure 11.
This was despite of the fatality reported on March 22, 2020 [6].

Fig 11: Chloroquine and Hydroxychloroquine started being discussed immediately
after the endorsement of the high profile personalities and remained in
discussion till July 2020.

Lastly, we applied thematic analysis to extract main topics in the WhatsApp
group. We started by applying an algorithmic technique called Latent Dirichlet
Allocation but did not find the results to be satisfactory. As a result, we
followed the 6-step process suggested by Braun and Clarke for thematic
analysis [62]. We note here that _thematic analysis_ studies are suited for
studies that are of exploratory nature and to generate hypotheses that can be
later tested from a representative sample. After initial scanning of the
messages for getting a high-level idea of about the issues being discussed in
this group, the first author went through each message and identified 11
categories labeling each message accordingly. This labeling was done to
uncover the repeated patterns of behavior i.e. _themes_ in the data. The
second author then reviewed this categorization by going through the file and
reviewing $40\%$ messages and their associated labels and suggested merging
some categories and also recategorized some other messages. After this
recategorization, we ended up with $6$ categorizes. The categories and their
brief explanation is given below:

  1. 1.

General Conversation: General conversations messages such as hellos, goodbyes,
greetings, etc .

  2. 2.

Virus Information: Messages that sought and shared information about the SARS-
CoV-2 virus, its origin, how it spreads etc.

  3. 3.

COVID-19 Information: In this category, the participants discussed the disease
its risks, possible treatments and shared information that can be helpful to
the patients of COVID-19.

  4. 4.

Fake Information: In this category, the participants discussed various fake or
dubious treatments like plasma therapy. We also included messages that
discussed or shared various conspiracy theories about the origin of the virus
or the nature of the disease.

  5. 5.

Social Distancing: Messages that discussed the importance of social distancing
and masks etc. or expressed concern about the non-compliance of social
distancing protocols.

  6. 6.

Anxiety/Stress: Messages that discussed high levels of anxiety and stress due
to the pandemic were categorized into this category. This category also
included suggestions intended to help cope with these issues, including those
of religious nature.

The variation of these themes with time is shown in Figure 12. We did not
include the category of general conversation in the interest of clarity of the
figure. Moreover, the plot does not show two very high peaks, but their values
are written where the peak is truncated. We can see from the figure that
discussions in this group started as soon as it was created and the initial
discussions revolved around basic information about the virus, its origin, the
disease, and social distancing other preventive measures. In about a week’s
time, the discussion also started to have significant messages on fake
information and also anxiety/stress issues. A large number of messages were
exchanged for about 2 months till about May 2020 and when the volume of
messages in all themes dropped. However, all themes started being discussed
again with renewed interest in the middle of June, but the level of interest
was lesser than the initial weeks of the pandemic and it dropped to
insignificant levels within a few weeks. Recall that mid-June was the time
when the total number of infections in Pakistan crossed $100,000$.
Interestingly, we observed that all themes tend to be discussed more or less
together with most of the discussion being focused on either information about
the virus or information about the disease.

Fig 12: Themes observed in the discussion on the public WhatsApp group. We can
see that there was a lot of interest at the start of the pandemic and
gradually interest reduced. Interestingly, we observed that all themes tend to
be discussed more or less together with most of the discussion being focused
on either information about the virus or information about the disease.

We plotted the trend of how discussions on fake and pseudo-medicinal remedies
identified in Section Method varied with time. We can see from Figure 13 that
this discussion seems to be bursty and the peaks seem to fall very close to
the time with the peak of pseudo-medicinal treatment on Google Trends. This
indicates that both the WhatsApp discussion as well as the search peak seem to
be related.

Fig 13: The trend of how various fake information was discussed on the
WhatsApp group. In this figure, the days when the search peak on Google Trends
was observed is labeled. Interestingly, we can see that the search peak
corresponds very near to a peak in the WhatsApp discussion about some fake
information.

## Conclusion

In this paper, we investigated the information-seeking behaviour of the
Pakistani public during the first wave of the COVID-19 pandemic in Pakistan.
We decided to focus on online resources and social media as they were a major
source of health-related information during the COVID-19 pandemic. The major
contributions of this paper are the following:

  1. 1.

We investigated how the public searched the web for various COVID-19 related
information as the pandemic progressed. We used data from Google Trends for
this purpose. Interestingly, our analysis seems to indicate that although the
number of infections and number of deaths due to COVID-19 was increasing, the
general public was searching lesser for COVID-19 related information. Analysis
of search trend of COVID-19 related treatments and prevention also indicated
that search volume seems to be influenced more by external factors like Donald
Trump and Elon Musk endorsing a drug; sanitizer not being available in the
market, or the psychological effect and media coverage when the number of
patients crossed $100,000$.

  2. 2.

For a high-level semantic understanding about the information sought during
the first wave, we analyzed data from a public WhatsApp group that was created
to share information about COVID-19. Similar to search trends, the group
members shared more information during the early weeks of the pandemic and
gradually the number of messages decreased despite the pandemic becoming more
widespread. Like search trends, discussion in this group seemed to be sparked
by external factors and associated media coverage. For example, a sudden
increase in the volume of messages was observed when the COVID-19 patient
died; when government-imposed countrywide complete lockdown; and when the
number of infected persons crossed the $100,000$ threshold.

The results discussed in Section Analysis $1$: Search Trends During COVID-19
Pandemic and Section Analysis $2$: WhatsApp Public Group Data seem to indicate
that the public was very concerned at the start of the pandemic, however with
time their level of concern gradually reduced. Their level of concern and
interest then seem to rise occasionally when an event that they perceived as
occurred. This seems to indicate habituation, here _habituation_ is
psychological behavior found in all living things and it informally means the
reduction of a particular response after repeated exposure of the same
stimulus [63]. A number of studies have been conducted to establish the
causality of repeated exposure of the same stimulus and resulting decrease in
response[64, 65]. In case our paper, the stimulus is the daily pandemic-
related news that everyone heard or read at various places and the response
was their level of concern and resulting online behavior that we tried to
observe. Habituation is also found to be _stimulus specific_ [66], hence the
response can be returned to previous levels when an individual who is already
habituated to one stimulus is presented with a novel stimulus [67].

We hence conjecture that in the case of the COVID-19 pandemic, the public
habituated after daily exposure of warnings and worrying news on traditional
media as well as on social media. They were, however, ‘dishabituated’ when the
stimulus changed, for example when news of the number of infections crossing
$100,000$. We check this conjecture by plotting the number of messages
exchanged on the WhatsApp group in the second wave as shown in Figure 14. Note
this plot only shows till December 2020 as the WhatsApp group was deactivated
soon after it. We can see that very few messages were exchanged, however, the
number of messages on the group experienced sudden and short peaks. On further
investigating, they all seem to be a result of something unusual or different,
explained as follows :

  1. 1.

First Peak: (September 04, 2020) At this time, the public was concerned about
the likely effect of government’s decision to open schools.

  2. 2.

Second Peak: (October 06, 2020) A famous newspaper Dawn publishes a gloomy
article[68] about second wave and how deadly it can be. There was a lot of
discussion about this article.

  3. 3.

Third Peak:  (November 04, 2020) There was discussion about the news[69] that
the government ruling out lockdown again.

  4. 4.

Fourth Peak: (December 04, 2020) Clinical trials of the COVID-19 vaccine
starts in Pakistan and it was discussed a lot in the group.

Fig 14: Number of Messages on the Whatsapp group in the second wave compared
to number of infections.

This same pattern can be seen for the search results data from Google Trends
as can be seen in Figure 15 that the keywords chosen in Section Analysis $1$:
Search Trends During COVID-19 Pandemic were very high from March 2020 till
about July 2020 probably due to initial scare and confusion. After the initial
scare at the height of the first wave they dropped to almost pre-pandemic
times. Hence supporting our conjecture that public’s habituation to COVID-19
after an initial scare and lack of information.

Fig 15: Figure showing Google Trends data of keywords selected in Section
Analysis $1$: Search Trends During COVID-19 Pandemic were only very high from
March 2020 till about July 2020 probably due to initial scare and lack of
information.

However, more work is needed to confirm whether there is a causal relationship
between online activity and external factors. It also needs to be investigated
how governments and health agencies should communicate with the general public
so habituation can either be avoided or reduced in future pandemics. Most
importantly there is a need for social media and communication applications
companies (such as Facebook) and search engines (such as Google), to control
the spread of misinformation, especially for the emergent user communities.

## References

  * 1. Wolf MS, Serper M, Opsasnick L, O’Conor RM, Curtis LM, Benavente JY, et al.  Awareness, attitudes, and actions related to COVID-19 among adults with chronic conditions at the onset of the US outbreak: a cross-sectional survey.  Annals of internal medicine. 2020;.
  * 2. Blandizzi C, Scarpignato C.  Gastrointestinal drugs.  In: Side effects of drugs annual. vol. 33. Elsevier; 2011. p. 741–767.
  * 3. Organization WH, et al.  Strengthening response to pandemics and other public-health emergencies: report of the review committee on the functioning of the International Health Regulations (2005) and on pandemic influenza (H1N1) 2009\.  World Health Organization; 2011.
  * 4. Moreno Á, Fuentes Lara CM, Navarro C, et al.  Covid-19 communication management in Spain: Exploring the effect of information-seeking behavior and message reception in public’s evaluation.  El profesional de la información. 2020;.
  * 5. Joseph A. CDC: Some Americans are misusing cleaning products — including drinking them — in effort to kill coronavirus; 2020 [Online].  https://www.statnews.com/2020/06/05/cdc-misusing-bleach-try-kill-coronavirus/.
  * 6. Liu M, Caputi TL, Dredze M, Kesselheim AS, Ayers JW.  Internet searches for unproven COVID-19 therapies in the United States.  JAMA Internal Medicine. 2020;.
  * 7. Coronavirus Attitude Tracker Survey Report – Wave 8; 2020 [Online].  Available from: https://gallup.com.pk/wp/wp-content/uploads/2020/10/Gallup-Pakistan-Coronavirus-Attitude-Tracker-Survey-Wave-8-.pdf.
  * 8. Pennycook G, McPhetres J, Zhang Y, Lu JG, Rand DG.  Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy-nudge intervention.  Psychological science. 2020;31(7):770–780.
  * 9. Ashrafi-rizi H, Kazempour Z.  Information diet in covid-19 crisis; a commentary.  Archives of Academic Emergency Medicine. 2020;8(1).
  * 10. Organization WH, et al.  Strengthening response to pandemics and other public-health emergencies: report of the review committee on the functioning of the International Health Regulations (2005) and on pandemic influenza (H1N1) 2009\.  World Health Organization; 2011.
  * 11. Genuis SK.  Constructing “sense” from evolving health information: A qualitative investigation of information seeking and sense making across sources.  Journal of the American Society for Information Science and Technology. 2012;63(8):1553–1566.
  * 12. Genuis SK.  Exploring the role of medical and consumer literature in the diffusion of information related to hormone therapy for menopausal women.  Journal of the American Society for Information Science and Technology. 2006;57(7):974–988.
  * 13. Sun Y, Zhang Y, Gwizdka J, Trace CB.  Consumer evaluation of the quality of online health information: Systematic literature review of relevant criteria and indicators.  Journal of medical Internet research. 2019;21(5):e12522.
  * 14. Boyer C, Selby M, Scherrer JR, Appel R.  The health on the net code of conduct for medical and health websites.  Computers in biology and medicine. 1998;28(5):603–610.
  * 15. Silberg WM, Lundberg GD, Musacchio RA.  Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor–Let the reader and viewer beware.  Jama. 1997;277(15):1244–1245.
  * 16. Rees CE, Ford JE, Sheard CE.  Evaluating the reliability of DISCERN: a tool for assessing the quality of written patient information on treatment choices.  Patient education and counseling. 2002;47(3):273–275.
  * 17. Cuan-Baltazar JY, Muñoz-Perez MJ, Robledo-Vega C, Pérez-Zepeda MF, Soto-Vega E.  Misinformation of COVID-19 on the internet: infodemiology study.  JMIR public health and surveillance. 2020;6(2):e18444.
  * 18. Kyriacou A, Sherratt C.  Online health information–seeking behavior by endocrinology patients.  Hormones. 2019;18(4):495–505.
  * 19. Chaudhri MS, LE MT, White MC, Thompson H, Demiris G.  Examining health information–seeking behaviors of older adults.  Computers, informatics, nursing: CIN. 2013;31(11):547.
  * 20. Gray NJ, Klein JD, Noyce PR, Sesselberg TS, Cantrill JA.  Health information-seeking behaviour in adolescence: the place of the internet.  Social science & medicine. 2005;60(7):1467–1478.
  * 21. Miller LMS, Bell RA.  Online health information seeking: the influence of age, information trustworthiness, and search challenges.  Journal of aging and health. 2012;24(3):525–541.
  * 22. McPherson AC, Gofine ML, Stinson J.  Seeing is believing? A mixed-methods study exploring the quality and perceived trustworthiness of online information about chronic conditions aimed at children and young people.  Health Communication. 2014;29(5):473–482.
  * 23. Carissa Bonner BN, Pickels K. Young men are more likely to believe COVID-19 myths. So how do we actually reach them?; 2020. [Online].  https://theconversation.com/young-men-are-more-likely-to-believe-covid-19-myths-so-how-do-we-actually-reach-them-143745/.
  * 24. Hu D, Lou X, Xu Z, Meng N, Xie Q, Zhang M, et al.  More effective strategies are required to strengthen public awareness of COVID-19: Evidence from Google Trends.  Journal of Global Health. 2020;10(1).
  * 25. Walker A, Hopkins C, Surda P.  The use of google trends to investigate the loss of smell related searches during COVID-19 outbreak.  In: International Forum of Allergy & Rhinology. Wiley Online Library; 2020.
  * 26. Kurian SJ, Alvi MA, Ting HH, Storlie C, Wilson PM, Shah ND, et al.  Correlations Between COVID-19 Cases and Google Trends Data in the United States: A State-by-State Analysis.  In: Mayo Clinic Proceedings. Elsevier; 2020. p. 2370–2381.
  * 27. Husnayain A, Fuad A, Su ECY.  Applications of google search trends for risk communication in infectious disease management: A case study of COVID-19 outbreak in Taiwan.  International Journal of Infectious Diseases. 2020;.
  * 28. Ayyoubzadeh SM, Ayyoubzadeh SM, Zahedi H, Ahmadi M, Kalhori SRN.  Predicting COVID-19 incidence through analysis of google trends data in iran: data mining and deep learning pilot study.  JMIR Public Health and Surveillance. 2020;6(2):e18828.
  * 29. Rovetta A, Bhagavathula AS.  Covid-19-related web search behaviors and infodemic attitudes in italy: Infodemiological study.  JMIR public health and surveillance. 2020;6(2):e19374.
  * 30. Badell-Grau RA, Cuff JP, Kelly BP, Waller-Evans H, Lloyd-Evans E.  Investigating the Prevalence of Reactive Online Searching in the COVID-19 Pandemic: Infoveillance Study.  Journal of medical Internet research. 2020;22(10):e19791.
  * 31. Rathore FA, Farooq F.  Information Overload and Infodemic in the COVID-19 Pandemic.  JPMA The Journal of the Pakistan Medical Association. 2020;70(5):S162–S165.
  * 32. Hernández-García I, Giménez-Júlvez T.  Assessment of health information about COVID-19 prevention on the internet: infodemiological study.  JMIR public health and surveillance. 2020;6(2):e18717.
  * 33. Gupta L, Gasparyan AY, Misra DP, Agarwal V, Zimba O, Yessirkepov M.  Information and misinformation on COVID-19: a cross-sectional survey study.  Journal of Korean medical science. 2020;35(27).
  * 34. Post S, Bienzeisler N, Lohöfener M.  A desire for authoritative science? How citizens’ informational needs and epistemic beliefs shaped their views of science, news, and policymaking in the COVID-19 pandemic.  Public Understanding of Science. 0;0(0):09636625211005334.  doi:10.1177/09636625211005334.
  * 35. Chou WYS, Oh A, Klein WM.  Addressing health-related misinformation on social media.  Jama. 2018;320(23):2417–2418.
  * 36. Datta SS, O’Connor PM, Jankovic D, Muscat M, Mamou MCB, Singh S, et al.  Progress and challenges in measles and rubella elimination in the WHO European Region.  Vaccine. 2018;36(36):5408–5415.
  * 37. Filia A, Bella A, Del Manso M, Baggieri M, Magurano F, Rota MC.  Ongoing outbreak with well over 4,000 measles cases in Italy from January to end August 2017- what is making elimination so difficult?  Eurosurveillance. 2017;22(37):30614.
  * 38. Chen L, Wang X, Peng TQ.  Nature and diffusion of gynecologic cancer–related misinformation on social media: Analysis of tweets.  Journal of Medical Internet Research. 2018;20(10):e11515.
  * 39. Waszak PM, Kasprzycka-Waszak W, Kubanek A.  The spread of medical fake news in social media–the pilot quantitative study.  Health policy and technology. 2018;7(2):115–118.
  * 40. Denworth L. Overcoming Psychological Biases Is the Best Treatment against COVID-19 Yet; 2020 [Online].  https://www.scientificamerican.com/article/overcoming-psychological-biases-is-the-best-treatment-against-covid-19-yet/.
  * 41. Shah M. The Failure of Public Health Messaging about COVID-19; 2020 [Online].  https://www.scientificamerican.com/article/the-failure-of-public-health-messaging-about-covid-19/.
  * 42. Moyer MW. People Drawn to Conspiracy Theories Share a Cluster of Psychological Features; 2019 [Online].  https://www.scientificamerican.com/article/people-drawn-to-conspiracy-theories-share-a-cluster-of-psychological-features/.
  * 43. Depoux A, Martin S, Karafillakis E, Preet R, Wilder-Smith A, Larson H. The pandemic of social media panic travels faster than the COVID-19 outbreak; 2020\.
  * 44. Wang C, Pan R, Wan X, Tan Y, Xu L, Ho CS, et al.  Immediate psychological responses and associated factors during the initial stage of the 2019 coronavirus disease (COVID-19) epidemic among the general population in China.  International journal of environmental research and public health. 2020;17(5):1729.
  * 45. Ghenai A, Mejova Y.  Catching Zika fever: Application of crowdsourcing and machine learning for tracking health misinformation on Twitter.  arXiv preprint arXiv:170703778. 2017;.
  * 46. Kostkova P, Mano V, Larson HJ, Schulz WS.  Who is Spreading Rumours about Vaccines? Influential user impact modelling in social networks.  In: Proceedings of the 2017 international conference on digital health; 2017. p. 48–52.
  * 47. Kostkova P, Mano V, Larson HJ, Schulz WS.  Vac medi+ board: Analysing vaccine rumours in news and social media.  In: Proceedings of the 6th International Conference on Digital Health Conference; 2016. p. 163–164.
  * 48. Ghenai A, Mejova Y.  Fake cures: user-centric modeling of health misinformation in social media.  Proceedings of the ACM on human-computer interaction. 2018;2(CSCW):1–20.
  * 49. Ginsberg J, Mohebbi MH, Patel RS, Brammer L, Smolinski MS, Brilliant L.  Detecting influenza epidemics using search engine query data.  Nature. 2009;457(7232):1012–1014.
  * 50. Polgreen PM, Chen Y, Pennock DM, Nelson FD, Weinstein RA.  Using internet searches for influenza surveillance.  Clinical infectious diseases. 2008;47(11):1443–1448.
  * 51. Teng Y, Bi D, Xie G, Jin Y, Huang Y, Lin B, et al.  Dynamic forecasting of Zika epidemics using Google trends.  PloS one. 2017;12(1):e0165085.
  * 52. Bento AI, Nguyen T, Wing C, Lozano-Rojas F, Ahn YY, Simon K.  Evidence from internet search data shows information-seeking responses to news of local COVID-19 cases.  Proceedings of the National Academy of Sciences. 2020;117(21):11220–11222.
  * 53. Kim KD, Hossain L.  Towards Early Detection of Influenza Epidemics by Using Social Media Analytics.  In: DSS; 2014. p. 36–41.
  * 54. Thapen N, Simmie D, Hankin C, Gillard J.  DEFENDER: detecting and forecasting epidemics using novel data-analytics for enhanced response.  PloS one. 2016;11(5):e0155417.
  * 55. Sharma K, Seo S, Meng C, Rambhatla S, Dua A, Liu Y.  Coronavirus on social media: Analyzing misinformation in Twitter conversations.  arXiv preprint arXiv:200312309. 2020;.
  * 56. Kušen E, Strembeck M.  Politics, sentiments, and misinformation: An analysis of the Twitter discussion on the 2016 Austrian Presidential Elections.  Online Social Networks and Media. 2018;5:37–50.
  * 57. VS Bursztyn LB.  Thousands of small, constant rallies: A large-scale analysis of partisan WhatsApp groups.  IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). 2019;.
  * 58. K Garimella GT.  Whatapp doc? a first look at whatsapp public group data.  twelfth International AAAI Conference on Web and Social Media. 2018;.
  * 59. khan S. Coronavirus in Pakistan: Blood plasma in high demand to cure COVID-19 patients; 2020. [Online].  https://www.dw.com/en/coronavirus-pakistan/a-54016553.
  * 60. Malani AN, Sherbeck JP, Malani PN.  Convalescent plasma and COVID-19.  Jama. 2020;324(5):524.
  * 61. Bilal A, Rextin A, Kakakhel A, Nasim M.  Roman-txt: forms and functions of roman urdu texting.  In: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services; 2017. p. 1–9.
  * 62. Braun V, Clarke V.  Using thematic analysis in psychology.  Qualitative research in psychology. 2006;3(2):77–101.
  * 63. Gluck MA, Mercado E, Myers CE.  Learning and memory: From brain to behavior.  Worth Publishers New York; 2008.
  * 64. Davis M.  Habituation and sensitization of a startle-like response elicited by electrical stimulation at different points in the acoustic startle circuit.  In: Sensory Functions. Elsevier; 1981. p. 67–78.
  * 65. Malcuit G, Bastien C, Pomerleau A.  Habituation of the orienting response to stimuli of different functional values in 4-month-old infants.  Journal of Experimental Child Psychology. 1996;62(2):272–291.
  * 66. Thompson RF, Spencer WA.  Habituation: a model phenomenon for the study of neuronal substrates of behavior.  Psychological review. 1966;73(1):16.
  * 67. Dewsbury DA.  Effects of novelty of copulatory behavior: The Coolidge effect and related phenomena.  Psychological Bulletin. 1981;89(3):464.
  * 68. Second Wave; 2020 [Online].  https://www.dawn.com/news/1583507/second-wave.
  * 69. NCC rules out comlete lockdown; 2020 [Online].  https://www.dawn.com/news/1588511.


Instructors appeared to be rarely aware of these experiences in their
classroom or addressed such issues superficially when informed about them so
that they persisted. Also, even if they communicated to their students that
they could choose to switch partners, the women still felt like the onus was
on them to bring up negative interactions with their peers to their
instructor’s attention.

We also saw that several women reflected on various aspects of their
identities that affected their experiences negatively, which may be useful for
faculty to consider. We encourage instructors to continue to contemplate upon
and consider various intersections of students’ identity and how those may
impact students in physics learning environments since students are not a
monolith and have a wide range of backgrounds and experiences. Our findings
could be useful useful for faculty professional develop workshop leaders in
educating and informing instructors on how to better support diverse groups of
students.

In our interviews, We also asked these women about things that would help
improve their experiences. Some of them provided advice based upon experiences
that have personally helped them. However, we note that their advice may not
be suitable for all students and some of them may put the onus on the
marginalized students to change in order to improve their experiences, rather
than focusing on the systemic issues that caused those negative experiences
and the need for the physics culture to change to support them. Moreover,
students who are marginalized in physics due to multiple marginalized
identities, such as Paulina, may not benefit from simply being advised to
change their mindset about their struggle unless the physics culture is
improved to support them as some of the white women in this study suggested
has helped them.

These women also had many suggestions for instructors, many of which were
about ways to change the structure of the classrooms. Others were more focused
on the responsibility of the instructor. We encourage physics faculty to
consider these suggestions as many of them do not require monetary resources,
but require serious reflection and time commitment on their part. In other
words, these suggestions are not impossible to implement, but may require
planning and collaboration amongst faculty and other experts in positions of
power. Faculty also do not need to be physics education researchers to carry
out these changes; instead, they must be committed towards equity and
inclusion.

With regard to the domains of power and HELPIEE frameworks, the power in
physics learning environments directly refers to the fact that physics faculty
and other individuals can make the learning environments equitable and
inclusive and take action and educate those who violate the equitable and
inclusive norms. Enforcing rules, such as those described by a code of
conduct, and even maintaining them through the repetitive use of discussions,
training and disciplinary action can potentially be useful. However, at this
small college where we conducted the interviews, from the perspective of these
undergraduate women, the instructors were not aware of students not treating
other students equally or if these issues were pointed out, they addressed
these issues superficially in class as opposed to in a way that would make
sustained difference. Thus, their actions in the disciplinary domain, if any,
were relatively minor. The cultural domain in the domains of power framework
is in reference to the physics culture, e.g., how members of the physics
community think about physics, and how those beliefs are reinforced by other
members. According to these undergraduate women, the physics culture at this
college is much like a prototypical physics culture, e.g., where doing physics
is associated with brilliance.

As discussed in the preceding section, if we were to rank this physics
department relative to the other two mentioned here [74, 23] in terms of the
physics culture, it would be somewhere in the middle. However, at a small
liberal arts institution, the culture may be easier to change because there
are fewer faculty who have to be on the same page, whereas at a large
university or a fragmented department, it would be more difficult to dismantle
an established physics culture with, for example, 40+ physics faculty [23]. In
such a large physics department with inequitable physics culture, there would
need to be a “critical” number of faculty who are committed towards equity and
inclusion in a physics department, who could work to change the physics
culture [23]. With enough faculty support and resources, we believe that there
can be a sustained and systemic effort in physics departments of any size and
demographic distribution to create equitable and inclusive learning
environments and support traditionally marginalized students (although some
efforts could require more effort than others).

In summary, through the experiences of undergraduate women presented here, we
illustrated how they navigated and perceived the physics culture and engaged
in doing physics. How power interacts in domains such as interpersonal,
cultural, and disciplinary is especially important as they are entangled with
each other. Because physics instructors and other administrators have large
roles, influence, and power, according to the HELPIEE framework, they should
take action, rather than expecting students to change or adapt to the physics
culture. These actions should focus on traditionally marginalized students,
e.g., they can focus on implementing interventions or workshops to dismantle
stereotypes in the classroom. There is also need for systemic institutional
changes. However, that would require more faculty and administrative support.
In summary, we encourage physics faculty to take these findings and
suggestions seriously and use their positions of power to implement positive
changes and create a supportive environment for traditionally marginalized
students.

###### Acknowledgements.

We are very grateful to the faculty member who helped us recruit participants
for this study and to all of the participants who volunteered for this study.

## References

  * Mulvey and Nicholson [2020] P. J. Mulvey and S. Nicholson, Physics Bachelor’s Degrees: 2018, retrieved from: https://www.aip.org/statistics/reports/physics-bachelors-degrees-2018 (2020).
  * AIP [2021] Percent of Physics Bachelor’s Earned by Women, Classes of 1976 to 2020, retrieved from: https://www.aip.org/statistics/data-graphics/percent-physics-bachelor’s-earned-women-classes-1976-2020 (2021).
  * APS [2020] Bachelor’s Degrees in Physics and STEM Earned by Women, retrieved from: http://www.aps.org/programs/education/statistics/womenstem.cfm (2020).
  * AIP [2020] The Proportion of Physics Bachelor’s Degrees Awarded to African Americans and Hispanic Americans, Classes 1994 to 2020, retrieved from: https://www.aip.org/statistics/data-graphics/proportion-physics-bachelor’s-degrees-awarded-african-americans-and (2020).
  * Boatman [2022] L. Boatman, Newest Data Shows Mixed Progress for Women and Marginalized Groups in Physics Higher Ed, retrieved from: https://www.aps.org/publications/apsnews/202211/newest-data.cfm (2022).
  * Seymour and Hewitt [1997] E. Seymour and N. M. Hewitt,  _Talking about Leaving: Why Undergraduates Leave the Sciences_ (Westview Press, Boulder, CO, 1997).
  * Seymour and Hunter [2019] E. Seymour and A.-B. Hunter,  _Talking about Leaving Revisited: Persistence, Relocation, and Loss in Undergraduate STEM Education_ (Springer, 2019).
  * Pollack [2016] E. Pollack,  _The Only Woman in the Room: Why Science Is Still a Boys’ Club_ (The Beacon Press, 2016).
  * Hazari  _et al._ [2007] Z. Hazari, R. H. Tai, and P. M. Sadler, Gender differences in introductory university physics performance: The influence of high school physics preparation and affective factors, Science Education 91, 847 (2007).
  * Sax  _et al._ [2016] L. J. Sax, K. J. Lehman, R. S. Barthelemy, and G. Lim, Women in physics: A comparison to science, technology, engineering, and math education over four decades, Physical Review Physics Education Research 12, 020108 (2016).
  * Potvin  _et al._ [2014] G. Potvin, Z. Hazari, and R. M. Lock, Exposure to underrepresentation discussion: The impacts on women’s attitudes and identities, in  _Proceedings of the Physics Education Research Conference (PERC)_ (2014) pp. 211–214.
  * Hazari  _et al._ [2020] Z. Hazari, D. Chari, G. Potvin, and E. Brewe, The context dependence of physics identity: Examining the role of performance/competence, recognition, interest, and sense of belonging for lower and upper female physics undergraduates, Journal of Research in Science Teaching 57, 1583 (2020).
  * Lorenzo  _et al._ [2006] M. Lorenzo, C. H. Crouch, and E. Mazur, Reducing the gender gap in the physics classroom, American Journal of Physics 74, 118 (2006).
  * Cheryan  _et al._ [2017] S. Cheryan, S. A. Ziegler, A. K. Montoya, and L. Jiang, Why are some STEM fields more gender balanced than others?, Psychological Bulletin 143, 1 (2017).
  * Jones  _et al._ [2000] M. G. Jones, A. Howe, and M. J. Rua, Gender differences in students’ experiences, interests, and attitudes toward science and scientists, Science Education 84, 180 (2000).
  * Hill  _et al._ [2010] C. Hill, C. Corbett, and A. St. Rose,  _Why So Few? Women in Science, Technology, Engineering, and Mathematics_ (ERIC, 2010).
  * Ganley  _et al._ [2018] C. M. Ganley, C. E. George, J. R. Cimpian, and M. B. Makowski, Gender equity in college majors: Looking beyond the STEM/Non-STEM dichotomy for answers regarding female participation, American Educational Research Journal 55, 453 (2018).
  * McCullough [2011] L. McCullough, Women’s leadership in science, technology, engineering and mathematics: Barriers to participation, in  _Forum on Public Policy Online_ , 2 (ERIC, 2011) pp. 1–11.
  * Barthelemy  _et al._ [2016] R. S. Barthelemy, M. McCormick, and C. Henderson, Gender discrimination in physics and astronomy: Graduate student experiences of sexism and gender microaggressions, Phys. Rev. Phys. Educ. Res. 12, 020119 (2016).
  * Traxler  _et al._ [2016] A. L. Traxler, X. C. Cid, J. Blue, and R. Barthelemy, Enriching gender in physics education research: A binary past and a complex future, Physical Review Physics Education Research 12, 020114 (2016).
  * Rosa and Mensah [2016] K. Rosa and F. M. Mensah, Educational pathways of black women physicists: Stories of experiencing and overcoming obstacles in life, Physical Review Physics Education Research 12, 020113 (2016).
  * Santana and Singh [2022] L. M. Santana and C. Singh, Investigating experiences of a black woman in physics and astronomy, in  _Proceedings of the Physics Education Research Conference (PERC_ (2022) pp. 419–424.
  * Santana and Singh [2023] L. M. Santana and C. Singh, Effects of male-dominated physics culture on undergraduate women, Physical Review Physics Education Research 19, 020134 (2023).
  * Leslie  _et al._ [2015] S.-J. Leslie, A. Cimpian, M. Meyer, and E. Freeland, Expectations of brilliance underlie gender distributions across academic disciplines, Science 347, 262 (2015).
  * Santos [2017] A. C. F. Santos, You don’t look like a physicist, The Physics Teacher 55, 524 (2017).
  * Marchand and Taasoobshirazi [2013] G. C. Marchand and G. Taasoobshirazi, Stereotype threat and women’s performance in physics, International Journal of Science Education 35, 3050 (2013).
  * Cwik and Singh [2021a] S. Cwik and C. Singh, Damage caused by societal stereotypes: Women have lower physics self-efficacy controlling for grade even in courses in which they outnumber men, Physical Review Physics Education Research 17, 020138 (2021a).
  * Maries  _et al._ [2018] A. Maries, N. I. Karim, and C. Singh, Is agreeing with a gender stereotype correlated with the performance of female students in introductory physics?, Phys. Rev. Phys. Educ. 14, 020119 (2018).
  * Hazari  _et al._ [2010] Z. Hazari, G. Sonnert, P. M. Sadler, and M.-C. Shanahan, Connecting high school physics experiences, outcome expectations, physics identity, and physics career choice: A gender study, J. Res. Sci. Teach. 47, 978 (2010).
  * Kalender  _et al._ [2020] Z. Y. Kalender, E. Marshman, C. D. Schunn, T. J. Nokes-Malach, and C. Singh, Damage caused by women’s lower self-efficacy on physics learning, Phys. Rev. Phys. Educ. Res. 16, 010118 (2020).
  * Kalender  _et al._ [2019a] Z. Y. Kalender, E. Marshman, C. D. Schunn, T. J. Nokes-Malach, and C. Singh, Why female science, technology, engineering, and mathematics majors do not identify with physics: They do not think others see them that way, Phys. Rev. Phys. Educ. Res. 15, 020148 (2019a).
  * Nissen and Shemwell [2016] J. M. Nissen and J. T. Shemwell, Gender, experience, and self-efficacy in introductory physics, Phys. Rev. Phys. Educ. Res. 12, 020105 (2016).
  * Zeldin  _et al._ [2008] A. L. Zeldin, S. L. Britner, and F. Pajares, A comparative study of the self-efficacy beliefs of successful men and women in mathematics, science, and technology careers, J. Res. Sci. Teach. 45, 1036 (2008).
  * Lock  _et al._ [2015] R. M. Lock, J. Castillo, Z. Hazari, and G. Potvin, Determining strategies that predict physics identity: Emphasizing recognition and interest, in  _Physics Education Research Conference Proceedings_ (2015) pp. 199–202.
  * Cwik and Singh [2022] S. Cwik and C. Singh, Not feeling recognized as a physics person by instructors and teaching assistants is correlated with female students’ lower grades, Physical Review Physics Education Research 18, 010138 (2022).
  * Li and Singh [2023] Y. Li and C. Singh, Impact of perceived recognition by physics instructors on women’s self-efficacy and interest, Physical Review Phys. Educ. Res. 19, 020125 (2023).
  * Cwik and Singh [2021b] S. Cwik and C. Singh, How perception of learning environment predicts male and female students’ grades and motivational outcomes in algebra-based introductory physics courses, Physical Review Physics Education Research 17, 020143 (2021b).
  * Sawtelle  _et al._ [2012] V. Sawtelle, E. Brewe, and L. H. Kramer, Exploring the relationship between self-efficacy and retention in introductory physics, J. Res. Sci. Teach. 49, 1096 (2012).
  * Good  _et al._ [2012] C. Good, A. Rattan, and C. S. Dweck, Why do women opt out? Sense of belonging and women’s representation in mathematics, Journal of Personality and Social Psychology 102, 700 (2012).
  * Li  _et al._ [2020] Y. Li, K. Whitcomb, and C. Singh, How perception of being recognized or not recognized by instructors as a “physics person” impacts male and female students’ self-efficacy and performance, Phys. Teach. 58, 484 (2020).
  * Li and Singh [2021] Y. Li and C. Singh, Effect of gender, self-efficacy, and interest on perception of the learning environment and outcomes in calculus-based introductory physics courses, Physical Review Physics Education Research 17, 010143 (2021).
  * Hazari  _et al._ [2015] Z. Hazari, C. Cass, and C. Beattie, Obscuring power structures in the physics classroom: Linking teacher positioning, student engagement, and physics identity development, Journal of Research in Science Teaching 52, 735 (2015).
  * Maries  _et al._ [2022] A. Maries, K. Whitcomb, and C. Singh, Gender inequities throughout STEM, Journal of College Science Teaching 51 (2022).
  * Kalender  _et al._ [2019b] Z. Y. Kalender, E. Marshman, C. D. Schunn, T. J. Nokes-Malach, and C. Singh, Gendered patterns in the construction of physics identity from motivational factors, Phys. Rev. Phys. Educ. Res. 15, 020119 (2019b).
  * Lewis  _et al._ [2016] K. L. Lewis, J. G. Stout, S. J. Pollock, N. D. Finkelstein, and T. A. Ito, Fitting in or opting out: A review of key social-psychological factors influencing a sense of belonging for women in physics, Physical Review Physics Education Research 12, 020110 (2016).
  * Doucette  _et al._ [2020] D. Doucette, R. Clark, and C. Singh, Hermione and the Secretary: How gendered task division in introductory physics labs can disrupt equitable learning, European Journal of Physics 41, 035702 (2020).
  * Bandura [1997] A. Bandura,  _Self-Efficacy: The Exercise of Control_ (W. H. Freeman, New York, 1997).
  * Masika and Jones [2016] R. Masika and J. Jones, Building student belonging and engagement: Insights into higher education students’ experiences of participating and learning together, Teach. High. Educ. 21, 138 (2016).
  * Goodenow [1993] C. Goodenow, Classroom belonging among early adolescent students: Relationships to motivation and achievement, J. Early Adolescence 13, 21 (1993).
  * Marshman  _et al._ [2018a] E. Marshman, Z. Y. Kalender, C. Schunn, and C. Singh, Female students with A’s have similar physics self-efficacy as male students with C’s in introductory courses: A cause for alarm?, Phys. Rev. Phys. Educ. 96, 391 (2018a).
  * Marshman  _et al._ [2018b] E. Marshman, Z. Y. Kalender, C. Schunn, T. Nokes-Malach, and C. Singh, A longitudinal analysis of students’ motivational characteristics in introductory physics courses: Gender differences, Can. J. Phys. 96, 391 (2018b).
  * Raelin  _et al._ [2014] J. A. Raelin, M. B. Bailey, J. Hamann, L. K. Pendleton, R. Reisberg, and D. L. Whitman, The gendered effect of cooperative education, contextual support, and self-efficacy on undergraduate retention, Journal of Engineering Education 103, 599 (2014).
  * Atwood  _et al._ [2010] S. Atwood, E. Patten, and L. Pruitt, Outreach teaching, communication, and interpersonal skills encourage women and may facilitate their recruitment and retention in the engineering curriculum, in  _2010 Annual Conference & Exposition_, 10.18260/1-2–15639 (ASEE Conferences, Louisville, Kentucky, 2010) pp. 15–933, https://peer.asee.org/15639.
  * Whitcomb  _et al._ [2020] K. M. Whitcomb, Z. Y. Kalender, T. J. Nokes-Malach, C. D. Schunn, and C. Singh, Comparison of self-efficacy and performance of engineering undergraduate women and men, Int. J. Eng. Educ. 36, 1194 (2020).
  * Felder  _et al._ [1995] R. M. Felder, G. N. Felder, M. Mauney, C. E. Hamrin Jr., and E. J. Dietz, A longitudinal study of engineering student performance and retention. III. Gender differences in student performance and attitudes, J. Eng. Educ. 84, 151 (1995).
  * Bian  _et al._ [2018] L. Bian, S.-J. Leslie, M. C. Murphy, and A. Cimpian, Messages about brilliance undermine women’s interest in educational and professional opportunities, Journal of Experimental Social Psychology 76, 404 (2018).
  * Allen [2016] M. Allen, Women discouraged by masculine culture in physics, Phys. World 29, 11 (2016).
  * Maries  _et al._ [2020] A. Maries, N. I. Karim, and C. Singh, Active learning in an inequitable learning environment can increase the gender performance gap: The negative impact of stereotype threat, Phys. Teach. 58, 430 (2020).
  * Karim  _et al._ [2018] N. I. Karim, A. Maries, and C. Singh, Do evidence-based active-engagement courses reduce the gender gap in introductory physics?, Eur. J. Phys. 39, 025701 (2018).
  * Walton  _et al._ [2015] G. M. Walton, C. Logel, J. M. Peach, S. J. Spencer, and M. P. Zanna, Two brief interventions to mitigate a “chilly climate” transform women’s experience, relationships, and achievement in engineering, Journal of Educational Psychology 107, 468 (2015).
  * Steele [2010] C. M. Steele,  _Whistling Vivaldi: How Stereotypes Affect Us and What We Can Do_ (WW Norton & Co, 2010).
  * Reuben  _et al._ [2014] E. Reuben, P. Sapienza, and L. Zingales, How stereotypes impair women’s careers in science, Proceedings of the National Academy of Sciences - PNAS 111, 4403 (2014).
  * Kelly [2016] A. M. Kelly, Social cognitive perspective of gender disparities in undergraduate physics, Physical Review Physics Education Research 12, 020116 (2016).
  * McCullough [2007] L. McCullough, Gender in the physics classroom, The Physics Teacher 45, 316 (2007).
  * Gonsalves  _et al._ [2016] A. J. Gonsalves, A. Danielsson, and H. Pettersson, Masculinities and experimental practices in physics: The view from three case studies, Physical Review Physics Education Research 12, 020120 (2016).
  * Francis  _et al._ [2017] B. Francis, L. Archer, J. Moote, J. DeWitt, E. MacLeod, and L. Yeomans, The construction of physics as a quintessentially masculine subject: Young people’s perceptions of gender issues in access to physics, Sex Roles 76, 156 (2017).
  * Danielsson [2012] A. T. Danielsson, Exploring woman university physics students ‘doing gender’ and ‘doing physics’, Gender Edu. 24, 25 (2012).
  * Dennehy and Dasgupta [2017] T. C. Dennehy and N. Dasgupta, Female peer mentors early in college increase women’s positive academic experiences and retention in engineering, National Academy of Sciences 114, 5964 (2017).
  * Doucette and Singh [2020] D. Doucette and C. Singh, Why are there so few women in physics? Reflections on the experiences of two women, Phys. Teach. 58, 297 (2020).
  * Rolin [2009] K. Rolin, Standpoint theory as a methodology for the study of power relations, Hypatia 24, 218 (2009).
  * Longino [1993] H. E. Longino, Feminist standpoint theory and the problems of knowledge, Signs 19, 201 (1993).
  * Harding [2004] S. G. Harding,  _The Feminist Standpoint Theory Reader: Intellectual and Political Controversies_ (Routledge, New York, 2004).
  * Blickenstaff [2005] J. C. Blickenstaff, Women and science careers: Leaky pipeline or gender filter?, Gender and Education 17, 369 (2005).
  * Johnson [2020] A. Johnson, An intersectional physics identity framework for studying physics settings (Springer, 2020) pp. 53–80.
  * Collins [2009] P. H. Collins,  _Another Kind of Public Education: Race, Schools, the Media, and Democratic Possibilities_ (Beacon Press, 2009).
  * Cwik and Singh [2023] S. Cwik and C. Singh, Framework for and review of research on assessing and improving equity and inclusion in undergraduate physics learning environments, in  _International Handbook of Physics Education Research: Special Topics_ (AIP Publishing, Melville, New York, 2023) pp. 2–1.
  * Saldaña [2013] J. Saldaña,  _The Coding Manual for Qualitative Researchers_ , 2nd ed. (SAGE, 2013).
  * Note [1] We note that Esme and Paulina have different college experiences in regard to the aspects of their identities that have negatively impacted them and one reason for this could be the differences in their visual characteristics, i.e., only Paulina visually appears to be non-White.
  * Heller and Hollabaugh [1992] P. Heller and M. Hollabaugh, Teaching problem solving through cooperative grouping. Part 2: Designing problems and structuring groups, American Journal of Physics 60, 637 (1992).