# Indicators for characterising online hate speech and its automatic detection

Erica Forzinetti1 Marco L. Della Vedova1,2, Corresponding author:
<EMAIL_ADDRESS>Stefano Pasta1 Milena Santerini1

( 1Osservatorio sull’odio on line - Mediavox
Università Cattolica del Sacro Cuore
Milano, Italy
2Chalmers University of Technology
Gothenburg, Sweden
)

###### Abstract

We examined four case studies in the context of hate speech on Twitter in
Italian from 2019 to 2020, aiming at comparing the classification of the 3,600
tweets made by expert pedagogists with the automatic classification made by
machine learning algorithms. Pedagogists used a novel classification scheme
based on seven indicators that characterize hate. These indicators are: the
content is public, it affects a target group, it contains hate speech in
explicit verbal form, it will not redeem, it has intention to harm, it can
have a possible violent response, it incites hatred and violence. The case
studies refer to Jews, Muslims, Roma, and immigrants target groups. We find
that not all the types of hateful content are equally detectable by the
machine learning algorithms that we considered. In particular, algorithms
perform better in identifying tweets that incite hatred and violence, and
those that can have possible violent response.
Keywords: hate speech, machine learning, social networks.

##  1 Introduction

In the public language, speeches and narratives of the infosphere and onlife
society [1], i.e., marked by the continuity of cross-references between online
and offline, discursive networks have been set up and they are able to build
new paradigms of hatred [2, 3]. Invectives, insults, offences, and rhetorics
of aversion follow human relations and conflicts throughout history, but a
singular reflection on this phenomenon is underway today, increasingly
perceived as a vital issue for the stability of democracy and social life. The
spread of Web 2.0 has opened up new dimensions to hate speech, transforming
its structure and syntax, but first of all its meaning and motivations. Most
of this communication, which is spreading in liquid, unstructured and
trivialised form, takes place under the banner of emotions, which orient and
direct our minds intelligently but also quickly and spontaneously [4, 5, 6].
From a legal perspective, hate speech has no clear definition [7, 8, 9]. For
the Council of Europe, it means “all forms of expression that spread, incite,
promote or justify racial hatred, xenophobia, anti-Semitism or other forms of
hatred based on intolerance, including: intolerance expressed by aggressive
nationalism and ethnocentrism, discrimination and hostility against
minorities, migrants and people of immigrant origin” [10]. On the other hand,
one could speak of hatred as a feeling, a persistent and systematic attitude
of aversion towards another person or group, or as an intrinsic psychological
dynamic that may or may not induce violent behavior [11]. Because of its
multidimensionality, the phenomenon requires a multidisciplinary approach.
From the legal field, the discussion has reached the humanities (sociological,
pedagogical, anthropological, philosophical, linguistic, semiotic), converging
in the interdisciplinary field of Hate Studies, which brings together
scholars, researchers, politicians, communication experts, human rights
activists, NGO leaders. From a pedagogical point of view, education for
citizenship is affected by hostile narratives, by the binary vision of the
world, divided into us/them, friend/enemy, inside/outside, by the affirmation
of processes of choosing a widespread and banal target group, not only in
words but also in images. Hate speech is considered a serious threat to social
cohesion, considering that the narrow boundary with freedom of expression and
the need to preserve this fundamental right sometimes provide a pretext for
allowing it to spread without limits. As can be seen from the qualitative-
textual and qualitative-motivational analysis and the following conversations
with young authors of hate performances, educational reflection has the need
to study the phenomenon in greater depth in order to prevent and combat the
various forms expressed in the form of online language and images [2]. The
research carried out so far in Italy has made it possible to analyze online
hate speech in different ways: in some cases, it has been studied from
geographical mapping, in others a general quantification has been attempted
using only offensive terms to identify online hate and the results indicate
that the phenomenon is more circumscribed and limited [12]. The linguistic
analyses that have developed on the basis of this assumption have made several
steps forward in identifying the lexical components of hate speech [13, 14].
From a methodological point of view, several analyses have also been made,
experimenting with different approaches for automatic speech processing
(sentiment analysis, text mining…) and on data collection and annotation to
capture the different facets of hate speech [15]. Because of the educational
implications of the phenomenon, these initial analyses suggest that the
detection of online hatred should be combined with a more in-depth study of
its characteristics, creating a synergy between computer science and the
humanities. In fact, it is not only a question of deciding ‘what is hate’, but
also of analyzing the many forms in which it is expressed, in order to better
identify subjects, targets and ways of expressing it, with a view to
developing more effective prevention and counteraction strategies. Previous
studies have examined and compared the websites of ideological groups from a
communications and media use perspective. The results indicated that group
type was predicted by the type of information presented, the difficulty of
becoming a member, and the amount of freedom members had on discussion boards.
These findings suggest that characteristics of violent ideological group
websites can be used to distinguish them from websites of both nonviolent
ideological and non-ideological groups [16].

##  2 Methodology

In this paper, we face the problem of classifying hate speech on
Twitter111Data has been collected in 2021, before the rebranding of Twitter to
X. by combining the socio-educational approach and automatic computer
processing. Thanks to the support of experts pedagogists we try to identify
specific features and indicators that characterize hate contents. The data has
been collected using the TwitterAPI with specific search queries detailed in
the next section. The expert annotators, given the text of the tweet, the
author, the date of publication, the number of likes and retweets, determined
whether the tweet contains hate or not. If it was hate content, they assigned
the corresponding indicators. The indicators that determine whether a text
contains hate are seven:

  1. 1.

Public: the content can be seen without limitation by users, so it is not a
private message in a restricted circle; note that in the specific case under
consideration, all tweets are public.

  2. 2.

Target group - other target: the hate content affects a target group, or an
individual related to that group, or an individual for what he or she
represents (e.g., the first nurse vaccinated to hit vaccines); often the
targets are minorities or their members, as indicated for example by the
definition of hate speech Council of Europe Recommendation (97)20, adopted on
October 30, 1997, and the texts of the same institution. This process of
election differentiates this content from the behaviors of the cyberbullying
spectrum.

  3. 3.

Hate expression: hate content contains hate speech in explicit verbal form,
i.e., hate words, verbally explicit insults, or speech in which one denies the
other person as a person, considering them inferior or attributing negative
qualities to them, insulting them, or humiliating them.

  4. 4.

Will not redeem: the speech is not interested in changing the mind of the hate
victim, but only in insulting and hurting him. It should also be considered
that very often the target of hate is not the interlocutor, but it is the
object of the speech, so rarely the purpose of the author of hate speech is to
"redeem" the target, also because the most structured forms of hate do not
allow, precisely, the redemption because the target is hated "as such" (Jew,
foreigner, black, Roma …) regardless of his individual behavior.

  5. 5.

Intention to harm: the content is marked by an intentionality of the author to
hit the victim (individual or target). Even in this case it should be noted
that often the target does not coincide with the interlocutor but rather with
the object of the speech, then often takes the form of speeches with the
intentionality to produce hostility towards the target.

  6. 6.

Possible violent response: the tones of speech are marked by violence or
intensity such that, following the typical mechanisms of toxic disinhibition
on the Web, they easily and quickly allow for a response of incitement to
hatred and violence.

  7. 7.

Inciting hatred and violence: hate contents that explicitly and directly
incite hatred and violence, in which the author aims to enlarge the co-
producers of hate speech. In these cases, online hate is a particularly
fertile background for offline hate actions and hate crimes.

These seven characteristics has been identified as the “Spectrum of Online
Hate Indicators”, in part to address a definitional ambiguity of the concept
of hate, and are presented in increasing intensity. The proposal draws from
the didactics of the Shoah the interpretive tool of the Anti-Defamation
League’s “Pyramid of Hate”, which interprets the extreme outcome of hate as
thresholds that are crossed and taboos that are broken down. However, although
the spectrum of indicators of online hate indicates a progression of
intensity, the sequence of indicators should not be understood rigidly, and
therefore one may find hate content that has some indicators but not others.
In any case, the presence of multiple indicators on the spectrum, and
particularly those of higher intensity, detects content with potentially very
dangerous effects in the communication ecosystem.

##  3 Case studies and data collection

This analysis has been tested on four datasets referring to four different
target groups: Jews, Muslims, Roma and immigrants. The first dataset, referred
to Jews, is composed by 160,646 Italian tweets published from 2019, September
1st to 2020, May 31st. The search query used to download the tweets is

ebrei OR Soros OR Israele OR sionismo OR sionista

(transl. Jews OR Soros OR Israel OR Zionist OR Zionism). Those terms were
identified from the working definition of Antisemitism by the International
Holocaust Remembrance Alliance, from the international reports of the
Observatory on Anti-Semitism of the Centre for Contemporary Jewish
Documentation, the Kantor Center for the Study of Contemporary European Jewry,
as well as on the basis of the literature on anti-Semitism [17, 11, 18]; in
this way, all tweets containing at least one of the words in the query were
downloaded.

The second dataset, referred to Muslims, is composed by 8,299 Italian tweets
published from 2019, September 1st to 2020, May 31st. The search query used to
download the tweets is

(islam OR musulman_ OR mussulman_) AND (radical_ OR integral_ OR terroris_ OR
valor_ OR immigra_ OR sharia OR velo)

(transl. islam OR musulman_ OR mussulman_ AND radical_ OR integral_ OR
terroris_ OR valu_ OR immigra_ OR sharia OR veil), in order to obtain all the
tweets containing at least one lemma identifying the target group (words in
the first bracket) and simultaneously with a reference to typical elements of
Islamophobia (words in the second bracket). These are, in particular, issues
at the basis of the most widespread forms of Islamophobia in Italy, associated
with the following “discourses”: the identification of all Muslims to
potential terrorists; anti-Muslim hostility that is combined with xenophobic
sentiment against immigrants; a clash of values as a form of reaction to the
supposed “Islamization” of Italy that confessionalizes the theory of ethnic
substitution closely linked to culturalist and differentialist racism, based
on the impossibility of coexistence due to an irreconcilable cultural
difference (“Muslims are too different, it is impossible to live together”);
Islam for essence anti-West, that is, the idea based on the accusation of
anti-democracy declined towards a secular defense of subjective rights
protected by the Constitution, to which all Muslims in Europe - according to
this “vulgate” - would not want to adapt, in the name of the alleged
superiority of religious laws.

The third dataset, referred to Roma, is composed by 1,191 Italian tweets
published from 2020, June 1st to 2020, December 31st. The search query used to
download the tweets is

(zingar_ OR rom OR nomad_) AND (feccia OR ladr_ OR rapi_ OR rolex OR bulldozer
OR strupr_ OR violent_ OR mendica_ OR elemosin_)

(transl. gypsy OR roma OR nomad_ AND scum OR thief OR kidnap_ OR rolex OR
bulldozer OR rap_ OR violent_ OR beg_ OR charity), in order to obtain all the
tweets containing at least one lemma identifying the target group (words in
the first bracket) and simultaneously "hostile words" or words evocative of
specific stereotypes and prejudices towards Roma and Sinti (words in the
second bracket) [19]. Such lemmas, rooted in the Italian “common sense”, are
also confirmed by the literature [20] and by European reports, such as the
Civil society monitoring report on implementation of the national Roma
integration strategy in Italy.

The fourth and last dataset, referred to people with a migration background,
is composed by 37,817 Italian tweets published from 2020, June 1st to 2020,
December 31st. The search query used to download the tweets is

(migrant_ OR profug_ OR stranier_ OR immigrat_ OR pover_ OR negr_) AND
(scimmia OR banan_ OR sbarc_ OR invasion_ OR razz_ OR invas_ OR stupr_ OR
violent_)

(transl. migrant_ OR refuge_ OR foreign_ OR immigra_ OR poor_ OR nigger_) AND
(monkey OR banan_ OR land_ OR invasion_ OR race_ OR invad_ OR rap_ OR
violen_), the first bracket refers to words connected to the migratory flows
towards Italy, while two other words have been inserted to evoke socio-
economic relations (“poor”) and the classic image of racism towards blacks
(“nigger”) [3]; the second bracket refers instead to racist images and
discourses, with particular reference to the landings that took place towards
the Italian coasts and to what the Council of Europe defined in 2021 as
“Afrophobia” or “anti-Black racism” [21].

For each of the four datasets, a random sample of 900 tweets has been selected
to be manually annotated by experts in each field. The annotators assigned to
each tweet a binary label (yes/no) for each of the seven indicators. The
resulting dataset of 3,600 annotated tweets is analyzed in what follows.

##  4 Results

The first result is related to the percentage of hate in the dataset. The
annotators classified 996 out of 3,600 tweets as containing hate, which
corresponds to 27.7% of the total. Fig. 1 shows the percentage of tweets
containing hate divided into the four case studies (grey bars). As can be
seen, in the Roma dataset more than 40% of tweets are considered as hateful
content, which is a much larger percentage with respect to the other test
cases. This is probably due to the condition used in the query for downloading
the tweets.

Figure 1: Percentage of hate tweets in the sample dataset

Some examples of tweets, which are labelled has hate content, are shown in
Table 1 with the corresponding English translation.

Original text of the tweet |  English translation
---|---
QUELLA ENORME METASTASI INFILTRATA DI GEORGE SOROS |  THAT HUGE INFILTRATED METASTASIS OF GEORGE SOROS
Sto dicendo che praticamente tutti i terroristi sono musulmani. Non ti risulta? |  I’m saying that basically all terrorists are Muslims. Don’t you know that?
ex zingari…ora sinti..domani rom…ma sempre ladri sono.. irrecuperabili!??? |  ex Gypsies … now Sinti … tomorrow Roma … but always thieves are … irretrievable!???
E questi sono i poveretti che salviamo perché hanno bisogno di aiuto?questi sono DELINQUENTI DELLA STESSA RAZZA PDche vengono per rubare spacciare e fottere |  And these are the poor people we save because they need help? These are CRIMINALS OF THE SAME KIND who come to steal, deal and fuck
Table 1: Examples of tweets containing hate, as classified by annotators

Once it was established that a tweet contained hate, the second step of the
analysis was to check whether the characteristics of the seven indicators were
present in the text. The annotators, for each of the 996 hate tweets, labelled
which indicator was present or not. The results are shown in Fig. 2. All
tweets are marked as public and almost all refer to a target group, and this
is obviously due to the way in which the dataset was made up. Moreover, only a
small percentage of hate tweets (< 20%) can have a possible violent response
and incite hatred and violence.

Figure 2: Percentage of tweets characterized by indicators within hate tweets

In order to make a more detailed comparison, the results obtained for each
target group (Jews, Muslims, Roma and people with a migration background) were
compared using the radarchart technique, a graphical method of visualizing
multivariate data. The results are shown in Figure 3. As can be seen, the
indicators homogeneously occurred in all four target groups. The most
substantial differences can be seen in the dataset referring to Jews, where,
unlike the others, a high percentage (more than 90%) of tweets has intention
to harm and will not redeem while only a little percentage can have possible
violent response. On the other hand, tweets referred to Muslims have a small
percentage (20%) of hate expression.

Figure 3: Radarchart of indicators for each target group

The last step of this analysis aims at evaluating automatic hate speech
classifiers based on machine learning. We chose four ML classifiers among the
few available open source, pre-trained models, specific for the Italian
language. The test consists of the comparison between the output of the
automatic classifiers and the evaluation of the expert annotators, which is
assumed to be ground-truth. The first model, Montanti [22], was developed for
EVALITA-2020 [23, 24], a periodic evaluation campaign of Natural Language
Processing (NLP) and speech tools for the Italian language222See
http://www.evalita.it. The main task of the competition was about the hate
speech detection, a binary classification task aimed at determining the
presence or the absence of hateful content in the text towards a given target
(among immigrants, Muslims and Roma). The other three algorithms are available
on the website HuggingFace. Dehatebert-mono-italian [25] algorithm is used for
detecting hate speech in Italian language, while POLIticBERT333Available at
https://huggingface.co/unideeplearning/polibert_sa and Neuraly444Available at
https://huggingface.co/neuraly/bert-base-italian-cased-sentiment performs
sentiment analysis on Italian sentences. With regard to the latter two
algorithms, in order to determine which tweets were to be considered as hate
content, an optimal threshold was chosen considering the highest balanced
accuracy score. Table 3 shows some of the results obtained from the four
algorithms and the label that was assigned by the annotators.

As can be seen in Table 3, the first tweet is classified correctly by all four
algorithms, probably because there are words like ’dickheads’, ‘terrorists’
and ‘Muslims’. The second tweet still contains negative words as ‘terrorists’,
‘enemy’, and ‘dead’ in fact all the four algorithms classify it as hate
content, however, according to the annotators, it is not a question of hatred
because the author is criticizing those who hate. Conversely, the third tweet
contains words as ‘children’ and ‘joking’, in fact all four algorithms
classify it as hate-free, but, according to the annotators, it is hate content
because it states a prejudice against Roma. These last two tweets are clear
examples that show how the identification of the context is one of the
principal problems for the machine learning algorithms. The fourth and fifth
tweets can be considered doubtful cases, in fact the fourth is a text that
contains real hate against Jews and the fifth, while having negative words,
describes a fact that really happened, so it cannot be considered a hateful
content. Note that in this example POLItic BERT algorithm correctly identified
all five tweets, but obviously there are other tweets not classified correctly
by this algorithm as well.

Indicator | [22] | [25] | -3 | -4
---|---|---|---|---
(i1) Public | 0.62 | 0.65 | 0.65 | 0.62
(i2) Target group - other target | 0.62 | 0.65 | 0.64 | 0.62
(i3) Hate expression | 0.62 | 0.66 | 0.66 | 0.66
(i4) Will not redeem | 0.61 | 0.64 | 0.65 | 0.61
(i5) Intention to harm | 0.60 | 0.63 | 0.65 | 0.61
(i6) Possible violent response | 0.70 | 0.68 | 0.68 | 0.65
(i7) Inciting hatred and violence | 0.63 | 0.70 | 0.69 | 0.69
Overall | 0.62 | 0.65 | 0.69 | 0.69
Table 2: Balanced accuracy score of the four ML algorithms

The last analysis is about the performance of automatic classifiers on the
seven hate indicators. Table 2 shows the balanced accuracy score for each
algorithm for each indicator. The balanced accuracy score is a standard
performance metric used when the dataset is not balanced, i.e., when the
number of observations with a label is very different to the number of
observations with another label, as it happens in our case. As can be seen,
all the algorithms have the highest balanced accuracy score for ‘Possible
violent response’ and ‘Inciting hatred and violence’ indicators. In this
regard, Table 4 contains anonymized examples of tweets which are labelled as
‘Possible violent response’ and/or ‘Inciting hatred and violence’.

As shown in Table 4, the text of these tweets contains many slang words (i.e.,
‘assholes’, ‘filthy scum’, ‘jail’, ‘disease’), which can be easily detected by
the algorithm. From the first two examples, it can be seen how these are real
hate contents. Both were classified by the annotators as ‘Inciting hate and
violence’ and ‘Possible violent response’, in fact they contain strong and
violent hatred and there is a risk that these will result in offline hate.
Moreover, in the third tweet there is not a violent tone (in fact the label
for ‘Possible violent response’ is 0) but it could be a background for offline
hate actions and hate crimes (‘Inciting hate and violence’ has label 1). On
the other hand, the fourth tweet has a rough tone (‘Possible violent response’
is 1) but, according to the annotators, the intent to spread hate is not
present (‘Inciting hate and violence’ has label 0).

##  5 Discussion and conclusion

One of the most important factors to emphasize before drawing conclusions from
these analyses is that the high percentage of hatred present in the dataset
(27,7%) is due to the way in which the queries to download the tweets were
created, in particular for the datasets referred to Muslims, Roma and people
with a migration background. In this study, the datasets were made up with all
those tweets that contained simultaneously "hostile words" or words evocative
of specific stereotypes and prejudices towards. At the end of this study, one
of the positive aspects that could be found is a low percentage of tweets
classified as ‘Inciting hate and violence’ and ‘Possible violent response’ by
the annotators (Figure 2). This clearly means that the tweets downloaded from
the web don’t allow to have a hate response easily and quickly (possible
violent response) and are not a fertile background for offline hate actions or
hate crimes (inciting hatred and violence). Most of the hate found online is
not to be considered dangerous, it is often an end in itself. This result is
in line with studies already done, since it has been proved that the websites
of violent ideological groups that use more fear appeals were less interactive
and less credible than non-ideological groups and nonviolent ideological
groups [26]. However, not having downloaded real-time data, it may be that
Twitter’s algorithms have promptly eliminated more dangerous hate contents.
This can also be noticed from the results shown in Table 2 about the balanced
accuracy scores. All four algorithms used in this study seem to perform better
in identifying tweets classified as ‘Inciting hate and violence’ and ‘Possible
violent response’. This means that machine learning algorithms are able to
better identify the tweets considered most dangerous by the annotators. On the
other hand, one of the biggest weaknesses of machine learning algorithms is
about the context detection. Too often it happens that the same words used in
different contexts have different meanings. This is the main reason why
automatic online hate speech detection still has many drawbacks. The main
problem causing the low reliability of these methods is due to the extremely
unbalanced presence of real hateful content in the datasets which are analyzed
and the lack of distinctive and discriminatory features.

## References

  * Floridi [2014] Luciano Floridi.  _The 4th Revolution: How the Infosphere Is Reshaping Human Reality_.  Oxford University Press, New York ; Oxford, first edition edition, 2014.  ISBN 978-0-19-960672-6.
  * Pasta [2018] Stefano Pasta.  _Razzismi 2.0: analisi socio-educativa dell’odio online_.  Scholè, Brescia, 2018.  ISBN 978-88-284-0030-1.
  * Santerini [2021] Milena Santerini.  _La mente ostile: forme dell’odio contemporaneo_.  Raffaello Cortina Editore, 2021.  ISBN 978-88-328-5300-1.
  * Damasio [1994] A. Damasio.  _Descartes’ Error: Emotion, Reason, and the Human Brain_.  Putnam, 1994.
  * Nussbaum [2008] Martha Craven Nussbaum.  _Upheavals of Thought: The Intelligence of Emotions_.  Cambridge University Press, Cambridge, 8. print edition, 2008.  ISBN 978-0-521-53182-5 978-0-521-46202-0.
  * Wallace [2015] Patricia Wallace.  _The Psychology of the Internet_.  Cambridge University Press, second edition, November 2015.  ISBN 978-1-107-07913-7 978-1-139-94096-2 978-1-107-43732-6.  doi: 10.1017/CBO9781139940962.
  * Ziccardi [2016] Giovanni Ziccardi.  _L’odio Online: Violenza Verbale e Ossessioni in Rete_.  Number 102 in Saggi. R. Cortina, Milano, 1. ed edition, 2016.  ISBN 978-88-6030-806-1.
  * Ziccardi [2020] Giovanni Ziccardi.  _Online Political Hate Speech in Europe: The Rise of New Extremisms_.  Edward Elgar Publishing, Northampton, August 2020.  ISBN 978-1-78811-365-6.
  * Faloppa [2020] Federico Faloppa.  _Odio: Manuale Di Resistenza Alla Violenza Delle Parole_.  UTET, Milan, prima edizione edition, 2020.  ISBN 978-88-511-7796-6.
  * Council of Europe (1997) [Committee of Ministers] Council of Europe (Committee of Ministers).  Recommendation No. R (97) 20 of the Committee of Ministers to member states on “hate speech”, 1997.
  * Santerini [2019] Milena Santerini, editor.  _Il nemico innocente: l’incitamento all’odio nell’Europa contemporanea =_.  Sguardi sul mondo attuale. Guerini e associati, Milano, prima edizione edition, 2019.  ISBN 978-88-6250-760-8.
  * D’Amico and Siccardi [2021] Marilisa D’Amico and Cecilia Siccardi.  _La Costituzione non odia: conoscere, prevenire e contrastare l’hate speech on line_.  Number 93 in Pubblicazioni del dipartimento di diritto pubblico italiano e sovranazionale. G. Giappichelli editore, Torino, 2021.  ISBN 978-88-921-3854-4.
  * Femia [2020] D. Femia.  Discorso dell’odio e risorse per il trattamento automatico delle lingue. Metodi, ipotesi, proposte.  In _Hate Speech: L’odio Nel Discorso Pubblico: Politica, Media, Società_. Round Robin ed., 2020.
  * Ferrini and Paris [2019] Caterina Ferrini and Orlando Paris.  _I Discorsi Dell’odio: Razzismo e Retoriche Xenofobe Sui Social Network_.  Number 1266 in Biblioteca Di Testi e Studi ; Linguistica. Carocci editore, Roma, 1a edizione edition, 2019.  ISBN 978-88-430-9854-5.
  * Sanguinetti et al. [2018] Manuela Sanguinetti, Fabio Poletto, Cristina Bosco, Viviana Patti, and Marco Stranisci.  An Italian Twitter Corpus of Hate Speech against Immigrants.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan, May 2018. European Language Resources Association (ELRA).
  * Byrne et al. [2013] Cristina L. Byrne, Darin S. Nei, Jamie D. Barrett, Michael G. Hughes, Joshua L. Davis, Jennifer A. Griffith, Lauren N. Harkrider, Kimberly S. Hester, Amanda D. Angie, Issac C. Robledo, Shane Connelly, H. Dan O’Hair, and Michael D. Mumford.  Online Ideology: A Comparison of Website Communication and Media Use.  _Journal of Computer-Mediated Communication_ , 18(2):25–39, January 2013.  ISSN 10836101.  doi: 10.1111/jcc4.12003.
  * Taguieff [2015] Pierre-André Taguieff.  _L’antisémitisme_.  Number n° 3734 in Que sais-je ? PUF, Paris, 2015.  ISBN 978-2-13-054909-3.
  * Pasta [2022] Stefano Pasta.  ANTISEMITISMO 2.0. La propagazione dell’odio online nel web social.  _Cultura tedesca_ , 63:81–99, 2022.
  * Pasta [2019] Stefano Pasta.  The media and the public perception of the Roma and the Sinti in Italy.  _Trauma and Memory_ , 7(1):46–52, April 2019.  ISSN 2282-0043.  doi: 10.12869/TM2019-1-08.
  * Claps and Vitale [2011] Enrico Claps and Tommaso Vitale.  Not Always the Same Old Story: Spatial Segregation and Feelings of Dislike against Roma and Sinti in Large Cities and Medium-size Towns in Italy.  In _Multi-Disciplinary Approaches to Romany Studies_ , pages 228–253. Central European University Press, February 2011.
  * Council of Europe [2021] Council of Europe.  Combating racism and racial discrimination against people of Africa descent in Europe, 2021.
  * Bisconti and Montagnani [2020] Elia Bisconti and Matteo Montagnani.  Montanti @ HaSpeeDe2 EVALITA 2020: Hate Speech Detection in online contents (short paper).  In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, _Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020)_ , volume 2765 of _CEUR Workshop Proceedings_ , Online event, December, December 2020. CEUR.
  * Basile et al. [2020] Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro.  EVALITA 2020: Overview of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian.  In Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro, editors, _Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020)_ , volume 2765 of _CEUR Workshop Proceedings_ , Online event, December, December 2020. CEUR.
  * Sanguinetti et al. [2020] Manuela Sanguinetti, Gloria Comandini, Elisa di Nuovo, Simona Frenda, Marco Stranisci, Cristina Bosco, Tommaso Caselli, Viviana Patti, and Irene Russo.  HaSpeeDe 2 @ EVALITA2020: Overview of the EVALITA 2020 Hate Speech Detection Task.  In Valerio Basile, Danilo Croce, Maria Maro, and Lucia C. Passaro, editors, _EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020_ , pages 93–101. Accademia University Press, 2020.  ISBN 9791280136329.  doi: 10.4000/books.aaccademia.6897.
  * Aluru et al. [2021] Sai Saketh Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee.  A Deep Dive into Multilingual Hate Speech Classification.  In Yuxiao Dong, Georgiana Ifrim, Dunja Mladenić, Craig Saunders, and Sofie Van Hoecke, editors, _Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track_ , Lecture Notes in Computer Science, pages 423–439, Cham, 2021. Springer International Publishing.  ISBN 978-3-030-67670-4.  doi: 10.1007/978-3-030-67670-4_26.
  * Dunbar et al. [2014] Norah E. Dunbar, Shane Connelly, Matthew L. Jensen, Bradley J. Adame, Bobby Rozzell, Jennifer A. Griffith, and H. Dan O’Hair.  Fear Appeals, Message Processing Cues, and Credibility in the Websites of Violent, Ideological, and Nonideological Groups: J Comput-Mediat Comm.  _Journal of Computer-Mediated Communication_ , 19(4):871–889, July 2014.  ISSN 10836101.  doi: 10.1111/jcc4.12083.

Original text of the tweet |  English translation | Hate annot. | [22] | [25] | -3 | -4
---|---|---|---|---|---|---
Minoranza forse, però grandissime teste di cazzo che hanno e stanno facendo guerre e attentati in nome di Allah! E guarda caso tutti i terroristi sono musulmani. |  A minority perhaps, but huge dickheads who have and are making wars and attacks in the name of Allah! And it just so happens that all terrorists are Muslims. | 1 | 1 | 1 | 1 | 1
Mah, passata la moda dei musulmani tutti terroristi, ora tocca agli ebrei, devono trovarsi un nemico, altrimenti non sanno come giustificare la propria esistenza, sono morti |  Well, after the fashion for Muslims to be all terrorists, now it’s the turn of the Jews, they have to find an enemy, otherwise they don’t know how to justify their existence, they’re dead. | 0 | 1 | 1 | 1 | 1
@user Da piccolo mia mamma diceva" stai lontano dagli zingari perché rapiscono i bambini" pensavo scherzasse!  |  @user When I was little my mum used to say "stay away from gypsies because they kidnap children" I thought she was joking!  | 1 | 0 | 0 | 0 | 0
Fatemi capire, Hitler si era affidato agli astri e ha sbagliato, sterminando 6 milioni di ebrei. E se ci avesse azzeccato invece? |  Let me get this straight, Hitler relied on the stars and got it wrong, exterminating 6 million Jews. But what if he got it right? | 1 | 0 | 0 | 1 | 1
Ieri 5 morti a #Parigi per mano di un immigrato convertito all’Islam, sempre ieri a #Fermo un immigrato girava con un machete pronto a uccidere. Oggi un immigrato ha ucciso due poliziotti a #Trieste. Serve prudenza e selezione degli ingressi, niente ingenuità. |  Yesterday 5 dead in #Paris at the hands of an immigrant convert to Islam, also yesterday in #Fermo an immigrant was walking around with a machete ready to kill. Today an immigrant killed two policemen in #Trieste. We need prudence and selection of entries, no naivety. | 0 | 1 | 0 | 0 | 1
Table 3: Examples of tweets classified by the four ML algorithms Original text of the tweet |  English translation | i7 | i6
---|---|---|---
@user @user Facciamo qualcosa per fermare questi coglioni che svendono il paese a negri, islamici e zingari. La feccia dell’umanità. |  @user @user Let’s do something to stop these assholes selling out the country to niggers, islamics and gypsies. The scum of humanity. | 1 | 1
Odio i rom, se potessi gli sparerei.. gentaccia di merda ladra, zozza, Hitler li doveva ammazzare tutti! |  I hate the Roma, I’d shoot them if I could… thieving, filthy scum, Hitler should have killed them all! | 1 | 1
Confidiamo in Putin che buone mettere in galera Soros e buttare la chiave |  We trust Putin who wants to put Soros in jail and throw away the key | 1 | 0
Nulla di nuovo! Sono anni che in inghiltera l‘Islam si diffonde come una malattia! Ci sono quartieri in cui la legge e la sharia.. dove dei barbuti vanno a imporre la morale islamica! L‘inghiltera ormai e andata… |  Nothing new! Islam has been spreading like a disease in England for years! There are neighbourhoods where the law is sharia… where bearded men go to impose Islamic morality! England is gone… | 0 | 1
Table 4: Examples of tweets annotated as containing the hate indicators:
possible violent response (i6) or inciting hatred and violence (i7)


  *

# Einsteinian gravitational concepts throughout secondary school

Corey McInerney and Phil Sutton  University of Lincoln, School of Mathematics
and Physics; Lincoln, LN6 7TS, UK<EMAIL_ADDRESS>
###### Abstract

Einstein’s theory of relativity is largely thought of as one of the most
important discoveries of the 20th century and continues to pass observational
tests over 100 years later. Yet, it is Newtonian gravity, a 350 year old
formalism proven to be less accurate than relativity, which is taught in
schools. It has been shown that Einsteinian gravitational concepts can be well
understood by students in both primary and secondary education. In this paper,
a cross-section of students from Yr 7-13 enrolled in an English secondary
school took part in an intervention designed to introduce the idea of gravity
from spacetime curvature. The overall aim of this work is to assess the
viability of including relativity in the secondary curriculum and to ascertain
which year this material would be best placed in. We determine that all year
groups where able to appreciate the effects of curvature to some extent.
Visual demonstrations aided conceptual understanding at Yr 7-8 level, but this
does not have a strong effect on their ideas around the source of the
gravitational force. Participants in Yr 9-13 were able to understand concepts
beyond those introduced in the demonstrations. However, a deeper understanding
of curvature as the source of the gravitational force is not seen until years
12 & 13\. We find that those in Yr 13 have the best overall understanding of
the concepts introduced during our intervention.

Keywords: Einstenian gravity, relativity, secondary school, conceptual
understanding

##  1 Introduction

In 1687, Isaac Newton published his law of universal gravitation[1], it is
commonly taught in schools because it combines the concept of mass and weight
with everyday phenomenon like falling objects[2]. Gravity is regarded as being
one of the key threshold concepts in education used as an indicator of a
students understanding of wider physics topics[3, 4]. However, research
suggests that a large number of misconceptions about gravity are held by
students. These include issues surrounding the direction of gravitational
force, gravity outside of the Earth and the role of the Sun in celestial
dynamics[5, 6, 7]. These issues are tied to students not being able to fully
grasp where the gravitational force comes from, with some students believing
that objects only have a gravitational pull if they are heavy and spherical,
or that air is needed in order for gravity to act between objects[8, 9].
Misconceptions can stay with students throughout their schooling, impacting
confidence, ability and thus students overall enjoyment of science[10].

A new approach pioneered by the Einstein-First project could eliminate some of
the conceptual issues faced by students. The idea is to introduce principals
from modern physics, such as Einstein’s Theory of General Relativity (GR) into
the Australian curriculum at primary and secondary level. A similar project,
ReleQuant, in Norway also pioneers the teaching of Relativity and quantum
mechanics at secondary school[11, 12].

Einstein’s Theory of General Relativity provides a mathematical description of
gravity and offers a qualitative description of how gravity works[13]. A key
feature of GR is the non-Euclidean geometry of space and the interplay between
time and space. The sophisticated mathematics involved in GR are seen as a
barrier to students below degree level. The ReleQuant project bypasses this
mathematical rigor by placing conceptual language at the forefront of the
learning process, exploring concepts via thought experiments and visual
demonstrations.

This study explores the impact of a one-off intervention on students’
understanding of Einsteinian gravitational concepts. Similar studies have been
performed in Lebanon[14], Australia[15, 16], Indonesia[17] and Italy[18, 19].
Each of those works report that students have a better understanding of
gravity from a GR stand point after being introduced to concepts such as
curvature, time dilation and spacetime diagrams.

The intervention used for this study was designed to fit into a standard 50
minute lesson period at a single English state funded secondary school. Within
that period, students were taught about how spacetime curvature produces the
gravitational force with the help of two hands on activities. The host school
is a fully government funded, selective, mixed grammar school located the town
of Gainsborough, County Lincolnshire, England. This study took place during
February and March 2023 and at that time a total of 1237 students were
enrolled at the school.

An identical intervention was delivered to all age groups (years 7 - 13) at
this school. Students were issued with questionnaires before and after the
intervention and by comparing the results across year groups, we hope to
identify the optimum place for GR to be introduced into the secondary science
curriculum.

In the following section we describe the research methodology in comparison
with other works, as well as the content and application method of our
intervention. In Section 3, we present the results of our research, which are
also discussed and analysed.

##  2 Method

###  2.1 The Host School

The school where this research was performed is a fully government funded,
selective, mixed grammar school located in England. At the time of this study,
a total of 1237 students were enrolled at the school. The UK government
website (https://www.find-school-performance-data.service.gov.uk) shows that
in 2022, the school was ranked in the top 25% in the UK for academic
performance at GCSE level and that 82% of pupils at the school achieved
A-Level grades between A* and C.

###  2.2 Comparison With Other Works

In other works, interventions range from short, one off workshops[20], to full
20 session programs[21, 22]. The offering a longer syllabus means more content
and concepts can be covered, students can learn at their own pace and there
are opportunities for deeper learning experiences. Nonetheless, it has been
shown that one-off interventions have a significant impact on students
engagement with science topics and consequently, their future career
choices[23, 24].

The intervention used in this paper was designed as a single, one-off session
to introduce students to the subject of GR and to explore gravitational
phenomena in the realm of GR. While other works have used participants from a
single class, a single year group or a particular Key Stage (KS), we offered
this intervention to all physics students in the school. A total of 183
students from years 7-13, choose to take part in this study, the breakdown of
which is shown in Table 1.

Table 1: Number of participants in this study by year group. Key Stage | Year Group | Number of Students
---|---|---
KS3 | 7 | 59
8 | 41
9 | 24
KS4 | 10 | 20
11 | 13
A-Level | 12 | 12
13 | 14
Total | 183

Surveying students from one single school allows us to see how students
understanding of gravity changes throughout the educational years via this
cross-sectional study and offers the opportunity for longitudinal studies in
the future following specific cohort(s). Gravity is taught in England at KS3
(11-14 years old), KS4 (14-16 years old) and A-Level (16-18 years old),
therefore, any misconceptions picked-up at KS3 have the potential to carry
forward and impact a students understanding at A-Level.

###  2.3 The Intervention

The intervention implemented in this work consisted of 33 identical 50 minute
workshops in the presence of 8 - 30 students from a single year group. The
same researcher conducted all sessions, covering the same material and
activities.

The intervention began with students being asked to describe gravity in their
own words before GR was introduced. Table 2 shows the order that topics were
introduced, the material spoken about and where any hands-on activities
involved tie-in (highlighted in blue).

Table 2: An outline of the different topics and concepts covered during our
intervention.

Topic Details Newton’s Law of Universal Gravitation Every particle attracts
every other particle in the universe. Gravitational field lines Field lines on
the surface of and around celestial bodies. $E=mc^{2}$ Relation between mass
and energy in Special Relativity, leading to relation between mass and gravity
in GR. General Relativity Space, time and spacetime curvature. Spacetime
simulator Gravitational force in GR Force increase with curvature. Triangles
on balloons Geometry in GR GPS corrections for Earth’s curvature. Positive and
negative curvature. The curvature of the universe.

####  2.3.1 The Spacetime Simulator

A common analogy for the for curvature of spacetime uses a sheet of stretchy,
elastic material such as Lycra, and some different massed objects. This
particular analogy is commonly referred to as ‘the spacetime simulator’[25]
and has proven to be effective at dispelling misconceptions around
gravity[26].

For our demonstration, a large Lycra sheet was stretched flat across a plastic
frame as seen in Figure 1. Three students were then invited forward to hold
the sheet in the air and it was explained to the group that the sheet
represents spacetime in the flat Euclidean plane (as per the Newtonian model).
Students were then invited forward to roll tennis, squash, and golf balls
across it, perceiving that they travel in straight lines and that mass does
not affect motion in flat space. A mass heavy enough to deform the sheet was
then placed on it and students observed that the balls now roll towards the
central mass much like a ball rolling towards the bottom of a hill. By adding
more mass to the centre of the sheet and increasing the curvature, students
observe that the balls roll towards the central mass much quicker than before.
This opens up a discussion around mass, curvature and gravitational
attraction.

Additionally, while the sheet is deformed, squash balls were given some
sideways velocity. Here, students observe the ball circling around the central
mass with motion analogous to an orbit. It was important here to draw
attention to the fact that the orbital motion decays due to the loss of energy
from friction between the ball and the sheet, whereas orbits in the universe
conserve energy in general and will be continuous.

To explore this further, students were shown a plot akin to Figure 1b from
Kaur et al. (2017)[27] which illustrates that increasing curvature (adding
mass to the sheet) changes the distance between two points in spacetime, thus
influencing the geometry of the surrounding spacetime.

Figure 1: The stretched sheet of Lycra used as a spacetime simulator. Three 1
kg masses placed in the centre of the sheet provide curvature.

####  2.3.2 The Geometry of General Relativity

Following the introduction to curvature, the intervention moved to exploring
the geometry of spacetime.

Pairs of students were issued with different sized balloons. Each balloon had
a triangle drawn on it and the students were tasked with carefully measuring
the internal angles of the triangles by placing protractors on the balloons
surface. Students then compared their results to the 180∘ of a triangle in
Euclidean space to that of one with increased curvature and how that changes
the internal angles. These results were then related to applications of non-
Euclidean geometry such as the spherical curvature of Earth, it’s effect on
lines of longitude (such as their convergence at the poles), GR corrections to
GPS and the potential geometries of the universe (hyperbolic, flat,
parabolic).

Figure 2: An example of some of the balloons used by students to demonstrate
the effects of curvature on geometry

###  2.4 Pre/Post Intervention Questionnaire

Table 3: Questions from the pre/post intervention questionnaire

Questions Pre Q1 Can parallel lines meet? Post Q2 Can the sum of the angles in
a triangle be different from 180$\circ$? Q3 What is gravity? Q4 How do objects
move in gravitational fields? Q5 Isaac Newton is famous for his laws of motion
and his law of gravity. What is Albert Einstein famous for? Q6 Does space have
a shape? What about the space around heavy objects like stars and planets? Q7
Do you prefer to learn about physics by listening to your teacher, watching
demonstrations or doing practical work? Q8 Did you enjoy learning about this
topic? What did you like/ not like? Q9 Are you interested in learning more
about gravity and general relativity? Q10 Should Einsteinian physics, like
general relativity be included in the curriculum?

Two sets of questionnaires were used in this research. One disseminated two
weeks before the intervention (pre-), and one given to students two weeks
after the intervention (post-). Following the format of related studies[20,
21, 15], the open-ended questions in the questionnaires examined conceptual
understandings and attitudes towards GR.

The pre-questionnaire comprised seven questions exploring students’
understanding of gravity, its origins, workings, their perspectives on space
shape, and opinions on physics teaching methods.

The post-questionnaire was nearly identical to the pre-questionnaire, allowing
direct comparison. It included three additional questions to gain insight into
students’ views on the intervention and Einsteinian gravity itself.

The questions themselves are open-ended and can be found in Table 3.

##  3 Results & Discussion

Figure 3: Responses to Q1. Can parallel lines meet? Table 4: Difference in pre- and post-questionnaire responses to Q1 by year group. | Pre | Post
---|---|---
| No | Yes | No | Yes
Yr 7 | 93.62% | 8.51% | 79.66% | 20.34%
Yr 8 | 100.00% | 0.00% | 75.76% | 21.21%
Yr 9 | 100.00% | 0.00% | 33.33% | 61.11%
Yr 10 | 78.95% | 21.05% | 6.25% | 75.00%
Yr 11 | 91.67% | 8.33% | 20.00% | 80.00%
Yr 12 | 91.67% | 8.33% | 0.00% | 100.00%
Yr 13 | 90.91% | 9.09% | 11.11% | 88.89%
Figure 4: Responses to Q2. Can the sum of the angles in a triangle be different from 180$\circ$? Table 5: Difference in pre- and post-questionnaire responses to Q2 by year group. | Pre | Post
---|---|---
| No | Yes | No | Yes
Yr 7 | 93.48% | 6.52% | 13.79% | 86.21%
Yr 8 | 88.89% | 11.11% | 33.33% | 63.64%
Yr 9 | 87.50% | 12.50% | 16.67% | 83.33%
Yr 10 | 84.21% | 15.79% | 6.25% | 75.00%
Yr 11 | 83.33% | 8.33% | 0.00% | 100.00%
Yr 12 | 83.33% | 16.67% | 0.00% | 100.00%
Yr 13 | 90.91% | 9.09% | 11.11% | 88.89%

(a)

(b)

Figure 5: Pre- (a) and post-questionnaire (b) responses to Q3. What is
gravity? Responses in which gravity is described as a force are shown in the
right-hand charts, with other opinions of gravity being shown in the left-hand
charts.

Figure 3 and Table 4 show the responses to Q1. The majority of participants
(93.8%) said that parallel lines cannot meet. Of the ten that said they can
(three from Yr 7, four from Yr 10 and one each from Yr 11, 12 & 13), five of
these responses (three Yr 10, one Yr 11 and one Yr 13) used the phrase ‘non-
Euclidean’ or ‘not on flat space’, thus showing good knowledge before the
intervention. It is likely that these participants have had previous exposure
to curved geometries. The total number of ‘yes’ responses moves from 6.2% to
44.2% after the intervention. This is not as much as the 11.3% to 83.6% rise
in correct responses to Q2, shown in Figure 4 and Table 5. The intervention
design contributed to this as Q2 relates to the balloon activity, whereas the
information about parallel lines also being influenced by curved geometry was
presented after this activity as part of the slideshow presentation. As such,
participants had to pay more attention to grasp this information. Nonetheless,
all year groups showed an increase in ‘yes’ responses for both Q1 and Q2 with
Yr 9, 11, and 12 having the highest increase in percentage. This is perhaps
expected as these cohorts have studied more geometry than younger students and
so are better prepared for discussion’s around non-Euclidean geometry.

These results indicate that participants comprehend information better through
visual and hands-on activities. Well-designed practical activities are
reported to increase students levels of understanding[28], with practical work
seen by students as more interesting and engaging than listening to their
teacher or watching demonstrations[29].

Table 6: Difference in pre- and post-questionnaire responses to Q3 by year
group.

Pre Post Force Spacetime Curvature Earth Force Spacetime Curvature Earth Earth
Other Earth Other Yr 7 25.53% 48.94% 0.00% 6.38% 17.24% 48.28% 1.72% 6.90% Yr
8 35.14% 54.05% 0.00% 2.70% 25.81% 51.61% 3.23% 6.45% Yr 9 29.17% 87.50% 0.00%
0.00% 38.89% 38.89% 5.56% 0.00% Yr 10 5.56% 72.22% 5.56% 0.00% 15.38% 53.85%
23.08% 0.00% Yr 11 58.33% 41.67% 0.00% 0.00% 30.00% 40.00% 10.00% 0.00% Yr 12
16.67% 75.00% 8.33% 0.00% 0.00% 42.86% 28.57% 0.00% Yr 13 0.00% 81.82% 9.09%
0.00% 0.00% 44.44% 44.44% 0.00%

Q3 is more open to predisposed misconceptions than Q1 or Q2. As seen in Figure
5, there were many different responses to this question. We have categorised
responses into those which described gravity as a force versus those which did
not. It was expected that post-intervention, participants would shift their
answers towards describing gravity as the ‘bending/warping of spacetime’. 82%
of the pre-questionnaire responses described gravity as a force, with
responses ranging from ‘the force that keeps us on Earth’, ‘the force that
stops us from floating/keeps us on the ground’ and ‘the force between
objects/masses’.

Only 1.8% of participants specified that gravity was the cause of spacetime
curvature pre-intervention, increasing to only 8.9% post-intervention. Similar
studies[26] also struggled to fully influence descriptions of gravity using
the spacetime simulator. However looking at Table 6 which shows the breakdown
of Q3 by year group, reveals that responses related to spacetime curvature
increased notably in years 10 (up 17.52%), 12 (up 20.24%), and 13 (up 35.35%).

One positive result from Table 6 is that participants associating gravity as
an Earth-bound force decreased across all year groups (except Yr 10),
particularly in Yr 8 and 11. The data indicates that the KS4 and A-Level
groups best understood the concept of gravity from spacetime curvature. The
majority of participants however either did not grasp the concepts introduced
(not indicated by the results to Q1 and Q2), or the intervention’s influence
was limited by their pre-existing classroom-based learning and opinions.

Figure 6: Responses to Q4. How do objects move in gravitational fields?

Q4 assess participants understanding of how gravity influences the motion of
objects. Presented in Figure 6, the responses to this question are extremely
varied, ranging from high level answers like ‘conic sections’ and ‘along
geodesics’ to less descriptive responses such as ‘they float’ or ‘fast’. Many
responses show an understanding of the attractive powers of gravity, as well
as knowledge of other aspects of physics such as kinetic and gravitational
potential energy. While many of these descriptions are not wholly incorrect,
they fail to properly describe the motion of objects in gravitational fields.
This is likely due to participants lack of formal education on gravitational
orbits, a topic not covered in detail until Yr 13. It is possible that
responses such as ‘around’ and ‘in circles’ are consequences of this as those
in lower year groups attempt to describe orbits without knowledge of the
correct scientific terminology. This is supported by the fact that we see a
slight increase in these descriptors post-intervention as participants watched
objects circle around masses on the spacetime simulator sheet.

Positively, phrases such as ‘they fall’, ‘towards ground/source’ and
‘pushed/pulled’ decrease post-intervention, whereas the number of participants
describing orbital motion doubles. Those describing motion in gravitational
fields as attractive also sees a significant increase post-intervention. This
again is likely due to the use of the spacetime simulator demonstration where
it was observed that objects on the sheet moved towards the source of
curvature. This is also evidenced by the added uses of terms such as ’oval’
and ’spirals’ which appear in post-questionnaire responses only.

Figure 7: Responses to Q6. Does space have a shape? What about the space
around heavy objects like stars and planets?

Pre-intervention, 58.2% of participants thought that space had no shape, often
saying it is infinite or expanding (‘no because it is infinite/expanding’) to
justify their response. This number decreases to 42.4% post-intervention,
indicating that the intervention had limited success in conveying the geometry
of space around celestial objects. Nonetheless, Figure 7 reveals an increase
in responses other than ‘yes’ or ‘no’ post-intervention with the addition of
more conceptualised answers like ‘the shape changes with gravity’ or ‘it
stretches’ demonstrating an understanding of the ideas presented using the
spacetime simulator. 9.2% of post-intervention participants shown deeper
understanding by correctly noting that space is curved around masses but flat
otherwise. Coupling these responses with the other ‘yes’ responses gives a
total percentage of conceptually correct answers of 58.9%.

Figure 8: Responses to Q9. Are you interested in learning more about gravity
and general relativity?

Figure 8 shows responses to Q9, where 77.1% of participants said they would
like to learn more about gravity and general relativity.

Figure 9: Responses to Q10. Should Einsteinian physics, like general
relativity be included in the curriculum?

The last question in our questionnaires addresses the main research question
of this work and asked ’Should Einsteinian physics, like general relativity be
included in the curriculum? A resounding 82.5% of participants said that it
should be included. Of those, 10% indicated that, if added to the curriculum,
it should be included at a specific level. 41% of those respondents said that
this content should not be taught before A-Level. The rest thought that it
should not be covered before KS4.

##  4 Conclusion

In this work, 183 students from a state-funded English secondary school
participated in a one-off intervention introducing them to General Relativity
and the concept that the gravitational force is caused by the curvature of
spacetime. This theory was introduced visually using the spacetime simulator
demonstration. Once this idea was established, participants performed an
experiment using balloons and triangles to investigate what happens to these
shapes in curved spaces.

The aim was to gauge students’ reception to GR and to investigate where these
concepts would be best placed into a school curriculum.

Results showed that 77.1% of our participants overall would like to delve
further into the subject. This is consistent with the opinions of teachers and
the public who also encourage the addition of Einsteinian physics into the
curriculum[30, 31]. Noting that the largest percentage of our participants
come from Yr 7 shows how eager young students are for opportunities to learn
about cosmology.

KS4 and A-Level students demonstrated better understanding of the presented
ideas through their responses to Q1-3 in the questionnaires. Years 11, 12 and
13 had the best improvement from pre- to post-questionnaire for Q1 and Q2,
with Yr 10 also showing a good increase in understanding for Q3. While many
participants from the other year groups liked the material covered, their
responses to Q3 showed little grasp of spacetime curvature. Years 7, 8 and 9
all showed good awareness of the deformation of triangles in curved spaces
after the intervention, only Yr 9 participants showed a grasp of the curving
of initially parallel lines.

While Einsteinian gravitational concepts are taught completely distinct from
the Newtonian gravity, it has been shown by other studies that students can
understand the ideas of GR even at primary level[18, 16, 15, 2]. While the
results to Q1 and Q2 show that participants were able to appreciate the
effects of curvature, the results of Q3 demonstrate that it would take more
than a one-off intervention to imbue a deeper understanding of GR. This result
aligns with that of other works[32]. Additionally, our intervention was not
successful at dispelling misconceptions around the gravitational force. A
multi-stage intervention may prove more beneficial for this[33].

Regarding GR and its associated geometry, our results show that this material
is better suited to the A-Level curriculum rather than lower year groups.
Therefore, we conclude that introducing GR in the English secondary curriculum
at Yr 13 aligns with students’ ability to recognize curvature as the source of
gravitational force and matches the current placement of gravity study in the
curriculum.

Further insights into the effects of teaching GR could be obtained through a
longitudinal study that follows a group or groups of students through late
primary/early secondary school to KS4 and A-Level.

Ethical Statement:

This study has been reviewed and given favourable opinion by a University of
Lincoln Research Ethics Committee. Reference Number: 12146.

The authors would like to thank the staff and students of the science
department at Queen Elizabeth’s High School for their support with and
participation in this work.

## References

  * [1] Isaac Newton, Andrew Motte and N.. Chittenden  “Newton’s Principia. The mathematical principles of natural philosophy”  New York: D. Adee, 1848
  * [2] Baldy Elise  “A New Educational Perspective for Teaching Gravity”  In _International Journal of Science Education_ 29.14, 2007, pp. 1767–1788
  * [3] J… Meyer and R. Land  “Threshold concepts and troublesome knowledge (2) : epistemological considerations and a conceptual framework for teaching and learning”  In _Higher Education_ 49.3, 2005, pp. 373–388
  * [4] A Ferreira and M Lemmer  “A fine-grained teaching activity to establish conceptual coherence in introductory physics concepts and representations”  In _Journal of Physics: Conference Series_ 1929, 2021
  * [5] S. Syuhendri  “Student teachers’ misconceptions about gravity”  In _Journal of Physics: Conference Series_ 1185.1, 2019
  * [6] Clifton Smith and David Treagust  “Not understanding gravity limits students’ comprehension of astronomy concepts”  In _Australian Science Teachers Journal_ 33.4, 1988, pp. 21–24
  * [7] Michael D. Piburn, Dale R. Baker and David F. Treagust  “Misconceptions about Gravity Held by College Students”  ERIC, 1988
  * [8] Igal Galili and Varda Bar  “Children’s operational knowledge about weight”  In _International Journal of Science Education_ 19, 1997, pp. 317–340
  * [9] S. Ruggiero, Antonio Cartelli, F. Duprè and M. Vicentini-Missoni  “Weight, gravity and air pressure: Mental representations by Italian middle school pupils”  In _International Journal of Science Education_ 7, 1985, pp. 181–194
  * [10] Rosalind Driver  “Pupils’ Alternative Frameworks in Science”  In _European Journal of Science Education_ 3.1, 1981, pp. 93–101
  * [11] Berit Bungum et al.  “ReleQuant – Improving teaching and learning in quantum physics through educational design research”  In _Nordina: Nordic studies in science education_ 11, 2015, pp. 153–168
  * [12] Magdalena Kersting, Ellen Karoline Henriksen, Maria Vetleseter Bøe and Carl Angell  “General relativity in upper secondary school: Design and evaluation of an online learning environment using the model of educational reconstruction”  In _Physical Review Physics Education Research_ 14, 2018
  * [13] Thomas A. Moore  “A General Relativity Workbook”  University Science Books, 2012
  * [14] Wadi’ D. Haddad and Milton O. Pella  “Relationship between Mental Maturity and Level of Understanding of Concepts of Relativity in Grades 4-8”  In _The Journal of Experimental Education_ 41.1, 1972, pp. 22–32
  * [15] Marina Pitts, Grady Venville, David Blair and M. Zadnik  “An Exploratory Study to Investigate the Impact of an Enrichment Program on Aspects of Einsteinian Physics on Year 6 Students”  In _Research in Science Education_ 44, 2014
  * [16] Tejinder Kaur et al.  “Teaching Einsteinian physics at schools: part 3, review of research outcomes”  In _Physics Education_ 52.6, 2017
  * [17] Y.S. Dua, D.G. Blair, T. Kaur and R.K. Choudhary  “Can Einstein’s Theory of General Relativity Be Taught To Indonesian High School Students?”  In _Jurnal Pendidikan IPA Indonesia_ 9, 2020, pp. 50–58
  * [18] Matteo Luca Ruggiero, Sara Mattiello and Matteo Leone  “Physics for the masses: teaching Einsteinian gravity in primary school”  In _Physics Education_ 56.6, 2021
  * [19] Matteo Luca Ruggiero  “Teaching Einsteinian concepts in the first years of Italian secondary school: some preliminary results”  In _The Fifteenth Marcel Grossmann Meeting_ , 2022, pp. 1329–1334
  * [20] Rahul K Choudhary et al.  “Can a short intervention focused on gravitational waves and quantum physics improve students’ understanding and attitude?”  In _Physics Education_ 53, 2018
  * [21] Tejinder Kaur et al.  “Evaluation of 14 to 15 Year Old Students’ Understanding and Attitude towards Learning Einsteinian Physics”  In _aXiv_ , 2017
  * [22] Tejinder Kaur et al.  “Determining the Intelligibility of Einsteinian Concepts with Middle School Students”  In _Research in Science Education_ 50, 2020, pp. 2505–2532
  * [23] Pooran Wynarczyk and Sarah Hale  “Improving Take up of Science and Technology Subjects in Schools and Colleges: A Synthesis Review”, Report prepared for the Economic and Social Research Council (ESRC) and the Department for Children, Schools and Families, 2009
  * [24] Sandra Laursen, Carrie Liston, Heather Thiry and Julie Graf  “What good is a scientist in the classroom? Participant outcomes and program design features for a short-duration science outreach intervention in K-12 classrooms”  In _CBE—Life Sciences Education_ 6, 2007, pp. 49–64
  * [25] Gary D. White, Tony Mondragon, David Slaughter and D.. Coates  “Modelling tidal effects”  In _American Journal of Physics_ 61, 1993, pp. 367–371
  * [26] A. Postiglione and I. Angelis  “Students’ understanding of gravity using the rubber sheet analogy: an Italian experience”  In _Physics Education_ 56, 2021
  * [27] Tejinder Kaur et al.  “Teaching Einsteinian physics at schools: part 1, models and analogies for relativity”  In _Physics Education_ 52, 2017
  * [28] Ian Abrahams and Robin Millar  “Does practical work really work? A study of the effectiveness of practical work as a teaching and learning method in school science”  In _International journal of science education_ 30, 2008, pp. 1945–1969
  * [29] Robin Millar  “Practical work”  In _Good practice in science teaching: What research has to say_ 2, 2010, pp. 108–134
  * [30] Alexander Foppoli et al.  “Public and teacher response to Einsteinian physics in schools”  In _Physics Education_ 54.1, 2018
  * [31] Nuri Balta, Ali Eryilmaz and Alandeom W. Oliveira  “Increasing the presence of Einsteinian physics in high school: the impact of a professional development program on teacher knowledge and practice”  In _Teacher Development_ 26, 2022, pp. 166–188
  * [32] Rahul Choudhary et al.  “Einsteinian Physics in the Classroom: Integrating Physical and Digital Learning Resources in the Context of an International Research Collaboration”  In _The Physics Educator_ 01, 2019
  * [33] Rahul Choudhary et al.  “A Comparison of Short and Long Einsteinian Physics Intervention Programmes in Middle School”  In _Research in Science Education_ 52, 2022, pp. 305–324
  * [34] Fanny Seroglou and Panagiotis Koumaras  “The Contribution of the History of Physics in Physics Education: A Review”  In _Science Education and Culture: The Contribution of History and Philosophy of Science_ Springer Netherlands, 2001, pp. 327–346
  * [35] Igal Galili  “History of Physics as a tool for teaching”  In _Connecting Research in Physics Education with Teacher Education_ , 2008
  * [36] David E Rowe  “Einstein and relativity: What price fame?”  In _Science in Context_ 25 Cambridge University Press, 2012, pp. 197–246
  * [37] E.. Brown and T… Bryce  “Thought-Experiments About Gravity in the History of Science and in Research into Children’s Thinking”  In _Science & Education_ 22, 2012
  * [38] Athanasios Velentzas and Krystallia Halkia  “The Use of Thought Experiments in Teaching Physics to Upper Secondary-Level Students: Two examples from the theory of relativity”  In _International Journal of Science Education_ 35.18, 2013, pp. 3026–3049
  * [39] Andrej Vidak, Iva Movre Šapić and Vanes Mešić  “An augmented reality approach to learning about the force of gravity”  In _Physics Education_ 56, 2021

##  Appendix A Supplimentary Results

Table 7: Number of questionnaire responses

| Pre | Post
---|---|---
Yr 7 | 47 | 59
Yr 8 | 37 | 33
Yr 9 | 24 | 18
Yr 10 | 19 | 16
Yr 11 | 12 | 10
Yr 12 | 12 | 7
Yr 13 | 11 | 9
(a) Number of responses by year group

| Pre | Post
---|---|---
Q1 | 162 | 147
Q2 | 161 | 147
Q3 | 161 | 146
Q4 | 131 | 124
Q5 | 147 | 140
Q6 | 139 | 141
Q7 | 150 | 144
Q8 |  | 138
Q9 |  | 143
Q10 |  | 141
(b) Number of responses by question

(a)

(b)

Figure 10: Pre- (a) and post-questionnaire (b) responses to Q5. Isaac Newton
is famous for his laws of motion and his law of gravity. What is Albert
Einstein famous for? Responses are loosely categorised into correct (right-
hand charts) and incorrect (left-hand charts) responses.

It is recognised that teaching the history of physics can be beneficial in
increasing students understanding of material[34, 35]. Q5 in our study was
designed to probe students background knowledge of one of the most famous
faces in all of science and the person behind the theories presented during
our intervention, Albert Einstein. Easily recognisable, with his large white
hair and distinctive mustache, Einstein’s image forms that of the
stereotypical ‘mad scientist’[36] and is likely to have been seen in some
capacity by many of our participants. But, how many of those familiar with
what the man looks like are also familiar with his works? That is the goal of
Q5, the results of which are presented in Figure 10. Here, questionnaire
responses are loosely sectioned into correct ($E=mc^{2}$, General and Special
Relativity etc.) and incorrect answers (light-bulb, electricity etc.).

Even pre-intervention, it can be seen that many participants were already
familiar with Einstein’s famous equation, $E=mc^{2}$, with a small amount also
familiar with the theory of relativity. It must be noted however, that during
the intervention few participants were able to define the terms of this
equation or knew what it related to.

62.5% of those with incorrect responses associated Einstein with the light-
bulb or electricity. It is difficult to know where these misconceptions
originated from. An argument could be made for Einstein’s discovery of the
photoelectric effect or his use of the speed of light in $E=mc^{2}$, but these
details were not present in the responses of these participants.

Post-intervention, correct responses about Einstein increase from 48.6% to
90.7%. Although some participants have missed the mark with responses such as
’advancing Newton’s theory’ or ’discovering gravity’.

Figure 11: Responses to Q7. Do you prefer to learn about physics by listening
to your teacher, watching demonstrations or doing practical work?

Q7 of our questionnaire, shown in Figure 11, asked participants about their
preferred method of learning physics and was included in both pre- and post-
questionnaires for consistency only. It was not expected that there would be
much difference in responses between questionnaires considering the short time
between them. A large majority of our participants enjoy learning physics via
practical work with the next most popular response being a mixture of teacher
demonstrations and practical activities. This was a consistent theme
throughout the year groups. Interestingly, the use of alternative techniques
such as thought-experiments[37, 38] and augmented reality technology[39] have
been successful in engaging students with gravity and GR lessons.

Figure 12: Responses to Q8. Did you enjoy learning about this topic? What did
you like/ not like?

Shown in Figure 12, 84.6% of participants said they enjoyed learning about the
curvature of spacetime and curved geometries. Of those responses to Q8 which
indicated which aspects of the intervention they enjoyed, 50% enjoyed the
spacetime simulator most, 36.4% enjoyed the practical with the balloons with
the rest enjoying being introduced to new and ‘interesting’ material.