# Longitudinal Sentiment Analyses for Radicalization Research

a Discussion Paper by Dennis Klinkhammer

### Abstract

_Sentiment analysis is a sub-discipline in the field of natural language
processing and computational linguistics and can be used for automated or
semi-automated analyses of qualitative data. The primary context of
application is to recognize an expressed attitude as positive or negative as
it can be contained in qualitative data, such as comments on social media
platforms. However, a cross-sectional perspective regarding sentiments within
social media comments has proven to be error prone when it comes to the
detection of radicalization, extremism and hate speech. Since these phenomena
are processes over time, there seems to be an increasing demand for
longitudinal perspectives in the context of radicalization research. This
discussion paper demonstrates how longitudinal sentiment analyses can depict
intertemporal dynamics on social media platforms, what challenges are inherent
and how further research could benefit from a longitudinal perspective.
Furthermore and since tools for sentiment analyses shall simplify and
accelerate the analytical process regarding qualitative data at acceptable
inter-rater reliability, their applicability in the context of radicalization
research will be examined regarding the Tweets collected on January 6th 2021,
the day of the storming of the U.S. Capitol in Washington. Therefore, a total
of 49,350 Tweets will be analyzed evenly distributed within three different
sequences: before, during and after the U.S. Capitol in Washington was
stormed. These sequences highlight the intertemporal dynamics within comments
on social media platforms as well as the possible benefits of a longitudinal
perspective when using conditional means and conditional variances.
Limitations regarding the identification of supporters of such events and
associated hate speech as well as common application errors will be
demonstrated as well. As a result, only under certain conditions a
longitudinal sentiment analysis can increase the accuracy of evidence based
predictions in the context of radicalization research._

### Keywords

Sentiment Analysis, Natural Language Processing, Social Media, Radicalization
Research

## (I) Introduction

Sentiment analyses can be used to capture the sentiment in qualitative data,
such as comments on social media platforms. Within comments on social media
platforms the polarity can be classified word by word as positive or negative
and in some cases as neutral via basic sentiment analysis. In addition,
different types of emotional states can be classified via advanced sentiment
analysis by using the NRC Word-Emotion Association Lexicon, the first word-
emotion lexicon with eight basic emotional states (Mohammad 2020). Despite
positive and negative sentiments, it is possible to classify anger, fear,
anticipation, trust, surprise, sadness, joy, and disgust as well.

Furthermore, sentiment analyses have a sufficient inter-rater-reliability with
less time requirements. The inter-rater-reliability is a degree of agreement
among independent observers who rate, code, or assess the same phenomenon
within qualitative data. While scientist usually achieve an inter-rater-
reliability up to 80%, sentiment analyses can achieve up to 70%. This appears
to be an acceptable value, because even if different types of sentiment
analyses would agree up to 100%, research indicates that scientists would
still disagree by 20% (Ogneva 2010). Therefore, sentiment analyses can be
found in a broad application context. However, the question regarding their
accuracy in the context of radicalization research has not yet been adequately
answered, but sentiment analyses are nevertheless increasingly used regarding
related phenomena on social media platforms (Torregrosa et al. 2022).

Especially social media platforms like Twitter offer several users a low-
threshold opportunity to exchange opinions and experiences. For example, these
opinions and experiences can affect various areas of society, such as
political and economical ones. Research indicates that, for example, comments
on social media platforms can be used to capture social issues like
radicalization and extremism (Tanoli et al. 2022), sexuality (Wood et al.
2017), side effects of medication and drugs (Korkontzelos et al. 2016) as well
as for the reflection of the offline political landscape (Tumasjan et al.
2010). A well known use-case is the political campaign of former U.S.
president Barack Obama, who used sentiment analysis back in 2012. There are
many possible use-cases, but also numerous challenges in the application of
sentiment analyses. Accordingly, research in this area continues (Klinkhammer
2022; Hamborg & Donnay 2021).

## (II) Theoretical Background

Social media platforms like Twitter have demonstrated a continuous increase of
active users over the most recent years (Pereira-Kohatsu et al. 2019). An
average of 500 million Tweets per day combined with a low threshold regarding
the participation leads to a high diversity of opinions (Koehler 2015). As a
result, Twitter is not to be interpreted as one singular social network, but
as several social sub-networks, who enable users to exchange information with
each other.

Some of these sub-networks are so-called echo chambers (Bright 2017). Echo
chambers can arise through an accumulation of thematically related Tweets,
replies, likes and followers. Since Twitter as social media platform allows
its users to switch quickly and uncomplicated from one social sub-network into
another (Prior 2005), it is to be assumed that echo chamber are most likely to
arise and unfold their dynamics. As a result, users usually participate within
echo chambers, which correspond with their own opinion and the so called echo
arises. Within an echo chamber the own opinion can be confirmed and this
confirmation bias can lead to distortions regarding the perception of social
phenomena outside a social media platform like Twitter (Cinelli et al. 2021;
Jacobs & Spierings 2018).

It has already be confirmed that these confirmation biases within echo
chambers with a political agenda can lead to a gradual accumulation from
radical to extreme to anti-constitutional opinions (O’Hara & Stevens 2015).
According to Neumann (2013), extremism is a context-specific term and must be
compared and adapted to the accepted socio-political realities of the observed
society. Extremism emerges from the process of radicalization and can be
divided into cognitive and violent extremism (Neumann et al. 2018): It can
also be stated that extremism is characterized by a willingness to act in
order to endanger life, freedom and rights of others.

Furthermore, these echo chambers enable users to perform a continuing
defamation of dissenters and in some cases these defamation strategies follow
the aim of political influence (Glaser & Pfeiffer 2017). This negative
communication is called hate speech and aims at the exclusion of single
persons or groups of persons, because of their ethnicity, sexual orientation,
gender identity, disability, religion or political views (Pereira-Kohatsu et
al. 2019; Warner & Hirschberg 2012). According to Kay (2011) and Sunstein
(2006), extremist networks show a low tolerance towards individuals and groups
who think differently and are generally less cosmopolitan.

As a result of these echo chambers, hate speech as well as radicalizing
elements show an increasing number on social media platforms (Reichelmann et
al. 2020; Barberá et al. 2015). Therefore, Twitter is often accused of being a
platform for polarizing, racist, antisemitic or anti-constitutional content
(Awan 2017; Gerstenfeld et al. 2003). This content is usually also freely
accessible to children and young people (Machackova et al. 2020). It could now
be hypothesized that certain messages will get more attention, even if only a
small minority of activists uses the echo chambers accordingly.

However, such social media elements could also be used to investigate
communications patterns inside social networks and social sub-networks, in
order to focus the role of individual users and what influence the content of
their comments and actions might have on the underlying structures of a social
network (Klinkhammer 2020). Therefore, social media platforms could also
become a sensor of the real world and provide important information for
criminological investigations and predictions (Scanlon & Gerber 2015; Sui et
al. 2014). Corresponding research papers have been published recently
(Hamachers et al. 2020) and five studies represent the scientific efforts
regarding the identification of hate speech and extremism on Twitter
(Charitidis et al. 2020; Mandl et al. 2019; Wiegand et al. 2018; Bretschneider
& Peters 2017; Ross et al. 2017).

Some of these research papers refer to mathematical and statistical methods in
order to identify hate speech as well as radicalizing elements. Regression
models and classification models are most commonly used in machine learning
based approaches and a few approaches are based on deep learning via logistic
regression models as basis for simple neural networks (Schmidt & Wiegand
2017), whereas more sophisticated approaches make use of convolutional neural
networks as well as sentiment analyses (Hamachers et al. 2020).

While methodologically it is feasible to count the number of hate speech and
radicalizing elements and, for example, to study the impact of anti-hate laws
regarding social media platforms by using semi-automated and merely
descriptive approaches (Wienigk & Klinkhammer 2021), the process of automated
identification without human supervision has proven to be error-prone. For
example, mean values and variance values as used as reference values in many
of these approaches only lead to a correct identification in the short term
(Klinkhammer 2020). However, the same approaches can lead to false positive or
false negative results when conducted again at a later point in time. The
decisive factor could be the intertemporal dynamics on social media platforms,
which also could impact approaches based upon sentiment analyses (Grogan
2020). Accordingly, in respect to the changing size and topics of echo
chambers over time and considering that radicalization is a process, a
longitudinal perspective seems recommended (Greipl et al. 2022). As an
intermediate result, two hypotheses can be derived for this discussion paper:

  1. (1)

Patterns of communication on social media platforms can be subject to
intertemporal dynamics.

  2. (2)

Phenomena in the context of radicalization research can be superimposed by
intertemporal dynamics.

Therefore, it seems necessary to consider these hypotheses, not only to
perform a longitudinal sentiment analyses, but also in order to implement the
outcomes of sentiment analyses as independent variables for evidence based
predictions in the context of radicalization research.

## (III) Software Requirements

This discussion paper bases upon previously published working papers and
tutorials on arXiv: _Analysing Social Media Network Data with R: Semi-
Automated Screening of Users, Comments and Communication Patterns_
(Klinkhammer 2020) and _Sentiment Analysis with R: Natural Language Processing
for Semi-Automated Assessments of Qualitative Data_ (Klinkhammer 2022). Both
are based on R, an object based programming language. Therefore, datasets,
variables, cases, values as well as functions can be applied as a combination
of objects. This longitudinal approach for sentiment analyses requires six
additional packages in order to expand the range of basic R functions, which
are listed here in order to make the requirements and the process of data pre-
processing and the subsequent analysis more transparent:

  1. (1)

Since the analysis of comments on social media platforms requires a focus on
every single element that is to be analysed, it is necessary to break down the
underlying data structure into manageable little pieces. A package that is
specifically designed to do so is called _dplyr_. It can split, apply and
combine qualitative data for further analytical steps (Wickham 2022).

  2. (2)

The second package is called _stringr_. Since qualitative data, like social
media comments, is represented by character variables in R, a package that can
process and - if necessary - manipulate individual characters within the
strings of a character variable is required (Wickham 2019); A string is marked
either by single quote signs or double quote signs.

  3. (3)

Another necessary package is called _textdata_. It contains several words as
references and sentiment libraries, such as the NRC Word-Emotion Association
Lexicon (Mohammad 2020), as well as the Bing Sentiment Lexicon (Hu & Liu
2004).

  4. (4)

Sentiment analysis is a text mining technique and the package _tidytext_ is
required in order to convert conventional qualitative data into tidy formats,
such as single words without punctuation or spaces (De Queiroz et al. 2022).
This allows scientists to focus on paragraphs or otherwise separated content
word by word. As a result, the tidy text format lists and counts all words
individually and assigns them a number according to the original Tweet.

  5. (5)

The package _tidytext_ provides a connection between most commonly used
packages like _dplyr_ and _ggplot2_ by using their basic formulas and
commands. The latter is responsible for the detailed visualisation of
sentiments and other types of results, based on “The Grammer of Graphics”
(Wickham et al. 2022). In particular, defining the details of a visualisation
enables scientists to create informative as well as attractive plots.

  6. (6)

Finally, the package _gridExtra_ enables scientists to arrange multiple
visualizations at once and to create dashboards for an intuitive display of
relevant information (Auguie & Antonov 2017).

The previously mentioned working papers and tutorials provide a step by step
guide through the process of data pre-processing as well as the analysis.
Furthermore, the process of collecting data regarding social media networks
and comments is demonstrated as well.

## (IV) Dataset and Data Pre-Processing

Data pre-processing is used to check datasets for irrelevant and redundant
information present or noisy and unreliable data. In a first step, 49.350
Tweets from January 6th 2021, the day the U.S. Capitol in Washington was
stormed, have been transformed into a tibble. Tibbles are plain and simple
datasets that can be processed by R. Furthermore, sentiment analyses require a
tidy data set. In a tidy data set, each word within a Tweet is separated
without losing connection to related words in the same Tweet. As a result, the
tidy dataset can be described as a listing of all words within different
Tweets. This allows the sentiment analysis to be performed word by word for
each Tweet. Since non-essential words, such as _and_ or _is_ , may be included
in this list, they need to be eliminated. In addition, custom words and
duplicates can also be excluded from this object.

In a final step of data pre-processing, the dataset (49.350 Tweets) will be
splitted into three evenly distributed sequences: Before (16.450 Tweets),
during (16.450 Tweets) and after (16.450 Tweets) the storming of the U.S.
Capitol in Washington. This subdivision is necessary because each point in
time is marked by different contextual conditions which might have impact on
the underlying dynamics of Tweets and thus for intertemporal sentiment
analyses. For example, if there is a strict social norm about a topic at a
given point in time, one might expect smaller variance than when there are
weak norms about a topic. However, social norms can vary over time and in
relation to acute events. Or one might observe that people who highly identify
with a group have a smaller variance than those who are weakly identified with
that group. Again, it must be taken into account, that the identification with
groups might be subject to temporal fluctuations as well.

## (V) Methodological Approach

In order to highlight the dynamics regarding the appearance of sentiments and
emotional states over time, a smoothed slope will be used. A smoothed slope
does not represent actual values, e.g. mean values, but expected values based
upon statistical modelling, e.g. conditional mean values. The conditional
expectation can be either a random variable, noted as $E[X|Y]$, or a function,
noted as $E[X|Y=y]$, where $Y$ respectively $Y=y$ represent the conditions. If
a random variable can take on only a finite number of values, the conditions
are that the variable can only take on a subset of those values which the
slope of the function would depict accordingly. One of the advantages of
expected values is their robustness to statistical outliers. Readers
interested in the history of statistics will recognize parallels to the
underlying work of Pierre-Simon Laplace, which was formalized by Andrey
Kolmogorov using the Radon-Nikodym theorem (Feller 1991).

Applying conditional expectations for longitudinal sentiment analyses is
necessary, because the slope of a function based on actual values could be
irratic due to sporadic occuring values. Nevertheless, only few longitudinal
sentiment analyses consider conditional values (Jacobs & Spierings 2018),
although this method was suitable to depict the dynamics of Tweets regarding
politicians from populist parties. Considering the large amount of Tweets
(here: 49.350) that are to be analyzed, sporadic occuring values are to be
expected and will be compensated by applying conditional expectations.
Conditional expectations will be calculated for each sequence of Tweets (here:
16.450) to establish intertemporal contextuality. It is to be assumed that
applying more small-stepped sequences could increase the accuracy of the
analysis further. The accuracy refers to the identification of individual data
points that could be relevant in the context of radicalization research, e.g.
radical comments regarding the storming of the U.S. Capitol in Washington. The
analysis will highlight that conditional means and conditional variances are
more suitable for that task than mean values and variance values.

In the actual analysis, not only the number of used sentiments over time is to
be analyzed, but also their summative score. The first analytical step focuses
the conditional mean values and conditional variance values regarding the
number of used sentiments over time, for which the NRC Word-Emotion
Association Lexicon is a suitable basis (Mohammad 2020). The NRC Word-Emotion
Association Lexicon differentiates between positive and negative sentiments as
well as eight emotional states: Anger, fear, anticipation, trust, surprise,
sadness, joy, and disgust. Since a Tweet can consist of zero up to several
sentiments, the expected values must be greater than or equal to zero within
each sequence of Tweets. Another way of analysing the intertemporal use of
sentiments and emotional states is based upon summative scores. A sentiment
score results from the sum of positive (_1_) and negative (_-1_) values in
respect to the underlying sentiment. Since the NRC Word-Emotion Association
Lexicon from the first analytical step differentiates between positive and
negative sentiments as well as eight emotional states, a validating lexicon
can be added that differentiates only between positive and negative
sentiments: The Bing Sentiment Lexicon (Hu & Liu 2004). For each Tweet with
sentiments the score of these sentiments is assigned as sum of all positive
(_1_) and negative (_-1_) sentiments and the conditional mean values and
conditional variance values will be plotted as smoothed slope again. Thus, the
number of used sentiments over time is to be analyzed via smoothed slopes and
based upon the NRC Word-Emotion Lexicon as well as their summative scores
based upon the Bing Sentiment Lexicon, each applied on all three sequences of
the dataset in order to frame the intertemporal dynamics. The possibility of a
specific identification of individual comments on social media platforms will
be demonstrated by comparing the values of means and variances within each
sequence with their expected values of the smoothed slope.

## (VI) Intertemporal Use of Sentiments

Focusing the intertemporal use of sentiments is supposed to reveal the
intertemporal dynamics within Tweets. Each sequence will compare the mean
values (red line) against the conditional mean values (blue slope). As can be
seen, the intertemporal dynamics of each sequence would not be represented by
one mean value for each sequence, whereas conditional means vary clearly below
and above average. Furthermore, two important points for such analyzes are
revealed: Intertemporal dynamics, as can be seen by focusing the below and
above average fluctuations, tend to be small-stepped. For example, the mean
values between the three sequences vary from 1.56 to 1.70 and finally 1.50. In
the present data, the second sequence stands for the storming of the U.S.
Capitol in Washington. Such an event is reflected in the intertemporal
dynamics by an increase in the use of sentiments. The conditional means
fluctuate a little bit more and run below and above the mean values, covering
a below average use of sentiments as well as an above average use of
sentiments. Corresponding analyzes must therefore be very sensitive in order
to cover these fluctuations. Based on these sequences, it can now be examined
which topic is depicted on Twitter with which intertemporal use of sentiments.
Finally, individual Tweets and their positioning along the smoothed slope can
be identified in order to specify the patterns of communication.

The first sequence is determined by two general topics: The necessity of
measurements regarding the COVID-19 pandemic as well as the elections won by
the Democratic Party in the federal state of Georgia. The lowest values of the
slope inidcate COVID-19 as dominant topic, whereas the highest values
represent the point in time the results of the elections in Georgia and their
confirmation were publicly announced. It becomes clear that the ongoing
COVID-19 pandemic as topic is characterized by a lower use of sentiments on
Twitter than the elections in Georgia. An example for that specific event and
the associated intertemporal dynamics is a Tweet that can be located at the
local maximum of the slope ([a] Tweet 8.022): _“jon ossoff will stand for all
of georgia in the fight for healthcare jobs justice and our nation will be all
the better for having him congratulations”_. However, when it comes to the use
of sentiments, differences between democratic and republican voters can hardly
be identified within the first sequence. Pro-democratic as well as pro-
republican voters vary regarding the use of sentiments only within the
confidence interval of each sequence (grey area). Furthermore, the course of
the slope, as well as the amplitude, seem also to be influenced by other
topics, as will be illustrated in the second and third sequence.

As expected, the use of sentiments increases within the second sequence and
during the storming of the U.S. Capitol in Washington. As a result, the
amplitude increases and leads to two local maxima right after each other. An
intensive use of sentiments can be monitored, either to support or to oppose
this event. As a result, when it comes to evaluating the use of sentiments
from a numerical point of view, supporters and opponents seem hardly to differ
regarding their patterns of communication. The following supportive Tweet is
representative for these developments ([b] Tweet 12.105): _“we the people love
you mr president we admire your courage and determination to listen to us and
defend our rights thank you president trump for defending the constitution
respecting us and fighting for us you are our president all 50 states are
red”_. Another Tweet represents the opponents ([c] Tweet 12.183): _“this isnt
a peaceful protest this is an attack on our democracy and domestic terrorism
to try to stop certifying elections this should be treated as a coup led by a
president that will not be peacefully removed from power”_. The majority of
Tweets speaks out against the storming of the U.S. Capitol in Washington and
the use of sentiments increases even more if people are directly affected by
the events ([d] Tweet 12.326): _“i just had to evacuate my office because of a
pipe bomb reported outside supporters of the president are trying to force
their way into the capitol and i can hear what sounds like multiple
gunshots”_. Again, all Tweets from supporters and opponents seem to vary
within the confidence interval of the sequence.

The third sequence is characterized by many soothing and thus less emotionally
charged Tweets. As a result, the intertemporal use of sentiments descends to
the lowest values of the slope. Instead of emotional outbursts, argumentative
Tweets appear on both sides. At the beginning of the third sequence some
supporters of the storming of the U.S. Capitol in Washington compose
emotionally charged Tweets, but it is important to note that they do not
affect the course of the slope ([e] Tweet 778): _“the corrupt democrats and
educational system efficiently spread their antiamerican cancer republicans
stood around like deer in the headlights you are the only one who fought it
total destruction of our democracy is ahead unless you continue fighting”_
Instead, the course of the slope is distorted by other topics, such as sexual
content ([f] Tweet 13.961): _“seems my sub is ready for a good dicking if i
get 100 retweets maybe ill upload my video of how i fucked his ass fuck
interracial bdsm slave master”_. The fact that such content has very different
patterns of communication than socio-political topics seems to shape a
corresponding intertemporal dynamic, which is illustrated by the amplitude. As
a result, individual Tweets as well as radical or extreme actors can be
overshadowed on Twitter and a case sensitive inspection seems required. The
intertemporal sentiment score can further clarify this.

## (VII) Intertemporal Sentiment Score

The same three sequences will be analyzed regarding the intertemporal
sentiment score. This is not about how many sentiments are used, but whether
they have a positive or negative connotation when it comes to their sum within
a Tweet. As a result, the first sequence shows that the COVID-19 pandemic has
more negative connotations than the outcome of the elections in the federal
state of Georgia. With a view to the first part of the analysis, this means
that fewer Tweets contain sentiments on the subject of COVID-19, but their
intertemporal score is quite negative. Overall, the mean values within each
sequence are negative, whereas the first sequence contains the weakest
negative mean value, because of the outcome of the elections in Georgia.
Again, the mean values do not cover the dynamics of the sequences, as the
amplitude of conditional means suggests.

For example, supporters and opponents of a strict COVID-19 policy slightly use
negative sentiments, as can be demonstrated for the supporters ([a] Tweet
1.717): _“when do all the right wing accounts and pundits paid by right wing
billionaires apologize for underplaying the covid virus”_. This can be shown
for the opponents as well ([b] Tweet 1.616): _“the new lockdown is primarily
about giving the police leeway to be more draconian in enforcing it expect
many more examples of wanton authoritarianism and brutality to follow”_. A
moderate, but nevertheless slightly negative use of language can be detected
by focusing the sentiment score. With regard to the outcome of the elections
in Georgia, not only more sentiments can be detected within Tweets, but these
are also significantly more positive overall, as could be shown in the
previous step of the analysis. The low confidence interval over the course of
the slope within the first sequence seems remarkable. This could mean that the
patterns of communication of supporters and opponents are similar regarding
these topics.

The second sequence confirms that the storming of the U.S. Capitol in
Washington goes along with the local minimum of the slope regarding the
intertemporal sentiment score. Accordingly, the amplitude is strongest in this
sequence. The dominant emotional state at the local minimum of this sequence
is fear, as can be demonstrated with two examples of opponents of the storming
of the U.S. Capitol in Washington ([c] Tweet 12.500): _“i am in the capitol i
am safe and my team and i are sheltering in place the president of the united
states has incited a riot that has now stormed the capitol there are rioters
roaming the halls of the capitol i saw them with my own eyes our country
deserves better”_ as well as ([d] Tweet 12.770) _“i am safe we are sheltering
in place make no mistake president trump and his enablers are directly
responsible for this violence”_. Accordingly, the lowest values of the slope
are not to be detected after the storming of the U.S. Capitol in Washington,
but almost simultaneously. Again, the intertemporal sentiment score shows a
parallel between opponents and supporters when it comes to the intensity of
shown sentiments and the underlying emotional states as well as their variance
around the slope. For example, supporters also refer to fear and trust as
emotional state and use it within their Tweets at a comparable rate ([e] Tweet
12.689): _“no matter what happens today stay strong do not waiver do not walk
in fear no matter what it looks like hold the line of faith trump is our
president”_. However, there is no difference between the Tweets of the so-
called Trumpists and other supporters of the event. Their patterns of
communication vary only in accordance with the course of the slope.

One of the most significant dynamics is revealed in the third sequence. The
confidence interval around the slope seems to increase significantly. This
does not primarily indicate that supporters and opponents after the storming
of the U.S. Capitol in Washington tend to have more differences regarding
their intertemporal sentiment score, but can rather be interpreted as a signal
for more emerging topics within the Tweets. As illustrated in the previous
step of the analysis, the intertemporal use of sentiments decreases in this
sequence. Therefore, the amplitude has decreased as well compared to the
previous sequence. This is another indication that sensitivity is required for
corresponding analyses. On the one hand, Twitter users seem to make use of
fewer sentiments, on the other hand, they seem to differ more clearly
regarding the strength of the sentiments expressed. This can be interpreted as
a slight after-effect in respect to the current events. Furthermore, the
length of the Tweets decrease and the topics return to the topics of the first
sequence, e.g. the necessity of measurements regarding the COVID-19 pandemic
in combination with the storming of the U.S. Capitol in Washington ([f] Tweet
13.987): _“not a mask to be seen at the trump sore loser rally”_. The contents
of the Tweets also seem to be devoted more to a substantive debate ([g] Tweet
14.069): _“if masks are so great why do the mask mandates show growth in the
graphs after implementation”_.

As a result, and in accordance with the first part of the analysis, events
deviating from the current events with more sentiments and emotional states
can lead to a significant distortion of the slope. This makes it harder to
spot promoters of events like the storming of the U.S. Capitol in Washington.
Likewise, it seems harder to differentiate between supporters and opponents by
relying solely on quantitative indicators.

## (VIII) Summary

The analysis has shown that differences between supporters and opponents of
relevant social affairs, like the storming of the U.S. Capitol in Washington,
can be similar in their patterns of communication. In fact, they can even be
so similar that it seems almost impossible to differentiate them solely based
upon quantitative characteristics. This similarity was determined via the
number of sentiments and emotional states used over time, but also via their
associated summative score. As a result, the values of individual Tweets vary
most likely within the conditional mean values and conditional variance
values, even if their content supports such events. Therefore, all Tweets from
supporters and opponents vary within the confidence interval of each sequence.
This is due to the fact that the intertemporal dynamics are affected by social
affairs and corresponding Tweets vice versa. The assumption that extremist
Tweets or hate speech can be identified by above-average quantitative values
would therefore be wrong. Furthermore it would be wrong to use mean values and
variance values without considering the intertemporal dynamics framed by the
context. This could result in false-positive identifications in the context of
radicalization research.

Furthermore, the analysis demonstrated that intertemporal dynamics influence
the course as well as the amplitute of the slope. This influence is not
exclusively due to social affairs or similar events. Topics with different
patterns of communication, like sexual content, can significantly influence
the intertemporal dynamics as well. Especially the permeability of social
media platforms like Twitter and the interaction between different echo
chambers could not only affect the course of the slope globally (Cinelli et
al. 2021), but also the intertemporal dynamics partially within the echo
chambers. Thereby, relevant phenomena for the context of radicalization
research can be overshadowed.

This confirms the two hypotheses that intertemporal dynamics can be traced and
obscure relevant phenomena in the context of radicalization research. As a
result, longitudinal sentiment analyses seem less suitable for the targeted
identification of individual Tweets, but more suitable for depicting a
development over time in the sum of all Tweets. This is in accordance with the
findings of Grogan (2020) as well as the suggestion made by Greipl et al.
(2022) to conduct longitudinal analyses in radicalization research. The
developments can be mapped almost in real time, which offers the possibility
for qualitative inspections of Tweets, which seems necessary. Accordingly, the
importance of qualitative perspectives was appropriately emphasized in the
anthology of Hamachers et al. (2020). Finally, the question arises, whether
the similarities found between the supporters and opponents of the storming of
the U.S. Capitol in Washington are not a result of the predefined structures
of social media platforms. As a social media platform, Twitter specifies the
same input format for all its users, whereby their patterns of communication
can be influenced as well.

## (IX) Recommendations

Taking into account the permeability of echo chambers on social media
platforms and their intertemporal dynamics, a longitudinal approach seems
necessary in order to depict process-based phenomena in the context of
radicalization research. With regard to sentiment analysis and in accordance
with current research, it can be stated that this methodological approach
seems not suitable for a precise identification of these phenomena outside
their echo chambers, either in a cross-sectional nor in a longitudinal
perspective. However, sentiment analyses seem rather suitable for depicting
the intertemporal dynamics on social media platforms in general. By doing so,
the use of several sentiment analysis dictionaries in order to validate the
findings has shown to be a beneficial factor.

It could also be shown that phenomena relevant for the context of
radicalization research do not necessarily have the strongest influence on the
intertemporal dynamics, but other phenomena could rather superimpose them.
Therefore, it would be negligent to identify radicalization, extremism and
hate speech solely on the basis of above-average quantitative values. The
distribution of these phenomena seems rather to depend on the fact that they
are within the range of conditional mean values and conditional variance
values. One could also say that these users seem to have mastered the rules of
social media platforms. As a result, in this example, the Tweets from
Trumpists, Republicans and Democrats are quite similar regarding the storming
of the U.S. Capitol in Washington. If a social event elicits increased
activity from one side, it appears to do the same for the other side.
Accordingly, the highs and lows of these intertemporal dynamics should be a
reminder for a deeper insight via qualitative research and human expertise.

## Sources

  * •

Auguie, Baptise & Anton Antonov (2017): “gridExtra: Miscellaneous Functions
for”Grid” Graphics”. Online: https://cran.r-project.org/package=gridExtra

  * •

Awan, Imran (2017): “Cyber-Extremism: Isis and the Power of Social Media”.
Society, 54(3). Online:
https://link.springer.com/article/10.1007/s12115-017-0114-0

  * •

Bache, Stefan Milton & Hadley Wickham (2020): “magrittr: A Forward-Pipe
Operator for R”. Online: https://cran.r-project.org/package=magrittr

  * •

Barberá, Pablo; Jost, John; Nagler, Jonathan; Tucker, Joshua; & Richard
Bonneau (2015): “Tweeting From Left to Right: Is Online Political
Communication More Than an Echo Chamber?” Psychological Science, 26(10).
Online: https://doi.org/10.1177/0956797615594620

  * •

Bretschneider, Uwe & Ralf Peters (2017): “Detecting Offensive Statements
towards Foreigners in Social Media”. International Conference on System
Sciences: http://dx.doi.org/10.24251/HICSS.2017.268

  * •

Bright, Jonathan (2017): “Explaining the emergence of echo chambers on social
media: the role of ideology and extremism”. Online:
https://arxiv.org/abs/1609.05003

  * •

Charitidis, Polychronis; Doropoulos, Stavros; Vologiannidis, Stavros;
Papastergiou, Ioannis, & Sophia Karakeva (2020): “Towards countering hate
speech against journalists on social media”. Online Social Networks and Media,
17. Online: https://arxiv.org/abs/1912.04106

  * •

Cinelli, Matteo; Morales, Gianmarco De Francisci; Galeazzi, Alessandro;
Quattrociocchi, Walter & Michele Starnini (2021): “The echo chamber effect on
social media”. Online: https://doi.org/10.1073/pnas.2023301118

  * •

De Queiroz, Gabriela; Fay, Colin; Hvitfeldt, Emil; Keyes, Os; Misra, Kanishka;
Mastny, Tim; Erickson, Jeff; Robinson, David & Julia Silge (2022): “tidytext:
Text Mining using”dplyr”, “ggplot2”, and Other Tidy Tools”. Online:
https://cran.r-project.org/package=tidytext

  * •

Feller, William (1991): “An Introduction to Probability Theory and its
Applications”. 2nd Edition. Wiley Series in Probability and Statistics. ISBN:
978-0-471-25709-7

  * •

Gerstenfeld, Phyllis; Grant, Diana & Chau-Pu Chiang (2003): “Hate Online: A
Content Analysis of Extremist Internet Sites”. Analyses of Social Issues and
Public Policy, 1. Online: https://doi.org/10.1111/j.1530-2415.2003.00013.x

  * •

Glaser, Stefan & Thomas Pfeiffer (2017): “Erlebniswelt Rechtsextremismus:
modern - subversiv - hasserfüllt. Hintergründe und Methoden für die Praxis der
Prävention”. 5. Auflage. Wochenschau Verlag. ISBN: 978-3-7344-0500-6

  * •

Greipel, Simon; Hohner, Julian; Schulze, Heidi & Diana Rieger (2022):
Radikalisierung im Internet: Ansätze zur Differenzierung, empirische Befunde
und Perspektiven zu Online-Gruppendynamiken. In: MOTRA-Monitor 2021\. ISBN:
978-3-9818469-4-2

  * •

Grogan, Michael (2020): “NLP from a time series perspective. How time series
analysis can complement NLP”. Towards Data Science. Online:
https://towardsdatascience.com/nlp-from-a-time-series-perspective-39c37bc18156

  * •

Hamachers, Annika; Weber, Kristin & Stephan Jarolimek (2020): “Extremistische
Dynamiken im Social Web”. Verlag für Polizeiwissenschaft. ISBN:
978-3-86676-671-6

  * •

Hamborg, Felix & Karsten Donnay (2021): “NewsMTSC: A Dataset for
(Multi-)Target-dependent Sentiment Classification in Political News Articles”.
Proceedings of the 16th Conference of the European Chapter of the Association
for Computational Linguistics. Online: https://aclanthology.org/2021.eacl-
main.142/

  * •

Hu, Minqing / Bing Liu (2004): “Mining and Summarizing Customer Reviews”.
University of Illinois at Chicago (Academic Press). Online:
https://www.cs.uic.edu/~liub/publications/kdd04-revSummary.pdf

  * •

Jacobs, Kristof & Niels Spierings (2018): “A populist paradise? Examining
populists’ Twitter adoption and use”. Information, Communication & Society,
22(12). Online: https://doi.org/10.1080/1369118X.2018.1449883

  * •

Kay, Jonathan (2011): “Among the Truthers: A Journey Through America’s Growing
Conspiracist Underground”. New York: HarperCollins. ISBN: 978-0062004819

  * •

Klinkhammer, Dennis (2020): “Analysing Social Media Network Data with R: Semi-
Automated Screening of Users, Comments and Communication Patterns”. Online:
https://arxiv.org/abs/2011.13327

  * •

Klinkhammer, Dennis (2022): “Sentiment Analysis with R: Natural Language
Procesing for Semi-Automated Assessments of Qualitative Data”. Online:
https://arxiv.org/abs/2206.12649

  * •

Korkontzelos, Ioannis; Nikfarjam, Azadeh; Shardlow, Matthew; Sarker, Abeed;
Ananiadou, Sophia & Graciela Gonzalez (2016): “Analysis of the effect of
sentiment analysis on extracting adverse drug reactions from tweets and forum
posts”. Journal of Biomedical Informatics, 62. Online:
https://doi.org/10.1016/j.jbi.2016.06.007

  * •

Machackova, Hana; Blaya, Catherine; Bedrosova, Marie; Smahel, David &
Elisabeth Staksrud (2020): “Children’s experiences with cyberhate”. Online:
https://www.lse.ac.uk/media-and-communications/assets/documents/research/eu-
kids-online/reports/euko-cyberhate-22-4-final.pdf

  * •

Mandl, Thomas; Modha, Sandip; Majumder, Prasenjut; Patel, Daksh; Dave, Mohana;
Mandlia, Chintak & Aditya Patel (2019): “Overview of the HASOC track at FIRE
2019: Hate Speech and Offensive Content Identification in Indo-European
Languages”. 11th Forum for Information Retrieval Evaluation. Online:
https://doi.org/10.1145/3368567.3368584

  * •

Mohammad, Saif M. (2020): “Sentiment Analysis: Automatically Detecting
Valence, Emotions, and Other Affectual States from Text”. Online:
https://arxiv.org/abs/2005.11882

  * •

Neumann, Peter (2013): “The Trouble with Radicalization”. International
Affairs, 89(4). Online: https://doi.org/10.1111/1468-2346.12049

  * •

Neumann, Peter; Winter, Charlie; Meleagrou-Hitchens, Alexander; Ranstorp,
Magnus & Lorenzo Vidino (2018): “Die Rolle des Internets und sozialer Medien
für Radikalisierung und Deradikalisierung”. PRIF Report, 9. ISBN:
978-3946459385

  * •

O’Hara, Kieron, & David Stevens (2015): “Echo Chambers and Online Radicalism:
Assessing the Internet’s Complicity in Violent Extremism”. Policy & Internet,
7(4). Online: https://doi.org/10.1002/poi3.88

  * •

Ogneva, Maria (2010): “How Companies Can Use Sentiment Analysis to Improve
Their Business”. Online: https://mashable.com/archive/sentiment-analysis

  * •

Pereira-Kohatsu, Juan Carlos, Quijano-Sánchez, Lara, Liberatore, Federico &
Miguel Camacho-Collados (2019): “Detecting and Monitoring Hate Speech in
Twitter”. Sensors, 19(21). Online: https://doi.org/10.3390/s19214654

  * •

Prior, Markus (2005): “News vs. Entertainment: How Increasing Media Choice
Widens Gaps in Political Knowledge and Turnout”. American Journal of Political
Science, 49(3). Online: https://doi.org/10.2307/3647733

  * •

Reichelmann, Ashley; Hawdon, James; Costello, Matt; Ryan, John; Blaya,
Catherine; Llorent, Vincente; Oksanen, Atte; Räsänen, Pekka & Izabela Zych
(2020): “Hate Knows No Boundaries: Online Hate in Six Nations”. Online:
https://doi.org/10.1080/01639625.2020.1722337

  * •

Ross, Björn; Rist, Michael; Carbonell, Guillermo; Cabrera, Ben; Kurowsky, Nils
& Michael Wojatzki (2017): “Measuring the Reliability of Hate Speech
Annotations: The Case of the European Refugee Crisis”. Duisburg-Essen:
University of Duisburg-Essen. Online: https://arxiv.org/abs/1701.08118

  * •

Scanlon, Jacob & Matthew Steven Gerber (2015): “Forecasting violent extremist
cyber recruitment”. IEEE Transactions on Information Forensics and Security,
10(11). Online: http://dx.doi.org/10.1109/TIFS.2015.2464775

  * •

Schmidt, Anna, & Wiegand, Michael (2017): “A Survey on Hate Speech Detection
using Natural Language Processing”. 5th International Workshop on Natural
Language Processing for Social Media. Online:
http://dx.doi.org/10.18653/v1/W17-1101

  * •

Sui, Xueqin; Chen, Zhumuin; Wu, Kai; Ren, Pengjie; Ma, Jun & Fenyu Zhou
(2014): “Social media as sensor in real world: Geolocate user with microblog”.
Communications in Computer and Information Science, 496\. Online:
http://dx.doi.org/10.1007/978-3-662-45924-9_21

  * •

Sunstein, Cass (2006): “Infotopia: How Many Minds Produce Knowledge”. Oxford:
Oxford University Press. ISBN: 978-0195340679

  * •

Tanoli, Irfan; Pais, Sebastiao; Cordeiro, Joao & Muhammad Luqman Jamil (2022):
“Detection of Radicalisation and Extremism Online: A Survey”. Online:
https://assets.researchsquare.com/files/rs-1185415/v1_covered.pdf

  * •

Torregrosa, Javier; Bello-Orgaz, Gema; Martínez-Cámara, Eugenio; Del Ser,
Javier & David Camacho (2022): “A survey on extremism analysis using natural
language processing: definitions, literature review, trends and challenges”.
Online: https://link.springer.com/article/10.1007/s12652-021-03658-z

  * •

Tumasjan, Andranik; Sprenger, Timm O.; Sandner, Philipp G. & Isabelle M. Welpe
(2010): “Predicting Elections with Twitter: What 140 Characters Reveal about
Political Sentiment”. Proceedings of the Fourth International AAAI Conference
on Weblogs and Social Media. ISBN: 978-1-57735-445-1

  * •

Warner, William & Julia Hirschberg (2012): “Detecting Hate Speech on the World
Wide Web”. Proceedings of the Second Workshop on Language in Social Media.
Online: https://aclanthology.org/W12-2103/

  * •

Wickham, Hadley (2019): “stringr: Simple, Consistent Wrappers for Common
String Operations”. Online: https://cran.r-project.org/package=stringr

  * •

Wickham, Hadley; François, Roman & Kirill Müller (2022): “dplyr: A Grammmar of
Data Manipulation”. Online: https://cran.r-project.org/package=dplyr

  * •

Wickham, Hadley; Chang, Winston; Henry, Lionel; Lin Pedersen, Thomas;
Takahashi, Kohske; Wilke, Claus; Woo, Kara; Yutani, Hiroaki & Dewey Dunnington
(2022): “ggplot2: Create Elegant Data Visualisations Using the Grammar of
Graphics”. Online: https://cran.r-project.org/package=ggplot2

  * •

Wiegand, Miachael; Siegel, Melanie & Josef Ruppenhofer (2018): “Overview of
the GermEval 2018 Shared Task on the Identification of Offensive Language”.
Saarbrücken: University of Saarland Press. Online: https://www.lsv.uni-
saarland.de/wp-content/publications/2018/germeval2018_wiegand.pdf

  * •

Wienigk, Ruben & Dennis Klinkhammer (2021): “Online-Aktivitäten der
Identitären Bewegung auf Twitter - Warum Kontensperrungen die Anzahl an
Hassnachrichten nicht reduzieren”. Forum Kriminalprävention. Online:
https://www.forum-kriminalpraevention.de/online-aktivitaeten-der-identitaeren-
bewegung.html

  * •

Wood, Ian B.; Varela, Pedro Leal; Bollen, Johan; Rocha, Luis M. & Joana
Gonçalves-Sá (2017): “Human Sexual Cycles are Driven by Culture and Match
Collective Moods”. Online: https://arxiv.org/abs/1707.03959

### Affiliations

Dennis Klinkhammer is Professor for Empirical Research at the FOM University
of Applied Sciences. He advises public as well as governmental organisations
on the application of multivariate statistics and limitations of artificial
intelligence by providing introductions to Python and R:
https://www.statistical-thinking.de (Homepage) https://github.com/statistical-
thinking (GitHub)


Cognitive control, the ability of a system to adapt to the demands of a task, is an integral part of cognition.
A widely accepted fact about cognitive control is that it is context-sensitive: Adults and children alike infer information about a task's demands from contextual cues and use these inferences to learn from ambiguous cues.
However, the precise way in which people use contextual cues to guide adaptation to a new task remains poorly understood. This work connects the context-sensitive nature of cognitive control to a method for meta-learning with context-conditioned adaptation.
We begin by identifying an essential difference between human learning and
current approaches to
meta-learning: In contrast to humans, existing meta-learning algorithms do not make use of task-specific contextual cues but instead rely exclusively on online feedback in the form of task-specific labels or rewards.
To remedy this, we introduce a framework for using contextual information about a task to guide the initialization of task-specific models before adaptation to online feedback.
We show how context-conditioned meta-learning can capture human behavior in a cognitive task and how it can be scaled to improve the speed of learning in various settings, including few-shot classification and low-sample reinforcement learning.
Our work demonstrates that guiding meta-learning with task information can capture complex, human-like behavior, thereby deepening our understanding of cognitive control.

§ INTRODUCTION

Flexibility is one of the defining features of cognition. Any intelligent organism must be able to adapt its behavior to continually changing and evolving environmental and task demands <cit.>. The processes behind such adaptability are collectively referred to as cognitive control <cit.>, and a primary goal of modern cognitive psychology and neuroscience involves understanding the mechanisms that underlie cognitive control in humans <cit.>.

A notable feature of cognitive control is the ability to derive complex rules from contextual cues <cit.>. As an example, consider a child raised in a bilingual environment with each parent speaking a different language. Upon learning that each parent speaks a different language, the child may come to expect that depending on the speaker (the context), the same object (the stimulus) will be labeled using different words (the response) <cit.>. In this manner, contextual information such as visual or linguistic cues enables adults and children alike to recognize the underlying structure of a new problem they face, which, in turn, enables them to decide on a strategy for interaction within the novel context  <cit.>.

Although it is well established that context-dependent adaptation is vital for flexible behavior, the computational mechanisms underlying how humans use contextual information to guide learning in a new situation are still poorly understood. While recent computational works have shed essential insights into understanding these mechanisms in simplified settings <cit.>, we lack computational models that can be scaled up to more realistic tasks.

In the present work, we offer a new perspective by proposing that context-dependent adaptation can be explained within a context-conditioned meta-learning framework. In standard meta-learning, a meta-learned global model determines the initialization of task-specific models, which are subsequently adapted to online feedback from each task.  Here, we propose maml-context, in which contextual cues about task structure–termed task information– guide the initialization of task-specific models, enabling the meta-learned prior over task structures to be informed by task information, similar to how human learning is guided by context.

We implement maml-context by augmenting a gradient-based meta-learning algorithm <cit.> with a context network that learns the relationship between task information and the initialization of task-specific models. We first use this implementation to demonstrate that the maml-context framework can capture the context-sensitivity of human behavior in a simple but well-studied cognitive control task. We then shift our focus to larger-scale simulations, where we demonstrate competitive performance against several baselines on supervised and reinforcement learning tasks. Our work thus contributes a framework for understanding key aspects of human adaptability and a cognitively-inspired algorithm that is competitive in realistic settings.

§ BACKGROUND

§.§.§ Computational accounts of context-specific adaptation in humans.

Although the importance of contextual cues in guiding human flexibility is well-established, very little work has looked into how contextual information guides such adaptability. Recent computational works have made progress towards understanding these mechanisms by suggesting that context-specific adaptation can be modeled using nonparametric Bayesian methods <cit.> as well as hierarchical reinforcement learning <cit.>. However, one limitation of these works is that the tasks modeled using these frameworks are relatively simple compared to the problems faced by humans. This limitation restricts our understanding of context-sensitive adaptation as we do not have models that can capture our everyday flexibility and adaptability. Despite this limitation, a critical insight from these models is that they suggest that the learning processes involved in cognitive control occur at multiple levels of abstraction in that prior knowledge and cognitive control constrain the lower-level, stimulus-response learning <cit.>. We take this insight as the motivation to pursue modeling context-specific adaptation under a meta-learning framework, which realizes an analogous hierarchical decomposition of learning.

§.§.§ Meta-learning.

Meta-learning aims to learn a model suitable for a distribution of tasks, which subsequently enables few-shot adaptation to new tasks sampled from the same distribution <cit.>, formulated in recent
works as the learning of global parameters that are shared
between independent, task-specific models <cit.>. While meta-learning algorithms can capture some elements of human adaptability (such as the ability to learn from very few examples), standard formulations of meta-learning are not sufficient to capture context-sensitive adaptation. This is because popular meta-learning approaches <cit.> and their derivatives learn in the absence of abstract task information by treating each task as a uniformly random draw from an underlying task distribution and do not use context to prime their learning.

§.§.§ Context-conditioning in meta-learning.

Recent works have explored augmenting meta-learning with conditioning information by modifying the meta-learner architectures to encode task-specific data into a latent task representation <cit.>. Analogous to the way learning loops occurring between abstract contexts and high-level rules constrain the lower-level learning loop in the brain, in these frameworks, outer learning loop between latent task representation and high-level rules constrain the inner learning loop.

However, one important distinction between context-conditioning meta-learning and context-specific human adaptation is that the former produces the task encoding using the support set using the same data over which the meta-learning objective is defined. For instance, <cit.> use a conditioning network to infer information about the task, but they do so without employing external contextual information. Similarly, <cit.> propose a meta-learning model that uses a transformation network to augment the base network with an implicit conditional network as a linear transformation on the weights but uses the same data as the base network. <cit.> also use contextual information at the instance or class level without any conditioning network. <cit.> linearly transform feature embeddings with a task-specific projection but does not employ contextual information or a conditioning network. This means that while context-conditioning meta-learning enables efficient few-shot learning, it cannot fully capture and explain context-sensitive adaptation in humans.

§ THE PRESENT RESEARCH

In this work, we consider meta-learning as a useful starting point towards modeling context-sensitive adaptation in humans. However, as noted previously, unlike humans, standard formulations of meta-learning do not employ contextual cues, and only in some cases, infer a task representation from task-specific data.

To account for human behavior, we instead propose to use contextual cues to guide meta-learning. Unlike prior works on meta-learning, we produce a task representation from the extra available contextual information and focus on the utility of this information in structuring learning at a higher level of abstraction rather than the increased expressiveness that architectural modifications bring. This structure is motivated by human learning, in which contextual cues serve to inform a prior about the task structure at hand, which then enables rapid adaptation to novel contexts. Our experiments show that this task-specific contextual-adaptation can not only capture human behavior but also improve the speed of learning of meta-learning in supervised and reinforcement learning tasks.

Our key contributions are as follows. First, to explain context-sensitive adaptation in humans, we introduce a framework that uses task information to guide meta-learning. Second, we demonstrate that our framework can successfully capture human behavior in a well-known cognitive control task. Modeling human behavior in this task allows us to understand important aspects of human flexibility and cognitive control. Third, and unusually for a cognitive modeling framework, we show that models implemented in our framework can outperform competitive baselines in more complex problem domains such as celeba and metaworld. Thus, our work also contributes towards developing a cognitively inspired meta-learning framework that can be applied to more realistic problem domains.

§ A META-LEARNING ACCOUNT OF CONTEXT-SPECIFIC ADAPTATION IN HUMANS

We now present our framework for capturing context-specific adaptation. In a standard meta-learning setup, a parametric meta-learner encodes information about the shared structure of the distribution of tasks, $p(\task)$, into a set of global parameters $\metaparams$ from which all task-specific predictors are derived. In particular, for each task $\task\taskidx{j} \sample p(\task)$, the meta-learner receives a task-specific dataset $\mathcal{D}_{j}=\left\{\mathbf{x}_{j_{i}}, \mathbf{y}_{j_{i}}\right\}$ and produces a predictive distribution $p_{\metaparams}(\hat{\mathbf{y}}_{j} \given \hat{\mathbf{x}}_{j}, \mathcal{D}\taskidx{j})$ for new examples $\hat{\mathbf{x}}_{j}$ from the same task.

Here, to capture context-sensitive adaptation, we propose to augment the standard meta-learning problem statement in a way that is analogous to the way contextual cues prime human learning in a new environment. In particular, we posit that the additional environmental contextual information, $\context_j$, can be leveraged as conditioning information in order to prime the initial state of the model $\metaparams$ for a specific task $\task_j$ (also refer to Figure <ref>). Formally, we implement conditioning on the task information $\context$ by parameterizing the initialization $\metaparams$ as the output of a context model $g$ with parameters $\contextparams$. Using experience from the task, $\metaparams$ is subsequently adapted with gradient descent to task specific parameters $\taskparams$, as in maml. In practice, we take $g$ to be a neural network with weights $\contextparams$, which we refer to as a context network, and update $\contextparams$ via back-propagation. Note that $\contextparams$ is updated only during the meta-update step and during the inner loop for task-specific adaptation, $\metaparams$ is used to initialize $\taskparams$ which is subsequently updated based on task-specific data.

§.§.§ Supervised meta-learning with task information.
We consider a family of tasks $\task$ with shared structure that enables a meta-learner to learn to solve a task from $\mathcal{T}_i\sim p(\mathcal{T})$. In the supervised learning setting, each task $\mathcal{T}_i$ consists of a set of examples $\mathbf{x}$ and annotations $\mathbf{y}$ (images with classification labels). Gradient-based meta-learning methods choose a parameterized model (base learner) and define an optimization objective over $\mathcal{T}$. For instance, the maml algorithm <cit.> uses the following objective:

\begin{equation}
\min_{\metaparams} \mathbb{E}_{\task_i}\left[\mathcal{L}_{\mathcal{T}_i} (f_{\metaparams'})\right] =  \mathbb{E}_{\task_i}\left[\mathcal{L}_{\mathcal{T}_i} (f_{\metaparams - \alpha \nabla_{\metaparams} \mathcal{L}_{\mathcal{T}_i}(f_{\metaparams}})\right]
\end{equation}

where $f$ is a parameterized function representing the base learner or policy, $\metaparams$ refers to the parameters that are optimized in the outer loop, and the $\taskparams$ parameters are used to compute the objective with respect to each task.
When employing task information, the meta-objective becomes

\begin{equation}\label{eq:maml-ctx}
\min_{\contextparams,\metaparams}\
\mathbb{E}_{\task_i \sample p(\task)}\left[
\mathcal{L}_{\mathcal{D}^\text{q}\taskidx{i}} \left(
f_{\{ g_{\contextparams}(\context_i),\metaparams - \nabla_{\metaparams}{ [\mathcal{L}_{\mathcal{D}^\text{s}\taskidx{i}}(f_{\{ g_{\contextparams}(\context_i),\metaparams\}})]
} \} }
\right)\right]~,
\end{equation}

where the principal difference is that the initial parameterization of the base network depends not only on global parameters $\metaparams$, but also task-information-dependent parameters produced as the output of $ g_{\contextparams}(\cdot)$. With this meta-objective, we can thus fully differentiate the objective with respect to $\metaparams$; we may make a further application of the chain rule to derive an update for $\contextparams$ also using the objective value at the last inner adaptation step.

§.§.§ Meta-policy search with task information.

rl assumes a mdp consisting of $(S, A, P, R, \gamma)$, and the goal is to discover a policy $\policy$ that maximizes the return $\sum_{k=0} \gamma^k R_{k+1}$, the sum of episodic rewards discounted by $\gamma \in [0, 1)$ <cit.>.
meta-rl generalizes this setting to a distribution $\mdpdistribution$ over mdps, with the aim of finding the policy that maximizes the expectation of returns under this distribution:
$\mathbb{E}_\mdpdistribution\left[\sum_{k=0} \gamma^k R_{k+1}\right]$.
Similar to the supervised scenario, we can decouple a solution to the meta-rl problem by performing an outer loop search procedure for parameters that maximized expected return across a distribution of control tasks,
\mathcal{L}_{\mathcal{T}_i} (f_{\metaparams})\right] = \mathbb{E}_{\task_i}\left[-\mathbb{E}_{(s_t, a_t) \sim q_{\mathcal{T}_i}} \big[\sum_t R(s_t, a_t)\big]\right]$
where $q_{\mathcal{T}_i}$ is the transition distribution of task $\mathcal{T}_i$, $s_t$ and $a_t$ are state and action at time $t$, respectively. The main difference from the supervised case is that we cannot explicitly differentiate through the dynamics of the environment, and so the standard approach is to use policy gradient methods to update meta-parameters $\metaparams$; we refer to <cit.> for more details.  With task information, algorithmically, updating $\psi$ and $\metaparams$ is similar to the supervised case. During the inner adaptation steps, only $\metaparams$ is updated to compute the task-specific parameters $\taskparams$. However, during the meta-update step, the gradient of the post-update objective value is used to update both $\contextparams$ and $\metaparams$, in a generalization of the maml algorithm.

§.§.§ Implementing a context-conditioning network.

Learning a function $g$ that produces a parameter initialization for a high-dimensional function $f$ such as a neural network poses problems of under-fitting and computational inefficiency.
There have been methods proposed to alleviate this issue <cit.>, all resting on the same premise (or empirical demonstration) that producing a subset of the parameter of $f$ is sufficient.
In all our large-scale experiments, we make use of the film parameterization from <cit.>; namely, the context network $g$ produces the shift and scale parameters in the hidden layers <cit.> of the base network $f$, thereby acting to linearly transform activations in the base network.

§ MODELING HUMAN BEHAVIOR

We begin by applying our proposed framework to capture human behavior in a well-known cognitive control experiment.

§.§.§ Task description.
We model our setup after the experiment in <cit.>. In their study, 8-month-old infants participated in a learning task followed by a violation-of-expectation inference test. In the learning task, infants viewed toy-word mappings that could be grouped into distinct rule sets using the faces and corresponding voices as higher-order contexts (refer to Figure <ref>). Each face-voice context labeled the toys using different words, similar to a bilingual environment in which one caregiver speaks English, and another caregiver, Spanish. Near the end of the learning task, a novel face-voice context was presented with several observed toy-word pairs and a novel toy-word pairing. This is akin to the infant observing a new caregiver introducing a new word in Spanish. During the inference test, infants were presented with the first two face-voice contexts from the learning task paired with the novel toy-word pairing presented at the end of the learning task (refer to Figure  <ref>). One of these presentations was consistent with the rule set structure formed during learning, while the other was inconsistent. Sensitivity to this contrast would demonstrate that the infant infers that the Spanish-speaking caregiver should use the novel object-label mapping introduced by the third caregiver, while the English-speaking caregiver should not. Infants looked longer at the inconsistent pairing compared to the consistent pairing, implying greater surprisal during inconsistent pairings [Looking time is a common metric used to study children's cognitive indications such as surprise and expectation violation]. If contextual cues did not help learn a hierarchical rule set, then no difference in the looking time would have been observed. Similar studies have also been undertaken with adults <cit.>, demonstrating that both adults and infants use contextual cues for faster task adaptation.

§.§.§ Experimental setup.
If our framework can capture context-sensitive adaptation, then we should be able to replicate the looking-time results from <cit.>. To test this, we created an analogous problem setting which consisted of a similar leaning task and inference test. During the learning task, we provided  tasks comprising a context, $\context \in \{0, 1, 2\}$ representing the speaker identity and two disjoint batches of stimulus-response pairs $(\mathbf{x}, \mathbf{y}) \in \{0, 1, 2\} \times \{0, 1, 2, 3, 4\}$, each representing an object identity paired with a word label. Like in the behavioral learning task, stimulus-response mappings appear only within valid contexts. Further, one of the stimulus-response pairs,
$(\mathbf{x}, \mathbf{y}) = (2, 4)$ is only presented in one context ($\context=2$) even though it is valid in another ($\context=0$). For the inference test, we create two conditions – consistent and inconsistent. In the consistent condition, the context network is presented with context $\context=0$, the produced parameters are adapted with seen examples from the context, and the adapted model's loss is evaluated on the held-out stimulus-response mapping, $(\mathbf{x}, \mathbf{y}) = (2, 4)$. In the inconsistent condition, the context network is presented with context $\context=1$, the produced parameters are adapted with seen examples from the context, and the adapted model's loss is evaluated on the held-out stimulus-response mapping, $(\mathbf{x}, \mathbf{y}) = (2, 4)$. Detailed data sampling procedure and worked-out task examples are included in the Supplementary.

§.§.§ Hyperparameters.
Both the base and context networks use a neural network with two hidden layers of size 10. Since $\metaparams$'s dimensionality is sufficiently low, the context network, which maps task information to base network weights, directly outputs $\metaparams$. For task-specific adaptation, we use one gradient update using a support set of size $10$. During inference, we present the model with $2$ support examples from a newly sampled task and measure mean-squared error over $1$ query example.

§.§.§ Results.
We compare our approach, which we term maml-context, against maml, the meta-learning method for sl in <cit.>. We hypothesize that our framework should be sensitive to the evaluation condition just like humans. Since maml-context uses the context as higher-order information, its error in the consistent condition should be lower compared to the error in the inconsistent condition (analogous to the difference in looking time/surprise in humans). On the other hand, because maml doesn't utilize contextual information, its error should not be influenced by the condition. Thus, its performance would serve as an ideal baseline to compare our framework. Note that absolute value of the validation errors are not particularly important, rather the relative difference in the validation errors across conditions is more important.

We first see that maml-context learns faster compared to maml (Figure <ref>). This is not surprising as maml-context employs the contextual information whereas maml does not. We further note that the variance in the performance of maml is quite high. Next, in Figure <ref> we see that maml-context can qualitatively reproduce the looking time results from <cit.> as the error of maml-context in the consistent condition is considerably lower than the error in the inconsistent condition ($3.65$ vs. $6.19$). A paired t-test revealed that this difference was statistically significant, $t(8) = -19.3, p < 0.001$. We further observe that as per our predictions, the baseline maml is not affected by the difference in condition as its error on the consistent condition is similar to the inconsistent condition ($3.9$ vs $3.7$). A paired t-test revealed that this difference was not significant, $t(8)  = 0.1, p = 0.5$. We also observe that the variance in the error of maml is quite high. This is partly driven by the high variance during learning – whenever maml reaches a lower error on the meta-training, it overfits on the training set (due to lack of context information) leading to a very high loss on the validation set. These results show that predictions made by our proposed framework are consistent with human behavior in a well-studied cognitive control task.

§ LARGE-SCALE EXPERIMENTS

The previous section showed that maml-context is consistent with psychological findings about context-sensitive adaptation on a controlled cognitive task. We now evaluate whether maml-context can perform competitively in more complex problem settings by guiding adaptation in meta-learning.

§.§.§ Overview of task information.
In the setting of mujoco, we explore task information as a diagnostic cue by using scalar parameter as task information. For the more challenging celeba dataset, we use a binary vector with attribute information as task information. For the metaworld tasks, we use the 3D goal position as task information.

§.§.§ Baseline comparisons.
We compare our approach of context-conditioned adaptation, maml-context, against three categories of baseline as described below. For hyperparameters that are common to all comparison methods, we use the same settings as are used in <cit.> and <cit.> where applicable.

maml is the meta-learning method for sl as described in <cit.> and promp is a policy-gradient meta-rl method that improves upon the initial application of maml to rl in <cit.> by combining maml and ppo <cit.>.
These methods make no use of task information and serve as lower bounds to task-information conditioning.

maml-static and promp-static are baselines with the same architecture as maml-context but do not depend on the context and instead replace the context $\context$ with a constant vector; this baseline is intended as a parameter count-equivalent baseline to maml-context in order to distinguish architectural differences in performance as compared to maml and promp.

maml-concat and promp-concat use the same architecture as the maml-context method but use task information in the form of concatenation to the observation; this setup is analogous to goal-conditioned RL, where policies are trained to reach a goal state that is provided as additional input <cit.>.
These baselines are provided with the same amount of information as maml-context but do not decouple context and task-specific feedback into initialization and adaptation phases, respectively, and therefore test the utility of task-information in priming meta-learning like humans do as opposed to simply being treated as extra observational information.

§.§ Ambiguous classification with celeba

§.§.§ Experimental setup.

We use a construction similar to <cit.> to generate an ambiguous binary classification task with the celeba dataset.
[We focused on the celeba dataset instead of mini, the usual dataset for the evaluation of few-shot classification methods <cit.>, as we can easily generate task descriptors.]
In particular, for each task, we sample 2 of the 40 attributes from celeba, then subsequently sample for the support set one image that contains these attributes (a positive example) and one image that does not contain these attributes (a negative example); this binary classification task is often ambiguous, as most images in celeba have more than two attributes active.
The task information is provided in the form of a two-hot vector that identifies the two attributes upon which the base network has to make a classification decision.
The query set comprises 15 examples as in the experimental setup in <cit.>.

§.§.§ Hyperparameters.

The context network pipeline embeds the two-hot task information vector via a learned embedding matrix; these embeddings are summed then fed as input to a two-layer feed-forward neural network with 40 hidden units.
As per the implementation of film-conditioning, the context network outputs a feature map that performs linear transformations to the base network's hidden activations.
The base network itself is a four-layer convolutional network with 32 filters applied at stride 2, similar to the small-scale convolution network employed in few-shot classification on the mini dataset <cit.>.
We set hyperparameters on the held-out validation set; all settings as well as details on the implementation of the context network are included in Supplementary.

§.§.§ Results.
As shown in Table 1, maml-static suffers from the need to fit the extra parameters and maml performs the task with a low degree of accuracy.
Next, we see that maml-context performs marginally better than maml-concat. These results suggests that for the highly-ambiguous few-shot celeba task, our cognitively-inspired method outperforms the context-independent method like maml while performing competitively (if not better) compared to context-concatenation method maml-concat.

§.§ Parameterized mujoco Tasks

We next comparing the above methods on simple continuous control tasks by using a set of three parameterized environments from the mujoco simulator <cit.>.
For the below results, the average return for the pre- and post-task-specific adaptation is computed from trajectories sampled before and after the inner loop for task-specific adaptation.

§.§.§ Environments.

In these environments, the underlying dynamics of the half-cheetah, walker-2d, and ant-goal environments depend on a randomly sampled scalar parameter: In half-cheetah and walker-2d, a scalar parameter controls the direction of motion (forward or backward) that is produced for a given action; for ant-goal, a randomly sampled 2D position defines a goal to which the actuator must be moved.
We use this scalar parameter as task information for this setting.

§.§.§ Hyperparameters.

For all method including the promp and promp-concat baselines, the base policy is a fully-connected network with two hidden layers of dimension $64$ and ReLU nonlinearities as in <cit.>.
For promp-context and the promp-static baseline, the base policy is conditioned with a film module; this module is fed contextual input and outputs a feature map that performs linear transformations on the policy network's hidden layers.
In our experiments, film is represented as a fully connected network with two hidden layers (of increasing dimension–32 and 64–to achieve up-sampling of the context) and outputs $W_i$ and $b_i$ for each hidden representation $h_i$ in the policy network, performing the transformation $h'_i = W_i \odot h_i + b_i$.
For other hyperparameters that are common to all four comparison methods, we use the same settings as are used in <cit.>.
In particular, the number of inner optimization steps is set to one, entailing two rollouts in the environment to evaluate the pre- and post-adaptation objectives.

§.§.§ Results.

Figure <ref> reports the post-adaptation performance of all methods.
First, task information is beneficial as promp-context consistently outperforms both promp and promp-static in all three environments.
Next, promp-context performs better than promp-concat in the half-cheetah environment and at least as well as promp-concat in walker-2d and ant-goal.
This suggests that our cognitively inspired approach of learning conditionally with the task information is a competitive parameterization compared to learning jointly (promp-concat, a standard in goal-conditioned rl setups).

§.§ metaworld manipulation tasks

We next investigate on a more challenging setting, using a set of five parameterized environments from the metaworld benchmark <cit.>.

§.§.§ Environments.

The metaworld benchmark presents a variety of simulated continuous control tasks that consist of a 4-dimensional robot actuator that is required to move from an initial position to a goal position.
In these environments, we use the goal position, a 3$\times$1 vector, as  task information; this goal position is re-sampled when a new task is encountered.
Similar to the mujoco environments, the goal position is normally treated as a direct concatenation to the state observation; we instead use the goal position as an input to context module in order to investigate the effect of context-conditional adaptation in a meta-policy search algorithm.
This use has two advantages: First, the goal position is readily available in different environments in metaworld.
Second, and more importantly, we hypothesize that goal information serves as an integral cue for reducing task ambiguity even in the various dense-reward environments in metaworld.
Furthermore, a task is defined as a fixed goal with varying initial states.
This makes these environments more challenging: rather than fixing the goal and initial state, the pre- and post-adaptation policies are evaluated with different initial states and goals.

§.§.§ Hyperparameters.

We use the same base policy and context network implementations as in the previous section.
Since metaworld environments are substantially more difficult to solve as compared to the mujoco environments, these environments needed more inner adaptation steps to show post-adaptation improvement; an inventory of hyperparameter settings are provided in Supplementary.

§.§.§ Results.

Results are shown in Figure <ref>.
We first observe that even for reach, a very simple environment, task information is necessary to perform well on the task, as evident by the superior performance of promp-context and promp-concat.
One possible explanation for this is that the reward available in reach is insufficient to guide meta-learning by itself, and that the goal information serves as a useful cue to guide meta-learning.
Additionally, we observe that in the reach environment,  context-conditioning is not especially beneficial compared to context-concatenation as both promp-context and promp-concat perform similarly on this task.

Next, we see that in both door-lock and door-unlock, task information is not necessarily crucial to perform well, as both promp-context and promp-concat perform similarly to promp.
Interestingly, the over-parameterized architecture in the promp-static method worsens the performance on the door-unlock environment.
The most interesting cases are the soccer and basket-ball environments: Here, we see that promp-context significantly outperforms all other methods.
Furthermore, we see that simply providing task-information as an extra input is not beneficial as evident from the performance of the promp-concat method on these two environments.

In summary, our proposed contextual-conditional meta-learning outperforms all the methods (including promp-concat) on both the soccer and basket-ball environments and performs as well as the other methods (if not better) in the remaining environments.
As a main takeaway, results from this experiment suggest that our cognitively inspired framework is a promising way to improve the performance of meta-learning on more challenging tasks such as metaworld.

§ CONCLUSION

An extensive literature in psychology and neuroscience demonstrates that context-specific adaptation is an integral component of cognitive control <cit.>. Here, we explain context-sensitive adaptation under a meta-learning framework that integrates task information to guide adaptation to new tasks. Our modeling results on a cognitive control task support existing theories that propose higher-order contextual information helps humans structure learning <cit.>. According to these theories, hierarchical learning based on contextual cues ensures that learning new information does not conflict with behaviors learned in other contexts; for instance, an infant in a bilingual environment receiving two different labels for the same word would not get confused when labels are consistent with the higher-order context provided by the identity of the speaker.

Our large-scale experiments further show that our cognitively inspired meta-learning framework is also a promising approach towards improved adaptation in meta-learning. Analogous to the way people use contextual cues as a prior over task structure, our framework thus highlights the value of task information in bringing meta-learning algorithms closer to human-like learning.

§ ETHICS STATEMENT

Our research contributes towards improving our understanding of cognitive control in humans as well as the field of meta-learning, which aims to emulate the ability of humans to learn new tasks rapidly. There are many benefits to such contributions, such as the development of automated systems that can quickly adapt and learn to solve a variety of tasks, although the current problem settings are simplistic as compared to the everyday variability that humans face. However, in the longer term, progress in adaptable and robust algorithms leads towards automation, which will disrupt the labor structures that many people rely on for employment.

§ EXTENDED RELATED WORK

Context-conditioning in meta-learning.

We now briefly review prior work on context-conditioned meta-learning before contrasting our contribution against these papers.

tadam <cit.> uses film conditioning to improve the performance of feature extractors for few-shot classification. However, tadam uses a very specific variable for conditioning – the mean of the class prototypes is the output from a conditional network which serves as inferred task context. On the other hand, we employ film with a variety of task-specific contextual cues and show how this can capture context-sensitive adaptation in humans. From our perspective, tadam serves as a useful starting point towards considering how context-conditioned meta-learning can be adapted to capture human-like behavior.

<cit.> learn task-specific balancing variables in order to balance between the meta-knowledge and task-specific update. However, in contrast to our work, they do not employ contextual cues.

<cit.> aim to control the influence of prior knowledge for each task and propose a method that performs selective forgetting by applying a task-dependent layer-wise attenuation on MAML initialization. This is in contrast to our proposal of utilizing the additional information provided by contextual cues to capture human behavior.

<cit.> is similar to our work from an architectural point of view as it employs a modulation network that produces a task embedding which is used to generate parameters that modulate the task network. However, the key difference is that while <cit.> generates the parameters by identifying the mode of tasks sampled from a multi-modal task distribution, we generate the parameters by utilizing contextual information. Future work could investigate the benefits of generating the parameters by utilizing both the multi-modal distribution as well as the auxiliary contextual information.

<cit.> show that using image tags as auxiliary information helps to learn a better representation for prototypical networks, enabling better generalization to unseen tasks. <cit.> learn both an image classifier and a text-conditioned image generator as a pre-training step; the generator is then used to provide auxiliary data during the few-shot adaptation stage. Both these approaches use contextual information at the instance or class level; in contrast, we operate over task-specific contexts, thus enabling to model human behavior.

<cit.> is based on the idea that task-specific learning should require fewer degrees of freedom compared to meta-learning and proposes a meta-learning model that determines a subspace and a corresponding metric that task-specific learners can learn in. This is in contrast with our main idea of contextual adaptation.

<cit.> linearly transform the network output with a task-specific projection; whereas we use contextual information to initialize the meta-learner.

<cit.> learn a policy that adapts to the task at hand by performing inference over a latent context variable on which the policy is conditioned. Here, context is defined as the history of past transitions, which is orthogonal to our setting of using the extra available contextual cues (and not the history of past transitions) to prime learning. Further, they do not investigate priming learning with context variables.

Lastly, the work of <cit.> uses auxiliary contextual information to constrain adaptation which makes it closer to our proposed method. However, while <cit.> perform task-specific parameter estimation in a linguistically structured latent space, we condition on arbitrary task information before interaction with a task, therefore combining more flexible adaptation of task-specific models with guidance provided by arbitrary (beyond linguistic) context.

Other uses of context. In addition to context-conditioned meta-learning, there has been a wide variety of work that study the utility of contextual information in decision-making. In the supervised setting, the use of descriptions or tags as extra inputs improves fine-grained image classification <cit.> and zero-shot learning <cit.>. Contextual information has also been used in sequential decision-making in the form of instruction following <cit.>, to guide learning of reward functions <cit.> and environment models <cit.>, or for better exploration <cit.>. While these methods make use of contextual information, they do so in parallel with concept or policy learning and usually do not deal with few-shot settings. This is analogous to the concat baseline used in our experiments and therefore cannot capture context-specific adaptation in humans. Here, we use contextual information to guide the initialization of task-specific parameters, followed by few-shot adaptation using feedback from the target task; this ordering enforces the use of the task information as a prime for interaction with the target task, similarly to context-sensitive adaptation in humans.

§ ADDITIONAL EXPERIMENTAL DETAILS

§.§ Modeling human behavior

For the cognitive modeling experiment, we report the average of five seeds.

During the learning task, to reproduce the behavioral task of <cit.>, we provided  tasks comprising a context, $\context \in \{0, 1, 2\}$ and two disjoint batches of stimulus-response pairs $(\mathbf{x}, \mathbf{y}) \in \{0, 1, 2\} \times \{0, 1, 2, 3, 4\}$, where each stimulus-response mappings appeared only within valid contexts. Table <ref> presents the training data sampling procedure in detail.

The hyperparameters are provided below. Further details can be determined by inspecting the attached code that reproduces all of our results ().

Cognitive     Hyperparameters

Gradient Clip Norm     10.0

Inner Loop Learning Rate     $0.1$

Outer Loop Learning Rate     $0.005$

Number of Meta-training Steps     $100$

Number of Inner Adaptation Steps     1

§.§ celeba

The hyperparameters for the celeba experiments are provided below.
Note that <cit.> hold out entire attributes at meta-test time, while we hold out combinations of attributes; our setup therefore treats the celeba attributes similarly to natural language descriptions with no unobserved vocabulary. An interesting next step would be to add in a component that extrapolates the context network to be applied to out-of-vocabulary items <cit.>.
Further details can be determined by inspecting the attached code that reproduces all of our results ().

celeba     Hyperparameters

Gradient Clip Norm     10.0

Inner Loop Learning Rate     $0.01$

Outer Loop Learning Rate     $0.001$

Number of Meta-training Steps     $10^{4}$

Number of Inner Adaptation Steps     5

Meta-batch Size     4

§.§ Reinforcement learning experiments

For all rl experiments, we report the average over three seeds. The hyperparameters for mujoco and metaworld are provided below.
Further details (and the environment-specific horizon length) can be determined by inspecting the attached that reproduces all of our results ().

mujoco     Hyperparameters

Clip Parameter     0.3

Discount ($\gamma$)     0.99

Lambda ($\lambda$)      1.0

KL Coeff     0.0

Learning Rate     $3.0\cdot 10^{-4}$

Tasks per Iteration     40

Trajectories per Task     20

Inner Step Size $\alpha$     0.1

Inner Adaptation Steps     1-2 (env-specific)

Grad Steps Per promp Iter     3-5 (env-specific)

metaworld     Hyperparameters

Clip Parameter     1.0

Discount ($\gamma$)     0.99

Lambda ($\lambda$)      1.0

KL Coeff     0.0

Learning Rate     $3.0\cdot 10^{-4}$

Tasks per Iteration     20

Trajectories per Task     5

Inner Step Size $\alpha$     0.05

Inner Adaptation Steps     4

Grad Steps Per promp Iter     5

§ ARCHITECTURE DETAILS

Table <ref> provides the architecture details for the different experiments. Note that FC(x,y) is a standard fully-connected network with two hidden layers of size x and y, Conv([x,y], s, f, n) is a n layer convolutional network with f kernels of  size [x,y] with stride length s, and LSTM([x, y], h) is a LSTM network with hidden layers of size x and y with a hidden state of size h.

Meta-train     Meta-test

10 points with $\context = 0, \mathbf{x} = 0, \mathbf{y} = 0$     2 points with $\context = 0, \mathbf{x} = 1, \mathbf{y} = 1$

10 points with $\context = 0, \mathbf{x} = 1, \mathbf{y} = 1$     2 points with $\context = 0, \mathbf{x} = 0, \mathbf{y} = 0$

10 points with $\context = 1, \mathbf{x} = 0, \mathbf{y} = 2$     2 points with $\context = 1, \mathbf{x} = 1, \mathbf{y} = 3$

10 points with $\context = 1, \mathbf{x} = 1, \mathbf{y} = 3$     2 points with $\context = 1, \mathbf{x} = 0, \mathbf{y} = 2$

5 points with $\context = 2, \mathbf{x} = 0, \mathbf{y} = 0$; 5 with $\context = 2, \mathbf{x} = 1, \mathbf{y} = 1$;     2 points with $\context = 2, \mathbf{x} = 2, \mathbf{y} = 4$

5 points with $\context = 2, \mathbf{x} = 0, \mathbf{y} = 0$; 5 with $\context = 2, \mathbf{x} = 2, \mathbf{y} = 4$;     2 points with $\context = 2, \mathbf{x} = 1, \mathbf{y} = 1$

5 points with $\context = 2, \mathbf{x} = 1, \mathbf{y} = 1$; 5 with $\context = 2, \mathbf{x} = 2, \mathbf{y} = 4$;     2 points with $\context = 2, \mathbf{x} = 0, \mathbf{y} = 0$

Detailed training procedure for the cognitive modeling experiment

Dataset     Base Network     Context Network     *-static Input     *-concat Input     maml-context Input

celeba     Conv([3,3], 2, 32, 4)     FC(40, 40) with film conditioning     Constant Vector embedded by a LSTM([40,40], 32)     Two-hot vector w/ attribute information     Two-hot vector w/ attribute information embedded by a LSTM([40,40], 40)

mujoco,metaworld     FC(64, 64)     FC(32, 64)     Constant Vector     Scalar parameter for mujoco, 3D goal position for metaworld)     Scalar parameter for mujoco, 3D goal position for metaworld)

Architectural details for the experiments. The first two columns correspond to the network architecture for the base and contextual network respectively. The last three columns describe the type of contextual input that is fed into the context network for Static, Concat, and MLTI baselines. Note that for maml and promp baselines, there is no contextual input.