# Can Social Media Platforms Transcend Political Labels? An Analysis of
Neutral Conservations on Truth Social

Chaitya Shah1, 5, Ritesh Konka1, 5, Gautam Malpani1, 5, Swapneel Mehta2, 4, 5,
Lynnette Hui Xian Ng3
1Dwarkadas J. Sanghvi College Of Engineering, Mumbai, India,
2Boston University, USA, 3Carnegie Mellon University, USA
4Massachusetts Institute of Technology, USA, 5SimPPL, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

> There is a prevailing perception that content on a social media platform
> generally have the same political leaning. These platforms are often viewed
> as ideologically congruent entities, reflecting the majority opinion of
> their users; a prime example of this is Truth Social. While this perception
> may exist, it is essential to verify the platform’s credibility,
> acknowledging that such platforms contain meaningful insights with neutral
> stances. To this end, we examine the dissemination of Wikipedia links on the
> alt-right platform, Truth Social. Wikipedia is recognized for enforcing
> content neutrality and serves as a unique lens to analyze the objectivity of
> user-generated content on Truth Social. By scrutinizing Truths with and
> without Wikipedia links, identifying toxicity trends & recognizing
> coordinated networks, we observe a lower level of engagement and a tendency
> for Truths shared on Truth Social to cover more neutral topics when it
> includes Wikipedia links (Wiki Truths). Given the significantly different
> engagement and nature of content shared of Wiki Truths against Non-Wiki
> Truths, we emphasize that we should not generalize the techno-political
> affiliation of a social media platform, but rather should investigate the
> content closely.

## Background

In February 2022, former US President Donald Trump introduced the Truth Social
platform after his account was suspended from Twitter. Truth Social allows
users to create posts termed as “Truths” that have text and media, and
interact with other content using favourites, replies and retruths (reposts).
Truth Social joins the set of social media platforms known as “alt-tech” that
includes Parler and Gab, labeled to serve users associated with the alt-right
political movement (Dehghan and Nagappa, 2022). However, while a platform is
labeled as “alt-right”, are the conversations within it consistently aligned
with right-wing ideologies? Or could there be space for neutral Truths? How do
the platform audiences respond when such links are shared within it? To answer
these questions, we look to investigate the sharing of Wikipedia links on
Truth Social. Wikipedia commits to providing objective information and
maintaining itself as a fair and impartial platform (Greenstein and Zhu,
2012). We assume that Truths that share Wikipedia links are generally neutral,
serving primarily to disseminate knowledge, because Wikipedia links are
generally not associated with a partisan meaning and has factual backing. The
scholarly works by Flisfeder (2018), Lamerichs et al. (2018) and Neiwert
(2017) provide compelling arguments suggesting that the primary beneficiary of
the 2016 US presidential elections was, assisted by an extreme right-wing
community, the “alt-right”. Consequently, the establishment of the Truth
Social platform was anticipated to serve as a sustaining force to this
community, providing a digital space conducive to their ideologies and
facilitating their continued influence in the socio-political landscape.

Sharot and Sunstein (2020) showcase that the importance of social media on
people’s intake of information has gained significant attention in recent
times. The platform’s distinct qualities, driven by hard-line supporters of
the former president, position it as an influential mouthpiece, yet little is
known about it (Gerard, Botzer, and Weninger, 2023). Thus, the way attention
was brought to the platform, we aim to analyse if the entire Truth Social
community is truly polarized? Or are there a group of topics where neutral
discussions occur?

In order to identify such a group, we need to target a set of truths that
share neutral nuances in general, and then check whether they perform
similarly or become biased inside the platform. Matei and Dobrescu (2011)
claims that Wikipedia shares a ”neutral point of view” (NPOV), but it is the
multiple ambiguous voices that contribute to the shape of neutrality on the
platform. Thus, with the help of these voices captured on Truth Social, we aim
to capture the objectivity of the audience and discern how it may respond to
different interactions within the platform.

In this paper, we address the Issue of Principle when studying social media
platforms. In particular, social media platforms are often labelled with
affiliations due to the alliances of their owners as stated in articles
rsf2022 (2022) and pewresearch_truthsocial (2022). We question this generic
labeling, for there are techno-political nuances within the content and the
users of the platforms. While Truth Social is termed as an alt-tech, past
analysis shows news reports of Truths, the stories shared on the platform
reflect a wide range of content placed across the political spectrum (Zhang et
al., 2024).

This paper builds on past work on the analysis of the Truth Social (TS)
(Wilber (2017)) and Wikipedia (Wiki) platforms (Soto (2012), Brandes et al.
(2009)), identifying the perceptions of content from one platform shared on
another. Using a combination of statistical and network analysis, we
demonstrate that Truths with neutral Wikipedia links exhibit varying levels of
engagement compared to Truths without such links. The coordination that exists
between these communities showcases a higher engagement on alt-right topics in
Truths without wiki-links, whereas Truths featuring wiki-links fosters more
discussion oriented interactions. Our work underscores the importance of
investigating this group of Truths.

## Retrieving Wikipedia Links on Truth Social

We collected Truths from Truth Social using a custom tool based on Stanford’s
open-source system111https://github.com/stanfordio/truthbrush to collect posts
on the dates August 17, 2023 and December 5, 2023. This helped us collect the
content of the Truths along with their creation time, author account details,
media attachments, tags, counts of the number of favourites, replies and
retruths. We separated this data manually into Wiki Truths, identified by the
presence of “en.wikipedia.org” in the post, and Non-Wiki Truths. This results
in a total of 1,790,749 Truths, of which there are 17451 unique Wiki Truths,
or 1% of the dataset, among 15313 unique users. Within these, we found out
that there were 13831 unique links shared. There were 14503 truths containing
only one Wikipedia link, 246 truths containing more than one Wikipedia link,
and 2138 truths containing a Wikipedia link along with atleast one other non-
Wikipedia link.

Figure 1: Count for Truths with Wikipedia Links across time

Our analysis delineates the dataset into three distinct segments illustrated
in Figure 2. Segment A, preceding a significant spike on May 28, 2023, is
marked by cnbc2023 (2023) reports of Truth Social facing financial losses and
a potential shutdown, prompting increased sharing of content from multiple
accounts. economicTimes2023 (2023) also states the platform losing $73 Million
since it’s launch. The Digital World Acquisition Corp states that a potential
merger has become a necessity for Trump Media & Technology Group, the owner of
Truth Social. August 17, 2023, witnesses Trump’s return to X after a prolonged
absence, diverting attention from Truth Social, resulting in Segment B
coinciding with a dip. Since then, his account on X has been inactive.
However, he also expressed his affection for Truth Social, stating that it is
his home directly on the platform (apnews2023, 2023). Segment C suggests a
period of reevaluation and adaptation for the platform, contending with
competition from Trump’s presence on other platforms. By segmenting the
dataset in this manner, we aim to gain insights, keeping an ever-evolving
social media landscape into consideration.

## Exploratory Data Analysis

Engagement Statistics | Segment A (wiki/non-wiki) | Segment B (wiki/non-wiki) | Segment C (wiki/non-wiki)
---|---|---|---
favourites count | 1.90 $\pm$ 13.84 /
6.83 $\pm$ 159.11 ** | 1.22 $\pm$ 13.31 /
4.84 $\pm$ 108.72 ** | 1.35 $\pm$ 7.43 /
8.68 $\pm$ 212.40 **
retruths count | 0.71 $\pm$ 5.01/
2.59 $\pm$ 47.39 * | 0.52 $\pm$ 5.49 /
1.79 $\pm$ 33.14 ** | 0.74 $\pm$ 4.21 /
2.91 $\pm$ 63.38 **
replies count | 0.52 $\pm$ 1.28 /
0.97 $\pm$ 45.07 ** | 0.52 $\pm$ 2.28 /
0.54 $\pm$ 8.99 ** | 0.52 $\pm$ 1.39 /
0.98 $\pm$ 45.08 **
average toxicity | 0.12$\pm$0.15 /
0.12$\pm$0.13 | 0.15$\pm$0.17 /
0.11$\pm$0.12 | 0.14 $\pm$0.16 /
0.14$\pm$ 0.14
Table 1: Comparison of engagement statistics of Truths with and without
Wikipedia Links. “*” means the engagement between the two groups of Truths are
significantly different at $p<0.05$, “**” is at the $p<0.01$ level.

Metrics for user engagement and content interaction across social media
platforms were analyzed between the Wiki/Non-Wiki Truths using the Kruskal-
Wallis test (Table 1). This test was chosen due to its non-parametric nature,
suitable for analyzing data that does not adhere to the normal distribution.
Wiki Truths consistently exhibited significantly lower engagement metrics
across all time segments, suggesting reduced user interaction compared to non-
Wiki Truths. The analysis revealed consistent levels of neutrality and
objectivity associated with Wikipedia content, as evidenced by stable, lower
toxicity scores across segments. The findings underscore that within
politically right-leaning platforms there are ”pockets of neutrality”
exhibiting behaviors that depart from platform norms, potentially related to
similar findings about scientifically organized pockets within COVID-skeptic
communities (Lee et al., 2021). Truths were labeled with toxicity values using
the Perspective API222https://perspectiveapi.com/. Both Wiki and Non-Wiki
Truths generally have a low average toxicity score, which suggests there are
nuances of alt-right platforms that are overlooked by current literature (e.g.
toxicity analysis) and may warrant a longitudinal study (Rieger et al., 2021).
There may be other subtle linguistic differences between Wiki and Non-Wiki
Truths, that suggests that deeper analysis into the evaluation of content in
alt-right platform might require specialized classifiers to identify topics
and language of interest.

Figure 2: Coordination Graphs by Common Links for Truths Containing Wikipedia
and Non-Wikipedia Links.

## Coordinated Sharing of Wikipedia Links

We then investigated coordinated link sharing behaviors on Truth Social,
focusing on users sharing the same Wikipedia link. Using temporal coordination
identification techniques (Ng and Carley, 2022), we constructed coordination
network graphs across each time segment. The time factor for analyzing the
coordination was each entire segment. In these graphs, nodes represent users,
and edges between two nodes indicate that they shared the same Wikipedia link
within the respective time segment.

In Segment A, minimal coordination was observed, characterized by numerous
small clusters. While Segment B showed a dominant large cluster along with
smaller clusters, Segment C demonstrated even more pronounced coordination,
with three to four significant clusters alongside smaller ones, yielding a
higher graph density. This variance in the presence of coordinated link
sharing across Segments A through C imply a shift in user behavior. Moreover,
the presence of coordination among users sharing neutral links in Segments B
and C suggests that not all coordination is inherently malicious. The
transient nature of coordinated clusters emphasizes the potential for co-
sharing behaviors to occur.

## Coordinated Sharing of Non-Wikipedia Links

The analysis of non-Wikipedia data reveals significant coordination across all
three segments, with all clusters showing notable coordination of non-
Wikipedia links. Given the inherently larger volume of Truths within the non-
wiki dataset compared to the Wikipedia dataset, it is unsurprising to observe
denser graphs for non-Wikipedia Data. This observation underscores the
platform’s active engagement in coordinated link-sharing activities for non-
Wikipedia topics, indicative of the community’s interest in such content.

We manually sieved through the Truths in the same cluster and provided labels
of the topics. The topics extracted in our study provided insights into
prevalent themes discussed within each community. Within the Non-Wiki Truths,
prominent topics included ”Lost Immigrant Children”, ”Government Propaganda”,
”COVID-19 Vaccine Safety””, and ”Political Conspiracy Theories”, reflecting a
focus on politically charged issues and societal concerns. Conversely, Wiki
Truths featured a broader range of topics, encompassing ”Survival Thriller
Films”, ”Investigative Journalism”, ”climate Change”, and ”Mathematical
Concepts”. This diversity suggested a neutral stance towards information
sharing within the Wikipedia community, highlighting the potential for
fostering balanced discourse and informed discussions on Truth Social. The
comparative analysis of topics extracted from Wiki and Non-Wiki underscores
the importance of studying neutral groups within alt-right platforms. While
Non-wiki clusters predominantly focused on politically charged issues and
conspiracy theories, Wiki clusters covered a broader spectrum of subjects with
a neutral stance. These findings offer valuable insights into the diverse
interests and engagement patterns within these communities, informing efforts
to promote informed discussions and balanced discourse on Truth Social.

## Discussion

Studying alt-right platforms helps in analyzing disinformation campaigns
(Bevensee and Ross, 2018). Continuous study of such social media platforms is
essential for effectively addressing the impact of extremist movements on
society and fosters a deeper understanding of societal vulnerabilities and
resilience strategies (Zannettou et al., 2018). Studying the Truths with
Wikipedia links on such platforms may indicate topics and issues significant
within the community of users that share the same ideology. This allows
understanding of user interests which can provide insights into the
ideological priorities and concerns of alt-right groups (Chadwick, Vaccari,
and Kaiser, 2022).

While our analysis offers valuable insights into the engagement dynamics on
Truth Social, it is important to acknowledge the inherent limitations of our
study. Search results on Truth Social gives us only a maximum of 10,000
Truths, therefore creating a data accessibility problem. By searching the
platform multiple times, we are able to collect a larger set of Truths. Our
method is also prone to sampling bias, for the returned set of Truths likely
contained the most popular Truths on the platform. However, given that the
most popular Truths are the most viewed and recommended Truths, we believe
that this set sufficiently represents the platform.

The analysis of engagement metrics between Wiki and Non-Wiki Truths, conducted
across three distinct time segments, revealed notable disparities in user
interaction on Truth Social. Notably, Wiki Truths consistently exhibited lower
engagement metrics across all time segments compared to Non-Wiki Truths,
indicating reduced interaction. Furthermore, the analysis highlighted
consistent levels of neutrality and objectivity as evidenced by stable
toxicity scores across segments. In addition to quantitative metrics, manual
labeling of topics within Truths clusters provided insights into prevalent
themes discussed within each community. Non-Wiki Truths Clusters primarily
addressed politically charged topics and conspiracy theories, whereas those
associated with Wiki Truths encompassed a wider range of subjects, reflecting
a more neutral stance. These findings underscore the importance of studying
neutral groups within alt-right platforms like Truth Social, offering valuable
insights into the diverse interests and engagement patterns within these
communities.

Given the differences in engagement statistics and the transient coordinated
link sharing behavior between Wiki/ Non-Wiki links, we urge the computational
social science community to not judge a platform by its label. This work draws
into the topic of theme of the issue of principle on social media platforms.
Very often, platforms are stereotyped with political alliances due to their
owners. These techno-political alliances undoubtedly create a filter for
social computing studies. Studies on the Truth Social platforms typically
revolve around its alt-right characteristics, such as content analysis
reflecting the support of the Republican party, or investigations of
conspiracy-laden narratives (Gerard, Botzer, and Weninger, 2023). Neither the
creator nor the owner of the platform should entirely determine the political
affiliation of a platform. Instead, the users themselves, and the
conversations within determine the platform’s character. In that aspect, we
put forth that every platform has a silver lining: the objective, innocuous,
non-toxic posts. Therefore, such social media platforms should not be judged
by their political labels, but we should study the platform by itself to
understand its character.

## Conclusion

The analysis conducted explored the engagement of the content shared on Truth
Social. Wiki Truths have a significantly less engagement, which might be
because they are neutral Truths and thus do not illicit much interactions. The
study also focuses on coordinated link sharing on Truth Social, extracting
topics from clusters identified in both Wikipedia (wiki) and non-Wikipedia
(non-wiki) Truths. Wiki and Non-Wiki Truths have different properties in terms
of engagement statistics, coordination patterns and topics shared, emphasizing
that the content on a social media platform transcends political labels and
should not be labeled with a sweeping statement. This work illustrates the
importance of scrutinizing different categories of content within a social
media platform, and opens discussion for future research into the
characteristics of sub-populations of political communities on social media.

## References

  * Lee et al. (2021) Lee, C.; Yang, T.; Inchoco, G. D.; Jones, G. M.; and Satyanarayan, A.  2021\.  Viral visualizations: How coronavirus skeptics use orthodox data practices to promote unorthodox science online.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–18.
  * Dehghan and Nagappa (2022) Dehghan, E., and Nagappa, A.  2022\.  Politicization and radicalization of discourses in the alt-tech ecosystem: A case study on Gab Social.  Social Media+ Society 8(3):20563051221113075.
  * Matei and Dobrescu (2011) Matei, S. A., and Dobrescu, C.  2011\.  Wikipedia’s “Neutral Point of View”: Settling Conflict through Ambiguity.  The Information Society 27(1):40–51.
  * Greenstein and Zhu (2012) Greenstein, S., and Zhu, F.  2012\.  Is Wikipedia Biased?  American Economic Review 102(3):343–348.
  * NYT (2022) NYT.  2022\.  Truth Social Review: Trump’s Uncensored Social App Is Incomplete (Published 2022) — nytimes.com.  https://www.nytimes.com/2022/04/27/technology/personaltech/truth-social-review-trump.html.
  * Rieger et al. (2021) Rieger, D.; Kümpel, A. S.; Wich, M.; Kiening, T.; and Groh, G.  2021\.  Assessing the extent and types of hate speech in fringe communities: A case study of alt-right communities on 8chan, 4chan, and Reddit.  Social Media+ Society 7(4):20563051211052906.
  * Ng and Carley (2022) Ng, L. H. X., and Carley, K. M.  2022\.  Online coordination: methods and comparative case studies of coordinated groups across four events in the United States.  In Proceedings of the 14th ACM Web Science Conference 2022, 12–21.
  * Gerard, Botzer, and Weninger (2023) Gerard, P.; Botzer, N.; and Weninger, T.  2023\.  Truth social dataset.  In Proceedings of the International AAAI Conference on Web and Social Media, volume 17, 1034–1040.
  * Zhang et al. (2024) Zhang, Y.; Lukito, J.; Suk, J.; and McGrady, R.  2024\.  Trump, Twitter, and Truth Social: how Trump used both mainstream and alt-tech social media to drive news media attention.  Journal of Information Technology & Politics 1–14.
  * (10) 2023\.  Trump’s micro-blogging platform Truth Social loses $73 million since launch.  Economic Times Telecom.
  * Lamerichs et al. (2018) Lamerichs, N.; Nguyen, D.; Melguizo, M. C. P.; Radojevic, R.; and Lange-Böhmer, A.  2018\.  Elite male bodies: The circulation of alt-Right memes and the framing of politicians on Social Media.  Participations 15(1):180–206.
  * Neiwert (2017) Neiwert, D.  2017\.  Alt-America: The rise of the radical right in the age of Trump.  Verso Books.
  * Flisfeder (2018) Flisfeder, M.  2018\.  “Trump”—What does the name signify? Or, Protofascism and the alt-right: Three contradictions of the present conjuncture.  Cultural Politics 14(1):1–19.
  * (14) 2023\.  Trump’s social media company has lost 31 million faces possible shutdown.  CNBC.
  * (15) 2023\.  Trump’s micro-blogging platform Truth Social loses $73 million since launch.  Economic Times.
  * (16) 2023\.  Trump returns to Twitter with familiar rants, grievances.
  * Bevensee and Ross (2018) Bevensee, E., and Ross, A. R.  2018\.  The alt-right and global information warfare.  In 2018 IEEE International Conference on Big Data (Big Data), 4393–4402.
  * Zannettou et al. (2018) Zannettou, S.; Bradlyn, B.; De Cristofaro, E.; Kwak, H.; Sirivianos, M.; Stringhini, G.; and Blackburn, J.  2018\.  What is gab: A bastion of free speech or an alt-right echo chamber.  In Companion Proceedings of the The Web Conference 2018, 1007–1014.
  * Chadwick, Vaccari, and Kaiser (2022) Chadwick, A.; Vaccari, C.; and Kaiser, J.  2022\.  The amplification of exaggerated and false news on social media: The roles of platform use, motivations, affect, and ideology.  American Behavioral Scientist 00027642221118264\.
  * Sharot and Sunstein (2020) Sharot, T., and Sunstein, C. R.  2020\.  How people decide what they want to know.  Nature Human Behaviour 4(1):14–19.
  * Wilber (2017) Wilber, K.  2017\.  Trump and a post-truth world.  Shambhala Publications.
  * Soto (2012) Soto, J. F. O., et al.  2012\.  Wikipedia: A quantitative analysis.
  * Brandes et al. (2009) Brandes, U.; Kenis, P.; Lerner, J.; and Van Raaij, D.  2009\.  Network analysis of collaboration structure in Wikipedia.  In Proceedings of the 18th International Conference on World Wide Web, 731–740.
  * (24) 2022\.  United States: Truth Social Highlights Need for Politically and Ideologically Neutral Social Media.  Reporters Without Borders, March 7, 2022.
  * (25) 2022\.  Key Facts About Truth Social as Donald Trump Runs for U.S. President Again.  Pew Research Center, November 18, 2022.


# Overestimation, Overfitting, and Plasticity in Actor-Critic:
the Bitter Lesson of Reinforcement Learning

Michal Nauman  Michał Bortkiewicz  Mateusz Ostaszewski  Piotr Miłoś  Tomasz
Trzciński  Marek Cygan

###### Abstract

Recent advancements in off-policy Reinforcement Learning (RL) have
significantly improved sample efficiency, primarily due to the incorporation
of various forms of regularization that enable more gradient update steps than
traditional agents. However, many of these techniques have been tested in
limited settings, often on tasks from single simulation benchmarks and against
well-known algorithms rather than a range of regularization approaches. This
limits our understanding of the specific mechanisms driving RL improvements.
To address this, we implemented over 60 different off-policy agents, each
integrating established regularization techniques from recent state-of-the-art
algorithms. We tested these agents across 14 diverse tasks from 2 simulation
benchmarks. Our findings reveal that while the effectiveness of a specific
regularization setup varies with the task, certain combinations consistently
demonstrate robust and superior performance. Notably, a simple Soft Actor-
Critic agent, appropriately regularized, reliably solves dog tasks, which were
previously solved mainly through model-based approaches.

Machine Learning, ICML

##  1 Introduction

In recent years, substantial improvements have been made in the domain of deep
reinforcement learning, as evidenced by breakthroughs such as mastering
complex games like Dota 2 (OpenAI et al., 2019), Go (Silver et al., 2017) and
achieving control over nuclear fusion plasma (Degrave et al., 2022). In
particular, off-policy RL has witnessed a surge of approaches reporting state-
of-the-art results (Li et al., 2022; Hafner et al., 2023; Lee et al., 2023),
including application to real robots (Smith et al., 2022). In general, those
approaches build upon Soft Actor-Critic (SAC) algorithm with increased number
of gradient steps per environment steps (Replay Ratio (RR)) used in
conjunction with some form of regularization that stabilizes the learning in
high RR setting (Janner et al., 2019; Chen et al., 2020; Hiraoka et al., 2021;
Nikishin et al., 2022; Li et al., 2022; D’Oro et al., 2022; Cetin &
Celiktutan, 2023). These approaches for regularization encompass
considerations of reducing overfitting (Li et al., 2022) (network
regularization), reducing critic overestimation (Cetin & Celiktutan, 2023)
(critic regularization) or reducing the rate of plasticity loss (Lee et al.,
2023) (plasticity regularization).

Despite significant advancements, the understanding of how different
regularization techniques synergistically improve off-policy agent performance
is still limited (Hiraoka et al., 2021; Lee et al., 2023). Moreover, most
methods are tested in narrow contexts, mainly in locomotion or manipulation
tasks, often restricted to a single simulation benchmark (Fujimoto et al.,
2018; Haarnoja et al., 2018; Chen et al., 2020; Moskovitz et al., 2021; D’Oro
et al., 2022), leading to questions about their broad applicability and
robustness. In this study, our goal is to consolidate these lessons and
address the following research questions: Which regularization techniques lead
to robust performance improvements across diverse tasks and agent designs? Can
generic regularization techniques outperform domain-specific RL techniques
that directly use the MDP structure?. We extend the scope of prior research by
examining over 60 design choices in a diverse array of tasks, including both
locomotion and manipulation, within two simulation benchmarks and two replay
ratio regimes. This comprehensive approach offers a deeper understanding of
the effectiveness of these regularization techniques in various settings.

Our main result reinforces the ’Bitter Lesson’ (Sutton, 2019): across varied
tasks, general neural network regularizers significantly outperform most RL-
specific algorithmic improvements in terms of agent performance. Notably,
network regularization enables agents to solve tasks previously impossible for
model-free agents, such as those in the dog domain. Our findings also show
that layer normalisation is more effective in reducing overestimation than
techniques specifically designed for mitigating Q-value overestimation in
critic networks. Consequently, we show that replacing the ubiquitous Clipped
Double Q-learning with network regularization techniques leads to significant
performance gains. Our research further explores the impact of overestimation,
overfitting, and plasticity loss on agent performance in a unified
experimental setup. We examine the correlation between these factors and agent
performance, showing a strong negative correlation for value overestimation
and agent plasticity metrics. These influences vary significantly across
environments, underscoring the complex nature of their effects on learning. A
key observation is the environment-dependent performance of various methods.
Strategies excelling in locomotion tasks may falter in manipulation scenarios
and vice versa. Comparisons between experiments on the DeepMind Control Suite
(Tassa et al., 2018) and MetaWorld (Yu et al., 2019) demonstrate the necessity
for diverse benchmarking in research, highlighting the value of expansive
experimental setups.

  1. 1.

Our study presents an extensive empirical analysis of various regularization
techniques in off-policy RL. We evaluate the effectiveness, robustness, and
generality of 12 SAC design choices derived from recent literature, examining
their diverse interactions. This encompasses testing 64 model designs across
14 tasks from two benchmarks under two replay ratio regimes.

  2. 2.

Our findings show that combining well-established network regularization
techniques with methods that prevent plasticity loss effectively addresses the
value estimation problem, eliminating the need for critic regularization.
Specifically, we observe that in network/plasticity regularized agents using
critic regularization often leads to significant performance degradation.
Leveraging these insights, we demonstrate that integrating specific
regularization methods into the basic Soft Actor-Critic framework leads to
state-of-the-art performance in dog domain tasks for model-free approaches.

  3. 3.

Our study investigates the correlation between overestimation, overfitting,
and plasticity proxies, and their impact on agent performance. We discover
that interventions aimed at one type of issue, such as full-parameter resets,
significantly affect proxies for issues other than plasticity such as
overestimation and overfitting, often more than interventions specifically
designed for those other issues. This suggests that RL agents encounter a
range of complex problems that collectively affect the learning process.

##  2 Background

We consider a Markov Decision Process (MDP) (Puterman, 2014; Sutton & Barto,
2018) which is described via a tuple $(S,A,r,p,\gamma)$, where states $S$ and
actions $A$ are continuous, $r(s,a)$ is the transition reward,
$p(s^{\prime}|s,a)$ is a transition mapping, $p_{0}$ is the starting state
distribution and $\gamma\in(0,1]$ is the discount factor. Policy, denoted as
$\pi(a|s)$ is a state-conditioned action distribution. Maximum Entropy
Reinforcement Learning (MaxEnt RL) objective (Ziebart et al., 2008; Haarnoja
et al., 2017) is to find a policy that maximizes the expected sum of
discounted returns and policy entropies, or equivalently expected initial
state values according to $\pi^{*}=\arg\max\mathbb{E}_{p_{0}}V^{\pi}(s_{0})$.
The Q-value is defined as $Q^{\pi}(s,a)=r(s,a)+\gamma V^{\pi}(s^{\prime})$.
State value is defined by
$V^{\pi}(s)=\mathbb{E}_{\pi}(Q^{\pi}(s,a)-\alpha\log\pi(a|s))$, where
$\alpha\log\pi(a|s)$ is the maximum entropy term. In actor-critic, policy and
Q-value functions are represented by parameterized function approximators
(Silver et al., 2014). Policy parameters $\theta$ are updated to maximize the
value approximation at sampled states $s$ from an off-policy replay buffer
$\mathcal{D}$ (Fujimoto et al., 2018; Haarnoja et al., 2018):

$\begin{split}&\theta^{*}=\arg\max_{\theta}\underset{\mathcal{D}}{\mathbb{E}}~{}Q_{\phi}(s,a)-\alpha\log\pi_{\theta}(a|s),~{}a\sim\pi_{\theta}.\end{split}$
(1)

The critic parameters $\phi$ are updated by minimizing the temporal-difference
(Silver et al., 2014):

$\begin{split}&\phi^{*}=\arg\min_{\phi}\underset{\mathcal{D}}{\mathbb{E}}\bigl{(}Q_{\phi}(s,a)-r_{(}s,a)-\gamma\bar{V}_{\phi}(s^{\prime})\bigr{)}^{2},\end{split}$
(2)

where $\bar{V}_{\phi}(s^{\prime})$ is the target network (Mnih et al., 2015).

###  2.1 Overestimation

Q-learning methods employing function approximation have been observed to
exhibit a bias toward overestimation, a phenomenon critical to the training
process (Thrun & Schwartz, 2014; Fujimoto et al., 2018). Positive bias stems
from the policy being trained to locally maximize action-value estimates,
leading its actions to exploit potential model errors for higher scores.
Modern actor-critic algorithms leverage a variety of countermeasures to
overestimation of Q-value targets, with Clipped Double Q-learning (CDQ)
(Fujimoto et al., 2018) being most used by many other algorithms (Haarnoja et
al., 2018; Chen et al., 2020; Hiraoka et al., 2021). In CDQ, the algorithm
maintains an two critics and uses their minimum as an approximate a Q-value
lower bound. The CDQ was generalized to the following pessimistic objective
(Ciosek et al., 2019; Moskovitz et al., 2021; Cetin & Celiktutan, 2023):

$\begin{split}&Q_{\phi}^{\beta}(s,a)=Q^{\mu}_{\phi}(s,a)-\beta
Q^{\sigma}_{\phi}(s,a).\end{split}$ (3)

We denote the level of pessimism as $\beta$, and the critic ensemble mean and
standard deviation as $Q^{\mu}_{\phi}$ and $Q^{\sigma}_{\phi}$ respectively.
In particular, for $\beta=1$, the above rule is exactly equal to the CDQ
minimum (Ciosek et al., 2019; Cetin & Celiktutan, 2023). The success of
pessimistic updates led to various methods for adjusting $\beta$ online. A
recent approach, Generalized Pessimism Learning (GPL) (Cetin & Celiktutan,
2023), estimates the critic approximation error and modifies $\beta$
accordingly. A different strategy, Tactical Optimism and Pessimism (TOP)
(Moskovitz et al., 2021), adjusts pessimism independent of the estimated
approximation error. Specifically, TOP uses an external bandit controller to
maximize online episodic rewards. Whereas this controller is aligned with the
RL objective, it only allows for discrete values of pessimism.

###  2.2 Overfitting

Overfitting, while not commonly scrutinized in reinforcement learning, has
gained attention in recent discussions (Li et al., 2022) as a phenomenon
correlated with performance decline in models characterized by a high ratio of
updates to data. To evaluate overfitting in agents, Li et al. (2022) utilize a
validation dataset that consists of samples gathered using the same policy as
the canonical replay buffer. The validation buffer is established to provide
an unbiased assessment of the critic error on experiences that were not used
in the learning. Although there are many strategies to deal with overfitting
in supervised learning, only few of them were applied in the context of RL. To
this end, application of Weight Decay (WD) (Schwarzer et al., 2023), Layer
Normalization (LN) (Ball et al., 2023) or Spectral Normalization (SN) (Cetin &
Celiktutan, 2023) was shown to greatly effect the performance of the
underlying agent.

###  2.3 Plasticity

Plasticity, in the context of models, refers to their ability to learn new
information. The concept of plasticity loss has recently gained prominence in
the deep learning community, particularly in supervised learning (Achille et
al., 2017; Ash & Adams, 2020; Dohare et al., 2021) and RL (Nikishin et al.,
2022; Dohare et al., 2021; Lyle et al., 2023; Lee et al., 2023; Kumar et al.,
2023; Nikishin et al., 2023). Numerous hypotheses have been proposed regarding
the sources of plasticity loss, including dead or dormant units, rank
collapse, and divergence due to large weight magnitudes (Lyle et al., 2022;
Sokar et al., 2023; Kumar et al., 2020; Dohare et al., 2021). However, none of
these mechanisms alone is sufficient to explain the phenomenon of plasticity
loss. Whereas the cause of plasticity loss remains to be discovered, various
approaches for regularizing the model plasticity have been proposed. For
example, full-parameter resets of actor-critic modules were shown to greatly
improve the agent’s ability to learn (Nikishin et al., 2022; D’Oro et al.,
2022). The problem of plasticity was also tackled at the level of the
activation function with Concatenated ReLU (CRLU) (Abbas et al., 2023) or the
optimizer with the Sharpness-Aware Minimization (SAM) (Foret et al., 2020).

##  3 Study Design

In this paper, we analyze the impact of various interventions on SAC
performance across seven DeepMind Control Suite (Tassa et al., 2018) (DMC)
tasks: acrobot-swingup, hopper-hop, humanoid-walk, humanoid-run, dog-trot,
dog-run, quadruped-run and seven MetaWorld (Yu et al., 2019) (MW) tasks:
Hammer, Push, Sweep, Coffee-Push, Stick-Pull, Reach, Hand-Insert. We chose a
wide spectrum of tasks, ranging from easy (acrobot-swingup, Reach) to barely
solvable (dog-run) for generic insights that are not overfitted to only a
specific group. We choose tasks that are not easily solved by the baseline
high replay SAC, as presented in D’Oro et al. (2022) and Hansen et al. (2022),
with added dog tasks (which are generally unsolved in state-based
representation). Finally, following (Li et al., 2022), we conduct experiments
in low 2 and high replay regimes. Such experimental design allows us to
pinpoint if specific regularization targets issues associated with high
replay, or if it is universally applicable across varying replay regimes.
Categorizing them based on current state-of-the-art methods, we identify three
intervention groups:

  * •

Critic Regularizations (CR),

    * –

Clipped Double Q-learning (CDQ) (Fujimoto et al., 2018),

    * –

Tactical Optimism Pessimism (TOP) (Moskovitz et al., 2021),

    * –

Generalized Pessimism Learning (GPL) (Cetin & Celiktutan, 2023),

  * •

Network Regularizations (NR),

    * –

Layer Norm (Ba et al., 2016),

    * –

Spectral Norm (Miyato et al., 2018; Zhang et al., 2018; Brock et al., 2018),

    * –

Weight Decay (Loshchilov & Hutter, 2017),

  * •

Plasticity Regularizations (PR),

    * –

Resets (Nikishin et al., 2022),

    * –

Concatenated ReLU activations (CRLU) (Abbas et al., 2023),

    * –

Sharpness-Aware Minimization Optimizer (SAM) (Foret et al., 2020).

To explore the interactions between interventions, we systematically run all
possible combinations of methods across groups, ensuring that methods from the
same group are not combined. Each configuration is evaluated on 10 seeds. In
the results analysis, we categorize marginalization into three levels:

First-order marginalization combines all results for a specific intervention.
For instance, the marginalized performance of layer norm will be computed as
the average performance across all combinations with interventions from other
groups (in this example, Critic Regularizations and Plasticity
Regularizations).

Second-order marginalization involves evaluating the performance of fixed
pairs of methods from two groups and marginalizing results from the third
group.

Third-order results involve no intervention marginalization and represent the
performance of a specific combination, including one method from each group.
The only marginalization is over all tested environments (we present these
results in Appendix D.1 due to space constraints). This cumulative result
provides insight into the overall impact of a given intervention.

Furthermore, we conduct an analysis of various proxy metrics associated with
the problems of overestimation, overfitting and plasticity loss. For
overestimation, we evaluate the state-action critic approximation error,
denoted as $b_{\phi}(s,a)$, is quantified as the disparity between the critic
output and the true on-policy Q-value according to
$b_{\phi}(s,a)=Q_{\phi}(s,a)-Q^{\pi}(s,a)$, where $Q_{\phi}$ denotes the
critic Q-value approximation and $Q^{\pi}$ represents the on-policy Q-value
which we estimate via a Monte-Carlo rollout with 5 samples. To calculate
overfitting, we compare average TD errors on evaluation trajectories (which
are not used for learning) to average TD errors observed in training according
to
$o_{\phi}=\frac{\mathbb{E}_{\mathcal{D}_{v}}TD_{\phi}}{\mathbb{E}_{\mathcal{D}}TD_{\phi}}$,
where $o_{\phi}$ denotes critic overfitting, $\mathcal{D}_{v}$ denotes
validation data, and $TD_{\phi}$ denotes the temporal difference loss. As
such, the extent of overestimation is then quantified by the ratio of
validation TD error to training TD error. We monitor plasticity loss by the
rank of penultimate layer representations (Kumar et al., 2020), dormant
neurons or dead units (Sokar et al., 2023), and the L2 norm of weights as a
proxy for plasticity loss.

##  4 Experiments

###  4.1 Combination of interventions – First-order marginalization

##### Study description:

First-order marginalization provides insights into the robust impact of a
given intervention on model performance, irrespective of what other type if
regularization it is paired with. To measure such robustness, we compare the
performance of the baseline SAC model augmented with one specific
regularization (e.g., SAC + WD) to the performance of SAC augmented this
regularization paired with some other technique (e.g. SAC + WD + Resets).

##### Results:

Examining the plots in Figure 1 shows that network and plasticity
regularization techniques are generally more effective than critic
regularization. We observe this results for both simple models using one
regularization technique and more complex agents leveraging many
regularizations at once. Most notably, avoiding the use of critic
regularization interventions (!CR) proves advantageous for both DMC and MW,
especially if some other type of regularization is used (such as layer norm or
full-parameter resets). This result is somewhat surprising, as critic
regularization methods where designed specifically for off-policy actor-critic
agents, whereas the network and plasticity regularization techniques are
general. Notably, TOP (Moskovitz et al., 2021) emerges as an exception,
particularly showcasing its effectiveness on the DMC benchmark. Conversely,
the GPL intervention exhibits the least robust performance in both DMC and MW.
Full-parameter resets (Nikishin et al., 2022), tailored for high replay ratio
regimes, proves to be one of the most robust approaches in this RR regime.
Further analysis reveals discrepancies in conclusions between benchmarks.
Clearly, LN is the most effective approach on the DMC benchmark, yet it
presents very poor performance on the MW benchmark. To this end, we find SN to
be more robust, which strongly helps the MW benchmark and moderately helps the
DMC.

##### Takeaways:

• Critic regularization methods exhibit limited effectiveness in enhancing
performance. When using network or plasticity regularization, critic
regularization leads to reduced performance. • Periodical network resetting is
the most robust intervention across two benchmarks in a high RR regime, and
highly surpasses other plasticity regularization techniques in both robustness
and performance. • When considering network regularization approaches, layer
norm is generally recommended for DMC, while layer norm is more effective for
MW benchmarks. When considering a diverse tasks, we find spectral norm to be
more robust than layer norm. Weight decay has generally low performance when
used alone with SAC.

Figure 1:  IQM performance of First-Order Marginalization. The left column
presents results for baseline SAC augmented with a single regularization
technique (and thus uses 10 seeds per task), and the right column presents the
aggregate performance of a specific regularization technique when paired with
other regularizations (and thus uses 640 seeds per task). Results are
presented for DMC (top row) and MW (bottom row) benchmarks. 14 tasks.

###  4.2 Combination of interventions – Second-order marginalization

##### Study description:

This study delves into second-order marginalization to pinpoint the most
effective combinations. Results are presented across various replay ratios (2
and 16) and benchmarks (DMC or MW). Given the limited impact of critic
regularizations like CDQ or GPL in the first order experiments, our focus is
on discerning the most advantageous combinations involving of regularization.

Figure 2:  Second-order results marginalizing critic regularization methods.
On the x-axis, we have different types of plasticity regularization, and each
color denotes network regularization. For better readability, points within
one plasticity regularization are spaced slightly horizontally. Vertical lines
indicate standard error.

##### Results:

On the DMC with RR=2 and RR=16 (top row in Figure 2) a clear hierarchy of
interventions is observed: layer norm and right below it, spectral norm
consistently outperforms others in mean return, irrespective of the plasticity
regularization (x-axis). Notably, the combination of layer norm and resets in
RR=16 (top-right plot in Figure 2) demonstrates exceptional performance across
all critic regularization variations. In contrast, the hierarchy of
interventions on the MW benchmark (bottom row in Figure 2) diverges
significantly from the DMC setup. Furthermore, a higher replay ratio
introduces shifts in training dynamics, as evidenced by SN transitioning from
a lower position on RR=2 (bottom-left plot in Figure 2) to nearly the most
versatile intervention on RR=16 (bottom-right plot in Figure 2). The results
generally align with first-order marginalization findings, emphasizing the
positive impact of SN, as well as using many different types of regularization
at once in general. Deeper analysis (see Appendix D.2) reveals that on MW,
indeed, the gradient norm in a higher RR regime is orders of magnitude bigger.
The finding that most contrasts with the first-order experiments, is that we
observe that weight decay can actually yield significant performance benefits,
under the condition that it is paired with other specific method, namely full-
parameter resets. In particular, we observe that this combination yields
synergies surpassing using any of these methods alone.

##### Takeaways:

• The DMC benchmark can be largely trivialized by using high RR agents
combined with layer norm and full-parameter resets. • Spectral normalization
intervention ranks best for the Meta World benchmark, but it’s not universally
applicable. Whereas weight decay does not perform when used alone, it seems to
have high synergy with full-parameter resets. • Resetting the network
significantly outperforms other plasticity-inducing interventions such as CRLU
and SAM.

###  4.3 A closer look on Dog environment

##### Study description:

In this study, we delve into the intricacies of two challenging Dog tasks,
Dog-Trot and Dog-Run, included in our DMC setup. These environments present
considerable difficulties for model-free approaches relying on proprioceptive
states, making them of particular interest within the research community. For
detailed experimental information, please refer to Appendix B.

##### Results:

Figure 3: IQM performance evolution across timesteps for Dog-Run (top row) and
Dog-Trot (bottom row) environments. Each plot showcases the top four
combinations alongside vanilla SAC (coded as: CDQ+!NR+!PR).

Specific intervention combinations effectively tackle the challenges posed by
the Dog environment, as depicted in Figure 3. Analyzing the top four
approaches for Dog-Run and Dog-Trot tasks reveals the prevalence of layer norm
in nearly all combinations. Additionally, each critic regularization approach
contributes to the leading group. Notably, in scenarios with high replay
ratios, resets emerge as a crucial intervention. These observations are
further substantiated by the analysis of second-order marginalization IQM
plots (see 4). Indeed, layer norm without critic regularization excels in
RR=2, and layer norm with resets outperforms all others convincingly.

This achievement is particularly notable as, to our best knowledge, no model-
free agent has previously solved the Dog environment using proprioceptive
states.

Figure 4: IQM performance of the top six intervention pair combinations. The
IQM is calculated based on the average of the last ten evaluation points in
each run, not the last evaluation point. Top row: Dog-Run environment. Bottom
row: Dog-Trot. Colors indicate hierarchy on the plot, not specific names.

##### Takeaways:

• Well-established network regularization techniques such as layer norm and
spectral norm enable solving Dog-Trot and Dog-Run effectively.  • The choice
of domain-specific RL critic regularization has little significance in dog
environments when layer norm and resetting interventions are employed.

###  4.4 Correlation of Overestimation, Overfitting and Plasticity metrics
with Performance

##### Study description:

This study analyses the relationships between Overestimation, Overfitting,
Plasticity, and model performance. Overestimation is quantified as an
approximation error, overfitting as the ratio of TD error on the validation
set to TD error on the training set. Expressing Plasticity loss is
challenging, so we utilize proxy metrics, including the percentage of Dormant
neurons, representations rank, gradient norm, and parameters norm.

We employ a Spearman correlation matrix to scrutinize these dependencies. This
statistic is chosen because we observe non-linear yet monotonic dependencies
between the mentioned metrics. We employ it on the data from all performed
experiments, i.e., form runs with different combinations of interventions.
Moreover, we do not have a division into RR=2 and RR=16, only the results from
both setups are combined, and analyses are made on them. Overestimation,
gradient norm, and parameters norm are analyzed in a logarithmic scale for
precision. We exclude metric pairs where the p-value of correlation is above
$5\%$ by whitening tiles in the correlation matrix.

Figure 5: Spearman correlation matrix for explanatory metrics on DMC
benchmark. Blank spaces are correlations that do not meet the p-value. Figure
6: Spearman correlation matrix for explanatory metrics on MetaWorld benchmark.
Blank spaces are correlations that do not meet the p-value.

##### Spearman correlation:

Figures 5 and 6 depict Spearman correlation matrices for DMC and MW
benchmarks, respectively. Notably, overestimation exhibits the strongest
correlation with agent returns on both benchmarks, offering insights into the
findings of previous sections regarding the limited robustness of critic
regularization methods designed to minimize overestimation.

Interestingly, as shown in Figure 7, layer norm and spectral norm effectively
mitigate approximation errors, outperforming CR methods. Further investigation
on the DMC benchmark (Figure 5) uncovers a strong negative correlation between
the percentage of dormant neurons and performance, closely tied to the rank of
representations on the critic’s penultimate layer.

Moreover, on the MW benchmark (Figure 6), the second-best correlated metrics
with performance are the critic gradient norm and the critic parameters norm.
This aligns with the results from section 4.2, highlighting the significant
performance boost provided by spectral norm and weight decay, particularly in
the RR=16 setup. An in-depth analysis of how RR increases the negative
correlation of gradient norm and return can be found in sections D.2 and D.3
of Appendix.

Figure 7: IQM overestimation in logarithmic scale. Each plot presents sorted
results. The IQM is calculated based on the average of the last ten evaluation
points in each run, not the last evaluation point. Left Figure: DMC benchmark.
Right Figure: MW benchmark. Colours indicate hierarchy on the plot, not
specific names.

##### Spearman correlation in dog environments:

To deepen analysis from the previous paragraph, we investigate the relations
between variables separately for every environment. Interestingly, for the DMC
benchmark, we find that overestimation is not a good predictor for dog
environments where we observe the highest values of overestimation (Figure 8).
We hypothesize that overestimation becomes a good predictor of performance
only when more fundamental issues, such as plasticity, are mitigated,
indicating a multifaceted learning problem in harder environments.

As an observation that supports this hypothesis, we refer to the critic
gradient norm, which exhibits the most monotonic relation with the return in
dog environments, as indicated by Spearman correlation (Figure 8). The high
gradient norm directly points to high curvature of the loss landscape, which,
as indicated by (Lee et al., 2023), describes low input plasticity. For this
reason, layer norm primarily smoothens the activation distribution and plays a
critical role in making SAC work in dog environments. Enviroments from the MW
benchmark also encounter challenges with high curvature loss landscapes (as
indicated by the Spearman correlation between gradient norm and return). This
suggests a resemblance between MW and DMC dog environments.

Figure 8: Explanatory metrics correlations for three different groups of
environment, namely: MetaWorld, DMC Dog environments, and DMC environments
without Dog environments. It’s important to observe that not only does the
main explanatory metric, gradient norm, vary for dog environments, but the
remaining DMC environments also exhibit a different correlation sign for this
metric.

##### Takeaways:

• Environments can differ significantly in terms of explanatory metrics
correlations, even those from the same benchmark suite.  • Techniques like
layer or spectral norm and resets are particularly effective in mitigating
overestimation also compared to methods specifically designed for that
purpose. • The negative correlation of the critic’s gradient norm and return
becomes more apparent in challenging environments and mainly in a high replay
ratio regime.

##  5 Related Works

The literature on deep reinforcement learning has long explored various
factors contributing to performance challenges, approaching the issue from
different perspectives. A notable study, akin to our pragmatic approach,
investigates the impact of diverse design choices in the training process of
on-policy methods (Andrychowicz et al., 2021).

In the realm of off-policy methods, numerous hypotheses have been proposed
regarding crucial factors. One key focus is on addressing the overestimation
problem, with attempts to harness its potential benefits (Ciosek et al., 2019)
and, more prominently, to mitigate the phenomenon by introducing novel loss
functions (Fujimoto et al., 2018; Moskovitz et al., 2021; Cetin & Celiktutan,
2023).

Regularization schemes have proven effective in enhancing deep reinforcement
learning methods. Neural network regularizations, such as Spectral Norm
(Gogianu et al., 2021; Bjorck et al., 2021), Layer Norm (Bjorck et al., 2021;
Ball et al., 2023), or weight decay (Liu et al., 2020), have yielded
significant improvements in results. In contrast, periodic resets of critic
weights were proposed by (Nikishin et al., 2022). An ensemble approach was
also suggested to address the challenge of determining the optimal number of
gradients per environment step, where decisions are based on the validation TD
error (Li et al., 2022).

##  6 Limitations

In this work, we found crucial choices that drive SAC effectiveness in a wide
range of control tasks from two popular benchmarks. Through extensive
experiments, we uncovered perplexities concerning explanatory metrics
correlations and complex dynamics of overestimation, which is successfully
mitigated by widely used regularizations. Nevertheless, our study has certain
constraints. Our empirical evaluations were limited to proprioceptive tasks on
DMC and MetaWorld benchmarks and only for SAC method.

##  7 Conclusions

This study explored different common RL design choices, as well as
interactions thereof, evaluating their impact on agent learning. Specifically,
we consider three types of regularization families: critic regularization
(motivated by value overestimation); network regularization (motivated by
model overfitting); and plasticity regularization (motivated by plasticity
loss). Our analysis revealed that generic network regularization methods such
as layer normalization, especially when paired with full-parameter resets, can
have vastly greater impact on the final performance than domain-specific RL
approaches. To this end, the same network regularization methods can lead to
strong performing policies on domains previously solved by model-based agents,
such as the dog domain. Furthermore, we studied a variety of metrics which
were shown to co-occur with deterioration of learning in low and high replay
regimes. Our analysis revealed the complex interactions of the considered
metrics with agent performance. Surprisingly, we found that interventions
motivated by a specific problem, for example overfitting, can have a
pronounced impact on the metrics associated with overestimation or plasticity.
Finally, we found that the effectiveness of considered critic, network and
plasticity regularization techniques is not only highly dependent on the
simulation benchmark, but also type of simulated task. The most prominent
example is Clipped Double Q-learning, a technique used in majority of modern
actor-critic algorithms, is effective in DMC locomotion tasks, but leads to
significant performance deterioration on the MetaWorld manipulation tasks. To
this end, we highlighted the need of testing new algorithms on a diverse set
of tasks, preferable stemming from more than one suite.

#### Impact Statement

This paper focuses on the issue of interplay between different design choices
within Reinforcement Learning (RL) algorithms. While the successful
application of RL has the potential to influence society in many ways, our
work, focused primarily on a technical advancement in RL algorithms, does not
introduce novel ethical considerations beyond those already inherent in the
broader field of RL.

## References

  * Abbas et al. (2023) Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C.  Loss of plasticity in continual deep reinforcement learning.  _arXiv preprint arXiv: 2303.07507_ , 2023.
  * Achille et al. (2017) Achille, A., Rovere, M., and Soatto, S.  Critical learning periods in deep neural networks.  _arXiv preprint arXiv: 1711.08856_ , 2017.
  * Agarwal et al. (2021) Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M.  Deep reinforcement learning at the edge of the statistical precipice.  _Advances in neural information processing systems_ , 34:29304–29320, 2021.
  * Andrychowicz et al. (2021) Andrychowicz, M., Raichuk, A., Stańczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., et al.  What matters in on-policy reinforcement learning? a large-scale empirical study.  In _ICLR 2021-Ninth International Conference on Learning Representations_ , 2021.
  * Ash & Adams (2020) Ash, J. and Adams, R. P.  On warm-starting neural network training.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 3884–3894. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/288cd2567953f06e460a33951f55daaf-Paper.pdf.
  * Ba et al. (2016) Ba, J. L., Kiros, J. R., and Hinton, G. E.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Ball et al. (2023) Ball, P. J., Smith, L., Kostrikov, I., and Levine, S.  Efficient online reinforcement learning with offline data.  _arXiv preprint arXiv: 2302.02948_ , 2023.
  * Bjorck et al. (2021) Bjorck, J., Gomes, C. P., and Weinberger, K. Q.  Towards deeper deep reinforcement learning with spectral normalization.  In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , pp. 8242–8255, 2021.  URL https://proceedings.neurips.cc/paper/2021/hash/4588e674d3f0faf985047d4c3f13ed0d-Abstract.html.
  * Brock et al. (2018) Brock, A., Donahue, J., and Simonyan, K.  Large scale gan training for high fidelity natural image synthesis.  _International Conference on Learning Representations_ , 2018.
  * Cetin & Celiktutan (2023) Cetin, E. and Celiktutan, O.  Learning pessimism for reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pp. 6971–6979, 2023.
  * Chen et al. (2020) Chen, X., Wang, C., Zhou, Z., and Ross, K. W.  Randomized ensembled double q-learning: Learning fast without a model.  In _International Conference on Learning Representations_ , 2020.
  * Ciosek et al. (2019) Ciosek, K., Vuong, Q., Loftin, R., and Hofmann, K.  Better exploration with optimistic actor critic.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Degrave et al. (2022) Degrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F., Ewalds, T., Hafner, R., Abdolmaleki, A., de Las Casas, D., Donner, C., Fritz, L., Galperti, C., Huber, A., Keeling, J., Tsimpoukelli, M., Kay, J., Merle, A., Moret, J.-M., Noury, S., Pesamosca, F., Pfau, D., Sauter, O., Sommariva, C., Coda, S., Duval, B., Fasoli, A., Kohli, P., Kavukcuoglu, K., Hassabis, D., and Riedmiller, M.  Magnetic control of tokamak plasmas through deep reinforcement learning.  _Nature_ , 602(7897):414—419, February 2022\.  ISSN 0028-0836.  doi: 10.1038/s41586-021-04301-9.
  * Dohare et al. (2021) Dohare, S., Sutton, R. S., and Mahmood, A. R.  Continual backprop: Stochastic gradient descent with persistent randomness.  _arXiv preprint arXiv: 2108.06325_ , 2021.
  * D’Oro et al. (2022) D’Oro, P., Schwarzer, M., Nikishin, E., Bacon, P.-L., Bellemare, M. G., and Courville, A.  Sample-efficient reinforcement learning by breaking the replay ratio barrier.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Foret et al. (2020) Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B.  Sharpness-aware minimization for efficiently improving generalization.  _arXiv preprint arXiv: 2010.01412_ , 2020.
  * Fujimoto et al. (2018) Fujimoto, S., Hoof, H., and Meger, D.  Addressing function approximation error in actor-critic methods.  In _International conference on machine learning_ , pp. 1587–1596. PMLR, 2018.
  * Gogianu et al. (2021) Gogianu, F., Berariu, T., Rosca, M. C., Clopath, C., Busoniu, L., and Pascanu, R.  Spectral normalisation for deep reinforcement learning: an optimisation perspective.  In _International Conference on Machine Learning_ , pp. 3734–3744. PMLR, 2021.
  * Haarnoja et al. (2017) Haarnoja, T., Tang, H., Abbeel, P., and Levine, S.  Reinforcement learning with deep energy-based policies.  In _International conference on machine learning_ , pp. 1352–1361. PMLR, 2017.
  * Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018.
  * Hafner et al. (2023) Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T.  Mastering diverse domains through world models.  _arXiv preprint arXiv:2301.04104_ , 2023.
  * Hansen et al. (2022) Hansen, N., Wang, X., and Su, H.  Temporal difference learning for model predictive control.  In _International Conference on Machine Learning, PMLR_ , 2022.
  * Hiraoka et al. (2021) Hiraoka, T., Imagawa, T., Hashimoto, T., Onishi, T., and Tsuruoka, Y.  Dropout q-functions for doubly efficient reinforcement learning.  In _International Conference on Learning Representations_ , 2021.
  * Janner et al. (2019) Janner, M., Fu, J., Zhang, M., and Levine, S.  When to trust your model: Model-based policy optimization.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Kumar et al. (2020) Kumar, A., Agarwal, R., Ghosh, D., and Levine, S.  Implicit under-parameterization inhibits data-efficient deep reinforcement learning.  _arXiv preprint arXiv: 2010.14498_ , 2020.
  * Kumar et al. (2023) Kumar, S., Marklund, H., and Roy, B. V.  Maintaining plasticity in continual learning via regenerative regularization.  _arXiv preprint arXiv: 2308.11958_ , 2023.
  * Lee et al. (2023) Lee, H., Cho, H., Kim, H., Gwak, D., Kim, J., Choo, J., Yun, S.-Y., and Yun, C.  Plastic: Improving input and label plasticity for sample efficient reinforcement learning.  _NEURIPS_ , 2023.
  * Li et al. (2022) Li, Q., Kumar, A., Kostrikov, I., and Levine, S.  Efficient deep reinforcement learning requires regulating overfitting.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Liu et al. (2020) Liu, Z., Li, X., Kang, B., and Darrell, T.  Regularization matters in policy optimization-an empirical study on continuous control.  In _International Conference on Learning Representations_ , 2020.
  * Loshchilov & Hutter (2017) Loshchilov, I. and Hutter, F.  Decoupled weight decay regularization.  _International Conference on Learning Representations_ , 2017.
  * Lyle et al. (2022) Lyle, C., Rowland, M., and Dabney, W.  Understanding and preventing capacity loss in reinforcement learning.  _International Conference on Learning Representations_ , 2022.  doi: 10.48550/arXiv.2204.09560.
  * Lyle et al. (2023) Lyle, C., Zheng, Z., Nikishin, E., Avila Pires, B., Pascanu, R., and Dabney, W.  Understanding plasticity in neural networks.  In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 23190–23211. PMLR, 23-29 Jul 2023.
  * Miyato et al. (2018) Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y.  Spectral normalization for generative adversarial networks.  _International Conference on Learning Representations_ , 2018.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * Moskovitz et al. (2021) Moskovitz, T., Parker-Holder, J., Pacchiano, A., Arbel, M., and Jordan, M.  Tactical optimism and pessimism for deep reinforcement learning.  _Advances in Neural Information Processing Systems_ , 34:12849–12863, 2021.
  * Nikishin et al. (2022) Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P.-L., and Courville, A.  The primacy bias in deep reinforcement learning.  In _International conference on machine learning_ , pp. 16828–16847. PMLR, 2022.
  * Nikishin et al. (2023) Nikishin, E., Oh, J., Ostrovski, G., Lyle, C., Pascanu, R., Dabney, W., and Barreto, A.  Deep reinforcement learning with plasticity injection.  In _Workshop on Reincarnating Reinforcement Learning at ICLR 2023_ , 2023.  URL https://openreview.net/forum?id=O9cJADBZT1.
  * OpenAI et al. (2019) OpenAI, :, Berner, C., et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv: 1912.06680_ , 2019.
  * Puterman (2014) Puterman, M. L.  _Markov decision processes: discrete stochastic dynamic programming_.  John Wiley & Sons, 2014.
  * Schwarzer et al. (2023) Schwarzer, M., Ceron, J. S. O., Courville, A., Bellemare, M. G., Agarwal, R., and Castro, P. S.  Bigger, better, faster: Human-level atari with human-level efficiency.  In _International Conference on Machine Learning_ , pp. 30365–30380. PMLR, 2023.
  * Silver et al. (2014) Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M.  Deterministic policy gradient algorithms.  In _International conference on machine learning_ , pp. 387–395. PMLR, 2014.
  * Silver et al. (2017) Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al.  Mastering the game of go without human knowledge.  _nature_ , 550(7676):354–359, 2017.
  * Smith et al. (2022) Smith, L., Kostrikov, I., and Levine, S.  A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning.  _arXiv preprint arXiv:2208.07860_ , 2022.
  * Sokar et al. (2023) Sokar, G., Agarwal, R., Castro, P. S., and Evci, U.  The dormant neuron phenomenon in deep reinforcement learning.  _International Conference on Machine Learning_ , 2023.  doi: 10.48550/arXiv.2302.12902.
  * Sutton (2019) Sutton, R.  The bitter lesson.  _Incomplete Ideas (blog)_ , 13(1), 2019.
  * Sutton & Barto (2018) Sutton, R. S. and Barto, A. G.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Tassa et al. (2018) Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al.  Deepmind control suite.  _arXiv preprint arXiv:1801.00690_ , 2018.
  * Thrun & Schwartz (2014) Thrun, S. and Schwartz, A.  Issues in using function approximation for reinforcement learning.  In _Proceedings of the 1993 connectionist models summer school_ , pp. 255–263. Psychology Press, 2014.
  * Yu et al. (2019) Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In Kaelbling, L. P., Kragic, D., and Sugiura, K. (eds.), _3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings_ , volume 100 of _Proceedings of Machine Learning Research_ , pp. 1094–1100. PMLR, 2019.  URL http://proceedings.mlr.press/v100/yu20a.html.
  * Zhang et al. (2018) Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A.  Self-attention generative adversarial networks.  _arXiv preprint arXiv: 1805.08318_ , 2018.
  * Ziebart et al. (2008) Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al.  Maximum entropy inverse reinforcement learning.  In _Aaai_ , volume 8, pp. 1433–1438. Chicago, IL, USA, 2008.

##  Appendix A Derivations - Variance

##  Appendix B Details of experiments

Results reporting Interquartile mean (IQM) are based on 500 boostrapping
points as calculated by rliable package (Agarwal et al., 2021). The final
performance is defined as average of last 10 policy evaluations.

##  Appendix C Architecture details

In all experiments, the Actor and Critic are represented by three-layer MLP
networks, each containing 256 neurons in the hidden layers, utilizing the ReLU
activation function (except for the CReLU variant, which effectively doubles
the number of activations).

In the scenario involving Layer Norm, it is applied to each hidden layer (Li
et al., 2022; Ball et al., 2023), while spectral norm is applied exclusively
to the last hidden layer (Gogianu et al., 2021; Li et al., 2022). Weight decay
is uniformly applied across all layers (Li et al., 2022). It’s important to
note that all network regularizations are exclusively applied to the Critic
network.

###  C.1 Hyperparameters

All hyperparameters are taken from original papers introducing given
intervention.

Table 1: Hyperparameter values used in the experiments. Hyperparameter | Notation | Value
---|---|---
Joint
Network Size | na | $(256,256)$
Action Repeat | na | $1$
Optimizer | na | Adam
Learning Rate | na | $3e-4$
Batch Size | $B$ | $256$
Discount | $\gamma$ | $0.99$
Initial Temperature | $\alpha_{0}$ | $1.0$
Initial Steps | na | $10000$
Target Entropy | $\mathcal{H}^{*}$ | $|\mathcal{A}|/2$
Polyak Weight | $\tau$ | $0.005$
TOP
Pessimism Values | $\beta$ | $\\{0,1\\}$
Bandit Learning Rate | na | $0.1$
GPL
Pessimism Learning Rate | na | $1e-5$

##  Appendix D Further Experiments

###  D.1 Third-order marginalization

Examining the plots without marginalization (Figure 9) provides further
insights into the conclusions drawn from the previous experiment.
Specifically, most combinations without critic regularization (red points)
consistently perform well across all setups. Additionally, the results for GPL
(pink points) affirm the overall subpar performance of this method.

Figure 9: DMC (top) MW (bottom).

On the DMC with RR=2 plot (top-left in Figure 9), layer norm (points labeled
”L”) consistently outperforms others in mean return, irrespective of the
Plasticity regularization (x-axis) or Critic Regularization (color). Notably,
the combination of layer norm and resets in RR=16 (top-right plot in Figure 9)
demonstrates exceptional performance across all critic regularization
variations.

For the MW benchmark with RR=16 (bottom-right plot in Figure 9), the results
align with first-order marginalization findings, highlighting the positive
impact of Spectral Normalization. Particularly interesting is the role of
Weight Decay, forming optimal combinations with the Reset method on RR=16 and
consistently performing well across various combinations on RR=2 (bottom-left
plot in Figure 9).

In a scenario of high replay ratio, the resets are the most important
interevention. They work best in combination with Layer norm, but other neural
network regularization methods combined with resets are at the forefront
regarding performance in the rr=16 scenario (see top of the Figure 9). We
present more results in Figures 20 and 21.

###  D.2 Gradient Norm Analysis

Drawing insights from the findings in section 4.2, where we highlighted the
significance of spectral norm in enhancing agent performance on the MW
benchmark, we now delve deeper into the behavior of the critic’s gradient
norm. In Figure 10, one can compare orders of magnitude of IQM of gradient
norm with respect to different replay ratio (RR) regimes and on different
benchmarks. Referring to results from section 4.2, Figure 10 underscores that
the gradient norm on MetaWorld, particularly in the RR=16 setup, exhibits
orders of magnitude higher values compared to the DMC benchmark.

A similar phenomenon can be observed on the DMC benchmark, but layer norm
proves more robust in mitigating exploding gradients than Spectral Norm.
Interestingly, training on very complex environments such as Dog causes
enormous gradient explosion, even in a small replay ratio regime 11.

Figure 10: IQM gradient norm of first-order results. The IQM is calculated
based on the average of the last ten evaluation points in each run, not the
last evaluation point. Colors indicate hierarchy on the plot, not specific
names.

Figure 11: IQM gradient norm of first-order results. The IQM is calculated
based on the average of the last ten evaluation points in each run, not the
last evaluation point. Colors indicate hierarchy on the plot, not specific
names.

###  D.3 Regression and Spearman correlation analysis

Deepening analysis from section 4.4, we investigate the coefficients of
Ordinary Least Square regression for overestimation, overfitting, dormant
neurons and critic’s gradient norm with return. Except for dormant neurons, we
take the symmetrical logarithm of every metric. We perform regression fitting
for every variable separately, which results in four different regression
analyses. As previously stated, because these relations are highly non-linear,
however, often monotonic, goodness of fit coefficients such as $R^{2}$ for
linear regression are low. Besides regression coefficients ($\beta$), we
report a standard error of regression ($se$), $R^{2}$, p-value $p$ in the plot
legend. In the axis titles, the Spearman correlation and its p-value are
shown.

In figures 12 and 13, we observe consistent negative coefficients, except for
the Coffee environment for overestimation and return. Interestingly, for the
MetaWorld benchmark, most environments exhibit a symmetrical degree of
overestimation and underestimation as two distinct clusters of well-performing
runs are mirrored by the Y axis. For the DMC environments, we observe a much
higher amount of runs in which overestimation is present.

For the gradient norm, we observe a higher value of this metric for runs with
RR=16 both for DMC and MW benchmarks (Figures 14 and 15. There is consistency
in coefficients for MW; however, in DMC, we observe that only four out of 7
environments exhibit negative signs of the coefficient. However, for most of
the extremely high values of gradient norm, we systematically observe low
returns regardless of the environment.

Dormant neurons, presented in Figures 16 and 17, clearly correlate negatively
with the return, especially for DMC environments. However, there are cases,
such as the Reach environment from MetaWorld, where a high percentage of
dormant neurons benefit the agent, probably because this particular
environment is especially easy.

Overfitting is the metric that exhibits the highest variance in coefficient
signs across environments and benchmarks. As shown in Figures 18 and 19, the
best-performing runs are located for low absolute values of overfitting. In
addition, RR=16 clearly results in higher overfitting values.

Figure 12: Overestimation logarithm scatter plots with regression line for DMC
environments. Figure 13: Overestimation logarithm scatter plots with
regression line for MW environments. Figure 14: Gradient norm logarithm
scatter plots with regression line for DMC environments. Figure 15: Gradient
norm logarithm scatter plots with regression line for MW environments. Figure
16: Dormant neurons scatter plots with regression line for DMC environments.
Figure 17: Dormant neurons scatter plots with regression line for MW
environments. Figure 18: Overfitting logarithm scatter plots with regression
line for DMC environments. Figure 19: Overfitting logarithm scatter plots with
regression line for MW environments.

###  D.4 Best combinations of intervention performance plots

Figure 20: Top performing configuration in the low replay regime. 10 seeds per
task per algorithm.

Figure 21: Top performing configuration in the high replay regime. 10 seeds
per task per algorithm.

###  D.5 Other

Figure 22: Top performing configuration in the low (left) and high (right)
replay regime. 10 seeds per task per algorithm. Figure 23: RR changes on
metrics - MOVE TO APPENDIX