textnode=[fill=blue!15, align=left, draw=black, outer sep=2pt, inner sep=2pt, rounded corners, text width=3.0cm, minimum height=1.0cm, minimum width=3cm]
responsenode=[fill=gray!5, align=left, draw=black, outer sep=2pt, inner sep=2pt, rounded corners, minimum height=1.0cm, text width=3.0cm, minimum width=2cm]
postedge=[- >, thick, draw=blue]
commentedge=[- >, thick, draw=red]
genericedge=[- >, thick]
genericnode=[fill=green!10, draw=black, rounded corners, outer sep=2pt, inner sep=2pt]
examplepostnode=[fill=blue!15, align=left, draw=black, outer sep=2pt, inner sep=2pt, rounded corners, text width=4.5cm, minimum size=1.0cm]
exampleresponsenode=[fill=gray!5, align=left, draw=black, outer sep=2pt, inner sep=2pt, rounded corners, minimum height=1.0cm, text width=4.5cm, minimum width=3cm]

Analysis of Moral Judgement on Reddit

Nicholas Botzer, Shawn Gu, and Tim Weninger
Department of Computer Science and Engineering
University of Notre Dame

{nbotzer, sgu3<EMAIL_ADDRESS>
Moral outrage has become synonymous with social media in recent years. However, the preponderance of academic analysis on social media websites has focused on hate speech and misinformation. This paper focuses on analyzing moral judgements rendered on social media by capturing the moral judgements that are passed in the subreddit /r/AmITheAsshole on Reddit. Using the labels associated with each judgement we train a classifier that can take a comment and determine whether it judges the user who made the original post to have positive or negative moral valence. Then, we use this classifier to investigate an assortment of website traits surrounding moral judgements in ten other subreddits, including where negative moral users like to post and their posting patterns. Our findings also indicate that posts that are judged in a positive manner will score higher.

§ INTRODUCTION

How do people render moral judgements of others?
This question has been pondered for millennia. Aristotle, for example, considered morality in relation to the end or purpose for which a thing exists. Kant insisted that one's duty was paramount in determining what course of action might be good. Consequentialists argue that actions must be evaluated in relation to their effectiveness in bringing about a perceived good.
Regardless of the particular ethical frame that one ascribes to, the common practice of evaluating others' behavior in moral terms is widely regarded as important for the well-being of a community. Indeed, ethnographers and sociologists have documented how these kinds of moral judgements actually increase cooperation within a community by punishing those who commit wrongdoings and informing them of what they did wrong [3].

The process of rendering moral judgement has taken an interesting turn in the current era with the adoption of the Internet and social media in particular. Online social systems allow people to encounter and consider the lives of others from around the world.
At no other time in history have so many people been able to examine  such a variety of cultures and viewpoints so readily. This increased sharing and mixing of viewpoints inevitably leads to online debates about various topics [29].
The content of these debates provides researchers with the opportunity to ask specific questions about argument, disagreement, moral evaluation, and judgement with the aid of new computational tools.

To that end, recent work has resulted in the creation of statistical models that can understand moral sentiment in text [25].
However, these models rely heavily on a gazette of words and topics and their alignment on moral axes.
The central motivation for these works are grounded in moral foundation theory [16] where studies also tend to investigate the use of morality as related to current events in the news such as politics or religion.
Despite their usefulness in understanding the moral valence of specific current events, the goal of the current work is to study moral judgements rendered on social media that apply to more common personal situations.

Example of (a) a post title and (b) a comment in the /r/AmItheAsshole subreddit. The NTA prefix and comment-score (not shown) indicates that the commenter judged the poster as “Not the Asshole”.

We focus on Reddit in particular, where users can create posts and have discussions in threaded comment-sections.
Although the details are complicated, users also perform curation of posts and comments through upvotes and downvotes based on their preference [12, 15].
This assigns each post and comment a score reflecting how others feel about the content.
Within Reddit there are a large number of subreddits, which are small communities dedicated to various topics.
The subreddit of interest for our question regarding moral judgements is called /r/AmItheAsshole.
Users of this subreddit post a description of a situation that they were involved in; they are also encouraged to explain details of the people involved and the final outcome of the situation.
Posters to /r/AmItheAsshole are typically looking to hear from other Reddit users whether or not they handled their personal situation in an ethically appropriate manner.
Other users then respond to the initial post with a moral judgement as to whether the original user was an asshole or not the asshole.
Figure <ref> shows an example of a typical post and one of its top responses.
One important rule of /r/AmItheAsshole is that top-level responses must categorize the behavior described in the original post to one of four categories: Not the Asshole (NTA), You're the Asshole (YTA), No assholes here (NAH), Everyone sucks here (ESH).
In addition to providing a categorical moral judgement, the responding user must also provide an explanation as to why they selected that choice.
Reddit's integrated voting system then allows other users to individually rate the judgements with which they most agree (upvote) or disagree (downvote). After some time has passed the competition among different judgements will settle, and one of the judgements will be rated highest. This top comment is then accepted as the judgement of the community.
This process of passing and rating moral judgement provides a unique view into our original question about how people make moral judgements.

Compared to other methodologies of computational evaluation of moral sentiments, collecting judgements from /r/AmItheAsshole (AITA) has some important benefits.
First, because posters and commenters are anonymous on Reddit, they are more likely to share their sensitive stories and frank judgements without fear of reprisal [21, 18].
Second, the voting mechanism of Reddit allows a large number of users to engage in an aggregated judgement in response to the original post  [13]. However, the breadth and variety of this data does pose additional challenges. For instance, judgements are provided without an explicit moral-framing, and, similarly, Reddit-votes do not explicitly denote moral valence and are susceptible to path dependency effects [14].

In the present work we use data from AITA to investigate how users provide moral judgements of others. We then extract representative judgement-labels from each comment and use these labels and comments to train a classifier. This classifier is then broadly applied to infer the moral valence of other Reddit comments from ten different subreddits and used to answer the following research questions:

RQ1: What language is most closely associated with positive and negative moral valence?

RQ2: Is moral valence correlated with the score of a post?

RQ3: Do certain subreddit-communities attract users whose posts are typically classified by more negative or positive moral judgements?

RQ4: Are self-reported gender and age descriptions associated with positive or negative moral judgements?

In summary, we find that posts that are judged to have positive moral valence (, NTA label) typically score higher than posts with negative moral valence. We also find that certain subreddit-communities where users confess to something immoral (, such as /r/confessions) tend to attract users whose posts are characterized by negative moral valence. Among these immoral users we show that their posting habits tend towards three different types. Finally, we show that self-described male-users are more likely to be judged an asshole than female-users.

§ METHODOLOGY

We retrieve moral judgements by collecting posts and comments from the subreddit /r/AmItheAsshole, taken from the Pushshift data repository [1].

The questions raised in the present work are considered human subjects research, and relevant ethical consideration are present. We sought and received research approval from the Institution Review Board at the University of Notre Dame under protocol #20-01-5751.

Label     Meaning          # Comments
NTA              Not the Asshole         717,006

YTA              You're the Asshole      372,850

NAH              No assholes here       91,903

ESH              Everyone sucks here     79,059

The four judgements that users can pass on the subreddit /r/AmItheAsshole.

We restricted our data collections to posts submitted between January 1, 2017, and August 31, 2019. In order to assure that labels reflected the result of robust discussion we excluded those posts containing fewer than 50 comments. Subreddit rules require that top-level comments begin with one of four possible prefix-labels indicated in Tab. <ref>. Because of this rule, we further restrict our data collection to contain only top-level comments and their prefix-label. Comments with the INFO prefix, which indicates a request for more information, and comments with no prefix are also removed from consideration. This methodology resulted in a collection of 7,500 posts and 1,260,818 comments with explicit moral judgements. Posters and commenters appear to put a lot of thought and effort into these discussions. Each post contains 381 words on average and each comment contains 57 words on average.

§.§ Linguistic Analysis of Moral Judgement

Before we introduce our classifier, we first consider RQ1: what linguistic cues are associated with positive and negative moral judgement?  To answer this question we split comments into two valence classes: positive and negative. The positive class contains comments that are labeled NTA or NAH; the negative class contains comments that are labeled YTA or ESH. Then we use the Allotaxonmeter system, which compares two Zipfian distributions using a scoring function called rank-turbulence divergence, to compare how terms are associated with these valence labels [9]. In our case, we constructed 1-gram multinomial distributions from each class, which, in the English language, is well known to exhibit a Zipfian distribution [22].

Negative Valence     Positive Valence
[remember picture, overlay]
[fill=gray!20, anchor=east, rectangle, minimum width=3cm, text width=3cm, inner sep=.5,outer sep=0, align=right]Tyyou;

[remember picture, overlay]
[fill=blue!10, anchor=west, rectangle, minimum width=3cm, text width=3cm, inner sep=.5,outer sep=0, align=left]toTy;

[remember picture, overlay]
[fill=gray!20, anchor=east, rectangle, minimum width=7.5mm, text width=7.5mm, inner sep=.5,outer sep=0, align=right]quilt;

[remember picture, overlay]
[fill=blue!10, anchor=west, rectangle, minimum width=7.2mm, text width=7.2mm, inner sep=.5,outer sep=0, align=left]sheTy;

[remember picture, overlay]
[fill=white, anchor=east, rectangle, minimum width=7.4mm, text width=18.3mm, inner sep=.5,outer sep=0, align=right]beiause;
[fill=gray!20, anchor=east, rectangle, minimum width=7.3mm, text width=7.3mm, inner sep=.5,outer sep=0, align=right]causeT;

[remember picture, overlay]
[fill=blue!10, anchor=west, rectangle, minimum width=7.1mm, text width=7.1mm, inner sep=.5,outer sep=0, align=left]myTy;

[remember picture, overlay]
[fill=white, anchor=east, rectangle, minimum width=7.1mm, text width=18.3mm, inner sep=.5,outer sep=0, align=right]iTynn;
[fill=gray!20, anchor=east, rectangle, minimum width=7.1mm, text width=7.1mm, inner sep=.5,outer sep=0, align=right]nternTy;

[remember picture, overlay]
[fill=blue!10, anchor=west, rectangle, minimum width=7.0mm, text width=7.0mm, inner sep=.5,outer sep=0, align=left]cornellTy;

[remember picture, overlay]
[fill=gray!20, anchor=east, rectangle, minimum width=6.8mm, text width=6.8mm, inner sep=.5,outer sep=0, align=right]suckT;

[remember picture, overlay]
[fill=blue!10, anchor=west, rectangle, minimum width=6.6mm, text width=6.6mm, inner sep=.5,outer sep=0, align=left]theyTy;

Rank divergence scores of terms associated with positive and negative moral valences. Color bars indicate the relative contribution of terms, ,  contributes about 4 times as much to negative valence as .

Method     Accuracy     Precision     Recall     F1

Doc2Vec Embeddings       65.92 $\pm$ 0.04     61.22 $\pm$ 0.15     13.5 $\pm$ 0.09     22.1 $\pm$ 0.09

BERT Embeddings         70.10 $\pm$ 0.07     64.28 $\pm$ 0.2     36.96 $\pm$ 0.14     46.96 $\pm$ 0.08

Multinomial Naïve Bayes     72.12 $\pm$ 0.17     62.58 $\pm$ 0.17     55.22 $\pm$ 0.07     58.66 $\pm$ 0.05

Judge-BERT     89.03 $\pm$ 0.13     85.57 $\pm$ 0.18     83.48 $\pm$ 0.27     84.51 $\pm$ 0.17

Classification results on the AITA dataset. Results are mean-averages and standard deviations over five-fold cross-validation.

Table <ref> shows the terms that contain the largest divergence contribution for each valence class. Words with the highest negative valence include , , and  in the top 5, but also ,  and  within the top 10 (not shown). Simply put, these are the top words that are used when assigning negative moral judgement. Words associated with positive moral valence consist mainly of functional terms, but also include the names several ivy league schools in the top 50[A brief survey of these posts reveals that many posters ask if they are wrong to attend one of these schools even though their family member or partner was not admitted.].

§.§ Prediction Model

Given the dataset, a dataset with textual posts and textual comments labeled with positive or negative moral judgements, our goal is to predict whether an unlabeled comment assigns a positive (NTA or NAH) or negative (YTA or ESH) moral judgement to the user of the post.
It is important to note that this classifier is classifying the judgement of the commenter, not the morality of the poster.

We define our problem formally as follows.

Problem Definition
Given a top level comment $C$ with moral judgement $A \in \{+,-\}$ that responded to post $P$ we aim to find a predictive function $f$ such that

\begin{equation}
f : (C) \rightarrow A
\end{equation}

Formally, this takes the form of a text classification task where class inference denotes the valence of a moral judgement.

The choice of classification model $f$ is not particularly important, but we aim to train a model that performs well and generalizes to other datasets.

We selected four text classification models for use in the current work:

* Multinomial Naïve Bayes [19]: Uses word counts to learn a text classification model and has shown success in a wide variety of text classification problems.

* Doc2Vec [20]: Create comment-embeddings, which are input into a logistic regression classifier that calculates the class margin.

* BERT Embeddings [8]: Uses word embeddings from BERT, which are averaged together and input into a logistic regression classifier that calculates the class margin.

* Judge-BERT: We fine-tune the BERT-base model using the class labels. Specifically, we added a single dropout layer after BERT's final layer, followed by a final output layer that consists of our two classes. The model is trained using the Adam optimizer and uses the cross entropy loss function. We then trained for 3 epochs as recommended by Devlin et al [8].

§.§ Judgement Classification Results

We evaluate our four classifiers using accuracy, precision, recall, and F1 metrics. In this context a false positive is the instance when the classifier improperly assigns a negative (, asshole) label to a positive judgement. A false negative is the instance when the classifier improperly assigns a positive (, non-asshole) label to a negative judgement. We perform 5-fold cross-validation and, for each metric, report the mean-average and standard deviation over the 5 folds.

The results in Table <ref> indicate that the Doc2Vec, BERT, and Multinomial Naïve Bayes classifiers do not perform particularly well at this task. Fortunately, the fine-tuned Judge-BERT classifier performs relatively well, with an accuracy near 90% and where type 1 and type 2 errors are relatively similar. Overall, these results indicate that the Judge-BERT classifier is able to accurately classify moral judgements.

§ ANALYSIS OF MORAL JUDGEMENT

Using the Judge-BERT classifier, our next tasks are to better understand moral judgement across a variety of online social contexts and to analyze various trends in moral judgement. In order to minimize the transfer-error rate it is important to select subreddit-communities that are similar to the training dataset. In total we chose ten subreddits to explore in our initial analysis. The subreddits we chose can be broken into three main stylistic groups and are briefly described in Table <ref>.

Subreddit     Description
64.3cmUsers pose questions in a scenario like the AITA dataset and receive advice or feedback on their situation.

54.3cmUsers confess to something that they have been keeping to themselves. Typically, confessions are about something immoral the poster has done.

64.8cmUsers engage in conversations with others to have a simple conversation or to here other opinions in order to change their worldview.

Subreddits used for analysis of moral judgement.

A post (in blue) made by a user along with the top response comment (white). The comment is then fed to our Judge-BERT classifier (green) to determine the moral valence of the post.

We applied the Judge-BERT classifier to the comments and posts of these ten subreddits. Specifically, given a post and its comment tree we identified the top-level comment with the highest score. This top-rated comment, which has received the most upvotes from the community, is considered to be the one passing judgement on the original poster. As illustrated in Fig. <ref>, this top-rated comment is then fed to our classifier and the resulting prediction is used to label the moral valence of the post and poster. It is important to be clear here: we are not predicting the moral valence of the comment itself, but rather the top-rated comment is used to pass judgement on the post.

Subreddit     $\mu^+$ Score     $\mu^-$ Score

/r/relationship_advice       44.32$\pm$1.77     23.90$\pm$1.81

/r/relationships     47.19$\pm$1.42     23.36$\pm$1.38

/r/dating_advice       43.49$\pm$2.51     33.45$\pm$3.03

/r/legaladvice      49.99$\pm$2.38     30.07$\pm$2.58

/r/dating      55.03$\pm$3.45     45.09$\pm$4.43

/r/offmychest      50.78$\pm$2.13     29.67$\pm$2.38

/r/TrueOffMyChest      47.93$\pm$1.40     23.95$\pm$1.38

/r/confessions      62.09$\pm$3.95     50.92$\pm$5.08

/r/CasualConversation       46.38$\pm$4.27     42.73$\pm$8.71

/r/changemyview      55.17$\pm$3.98     56.45$\pm$6.52

Mean scores and 95% confidence intervals of posts that were classified as having positive moral valence and negative moral valence. Positive posts are statistically significantly higher than negative posts ($p<0.001$).

§.§ Moral Valence and Popularity

Here we can begin to answer RQ2: Is moral valence correlated with the score of a post? In other words, do posts with positive moral valence score higher or lower than posts with negative moral valence. To answer this question, we extracted all posts and their highest scoring top-level comment from 2018 from each subreddit in Table <ref>.

Posts judged to have positive valence as a function of post score. Higher indicates more positive valence. Higher post scores are associated with more positive valence (Mann Whitney $\tau\in[0.395,0.41]$, $p<0.001$ two-tailed, Bonferroni corrected).

Popularity scores on Reddit exhibit a power-law distribution, so the mean-scores and their differences will certainly be misleading. Instead, we plot the ratio of comments judged to be positive against all comments as a function of the post score cumulatively in Fig. <ref>. Higher values in the plot indicate more positive valence. The results here are clear: post popularity is associated with positive moral valence. Most of the subreddits appear to have similar characteristics except for /r/CasualConversation, which has a much higher positive valence (on average) than the other subreddits. Mann-Whitney Tests for statistical significance on individual subreddits as well the aggregation of these tests with Bonferroni correction found that posts with positive valence have significantly higher scores than posts with negative valence ($\tau\in[0.395,0.41]$, $p<0.001$ two-tailed).

We take the additional step to argue that correlation does indeed imply causation in this particular case. Because posts are made before votes are cast, and because the text of a post is (typically) unchanged, and if we assume that scores are causally related to the text of the post, then the causal arrow can only point in one direction, , posts with positive moral valence result in higher scores than posts with negative moral valence.

These findings appear to conflict with other studies that have shown how negative posts elicit anger and encourage a negative feedback loop on social media [2, 7]. A further inspection of the posts indicated that posts classified as having positive moral valence often found users expressing that a moral norm had been breached. The difference in our results compared to others may be explained by perceived intent, that is, whether or not the moral violation occurred from an intentional agent towards a vulnerable agent, , dyadic morality [27]. Our inspection of comments expressing negative moral judgement confirms that the perceived intent of the poster is critical to the judgement rendered. These negative judgements typically highlight what the poster did wrong and advise the poster to reflect on their actions (or sometimes simply insult the poster). Conversely, we find that many posts judged to be positive clearly show that the poster is the vulnerable agent in the situation to some other intentional agent. The responses to these posts often display sympathy towards the poster and also outrage towards the other party in the scenario. These instances are perhaps best classified as examples of empathetic anger [17], which is anger expressed over the harm that has been done to another. We also note that some of the content labelled to have positive moral valence is simply devoid of a moral scenario. Examples of this can be primarily seen in /r/CasualConversation where the majority of posts are about innocuous topics.

Another possible explanation for our findings is that users on other online social media sites like Facebook and Twitter are more likely to like and share news headlines that elicit moral outrage; these social signals are then used by the site's algorithms to spread the headline further throughout the site [4, 15]. Furthermore, the content of the articles triggering these moral responses often covers current news events throughout the world. Our Reddit dataset, on the other hand, typically deal with personal stories and therefore tend to not have the same in-group/out-group reactions as those found on viral Facebook or Twitter posts.

§ ASSIGNING JUDGEMENTS TO USERS

Next we investigate RQ3: Do certain subreddit-communities attract users whose posts are typically classified by more negative or positive moral judgements? To answer this question we need to reconsider our unit of analysis. Rather than assigning moral valence to the individual post, in this analysis we consider the moral valence of the user who committed the post. To do this, we again find all posts and comments of the ten subreddits and find the highest scoring top-level comment; we classify whether that comment is judging the post to have positive or negative moral valence and then tally this for the posting user.

Of course, users are also able to post comments and sub-comments. So we expand this analysis to include judgements of users from throughout the entire comment tree. Each comment can have zero or more replies each with its own score. So, for each comment we identify the reply with the highest score and classify whether that reply is judging the comment to have positive or negative moral valence, and then tally this for the commenting user. We do this for each comment that has at least one reply at all levels in the comment tree.

By assigning moral valence scores to users we are able to capture all judgements across the ten subreddits and better-understand their behavior. It is important to remember that the classifier classifies the moral valence of text – with some amount of uncertainty – not the user specifically. So we emphasize that we do not label users as “good” or “bad” explicitly; rather, we identify users as having submitted posts and comments that were similar to comments that previously received positive or negative moral judgement.

Lorenz curve depicting the judgement inequality among users; Gini coefficient = 0.515

We include only users that were judged at least 50 times. Each user therefore has an associated count of positive and negative judgements. This begs an interesting question: are some users judged more positively or negatively than others? What does that distribution look like? To understand this breakdown we first plot a Lorenz curve in Fig. <ref>. We find that the distribution of moral valence is highly unequal: about 10% of users receive almost 40% of the observed negative judgements (Gini coefficient = 0.515).

This clearly indicates that there are a handful of users that receive the vast majority of negative judgements. To identify those users which receive a statistically significant proportion of negative judgements we perform a one-sided binomial test on each user. Simply put, this test emits a negativity probability, , the probability (p-value) that the negativity of a user is not due to chance.

Number of comments (normalized) as a the negativity threshold is raised. As the negativity threshold is raised the fraction of comments revealed tends towards 1. Higher lines indicate a higher concentration of negative users and vice versa.

Finally, we can illustrate the membership of each subreddit as a function of users' negativity probability. As expected, Fig. <ref> shows that as we increase the negativity threshold from almost certainly negative to uncertainty (from left to right) we begin to increase the fraction of comments observed. These curves therefore indicate the density of comments that are made from negative users (for varying levels of negativity); higher lines (especially on the left) indicate higher concentration of negativity. We find that /r/confessions, /r/changemyview, and /r/TrueOffMyChest contain a higher concentration of comments from more-negative users. On the opposite side of the spectrum, we find that /r/CasualConversation and /r/legaladvice have deep curves, which implies that these communities have fewer negative users than others.

§.§ Three Types of Negative Users

We select our group of users that have a statistically significant negative moral valence as those that were found to have a $p$-value less than 0.05 from our one-tailed binomial test.
Within this group we investigated into their posting habits to determine what types of posts they make to garner such a large number of negative judgements. From our analysis of these users we determined that they fall into three different stylistic groups.

* Explainer: These users will argue that what they did isn't that wrong.

* Stubborn Opinion: Users that do not acquiesce to the prevailing opinion of the responders.

* Returner: Users that repeatedly post the same situation hoping to elicit more-favorable responses.

The first type of user that we observe is the Explainer. The explainer typically makes a post and receives comments that condemn their immoral actions. In response to this judgement, the explainer will reply to many of the comments in an attempt to convince others that what they did was in fact moral. Often, this only serves to exacerbate the judgements made against them. This then leads to further negative judgements. In fact, we found that many of these users have only made a handful of posts that each have a large number of comments in self-defense. The large number of users that respond to these comments and with negative judgements is similar to the effect of online firestorms [24] but at a scale contained to only an individual post. For these types of posts we also note that some people do come to the defense of the poster, which follows similar findings that people show sympathy after a person has experienced a large amount of outrage [26].

A diagram showing the posting habits of a Returner. Posts are in the light blue boxes with blue arrows represent the order of posts. An example of a post response is shown in the white box with the red arrow representing the post it came from. Each post is prefaced with the overarching title, "Me and my partner are having a baby." followed by the current update on the situation. The response comments have also been condensed from their full length.

The second type of user we observe is the Stubborn Opinion user. These users are similar to but opposite from the Explainers. Rather than trying to change their perspective, the Stubborn Opinion user refuses to acquiesce to the prevailing opinion of the comment thread. For example, users posting to /r/changemyview that do not express a change of opinion despite the efforts and agreement of the commenting users often incur comments casting negative judgement. This back-and-forth sometimes becomes hostile. Many of these conversations end in personal attacks from one of the participants, which has also been shown in previous work on conversations derailing in /r/changemyview [5].

The third type of user is the Returner. The returner seeks repeated feedback from Reddit on the same subject. For example, when returners make posts seeking moral judgement, they will often engage in some of the discussion and may even agree with some of the critical responses. Some time later, the user will return and edit their original post or make another post providing an update about their situation. An example of a Returner is illustrated in Figure <ref>. In this case, a user continues to request advice after recently impregnating their partner. In these situations responding users often find previous posts on the same topic made by the same user and then use this information and highlight commentary from the previous post to build a stronger case against the user or highlight how the new post is nothing but a thinly-veiled attempt to shine a more-favorable light on their original situation. These attempts usually backfire and result in more negative judgments being cast against the user.

Positive   Negative
Male   53,416   26,281

Female   57,126   20,714

Positive   Negative
Male   139,163   74,384

Female   216,190   78,823

Contingency tables showing the number of positive and negative judgements for each self-reported gender. Moral valence has a small association with gender on /r/relationship_advice $\phi=0.07, p<0.01$ and /r/relationships $\phi=0.09, p<0.01$

§.§ Gender and Age Analysis

Our final task investigates RQ4: Are self-reported gender and age descriptions associated with positive or negative moral judgements? Recent studies on this topic have found that gender and moral judgements have a strong association [23]. Specifically, women are perceived to be victims more often than men and harsher punishments are sought for men. The rates at which men commit crimes tends to be higher than the rates of female crime and society generally views crimes as a moral violation [6]. If we apply these recent findings to our current research question we expect to find that male users will be judged negatively more often than females.

This task is not usually available on public social media services because gender and age are not typically revealed, while also allowing for anonymous posting. Fortunately, the posting guidelines of /r/relationships and /r/relationship_advice requires posters to indicate their age and gender in a structured manner directly in the post title. An example of this can be seen here:

where the poster uses [M27] to indicate that they identify as male aged 27 years and that their partner [F25] identifies as female aged 25 years. Using these conventions we are able to reliably extract users' self-reported age and gender.

We again apply our Judge-BERT model to assign a moral judgement to the post based on the top-scoring comment. In total we extracted judgements from 508,560 posts on /r/relationships and 157,537 posts on /r/relationship_advice. Because the posting age of a user may not be above the age of majority, we are careful to only collect data from users that are aged 18 and older via our research ethics guidelines. In general, the age breakdown appears to closely follow Reddit's age demographic. 90% of posters were between 18-30 years old.

Our first task is to determine if any association exists between moral judgement and gender. To answer this question we performed a $\chi^2$ test of independence. Contingency tables for this test are reported in Table <ref>. The $\chi^2$ test reports a significant association between gender and moral judgement in /r/relationships ($\chi^2 (1, 508,560) = 3874.6, p < .0001$) and in /r/relationship_advice ($\chi^2 (1, 157,537) = 762.2, p < .0001$). However, the $\chi^2$ test on such large sample sizes usually results in statistical significance; in fact, the $\chi^2$ test tends to find statistical significance for populations greater than 200 [28]. So we verify this association using $\phi$, which measures the strength of association controlled for population size. In this case, $\phi$ = 0.09 for /r/relationships and $\phi$ = 0.07 for /r/relationship_advice. These low values indicate that there is only a small association between gender and moral judgement.

Variable     Coefficient     p-value     95% CI

(Constant)     -1.1575      $<$0.001     (-1.2093, -1.1058)

Gender     0.3076     $<$0.001     (0.2806, 0.3241)

Age     0.0059     $<$0.001     (0.0039, 0.0080)

Variable     Coefficient     p-value     95% CI

(Constant)     -1.0923     $<$0.001     (-1.1214, -1.0631)

Gender     0.3814      $<$0.001     (0.3693, 0.3935)

Age     0.0034      $<$0.001     (0.0023, 0.0046)

Results for the logistic regression analysis for both subreddits.

§.§ Logistic Regression Analysis

Our second task is to determine if gender and age are associated with moral judgement. In other words, are young females, for instance, judged more positively than, say, old males?
To answer this question, we fit a two-variable logistic regression model where the binary-variable gender is encoded as 0 for female and 1 for male.

We report the findings from the logistic regressor for each subreddit in Table <ref>. These results indicate that males are judged more negatively than females. Specifically, in /r/relationship_advice being male is associated with a 35% increase in receiving a negative judgement. Similarly, in /r/relationships being male is associated with a 46% increase in receiving a negative judgement.

We also find that age has a relatively small effect on moral judgement; increased age is slightly  correlated with negative judgement. Specifically, in /r/relationship_advice an increase in age by one year is associated with a 0.59% increase in receiving a negative judgement. In /r/relationships an increase in age by one year is associated with a 0.34% increase in receiving a negative judgement.

Simply put, those who are older and those who are male (independently) are statistically more likely to receive negative judgements from Reddit than those who are younger and female. Although gender is much more of a contributing factor than age and neither association is particularly strong.

An interesting avenue of work that has been investigated recently is the ability to preempt a conversation if you know further discussion will result in a personal attack <cit.>.
This differs from other frameworks that try to detect these personal attacks after they have already happened.
In this task two datasets have been used one from Wikipedia discussion pages and the other from the subreddit .
Each dataset consists of a back and forth conversation between users with some ending in a personal attack.
With this task in mind we wondered whether our asshole predictor could be transferred and used to predict these personal attacks as well.
The judgements user give on  can be viewed as a form personal attacks when someone calls the original poster an asshole, since the user is expected to explain and give reasoning as to why this person is an asshole.
Due to this we wondered whether our classifier could be used on the conversations gone awry dataset to perform the same prediction task.

There are three different methods that have been used to solve this problem so far.
The first method uses a TF-IDF weighted bag-of-words representation extracted from the first comment-reply pair in the conversations.
The second method, called Awry, uses the first comment-reply pair of the conversation and extracts pragmatic features to use.
These pragmatic features include 38 different politeness strategies and 12 different prompt types.
The final model is called CRAFT (Conversational Recurrent Architecture for Forecasting) and is the current state of the art for this problem.
CRAFT consists of two parts, a pre-trained hierarchical recurrent encoder-decoder <cit.> and a prediction component that is just a multilayer perceptron with three fully-connected layers.
One major difference of the CRAFT model is that it works in an online fashion. This allows it to process comments in order and gain a context of what is happening in the conversation to signal it will de-reail.
Due to this factor the authors of CRAFT re-implemented the prior methods to also handle the comments in an online manner.

For our method we utilize our AITA classifier in both ways for this task. We first experiment with just using the second comment in each of the conversations. If our classifier predicts they are calling the original poster an asshole then we believe the conversation will de-rail. Otherwise, our prediction believes the conversation will proceed normally.
We also modified our model to stream comments in an online fashion as was done in the previous works.
Once our model makes a prediction that someone is an asshole we stop predicting and believe the conversation will end in a personal attack. Otherwise we force the model to continue down through all the comments and continue predicting.

We present our results in Table <ref>.
The results for our AITA classifier using just the second comment performs almost as well as the original Awry method on the Wikipedia talk pages. It does outperform a simple bag-of-words model. Part of the reason for it's performance is that it does not guess many of the conversations to de-rail. We can see a degradation in performance when we shift to the online model. In our online version our model ends up detecting too many of the posts as calling the other user an asshole to signal the derailment. This stands in contrast to all of the other methods that saw an increase in performance when shifting to an online version.
Overall our model does not seem to generalize to this other task very well. Our results do show that some of the language tendencies between calling someone an asshole and a conversation derailing may overlap.

5cWikipedia Talk Pages     5cReddit CMV

3-5 8-10
Model     A     P     R     FPR     F1     A     P     R     FPR     F1

BoW     56.5     55.6     65.5     52.4     60.1     52.1     51.8     61.3     57.0     56.1

Awry     58.9     59.2     57.6     39.8     58.4     54.4     55.0     48.3     39.5     51.4

Cumul. BoW     60.6     57.7     79.3     58.1     66.8     59.9     58.8     65.9     46.2     62.1

Sliding Awry     60.6     60.2     62.4     41.2     61.3     56.8     56.6     58.2     44.6     57.4

CRAFT     66.5     63.7     77.1     44.1     69.8     63.4     60.4     77.5     50.7     67.9

AITA     57.3     57.9     57.3     14.4     57.5     53.3     53.3     53.3     22.4     53.3

AITA Online     54.8     58.6     54.8     66.8     56.6     52.1     55.9     52.1     44.1     53.9

Comparison of our AITA classifier used on the conversations gone awry dataset. Evaluation is done using accuracy, precision, recall, false positive rate, and F1 score.

The dynamics of discussion have been studied in many ways. One common way of doing so is analyzing the sentiment of the text in question. To do so, the text must first be embedded as a low dimensional feature vector. Then, given the sentiment of the text, e.g., positive or negative, one can train a classifier over the feature vectors, using the sentiment as the labels. As such, we discuss relevant text embedding methods, as follows.

Many text embedding methods focus on obtaining feature vectors for single words at a time <cit.>. However, it is also important to study groups of words (sentences, paragraphs) <cit.>. While simply combining feature vectors of each individual word in a sentence is one way to obtain group-level embeddings, this is often lacking. Doc2Vec [20] introduces an approach to embed groups of words at a time. It does so using a similar approach to its predecessor, word2vec <cit.>. However, Doc2Vec is also able to take into account the word order, and better embeds semantically similar words to each other. Several approaches have been proposed since, including many deep learning-based models <cit.>. However, BERT [8] is the current state-of-the-art. It uses a multilayer bidirectional Transformer encoder to learn the representation of a sequence of text, making it applicable to both words and groups of words.

The main drawback of sentiment analysis in the context of discussions, however, is that they do not account for how a given piece of text is responding to another. Presumably, the words people use in reaction to a situation depend on the morality of that situation. One approach does consider the response dynamic, though it does so in a general sense and not specifically for morality, described below.

Guimarães et al. <cit.> are interested in understanding different types of discussions that occur in the subreddits  and . They propose four “conversational archetypes” that describe how a discussion progresses, and aim to test 10 hypotheses associated with them. For example, they hypothesize that “Harmonious” discussions, ones in which all the posts are not marked as “controversial” by Reddit standards, have high positive sentiment scores.

§ CONCLUSIONS

In this study, we show that it is possible to learn the language of moral judgements from text taken from /r/AmITheAsshole. We demonstrate that by extracting the labels and fine-tuning a BERT language model we can achieve good performance at predicting whether a user is rendering a positive or negative moral judgement. Using our trained classifier we then analyze a group of subreddits that are thematically similar to /r/AmITheAsshole for underlying trends. Our results showed that users prefer posts that have a positive moral valence rather than a negative moral valence. Another analysis revealed that a small portion of users are judged to have substantially more negative moral valence than others and they tend towards subreddits such as /r/confessions. We also show that these highly negative moral valence users fall into three different types based on their posting habits. Lastly, we demonstrate that age and gender have a minimal effect on whether a user is judged to be have positive or negative moral valence.

Although the Judge-BERT classifier enabled us to perform a variety of analysis it does pose some limitations. We are unable to verify if the classifier generalizes well to the other subreddits in our study. The test-subreddits do deviate from the types of moral analysis observed in the training data. Moral judgement is not the focus of /r/CasualConversation, for example.

In the future we hope to implement argument mining in order to gain a better understanding of the reasons for these judgements by extracting the underlying arguments given by users. Other works have done this by extracting rules of thumb through human annotation [11] but this limits the ability to perform a large scale analysis. Argument mining has shown success with extracting the persuasive arguments from subreddits like /r/changemyview [10] and would enable us to get a better understanding of moral judgements on social media. This would also allow us to aggregate the underlying themes from these judgements for further analysis.

§ ACKNOWLEDGEMENTS

We would like to thank Michael Yankoski and Meng Jiang for their help preparing this manuscript. This work is funded by the US Army Research Office (W911NF-17-1-0448) and the US Defense Advanced Research Projects Agency (DARPA W911NF-17-C-0094).

[1]
J. Baumgartner, S. Zannettou, B. Keegan, M. Squire, and J. Blackburn.
The pushshift reddit dataset.
ICWSM, Jan 2020.

[2]
K. Bebbington, C. MacLeod, T. M. Ellison, and N. Fay.
The sky is falling: evidence of a negativity bias in the social
transmission of information.
Evolution and Human Behavior, 38(1):92–101, 2017.

[3]
R. Boyd and P. J. Richerson.
Punishment allows the evolution of cooperation (or anything else) in
sizable groups.
Ethology and Sociobiology, 13(3):171–195, May 1992.

[4]
W. J. Brady, M. Crockett, and J. J. Van Bavel.
The mad model of moral contagion: The role of motivation, attention,
and design in the spread of moralized content online.
Perspectives on Psychological Science, 15(4):978–1010, 2020.

[5]
J. P. Chang and C. Danescu-Niculescu-Mizil.
Trouble on the horizon: Forecasting the derailment of online
conversations as they develop.
In Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on
Natural Language Processing (EMNLP-IJCNLP), pages 4745–4756, 2019.

[6]
O. Choy, A. Raine, P. H. Venables, and D. P. Farrington.
Explaining the gender gap in crime: The role of heart rate.
Criminology, 55(2):465–487, 2017.

[7]
M. J. Crockett.
Moral outrage in the digital age.
Nature human behaviour, 1(11):769–771, 2017.

[8]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.
Bert: Pre-training of deep bidirectional transformers for language
arXiv preprint arXiv:1810.04805, 2018.

[9]
P. S. Dodds, J. R. Minot, M. V. Arnold, T. Alshaabi, J. L. Adams, D. R.
Dewhurst, T. J. Gray, M. R. Frank, A. J. Reagan, and C. M. Danforth.
Allotaxonometry and rank-turbulence divergence: A universal
instrument for comparing complex systems.
arXiv preprint arXiv:2002.09770, 2020.

[10]
S. Dutta, D. Das, and T. Chakraborty.
Changing views: Persuasion modeling and argument extraction from
online discussions.
Information Processing & Management, 57(2):102085, 2020.

[11]
M. Forbes, J. D. Hwang, V. Shwartz, M. Sap, and Y. Choi.
Social chemistry 101: Learning to reason about social and moral
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 653–670, 2020.

[12]
E. Gilbert.
Widespread underprovision on reddit.
In CSCW, pages 803–808, 2013.

[13]
M. Glenski, C. Pennycuff, and T. Weninger.
Consumers and curators: Browsing and voting patterns on reddit.
IEEE Transactions on Computational Social Systems,
4(4):196–206, Dec 2017.

[14]
M. Glenski, G. Stoddard, P. Resnick, and T. Weninger.
Guessthekarma: a game to assess social rating systems.
CSCW, 2:1–15, 2018.

[15]
M. Glenski and T. Weninger.
Rating effects on social news posts and comments.
TIST, 8(6):1–19, 2017.

[16]
J. Graham, J. Haidt, S. Koleva, M. Motyl, R. Iyer, S. P. Wojcik, and P. H.
Chapter Two - Moral Foundations Theory: The Pragmatic Validity
of Moral Pluralism, volume 47, page 55–130.
Academic Press, Jan 2013.

[17]
S. Hechler and T. Kessler.
On the difference between moral outrage and empathic anger: Anger
about wrongful deeds or harmful consequences.
Journal of Experimental Social Psychology, 76:270–282, 2018.

[18]
J. J. Jordan and D. G. Rand.
Signaling when no one is watching: A reputation heuristics account of
outrage and punishment in one-shot anonymous interactions.
Journal of personality and social psychology, 118(1):57, 2020.

[19]
A. M. Kibriya, E. Frank, B. Pfahringer, and G. Holmes.
Multinomial naive bayes for text categorization revisited.
In Australasian Joint Conference on Artificial Intelligence,
pages 488–499. Springer, 2004.

[20]
Q. Le and T. Mikolov.
Distributed representations of sentences and documents.
In ICML, pages 1188–1196, 2014.

[21]
A. D. Ong and D. J. Weiss.
The impact of anonymity on responses to sensitive questions 1.
Journal of Applied Social Psychology, 30(8):1691–1708, 2000.

[22]
S. T. Piantadosi.
Zipf’s word frequency law in natural language: A critical review
and future directions.
21(5):1112–1130, 2018.

[23]
T. Reynolds, C. Howard, H. Sjåstad, L. Zhu, T. G. Okimoto, R. F.
Baumeister, K. Aquino, and J. Kim.
Man up and take it: Gender bias in moral typecasting.
Organizational Behavior and Human Decision Processes,
161:120–141, 2020.

[24]
K. Rost, L. Stahel, and B. S. Frey.
Digital social norm enforcement: Online firestorms in social media.
PLoS one, 11(6):e0155923, 2016.

[25]
E. Sagi and M. Dehghani.
Measuring moral rhetoric in text.
Social Science Computer Review, 32(2):132–144, Apr 2014.

[26]
T. Sawaoka and B. Monin.
The paradox of viral outrage.
Psychological science, 29(10):1665–1678, 2018.

[27]
C. Schein and K. Gray.
The theory of dyadic morality: Reinventing moral judgment by
redefining harm.
Personality and Social Psychology Review, 22(1):32–70, 2018.

[28]
K. Siddiqui.
Heuristics for sample size determination in multivariate statistical
World Applied Sciences Journal, 27(2):285–287, 2013.

[29]
S. Yardi and D. Boyd.
Dynamic debates: An analysis of group polarization over time on
Bulletin of Science, Technology & Society, 30(5):316–327,