A Hybrid Intelligence Method for Argument Mining

Michiel van der Meer<EMAIL_ADDRESS>
Leiden Institute for Advanced Computer Science (LIACS),

Leiden University
Enrico Liscio<EMAIL_ADDRESS>
Interactive Intelligence (II),

Delft University of Technology
Catholijn M. Jonker<EMAIL_ADDRESS>
Interactive Intelligence (II),

Delft University of Technology
Aske Plaat<EMAIL_ADDRESS>
Leiden Institute for Advanced Computer Science (LIACS),

Leiden University
Piek Vossen<EMAIL_ADDRESS>
Computational Linguistics & Text Mining Lab (CLTL),

Vrije Universiteit Amsterdam
Pradeep K. Murukannaiah<EMAIL_ADDRESS>
Interactive Intelligence (II),

Delft University of Technology

Large-scale survey tools enable the collection of citizen feedback in opinion corpora. Extracting the key arguments from a large and noisy set of opinions helps in understanding the opinions quickly and accurately. Fully automated methods can extract arguments but (1) require large labeled datasets that induce large annotation costs and (2) work well for known viewpoints, but not for novel points of view. We propose , a hybrid (human + AI) method for extracting arguments from opinionated texts, combining the speed of automated processing with the understanding and reasoning capabilities of humans. We evaluate on three citizen feedback corpora. We find that, on the one hand, achieves higher coverage and precision than a state-of-the-art automated method when compared to a common set of diverse opinions, justifying the need for human insight. On the other hand, requires less human effort and does not compromise quality compared to (fully manual) expert analysis, demonstrating the benefit of combining human and artificial intelligence.

§ INTRODUCTION

To make decisions on large public issues, such as combating a pandemic and transitioning to green energy, policymakers often turn to the citizens for feedback kythreotis2019citizen,lee2020policy. This feedback provides insights into public opinion and contains viewpoints from many individuals with different perspectives. Involving the public in the decision-making process helps in gaining their support when the decisions are to be implemented, fostering the legitimacy of the process ostrom1990governing.

In the face of crises, decisions must be made swiftly. Thus, collecting feedback, analyzing it, and making recommendations ought to be performed under tight time constraints. For example, when deciding on relaxing COVID-19 measures in the Netherlands, researchers had one month to design the experiment, collect public feedback, and make recommendations to the government mouter2021public.
The time constraint limits the amount of information researchers can analyze, potentially painting an incomplete picture of the opinions. In the scenario above, researchers processed data manually and they could only analyze less than 8% of the qualitative feedback provided by more than 25,000 participants.

Argument Mining (AM) lawrence2020argument methods can assist in increasing the efficiency of feedback analysis by, e.g., locating and interpreting argumentative feedback and classifying statements as supporting or opposing a decision. However, applying automated AM methods for feedback analysis poses three main challenges. First, AM methods generalize poorly across domains stab2018cross,jakobsen2021spurious. Thus, they require large amounts of domain-specific training data, which is often not available. The use of pretrained language models, with the pre- or fine-tuning paradigm, mitigates but does not solve the reliance on large domain-specific training datasets reimers2019classification,ein2020corpus.
Second, although AM methods can identify argumentative content, they do not compress the information. That is, they do not recognize whether two arguments describe the same point of view, leaving the policymakers with the significant manual labor of aggregating arguments.
Finally, naively relying on a small sample of labeled data might cause minority opinions to be ignored as they are not well represented klein2012enabling, creating a bias toward popular (repeated) arguments, which can perpetuate echo chambers and filter bubbles price1989social,schulz2000biased.

The key point analysis (KPA) task bar2020arguments seeks to automatically compress argumentative discourse into unique key points, which can be matched to arguments. However, synthesizing key points is a significant challenge. In the ArgKP dataset, domain experts (skilled debaters) were asked to generate key points. Subsequently, a model was trained to take over the task bar2020quantitative.
However, the reliance on few human expert annotators introduces biases of the human experts and may not be representative of the opinions of the larger population. This defeats the purpose of engaging the larger public in a bottom-up deliberative decision-making process.

We argue for a crowd-sourced human-machine approach for argument extraction, combining the scalability of automated methods and the human understanding of others' perspectives. We propose (Hybrid Extraction of Arguments), a hybrid (human + AI) method for extracting a diverse set of key arguments from a textual opinion corpus. breaks down the argument extraction task into argument annotation, consolidation, and selection phases. employs human (crowd) annotators and supports them via intelligent algorithms based on natural language processing (NLP) techniques for analyzing opinions provided by a large audience, as shown in Figure <ref>.

In a democratic cycle, citizens provide their opinions on options for governmental decision-making and their opinions need to be interpreted. Insights into the arguments embedded in their comments can be provided by Key Point Analysis (KPA). To perform KPA, most analysis is performed either manually or automatically. In our work, we propose , a hybrid method.

is evaluated on three corpora, each containing more than 10K public opinions on relaxing COVID-19 restrictions mouter2021public.  We compare with an automated approach bar2020quantitative performing the KPA task. In addition, we compare the key arguments generated by with manually obtained insights identified by experts mouter2021public. We find that outperforms the automated baseline in terms of precision and diversity, specifically when confronted with a set of varied perspectives. also yields better results than manual analysis, as fewer opinions needed to be analyzed in order to obtain a wider set of key arguments.

C1 We present a hybrid method for key argument extraction, which generates a diverse set of key arguments from a collection of opinionated user comments.

C2 We evaluate our method on real-world corpora of public feedback on policy options. Compared to an automated baseline, increases the precision of the key arguments produced and improves coverage over diverse opinions. Compared to the manual baseline, identifies a large portion of arguments identified by experts as well as new arguments that experts did not identify.

C3 We extensively discuss the implications of incorporating recent advances in NLP, such as Large Language Models (LLMs), into the workflow of our hybrid method.

In this paper, we extend the method vandermeer2022hyena to include argument selection. The original method outputs argument clusters, and leverages manual annotations from the first two phases to select arguments from argument clusters. In this extension, we introduce a method for selecting the most representative argument from each cluster. The need to summarize argument clusters is not specific to , as previous AM applications also retrieve clusters instead of singular arguments boltuvzic2015identifying,wachsmuth2018retrieval,daxenberger2020argumentext. We compare various techniques to accomplish this task, including generative large language models. Furthermore, we run additional experiments to demonstrate how the new argument selection step can be incorporated in the pipeline. Finally, we perform additional analyses to derive further insights from annotators in .

Organization Section <ref> provides background on Argument Mining for public opinions, and Section <ref> introduces the method for extracting arguments. We outline the experimental setup in Section <ref> and provide extensive results in Section <ref>. A discussion of our work is given in Section <ref> and we conclude with Section <ref>.

§ RELATED WORK

We describe related work on Argument Mining, methods for summarizing arguments, and their application to opinion analysis.

§.§ Computational Argument Analysis

Argument Mining (AM) methods cabrio2018five,lawrence2020argument focus on the recognition, extraction, and computational analysis of arguments presented in natural language. They seek to discover arguments brought forward by speakers and identify connections between them. Typically, AM techniques concern themselves with finding the structure of arguments eemeren1987handbook, with the goal of finding premises for supporting or refuting conclusions.

AM is a challenging problem. The ability to recognize and extract arguments from text (for humans and machines, alike) is dependent on the argumentativeness of the underlying data. Often, significant effort is required by human annotators for reaching moderate inter-rater agreement when annotating arguments teruel2018increasing. Given argumentative texts, modern NLP models are reasonably good at recognizing argumentative discourse within specific contexts niculae2017argument,eger2017neural,reimers2019classification.

Typically, the first step of AM is to identify the elemental components of arguments (e.g., claims and premises) in text palau2009argumentation. The combination of such components forms a structured argument. However, there is currently no consensus on the exact linguistic notion of such elemental components, with multiple levels of granularity being proposed daxenberger2017essence,walton2008argumentation,freeman2011argument,bentahar2010taxonomy. Nonetheless, a few characteristics have been recognized as important for recognizing arguments, namely that arguments

* contain (informal) logical reasoning stab2014annotating,

* address a why question biran2011identifying, and

* have a non-neutral stance towards the issue being discussed stab2014annotating.

is a novel AM method that combines human annotators and automated NLP models. By splitting up the argument extraction task into distinct phases, we take advantage of the diverse human perspectives, while addressing scalability through automation.

§.§ Summarization of Arguments

Automated methods have been proposed to create a core set of key points from a large corpus of individual comments bar2020quantitative. In this paradigm, comments are filtered by a manually tuned selection heuristic, resulting in a list of key point candidates. The candidates are matched against all comments, based on a classifier trained for the argument–key point matching task bar2020arguments.  We evaluate the performance of this approach on a novel domain on COVID-19 measures and compare it against .

Additionally, there exists an extended body of work on Natural Language Inference (NLI) and Semantic Textual Similarity (STS). In these works, models are trained to indicate semantic similarity or logical entailment between two sentences conneau2017supervised,reimers2019sentence. They have made a significant impact across a range of tasks xu2018deep,zhong2020extractive. However, downstream applications often need additional fine-tuning howard2018universal in order to perform a task well. They also capture generic aspects of semantic similarity and entailment, which may not be applicable to arguments reimers2019sentence, or overfit to spurious patterns in the data mccoy2019right. Thus, such methods require significant adaptation to effectively compress information in particular domains. Recently, Large Language Models (LLMs) have been shown to perform well on inference tasks with out-of-distribution data wang2023robustness. However, we argue that a plurality of (human) perspectives is necessary to perform sensitive tasks such as the summarization of arguments, which may in turn be used to inform policy-makers about the sentiment of a population Talat2022OnLanguage. Yet, LLMs might be adequate for specific subtasks, as we showcase in the third phase of the method.

§ METHOD

is a hybrid method since it combines automated techniques and human judgment akata2020research. guides human annotators in synthesizing key arguments (i.e., semantically distinct arguments that describe relevant aspects of the topic under discussion) from an opinion corpus composed of individual opinions (textual comments) on a topic.

consists of three phases (Figure <ref>). In the first phase (Key Argument Annotation), an intelligent sampling algorithm guides human annotators individually through an opinion corpus to extract high-level information from the opinions.
In the second and third phases, aims to reduce the subjectivity in the first phase annotations by combining and rewriting arguments that were individually annotated.
In the second phase (Key Argument Consolidation), an intelligent merging strategy supports a new group of annotators in merging the results from the first phase into clusters of arguments, combining manual and automatic labeling. In the third phase (Key Argument Selection), employs an automated method to synthesize a single argument that represents the arguments belonging to the same merged argument cluster. The final output of is a list of key arguments grounded on the opinions in the corpus.

Overview of the method.

§.§ Opinion Corpora

Our opinion corpora are composed of citizens' feedback on COVID-19 relaxation measures, a contemporary topic.
The feedback was gathered in April and May 2020 using the Participatory Value Evaluation (PVE) method mouter2021public. In a PVE, participants are offered a set of policy options and asked to select their preferred portfolio of choices. Then, the participants are asked to motivate why they picked certain options (pro stance) and not pick the other options (con stance) via textual comments. Pro- and con-opinions together form the opinion corpus.
We analyze feedback from 26,293 Dutch citizens on three of these options, treating comments on each option as an opinion corpus. Table <ref> shows examples of opinions provided for each different policy option. In our experiments, the method is applied to one corpus at a time. Since we use data from a publicly run citizen feedback experiment, we observe that some options attracted more pro comments than others. We picked these three options with different pro/con ratios to investigate their impact on the key argument extraction task. The opinions in these corpora are similar to noisy user-generated web comments habernal2017argumentation, may span multiple sentences, and contain more than one argument at a time. For each policy option, we use the keyword in uppercase as the option identifier in the remainder of the paper.

Policy option (Corpus)     Example opinion     # Opinions     Pro/Con Ratio

people may come together in small groups     Then they can go back to school (Pro)     13400     0.66/0.34

All restrictions are lifted for persons who are     Encourages inequality (Con)     10567    0.17/0.83

hospitality and entertainment industry     The economic damage is too high (Pro)     12814     0.55/0.45

Example opinions in the COVID-19 corpora.

The original opinions were provided in Dutch. To accommodate a diverse set of annotators in our experiments, we translated all comments to English using the Microsoft Azure Translation service.
All experiments are performed with the translated opinions. Mixing (pretrained) embeddings and machine-translated comments has a minimal impact on downstream task performance sennrich2016improving,eger2018cross,daza2020x. Although all experiments are conducted in English, the link to the original Dutch text is preserved for future applications.

§.§ Key Argument Annotation

In the first phase of , human annotators extract individual key argument lists by analyzing the opinion corpus. Since a realistic corpus consists of thousands of opinions, it is unfeasible for an annotator to read all opinions. Thus, proposes a fixed number of opinions to each annotator. employs NLP and a sampling technique to select diverse opinions to present to an annotator.

Intelligent Opinion Sampling
Each annotator is presented, one at a time, with a fixed number of opinions. To sample the next opinion, we embed all opinions and arguments observed thus far using the S-BERT model ($M_S$) reimers2019sentence. S-BERT converts sentences into fixed-length embeddings, which can be used to compute semantic similarities between pairs of sentences.

Then, we select a pool of candidate opinions using the Farthest-First Traversal (FFT) algorithm Basu2004. FFT selects the candidate pool as the $f$ farthest opinions in the embedding space from the previously read opinions and annotated arguments (in our experiments, we empirically select $f=5$). Next, we use an argument quality classifier trained on the ArgQ dataset gretz2020large to select the opinion most clear and related to the policy option. In this way, we aim at increasing both the diversity and quality of the opinions presented to each annotator.

Upon reading an opinion, the annotator is asked, first, to identify whether the opinion contains an argument or not. If so, the annotator is asked to check whether the argument is already included in their current list of key arguments. If it is not, the annotator should extract the argument into a standalone expression (i.e., into a key argument), and add it to the list of key arguments. When adding a new argument, the annotator is asked to indicate the stance of the opinion (i.e., whether it is in support or against the related policy option). To facilitate this task, highlights the most probable stance for the user as a label suggestion schulz2019analysis,beck2021investigating.

Topic Assignment
We use a BERTopic grootendorst2020bertopic model $\mathcal{T}$ to extract clusters of topics from the corpus.
We train $\mathcal{T}$ on all opinions in the corpus and select the most frequent topics found by $\mathcal{T}$, with duplicates and unintelligible topics manually removed by two experts. We ask human annotators to associate the topics from the generated shortlist to each argument. This topic assignment $T$ is used in the second phase to compute argument similarity. Thus, in the first phase, yields multiple key argument lists (one per annotator), each containing key arguments and their stances, and an assignment of pre-selected topics to key arguments.

§.§ Key Argument Consolidation

In the first phase,

* the annotators are exposed to a small subset of the opinions in the corpus, and

* the interpretation of arguments is subjective.

In the second phase, seeks to consolidate the key argument lists generated in the first phase.
Our goal is to increase the diversity of the resulting arguments and compensate for individual biases.

First, we create the union of all lists of key arguments generated in the first phase of .
Then, we ask the annotators to evaluate the similarity of the key argument pairs in the union list. Based on the similarity labels, we employ a clustering algorithm to group similar key arguments, producing a consolidated list of key arguments.

Pairwise Annotation
To simplify the consolidation task, the annotators are presented with one pair of key arguments at a time and asked whether the concepts described by the key arguments in the pair are semantically similar.
To reduce human effort, we select only the most informative key argument pairs for manual annotation and automatically annotate the remaining pairs. To select the most informative pairs, we adopt a Partial-Ordering approach, Power chai2016cost, as described below.

Let $p_{ij}$ be a pair of key arguments $\langle a_i, a_j\rangle$. The similarity between the two key arguments in the pair is described by two similarity scores, $s^1_{ij}$ and $s^2_{ij}$. By using multiple scores, we seek to make the similarity computation robust.
For each $p_{ij}$, we compute the two similarity scores described in Table <ref>. We use cosine similarity for $s^1_{ij}$ since the angular distance describes the semantic textual similarity between two arguments. In contrast, we use Euclidean distance for $s^2_{ij}$ since the absolute values of the topic assignment are relevant.

Measure     Description

$s^1_{ij} = \frac{\mathbf{i} \cdot \mathbf{j}}{\Vert\mathbf{i}\Vert \Vert\mathbf{j}\Vert}$     Cosine similarity between embeddings $\mathbf{i} = M_S(a_i)$ and $\mathbf{j} = M_S(a_j)$

$s^2_{ij} = \frac{1}{d(T(a_i),T(a_j))}$     Inverse of the Euclidean distance $d$ between manual topic assignments $T$ of $a_i$ and $a_j$

The similarity scores between key argument pairs used to create the pairwise dependency graph.

Given the similarity scores, we construct a dependency graph $G$ (as in the top-left part of Figure <ref>), where each key argument pair is a node in $G$ and the edges indicate a Pareto dependency ($\succ$) between two pairs as follows:

\begin{align}
\label{eq:pareto0}
p_{ij} & \succeq p_{i'j'} & \text{if}  & \quad \forall n \quad s^n_{ij} \geq s^n_{i'j'}\\
\label{eq:pareto1}
p_{ij} & \succ p_{i'j'}   & \text{if}  & \quad p_{ij} \succeq p_{i'j'} \quad\text{and} \quad \exists n\quad s^n_{ij} > s^n_{i'j'}
\end{align}

Next, we follow Power to extract disjoint paths from $G$. The highlighted path in the bottom-left part of Figure <ref> is an example disjoint path.
For every path, we perform a pairwise annotation as in the right part of Figure <ref>. We select the vertex at the middle of the unlabeled portion of the path and ask multiple (7) humans to indicate whether the concepts described by the two arguments in the pair are similar on a binary scale, and select the label with the majority vote. Given the annotation, we can automatically label

* all following pairs in the path as similar (green) in case the vertex is labeled as similar or

* all preceding pairs in the path as non-similar (red) in case the vertex is labeled as non-similar.

In essence, using the Pareto dependency, we search for threshold similarity scores for each path, above which all pairs are considered similar, and below which all pairs are non-similar. Because this is a local threshold, we prevent over-generalization. To annotate the complete graph efficiently, we employ the parallel Multi-Path annotation algorithm chai2016cost.

[mode=tex, width=]tikz/pairwise
Pairwise annotation of the dependency graph, combining human and automatic judgments.

Given a similarity label for each key argument pair, our goal is to identify groups of similar key arguments. However, the similarity among key arguments may not be transitive—given $\langle a_1, a_2\rangle$ as similar and $\langle a_2, a_3\rangle$ as similar, $\langle a_1, a_3\rangle$ may be labeled as dissimilar. This can happen because

* the interpretation of similarity can be subjective (for manually labeled pairs), and

* the automatic approach is not always accurate (for automatically labeled pairs).

Thus, we employ a clustering algorithm for identifying a consolidated list. First, we construct a similarity graph, where each key argument is a node and there is an edge between two arguments if they are labeled as similar. Then, we employ out-of-the-box graph clustering algorithms for constructing argument clusters. These clusters form the key argument lists.

§.§ Key Argument Selection

In the third step of , we extract a single argument from each cluster, obtaining the final list of key arguments for the opinion corpus. Formally, for every cluster $k \in K$, we create an argument $a_k$ that is representative of that cluster. Argument selection methods can be extractive (select an argument from the cluster) or abstractive (generate a new argument that summarizes the cluster). Since there are many methods available for selecting arguments, we can experiment with multiple, and pick the best-performing method. In that case, we again pick an intermediate evaluation metric, which we use to select the best selection method. While there is no human annotation involved in this step, we still consider this higher-level algorithmic design a hybrid process, and thus a collaboration between humans and AI. For the task of selecting relevant arguments, we compare the following four types of approaches.

Centroids For every cluster $k$, we compute a sentence embedding of every argument $a_k$ using $M_{S}$. Then, we compute pairwise distances between all arguments inside the same cluster. We select the argument with the lowest average distance, measured using cosine similarity, to all other arguments.

Argument Quality We use a model that measures argument quality to select the argument from each cluster with the highest quality. We use the same argument classifier as in the Key Argument Annotation phase, trained on the ArgQ dataset gretz2020large.

Prompting We prompt an LLM to synthesize a single argument out of the arguments provided in the argument cluster brown2020language. We experiment with an open-source and a closed-source model.

Random As a baseline, we randomly select an argument from the cluster to represent the entire argument cluster.

§ EXPERIMENTAL SETUP

We involve 378 Prolific (<www.prolific.co>) crowd workers as annotators to evaluate . We required the workers to be fluent in English, have an approval rate above 95%, and have completed at least 100 submissions. Our experiment was approved by an Ethics Committee and we received informed consent from each subject. We provide supplemental material, containing instructions provided to the annotators, experiment protocol, experiment data, analysis code, and additional details on the experiment van2022hyenasuppl.

Table <ref> shows an overview of the tasks in the experiment. First, we ask annotators to perform the method to generate key argument lists for three corpora. Then, we compare the quality of the obtained lists with lists generated for the same corpora via two baselines. All tasks except topic generation were performed by the crowd workers, with most of the task instances annotated by multiple annotators to investigate the agreement between annotators.

Task     Option     # Items     # Annotators     Overlap

3*Key argument annotation            255           5     3*1

255           5

255           5

Topic generation                                                all              45            2     2

3*Topic assignment                   91            10     3*5

66            5

69            5

3*Key argument consolidation         1538        99     3*3

824         57

940         87

3*Key argument evaluation        248         42     3*7

193         29

221         29

Overview of the tasks in the experiment. Items to be annotated can be opinions , arguments , topics , or combinations.

§.§ Phase 1: Key Argument Annotation

In the first phase of , each annotator extracts a key arguments list from an opinion corpus. In each corpus, five annotators annotated 51 opinions each, for a total of 255 opinions per corpus. Of the 51 opinions, the first is selected randomly, and the following 50 are selected by FFT. This number of opinions was empirically selected to make the annotation feasible within a maximum of one hour. We instantiate the S-BERT model $M_S$ using the Huggingface Model Hub[<https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2>].

We train a BERTopic model on each opinion corpus, generating 59, 56, and 72 topics for the , , and corpora, respectively. Since the number of resulting topics is too high for the manual assignment of arguments to topics, we curate a short list of topics per corpus. We select the 15 most frequent topics in a corpus and ask two experts, the first two authors, to remove duplicates (i.e., topics covering the same semantic aspect) and rate the clarity (i.e., how well the topic describes a relevant aspect of the discussion in the corpus) of each topic. Unique topics with an average clarity score above 2.5 compose the shortlist of topics. Then, we ask crowd annotators to assign topics to each key argument generated in the first phase of .

§.§ Phase 2: Key Argument Consolidation

In the second phase of , we obtain similarity labels $y(a_i,a_j)$ (1 if similar, 0 if not) for all key argument pairs $\langle a_i, a_j\rangle$—some pairs are labeled by the annotators and others are automatically labeled. Given the similarity labels, we construct an argument similarity graph and cluster the graph to identify a consolidated list of key arguments.

We experiment with two well-known graph clustering algorithms:

* Louvain clustering blondel2008fast uses network modularity to identify groups of vertices based on a resolution parameter $r$.

* Self-tuning spectral clustering zelnik2004self uses dimensionality reduction in combination with $k$-means to obtain clusters, where $k$ is the desired number of clusters.
We select the parameters of these algorithms to minimize the error metric $E$ shown in Eq. <ref>. The metric penalizes clusters having dissimilar argument pairs. That is, for a cluster $k \in K$ and $\forall a_i, a_j \in k$, if $y(a_i, a_j) = 1$, the error for that cluster is 0. If a cluster contains only a single element, we manually set the error for that cluster to 1, to discourage creating single-member clusters.

\begin{gather}
\label{eq:cluster-error}
E = \frac{1}{|K|}\mathlarger{\sum}_{k \in K} \frac{\sum\limits_{a_i,a_j \in k} \mathbbm{1}_{y(a_i,a_j) = 0} }{\binom{|k|}{2}}
\end{gather}

§.§ Phase 3: Key Argument Selection

In the third phase, we use a mechanism for selecting single arguments per argument cluster. We experiment with multiple methods and different models for selecting arguments. An overview of the methods used is given in Table <ref>. Below, we explain the setup for each method, and how we select the best-performing method to be used in the final output for .

Method     Model     Type     Open     Size

Random     –     extractive      –     –

Centroid     S-BERT     extractive      yes     22M

Prompting     ChatGPT     abstractive     no    175B

Llama     abstractive     yes     7B

Quality     ArgQ     extractive     no     125M

Argument selection algorithms.

Prompts We construct different prompts for the two models to extract the desired argument selection output. ChatGPT is an instruction-tuned model and can be prompted to answer questions or follow instructions ouyang2022training. Llama lacks instruction-tuning, and thus requires prompts designed for next-token generation touvron2023llama.

Consider the context of the COVID-19 pandemic and the following arguments:

- Argument 1

- Argument $k$

Write a key argument that summarizes the above arguments, and make it short and concise.

Consider the context of the COVID-19 pandemic and the following arguments:

- Argument 1

- Argument $k$

A short and concise key argument that summarizes the above arguments is:

Testing Cluster Coherence First, we investigate the coherence of the clusters generated in Phase 2 according to each argument selection method, with the intent of measuring how each (automated) method aligns with the results of the first two phases of the (hybrid) process. In cases of low coherence, semantically different arguments may end up together. Vice versa, in highly coherent clusters, only arguments that are the same are actually put together. While the error metric $E$ (Equation <ref>) gives an error rate, it is mostly a comparative method, designed to select the best clustering method. Whether or not the clusters make sense to a human interpreter remains unclear. As such, we devise a so-called odd-one-out task, in which we use the Argument Selection methods for selecting arguments from a triple of arguments. In this triple, two arguments stem from the same cluster, and the third from a different clsuter. The task for each argument selection method is to select which is the deviating argument. Here, we expect an adequate method to succeed well beyond random performance. Because Argument Quality is not intended for pairwise comparisons of arguments, we omit it in the odd-one-out task. We evaluate the remaining methods on a sample of 1K triples uniformly chosen from all possible triple combinations.

Evaluating Argument Selection We use different methods and different models for experimenting with the argument selection phase. As before, we employ an error metric to select the best-performing method, which we then inspect through a human evaluation. We use BERT score zhang2020BERTScore, a metric designed for model selection that uses a trained BERT model to compare the semantic similarity between the selected argument and the original opinions. Specifically, BERT score recall correlates well with human consistency judgments, the factual alignment between selected argument and references (original opinions) fabbri2021summeval. We pick the best-performing method for argument selection based on this metric. This way, we penalize any possible effect of hallucinations of LLMs on the method. We take the argument selected by each approach in the Key Argument Selection phase of the procedure. As references, we take all comments that were involved in the creation of the cluster. We compute BERTscore and compare it across our approaches.

§.§ Baselines

We compare the output of to the results of an automated and a manual approach to key argument extraction.

§.§.§ Comparison to Automated Baseline

We use the  argument matching model bar2020quantitative to automatically extract key points from the corpus. selects candidate key points from opinions using a manually-tuned heuristic, which filters opinions on their length, form, and predicted argument quality gretz2020large.
The original approach suggests relaxing heuristic parameters such that 20% of the opinions are selected as candidates. However, this caused  overly specific arguments as candidates. Instead, we departed from the parameters used for the ArgKP dataset bar2020quantitative, and only relax them slightly such that $\sim$10% of opinions are selected as key point arguments.

Candidate key points and opinions are assigned a match score using a model trained for matching arguments based on RoBERTa liu2019roberta. Opinions only match the highest-scoring candidate key points if their match score exceeds a threshold $\theta$, corresponding to the best match and threshold (BM+TH) approach. After deduplication, this results in a single list of key arguments per option. We use three metrics, coverage ($C$), precision ($P$), and diversity ($D$) to compare and .

Coverage ($C$) is defined as the fraction of opinions mapped to an argument out of all the processed opinions bar2020quantitative. To compute $C$, first, we extract the set of key arguments $\mathcal{A}_H$ from based on opinions $O_H^{obs}$ ($\subset O$) observed by the annotators. Further, if an argument is extracted from an observed opinion $o_i \in O_H^{obs}$, we add $o_i$ to the set of annotated opinions $O^{ann}_H$. Similarly, we extract the set of key arguments $\mathcal{A}_A$ from based on its observed set of opinions $O_A^{obs} (\equiv O)$, producing a set of annotated opinions $O_A^{ann}$. Then, the coverage with respect to all observed opinions is:

\begin{equation}
\label{eq:coverage-hyena-all}
C_H = \frac{|O^{ann}_H|}{|O^{obs}_{H}|}
\end{equation}

\begin{equation}
\label{eq:coverage-argkp-all}
C_A = \frac{|O^{ann}_A|}{|O^{obs}_{A}|}
\end{equation}

Comparing the coverage scores as defined above naively may not be fair since the set of observed opinions (i.e., the denominators of Equations <ref> and <ref>) are not the same for and . Thus, we also compute coverage with respect to a set of common opinions, $O^{obs}_{H} \cap O^{obs}_{A}$, observed by both methods, as:

\begin{equation}
\label{eq:coverage-hyena-common}
C_H^{common} = \frac{|O^{ann}_H \cap O^{obs}_{A}|}{|O^{obs}_{H} \cap O^{obs}_{A}|}
\end{equation}

\begin{equation}
\label{eq:coverage-argkp-common}
C_A^{common} = \frac{|O^{ann}_A \cap O^{obs}_{H}|}{|O^{obs}_{H} \cap O^{obs}_{A}|}
\end{equation}

We add the same term to both denominator and numerator in Equations <ref> and <ref> so that the coverage stays in the range [0, 1]. Note that $C_H^{common} = C_H$ since $O_H^{obs}, O_H^{ann} \subset O_A^{obs} (\equiv O)$.

Precision ($P$) is the fraction of mapped opinions for which the mapping is correct bar2020quantitative.
Thus, we must map a set of opinions to arguments in order to compute precision. For this mapping, we select the common opinions, $O^{ann}_{H} \cap O^{ann}_{A}$, that are annotated in both and . Then for each $o_i \in O^{ann}_{H} \cap O^{ann}_{A}$, we create two pairs $\langle o_i, \mathcal{A}_H(o_i)\rangle$ and $\langle o_i, \mathcal{A}_A(o_i)\rangle$, where $\mathcal{A}_H(o_i)$ and $\mathcal{A}_A(o_i)$ are the arguments associated with $o_i$ by and , respectively. Then, we ask annotators to label $z(o_i, a_i)=1$ for all matching pairs and $z(o_i, a_i)=0$ for all non-matching pairs, and keep the majority consensus from multiple annotators. Given the opinion-argument mapping, we compute precision as:

\begin{equation}
\centering
P_H^{common} = \frac{\sum\limits_{o_i \in O_H^{ann} \cap O_A^{ann}}z(o_i, \mathcal{A}_H(o_i))}{|O_H^{ann} \cap O_A^{ann}|} \label{eq:precision-same-hyena}
\end{equation}

\begin{equation}
\centering
P_A^{common} = \frac{\sum\limits_{o_i \in O_H^{ann} \cap O_A^{ann}}z(o_i, \mathcal{A}_A(o_i))}{|O_H^{ann} \cap O_A^{ann}|} \label{eq:precision-same-argkp}
\end{equation}

Diversity ($D$) is defined as the ratio of key arguments and the number of comments seen by the method. We use diversity to signify how well our method is able to preserve the perspectives present in the opinions seen by the method. In order to compare across methods, we take (1) only correct mappings ($z(o_i, a_i)=1$) using the labels from $P$ and (2) take the opinions seen by both $A$ and $H$. We define diversity as follows:

\begin{equation}
\centering
D_H = \frac{\mathcal{A}_H}{|O^{obs}_{H} \cap O^{obs}_{A}|}
\end{equation}

\begin{equation}
\centering
D_A = \frac{\mathcal{A}_A}{|O^{obs}_{H} \cap O^{obs}_{A}|}
\end{equation}

§.§.§ Comparison to Manual Baseline

A manual analysis involving six experts analyzed the feedback from a sample of participants (2,237 out of 26,293) over all policy options and identify key arguments mouter2021public. However, they do not report the exact number of opinions analyzed. Since there are 36,781 opinions for the three options we analyze (Table <ref>), we estimate the number of opinions the six experts would have analyzed to be 3,129 across the three options. In contrast, annotators analyze 765 intelligently selected opinions across the three options.

It is evident that reduces the number of opinions analyzed. Further, we investigate the extent to which the key argument lists generated by and the manual baseline have comparable insights. To do so, we report the number of key arguments that are overlapping, missing, and new compared to the expert-identified key arguments. We cannot compute precision and coverage for the manual baseline because it does not include a mapping between key arguments and opinions.

§ RESULTS

First, we analyze the inter-rater reliability of annotations. Then, we analyze the intermediate results of the three phases of . Finally, we compare our hybrid approach with the automated and manual baselines.

§.§ Annotator Agreement

Table <ref> shows the inter-rater reliability (IRR) for four steps with overlapping human annotations. We don't obtain IRR ratings for the argument extraction task in Phase 1 since the annotation is designed to be disjoint, and raters had little to no overlap in their extractions. In the topic generation phase (Section 4.1), we use the intraclass correlation coefficient ICC$(3,k)$ shrout1979intraclass since it involves ordinal ratings. In the other three tasks, multiple binary labels are obtained for the same subjects. In these tasks, we use prevalence- and bias-adjusted $\kappa$ (PABAK) sim2005kappa, which adjusts Fleiss' $\kappa$ for prevalence and bias resulting from small or skewed distribution of ratings.

Task     ICC3k     PABAK

Topic generation           0.66 (0.14)     –

Topic assignment           –     0.81 (0.10)

Key argument consolidation     –     0.34 (0.03)

Key argument evaluation      –     0.36 (0.04)

IRR scores per task in . We show the average (and standard deviation) over the three option corpora.

The lowest reliability score we obtain for the last two annotation tasks, Key argument consolidation and Key argument evaluation task. The obtained scores may be due to the difficulty of the task—for instance, lay annotators are asked to characterize the similarity between two arguments, and they may not stick to the provided definition of argument similarity. However, task difficulty may not be the only factor at play here. Argument comparisons are made with limited context, and the personal perspective or background of the annotator may influence their judgment. Thus, the low IRR scores may indicate a combination of task difficulty and the relatively subjective nature of the task aroyo2015truth. Similar reasoning holds for the task of evaluating arguments with respect to the original opinions.

Prolific annotators were generally young (M=29.2, SD=7.8) and typically active users with a median of over 300 tasks completed (M=404, SD=418). A little over half of our annotators were male (58.8%), another 38.6% reported as female, and the rest had no data available. 76.7% reported a language other than English as their native language (we did require all annotators to be fluent in English). Annotators mostly resided in European countries, with the UK, Mexico, and the US being the only non-EU countries with more than 10 annotators. 23.8% reported as being a full-time student, with the rest either reporting as not being a student or having no data available. Further work is required in order to investigate the impact of demographic factors on the subjective interpretation of the opinions and arguments involved shortall2022reason.

§.§ Phase 1: Key Argument Annotation

In Phase 1, individual annotators were guided through 51 opinions each and asked to annotate the observed arguments.
Table <ref> shows the number of different operations annotators perform over the 51 opinions. On average, the annotators identified 15 unique key arguments per option. About half of the opinions were skipped, mainly because the opinion lacked a clear argument.
Since the opinions had been automatically translated, we also provided annotators with the option to skip an opinion due to an unclear translation. Out of 51 actions, annotators reported mistranslations in 6, 7, and 2 opinions on average for , , and , respectively.

3cPhase 1     2cPhase 2

Option     # Args    # Skip     # Already     $\Delta$     $\tau$

18.0 (5.5)     23.4 (5.4)      11.4 (9.0)     -61.6%     0.34

12.8 (2.6)     31.4 (4.5)      8.6 (4.4)      -59.1%     0.42

13.8 (7.6)     29.2 (11.5)     10.2 (7.6)      -59.8%     0.41

The average annotation operations (and their standard deviation) in Phase 1, and obtained statistics for Phase 2.

This is a positive result since the noise (i.e., irrelevant or non-argumentative opinions) in public feedback can be much higher. Thus, the argument quality classifier we incorporate for opinion sampling is effective in filtering noise. Further, the annotators marked only about 15% of the encountered opinions as already annotated key arguments, which shows that the FFT approach is effective in sampling a diverse set of opinions for annotation.

To further examine the behavior of the annotators, we measured the amount of text that was literally copied from the opinions. In Figure <ref>, we show the distribution of overlap ratios across all extracted arguments. While some arguments do get copied verbatim (overlap ratio of 1), across all three corpora annotators generally rephrase the arguments themselves. This shows that, in , human intervention acts in shaping the arguments extracted from the opinions, rather than simply copying part of an opinion (as automated methods would do). Table <ref> shows some examples of arguments extracted with different overlap ratios.

Distribution of argument overlap ratio for arguments generated by Key Argument Annotation in Phase 1.

Option     1cOpinion Text     1cExtracted Argument     [c]Overlap

Our daughter misses her friends so much and I notice that she really needs it     Positive for the psychological health of children     0.060

Keep one system, keep it simple. Not too many deviations.     Everyone should be subject to the same set of rules/restrictions.      0.091

Too little research has been done to limit the measures for people who are immune and too few opportunities to test it. In addition, it is difficult to control.     It is difficult to control.     1.000

These measures are quite easy to take compared to the unselected measures.     Measures are easy to take compared to the unselected measures     0.820

Examples of copied and paraphrased arguments in Phase 1 of .

The topic models for each option generated a large variety of topics. After the generation of the topic models $\mathcal{T}$, we retain only the top-15 most frequent topics to make the annotation feasible. Our experts eliminated one, two, and zero topics as duplicates in the three options (Table <ref>). On average, the coherence scores—ranging from 1 (low) to 5 (high)—are high. This suggests that these topics were suitable for assignment to the arguments stemming from the crowd-extracted arguments. Table <ref> shows the final list of topics, with low-scoring topics removed.

Option     $|\mathcal{T}|$      [b]Number of
duplicates     Kept     [b]Average

59     1     12     4.4

56     2     12     4.4

72     0     14     4.0

Expert topic generation statistics in Phase 1.

Option     Topics

0.27immune entertainment hospitality restrictions 0.27infection immunity risk infected 0.27virus susceptible spread transmit 0.27schools reopen education students 0.27risk limited low dangerous 0.27group risk target least 0.27enforcement rules enforce measures 0.27pressure care effect increase 0.27distance keep difficult keeping 0.27exercise sports play exercising 0.27easing relaxation options effects 0.27mental development life psychological

0.27homes nursing care vulnerable 0.27netherlands country provinces dutch 0.27risk contamination danger dangerous 0.27work companies home economy 0.27entertainment hospitality catering industry 0.27infect sick cant anymore 0.27virus spread transmit young 0.27testing test immunity antibodies 0.27pressure care healthcare increase 0.27young distance age 15 0.27exceptions rules enforcement same 0.27easing options relaxations step

0.27homes nursing care vulnerable 0.27netherlands friesland groningen dutch 0.27risk hospitality entertainment dangerous 0.27mental health income decrease 0.27measures rules these should 0.27entrepreneurs work working industry 0.27care healthcare psychological decrease 0.27life economy social entertainment 0.27relaxation easing options relax 0.27corona immune had immunity 0.27bankrupt companies bankruptcies hospitality 0.27infections increase risk outbreak 0.27economy needs soon now 0.27meters distance possible industry

Top 4 words per topic generated in Phase 1.

§.§ Phase 2: Key Argument Consolidation

In Phase 2, uses the Power algorithm to guide human annotations on arguments similarity, with the intent of creating clusters of similar arguments across all arguments individually annotated in Phase 1.
Table <ref> (right side) shows the benefit of the Power algorithm—the number of pairs requiring human annotation ($\Delta$) was on average reduced by $60\%$.
The transitivity scores $\tau$ newman2002random indicate the extent to which transitivity holds among the similarity labels of argument pairs. The relatively low $\tau$ scores justify the subsequent clustering we perform.

Figure <ref> compares Louvain and spectral clustering for extracting argument clusters. Generally, both methods show a clear minimum for obtaining the final argument clusters.
Louvain clustering yields the smallest error for the and corpora, and spectral clustering for corpus. These methods create 20, 14, and 18 clusters respectively. We pick these clusters as input to the argument selection phase.

Error rate $E$ for different parameters per clustering method for each subcorpus in Phase 2.

Not all arguments inside the same cluster are constrained to have the same stance (pro or con) towards the policy option. We count what proportion of arguments in the cluster do not adhere to the majority stance. The distribution of stances scores is visualized in  Figure <ref>. While we see that the upper limit is that half the arguments in each cluster are not agreeing with the majority label, the average ratio denotes that only a small fraction of argument stances do not agree with the majority stance label. This shows that the clusters generally represent a coherent distribution of arguments with similar stances to each policy option. The ratio on average is lowest for , which is the option with the highest ratio of con opinions.

Stance distribution for clusters extracted for each corpus in Phase 2. A ratio of 0.5 denotes an equal number of pro and con arguments inside a cluster.

§.§ Phase 3: Key Argument Selection

In Phase 3, we compare five Argument Selection methods for extracting a representative argument for each of the clusters obtained in Phase 2. We first perform an odd-one-out task to evaluate the coherence of the clusters according to each tested Argument Selection method (see Section <ref> for additional details). Then, we evaluate the quality of the arguments that are selected to represent clusters.

Odd-one-out task
Figure <ref> shows the results of the odd-one-out evaluation. Here, we see that two methods outperform the random baseline. This indicates that these methods identify cluster membership relatively consistently with the results of , although with considerable error rates. For Llama, we encountered a strong position bias with respect to the ordering of the triple: independently of which was the odd-one-out argument, the model primarily picks arguments at a specific index. This causes its performance to be similar to random picking. We attribute this to the lack of instruction tuning for the Llama model.

Accuracy on the odd-one-out task per method.

Evaluating Argument Selection
To select the best-performing Argument Selection method, we compare BERTscores in Figure <ref>. We observe that all extractive methods have a higher standard deviation when compared to the generative methods. The high maxima for extractive methods are likely caused by the fact that some selected representative arguments were verbatim copied from an opinion in the corpus. The low minima are instead due to the fact that the extractive methods are required to use an argument from the cluster, which may not be representative of the cluster (since there may be noise in the clusters, see Figure <ref>). Conversely, the lower bound for the abstractive methods is higher, showing how rephrasing the selected argument allows it to be more related to all arguments inside a cluster. Between the abstractive methods, ChatGPT achieves a better score than Llama on average but has a higher standard deviation as well. Since we did not perform extensive prompt engineering, there is room for improvement in both methods with better-crafted prompts.

Aggregated BERTscore for the different Key Argument Selection methods across all corpora and argument clusters (Phase 3).

All methods, except for Llama, achieve similar average BERTscores. Surprisingly, none of the approaches on average performs considerably better than random. This suggests that selecting a representative argument from the cluster is relatively simple—in practice, since the argument clusters are coherent, a random selection is generally sufficient. However, in the final evaluation, humans will be judging the match between selected arguments and individual opinions. In this case, we strive for a better worst-case performance—we care less about having perfect matches, but rather wish to have fewer misrepresentation errors. Thus, given the comparable averages, we opt for the method with the highest lower boundary and higher media score (ChatGPT), which we use for the remainder of the experiments.

Finally, we compare the output of Phase 3 of against a version where the selection was manual. In particular, we take the extractions from Phase 1 and re-evaluate them using a new set of annotators. In Table <ref>, we show the difference in Precision (Equation <ref>).

We find that the addition of Argument Selection has some impact on the ability of annotators to match opinions and arguments, although, generally, the decrease in performance is minor. Most interestingly, when comparing argument matches for the same set of opinions before and after the addition of Argument Selection, we find that there is only fair agreement between the re-matched labels (Cohen $\kappa=0.255$). This indicates that the argument selection phase causes some opinions easier to be matched to selected key arguments while making others more difficult. ChatGPT generated as a selected argument a key argument that is representative of the cluster, which sometimes is more general than the arguments extracted from individual opinions. As such, for these opinions, external annotators do not recognize the specific argument being present, but some others are now captured in the selected argument.

Method                 Overall

0.816     0.833      0.641     0.765

w/o Phase 3     0.787      0.848     0.739     0.789

Comparing Precision ($P$) scores with and without Phase 3 (Key Argument Selection phase).

§.§ Comparison with Automated Baseline

Figure <ref> compares the coverage, precision, and diversity scores of and . The low coverage (for both methods) indicates that a large number of opinions do not map to a key argument. This is not surprising since real-world opinions are noisy.

Considering all observed opinions ($C_H$ and $C_A$), yields slightly higher coverage than in the and corpora. In contrast, yields higher coverage than in the corpus. We attribute this to the repeated arguments in the corpus. As 83% of opinions are con-opinions, the policy option (Table <ref>) was highly opposed and its corpus contains many repeated arguments. Since the set of all observed opinions is the entire corpus for , the repeated arguments inflate its coverage. However, since is designed to observe only a small subset of diverse opinions from the corpus, the repeated arguments do not influence its coverage significantly. This is corroborated in the diversity scores, where we observe to consistently output a set of arguments that is more diverse than the ones produced by .

In addition to comparing coverage over observed opinions, we compare the coverage of and with respect to a common set of diverse opinions. In this comparison ($C_H^{common}$ and $C_A^{common}$), yields consistently higher coverage (0.34 on average) than (0.16 on average) in all three corpora. often fails to recognize the key arguments in the diverse set of opinions included by .

Comparing and ArgKP.

yields a larger number of key arguments (around 30 for each option) than . However, these arguments lead to an average precision of 0.56. In contrast, extracts fewer argument clusters (on average 17 per option), but with higher precision (0.80).

§.§ Comparison with Manual Baseline

Table <ref> shows counts of overlapping (yes, yes), missing (no, yes), and new (yes, no) key arguments between and the manual baseline.
required an analysis of 765 opinions, compared to the estimated 3,000 opinions seen in the manual baseline. Despite the lower human effort, the lists largely overlap with the expert lists.

5cManual baseline

2c     2c     2c

(lr)3-4(lr)5-6 (lr)7-8
yes     no     yes     no     yes     no

2*     yes      8     7      7     2     10     1

no        1     –     0     –     4     –

A confusion matrix comparing the key argument lists generated by and manual baseline.

missed some key arguments that the experts identified, e.g., a key argument about building herd immunity was not in the list for the option.
We conjecture that increasing the number of opinions annotated in would subsequently yield the missing insights.
also led to new insights that experts missed, e.g., an argument about the physical well-being of young people was not on the expert list for the option.
Likely, the larger (random) sample of opinions experts analyzed did not include opinions supporting this argument, whereas the smaller (intelligently selected) set sampled in did.

§ DISCUSSION

We find that exploits the strengths of automated methods and the insights from human annotation. outperformed an automated KPA model in terms of precision and diversity, and on a diverse set of opinions, is able to capture more nuanced arguments. Further, expanded beyond an expert analysis, showing how a fully manual procedure may also be limited. In the remainder of this section, we expand on three specific aspects.

Limitations Our experimental setup and comparisons are limited in their scope in multiple ways, thus making our conclusions hard to generalize. Our choice of baseline is the ArgKP model, which was optimized for the task of extracting Key Arguments from a corpus of opinions. However, other automated baselines are conceivable, especially with the introduction of the current generation of flexible LLMs (e.g., ChatGPT, Bard). Those models may be employed for KPA by using prompting techniques liu2023pre. The capabilities of these models seem to imply that they have access to higher order argumentation knowledge lauscher2022scientia, and thus would fare better than the basic ArgKP model. However, having such LLMs reliably process large amounts of citizen feedback without hallucinations is a nontrivial task. In this process, due diligence to preserve a variety of perspectives is required (e.g., by optimizing for a range of opinions instead of single-annotator labels bakker2022fine) in order to prevent rampant misrepresentation of marginalized demographics.

Instead of relying solely on the judgment of an LLM for the task of KPA, we opted to include one in the final step of . While some of the criticism for using an LLM for end-to-end KPA still holds for the Argument Selection step as well, our method investigated a more controlled setup, supported by an objective task definition. Through our comparisons with random and human-generated labels, we aim to show where, how, and to what extent LLMs may aid in the KPA process. As ever, the choice of metrics remains important for measuring the effect size.

Balancing Task Allocation The pairwise comparison in the consolidation phase is the most human-intensive task in , and the effort increases with the number of analyzed opinions. Also, comparing arguments is cognitively demanding, partly evidenced by the low IRR. While reduced the number of comparisons required in the consolidation phase by 60%, we may experiment with different setups or other techniques for comparing arguments to remove this overhead. For example, first clustering the key arguments and then consolidating the arguments within these clusters (reverse order as ) may drastically reduce the number of judgments required in the second phase.

We place human efforts in places where there are multiple bidirectional benefits possible stemming from performing the task. For instance, the Argument Annotation task both serves the purpose of analyzing the opinions to progress our method, as well as actively making annotators perform perspective-taking. On multiple occasions, annotators noted on their increase in sympathy and recognition of the issues raised in the comments, showcasing how the task could further help bring understanding to a group of citizens.

Ablations studies Any part of the pipeline is open to adjustment and can be performed by humans, machines, or a combination of both. presents a general framework that allows individual phases to be supported by different types of technologies.
Within this hybrid framework, we considered the following criteria when deciding to allocate tasks to humans or to AI methods:

* let humans read other's opinions to promote perspective-taking,

* use humans to solve tasks where AI methods may incur considerable error,

* leverage AI methods for routine tasks, and

* use task-specific intrinsic evaluation metrics for selecting the right method.

In each phase, we perform both intrinsic evaluations (e.g., observe error rates for particular tasks or annotator behavior) and extrinsic evaluation against two baselines. This fits a standardized machine learning pipeline, except that we are now able to [label=(*)]

* evaluate annotator behavior and model performance jointly, and

* make decisions on which techniques to use based on some intermediate statistic.
We believe this setup to be generalizable for Hybrid Intelligence systems, as it makes the role of the designer and their decisions explicit akata2020research. Furthermore, the results remain interpretable, as any decision made by either annotators or models can be traced from opinion to selected key argument.

Different configurations of the framework are possible, and the one we have presented is an instance that tackles the problem of policy feedback analysis. is a complex combination of AI methods and human annotation. Our main objective was to present the framework, as well as a real-world use case to show the benefit of using a Hybrid Intelligent methodology. However, other choices for individual components of can be used, or parts of the method can be performed solely by humans or AI methods. We leave this open for future work, as running ablations requires considerable amounts of work, and we envision research to come up with similar use cases where HI can make a significant impact.

§ CONCLUSION AND FUTURE DIRECTIONS

We develop and evaluate , a hybrid method that combines human judgments with automated methods to generate a diverse set of key arguments.
extracts key arguments from noisy opinions and achieves consistent coverage, whereas the coverage of a state-of-the-art automated method drops by 50% when switching from all (containing repeated) opinions to diverse opinions.
Moreover, the key arguments extracted by are more precise than those extracted by the automated baseline.
Additionally, provides important insights that were not included in an expert-driven analysis of the same corpus, despite requiring fewer opinions to be analyzed.

Finding arguments in a discourse is only one aspect that constitutes the perspectives in a discussion. Future work can incorporate analysis of other perspective factors, such as values liscio2021axies,liscio2022values, sentiment, emotion, and attribution van2016grasp. By combining these rich aspects with arguments, we can merge the logical basis of the discussion with other semantic and syntactic information, allowing close scrutiny of the perspectives in opinions.

This research was (partially) funded by the Hybrid Intelligence Center, a 10-year program funded by the Dutch Ministry of Education, Culture, and Science through the Netherlands Organisation for Scientific Research.