# VECHR: A Dataset for Explainable and Robust Classification of Vulnerability
Type in the European Court of Human Rights

Shanshan Xu1, Leon Staufer1,2, Santosh T.Y.S.S1,
Oana Ichim3, Corina Heri4, Matthias Grabmair1
1Technical University of Munich, Germany, 2LMU Munich, Germany
3Graduate Institute of International and Development Studies, Switzerland
4Faculty of Law, University of Zürich, Switzerland
{shanshan.xu, leon.staufer, santosh.tokala<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recognizing vulnerability is crucial for understanding and implementing
targeted support to empower individuals in need. This is especially important
at the European Court of Human Rights (ECtHR), where the court adapts
convention standards to meet actual individual needs and thus to ensure
effective human rights protection. However, the concept of vulnerability
remains elusive at the ECtHR and no prior NLP research has dealt with it. To
enable future work in this area, we present VECHR, a novel expert-annotated
multi-label dataset comprised of vulnerability type classification and
explanation rationale. We benchmark the performance of state-of-the-art models
on VECHR from both the prediction and explainability perspective. Our results
demonstrate the challenging nature of the task with lower prediction
performance and limited agreement between models and experts. We analyze the
robustness of these models in dealing with out-of-domain (OOD) data and
observe limited overall performance. Our dataset poses unique challenges
offering a significant room for improvement regarding performance,
explainability, and robustness.

##  1 Introduction

Vulnerable Type | Description
---|---
Dependency |  | Including that of minors, the elderly, and those with physical, psychosocial
---
and cognitive disabilities (i.e. mental illness and intellectual disability)
State Control | Including that of detainees, military conscripts, and persons in state institutions
Victimisation |  | Due to victimisation, including by domestic and sexual abuse, other violations,
---
or because of a feeling of vulnerability
Migration | In the migration context, applies to detention and expulsion of asylum-seekers
Discrimination |  | Due to discrimination and marginalisation, which covers ethnic, political and
---
religious minorities, LGBTQI people, and those living with HIV/AIDS
Reproductive Health | Due to pregnancy or situations of precarious reproductive health
Unpopular Views | Due to the espousal of unpopular views
Intersection | Intersecting vulnerabilities

Table 1: Description of each vulnerability type. For more details, see App A.

Vulnerability encompasses a state of susceptibility to harm, or exploitation,
particularly among individuals or groups who face a higher likelihood of
experiencing adverse outcomes due to various factors such as age, health,
disability, or marginalized social position Mackenzie et al. (2013); Fineman
(2016). While it is impossible to eliminate vulnerability, society has the
capacity to mitigate its impact. The European Court of Human Rights (ECtHR)
interprets the European Convention of Human Rights (ECHR) to address the
specific contextual needs of individuals and provide effective protection.
This is achieved through various means, such as displaying flexibility in
admissibility issues, and shifting the burden of proof Heri (2021).

(a) Evolving distribution of types of vulnerability.

(b) Increase in application of the vulnerability type “migration” between 2010
and 2018.

Figure 1: Distribution changes of vulnerability types.

However, the concept of vulnerability remains elusive within the ECtHR. While
legal scholars have explored vulnerability as a component of legal reasoning
Peroni and Timmer (2013), empirical work in this area remains scarce and
predominantly relies on laborious manual processes. To address this challenge,
NLP can offer valuable tools to assist experts in efficiently classifying and
analyzing textual data. Besides high classification performance, the true
utility of NLP in the legal field is its ability to identify relevant aspects
related to vulnerability in court cases. These aspects can be extracted,
grouped into patterns, and used to inform both litigation strategy and legal
policy. Even so, a significant obstacle to progress in this area is the lack
of appropriate datasets. To bridge these research gaps, we present the dataset
VECHR111VECHR stands for Vulnerability Classification in European Court of
Human Rights. Our dataset and code is available at
https://github.com/TUMLegalTech/vechr_emnlp23, which comprises cases dealing
with allegation of Article 3 “Prohibition of torture” and is obtained from
legal expert’s empirical study222Heri 2021. Heri is the fifth author of this
work.. Our proposed task is to identify which type of vulnerability (if any)
is involved in a given ECtHR case.

As model explainability is crucial for establishing trust, we extend the
dataset with VECHRexplain, a token-level explanation dataset annotated by
domain experts on a subset of VECHR. Its fine-grained token-level design
mitigates performance overestimation of explainability when evaluated at the
coarse paragraph level, as shown in previous works Chalkidis et al. (2021);
Santosh et al. (2022); Xu et al. (2023). Further, the understanding and
application of vulnerability in court proceedings change over time, reflecting
societal shifts and expanding to encompass a wider range of types (1(a)). The
volume of cases also fluctuates significantly in response to social and
political events (1(b)). To evaluate the model’s robustness against
distribution shifts, we further collect and annotate an additional out-of-
domain (OOD) test set from cases involving non-Article 3 allegations, called
VECHRchallenge.

We present comprehensive benchmark results using state-of-the-art (SOTA)
models, revealing limited performance in vulnerability type classification in
VECHR. We assess the models’ alignment with expert explanations in
VECHRexplain, and observe limited agreement. Experiment results on
VECHRchallenge indicate that, although incorporating description of the
vulnerability type helps to improve the models’ robustness, the performance
remains low overall due to the challenges posed by the distribution shift. Our
experiments underscore the difficulty of vulnerability classification in
ECtHR, and highlight a need for further investigation on improve model
accuracy, explainability, and robustness.

##  2 Vulnerability Typology in ECtHR

The inescapable and universal nature of vulnerability, as posited by Fineman
(2016), underscores its significance in legal reasoning. For instance, the
European Union has acknowledged the concept by establishing a definition for
vulnerable individuals Dir (2013). However, it remains undefined within the
context of ECtHR. To facilitate an examination of vulnerability and its
application within the ECtHR, it is crucial to establish a typology recognized
by the Court. Several scholars have endeavored to effectively categorize
vulnerability for this purpose Timmer (2016); Limantė (2022). One notable
study is conducted by Heri (2021), which provides a systematic and
comprehensive examination of the concept of vulnerability under ECHR Article
3. Heri proposes a complete typology encompassing eight types: dependency,
state control, victimization, migration, discrimination, reproductive health,
unpopular views and intersections thereof. Tab 1 gives a description for each
type.

##  3 Data Collection and Annotations

###  3.1 Data Source and Collection Process

VECHR consists of 788 cases under Article 3, which were collected based on
Heri’s study of the Court’s case law references of vulnerability. See App B
for details on Heri’s case sampling methodology and our post-processing
procedures. We divided the dataset chronologically into three subsets:
training (–05/2015, 590 cases), validation (05/2015–09/2016, 90 cases) and
test (09/2016–02/2019, 108 cases).

VECHRexplain: We selected 40 cases (20 each) from the val and test splits for
the explanation dataset. Within each split, our sampling procedure involved
two steps. First, we ensured coverage of all seven types by sampling one case
for each type. Subsequently, we randomly selected an additional 13 cases to
supplement the initial selection.

VECHRchallenge: To test the model’s ability to generalize across distribution
shifts, we extend VECHR by collecting and annotating additional cases _not_
related to Article 3. Following Heri’s method, we used the regular expression
“vulne*” to retrieve all English relevant documents from the ECtHR’s public
database HUDOC333https://hudoc.echr.coe.int and exclude cases related to
Article 3. We restricted the collection to the time span from 09/2016
(corresponding to start time of the test set) to 07/2022. In cases where
multiple documents existed for a given case, we selected only the most recent
document, resulting in a dataset consisting of 282 judgments. VECHRchallenge
can be regarded as an out-of-domain topical (OOD) scenario. The in-domain
train/val/test of VECHR are all from the same text topic cluster of Article 3.
The OOD VECHRchallenge consists of non-Article 3 cases from different topic
clusters (e.g. Article 10: freedom of expression), which involves different
legal concepts and language usage.444For example, the Court recognizes the
vulnerability of an elderly woman and provides her with protection under
Article 3 (prohibiting torture) rather than Article 10 (freedom of expression)

###  3.2 Vulnerability Type Annotation

We follow the typology and methodology presented by Heri 2021. She considered
cases as “vulnerable-related”, only when “vulnerability had effectively been
employed by the Court in its reasoning”. These cases are further coded
according to the trait or situation (vulnerable type) giving rise to the
vulnerability. In situations where the Court considered that multiple traits
contributed to the vulnerability, she coded the case once for each relevant
category. The resulting dataset comprises 7 labels555See App D for our
justification for excluding the type “intersectionality”.. Cases in which
vulnerability was used only in its common definition, e.g. “financially
vulnerability”, were regarded as ‘non-vulnerable’ and were labelled none

Split | $\\#C$ | $T/C$ | $P/C$ | $\\#C_{\lnot V}$ | $L/C$ | $L/C_{V}$
---|---|---|---|---|---|---
Train | 590 | 5140 | 83 | 325 | 0.70 | 1.55
Validation | 90 | 5077 | 77 | 27 | 1.41 | 2.02
Test | 108 | 3992 | 57 | 34 | 1.13 | 1.65
Challenge | 282 | 4176 | 51 | 144 | 0.61 | 1.24
Total | 1070 | 4765 | 72 | 530 | 0.78 | 1.54

Table 2: Dataset statistics for each split, with number of cases ($C$), number
of non-vulnerable cases ($C_{\lnot V}$), mean tokens ($T$) per case, mean
paragraphs per case, mean labels ($L$) per case, and mean labels per case when
only considering positive vulnerability cases ($C_{V}$).

of the 7 types. See App C for more details of the definition of “vulnerable-
related”.

For cases under Article 3, we adopted the labelling provided by Heri’s
protocol. For VECHRchallenge, we ask two expert annotators666See App E for
annotators’ background and expertise.  to label the case following Heri’s
methodology777For the reason of why Heri confined her study to Article 3 and
why the typology also applies to cases under other articles, please refer to
App F.. Each annotator has annotated 141 cases.

Inter-Annotator Agreement To ensure consistency with Heri’s methodology, we
conducted a two-round pilot study before proceeding with the annotation of the
challenge set (details in App G). In each round, two annotators independently
labelled 20 randomly selected cases under Article 3, and we compared their
annotations with Heri’s labels. The inter-annotator agreement was calculated
using Fleiss Kappa, and we observed an increase from 0.39 in the first round
to 0.64 in the second round, indicating substantial agreement across seven
labels and three annotators.

###  3.3 Explanation Annotation Process

The explanation annotation process was done using the GLOSS annotation tool
Savelka and Ashley (2018), see App H for details. Based on the case facts, the
annotators was instructed to identify relevant text segments that indicate the
involvement of a specific vulnerability type in the Court’s reasoning. The
annotators was permitted to highlight the same text span as an explanation for
multiple vulnerable types.

##  4 Dataset Analysis

(a) Hierarchical Classification Variant

(b) Concept-aware Classification Variant

Figure 2: Visualization of Hierarchical and Concept-aware Hierarchical Model
architectures.

Tab 2 presents the key statistics of our dataset. VECHR comprises a total of
1,070 documents, with an average of 4,765 tokens per case ($\sigma=4167$). 788
and 282 cases fall under the Article 3 and non-Article 3 partitions,
respectively. Among all, 530 documents are considered as “non-vulnerable”,
meaning they are not labelled as any of the seven vulnerable types. In the
vulnerable-related cases, the average number of labels assigned per document
is 1.54.

We observe a strong label distribution imbalance within the dataset. The label
“state control” dominates, accounting for 33% of the cases, while the least
common label, “reproductive health”, is present in only 3% of the cases. For
more detailed statistics of our dataset, including details regarding the label
imbalances in Tab 6, please refer to App I.

##  5 Experiments

###  5.1 Vulnerability Type Classification

Task: Our objective is to predict the set of specific vulnerability type(s)
considered by the Court based on the factual text of a case.

Models: We finetune pre-trained models _BERT_ Devlin et al. (2019),
_CaselawBERT_ Zheng et al. (2021), _LegalBERT_ Chalkidis et al. (2020): on our
dataset with a multi-label classification head, truncating the input to the
maximum of 512 tokens.

Model | Classification | Explanation
---|---|---
| mac-F1 | mic-F1 | Kappa
random | 19.02 | 25.07 | -0.11 ± 0.02
BERT | 24.31 | 41.78 | 0.02 ± 0.06
CaselawBERT | 27.31 | 45.16 | 0.04 ± 0.08
LegalBERT | 27.34 | 42.47 | 0.04 ± 0.07
Longformer | 31.49 | 46.21 | 0.11 ± 0.11
Hierachical | 31.46 | 45.32 | 0.10 ± 0.08

Table 3: Classification and explanation results. We report F1s for
classification performance and Kappa score with standard error for explanation
agreement.

We finetune the _Longformer_ model Beltagy et al. (2020) on our dataset that
allows for processing up to 4,096 tokens, using a sparse-attention mechanism
which scales linearly, instead of quadratically.

We further employ a _hierarchical_ variant of pre-trained LegalBERT to deal
with the long input limitation. We use a greedy input packing strategy where
we merge multiple paragraphs888Details and statistics on paragraphs are
reported in App I. into one packet until it reaches the maximum of 512 tokens.
We independently encode each packet of the input text using the pretrained
model and obtain representations ($h_{[CLS]}$) for each packet. Then we apply
a non-pretrained transformer encoder to make the packet representations
context-aware. Finally, we apply max-pooling on the context-aware packet
representations to obtain the final representation of the case facts, which is
then passed through a classification layer. 2(a) illustrates the detailed
architecture of the hierarchical model.

For details on all models’ configuration and training, please refer to App J.

Evaluation Metrics: we report micro-F1 (mic-F1) and macro-F1 (macF1) scores
for 7+1 labels, where 7 labels correspond to 7 vulnerability types under
consideration and an additional augmented label during evaluation to indicate
non-vulnerable.

Results: Tab 3 reports the results of classification performance. We observe
that legal-specific pre-training improved the performance over general pre-
training. However, BERT models still face the input limitation constraint.
Both Longformer and Hierarchical models improved compared to truncated
variants and are comparable to each other. Overall, we see low overall
performance across models, highlighting the challenging task.

###  5.2 Vulnerability Type Explanation

We use Integrated Gradient (IG) Sundararajan et al. (2017) to obtain token-
level importance from the model with respect to each vulnerable type under
consideration. We max pool over sub-words to convert token-level IG scores
into word-level scores, followed by a threshold-based binarization. Tab 3
reports explainability performance expressed as the average of Cohen’s
$\kappa$ between the models’ focus and the experts’ annotations for the test
instances. We observe that the low explainability scores among different
models reflect their trend in classification scores and also echo the
challenging nature of the task.

###  5.3 Robustness to Distributional Shifts

Model | VECHRchallenge
---|---
| mac-f1 | mic-f1
random | 12.75 | 14.61
BERT | 20.51 | 43.48
CaselawBERT | 24.55 | 57.51
LegalBERT | 22.60 | 50.77
Longformer | 25.24 | 55.71
Hierarchical |  26.43 | 58.46
Concept-aware Hierarchical | 33.11 | 49.62

Table 4: Results on the challenge dataset.

We assess the robustness of models to distributional shift using the
VECHRchallenge and present the performance in Tab 4. Notably, we observe a
drop in macro-F1 score on VECHRchallenge compared to the test set. We
attribute this to the models relying on suboptimal information about
vulnerability types, which is primarily derived from the factual content
rather than a true understanding of the underlying concept. To address this
limitation, we propose a Concept-aware Hierarchical model that considers both
the case facts and the description of vulnerability type to determine if the
facts align with the specified vulnerability type999We cast the multi-label
task into a binary classification setup by pairing the text with each
vulnerability type. These binary labels are transformed into a multi-label
vector for performance evaluation, to produce a fair comparison to multi-label
models on the same metric., inspired by Tyss et al. 2023a. We employ a greedy
packing strategy as described earlier and use a hierarchical model to obtain
the context-aware packet representations for each packet in the facts and
concept description separately. Subsequently, we apply scaled-dot-product
cross attention between the packet vectors of the facts (as Query) and
concepts (as Keys and Values), generating the concept-aware representation of
the facts section packets. A transformer layer is used to capture the
contextual information of the updated packet vectors. Then we obtain the
concept-aware representation of the case facts via max pooling and pass it
through a classification layer to obtain the binary label. 2(b) illustrates
the detailed architecture of the concept-aware model. For more details, see
App K.

The concept-aware model exhibits increased robustness to distributional shift
and shows an improvement on the challenge set, owed to the incorporation of
the vulnerability type descriptions. Overall, our results show promise for the
feasibility of the task yet indicate room for improvement.

##  6 Conclusion

We present VECHR, an ECtHR dataset consisting of 1,070 cases for vulnerability
type classification and 40 cases for token-level explanation. We also release
a set of baseline results, revealing the challenges of achieving accuracy,
explainability, and robustness in vulnerability classification. We hope that
VECHR and the associated tasks will provide a challenging and useful resource
for Legal NLP researchers to advance research on the analysis of vulnerability
within ECtHR jurisprudence, ultimately contributing to effective human rights
protection.

## Limitations

In our task, the length and complexity of the legal text require annotators
with a deep understanding of ECtHR jurisprudence to identify vulnerability
types. As a result, acquiring a large amount of annotation through
crowdsourcing is not feasible, leading to limited-sized datasets.
Additionally, the high workload restricts us to collecting only one annotation
per case. There is a growing body of work in mainstream NLP that highlights
the presence of irreconcilable Human Label VariationPlank (2022); Basile et
al. (2021) in subjective tasks, such as natural language inference Pavlick and
Kwiatkowski (2019) and toxic language detection Sap et al. (2022). Future work
should address this limitation and strive to incorporate multiple annotations
to capture a more and potentially multi-faceted of the concept of
vulnerability.

This limitation is particularly pronounced because of the self-referential
wording of the ECtHR Fikfak (2021). As the court uses similar phrases in cases
against the same respondent state or alleging the same violation, the model
may learn that these are particularly relevant, even though this does not
represent the legal reality. In this regard, it is questionable whether cases
of the ECtHR can be considered “natural language”. Moreover, the wording of
case documents is likely to be influenced by the decision or judgement of the
Court. This is because the documents are composed by court staff after the
verdict. Awareness of the case’s conclusion could potentially impact the way
its facts are presented, leading to the removal of irrelevant information or
the highlighting of facts that were discovered during an investigation and are
pertinent to the result Medvedeva et al. . Instead, one could base the
analysis on the so-called “communicated cases”, which are often published
years before the case is judged. However, these come with their own
limitations and only represent the facts as characterized by the applicant
applicant and not the respondent state. There are also significantly fewer
communicated cases than decisions and judgements.

One of the main challenges when working with corpora in the legal domain is
their extensive length. To overcome this issue, we employ hierarchical models,
which have a limitation in that tokens across long distances cannot directly
interact with each other. The exploration of this limitation in hierarchical
models is still relatively unexplored, although there are some preliminary
studies available (e.g., see Chalkidis et al. 2022). Additionally, we choose
to freeze the weights in the LegalBERT sentence encoder. This is intended to
conserve computational resources and reduce the model’s vulnerability to
superficial cues.

## Ethics Statement

Ethical considerations are of particular importance because the dataset deals
with vulnerability and thus with people in need of special protection. In
general, particular attention needs to be paid to ethics in the legal context
to ensure the values of equal treatment, justification and explanation of
outcomes and freedom from bias are upheld Surden (2019).

The assessment of the ethical implications of the dataset is based on the Data
Statements by Bender and Friedman (2018). Through this, we aim to establish
transparency and a more profound understanding of limitations and biases. The
curation is limited to the Article 3 documents in English. The speaker and
annotator demographic are legally trained scholars, proficient in the English
language. “Speaker” here refers to the authors of the case documents, which
are staff of the Court, rather than applicants. We do not believe that the
labelling of vulnerable applicants is harmful because it is done from a
legally theoretical perspective, intending to support applicants. The
underlying data is based exclusively on the publicly available datasets of
ECtHR documents available on HUDOC101010https://hudoc.echr.coe.int. The
documents are not anonymized and contain the real names of the individuals
involved. We do not consider the dataset to be harmful, given that the
judgments are already publicly available.

We are conscious that, by adapting pre-trained encoders, our models inherit
any biases they contain. The results we observed do not substantially relate
to such encoded bias. Nonetheless, attention should be paid to how models on
vulnerability are employed practically.

In light of the aforementioned limitations and the high stakes in a human
rights court, we have evaluated the potential for misuse of the vulnerability
classification models. Medvedeva et al. (2020) mention the possibility of
reverse engineering the model to better prepare applications or defences. This
approach is, however, only applicable in a fully automated system using a
model with high accuracy towards an anticipated decision outcome. As this is
not the case for the models presented, we assume the risk of circumventing
legal reasoning to be low. On the contrary, we believe employing a high recall
vulnerability model could aid applicants and strengthen their legal reasoning.
In a scholarly setting focused on vulnerability research, we do not think the
model can be used in a detrimental way. Our research group is strongly
committed to research on legal NLP models as a means to derive insight from
legal data for purposes of increasing transparency, accountability, and
explainability of data-driven systems in the legal domain.

There was no significant environmental impact, as we performed no pre-training
on large datasets. Computational resources were used for fine-tuning and
training the models, as well as assessing the dataset. Based on partial
logging of computational hours and idle time, we estimate an upper bound for
the carbon footprint of 30 kg $\mathrm{CO_{2}}$ equivalents. This is an
insignificant environmental impact.

## References

  * Dir (2013) 2013\.  Directive 2013/33/EU of the European Parliament and of the Council of 26 June 2013 laying down standards for the reception of applicants for international protection (recast).
  * Basile et al. (2021) Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021.  We need to consider disagreement in evaluation.  In _Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future_ , pages 15–21, Online. Association for Computational Linguistics.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Bender and Friedman (2018) Emily M. Bender and Batya Friedman. 2018.  Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604.
  * Chalkidis et al. (2022) Ilias Chalkidis, Xiang Dai, Manos Fergadiotis, Prodromos Malakasiotis, and Desmond Elliott. 2022.  An exploration of hierarchical attention transformers for efficient long document classification.  _arXiv preprint arXiv:2210.05529_.
  * Chalkidis et al. (2020) Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020.  Legal-bert: The muppets straight out of law school.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2898–2904.
  * Chalkidis et al. (2021) Ilias Chalkidis, Manos Fergadiotis, Dimitrios Tsarapatsanis, Nikolaos Aletras, Ion Androutsopoulos, and Prodromos Malakasiotis. 2021.  Paragraph-level rationale extraction through regularization: A case study on european court of human rights cases.  In _Proceedings of the NAACL-HLT 2021_ , pages 226–241.
  * Charte et al. (2013) Francisco Charte, Antonio Rivera, María José del Jesus, and Francisco Herrera. 2013.  A First Approach to Deal with Imbalance in Multi-label Datasets.  In _Hybrid Artificial Intelligent Systems_ , Lecture Notes in Computer Science, pages 150–160, Berlin, Heidelberg. Springer.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fikfak (2021) Veronika Fikfak. 2021.  What Future for Human Rights? Decision-making by algorithm.
  * Fineman (2016) Martha Fineman. 2016.  The Vulnerable Subject: Anchoring Equality in the Human Condition.  _Yale Journal of Law & Feminism_.
  * Heri (2021) Corina Heri. 2021.  _Responsive Human Rights: Vulnerability, Ill-treatment and the ECtHR_.  Bloomsbury Academic.
  * Limantė (2022) Agnė Limantė. 2022.  Vulnerable Groups in the Case Law of the European Court of Human Rights.  In Agnė Limantė and Dovilė Pūraitė-Andrikienė, editors, _Legal Protection of Vulnerable Groups in Lithuania, Latvia, Estonia and Poland_ , volume 8, pages 29–54. Springer International Publishing, Cham.
  * Mackenzie et al. (2013) Catriona Mackenzie, Wendy Rogers, and Susan Dodds. 2013.  Introduction: What Is Vulnerability, and Why Does It Matter for Moral Theory? In _Vulnerability: New Essays in Ethics and Feminist Philosophy_ , page 0. Oxford University Press.
  * (15) Masha Medvedeva, Michel Vols, and Martijn Wieling.  Judicial Decisions of the European Court of Human Rights:.
  * Medvedeva et al. (2020) Masha Medvedeva, Martijn Wieling, and Michel Vols. 2020.  The Danger of Reverse-Engineering of Automated Judicial Decision-Making Systems.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Pavlick and Kwiatkowski (2019) Ellie Pavlick and Tom Kwiatkowski. 2019.  Inherent disagreements in human textual inferences.  _Transactions of the Association for Computational Linguistics_ , 7:677–694.
  * Peroni and Timmer (2013) Lourdes Peroni and Alexandra Timmer. 2013.  Vulnerable groups: The promise of an emerging concept in European Human Rights Convention law.  _International Journal of Constitutional Law_ , 11(4):1056–1085.
  * Plank (2022) Barbara Plank. 2022.  The “problem” of human label variation: On ground truth in data, modeling and evaluation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Santosh et al. (2022) T.y.s.s Santosh, Shanshan Xu, Oana Ichim, and Matthias Grabmair. 2022.  Deconfounding legal judgment prediction for European court of human rights cases towards better alignment with experts.  In _Proceedings of EMNLP 2022_.
  * Sap et al. (2022) Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022.  Annotators with attitudes: How annotator beliefs and identities bias toxic language detection.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5884–5906, Seattle, United States. Association for Computational Linguistics.
  * Savelka and Ashley (2018) Jaromir Savelka and Kevin D Ashley. 2018.  Segmenting us court decisions into functional and issue specific parts.  In _JURIX_ , pages 111–120.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _ICML_ , pages 3319–3328. PMLR.
  * Surden (2019) Harry Surden. 2019.  The Ethics of Artificial Intelligence in Law: Basic Questions.
  * Timmer (2016) Alexandra Timmer. 2016.  A Quiet Revolution: Vulnerability in the European Court of Human Rights.  In Martha Albertson Fineman and Anna Grear, editors, _Vulnerability: Reflections on a New Ethical Foundation for Law and Politics_ , zeroth edition. Routledge.
  * Tyss et al. (2023a) Santosh Tyss, Oana Ichim, and Matthias Grabmair. 2023a.  Zero-shot transfer of article-aware legal outcome classification for european court of human rights cases.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 593–605.
  * Tyss et al. (2023b) Santosh Tyss, Marcel Perez San Blas, Phillip Kemper, and Matthias Grabmair. 2023b.  Leveraging task dependency and contrastive learning for case outcome classification on european court of human rights cases.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1103–1103.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2023) Shanshan Xu, Santosh T.Y.S.S, Oana Ichim, Isabella Risini, , and Matthias Grabmair. 2023.  From dissonance to insights: Dissecting disagreements in rationale dataset construction for case outcome classification.  _arXiv preprint arXiv2310.11878_.
  * Zheng et al. (2021) Lucia Zheng, Neel Guha, Brandon R Anderson, Peter Henderson, and Daniel E Ho. 2021.  When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings.  In _Proceedings of the eighteenth international conference on artificial intelligence and law_ , pages 159–168.

##  Appendix A Vulnerability Typology and Descriptions

Here is the typology of vulnerability in ECtHR Heri (2021):

  * •

Dependency: dependency-based vulnerability, which concerns minors, the
elderly, and those with physical, psychosocial and cognitive disabilities
(i.e., mental illness and intellectual disability).

  * •

State Control: vulnerability due to state control, including vulnerabilities
of detainees, military conscripts, and persons in state institutions.

  * •

Victimisation: vulnerability due to victimisation, including by domestic and
sexual abuse, other violations, or because of a feeling of vulnerability.

  * •

Migration: vulnerability in the migration context, applies to detention and
expulsion of asylum-seekers.

  * •

Discrimination: vulnerability due due to discrimination and marginalisation,
which covers ethnic, political and religious minorities, LGBTQI people, and
those living with HIV/AIDS.

  * •

Reproductive Health: vulnerability due to pregnancy or situations of
precarious reproductive health.

  * •

Unpopular Views: vulnerability due to the espousal of unpopular views.

  * •

Intersection: intersecting vulnerabilities.

Following is a detailed description of each type:
Dependency Dependency-based vulnerability derives from the inner
characteristics of the applicant and thus concerns minors, elderly people, as
well as physical, psychosocial and cognitive disabilities (i.e., mental
illness and intellectual disability). The Court has built special requirements
around these categories to be fulfilled by States as part of their obligations
under the Convention. Minors: The Court often refers to children as a
paradigmatic example of vulnerable people and made use of the concept of
vulnerability to require States to display particular diligence in cases
imposing child protection given, on the one hand, their reduced ability and/or
willingness of complaining of ill-treatment and, on the other hand, their
susceptibility to be exposed to traumatic experiences/treatment. Elderly: In
many ways, vulnerability due to advanced age is a continuation of the
vulnerability of children. All humans experience dependency at the beginning
of life, and many experience it near the end. Intellectual and Psychosocial
Disabilities: Intellectual disability may render those affected dependent on
others – be it the state or others who provide them with support. Having
regard to their special needs in exercising legal capacity and going about
their lives, the Court considered that such situations were likely to attract
abuse. Persons living with intellectual disabilities experience difficulties
in responding to, or even protesting against, violations of their rights. In
addition, persons with severe cognitive disabilities may experience a legal
power imbalance because they do not enjoy legal capacity.

State Control Vulnerability due to state control includes detainees, military
conscripts, and persons in state institutions. This type of vulnerability
includes persons in detention, but also those who are institutionalised or
otherwise under the sole authority of the state. When people are deprived of
their liberty, they are vulnerable because they depend on the authorities both
to guarantee their safety and to provide them with access to essential
resources like food, hygienic conditions, and health care. In addition, the
state often controls the flow of information and access to proof. Hence, the
Court automatically applies the presumption of state responsibility when harm
comes to those deprived of their liberty.

Victimisation Vulnerability due to victimisation refers to situations in which
harm is inflicted by someone else. This type of vulnerability applies to
situations of domestic and sexual abuse, and other type of abuse. The Court
has also found that a Convention violation may, in and of itself, render
someone vulnerable. Crime victims who are particularly vulnerable, through the
circumstances of the crime, and can benefit from special measures best suited
to their situation. Sexual and domestic violence are expressions of power and
control over the victim, and inflict particularly intense forms of trauma from
a psychological standpoint. Failing to recognise the suffering of the victims
of sexual and domestic violence or engaging in a stigmatising response – such
as, for example, the perpetuation of so-called ‘rape myths’ – represents a
secondary victimisation or ‘revictimisation’ of the victims by the legal
system

Migration Vulnerability in the context of migration applies to detention and
expulsion of asylum-seekers. Applicants as asylum-seeker are considered
particularly vulnerable based on the sole experience of migration and the
trauma he or she was likely to have endured previously’. The feeling of
arbitrariness and the feeling of inferiority and anxiety often associated with
migration, as well as the profound effect conditions of detention in special
centres, indubitably affect a person’s dignity. The status of the applicants
as asylum-seekers is considered to require special protection because of their
underprivileged (and vulnerable) status.

Discrimination Vulnerability due to discrimination and marginalisation covers
ethnic, political and religious minorities, LGBTQI people, and those living
with HIV/AIDS. The Court recognises that the general situation of these groups
– the usual conditions of their interaction with members of the majority or
with the authorities – is particularly difficult and at odds with
discriminatory attitudes. Similarly to dependency-based vulnerability, this
type of vulnerability imposes special duties on states and depends not solely
on the inner characteristics of applicants but also on their choices which, in
most cases, states have to balance against other choices and interests.

Reproductive Health Vulnerability due to pregnancy or situations of precarious
reproductive health concerns situations in which women may find themselves in
particular vulnerable situations, even if the Court does not consider women
vulnerable as such. This may be due to an experience of victimisation, for
example in the form of gender-based violence, or due to various contexts that
particularly affect women. Sometimes, depending on the circumstances,
pregnancy may be reason enough for vulnerability while other times
vulnerability is linked to the needs of the unborn children.

Unpopular Views Vulnerability due to the espousal of unpopular views includes:
demonstrators, dissidents, and journalists exposed to ill-treatment by state
actors. Where an extradition request shows that an applicant stands accused of
religiously and politically motivated crimes, the Court considers this
sufficient to demonstrate that the applicant is a member of a vulnerable
group. Similarly to the case of victimisation, it is also the applicant’s
choice to display such views and vulnerability comes from particular measures
that state undertake when regulating or disregarding such choices.

##  Appendix B More details on Data Source and Collection Process

Heri 2021 reported the details of the case sampling process. The following
serves as a summary of her case sampling methodology: She used the regular
expression “vulne*” to retrieve all relevant English documents related to
Article 3 from HUDOC, the public database of the ECtHR, excluding communicated
cases and legal summaries, for the time span between the inception of the
Court and 28 February 2019. This yielded 1,147 results.

Heri recorded her labeling in an Excel sheet, including the application number
for each case. The application number serves as a unique identifier for
individual applications submitted to the ECtHR. To create VECHR, we fetch all
relevant case documents from HUDOC, including their metadata. During the post-
processing, when one case has multiple documents, we keep the latest document
and discard the rest, which yields 788 documents.

##  Appendix C Definition of “Vulnerable-related”

Heri (2021) specified that only cases where “vulnerability had effectively
been employed by the Court in its reasoning” are regarded as “vulnerable-
related”. Cases in which vulnerability was used only in its common definition,
or used only in the context of other ECHR rights, or by third parties, were
excluded. To ensure clarity and alignment with Heri’s perspective, we
communicated with her during the annotation process to clarify the definition
of “vulnerable-related”. As a result, we determined that vulnerability is
labeled (primarily) in situations where:

  * •

Vulnerability is part of the Court’s main legal reasoning.

  * •

The alleged victims (or their children) exhibit vulnerability.

  * •

The ECtHR, rather than domestic courts or other parties, consider the alleged
victims vulnerable.

##  Appendix D Omitting “Intersectionality” Label

The vulnerability type “intersectionality” was omitted because of its unclear
usage in cases. Even more than the other typologies, it is a highly nuanced
concept without a clear legal definition. Furthermore, Heri (2021, 117) says
that the ECtHR does not engage with the concept of intersectionality in this
form. Given that there are only 11 cases exclusively annotated as
“intersectional” (out of a total of 59), the effect of disregarding it in this
work is negligible. Omission does not suggest that intersectionality fails to
play a role in the reasoning of the ECtHR or that we deem it irrelevant.
Instead, we suggest exploring the concept of vulnerability as a combination of
the other seven vulnerabilities.

##  Appendix E Annotator Background & Expertise

Two annotators performed the classification task. Annotator 1 (the fourth
author) is a Post-doctoral Researcher at a European Research Centre, who has
worked at the European Court of Human Rights as a case lawyer. Annotator 2
(the second author) is a law student, with also a philosophy and computer
science background. Prior to annotation, both annotators had read Heri’s book,
and Annotator 2 had received a training session on the ECHR from Annotator 1.

The explanation dataset was annotated by Annotator 1.

##  Appendix F Justification of Article Applicability

Heri’s (2021) typology is limited to Article 3 of the ECHR, which pertains to
the Prohibition of Torture. Under Article 3, the concept of vulnerability was
first coined by the ECtHR, given that it deals with inhuman, degrading
treatment and torture, which represent prototypical contexts of vulnerability.
As such, an initial exploration under Article 3 is reasonable.

Applying Heri’s procedure to non-Article 3 cases is nonetheless justified
according to our legal expert because Heri’s underlying typology is based on
Timmer (2016) and relates to all articles. Furthermore, vulnerability is now a
concept that is not limited to a single article, and which the ECtHR applies
across articles.

##  Appendix G Pilot study for Annotating Non-Article 3 Cases

In the first round, both annotators independently labeled 20 randomly selected
cases under Article 3. After completing the labeling process, they compared
their annotations with Heri’s labels and engaged in a discussion to address
any discrepancies and clarify their understanding of the vulnerability
concept. In the second round, the annotators independently labeled another 20
randomly selected cases. We calculated the inter-annotator agreement using
Fleiss Kappa to measure the consistency between Heri’s labels and the
annotations provided by our two annotators. The Fleiss Kappa agreement
increased from 0.39 in the first round to 0.64 in the second round, which we
consider to be substantial agreement in a multi-label setting involving seven
vulnerable types and three annotators.

##  Appendix H GLOSS Annotation Tool

The task of explanation annotation was done using the GLOSS annotation tool
Savelka and Ashley (2018). Fig 7 demonstrates the GLOSS annotation interface.

Case fact
---
# cases | 40
Avg. # vulnerable type per case | 1.3
Avg. length per case | 2964 ±1991 words
Rationals from annotator 1
Avg. length case-allegation pair | 630 ± 551 words

Table 5: Statistics for the explanation dataset.

| Count | % | IRLbl
---|---|---|---
Dependency | 254 | 23.74 | 1.41
State Control | 358 | 33.46 | 1.00
Victimisation | 70 | 6.54 | 5.11
Migration | 71 | 6.64 | 5.04
Discrimination | 34 | 3.18 | 10.53
Reproductive Health | 11 | 1.03 | 32.55
Unpopular Views | 33 | 3.08 | 10.85
Non-Vulnerable | 530 | 49.53 | 0.68

Table 6: Count, percentage of total documents (%), imbalance ratio per label
(IRLbl) Charte et al. (2013) for each type of vulnerability.

##  Appendix I Dataset Statistics

The dataset comprises 1070 cases. On average, each case involves 0.78
vulnerable types and 1.54 vulnerable types for non-negative cases. Fig 3
presents the distribution of the number of annotated labels per document. We
report the imbalance characteristics for each label in Tab 6. Fig 4 shows the
difference in frequency of vulnerability annotations between Article 3 and
non-Article 3. Fig 5 illustrates the difference in frequency of each
vulnerability label between all four datasets. Tab 5 shows the statistics for
the explanation dataset.

The hierarchical nature of the dataset is based on the naturally occurring
paragraphs in the judgment texts. On average, each case consists of 71.54
paragraphs ($\sigma=67.54$). The distribution of the number of paragraphs by
vulnerability type is shown in 6(a). The mean token count is 4,765; its
distribution by vulnerability type is depicted in 6(b). The distribution of
the mean token count per paragraph by vulnerability type is shown in 6(c).

##  Appendix J Implementation Details

We use _BERT_ "bert-base-uncased" Devlin et al. (2019), _CaselawBERT_
"casehold/legalbert" Zheng et al. (2021), _LegalBERT_ "nlpaueb/legal-bert-
base-uncased" Chalkidis et al. (2020), and _Longformer_ "allenai/longformer-
base-4096" Beltagy et al. (2020). We finetune pre-trained models from the
Transformers Hub Wolf et al. (2020) on our dataset with a multi-label
classification head, truncating to maximum lengths of 512 and 4096 tokens for
BERT and Longformer models, respectively.
Hyperparameter & Overfitting Measures: For the BERT-based models, we perform a
grid search for hyperparameters across the search space of batch size [4, 8,
16] and learning_rate [5e-6, 1e-5, 5e-5, 1e-4]. We train models with the Adam
optimizer up to 8 epochs. We determine the best hyperparameters on the dev set
and use early stopping based on the dev set macro-F1 score. For the
Hierarchical models, we employ a maximum sentence length of 128 and document
length (number of sentences) of 80. The dropout rate in all layers is 0.1. We
perform a grid search for the hyperparameters across the search space of batch
size [2, 4] and learning_rate [1e-6, 5e-6, 1e-5]. We train models with the
Adam optimizer up to 20 epochs. We determine the best hyperparameters on the
dev set and use early stopping based on the dev set macro-F1 score. We use
_PyTorch_ Paszke et al. (2019) 2.0.1.

Figure 3: Distribution of number of annotated labels per document. Figure 4:
Difference in frequency of vulnerability annotations between Article 3 and
non-Article 3. Figure 5: Difference in frequency of vulnerability type between
“train”, “validation”, “test”, and “challenge” datasets.

##  Appendix K Model Architecture

(a) Box plot of the number of paragraphs per type.

(b) Box plot of the token count per type.

(c) Box plot of the mean number of tokens per paragraph per type.

Figure 6: Box plots on paragraph and token information for each vulnerability
type.

Hierarchical Model: Greedy packing turns the case facts input into $m$ packets
as $x=\\{x_{1},x_{2},\ldots,x_{m}\\}$, where packet
$x_{i}=\\{x_{i1},x_{i2},\ldots,x_{in}\\}$ consists of $n$ tokens. We pass each
packet $x_{i}$ independently into the pre-trained LegalBERT model Chalkidis et
al. (2020) to extract the $h^{cls}_{i}$ representation for each packet. All
packet representations $h=\\{h_{1},h_{2},\ldots h_{m}\\}$, along with
learnable position embeddings, are passed through a transformer encoder to
make them aware of the surrounding context. These context-aware packets
representations are then max pooled to obtain the final representation of the
case facts, which then pass through a classification layer. Given the multi-
label nature of the task, we employ a binary cross-entropy loss over each
vulnerability type label. 2(a) illustrates the detailed architecture of our
hierarchical model, which is inspired by Santosh et al. (2022); Tyss et al.
(2023b).

Concept-aware Hierarchical model: We cast the multi-label task into a binary
classification setup where we pair case facts with each vulnerability type to
predict whether this vulnerability type was raised by court in the case from
which the facts section stems. Note that we transform these binary labels into
a multi-label vector for evaluation, to make a fair comparison with multi-
label models. This is based on the article-aware outcome prediction setting in
Tyss et al. 2023a.

The concept-aware model also takes the case facts as input, which after greedy
packing form $m$ packets $x=\\{x_{1},x_{2},\ldots,x_{m}\\}$ and vulnerability
concept description text $c=\\{c_{1},c_{2},\ldots,c_{k}\\}$ with $k$ packets.
Packet $x_{i}=\\{x_{i1},x_{i2},\ldots,x_{in}\\}$ has $n$ tokens and packet
$c_{i}=\\{c_{i1},c_{i2},\ldots,c_{ip}\\}$ has $p$ tokens.

Similar to the hierarchical model, we use a pre-trained LegalBERT model
Chalkidis et al. (2020) to encode each packet in the case facts and concept
description independently, and extract the $h^{cls}$ representation for each
packet. These are passed through non-pretrained transformer model to obtain
context-aware representations $f$ = $\\{f_{1},f_{2},\ldots,f_{m}$} and
$g=\\{g_{1},g_{2},\ldots g_{k}$} for case facts and concept descriptions,
respectively.

The obtained packet representations of facts and concept description interact
with each other using a multi-head scaled dot product cross attention Vaswani
et al. (2017) similar to the decoder in the transformer layer by treating case
facts packets ($f$) as the queries (Q) and the keys (K) and values (V) come
from the concept description packets ($g$).

$\operatorname{Attention}(Q,K,V)=\operatorname{softmax}(\frac{QK^{\top}}{\sqrt{d_{k}}}\bigg{)}V$
(1)

Thus we obtain a concept-aware representations of the fact description packets
$d=\\{d_{1},d_{2},\ldots,d_{m}\\}$, which are once again passed through non-
pretrained transformer module to enhance them with the surrounding contextual
information, and max-pooled operation to obtain the final concept-aware case
fact representation. A classification layer then predicts the binary label
indicating whether these facts give rise to the given vulnerability type. 2(b)
displays the architecture of the concept-aware hierarchical model in detail.

Figure 7: Screenshot of the GLOSS annotation interface.


11institutetext: University of Pittsburgh, Pittsburgh, USA
11email<EMAIL_ADDRESS>

# Utilizing Natural Language Processing for Automated Assessment of Classroom
Discussion††thanks: Supported by a grant from the Learning Engineering Tools
Competition.

Nhat Tran  Benjamin Pierce  Diane Litman  Richard Correnti  Lindsay Clare
Matsumura

###### Abstract

Rigorous and interactive class discussions that support students to engage in
high-level thinking and reasoning are essential to learning and are a central
component of most teaching interventions. However, formally assessing
discussion quality ‘at scale’ is expensive and infeasible for most
researchers. In this work, we experimented with various modern natural
language processing (NLP) techniques to automatically generate rubric scores
for individual dimensions of classroom text discussion quality. Specifically,
we worked on a dataset of 90 classroom discussion transcripts consisting of
over 18000 turns annotated with fine-grained Analyzing Teaching Moves (ATM)
codes and focused on four Instructional Quality Assessment (IQA) rubrics.
Despite the limited amount of data, our work shows encouraging results in some
of the rubrics while suggesting that there is room for improvement in the
others. We also found that certain NLP approaches work better for certain
rubrics.

###### Keywords:

Classroom discussion Quality assessment NLP

##  1 Introduction and Background

Instructional quality has been of great interest to educational researchers
for several decades. Due to their cost, measures of instructional quality that
can be obtained at-scale remain elusive. Previous work [5] has shown that
providing automated feedback on teachers’ talk moves can lead to positive
instructional changes. We report here on initial attempts to apply Natural
Language Processing (NLP) methods such as pre-trained language models or
sequence labeling with BiLSTM [4] to automatically produce rubric scores for
individual dimensions of classroom discussion quality from transcripts,

Table 1: Data distribution and mean (Avg) of 4 focused IQA rubrics for Teacher ($T$) and Student ($S$) with their relevant ATM codes. An IQA rubric’s distribution is represented as the counts of each score (1 to 4 from left to right) (n=90 discussions). Rubric | Relevant ATM code
---|---
Short Description | Distribution | Avg | Code Label | Count
S1: T connects Ss | [51, 19, 9, 11] | 1.8 | Recap or Synthesize S Ideas | 75
S2: T presses $S$ | [7, 7, 9, 67] | 3.6 | Press | 927
S3: $S$ builds on other’s idea | [65, 6, 8, 11] | 1.6 | Strong Link | 101
S4: $S$ support their claims | [28, 12, 8, 42] | 2.7 | Strong Text-based Evidence | 403
Strong Explanation | 286
- |  | Others | 52687

building upon two established measures that have shown high levels of
reliability and validity in prior learning research – the Instructional
Quality Assessment (IQA) and the Analyzing Teaching Moves (ATM) rubrics [2,
6].

Our corpus consists of 170 videos from 31 English Language Arts classrooms in
a Texas district. 18 teachers taught fourth grade, 13 taught fifth grade, and
on average had 13 years of teaching experience. The student population was
considered low income (61%), with students identifying as: Latinx (73%),
Caucasian (15%), African American (7%), multiracial (4%), and Asian or Pacific
Islander (1%). The videos were manually scored holistically, on a scale from 1
to 4, using the IQA on 11 dimensions for both teacher and student
contributions. They were also scored using more fine-grained talk moves at the
sentence level using the ATM discourse measure. The current work focuses on
only the 90 discussion transcripts that have already been converted to text-
based codes.

As summarized in column 1 of Table 1, our classifiers are trained to predict 4
of the 11 IQA rubrics containing aspirational teacher (T) and student (S)
‘talk moves’ – T Links Student Contributions (score S1), T Presses for
Information (S2), S Link Contributions (S3), S Support Claims with Evidence
(S4). Besides the 5 ATM codes (column 4 in Table 1) related to these 4 IQA
rubrics the rest are labeled as Others. The distributions of IQA scores for
each rubric and of relevant ATM codes are summarized in columns 2 and 5 of
Table 1, respectively. We notice that the frequencies of ATM codes related to
S1 and S3 are very low (less than or approximately 1 per transcript). Below is
an example excerpt with annotated ATM codes from our data:

> Teacher: [The girls get the water and the boys do the herds, right?]Others
> [Where did you get that from the text?]Press
>
> Student: [Other people, mostly women and girls who had to come fill their
> own containers, many kinds of birds, all flap, twittering and cawing. Herds
> of cattle had been brought to good grazing by the young boys who looked
> after them.]Strong Text-Based Evidence

In this paper we present several IQA classifiers, and show that using
predicted ATM codes as features outperforms an end-to-end model. The long-term
goal of our work is to use such classifiers in a tool for automated IQA
assessment so that teachers and coaches can evaluate classroom discussion
quality in real-time.

##  2 Methods

We train different models for IQA assessment to explore tradeoffs between
scoring performance, explainability, and training set. Our baseline is a
neural end-to-end model, as neural models often have high performance and do
not require feature engineering. However, since the IQA score of a specific
rubric can be inferred from the number of times the relevant ATM codes are
used (Section 1), we also develop IQA prediction models using ATM codes as
predictive features. This in turn requires ATM models to predict the relevant
6 ATM codes. Specifically, for each sentence, these models will predict 1 of 6
ATM code labels in column 4 of Table 1. This is a 6-way classification task.

Hierarchical ATM Classification. We hypothesize that it would be easier to
separate Others from the 5 focal ATM codes as they have specific usages. We
perform a 2-step hierarchical classification at sentence level as follows.
Step 1, binary classification, is to classify Other versus 5 focal ATM Codes.
If the code is not Others, step 2 is to perform another 5-way classification
to identify the final label. We train separate BERT-based classifiers for each
step. The input for the classifiers is the current sentence concatenated with
previous sentences in the same turn and sentences from one previous turn.
Because each ATM code except Others can be only from one speaker, either
Teacher or Student, we train two classifiers for the 5-way classification of
Step 2, one classifier used to predict teacher codes (Recap or Synthesize S
Ideas and Press) and one classifier specialized in student codes (Strong Link,
Strong Text-Based Evidence and Strong Explanation). Depending on the speaker,
only one of them is called for Step 2.

ATM Sequence Labeling. A classroom discussion can be considered as a sequence
of sentences. This approach assigns a label (1 out of the 6 ATM codes) to each
sentence in a conversation. Unlike the Hierarchical Classification approach
that predicts the label of each sentence independently, in this approach, the
label of a given sentence is more dependent on the labels of nearby sentences.
We use BERT-BiLSTM-CRF as our sequence labeling model. BiLSTM-CRF has been
widely used for sequence labeling tasks [4] and BERT [3] provides a powerful
tool for sentence representation that can work well with that architecture.

Additional Techniques. During ATM classification, since Others constitutes
more than 90% of the total labels, we downsample the training data to reduce
the imbalance. For IQA classification, annotators tend to group consecutive
sentences sharing the same functionality in one turn as one ATM code (e.g.,
one Strong Texted-Based Evidence code is used for two sentences in the excerpt
in Section 1), but our prediction is done on sentence level. Merging adjacent
ATM predictions that are the same into one code in the inference phase helps
preserve this nature. Also, since the range of the IQA scores is very small (1
to 4), translating the absolute counting of ATM codes to IQA scores (see
Section 1) can drastically shift the IQA scores due to misclassification of
the ATM codes. To alleviate this sensitivity, we build separate linear
regression models to estimate each IQA score from the counting of relevant ATM
codes, then use the nearest integers as the IQA scores.

Table 2: ATM Codes 6-way Classification Results ($\mathrm{F_{1}}$ scores over 5-fold cross-validation with standard deviations in parentheses). Method | Step 1 | 6-way
---|---|---
Non-hierarchical Classification (All Data) | - | 0.29 (0.04)
/w 60% downsampling | - | 0.49 (0.03)
Hierarchical Classification (All Data) | 0.56 | 0.41 (0.04)
/w 60% downsampling | 0.72 | 0.65 (0.02)
Sequence Labeling (All Data) | - | 0.45 (0.03)
/w 60% downsampling | - | 0.62 (0.01)
Hierarchical with Oracle for Step 1 (All Data) | 1 | 0.57 (0.02)
/w 60% downsampling | 1 | 0.68 (0.04)

##  3 Results

ATM Code Prediction results (macro average $\mathrm{F_{1}}$ scores) are shown
in Table 2. The Non-hierarchical baseline is a BERT-based 6-way classifier
given the same input as our hierarchical approach. Both Hierarchical
Classification and Sequence Labeling outperform the Non-hierarchical baseline,
whether using all data or downsampled data for training. The numbers also show
that downsampling the proportion of the most popular class (Others) to 60%
increases the performance of all models111We tried different ratios and 60%
provides the best results.. For the 2-step Hierarchical Classification
approach, it improves the performances of both step 1 and the final 6-way
classification. Additionally, the Sequence Labeling and Hierarchical
approaches perform similarly. Although Sequence Labeling has a slightly higher
score when all training data is used (0.45 vs. 0.41, $\rho=0.039$), it is
inferior to Hierarchical Classification with 60% downsampling (0.62 vs. 0.65,
$\rho=0.046$). Using a perfect Oracle model with 100% accuracy for Step 1
(Others versus 5 ATM codes) does not lead to a large gain in the 6-way
classification results compared to our fully automated Hierarchical approach
in the downsampled version (0.68 vs 0.65). We thus use the non-oracle models
built with 60% downsampling rate for the inference of the classroom discussion
quality (IQA scores) below.

IQA Score Prediction is performed based on the models for ATM codes
prediction. The Quadratic Kappa (QK) scores for the estimations of the four
rubrics of classroom discussion quality are reported in Table 3. The baseline
for each rubric is an end-to-end Longformer model [1] which directly predicts
the IQA scores given the raw text transcripts using a linear layer on top of
the hidden representation of [CLS], ignoring the ATM codes. The results show
that all variations of Hierarchical Classification and Sequence Labeling
outperform the baselines, which emphasizes the importance of utilizing ATM
codes to infer IQA scores. Besides increasing performance, the ATM-based
models also increase model interpretability, useful for generating formative
feedback in the future.

Table 3: IQA Scores Estimation Results in Quadratic Kappa (QK) averaged over 5-fold cross-validation, inferred from Absolute Counting (A.Count) and Linear Regression (Regression) after ATM prediction. Bold numbers are the best results for each rubric. Rubric | Baseline | Hierarchical | Sequence
---|---|---|---
A.Count | Regression | A.Count | Regression
S1: Teacher connects Students | 0.34 | 0.43 | 0.54 | 0.50 | 0.55
/w merged codes | 0.48 | 0.55 | 0.52 | 0.57
S2: Teacher presses Student | 0.35 | 0.60 | 0.65 | 0.55 | 0.62
/w merged codes | 0.64 | 0.68 | 0.57 | 0.63
S3: Student builds on each other | 0.30 | 0.42 | 0.51 | 0.47 | 0.51
/w merged codes | 0.49 | 0.54 | 0.50 | 0.53
S4: Student support their claims | 0.36 | 0.60 | 0.65 | 0.57 | 0.61
/w merged codes | 0.65 | 0.70 | 0.61 | 0.63

One notable observation is that using regression to estimate the IQA scores is
always better than using absolute counting. This supports our assumption that
regression will alleviate the sensitivity of miscounting and provides a
smoother transition from the number of times ATM codes appear to the actual
IQA scores. Using regression, the highest gain in QK scores for Hierarchical
Classification (0.09) and Sequence Labeling (0.07) are from S3 and S2,
respectively.

Merging consecutive same ATM codes into one also improves the performance of
IQA score estimation as expected. For the same approach, the increases from
this technique are mostly larger when absolute counting is used. The ATM
classification was on sentence level and absolute counting is more sensitive
to over-counting, so the merging technique is more effective for this method.

While Sequence Labeling yields the best S1 results, the best results for the
other rubrics come from Hierarchical Classification. Our reasoning is that for
S1, the relation to adjacent sentences plays a more important role to identify
the relevant ATM code as there should be multiple students speaking out their
ideas before the teacher can connect/synthesize them. Thus, Sequence Labeling,
which focuses more on dependencies between sentences, performs better. This
suggests that certain approaches are more suitable for certain rubrics.

Figure 1: IQA Score Estimation Results (QK) in relation to the amount of
training data.

Finally, Figure 1 demonstrates the performance of our best models (with
regression and merged codes) over training size. While the baselines do not
improve much after a certain size of training data, the lines of Hierarchical
Classification and Sequence Labeling maintain upward trends, suggesting that
these models will continue to benefit from more data as we complete our video
transcription. Even using only 90 discussions, the QK results show that ATM-
based model reliability is already substantial for S2 and S4, and moderate for
S1 and S3, even though there were infrequent instances of relevant codes in
the corpus.

##  4 Conclusion and Future Directions

We experimented with NLP approaches to automatically assess discussion quality
using the IQA, and to deal with imbalanced ATM data and imperfect ATM code
prediction. Our results show that IQA models using either Hierarchical
Classification or Sequence Labeling to first predict ATM codes outperform
baseline end-to-end IQA models, while each of the ATM-based IQA models
performs better than the other in certain IQA rubrics. Once the full corpus is
available, we will generate a validity argument for whether automated scoring
replicates known associations in the corpus, and incorporate the demographic
information into our analyses. We will also utilize ATM codes beyond the 5
relevant to the focused rubrics to add more context for ATM prediction. To
mitigate the limited size of even the full corpus, we will explore whether
techniques such as transfer learning that can take advantage of classroom
discussion data from math [8] or high school [7] that are now being made
available to the community.

## References

  * [1] Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document transformer. arXiv:2004.05150 (2020)
  * [2] Correnti, R., Matsumura, L.C., Walsh, M., Zook-Howell, D., Bickel, D.D., Yu, B.: Effects of online content-focused coaching on discussion quality and reading achievement: Building theory for how coaching develops teachers’ adaptive expertise. Reading Research Quarterly 56(3), 519–558 (2021)
  * [3] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Conf. of the North American Chapter of ACL: Human Language Technologies. pp. 4171–4186. Minneapolis (2019)
  * [4] Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. CoRR abs/1508.01991 (2015), http://arxiv.org/abs/1508.01991
  * [5] Jacobs, J., Scornavacco, K., Harty, C., Suresh, A., Lai, V., Sumner, T.: Promoting rich discussions in mathematics classrooms: Using personalized, automated feedback to support reflection and instructional change. Teaching and Teacher Education 112, 103631 (2022)
  * [6] Matsumura, L.C., Garnier, H.E., Slater, S.C., Boston, M.D.: Toward measuring instructional interactions “at-scale”. Educational Assessment 13(4), 267–300 (2008)
  * [7] Olshefski, C., Lugini, L., Singh, R., Litman, D., Godley, A.: The Discussion Tracker corpus of collaborative argumentation. In: Language Resources & Evaluation (2020)
  * [8] Suresh, A., Jacobs, J., Clevenger, C., Lai, V., Tan, C., Martin, J.H., Sumner, T.: Using AI to promote equitable classroom discussions: The TalkMoves application. In: International Conf. on Artificial Intelligence in Education. pp. 344–348 (2021)