# One Size Does Not Fit All: Customizing Open-Domain Procedures

Yash Kumar Lal13, Li Zhang23, Faeze Brahman3,
Bodhisattwa Prasad Majumder3, Peter Clark3, Niket Tandon3

1 Stony Brook University, 2 University of Pennsylvania
3 Allen Institute for Artificial Intelligence
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

How-to procedures, such as how to plant a garden, are ubiquitous. But one size
does not fit all - humans often need to customize these procedural plans
according to their specific needs, e.g., planting a garden without pesticides.
While large language models (LLMs) can fluently generate generic procedures,
we present the first study on how well LLMs can customize open-domain
procedures. We introduce CustomPlans, a probe dataset of customization hints
that encodes diverse user needs for open-domain How-to procedures. Using LLMs
as CustomizationAgent and ExecutionAgent in different settings, we establish
their abilities to perform open-domain procedure customization. Human
evaluation shows that using these agents in a Sequential setting is the best,
but they are good enough only ~51% of the time. Error analysis shows that LLMs
do not sufficiently address user customization needs in their generated
procedures.

One Size Does Not Fit All: Customizing Open-Domain Procedures

Yash Kumar Lal13, Li Zhang23, Faeze Brahman3, Bodhisattwa Prasad Majumder3,
Peter Clark3, Niket Tandon3 1 Stony Brook University, 2 University of
Pennsylvania 3 Allen Institute for Artificial Intelligence
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

How-to procedures (or procedural plans) serve as building blocks in decision-
making processes, even making up for a large fraction of search engine
queries111https://observer.com/2018/12/google-most-searched-2018-report-top-
questions/. WikiHow contains a large collection of high-quality, expert-
authored How-to procedures about diverse topics. There are a myriad of ways to
tailor these generic How-to procedures to suit users by using different
customization hints. For example, for a How-to procedure like “How to stay
healthy”, users can prefer yoga, or a Mediterranean lifestyle, or be dealing
with an injury, or more. It is infeasible to manually compile innumerable
customizations of a procedural plan, and thus it is important to build an
automated system to do this.

Customization hints implicitly encode multiple requirements and are subjective
— for example, a Mediterranean lifestyle can involve a diet of whole grains,
using healthy fats like olive oil, the occasional glass of wine, walking, and
more. These implicit customizations are best expressed in natural language
since it is more practical in a real-world, open-ended setting. Thus, we need
an automated method that can understand these implicitly expressed needs and
make consistent revisions to customize how-to procedures.

Prior work has focused on building supervised customization models for limited
domains such as shopping, news and video games Rashid et al. (2002); Togelius
et al. (2007). Majumder et al. (2019) focus on just customizing cooking
recipes while Antognini et al. (2023) are limited to certain aspects of
customization. An open-domain setting with natural language customization
hints requires a supervision-free model such as LLMs.

Figure 1: There are various ways to customize a procedure for a goal. Each hint leads to a different customized procedure. For each customization, different aspects of the hint need to be considered.  Goal |  Uncustomized Procedure |  Customization Hint |  Customization Metadata
---|---|---|---
Knit a hat |  \- Get knitting needles, yarn …
\- Learn basic stitches … |  I want to take up
knitting as a hobby |  Constraint: Refinement
Make banana
smoothie |  \- Look up recipe …
\- Go to store … |  It is 4am in the
morning |  Constraint: Prerequisite
Stay healthy |  \- Eat a balanced diet …
\- Exercise regularly … |  I like the
Mediterranean
style of living |  Constraint: Preference
Bake bread |  \- Dissolve yeast in water …
\- Stir in sugar, salt, oil … |  I have been baking
for years |  Expertise: Expert
Table 1: Examples from our CustomPlans dataset. Each data point contains a
goal, the uncustomized procedure and a customization hint.

This paper presents the first study of using LLMs to customize open-domain
How-to procedures. First, we compile a new dataset, CustomPlans, that contains
diverse procedures and customization hints. We manually collect a small, high-
quality dataset with diverse procedures and customization hints and also re-
formulate two existing WikiHow based datasets — Choice-75 Hou et al. (2023)
and CoPlan Brahman et al. (2023) — that contain user scenarios which can serve
as customization hints.

We then evaluate how well LLMs can customize procedures. We conduct a set of
experiments inspired by prior work on building supervised models that can edit
procedures according to constraints. Through human evaluation, we establish
that using first customizing the original procedure for a goal and then fixing
issues related to execution is the best method to generate procedures that
satisfy customization hints. Even so, we find that LLMs fail to sufficiently
address the customization needs, often generating procedures that miss
important steps.

In summary, our contributions are:

  * •

CustomPlans, a probe dataset to study how well LLMs can customize open-domain
procedures when provided hints from users.

  * •

Using a multi-agent framework, LLMs are able to generate executable customized
procedures 51% of the time, leaving significant room for improvement.

  * •

Models are unable to fully address the implicit requirements of user
customization hints in their generated procedures.

##  2 The CustomPlans dataset

Prior work on editing and customizing procedures has focused on specific
domains. So, data for the task of procedure customization is limited. To
alleviate this problem, we re-purpose two related datasets and collect more
high-quality data from WikiHow to create the CustomPlans probe dataset.

###  2.1 CustomPlansReal

Each WikiHow article consists of a goal (indicated in the title starting with
“How to”) and ordered, richly described steps that lead to that goal. It
already contains articles that include multiple methods for a task.
Additionally, search queries can be used to retrieve and select articles
broadly related to the same goal. However, user interaction signals like fine-
grained querying related to specific versions of procedures are not available
publicly. Such signals can be used to improve user experiences. Therefore, we
create a new probing set that contains a procedural goal and an individual
user’s relevant context.

Figure 2: Types of customization hints present in CustomPlans, enabling
analysis of diverse customization hints. Prerequisites are usually expressed
explicitly, while preferences and refinements may be expressed implicitly in
the hint. Expertise usually needs to be inferred from the hint, except in
hints collected experts. Some hints also encode both user expertise as well as
constraints in their scenarios.

To compile this data, we start by identifying 9 diverse domains. For each
domain, we employ two methods to curate WikiHow articles. 1) Search Results:
We devise relevant, popular and pragmatic goals which have relevant
instructions present in WikiHow. We present each goal to users of relevant
persona and ask them to provide their feedback according to their personal
preferences. For example, for the broad goal of staying healthy, a collected
user persona was that of a teenager looking to build long-term healthy habits.
2) Multiple Methods: For the same set of broad goals, we filter out WikiHow
articles that contain multiple methods to achieve the same goal. Given the
broad goal, users are required to provide situational feedback corresponding
to each method. Using this approach, we collect 106 customized goals
(corresponding to user feedback) over 9 domains and build CustomPlansReal,
which contains customization hints collected from humans.

###  2.2 CustomPlansSimulated

To study decision branching in scripts, Hou et al. (2023) created Choice-75, a
benchmark of 565 data points which requires selecting the next step in a
procedure given descriptive scenarios. These scenarios are valid only at a
step level. However, they can also be incorporated into the full procedure if
treated as constraints. We treat these scenarios as customization hints.

CoPlan is a dataset of machine-generated, human-verified scripts. We use the
test set of 861 data points for the counterfactual plan revision task for our
purposes. Brahman et al. (2023) collect goals from diverse topics and prompt
text-curie-001 to generate a set of ordered steps as a plan to achieve that
goal. text-curie-001 is also used, in a few-shot manner, to generate
conditions that can alter these plans. Human verification of these components
leads to the creation of a test set that contains a goal, an uncustomized
procedure and a relevant condition. These conditions describe prerequisites to
be fulfilled that require the original procedure to be customized in a
specific manner. We treat these conditions as customization hints. The
combination of data from these two sources is referred to as
CustomPlansSimulated since it contains customization hints generated
automatically.

###  2.3 Statistics: CustomPlans dataset

We randomly select 100 procedures and their corresponding customization hints
from CustomPlansSimulated and add them to CustomPlansReal to form the
CustomPlans evaluation set. Overall, CustomPlans contains 206 data points,
each of which is made up of a goal, a list of steps to achieve that goal and a
customization hint from a user according to which the procedure should be
modified. It contains 106 unique goals and 203 unique customization hints.
Table 1 shows examples from the dataset.

Figure 3: Given a goal $G$, the uncustomized procedure $P$ to achieve that
goal and a customization hint $H$ from a user, models are required to generate
a modified procedure $P^{\prime}$ to achieve the same goal by taking into
account the customization hint.

(a) Uniform

(b) Sequential

(c) Parallel

Figure 4: Different settings of CustomizationAgent and ExecutionAgent for
procedure customization

We also store relevant metadata for each user persona providing the annotation
such as their level of expertise and their constraints like dietary
preferences or availability of tools. 1.2% of these customized procedures can
only be performed by domain experts, 7.6% need an intermediate level of
expertise, and the rest are beginner-friendly. The user comments contain
implicit or explicit constraints — 68% contain hard prerequisites, 30% reflect
some user preference and the rest mention a target refinement to be achieved.
The types of customization hints, as well as corresponding examples, in this
dataset are presented in Figure 2.

##  3 Experiment Details

###  3.1 Task

The task, illustrated in Figure 3, involves taking as input the goal $G$ of
the procedure $P$ as well as a user customization hint $H$. The output should
be a procedure to fulfill the goal while also taking into account the user
scenario described in the hint $P^{\prime}$. To tailor procedures according to
customization hints, one needs to make implicit inferences out of the hints,
identify the steps that require changes, and finally consistently apply
changes to the different steps. We introduce a multi-agent prompting framework
to establish the abilities of state-of-the-art LLMs to customize procedures
according to user feedback using the CustomPlans dataset. Through our
experiments, we aim to understand how well models can modify generic
procedures to incorporate aspects provided in customization hints.

###  3.2 Models

Recently, model-based agents have been used to perform different types of
reasoning in service of achieving a larger goal Yoran et al. (2023). To that
end, we disentangle the task of generating customized procedures, delegating
customization to CustomizationAgent and ensuring the executability of the
generated procedure to ExecutionAgent. However, it is unclear how these agents
can interact with each other to best generate customized procedures.

Each agent produces a bag of semi-structured edits $E$ that indicate the
operation to be performed (they are only allowed to perform the insert or
replace operation), the step in the original procedure $P$ to perform it
on/after, and the new step to replace it with, or insert after it. We then
apply these edits deterministically on $P$ to obtain $P^{\prime}$ i.e., we
find the step number in $P$ that corresponds to a suggested edit and
insert/replace it with the edited text. We experiment with three settings for
the defined agents: Uniform, Sequential, and Parallel.

Uniform \- We first define a single agent that performs the task of suggesting
edits about customization as well as executability on $P$ to generate $P_{c}$.
This agent is required to understand how to perform both customization towards
a hint $H$ as well as execution to achieve the goal $G$.

Sequential \- In this setting, we first obtain a set of edits $E_{c}$ from
CustomizationAgent, and apply $E_{c}$ to obtain $P_{c}$. Next, we use $P_{c}$
to extract another set of edits $E_{e}$ from ExecutionAgent. Finally, we
deterministically apply $E_{e}$ on $P_{c}$ to obtain the resulting procedure
$P^{\prime}$. Here, the agents are used in a sequential order, first
suggesting edits to meet customization requirements and next to address any
execution-related issues that may arise from those edits.

Parallel \- Both CustomizationAgent and ExecutionAgent propose their set of
edits, $E_{c}$ and $E_{e}$, upon $P$. However, this gives rise to conflicting
edits according to different objectives, both of which are important to
generate $P^{\prime}$. To address this, we use ResolutionAgent, which takes as
input two bags of edits, $E_{c}$ and $E_{e}$, and produces a merged set of
edits $E$. Finally, applying $E$ on the original procedure $P$ results in
$P^{\prime}$.

Method | Customized | Executable | FullyCorrect
---|---|---|---
Sequential | 60.68% | 72.33% | 51.94%
Uniform | 54.85% | 71.36% | 47.09%
Parallel | 53.88% | 70.87% | 45.63%
Table 2: Customizability, executability and overall correct (strictest
measure) of procedures generated by different approaches as judged by majority
of human evaluators.

###  3.3 Evaluation

For open-ended text generation tasks like procedure customization, the absence
of an automatic evaluation that correlates well with human judgments is a
major challenge Chen et al. (2019); Ma et al. (2019); Caglayan et al. (2020);
Howcroft et al. (2020); Lal et al. (2021).

Figure 5: MTurk interface presented to crowdsource workers to judge model
generated procedures.

We conduct a human evaluation with a standardized interface to compare
different models. We want to measure whether a model produces procedures that
meet the requirements of the customization hint, as well as whether the
generated procedures can be executed to achieve the goal. To this end, we pose
questions about whether a procedure can be performed as is to accomplish the
given goal (Executable), and whether it satisfies a presented customization
constraint (Customized). This serves as the human evaluation interface for our
task. A sample HIT can be seen in Figure 5.

For each goal, customization hint, and a model’s generated procedure, we ask 3
distinct annotators to provide judgments about customizability and
executability of that procedure. A procedure can be complete in an aspect,
missing steps, have extra steps, have underspecified steps, or have incorrect
steps. We take majority vote of annotator judgments for the Customized and
Executable criteria. If a plan is judged to be both Customized and Executable
by a majority of annotators, we consider it to be FullyCorrect.

##  4 Results

We report majority statistics for customizability, executability and
correctness of different models in Table 2.

Using CustomizationAgent and ExecutionAgent in Sequential order is the best at
producing customized procedures $P^{\prime}$. It performs better than using
just one agent in Uniform, which, in turn, is better than using both agents in
Parallel setting. We note that execution accuracy for all models are similar.
However, performance differs based on how well a model can address implicit
and explicit customization needs.

Procedures are not sufficiently customized. All methods suffer from the
problem of missing steps, indicating that none of them adequately address the
requirements stated implicitly or explicitly in the customization hint for the
procedure.

Procedures are not direct. For the executability aspect, each method suffers
the most from generating unnecessary steps, which often hinder achieving the
goal of the procedure $G$. Furthermore, procedures generated by Parallel also
suffer from missing steps. We hypothesize that this problems arises because
ResolutionAgent is unable to correctly merge bags of edits from
CustomizationAgent and ExecutionAgent.

##  5 Related Work

Over the past few years, as information about users has been readily
available, user-facing technology has become increasingly customized.
Customization has been widely studied in the context of such technology, like
search engines Rashid et al. (2002), chatbots Majumder et al. (2020) and game
content Shaker et al. (2010); Togelius et al. (2007). There is some work on
customized procedural content generation Togelius et al. (2011); Yannakakis
and Togelius (2011). However, they focus on video game content rather than
everyday procedures related to pragmatic goals. Previous research has shown
the value of customization in various settings. Kapusta et al. (2019) present
a method to customize collaborative plans for robot-assisted dressing.
Building customized care plans is crucial for patients with severe illnesses
Lin et al. (2017); Anbari et al. (2020).

Procedural text understanding addresses the task of tracking entity states
throughout the text Bosselut et al. (2018); Henaff et al. (2017). Generating
such texts Aouladomar and Saint-Dizier (2005) involves different types of
understanding such as temporal reasoning and entity linking. Mori et al.
(2014) generated a procedural text from a meaning representation taking
intermediate states into account. ChattyChef Le et al. (2023) uses the
conversational setting to generate cooking instructions and iteratively refine
its step ordering. To study decision branching in procedural text, Hou et al.
(2023) generated user scenarios and choices for various steps in a procedure
and presented Choice-75. For the task of counterfactual planning, CoPlan
Brahman et al. (2023) collects conditions and combines them with a revised
list of steps. Majumder et al. (2019) use historical user preferences to
generate customized recipes. But they only focus on one domain (here, cooking)
and only on one dimension to model customization. LLMs have been shown to
generate procedural text well but it is unclear how they can be customized for
diverse user preferences.

Customized language models are designed and trained to capture language
patters specific to individual users. Such models are often better than using
more general models for human-level NLP tasks Lynn et al. (2017). King and
Cook (2020) examined methods for creating customized language models using
interpolating, fine-tuning and priming. Similarly, Welch et al. (2022) present
another approach for fine-tuning and interpolation to customize LMs. In the
context of LLMs, role-based prompting is a recent trend. Pseudo Dialog
Prompting Han et al. (2022) is a method to build prompts for LLMs so that
chatbots mimic fictional characters, while Vincent et al. (2023) release a
dataset of character annotations to induce personas in LLM output. However, it
is still unclear how to encapsulate more fine-grained aspects of
customization.

##  6 Conclusion

This paper studies the capabilities of current LLMs to customize open-domain
How-to procedures. We introduce a probe dataset, CustomPlans, for this task.
Using LLMs as CustomizationAgent and ExecutionAgent in different settings, we
establish their abilities to perform open-domain procedure customization.
Human evaluation shows that using these agents in a Sequential setting is the
best, but they are good enough only ~51% of the time. Error analysis shows
that LLMs do not sufficiently address user customization needs in their
generated procedures. Further, the generated plans contain extra steps because
of which they cannot be often executed.

## References

  * Anbari et al. (2020) Allison Brandt Anbari, Pamela L. Ostby, and Pamela Ginex. 2020.  Breast cancer–related lymphedema: Personalized plans of care to guide survivorship.  _Current Breast Cancer Reports_ , 12:237 – 243.
  * Antognini et al. (2023) Diego Antognini, Shuyang Li, Boi Faltings, and Julian McAuley. 2023.  Assistive recipe editing through critiquing.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 375–384, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Aouladomar and Saint-Dizier (2005) Farida Aouladomar and Patrick Saint-Dizier. 2005.  Towards generating procedural texts: An exploration of their rhetorical and argumentative structure.  In _Proceedings of the Tenth European Workshop on Natural Language Generation (ENLG-05)_ , Aberdeen, Scotland. Association for Computational Linguistics.
  * Bosselut et al. (2018) Antoine Bosselut, Omer Levy, Ari Holtzman, Corin Ennis, Dieter Fox, and Yejin Choi. 2018.  Simulating action dynamics with neural process networks.  _ICLR_.
  * Brahman et al. (2023) Faeze Brahman, Chandra Bhagavatula, Valentina Pyatkin, Jena D. Hwang, Xiang Lorraine Li, Hirona J. Arai, Soumya Sanyal, Keisuke Sakaguchi, Xiang Ren, and Yejin Choi. 2023.  Plasma: Making small language models better procedural knowledge models for (counterfactual) planning.
  * Caglayan et al. (2020) Ozan Caglayan, Pranava Madhyastha, and Lucia Specia. 2020.  Curious case of language generation evaluation metrics: A cautionary tale.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2322–2328, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Chen et al. (2019) Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  Evaluating question answering evaluation.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 119–124, Hong Kong, China. Association for Computational Linguistics.
  * Han et al. (2022) Seungju Han, Beomsu Kim, Jin Yong Yoo, Seokjun Seo, Sangbum Kim, Enkhbayar Erdenee, and Buru Chang. 2022.  Meet your favorite character: Open-domain chatbot mimicking fictional characters with only a few utterances.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5114–5132, Seattle, United States. Association for Computational Linguistics.
  * Henaff et al. (2017) Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2017.  Tracking the world state with recurrent entity networks.  _ICLR_.
  * Hou et al. (2023) Zhaoyi Joey Hou, Li Zhang, and Chris Callison-Burch. 2023.  Choice-75: A dataset on decision branching in script learning.  _arXiv preprint arXiv:2309.11737_.
  * Howcroft et al. (2020) David M. Howcroft, Anya Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A. Hasan, Saad Mahamood, Simon Mille, Emiel van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020.  Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 169–182, Dublin, Ireland. Association for Computational Linguistics.
  * Kapusta et al. (2019) Ariel Kapusta, Zackory M. Erickson, Henry M. Clever, Wenhao Yu, C. Karen Liu, Greg Turk, and Charles C. Kemp. 2019.  Personalized collaborative plans for robot-assisted dressing via optimization and simulation.  _Autonomous Robots_ , 43:2183 – 2207.
  * King and Cook (2020) Milton King and Paul Cook. 2020.  Evaluating approaches to personalizing language models.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 2461–2469, Marseille, France. European Language Resources Association.
  * Lal et al. (2021) Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2021.  TellMeWhy: A dataset for answering why-questions in narratives.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 596–610, Online. Association for Computational Linguistics.
  * Le et al. (2023) Duong Minh Le, Ruohao Guo, Wei Xu, and Alan Ritter. 2023.  Improved instruction ordering in recipe-grounded conversation.
  * Lin et al. (2017) Hung-Hsin Lin, Nien Wei, T-Y. Chou, Chun-Chi Lin, Yuan-Tsu Lan, Shin-Ching Chang, Huann-Sheng Wang, Shung-Haur Yang, Wei-Shone Chen, Tzu chen Lin, Jen-Kou Lin, and Jeng-Kai Jiang. 2017.  Building personalized treatment plans for early-stage colorectal cancer patients.  _Oncotarget_ , 8:13805 – 13817.
  * Lynn et al. (2017) Veronica Lynn, Youngseo Son, Vivek Kulkarni, Niranjan Balasubramanian, and H. Andrew Schwartz. 2017.  Human centered NLP with user-factor adaptation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1146–1155, Copenhagen, Denmark. Association for Computational Linguistics.
  * Ma et al. (2019) Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019.  Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 62–90, Florence, Italy. Association for Computational Linguistics.
  * Majumder et al. (2020) Bodhisattwa Prasad Majumder, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Julian McAuley. 2020.  Like hiking? you probably enjoy nature: Persona-grounded dialog with commonsense expansions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9194–9206, Online. Association for Computational Linguistics.
  * Majumder et al. (2019) Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. 2019.  Generating personalized recipes from historical user preferences.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5976–5982, Hong Kong, China. Association for Computational Linguistics.
  * Mori et al. (2014) Shinsuke Mori, Hirokuni Maeta, Tetsuro Sasada, Koichiro Yoshino, Atsushi Hashimoto, Takuya Funatomi, and Yoko Yamakata. 2014.  FlowGraph2Text: Automatic sentence skeleton compilation for procedural text generation.  In _Proceedings of the 8th International Natural Language Generation Conference (INLG)_ , pages 118–122, Philadelphia, Pennsylvania, U.S.A. Association for Computational Linguistics.
  * Rashid et al. (2002) Al Mamunur Rashid, Istvan Albert, Dan Cosley, Shyong K. Lam, Sean M. McNee, Joseph A. Konstan, and John Riedl. 2002.  Getting to know you: Learning new user preferences in recommender systems.  In _Proceedings of the 7th International Conference on Intelligent User Interfaces_ , IUI ’02, page 127–134, New York, NY, USA. Association for Computing Machinery.
  * Shaker et al. (2010) Noor Shaker, Georgios Yannakakis, and Julian Togelius. 2010.  Towards automatic personalized content generation for platform games.  _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , 6(1):63–68.
  * Togelius et al. (2007) Julian Togelius, Renzo De Nardi, and Simon M. Lucas. 2007.  Towards automatic personalised content creation for racing games.  In _2007 IEEE Symposium on Computational Intelligence and Games_ , pages 252–259.
  * Togelius et al. (2011) Julian Togelius, Georgios N. Yannakakis, Kenneth O. Stanley, and Cameron Browne. 2011.  Search-based procedural content generation: A taxonomy and survey.  _IEEE Transactions on Computational Intelligence and AI in Games_ , 3(3):172–186.
  * Vincent et al. (2023) Sebastian Vincent, Rowanne Sumner, Alice Dowek, Charlotte Blundell, Emily Preston, Chris Bayliss, Chris Oakley, and Carolina Scarton. 2023.  Personalised language modelling of screen characters using rich metadata annotations.  _arXiv preprint arXiv:2303.16618_.
  * Welch et al. (2022) Charles Welch, Chenxi Gu, Jonathan K. Kummerfeld, Veronica Perez-Rosas, and Rada Mihalcea. 2022.  Leveraging similar users for personalized language modeling with limited data.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1742–1752, Dublin, Ireland. Association for Computational Linguistics.
  * Yannakakis and Togelius (2011) Georgios N. Yannakakis and Julian Togelius. 2011.  Experience-driven procedural content generation.  _IEEE Transactions on Affective Computing_ , 2(3):147–161.
  * Yoran et al. (2023) Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant. 2023.  Answering questions by meta-reasoning over multiple chains of thought.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_.

##  Appendix A Mechanical Turk tasks

###  A.1 Instructions

We present the instructions given to annotators for both the tasks in Figure
6. Annotators were given clear direction for the task, as well as provided
examples for various cases of desirable and undesirable characteristics of a
procedure. We restricted the task to master turkers. The master turkers are
paid 0.30$ per HIT, which translates to 17$/hr according to average time
needed for completion.

Figure 6: Instructions for MTurk tasks


# Transparency, Governance and Regulation of Algorithmic Tools Deployed in the
Criminal Justice System: a UK Case Study

Miri Zilka  0000-0001-9640-8139 University of CambridgeCambridgeUK The Alan
Turing InstituteLondonUK<EMAIL_ADDRESS>,  Holli Sargeant
0000-0003-3482-7789 University of CambridgeCambridgeUK<EMAIL_ADDRESS>and
Adrian Weller  0000-0003-1915-7158 University of CambridgeCambridgeUK The Alan
Turing InstituteLondonUK<EMAIL_ADDRESS>
(2022)

###### Abstract.

We present a survey of tools used in the criminal justice system in the UK in
three categories: data infrastructure, data analysis, and risk prediction.
Many tools are currently in deployment, offering potential benefits, including
improved efficiency and consistency. However, there are also important
concerns. Transparent information about these tools, their purpose, how they
are used, and by whom is difficult to obtain. Even when information is
available, it is often insufficient to enable a satisfactory evaluation. More
work is needed to establish governance mechanisms to ensure that tools are
deployed in a transparent, safe and ethical way. We call for more engagement
with stakeholders and greater documentation of the intended goal of a tool,
how it will achieve this goal compared to other options, and how it will be
monitored in deployment. We highlight additional points to consider when
evaluating the trustworthiness of deployed tools and make concrete proposals
for policy.

Criminal Justice, Trustworthy AI, Policy, Governance and regulation,
Algorithms in deployment

††journalyear: 2022††copyright: rightsretained††conference: Proceedings of the
2022 AAAI/ACM Conference on AI, Ethics, and Society; August 1–3, 2022; Oxford,
United Kingdom††booktitle: Proceedings of the 2022 AAAI/ACM Conference on AI,
Ethics, and Society (AIES’22), August 1–3, 2022, Oxford, United Kingdom††doi:
10.1145/3514094.3534200††isbn: 978-1-4503-9247-1/22/08††submissionid:
aies227††ccs: Social and professional topics††ccs: Social and professional
topics Governmental regulations††ccs: Social and professional topics Race and
ethnicity††ccs: Security and privacy Privacy protections††ccs: Computing
methodologies Artificial intelligence††ccs: Computing methodologies Machine
learning algorithms††ccs: Computing methodologies Machine learning approaches

##  1\. Introduction

Law enforcement faces intense scrutiny with important questions being raised
about the extent to which algorithmic tools might explain instances of over-
policing or other negative community interactions with the police (Sankin et
al., 2021). More and more algorithms are in use within criminal justice
systems around the world, deployed by law enforcement and often designed by
the private sector (Ensign et al., 2018; Mamalian and La Vigne, 1999; Weisburd
et al., 2008; Babuta and Oswald, 2020; Jansen, 2018; Sprick, 2019). There is a
potential for promising benefits for law enforcement, including support for
improved decision-making, efficiency, and consistency. However, there are many
risks to their deployment. These systems have super-human abilities to analyse
vast amounts of data which could be a benefit; however, they have significant
drawbacks, such as a lack of contextual understanding or common-sense
reasoning, leading to concerns raised in this paper. While prior work has
mainly focused on the analysis of single tools, this paper aims to consider
the deployment of these tools within the bigger picture of the criminal
justice system. As a case study, we focus on algorithmic tools used in the
criminal justice system in the UK. As we address key issues based on use cases
in the UK, we reveal principles and considerations that will be relevant for
law enforcement agencies across jurisdictions.

Academics have been bringing examples of algorithmic policing to light in
recent years from the US (Eubanks, 2018; Pasquale, 2019; Selbst, 2018;
ProPublica, 2016) to Australia (Santow, 2021; Australian Human Rights
Commission, 2021). Recent court cases have demonstrated the challenges of
using algorithmic systems in the criminal justice process.111R (on the
application of Edward BRIDGES) v The Chief Constable of South Wales Police
[2020] EWCA Civ 1058 (‘Bridges v South Wales Police’); State of Wisconsin v.
Loomis 881 N.W.2d 749 (2016); State of Kansas v. John Keith Walls, (2017). In
the UK, academics, charity commissioned reports and freedom of information
requests have identified some of these relevant algorithmic tools (cited
within this paper). However, significant opacity remains along with confusion
about what tools are available, which are in use and how they are deployed.
This paper aims to offer a non-exhaustive survey of tools used in the criminal
justice system in the UK in the following categories: data infrastructure,
data analysis, and risk prediction.

Policy and law-makers are struggling to keep pace with the rapid changes in
technology. In particular, current regulation does not capture the different
risks present in deploying algorithmic tools at different stages of the
criminal justice pipeline. In this work, first, we define specific
technologies. Second, we survey and identify the use of these technologies at
different stages in the UK criminal justice system. Then, we consider the
implications for policy and regulation in the context of design principles for
trustworthy algorithmic systems.

##  2\. Definitions

In this work, we focus on specific data-driven technologies that are deployed
throughout the criminal justice system in the UK, in the following categories:

  1. (1)

Data infrastructure refers to software and tools primarily used to record,
store or organise data. The ability to filter and perform basic search queries
on data is included in this category. More advanced options such as
visualisations, statistical analyses, risk scores or predictions are included
in the next categories.

  2. (2)

Data analysis refers to software and tools primarily used to analyse data to
create insights informing practitioners and stakeholders. This category
includes custom data dashboards, visualisations and forms of statistical
analysis that do not include machine learning, predictions or automated
creation of risk scores.

  3. (3)

Risk prediction refers to software and tools primarily used for predicting
future risk based on analysis of past data. This category can be split into
two subcategories:

    1. (a)

Predicting future crime spatiotemporally includes software and tools primarily
used to estimate the volume, location and time of criminal activity that has
yet to occur. The predictive estimate is usually based on analysing spatial
and temporal patterns in past data.

    2. (b)

Predicting future individual crime risk includes software and tools, including
manual tools, primarily used to estimate the risk of an individual offending
in the future based on previous offending and/or other attributes of the
individual.

Other technologies that may be of interest to the community, which we do not
cover in this work, include facial recognition, gait identification, number
plate recognition, speaker identification, speech identification, lip-reading
technologies, gunshot detection algorithms and social media monitoring. While
The broad principles discussed in this work apply to these technologies as
well, in this work we do not elaborate on any of their specific challenges.

##  3\. Algorithmic decision aids in the criminal justice pipeline: UK case
study

The criminal justice system in the UK has many players and components. Here we
focus on four main components: law enforcement agencies, specifically the 43
police forces operating in England and Wales; the Crown Prosecution Service
(CPS); the courts, primarily referring to the magistrates’ court and the crown
court; and Her Majesty’s (HM) Prison and Probation Service. See Figure 1 for
an illustration, to give context to the decisions that algorithmic systems are
meant to assist.

Only a portion of all illegal activity becomes known to law enforcement
agencies. There are two main ways a criminal offence may become known: (i) A
victim or a witness can report the crime to the police, or (ii) it can be
discovered through proactive policing efforts, e.g., stop and search. The
first decision point for the police after a crime is recorded is whether to
dedicate resources to investigate the crime. Then, assuming an investigation
was carried out and a suspect was arrested, the police need to decide whether
to charge the suspect. The police can receive advice from the Crown
Prosecution Service (CPS) and make the decision, or, more commonly, the case
is transferred to the CPS, where a prosecutor decides if the case proceeds to
court.

Charges become convictions if the court finds the suspect guilty. In the UK,
most offences ($\sim 95\%$) are heard in the Magistrates’ Court. All offences
start at the Magistrates’ Court, where pre-trial custody status is decided.
Serious criminal offences are tried in the Crown Court, where a judge presides
over a jury trial. If the suspect is found guilty, a sentence will be decided
by the magistrate or by a judge. In the UK, sentencing guidelines, setting
minimum and maximum terms for the sentence, including limits on the time of
imprisonment and the amount that can be set for monetary fines, must be
followed unless “it is contrary to the interests of justice to do so”. If the
sentence includes an incarceration or probation period, the case is
transferred to the HM Prison and Probation Service. In due course, the
offender may be granted parole and released before serving their full
sentence.

This section includes representative examples of the tools and technologies
described in Section 2, and examines how they are used across the criminal
justice pipeline. To the best of the authors’ knowledge, there is no
comprehensive list of the technologies deployed within criminal justice in the
UK. This work is not a complete list of such tools either – there are many
more tools in use even just by the police forces. However, the authors believe
these examples are representative of the types of tools in use and the context
of their deployment, in the UK, and likely beyond. The information we present
is curated from government websites, published reports, peer-reviewed
articles, news articles, freedom of information requests and conversations
with police officers. We provide a reference for each tool. All tools we list
have been in use, though some may no longer be in active use.

Figure 1 illustrates where these tools are deployed in the criminal justice
pipeline.

Figure 1. Stages in the criminal justice pipeline where predictive tools are
used. In the top layer, tools are used to help decide where to look for crime.
In all other layers, tools are used to help decide how to process an
individual.

###  3.1. Law Enforcement Agencies

####  3.1.1. Crime discovery and patrol planning

Mapinfo uses crime and incident data to map where crimes have occurred so
police can monitor these ‘hotspots’. As this tool maps past crime, we classify
it as a data analysis tool. West Midlands Police reported using this tool in
answer to a freedom of information request from 2016 (Society, 2018). Patrol-
Wise is an algorithm developed by University College London (UCL) in a
research collaboration with West Yorkshire Police to predict burglaries on a
street-by-street level (Newsroom, 2017). iHotSpot is an AI-based predictive
analytics engine used to predict daily crime incident hotspots. This tool was
developed by SpaceTimeAI, a spin-out company of UCL’s SpaceTimeLab. The system
is integrated into the London Metropolitan Police as part of an EPSRC funded
research collaboration (SpaceTimeAI, [n. d.]; Cheng, 2021). Both Patrol-Wise
and iHotSpot are predictive policing tools which aim to predict future crime
spatiotemporally. We note the use of similar tools in the US, particularly
Perdpol, which was also trialled in the UK by Kent Police and the Metropolitan
Police and was criticised for potential racial bias (Lum and Isaac, 2016;
Sankin et al., 2021).

####  3.1.2. Crime recording and investigation aids

##### Data infrastructure.

The 43 police forces in England and Wales do not use a single primary data
collection tool. For recording incidents as they are reported, and managing
the first response, many forces use a platform called STORM Command and
Control system (Devon and Police, 2020; Police, 2020a, b, c; Steria, 2019;
Police, 2020d, e, 2019a, 2019b).222STORM is a software product from Sopra
Steria designed for emergency services, see
https://www.soprasteria.co.uk/industries/public-safety. If the reported
incident is a crime or a recordable incident, it is also logged and managed in
a record management system. Several of the forces use Niche RMS333Niche RMS is
a private Canadian software company, see https://nicherms.com/. for this
purpose (Technology, [n. d.]; Babuta and Oswald, 2018; Police, 2019a). The
data collected in relation to crime and reported incidents follows
comprehensive national standards set by the Home Office.444The national
standards for incidents and crime records set by the Home Office include The
National Standard for Incident Recording (NSIR), National Crime Recording
Standard (NCRS) and the Home Office Counting Rules (HOCR) (Police, 2019a). To
our knowledge, there are no national standards specifying which additional
data fields may be collected by the forces, who each use their discretion.
Information regarding which additional data fields are collected by each force
is not immediately available. In some cases, information may exist within
published police policies or can be accessed through freedom of information
requests. The raw data collected by each force is not routinely accessible to
other forces, but data sharing and data-based collaborations exist between
forces. All forces make a series of annual data returns to the Home Office,
containing aggregated data about crime, police workforce, arrests and stop and
search (Office, 2021a).

Police forces have access to several national datasets: The National Police
Computer (NPC) stores and shares criminal record information across law
enforcement agencies in the UK (Office, [n. d.]). The back-end data is not
visible to individual forces. However, the forces routinely request
information about an individual or against a vehicle registration number from
the NPC. The forces can also request information from the Fingerprint and DNA
national databases. These requests must be justified.

##### Allocating investigative resources.

The Evidence-Based Investigative Tool (EBIT) is a tool deployed by Kent Police
to predict investigative success for minor, non-domestic assault and public
order offences. A solvability assessment uses a logistic regression model,
followed by a two-step review of the case. Using software built by Kent
Police, the EBIT user answers eight questions and is provided with one of the
following recommendations: further investigation, close the case pending
further evidence, or further review by a supervisor (McFadzien et al., 2020).

##### Data analytics

The National Data Analytics Solution (NDAS) is a project sponsored by the Home
Office and led by West Midlands Police in collaboration with Accenture
(Police, [n. d.]b).555Accenture is a multinational company that provides
consulting and professional services, see https://www.accenture.com/gb-en. The
project aims to develop “algorithms on specific crime types that are applied
to police-controlled data to produce insights and analysis which inform
strategic and operational decision making” (Office, 2021b). Currently, it is
hosted by West Midlands Police and is used by seven additional police forces
and the National Crime Agency.

Sussex police use customisable data analytic dashboards developed for them by
a private company. For example, two dedicated dashboards were created in
response to the Covid-19 pandemic: one to track employee self-isolation and
one to track the daily changes in reported crimes, incidents, calls and
arrests (Police, 2021).

####  3.1.3. Predicting risk associated with an individual.

Not all individual assessment tools involve the use of computational tools.
Two manual risk assessment tools used nationally by the forces are THRIVE and
DASH. Thrive stands for Threat, Harm, Risk, Investigation opportunities,
Vulnerability of the victim and the Engagement level required to resolve the
issue. It is used to assign a priority level to an incident (Inspectorates,
2019a). DASH stands for Domestic Abuse, Stalking and Harassment. It is a
checklist tool used to assess the risk of cases involving domestic abuse,
stalking, harassment and so-called honour-based violence (Inspectorates,
2019b).

The gangs violence matrix (GVM) is a dataset of suspected London based gang
members used by the Metropolitan Police. It is used as an intelligence tool
“to reduce gang-related violence, safeguard those exploited by gangs and
prevent young lives from being lost” (Police, [n. d.]a). Each individual in
the dataset is assigned a harm score and a victim score to indicate if they
are likely to deliver or receive harm, respectively. Both scores are graded as
Red, Amber or Green. Based on a critical report published by Amnesty
International in 2018, the scores are automated risk scores, but no additional
details are provided (Police, [n. d.]a; International, 2018). The report also
highlighted concerns regarding stigma and bias in the database (International,
2018).

West Yorkshire Police uses an Integrated Offender Management (IOM) software
system called Corvus IOM Case.666The tool is provided by Bluestar, a UK based
company, see https://bluestar-software.co.uk/products/offender-management-
iom/. The system draws data from other sources, including STORM and Niche RMS
(see Section 3.1.2), analysing intelligence, crimes, arrests and substance
misuse in order to derive an individualised score aimed at providing an
indication of an individual’s likelihood to re-offend. Different crime types
are scored differently, and the Risk of Re-Offending Cohort scores are
categorically displayed as low, medium or high (Police, 2020f).

Avon and Somerset Police use a number of bespoke risk assessment tools. These
tools are built within Qlik Sense, a data analytics platform developed by a
private company.777Qlik Sense is developed by QlikTech International AB, see
https://www.qlik.com/us/products/qlik-sense. The tool produces an offender
risk score in the range 0-100, which combines two predictions: the likelihood
of offending combined with the potential harm if the re-offending does occur;
and a vulnerability risk score to predict the likelihood of an individual
becoming a victim of crime (Dencik et al., 2018).

##### Deciding whether to charge an individual.

Durham Constabulary uses an individualised Harm Assessment Risk Tool (HART).
The tool was designed in collaboration with the University of Cambridge as
part of the Checkpoint program to offer an alternative to prosecution for
individuals classified as a medium risk – likely to re-offend, but not in a
serious violent manner. Qualifying individuals are offered a four month
intervention program tailored to their needs and are not charged if they
successfully complete it (Oswald et al., 2018; Society, 2018).

###  3.2. The Crown Prosecution Service and the Courts

##### Data infrastructure.

A report on efficiency in the criminal justice system has highlighted that
“Problems occur as cases enter and progress through the system, including at
crucial handover points between organisations, such as where information
passes between police and the Crown Prosecution Service (CPS)” (the
Comptroller and Genera, 2016). In an attempt to improve efficiencies, a single
digital case management system was developed for the prosecution service and
the courts, named the Common Platform, at an investment of hundreds of
millions of pounds. The common platform was expected to be used in criminal
hearings in all Crown and magistrates’ courts in England and Wales by the end
of 2021 (Service, [n. d.]).

##### Data analytics.

The Ministry of Justice has a “largely centralised analytical function that
provides wide-ranging analytical support to all of the Ministry’s activities”
(of Justice, 2021), including analysing data from the courts. The Ministry is
taking steps to ensure the production of quality analysis with governance and
assurance frameworks, open-source coding standards,888https://moj-analytical-
services.github.io/our-coding-standards/, and a Government Digital Service
(GDS) data science cookie
cutter.999https://github.com/ukgovdatascience/cookiecutter-data-science-gds
The latter is a standardized but flexible project structure for data science
projects.

####  3.2.1. Charging Decisions

Currently, the decision of whether to charge an individual with an offence
within the Crown Prosecution Service (CPS) is done without algorithmic aids.
The CPS plans to increase its use of AI, predominately to enable a more
efficient search of text and video files, but there are no published plans to
employ algorithms to predict risk or prosecution success (Service, 2021c).

####  3.2.2. Predicting future risk associated with an individual.

If a defendant pleads guilty or is found guilty during trial, the court can
request a Pre-Sentence Report (PSR) to be prepared. The probation service
writes the report after a private interview with the offender. Reports for
magistrates’ court cases are usually same day PSRs, presented orally or in
written format. Crown court cases may require a written report containing a
full Offender Assessment System (OASys) assessment (see Section 3.3.2), which
requires the court to adjourn for 3 weeks. All reports include the following
predictive scores: Offender Group Reconviction Score (OGRS); Risk of Serious
Recidivism (RSR) Score; and a Risk of Serious Harm (RoSH) screening (Service,
2017).

Offender Group Reconviction Score (OGRS) is a predictor of proven re-offending
within 1 and 2 years of noncustodial sentence or discharge from custody.
Proven re-offending only measures re-offending known to the authorities within
the specified time frame and does not include arrests and offences pending
prosecution. The OGRS4/G101010OGRS4 is the fourth version of the tool. OGRS3
may still be in deployment. includes all recordable offending and does not
indicate the severity of predicted re-offending. Offenders with high OGRS4/G
scores may re-offend rapidly but are comparatively unlikely to be involved in
serious further offending. OGRS4/V includes violent proven re-offending only
and outperforms OGRS4/G in predicting violent re-offending. OGRS prediction
scores are based on a limited number of static risk factors111111Static risk
factors are factors that do not change or which change in only one direction.
Examples include age and past criminal offences. including age, gender and
criminal history (Service, 2015).

The Risk of Serious Recidivism (RSR) was introduced in 2014 to predict the
likelihood of an offender committing a seriously harmful re-offence within two
years. The RSR tool provides three sub-scores, one for contact sexual re-
offending, one for indecent image re-offending and one for non-sexual
violence. All scores can be calculated from static variables, but non-sexual
violence has an extended static and dynamic121212Dynamic risk factors are
factors about individuals or their environments that can change in either
direction, e.g., employment, relationships, and substance misuse. version that
performs better than the brief static score (Service, 2015).

Risk of Serious Harm (RoSH) Serious harm is defined in this context as an
“event that is life-threatening and/or traumatic and from which recovery,
whether physical or psychological, can be expected to be difficult or
impossible” (Service, 2015). RoSH screening is done in all cases to indicate
if a full analysis should be completed. The levels of RoSH are Low, Medium,
High and Very High. Medium level indicates serious harm is unlikely unless
there is a change in the offender’s circumstances. Very high indicates a high
and imminent risk of serious harm. In the full RoSH analysis, there are
separate scores indicating risk to children, known adults, the general public,
staff and other prisoners. For all but the latter, there is a separate risk
level for community and custodial settings (Service, 2015).

###  3.3. HM Prison and Probation Service

####  3.3.1. Security categorisation

The digital categorisation service (DCS) is an algorithmic tool used to make,
record and justify prison security categorisation decisions. The tool
“highlights risk information” and suggests an initial categorisation for the
prisoner that is reviewed and can be changed by the staff (Trust, 2021b).

####  3.3.2. Predicting future risk associated with an individual.

The prison and probation services in England and Wales use the actuarial risk
and needs assessment tool OASys. All offenders undergo a basic screening, and
offenders with a sentence of 12 months or more undergo a full OASys
assessment. OASys generates several risk scores based on static and dynamic
risk factors to assess the likelihood of re-offending and risk of harm to self
and others. The OGRS and RoSH scores mentioned in section 3.2.3 form part of
the full OASys assessment (Howard and Dixon, 2013; Trust, 2018).

##  4\. Implications for policy and regulation

The use of algorithmic systems – whether computerised or manual – raises the
hope for significant benefits in terms of efficiency and consistency beyond
those achievable by standard human methods. However, they also have
significant drawbacks, such as a lack of contextual understanding or common-
sense reasoning, leading to important concerns.

In the UK, algorithmic decision aids are in active deployment at almost every
decision point in the criminal justice pipeline described above. The
incorporation of new technologies is further increasing in the following
years. The Home Office has indicated that they “will therefore back and
empower the police to use new technologies to deliver operational effect in a
way that maintains public trust” ((UK), 2021). Given the high-stakes nature of
law enforcement decision-making, it is critical to design trustworthy
algorithmic tools. In this section, we identify points to consider when
evaluating the benefits and costs of deploying algorithmic tools compared to
traditional analogue policing. We highlight key aspects required for
evaluating the system as a whole, in addition to the individual tools. We
suggest policy considerations for different stages in the data and criminal
justice pipeline and in the context of different stakeholders.

###  4.1. Design Principles for trustworthy algorithmic systems

We consider the following principles of trustworthy use of tools in respect of
different stages and stakeholders in the UK criminal justice system. Many of
the points we mention have been discussed previously, e.g. see the ALGO-CARE
framework for algorithms in policing (Oswald et al., 2018) and a 10-point
guide for evaluating if risk assessment tools are fit for purpose (Fazel and
Wolf, 2018).

Fit for purpose: A specific purpose for the collection of data and deployment
of tools is necessary to justify whether the expected outcomes yield clear
societal benefits. The appropriateness of its use must be assessed with
respect to how it performs in deployment. The Metropolitan Police has stated
that “When policing uses technology, it does so to meet a defined policing
purpose which must justify any privacy intrusion. The technology’s
effectiveness and demographic performance (where relevant) must be assured as
fit for its intended purpose, its use must be transparent to the degree
possible within a policing context, and there must be safeguards, supported by
community engagement and suitable oversight” (Service, 2021b). It is also
important for the goal to be measurable. For example, Avon and Somerset
Constabulary state they use “advanced analytics along with data visualisation
dashboards” in order to “support more effective decision making” and improve
“understanding around demand, risk and threat” (Avon and Police, 2021). In
this example, the aim is articulated; however, it is not clear if or how
success is measured. We note that user satisfaction has been monitored and is
reported to be high (Avon and Police, 2021).

Data quality: The quality of data and the design choices made at the
collection stage can dramatically impact an algorithm’s intended function.
When in training and in use, data used in tools is often collected in a way
that involves some variety of selection bias. For example, the digital
categorisation tool used by the prison service in the UK relies on the quality
of intelligence fed into it. However, it is not clear which, if any, data
quality control procedures have been put in place. At the point of use, the
intelligence cannot be easily verified or challenged. As a result, the
automated categorisation produced by the tool cannot be easily verified or
challenged either. Specific concerns have been raised around “intelligence
about BAME and Muslim men which cannot be effectively challenged, and which
may reflect conscious or unconscious bias may be driving unfair treatment
which the categorisation algorithm then cements in a way which will follow the
prisoner throughout their custodial history” (Trust, 2021b).

Fairness: There is extensive literature reporting bias against minority groups
within algorithms deployed in the criminal justice system (Selbst, 2017;
Chouldechova, 2017). Many concerns focus on tools deployed in the US
(ProPublica, 2016), but there is also significant worry about tools deployed
in the UK (International, 2018; Watch, 2019). A common concern is that an
algorithm trained on biased data may learn to repeat or even reinforce the
same bias unless steps are taken to avoid it. The baseline often used in this
context is whether using the tool increases disproportionate outcomes. For
example, a pilot study of a prison security categorisation tool didn’t seem to
suggest that using the new tool increased BAME disproportionately in higher-
risk categories, but nor did it reduce it (Trust, 2021b). We note even when
disproportionately does not increase following the adoption of a tool, it does
not necessarily mean the tool is unbiased. Avon and Somerset police highlight
the exclusion of ethnicity as an input variable in their tools (Avon and
Police, 2021). However, not only does this not guarantee the tool is unbiased
with respect to ethnicity but excluding the variable can sometimes aggravate
the disproportionality in outcomes (Simons et al., 2021).

Transparency and accountability: For many reasons, it is difficult to ensure
algorithms in criminal justice are fully transparent.131313For further
discussion of the challenges surrounding algorithmic transparency, see
(Weller, 2019; Coyle and Weller, 2020; Bhatt et al., 2020) It is important,
however, that users and stakeholders have a sufficient understanding of the
underlying mechanisms. It is key that accountability for decisions always
remains with human decision-makers. Ensuring this requirement is met can prove
more difficult when the tool is proprietary and is supplied by an external
company, which is often the case.

Privacy: The availability of data on many individuals enables useful patterns
to be discovered by statistical or machine learning methods. However, from a
legal and policy perspective, the protection of individual privacy is also
important, and it can be hard to strike the right balance. It is important to
protect individuals from invasive or disproportionate data collection,
especially when the data is of a sensitive nature. This is of particular
concern when data is shared between different agencies. For example, Avon and
Somerset Police are sharing data with local agencies to “collectively be
better at identifying risk and vulnerability to support early intervention
initiatives”. Specifically, they give an example that “a model identifying a
child at high risk of sexual exploitation would be validated through
additional information such as referrals, social workers, analysts and other
data held on systems before a practitioner makes a decision” (Avon and Police,
2021). This example highlights the high level of causation required due to the
level of sensitivity of the information and the vulnerability of the
individuals in question. Avon and Somerset police state that “Data Privacy
Impact Assessments (DPIA)” are filled for every tool (Avon and Police, 2021).
It is not clear if these assessments are publicly available and whether
individuals can request to be excluded.

Robustness: If an algorithmic system performs well in some settings, it can
create the impression that it will perform well across all settings. However,
algorithms can behave in fragile, hard to predict ways, such that small
changes in input features can lead to unexpectedly large changes in model
outputs (Szegedy et al., 2014). Currently, in the UK, a new algorithmic tool
can be introduced without a pilot or a trial period. In practice, many tools
have been trialled by the different police forces and later abandoned.
Unfortunately, the results and conclusions of these pilots are not often
published.

Contestability: The ability to contest decisions made or assisted by an
automated system is an important safeguard in the trustworthy deployment of
algorithmic tools (Mulligan et al., 2019; Ploug and Holm, 2020; Lyons et al.,
2021).

Granting an individual recourse for decisions that they believe were not
granted fairly requires a well-designed process. It may involve a formal
internal review and a clear cause of action for judicial review. Such
processes requires a level of transparency and explainability to ensure
understanding of the algorithmic process and outcome.

###  4.2. Data infrastructure

Data infrastructure and data collection standards are inconsistent across the
43 police forces in England and Wales, although there are some data sharing
and database collaborations between forces. The Home Office has stated that
work is underway to establish a Central Data Office within the Police Digital
Service, which will “provide the essential infrastructure for the sector”
(Office, 2021a). Where data infrastructure is not fit for purpose,
organisations cannot benefit from the potential to improve criminal justice
processes. Better data infrastructure would allow better information sharing
between all 43 law enforcement agencies to the make performance of their
duties easier and potentially improve public safety. The UK National Policing
Digital Strategy recognises the need to modernise legacy data infrastructure
as data is a “strategic asset” ((NPCC) et al., 2020).

Although data infrastructure systems do not usually include complicated
algorithms or artificial intelligence, the quality and consistency of the data
collected are essential prerequisites for reliable analytical analysis or
predictions. It is, therefore, crucial to ensure that the data collection
process achieves good quality, fit for purpose data and does not unnecessarily
infringe on individual privacy. To examine the application of such principles,
we consider the West Midlands Police data-driven policing project with
Accenture as a data infrastructure case study.

As noted above, this data infrastructure collates various datasets on police
data, including ‘stop and search’. Consider, for example, the way information
about race and ethnicity may be collected by law enforcement in ‘stop and
search’ datasets. In the USA, data collected includes separate fields for race
and ethnicity, yet not all agencies fill in the ethnicity field (Fogliato et
al., 2021). This creates a bias in the data, which can be very difficult to
fix post hoc. In addition, race and ethnicity as defined by the individual
might be different from the race and ethnicity observed and recorded by the
arresting officer. In the UK, it is clearly stated if the race and ethnicity
field is based on the individual’s statement or assumed by the officer, with
preference given to the former. Further, on a societal level, where certain
demographics have higher arrest rates, police may be more likely to stop and
search based on observed race, compounding and reinforcing existing
unfairness.141414Issues of race disproportionality within policing were
discussed in the Lammy Review, The Stephen Lawrence Inquiry and Race Disparity
Audit, and in the discussion in (for Data Ethics and Innovation, 2021). The
report on the project does not discuss principles of trustworthy design
(Accenture, 2020). The West Midlands Police and Crime Commissioner Ethics
Committee raised potential bias as an issue arising from the data
infrastructure set up by Accenture (Police and Commissioner, 2021).

Data infrastructure that reinforces existing bias will not be fit for purpose
or fair. The Accenture project does not clarify how the use of data from
various sources is being dealt with. Specifically, different types of data
have different levels of reliability. For example, intelligence reports cannot
be taken on par with proven offending data.

Law enforcement recognises that privacy and security are issues with modern
data infrastructures (Service, 2021b). Recent developments in privacy-
enhancing technologies, including differential privacy and secure multi-party
computation, can sometimes enable useful information to be discovered without
inappropriate access to private data (Society, 2021).

###  4.3. Data analysis

The use of data analysis as part of decision-making processes may give rise to
certain risks. As noted above, there are various data analysis tools in use
across the UK law enforcement agencies. However, the Home Office recently
submitted that “our analysis suggests that most data analytics used by police
forces is currently used to enable organisational effectiveness and resource
planning rather than directly to tackle crime” (Office, 2021a). It appears
there is a lack of central understanding of what algorithmic tools are in use.
A lack of national standards creates concerns about fairness, transparency and
accountability.

MAPinfo is an example of an analytics tool that uses past incident reports to
identify crime ‘hotspots’. Police officers are alerted to these areas for
patrols and monitoring. Although not strictly a predictive tool, careless use
of hotspot mapping can still result in over-policing of local communities –
such that a level of policing “is disproportionate to the level of crime that
actually exists in a given area” (Liberty, 2021). This follows a similar
concern as raised in respect of data collection above. If data collection is
not fit for purpose, this will flow through to later stages of the data
pipeline. For example, an algorithm trained on biased data will learn to
repeat or even reinforce the same bias unless steps are taken to avoid it.
There is potential for bias to be further aggravated by algorithmic tools –
for example, via feedback loops (Ensign et al., 2018).

###  4.4. Risk Prediction

Predicting future crime has been the focus of much research and legal
development (ProPublica, 2016; Berk and Bleich, 2013; Flores et al., 2016;
Fazel et al., 2021). The high-risk nature of predicting future criminal
offences has induced procedural safeguards that ensure law enforcement cannot
depend on entirely automated risk predictions.151515Data Protection Act 2018
(UK) section 50; General Data Protection Regulations (EU) section 22. See also
draft provisions in Draft Artificial Intelligence Act 2021 (EU) section 19 and
Bill for an Algorithmic Accountability Act of 2022 (USA).

Even where such tools are not solely used to make decisions, further
consideration should be given to fairness generally in risk prediction and
ensure it is not disproportionate to individual rights and freedoms. Risk
predictions about a persons’ future behaviour may have long-lasting
ramifications. Unlike the other two categorisations of algorithmic tools, this
type of tool may be relevant to court proceedings. Transparency is a
prerequisite for individuals to understand the way algorithmic tools are
designed and used, a necessity to be able to challenge this use. However, the
inference rules algorithmic risk prediction models ‘learn’ when training may
not be easily understood by humans (Vestby and Vestby, 2021).

HM Prison and Probation Service have advanced a policy to “deploy an advanced
algorithm which draws on data from the whole of the prisoner’s journey through
the criminal justice system to produce a recommended initial categorisation”
(Trust, 2021a; Service, 2021a). This categorisation is reviewed and can be
changed by the staff. The Prison Reform Trust, an independent UK charity,
highlights the welcome prospects of “more consistent application of the
criteria that determine categorisation” (Trust, 2021a). However, the quality
of the prediction depends on the quality of information that is fed into the
system which cannot easily be validated or challenged. This also puts in
question the ability of the staff to effectively challenge the categorisation
suggested by the tool. The responsibility of ensuring the tool works properly
falls on a manager appointed by the Prison Governor. Specifically, the manager
is required to “that the categorisation/recategorisation process is
functioning effectively; that decisions are fair, consistent and taken without
bias; to provide quality assurance of decision making; to collect and analyse
data in terms of protected characteristics alongside other equalities data to
ensure that there is a complete picture of any disproportionate impact, and to
implement change where necessary” (Trust, 2021a). However, it is unlikely
local managers are equipped to do this.

###  4.5. Implication of the deployment of multiple algorithms simultaneously

The UK criminal justice system is not unusual in having many players and
components, such as regional law enforcement agencies. Ensuring trustworthy
deployment within the criminal justice system requires considering the
deployment of other algorithmic tools in the same jurisdiction. Specifically,
the deployment of algorithmic systems in earlier stages of the pipeline, i.e.,
policing, can introduce both noise and bias to algorithms deployed later
within the pipeline, e.g., within the courts. Consider, for example, tools
that predict risk associated with an individual, like those used in the pre-
sentencing report in the UK (see Section 3.2.2). These tools often take into
account the number of prior arrests in determining the level of risk. Even if
those algorithms are well designed and thoroughly tested, they may implicitly
depend on the assumption that an individual’s likelihood of arrest is
independent of, for example, the neighbourhood where they reside. If this
stops being the case at any point of deployment, potentially due to a non-
reliable deployment of a predictive policing algorithm within one or more
jurisdictions, this may affect the reliability of the individualised risk
prediction tool used in the courts.

###  4.6. Ethics, governance and regulation

Law enforcement agencies have a positive duty to “satisfy themselves, either
directly or by way of independent verification, that the software program in
this case does not have an unacceptable bias on the grounds of race or
sex”.161616Bridges v South Wales Police para 199. However, it is not clear
that individual agencies have the resources, skills and understanding to
verify an absence of bias. For example, merely removing protected
characteristics from data analytics tools is not sufficient (Simons et al.,
2021; Office, 2021a). There is often a need for law enforcement to find
commercial partners to develop algorithmic tools. However, undertaking
verification of an absence of bias may be particularly difficult when using
proprietary software. The Metropolitan Police have highlighted “vendors being
reluctant to share enough information citing reasons of commercial
confidentiality” as a challenge (Service, 2021b). It is important to remember
that even where vendors claim commercial confidentiality over algorithmic
tools, a public authority cannot neglect its non-delegable Public Sector
Equality Duty.171717Bridges v South Wales Police para 199, see Public Sector
Equality Duty in Equality Act 2010 section 149.

The Home Office states that “we strongly disagree with the characterisation
put forward by some that these technologies are being deployed without regard
to proper evaluation or engagement, or that we are allowing sensitive
decisions to be delegated to machines in a way that is either contrary to the
law or the core principles of the CJS (criminal justice system)” (Office,
2021b). However, it is unclear what these proper evaluations are and who is in
charge of ensuring they are properly executed. From the evidence provided by
Home Office and other law enforcement agencies, it is difficult to establish
the governance structure with respect to ensuing proper evaluation of new
technologies. For example, both Kent Police and the Metropolitan Police have
previously deployed and later abandoned the predictive policing software
Predpol (Predpol, [n. d.]; Nilsson, 2018), a tool still widely used in the US,
with strong claims that its use can potentially increase racial bias (Sankin
et al., 2021).

Although it is often stated that all algorithmic decisions are advisory and
that accountability remains with the practitioners, this is only typically
valuable when practitioners are in a position to effectively verify or
challenge the decision, which is often not the case. Furthermore, the decision
to incorporate tools is often not in the hands of individual officers or
practitioners, adding to the limits of holding them accountable for mistakes
caused by using the tool. The Metropolitan Police highlight the challenge of
internally providing effective oversight for the systems: “Policing would
welcome the support from experts in Government and beyond to help meet this
challenge”. Specifically, they highlighted “A code of practice would provide a
framework for ethical decision making when considering whether to use a new
technology” (Service, 2021b). We note that there are existing frameworks
available (Oswald et al., 2018; Fazel and Wolf, 2018), and that a national
framework for policing is being developed by the National Police Chiefs’
Council and the Centre for Data Ethics and Innovation.

Currently, several police forces take their data analytics products to
independent ethics committees and scrutiny panels prior to deployment. These
are often comprised of volunteers and operate in an advisory capacity. Meeting
notes are typically made public and can be further scrutinised by interested
parties. However, projects are often considered by ethics committees after
much time and resources have been invested in them. As such, there may be an
implicit pressure to approve or to find post hoc solutions to any problems
found. Governance structures are needed to ensure that teams deploying data
analytics have the knowledge and resources to intentionally design trustworthy
systems from the beginning. Demonstrating an intentionally trustworthy design
is likely to both save resources and provide better results. In addition,
inter-agency collaboration is important to ensure a consistent approach toward
ethics and evaluation of advanced algorithms. A national Data Ethics
Governance model, which the Home Office stated is planned, is needed (Office,
2021b). It is crucial to ensure that governance structures allocate
responsibilities to those with the resources, support and ability to undertake
appropriate due diligence (Trust, 2021a).

##  5\. Discussion

Data-driven technologies are increasingly an important part of the criminal
justice system. Avon and Somerset Constabulary stated they “believe that data
is a critical organisational asset and, like our staff, is at the core of
delivering efficient and effective local policing services” (Avon and Police,
2021). The Metropolitan Police stated that “To declare technologies as being
‘off limits’ to policing risks denying law enforcement the tools it needs to
keep the public safe whilst leaving these tools easily available for criminals
and commercial users to consume and exploit” (Service, 2021b). The Home Office
endorses the use of technology to improve effectiveness and efficiency within
the criminal justice system ((UK), 2021).

While specific tools such as Predpol and COMPAS have received significant
attention from the algorithmic fairness community, there are many more
algorithmic systems trialled and deployed around the world. In this work, we
use the UK as a case study to step back and consider the deployment of
algorithmic tools at different decision points in the criminal justice system.
We consider how different trustworthy design principles might apply to
examples of tools in deployment and examine the current state of guidance and
regulation of these tools in deployment. Although ethical decision making and
evaluation frameworks are available and more are in development, it is unclear
who is able and accountable to perform the required due diligence. It is
implied that: the practitioner is responsible for validating or challenging
the automated decision; local management is responsible for ensuring the tool
is fit for purpose, free of bias and is working as expected; and in some
cases, it is the responsibility of advisory ethics committees, composed of
volunteers, to ethically scrutinise and approve the use of operational
systems. Unfortunately, it is often the case that none of the parties above
has the ability in terms of access to data, skill, time and resources to
perform the required level of due diligence.

We conclude with the following proposals: (1) A clear governance structure is
required, identifying an appropriate body able to assist agencies in choosing
or building trustworthy tools and systems, and ensuring proper testing and due
diligence occur prior to deployment. (2) Outcomes of testing and evaluation of
new tools should be made public unless security prohibits this, and mechanisms
for independent scrutiny of the systems should be made available wherever
possible (for example, by academic-led teams).

The House of Lords Justice and Home Affairs Committee published its report on
Technology Rules? The advent of new technologies in the justice system
(Justice and Committee, 2022). The Committee examined the use of algorithmic
tools throughout the criminal justice pipeline and proposed many
recommendations that touch on issues identified in this paper, including, but
not limited to: legislation establishing clear principles and standards for
the use of technologies in the justice system (recommendations 9 and 12);
establishing a proper governance structure for the use of technologies in the
application of the law (recommendation 5); the need for an independent body to
support the various entities involved in designing and deploying new
technologies (recommendation 6); empowering local specialist ethics committees
in the law enforcement community (recommendation 35). In addition, the
Committee concludes that humans should always be the ultimate decision-maker
and that there should be more clarity on the possible duty of candour for
police to ensure appropriate transparency over their use of AI (recommendation
18).

We welcome these recommendations and hope that further research, law reform
and governance oversight will bring these recommendations into effective
practice.

###### Acknowledgements.

The authors thank Detective Sergeant Laurence Cartwright for valuable comments
and discussion. We thank the reviewers for their helpful comments and
suggestions. M.Z. acknowledges support from EPSRC grant EP/V025279/1, The Alan
Turing Institute and the Leverhulme Trust grant ECF-2021-429. H.S.
acknowledges support from the General Sir John Monash Foundation. A.W.
acknowledges support from a Turing AI Fellowship under grant EP/V025279/1, The
Alan Turing Institute, and the Leverhulme Trust via CFI.

## References

  * (1)
  * Accenture (2020) Accenture. 2020\.  West Midlands Police: Serving and protecting with cloud.  https://www.accenture.com/_acnmedia/PDF-142/Accenture-West-Midlands-Police-Serving-and-protecting-with-cloud.pdf#zoom=40
  * Australian Human Rights Commission (2021) Australian Human Rights Commission. 2021\.  _Human Rights and Technology_.  Final Report. Australian Human Rights Commission.
  * Avon and Police (2021) Avon and Somerset Police. 2021.  Avon and Somerset Police — Written evidence (NTL0052).  https://committees.parliament.uk/writtenevidence/40328/pdf/
  * Babuta and Oswald (2018) Alexander Babuta and Marion Oswald. 2018.  _Machine Learning Algorithms and Police Decision-Making: Legal, Ethical and Regulatory Challenges_.  Technical Report. Royal United Services Institute.  https://rusi.org/explore-our-research/publications/whitehall-reports/machine-learning-algorithms-and-police-decision-making-legal-ethical-and-regulatory-challenges
  * Babuta and Oswald (2020) Alexander Babuta and Marion Oswald. 2020.  Data analytics and algorithms in policing in England and Wales: Towards a new policy framework.
  * Berk and Bleich (2013) Richard A Berk and Justin Bleich. 2013.  Statistical procedures for forecasting criminal behavior: A comparative assessment.  _Criminology & Pub. Pol’y_ 12 (2013), 513.
  * Bhatt et al. (2020) Umang Bhatt, McKane Andrus, Adrian Weller, and Alice Xiang. 2020.  Machine learning explainability for external stakeholders.  _arXiv preprint arXiv:2007.05408_ (2020).
  * Cheng (2021) Tao Cheng. 2021\.  Crime, Policing and Citizenship (CPC) - Space-Time Interactions of Dynamic Networks.  https://gtr.ukri.org/projects?ref=EP%2FJ004197%2F1 Accessed: 2021.
  * Chouldechova (2017) Alexandra Chouldechova. 2017\.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  _Big data_ 5, 2 (2017), 153–163.
  * Coyle and Weller (2020) Diane Coyle and Adrian Weller. 2020.  “Explaining” machine learning reveals policy challenges.  _Science_ 368, 6498 (2020), 1433–1434.
  * Dencik et al. (2018) Lina Dencik, Arne Hintz, Joanna Redden, and Harry Warne. 2018\.  Data scores as governance: Investigating uses of citizen scoring in public services project report.
  * Devon and Police (2020) Devon and Cornwall Police. 2020.  Force Call Handling and Contact Policy.  https://www.devon-cornwall.police.uk/FOI/Doc/d4999792-4609-4f1c-aba2-2948f7bba40a/p?D268open.pdf
  * Ensign et al. (2018) Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2018\.  Runaway feedback loops in predictive policing. In _Conference on Fairness, Accountability and Transparency_. PMLR, 160–171.
  * Eubanks (2018) Virginia Eubanks. 2018\.  _Automating Inequality_.  St. Martin’s Press.
  * Fazel et al. (2021) Seena Fazel, Matthias Burghart, Thomas Fanshawe, Sharon Danielle Gil, John Monahan, and Rongqin Yu. 2021.  The predictive performance of criminal risk assessment tools used at sentencing.
  * Fazel and Wolf (2018) Seena Fazel and Achim Wolf. 2018.  Selecting a risk assessment tool to use in practice: a 10-point guide.  _Evidence-based mental health_ 21, 2 (2018), 41–43.
  * Flores et al. (2016) Anthony W Flores, Kristin Bechtel, and Christopher T Lowenkamp. 2016\.  False positives, false negatives, and false analyses: A rejoinder to machine bias: There’s software used across the country to predict future criminals. and it’s biased against blacks.  _Fed. Probation_ 80 (2016), 38.
  * Fogliato et al. (2021) Riccardo Fogliato, Alice Xiang, Zachary Lipton, Daniel Nagin, and Alexandra Chouldechova. 2021\.  On the Validity of Arrest as a Proxy for Offense: Race and the Likelihood of Arrest for Violent Crimes. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_. ACM.  https://doi.org/10.1145/3461702.3462538
  * for Data Ethics and Innovation (2021) Centre for Data Ethics and Innovation. 2021.  Review into bias in algorithmic decision-making.
  * Howard and Dixon (2013) Philip D Howard and Louise Dixon. 2013.  Identifying change in the likelihood of violent recidivism: Causal dynamic risk factors in the OASys violence predictor.  _Law and Human Behavior_ 37, 3 (2013), 163\.
  * Inspectorates (2019a) Criminal Justice Inspectorates. 2019a.  DASH.  https://www.justiceinspectorates.gov.uk/hmicfrs/glossary/thrive/
  * Inspectorates (2019b) Criminal Justice Inspectorates. 2019b.  DASH.  https://www.justiceinspectorates.gov.uk/hmicfrs/glossary/dash
  * International (2018) Amnesty International. 2018\.  Trapped in the Matrix.  https://www.amnesty.org.uk/files/reports/Trapped%20in%20the%20Matrix%20Amnesty%20report.pdf
  * Jansen (2018) Fieke Jansen. 2018\.  Data Driven Policing in the Context of Europe.  _Data Justice Lab_ (2018).
  * Justice and Committee (2022) Justice and Home Affairs Committee. 2022.  Technology Rules? The advent of new technologies in the justice system.  https://publications.parliament.uk/pa/ld5802/ldselect/ldjusthom/180/18002.htm
  * Liberty (2021) Liberty. 2021.  New technologies and the application of the law: Written evidence.  https://committees.parliament.uk/writtenevidence/38701/pdf/
  * Lum and Isaac (2016) Kristian Lum and William Isaac. 2016.  To predict and serve?  _Significance_ 13, 5 (2016), 14–19.
  * Lyons et al. (2021) Henrietta Lyons, Eduardo Velloso, and Tim Miller. 2021\.  Fair and Responsible AI: A focus on the ability to contest.  _arXiv preprint arXiv:2102.10787_ (2021).
  * Mamalian and La Vigne (1999) Cynthia A Mamalian and Nancy Gladys La Vigne. 1999.  The use of computerized crime mapping by law enforcement: Survey results.
  * McFadzien et al. (2020) Kent McFadzien, Alan Pughsley, Andrew M Featherstone, and John M Phillips. 2020.  The evidence-based investigative tool (EBIT): a legitimacy-conscious statistical triage process for high-volume crimes.  _Cambridge Journal of Evidence-Based Policing_ 4, 3 (2020), 218–232.
  * Mulligan et al. (2019) Deirdre K Mulligan, Daniel Kluttz, and Nitin Kohli. 2019\.  Shaping our tools: Contestability as a means to promote responsible algorithmic decision making in the professions.  _Available at SSRN 3311894_ (2019).
  * Newsroom (2017) The Newsroom. 2017\.  I predict a break-in: Yorkshire police use cutting-edge technology to deter burglars.  https://www.yorkshirepost.co.uk/news/crime/i-predict-break-yorkshire-police-use-cutting-edge-technology-deter-burglars-595904
  * Nilsson (2018) Patricia Nilsson. 2018\.  First UK police force to try predictive policing ends contract.  https://www.ft.com/content/b34b0b08-ef19-11e8-89c8-d36339d835c0 Accessed: 2022.
  * (NPCC) et al. (2020) The National Police Chiefs Council (NPCC), Association of Police, and Crime Commissioners. 2020.  National Policing Digital Strategy 2020-2030.  https://www.apccs.police.uk/media/4886/national-policing-digital-strategy-2020-2030.pdf
  * of Justice (2021) The Ministry of Justice. 2021\.  Ministry of Justice — Written evidence (NTL0053).  https://committees.parliament.uk/writtenevidence/40365/pdf/
  * Office ([n. d.]) ACRO Criminal Records Office. [n. d.].  Police National Computer Services.  https://www.acro.police.uk/PNC-services Accessed: 2022.
  * Office (2021a) Home Office. 2021a.  New technologies and the application of the law: Written evidence.  https://committees.parliament.uk/writtenevidence/40975/pdf/
  * Office (2021b) The Home Office. 2021b.  Home Office — Written evidence (NTL0055).  https://committees.parliament.uk/writtenevidence/40975/pdf/
  * Oswald et al. (2018) Marion Oswald, Jamie Grace, Sheena Urwin, and Geoffrey C. Barnes. 2018.  Algorithmic risk assessment policing models: lessons from the Durham HART model and ‘Experimental’ proportionality.  _Information & Communications Technology Law_ 27, 2 (2018), 223–250.  https://doi.org/10.1080/13600834.2018.1458455
  * Pasquale (2019) Frank Pasquale. 2019\.  _The Black Box Society_.  Harvard University Press, Cambridge, MA.
  * Ploug and Holm (2020) Thomas Ploug and Søren Holm. 2020.  The four dimensions of contestable AI diagnostics-A patient-centric approach to explainable AI.  _Artificial Intelligence in Medicine_ 107 (2020), 101901.
  * Police (2020a) Dorset Police. 2020a.  Freedom of Information Act Request No: 2020-799.  https://www.dorset.police.uk/media/64900/record-1-2020-396.doc
  * Police (2020b) Essex Police. 2020b.  D0503 Procedure - Responding to Incidents.  https://www.essex.police.uk/foi-ai/essex-police/our-policies-and-procedures/d/d0503-procedure---responding-to-incidents/
  * Police (2020c) Leicestershire Police. 2020c.  Freedom of Information 003528/20.  https://www.leics.police.uk/SysSiteAssets/foi-media/leicestershire/disclosure_2020/11.-november/3528-20-iccs-and-cad-systems.pdf
  * Police (2019a) Sussex Police. 2019a.  Crime and incident disposal recording and auditing policy.  https://www.sussex.police.uk/SysSiteAssets/foi-media/sussex/policies/crime-and-incident-disposal-recording-and-auditing-policy-7572019.pdf
  * Police (2020d) Staffordshire Police. 2020d.  Freedom of Information request: reference 12562.  https://www.staffordshire.police.uk/SysSiteAssets/foi-media/staffordshire/published-foi-request-responses-december-2020/foi-12562-iccs--cad-systems.pdf
  * Police (2020e) Suffolk Police. 2020e.  Freedom of Information Request Reference No : FOI 003542/20.  https://www.suffolk.police.uk/sites/suffolk/files/003542-20_-_iccs_and_cad_contract.pdf
  * Police (2021) Sussex Police. 2021\.  Freedom of Information ref 0162/21.  https://www.sussex.police.uk/SysSiteAssets/foi-media/sussex/other_information/freedom-of-information---foi-0162.21-covid.pdf
  * Police ([n. d.]a) The Metropolitan Police. [n. d.]a.  Gangs violence matrix.  https://www.met.police.uk/police-forces/metropolitan-police/areas/about-us/about-the-met/gangs-violence-matrix/ Accessed: 2021.
  * Police and Commissioner (2021) West Midlands Police Police and Crime Commissioner. 2021.  Ethics Committee Minutes and Advice.  https://www.westmidlands-pcc.gov.uk/wp-content/uploads/2021/04/05032021-EC-Minutes-and-Advice.pdf?x86241
  * Police ([n. d.]b) West Yorkshire Police. [n. d.]b.  National Data Analytics Solution (NDAS) Privacy Notice.  https://www.westyorkshire.police.uk/advice/modern-slavery/national-data-analytics-solution-ndas-privacy-notice Accessed: 2022.
  * Police (2019b) West Yorkshire Police. 2019b.  Information and Data Management.  https://www.westyorkshire.police.uk/sites/default/files/2019-09/information_and_data_management_q.pdf
  * Police (2020f) West Yorkshire Police. 2020f.  Integrated Offender Management (IOM).  https://www.westyorkshire.police.uk/sites/default/files/2020-07/integrated_offender_management_iom.pdf
  * Predpol ([n. d.]) Predpol. [n. d.].  Kent Police Use PredPol To Prevent Violent Crime.  https://www.predpol.com/kent-police-use-predpol-to-prevent-violent-crime/ Accessed: 2022.
  * ProPublica (2016) ProPublica. 2016\.  Machine Bias.
  * Sankin et al. (2021) Aaron Sankin, Dhruv Mehrota, Surya Mattu, and Annie Gilbertson. 2021.  Crime Prediction Software Promised to Be Free of Biases. New Data Shows It Perpetuates Them.  https://themarkup.org/prediction-bias/2021/12/02/crime-prediction-software-promised-to-be-free-of-biases-new-data-shows-it-perpetuates-them
  * Santow (2021) Edward Santow. 2021\.  Intelligence: Coding calues into the future.  In _The Public Square Project_ , Peter Lewis and Jordan Guiao (Eds.). Melbourne University Press, Melbourne.
  * Selbst (2017) Andrew Selbst. 2017\.  Disparate Impact in Big Data Policing.  _Georgia Law Review_ 52 (2017), 3373.  Issue 1.
  * Selbst (2018) Andrew Selbst. 2018\.  Disparate Impact in Big Data Policing.  _Georgia Law Review_ 52, 1 (2018), 3373.
  * Service ([n. d.]) HM Courts & Tribunals Service. [n. d.].  HMCTS services: Common Platform.  https://www.gov.uk/guidance/hmcts-services-common-platform#looking-ahead-in-2021 Accessed: 2022.
  * Service (2021a) HM Prison & Probation Service. 2021a.  Security Categorisation Policy Framework.  https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1011502/security-categorisation-pf.pdf
  * Service (2021b) Metropolitan Police Service. 2021b.  Metropolitan Police Service — Written evidence (NTL0031).  https://committees.parliament.uk/writtenevidence/38736/pdf/
  * Service (2015) National Offender Management Service. 2015\.  A compendium of research and analysis on the Offender Assessment System (OASys).  https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/449357/research-analysis-offender-assessment-system.pdf
  * Service (2017) National Offender Management Service. 2017\.  Determining Pre Sentence Reports - Sentencing within the new framework. PI 04/2016.  https://www.gov.uk/government/publications/determining-pre-sentence-reports-pi-042016
  * Service (2021c) The Crown Prosecution Service. 2021c.  Crown Prosecution Service — Written evidence (NTL0018).  https://committees.parliament.uk/writtenevidence/38677/pdf/
  * Simons et al. (2021) Joshua Simons, Sophia Adams Bhatti, and Adrian Weller. 2021\.  _Machine Learning and the Meaning of Equal Treatment_.  Association for Computing Machinery, New York, NY, USA, 956–966.  https://doi.org/10.1145/3461702.3462556
  * Society (2018) The Law Society. 2018\.  _Algorithms in the Criminal Justice System_.  Technical Report. The Law Society.  https://www.lawsociety.org.uk/en/topics/research/algorithm-use-in-the-criminal-justice-system-report
  * Society (2021) The Royal Society. 2021\.  Privacy Enhancing Technologies.  https://royalsociety.org/topics-policy/projects/privacy-enhancing-technologies/
  * SpaceTimeAI ([n. d.]) SpaceTimeAI. [n. d.].  iHotSpot.  http://spacetimeai.com/iHotSpot.html Accessed: 2021.
  * Sprick (2019) Daniel Sprick. 2019\.  Predictive Policing in China: An Authoritarian Dream of Public Security.  _Naveiñ Reet: Nordic Journal of Law and Social Research (NNJLSR) No_ 9 (2019).
  * Steria (2019) Sopra Steria. 2019\.  Merseyside Police sign contract for STORM Command and Control system.  https://www.soprasteria.co.uk/thinking/blogs/details/merseyside-police-sign-contract-for-storm-command-and-control-system
  * Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014\.  Intriguing properties of neural networks.  arXiv:1312.6199 [cs.CV]
  * Technology ([n. d.]) Niche Technology. [n. d.].  Niche RMS Customer Profiles.  https://nicherms.com/region/uk/ Accessed: 2021.
  * the Comptroller and Genera (2016) the Comptroller and Auditor Genera. 2016.  _Efficiency in the criminal justice system_.  Technical Report. National Audit Office.  https://www.nao.org.uk/wp-content/uploads/2016/03/Efficiency-in-the-criminal-justice-system.pdf
  * Trust (2021a) Prison Reform Trust. 2021a.  New technologies and the application of the law: Written evidence.  https://committees.parliament.uk/writtenevidence/38265/pdf/
  * Trust (2018) The Prison Reform Trust. 2018\.  Offender Management and Sentence Plan.  http://www.prisonreformtrust.org.uk/ForPrisonersFamilies/PrisonerInformationPages/OffenderManagementandsentenceplanning
  * Trust (2021b) The Prison Reform Trust. 2021b.  Prison Reform Trust — Written evidence (NTL0004).  https://committees.parliament.uk/writtenevidence/38265/pdf/
  * (UK) (2021) The Home Office (UK). 2021\.  Home Office — Written evidence (NTL0055).  https://committees.parliament.uk/writtenevidence/40975/pdf/
  * Vestby and Vestby (2021) Annette Vestby and Jonas Vestby. 2021.  Machine Learning and the Police: Asking the Right Questions.  _Policing: A Journal of Policy and Practice_ 15 (2021), 44.  Issue 1.
  * Watch (2019) Big Brother Watch. 2019\.  Big Brother Watch’s written evidence on algorithms in the justice system for the Law Society’s Technology and the Law Policy Commission.  https://bigbrotherwatch.org.uk/wp-content/uploads/2019/02/Big-Brother-Watch-written-evidence-on-algorithms-in-the-justice-system-for-the-Law-Societys-Technology-and-the-Law-Policy-Commission-Feb-2019.pdf
  * Weisburd et al. (2008) David Weisburd, Rosann Greenspan, Stephen Mastrofski, and J Willis. 2008.  Compstat and organizational change: A national assessment.
  * Weller (2019) Adrian Weller. 2019\.  Transparency: motivations and challenges, in ‘Explainable AI: Interpreting, Explaining and Visualizing Deep Learning’.