# The Dangers of Computational Law and Cybersecurity; Perspectives from
Engineering and the AI Act

Kaspar Rosager Ludvigsen  Department of Computer and Information Sciences,
University of Strathclyde<EMAIL_ADDRESS>Shishir
Nagaraja  Department of Computer and Information Sciences, University of
Strathclyde<EMAIL_ADDRESS>Angela Daly  Leverhulme Research
Centre for Forensic Science and Dundee Law School<EMAIL_ADDRESS>
(June 2022)

###### Abstract

Computational Law has begun taking the role in society which has been
predicted for some time. Automated decision-making and systems which assist
users are now used in various jurisdictions, but with this maturity come
certain caveats. Computational Law exists on the platforms which enable it, in
this case digital systems, which means that it inherits the same flaws.
Cybersecurity is one framework which addresses these potential weaknesses, and
in this paper we go through known issues and discuss them in the various
levels, from design to the physical realm. We also look at machine-learning
specific adversarial problems, which entail further weaknesses. Additionally,
we make certain considerations regarding computational law and existing and
future legislation. Finally, we present three recommendations which are
necessary for computational law to function globally, and which follow ideas
in safety and security engineering. As indicated, we find that computational
law must seriously consider that not only does it face the same risks as other
types of software and computer systems, but that failures within it may cause
financial or physical damage, as well as injustice. The consequences of
Computational Legal systems failing are in this sense greater than if they
were merely software and hardware. And if the system employs machine-learning,
it must take note of the very specific dangers which this brings, of which
data poisoning is the classic example. Computational law must also be
explicitly legislated for, which we show is not the case currently in the EU,
and this is also true for the cybersecurity aspects that will be relevant to
it. But there is great hope in EU’s proposed AI Act, which makes an important
attempt at taking the specific problems which Computational Law bring into the
legal sphere. Lastly, our recommendations for Computational Law and
Cybersecurity are: Accommodation of threats, adequate use, and that humans
must remain in the centre of their deployment. The latter is primarily for the
abilities humans process and which allow them to handle emergencies.

##  1 Introduction

Despite law being established as an academic discipline for a significant
period of time, it is still rather undefined and lacks the rigour that other
disciplines possess. There are ongoing debates and unresolved questions as to
when law is deductive or inductive111And while these are vital, we cannot
discuss them further in this paper. For a contrary opinion, see footnote 4,
page 98 in [Gaudencio2022]., so for computational law (CL) to even claim being
future-proof misses the mark. Like the existence of exotic matter in
astronomy, you may deduce or assume their existence, but empirical evidence
will eventually prevail and show whether it is worthwhile222Past questions on
this are still not fully answered, see [Malaney1988].. The same can be said
for CL, as its implementation should not be forced because of powers outside
the academic and legal sphere[Hoffmann-Riem2021, 10 - 11], or for the sake of
profit. Increased use must be justified, regardless of whether some types of
technology are forced upon everyone without any other reason than power333See
any work on facial recognition or contract tracing, e.g., [Rezende2020,
Siatitsa2020, White2021]..

CL can be defined by its usage, like many other fields of law444See ongoing
work on how this is understood, like [Schafer2022].. Law in general, and CL
too, contain an important modifier:

They are affected by the technological systems they exist in555There are
diverging opinions on this, see [Koops2006] versus [Ohm2010]., and CL
expresses this in a much more extreme manner than any other field. It would
not even exist without its extra-legal roots666See, e.g., [Kelso1946,
DanielHillis1982].. Computer science provides the basis and logic for CL, and
in turn brings the same drawbacks777While this is not a new or unknown
phenomena, it is worth discussing and elaborating on, both to make it clearer
for policymakers, users and the public in general..

A good parallel to describe this, would be the event of IoT devices888IoT
devices can be part of a CL system, e.g., the data could feed into a system
which decides on the basis of it.. The increased adoption of these brings in a
new danger at every step, as all of them are (usually) connected to a network
and make use of software. Every danger posed to computers and digital systems
therefore exist in these devices, which are often used close to humans or for
critical infrastructure, increasing the amount of possible accidents and the
risk of damage999See any literature on this problem, [Ludvigsen2022, Fu2020,
Rizvi2018]..

CL shares this, every logical or otherwise systemic pitfall can pose a danger.
For example, circumvention and adversarial law usually relies on a system that
consists of courts and subjects seeking to abuse it101010Extra-legal means to
circumvent or abuse the legal system exist too, very much analogously to how
many different ways an adversary can attack digital systems.. Doing the same
in CL only requires obfuscating the actions within the logic of the code or
otherwise abusing technical limitations or faults of the system. The different
types of parties which can do this to these systems are many times larger,
than the sum of money and influence required to do this within pre-existing
legal systems.

This brings us to some unique points about what kind of vulnerabilities CL
will always have. Cybersecurity (which now applies to CL) consists of certain
assumptions, either expressed directly or seen but not necessarily explicitly
mentioned by those who practice it111111We will not provide further
argumentation for the assumptions, as these are not stringently codified in
cybersecurity at the time of writing, but matter to how it is deployed in
practice. For example, zero-day exploits may be dangerous, but are something
you must prepare for and no one expects everything to be perfectly
preventable.. One of these is the assumption that there will never be a
perfect defence. We are never reaching an equilibrium of defences and attacks,
and the risk of incursions that can succeed will always exist. A major
consequence of CL either becoming or already being implemented into national
legal systems, is that they can now suffer adversarial failures (from
adversarial attacks). Therefore, ironically, CL as a tool for further
automation may be a target for automated adversarial attacks and loophole
analysis itself.

Like in law, adversary merely refers to the subject being against the target,
but the attacks and failures entail weaknesses which only increase in
sophistication and consequences as the systems or technology used becomes more
complicated and increasingly used. To illustrate this, imagine the difference
between a court system which only partially or does not rely on CL, and one
which does. The former cannot be brought to a halt or fully hijacked by an
adversary. But a court system which makes fully or mostly use of CL definitely
can, unless it has backups and redundancies that enables it to revert to a
non-CL state.

In this sense, CL inherently makes a legal system more vulnerable, in turn
potentially damaging rights, individuals, corporations and even the state
itself. It is from here its relationship with cybersecurity must be
scrutinised, understood and realised, which we can only discuss so much here.

In this paper, we take a very narrow look at one case, which is adversarial
attacks on machine-learning (ML) models and any system that makes use of them.
Computational legal systems that consist partially or fully of these, in any
shape or form, will be vulnerable to attacks on the
classifiers[Fredrikson2015], one of the fundamental parts of these systems,
which all the learning is used on and to form, and attacks to extract the
data[Tramer2016] which they are created on, or the source code of the entire
system. These are all well known, but for CL, they create barriers which make
the practical implementation of these systems potentially unsafe.

Safety here refers to accidents or losses, while cybersecurity refers to
lowering the risk of adversarial failures. It is said that safety is
increasingly covering cybersecurity121212Acknowledged by the European Union,
see page 10 of ‘MDCG 2019-16 Guidance on Cybersecurity for medical devices’,
<https://ec.europa.eu/docsroom/documents/41863>, last accessed 30 June 2022.,
as seen with the IoT example above, and this implies that CL must develop
defences in its technical implementation and logic. Limiting itself to ways
where risks can be mitigated or hazards can be controlled is suitable, and
follow the tradition of anything that absorbs cybersecurity as part of its
being131313Ibid. But any type of product which gains network connectivity
could be a good example. For a general overview in the EU, see
[Banasinski2021].

Section 2 defines what we see as CL and cybersecurity. After this, we discuss
the weaknesses which CL contains in 5 levels. In Section 3, we dive further
into machine-learning specific issues with CL, and we also comment on the
problems which black boxes and authentication bring. Section 4 contains a
brief legal commentary on how cybersecurity is regulated in the EU, with a
focus on its effect on CL, and how the EU’s future AI Act141414Proposal for a
Regulation of the European Parliament and of the Council laying Down
Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and
Amending Certain Union Legislative Acts, COM/2021/206. may affect CL as a
field. We then provide general recommendations for the future, because of the
obvious consequences which CL and cybersecurity together bring in Section 5,
Section 6 has some ideas for future work, and finally, Section 7 concludes the
paper.

##  2 Computational Law and Cybersecurity

CL and cybersecurity naturally fit together, because the former interacts or
exists in some digital space which requires software and/or hardware. This
section combines the two and looks at weaknesses.

###  2.1 Definitions

CL can be defined as:

_The techniques of computational logic, applied to the semantic rules as well
as the data, form the basis of a computational law system.[Love2005, 206]_

CL is considered to be automation of legal reasoning. But from the quote by
Love, we can see that there are essentially two broad definitions, either
purely automation or broadly applying computational logic to law. We will
consider both in this paper.

####  2.1.1 Forms of Computational Law

To better contextualise CL, we mention selected forms here. This is not an
exhaustive list, and some applications will cover several areas.

  * •

Automated assistance within legal systems. TurboTax151515See [Marr2012], for
an example of function and issues. (while not developed by a State) is always
brought up, but the Danish state is another example where income and other
relevant information is automatically sought and filled
in161616https://lifeindenmark.borger.dk/economy-and-tax/the-danish-tax-
system/a-general-introduction-to-the-danish-tax-system, last accessed 30 June
2022. in tax returns and similar forms, leading to automated experiences for
most subjects. On the back-end, this type of CL is also used by the
authorities.

  * •

Contract related fields, like smart contracts[Dixit2022], as well as other
types or derivatives, are considered CL, but are often found in their own sub
genres in relation to the technology which they exist in. These see primarily
use in private law171717However, even public authorities and states will be
private law subjects to the companies they create contractual relations with.,
and will not always be able to execute their contents fully without assistance
from litigation or other means181818For an elaboration of these issues, see
[Pitschas2011, DiAngelo2019, Werbach2017].. The latter comes down to needing
state or legal system assistance to fulfill them, in case of disputes or
disagreements.

  * •

Automated legal decision-making. This can range from public legal decisions on
subsidies, pension, automated payments191919The wrongful texts on Estonian AI
should have referred to the development of automated systems for certain types
of payment, see https://www.just.ee/en/news/estonia-does-not-develop-ai-judge,
last accessed 30 June 2022. or company registration, with a much bigger
potential in the future202020For an overview in public legal systems, see
[Roehl2022].. For future applications, the research community and the industry
as such tends to focus on automated tools or AI judges212121Some examples
could be [Morison2019, Wang2020a, Muhlenbach2020]. but this seems very far-
fetched with the capabilities of current CL systems.

  * •

Quantum CL, if the event of quantum computing occurs, should be its own
separate field [Atik2021, 305]. Primarily due to the power imbalance, but also
the new physical constraints of the system. The former refers to the
substantially increased amount of computing power these will have over
conventional hardware, the latter to the real tangible physical difference
which quantum computers contain, meaning that new threat models and measures
will have to be developed222222There is a wealth of post-quantum research
which is ongoing, see, e.g., [Ahn2022]..

What these all have in common, is that they exist in or as software and make
use of hardware, while containing a level of automation and operate within or
as the legal system. This entails that cybersecurity is an issue for all of
them.

####  2.1.2 Cybersecurity

Cybersecurity can be defined as freedom from adversarial
failures[Leveson1995], but there exists other definitions outside of
traditional security engineering. We chose this because it is narrow and
focused on mitigating or limiting the damage of what failure to defend against
attacks may cause.

As mentioned, this freedom is hard to attain, and it is therefore more of a
goal to strive for, than a goal to reach232323And can be expanded to include
the process of attaining security, see [Radoniewicz2022, 17 - 18]..
Adversaries successfully attacking a system does not always lead to safety
failures, but with the few examples we have mentioned, there is a risk they
will. Safety is the general idea of freedom from accidents or
losses[Leveson1995], where both financial as well as human losses are
included.

Cybersecurity itself is very diverse, and includes everything from encryption,
good practice for building databases[Ibrahim2020] and naming
conventions[Anderson2020, 259-271], to physical elements like air
gaps242424Physical gaps between the system and the internet or just public
space. Guri has written a range of papers on how to mitigate and understand
air gaps, see e.g., [Guri2021]., or access control[Valenzano2014] to both the
servers or robots252525Each area could be analysed specifically with regards
to CL, but this is not the paper for that..

What we focus on is the additional layer of complexity and potential risk
which cybersecurity adds to CL. Very much like safety engineering with the
event of the industrial revolution, cybersecurity needs to be integrated and
considered when designing and also deploying CL systems. Gone are the days
where the application and idea behind CL is purely discussed, as there are
weaknesses which certain types of logic or choices add, giving rise to
practical impacts.

###  2.2 Weaknesses

CL can exist in a theoretical manner without considering cybersecurity, but
when deployed in practice, defences are needed in various ways. Stochastic or
completely unpredictable issues are not considered because of their
extraordinary nature. But the limitations to what can be considered special
enough to warrant lack of liability depends on the jurisdiction, to this, see
our past work[Ludvigsen2022], for an analysis combining private law and
cybersecurity within a Danish context.

####  2.2.1 General Issues

With inspiration from existing literature, we sketch an overarching conceptual
understanding of how cybersecurity matters to CL.

Weakness in CL can be described in the following levels:

  1. 1.

Logic or design.

  2. 2.

Software.

  3. 3.

Hardware.

  4. 4.

System or network.

  5. 5.

Physical.

If the logic or the design of the system does not consider most common
threats, and has not been thoroughly tested, cybersecurity adds a layer of
weakness to all levels. Testing for hardware is the same as software, but
level 4 is different.

Broadly, all the attacks and defences work on individual levels (each
substation), but if these are not applied uniformly, the weakness will consist
of other systems (not those related to the CL), because individual weaknesses
will impose a risk on everyone in the system, making level 4 exceptionally
difficult. This includes the very network(s) that CL use.

Atttacks on the physical layer, level 5, is to usually enable an attack on a
lower level, except for those unique to it such as physical destruction, but
shoulder-surfing262626Refers to snooping, observing literally over a shoulder
or otherwise, to deduce or directly observe passwords or other information.
There exists plenty of research on the matter, see, e.g., [bace2022]. or
physical side-channel attacks272727Physical, unlike digital side-channel
attacks, refers to readings which then allow an adversary to manipulate or
otherwise know something they should not. This could be electromagnetic and so
on, see following article for a great overview with a focus on neural
networks, [Real2021]. are examples of the first.

Most levels are described or specified better in existing literature.
Regardless of this, there are some unique additional explanations needed for
each.

To illustrate the division, we include an visualisation which shows the
interactions between the different levels.

####  2.2.2 Logic

Cybersecurity confer devastating and potentially permanent weaknesses to CL
systems through its choice of logic, as it is what defines what should be
understood and done. This is explained by how it is constructed, e.g., through
the use of machine-learning or expert decision-making, or through choices of
programming language or other technical design decisions. These must be not
only uncovered and possibly mitigated, but some may warrant the removal of the
system entirely, if any of the core weaknesses compromise any basic
cybersecurity attributes at large. For example, machine-learning based systems
may contain weaknesses that allow both data and the code behind it to be
easily retrievable, and if defences against this are not vigilantly updated
and improved throughout the life-cycle of the system, they should not even be
used in the first place.

Another could be the logic behind what constitutes a decision in the
system282828This applies to all types of platforms CL can use., becoming law,
which may be circumventable or possible to be hijacked, either through known
techniques or some discovered by fuzzing292929Fuzzing is the practice of
testing arbitrary or deliberate inputs or actions, and seeing how the system
reacts to it. Some that can cause adversarial attacks may be discovered at the
design stage, others later. For an overview, see [Mcnally2012].. Circumvention
or hijacking can be seen as obfuscation or abuse by known constraints of the
system, which is known in law as:

Loopholes303030There exists an ocean of research on the concept, good
specialised examples could be [espionosa2022, Butler2022]., deliberate non-
compliance without enforcement313131For an empirical study with interesting
perspectives on this, see [Shodunke2022]. or stalling for time[Mazur2022]
through litigation or in public law323232This list in not exhaustive, but
represents common areas.. CL allows circumvention etc., that does not
necessarily require humans or much effort measured by time, which increases
the risk as to whether it will be used333333Cheap externalities usually leads
to increased use, as we have seen with the data sharing, see, e.g.,
[Ichihashi2021]. Worth noting that even if elements make higher profits for
shareholders, this does not mean decreased costs for users or better rights,
see [Deibert2022]. This analogy persists with CL as well, as it will end up
being extremely cheap to attack and abuse the systems going forward..

####  2.2.3 Software

CL is expressed in and usually executes within the software level. Pandora’s
box is therefore opened in regards to weaknesses, anything that is possible
with the very software343434This could be the AI that decides, or one of
several subsystems. that is or the CL resides in, expresses the opportunities
of attack and failures. This could be in the form of denial-of-service of
various kinds, leading to loss of integrity or availability. The latter is
especially important if the CL is the only source of decisions on for example
legal subsidies or permits, and if such a system is taken out, and there are
no redundancies, systemic financial loss is very possible353535As is mental
damage to those who are denied finances to live from, as could be the case
with pension or other types of social services.. Software is usually also the
target of attacks from the other levels, and attacking the CL software is very
possible through other software in the system somewhere, which we focus on
further below.

####  2.2.4 Hardware

Attacking software through hardware is classic, as seen with the Spectre
attack on CPUs363636There exist many good papers on defences and solutions
against it, see e.g., [Kadir2019]., and recently Hertzbleed[Wang2022], allow
for various actions going from hardware to software. In a CL context, this
would allow attackers everything from stealing data, to escalation attacks
that would give them control of the software, or simply a way to destroy it
through ransomware attacks with no way to reverse the encryption of files.

Defences against this are often physical, but as attacks like
Spook.js[Agarwal2022] show, you do not even need to have such access to
perform the attack. In this sense, hardware only adds to the burden of
defending CL systems, and the most important detail of all here, is there is
no way to prevent all side or covert channels from being exploited, as they
often are caused by deliberate decisions in hardware architecture. This is
best seen with the Spectre attack and its derivatives, as the weakness that
allows it also increases CPU performance immensely, and will therefore not
(unless by law) be changed or removed. Analogies to this will exist in many
types of hardware.

####  2.2.5 System

As mentioned above, attacks from other types of software towards the one which
the CL system resides in is the biggest threat overall. Spook.js[Agarwal2022]
does this partially too, as it enables attacks in the browser, which in a CL
context could result in an adversary stealing information from singular users
accessing CL decisions or data which is being provided to such a a system. On
the provider end, operation systems, proprietary software or even malicious
antivirus[Prakash2016] or just plain malware, can all cause one or several
types of software or potentially hardware to be compromised. In this context,
the weakness is the entire infrastructure where we rely on many types of
software and hardware at once.

Solutions like trusted or trustworthy373737Trustworthy must be made to never
fail, whereas trusted merely is the idea that it should not fail, but there is
no assurance it never will, [Anderson2020, 13]. hardware are not enough, and
attackers merely need to exploit or use well known techniques to gain some
type of access, then lie in wait until they can use their position to
potentially reach the system which houses the CL. Even from the perspective of
the user, weaknesses on their side may allow adversaries to manipulate or
violate the confidentiality of CL system, adding an additional attack venue.
In this sense, the system level weakness of CL is most likely the most severe,
as the potential battlefield of different options and tools which adversaries
can use are frankly many times greater than any of the others.

####  2.2.6 Physical

Destroying servers physically, or abusing physical interfaces such as
USB[Thomas2021] all constitute primary reasons to have backups and
redundancies for CL systems. Physical attacks, unless aimed at destruction,
will be means to escalate and gain access to hardware and software. It may
seem like the simplest area, but defending and perhaps expecting employees to
abuse their knowledge of location of servers and so on, is in its own way
paramount in cybersecurity regarding CL systems.

As mentioned, physical side-channel attacks are another matter where defences
must be erected, but many types will regrettably always exist due to their
passive or irreplaceable nature383838Electromagnetic emissions are close to
impossible to prevent efficiently in all systems, which means this can usually
always be deployed. For insights into this, see [Sayakkara2021,
Sayakkara2019]. It could also be audio, anything that can reveal the inner
workings of hardware is possible to make use of., either due to the costs for
preventing any leakage or disabling designs which are necessary for the
functioning of the hardware.

###  2.3 Strengths

On the contrary, CL does contain certain strengths over non-computational
systems. Distributed and massive decision-making393939On the contrary, see
[Cobbe2019a]., simplified and efficient support and search powers (to help
users or citizens) and the other classic benefits of automation
apply404040While this is contentious, and requires more empirical research to
confirm, there are indications towards this direction, see [Zalnieriute2019]..
However, all of it requires the right kind of humans in-the-loop and
legislative framework. Assuming the latter is true in a system, the strengths
of CL are quite clear. Regardless of this, the strengths may not outweigh the
costs, which we will comment on later.

##  3 Cases

To specifically illustrate the interaction between CL and cybersecurity, we
take a look at some examples.

###  3.1 Machine-Learning Specific Issues

Existing taxonomies and threat models for machine-learning (ML) models and
systems sufficiently describe the problem 414141See examples like [Gupta2020,
Papernot2018]. For a practical summary of the current threat model landscape,
see https://docs.microsoft.com/en-us/security/engineering/threat-modeling-
aiml, last accessed 30 June 2022.. But in the context of CL, we can
deliberately go past the CIA triad, confidentiality, integrity and
availability424242Failures in each may overlap during specific successful
attacks. The CIA triad is a core concept in cybersecurity at large, and is
understood in a literal sense., and focus on:

  1. 1.

Poisoning (modification) or possession of the training data.

  2. 2.

Attacks on the model or during the creation of it.

  3. 3.

Hijacking of the communication from or to the model or where it operates.

####  3.1.1 Poisoning or possession of data

For ML based CL systems, this is where the biggest risk lies. Not only can all
training data, personal or not, be stolen and published or used to make a
competitive model, but these can also result in consequences which cannot be
seen until the system has made an unfair or outright dangerous decision. Even
if it did not matter whether the SyRi system was ML based or
not[Rachovitsa2022], an attack with similar or more severe consequences like
in the Post Office case434343[Christie2020]. The loss of lives were a later
consequence, but are still significant., could be caused by deliberately
changing the data to make a system commit to wrong decisions or actions. There
are defences, but vigilance by every human involved, combined with an
understanding that data may be poisoned in the first place, and regular
auditing, must be prioritised to minimize the threat which this constitutes.

####  3.1.2 Model Attacks

Not unlike other types of software, protecting the model at the design stage
is vital, as adversaries can inject or modify parts which may, like above,
cause decisions with very legal or physical consequences due to its CL
nature444444Or simply steal the model and use it for themselves.. Unlike
poisoning, model attacks may be used by journalists and competitors to attempt
to reveal the contents of the model for various purposes. Activism in
cybersecurity is well known454545There exists plenty of research on this, from
many different types of sciences, e.g., [George2018]., and may play a role
here and elsewhere, and with the position CL takes in society, this clash may
be further exacerbated. Like other types of activism, which include activities
that may be considered criminal, this is typically divided into
black[Kwon2018], white[Schrock2016] and grey hatted[Thomas2019], but what
matters the most, is the danger that each possess. This must be considered
before using CL with ML, as the adversaries may cause undesirable outcomes for
the provider, user and citizen respectively, explicitly by revealing the model
illegally, ethically or unethically.

####  3.1.3 Communication Attacks

While less dangerous than the two others, and seen heavily elsewhere in
various ways, communication attacks, if done in a CL system with multiple
steps, could potentially cause some of the same issues as above.
Traditionally, these are divided into several categories, such as replay,
hijacking or modification. The central part is that what is sent and received
is attacked, not necessarily the software itself, in this sense its
actions464646Papers which give an overview on this are, e.g., [Syverson1994,
Xenofontos2022, Huitsing2008].. Contents of the communication could lead to
changes in decisions, or the system could just stop functioning from wrong
received input.

###  3.2 Black Boxes

Black boxes may exist deliberately, or for political or unknown reasons, but
from a cybersecurity perspective, they are not wanted or useful. Trade-secrets
or patents can warrant this, but the general idea is strong security, not
secrecy[Kerckhoffs1883]. CL suffers further from this through the loss of
legitimacy. Transparency and openness about what a system can and should do
will increase trust and confidence and vice versa. The exceptions for this
would be surveillance or military purposes, but even these should still be as
strong as possible [Kerckhoffs1883]. ML based solutions may be Black Boxes for
the two reasons mentioned above, but another could be the complexity of the
neural network which supports it. The question then becomes whether CL systems
should make use of such technology, if it is not possible to comprehend or in
a transparent manner show what goes on inside of it. Considering the weight
which CL systems in the future may have, in both a human and legal manner, it
may simply not be advisable to use them on this basis alone. Outside the
reasoning above, it could also be due to the security worries which a system
you do not know or understand can have.

###  3.3 Authentication

When authentication474747For good taxonomy based overviews, see [El-Hajj2017,
Challa2018]. is automated in a digital manner, it will be considered as being
part of CL, regardless of whether it is incorporated or interpreted into the
legal system.

But, authentication has a known amount of problems, such as proliferation of
new technology, which does not solve its issues[Joos2022], its practical
constraints through circumvention[Blythe2013] or coercion, and lack of
empirical research that really considers the population at large and not just
students or paid individuals as the data source.

CL in authentication refers to the legal identification of the individual,
either through a legal decision or as part of a process, and includes access
to the state, banks and other companies which may constitute critical or
important digital infrastructure in the lives of individuals. The conflict
occurs when the CL is faulty, either deliberately or not, and leads to
injustice or financial losses. Not unlike law in general, there is no quick
solution, and CL must guarantee humans-in-the-loop to function in practice.

In terms of cybersecurity, authentication may be an enabler for fraud through
impersonation or integrity loss. Unlike past means of fraud, authentication in
CL enables these actions to be done remotely and at a massive scale, which
lowers the financial and practical costs and increases the risk of occurrence
dramatically. This must be countered on a design or software level, but is
rarely the focus of developers and users of these systems.

##  4 Cybersecurity Legislation

While there is little concrete and hard law which directly regulates
cybersecurity484848This statement can be contested, on the basis that many
jurisdictions will have overarching rules and legislation which may sound like
it regulates it. The problem with a majority of these is that they leave the
technical questions to guidance or worse, standards, without enforcement by
professionals who actually understand these attributes. A further comparative
legal analysis of this should be done elsewhere., we do have certain EU
legislation which specifies elements of it.

###  4.1 The Cybersecurity Regulation and Future Legislation

Currently, cybersecurity in the EU is regulated on a product to product type
basis via guidance494949If medical devices are used as an example, see our
commentary on the role which cybersecurity guidance has in: [Ludvigsen2022,
Ludvigsen2021]. For other perspectives, see [Banasinski2021, Chiara2022].,
somewhat dedicated rules505050Each type has its own Regulations and
Directives. For a good example, see Regulation 2019/941 on risk-preparedness
in the electricity sector and repealing Directive 2005/89/EC regarding
critical infrastructure, [2019], L 158/1. Note that this is still not
centralised hard law, but delegation. For general commentary, see
[Maglaras2018]., through the NIS directive515151Directive 2016/1148,
concerning measures for a high common level of security of network and
information systems across the Union, [2016] L 194/1. in a national and
fragmented manner, strictly by the Cybersecurity Act525252Through Regulation
2019/81 on ENISA (the European Union Agency for Cybersecurity) and on
information and communications technology cybersecurity certification and
repealing Regulation (EU)No 526/2013 (Cybersecurity Act), [2019] L 151/15.
which only applies to EU institutions, and coordinated in a semi-volunteered
manner by ENISA via the Cybersecurity Act. On the sidelines, we do have
standards and other measures that may be enforced on a contractual or national
basis, but they come with the usual caveats. For CL, this means that until
there is harder law specifying which techniques should and should not be used,
it will exist in a grey area. Yet again, CL will be treated like other types
of software and systems, even if there should perhaps be specialised rules to
accommodate the increased risks caused by failure, not unlike the
cybersecurity requirements that exist for critical infrastructure like
telecommunication535353See [Szczepaniuk2022] for analysis on human factors in
it..

####  4.1.1 Cybersecurity Resilience Act

Earlier in 2022, the Commission called for evidence for a future Cybersecurity
Resilience Act545454See [Ludvigsen2022b].. We provided commentary for this,
but there are some further considerations when discussing CL. Resilience is
usually defined as a system that enables detection, tolerance and recovery
from issues555555Defined more narrowly and precisely in [Anderson2020, 251 -
252].. The problem with CL, is that there must be adequate legislation and
redundancies and mechanisms, not just technical or engineering based
solutions. Real resilience in CL is thus only attained when we have had to
time study the failures and issues of past systems, such as the SyRi
case[Rachovitsa2022], and this is traditionally how we improve both safety and
security. Resilience in CL must therefore be a matter of increased redundancy,
in the form of subsidiary systems or humans which can take over roles of the
CL system in an emergency, strict logic and design which focuses heavily on
preventing adversarial or non-adversarial failures through standards or
existing research on issues and problems with the hardware or software used,
and recovery mechanisms which allow the CL system to keep functioning either
partially or fully after disruption of any kind. The latter should be both
recovery from actions of the system, so that the resources (time, financial)
are not wasted, but also in a classic sense, where it can recover from attacks
which bring down the or partially hijacks the system. There is currently no
interest in creating these in the future Cybersecurity Resilience Act, but
there may be another approach to this problem.

###  4.2 Artificial Intelligence Act

CL will become part of the European product legislation world via the proposed
AI Act565656Proposal for a Regulation of the European Parliament and of the
Council laying Down Harmonised Rules on Artificial Intelligence (Artificial
Intelligence Act) and Amending Certain Union Legislative Acts, COM/2021/206..
This means that both its cybersecurity575757Art 15. and its mechanisms as
artificial intelligence585858Through what kind of risk it constitutes, which a
lot of the time will be high, see Art 6(2) and Annex III. CL can fit many of
the categories specified in Annex III, which means it will mostly be
considered High Risk AI. will be the subject of regulation. How much and and
of which kind remains to be seen, as the Act has not been finalised yet.
Irrespective of the progress of the Act, we can draw some general
considerations which CL bring. The proposed AI Act allows the EU to regulate
CL in a practical manner, and since the type depends on the area which the CL
system functions in, it can either draw rules from there595959Various product
legislation will work alongside the AI Act, see Art 6(2) and Annex II., or
perhaps be granted new guidance which it must follow under the supervision of
National Competent Authorities (authorities). Sadly, if the same type of loose
enforcement seen in other types of product legislation continues, there is a
risk of easy circumvention or mere loose slaps on the wrists for private or
public providers of CL, which may not be very beneficial for the subjects
which will be affected by poorly secured CL systems.

####  4.2.1 Cybersecurity and Resilience

Let us _firstly_ take a closer look at Article 15:

_1\. High-risk AI systems shall be designed and developed in such a way that
they achieve, in the light of their intended purpose, an appropriate level of
accuracy, robustness and cybersecurity, and perform consistently in those
respects throughout their lifecycle. …_

The AI Act considers robustness and cybersecurity here, and the wording is
much clearer and closer to the expectations of those that build these systems.
The Article continues with:

_… 3. High-risk AI systems shall be resilient as regards errors, faults or
inconsistencies that may occur within the system or the environment in which
the system operates, in particular due to their interaction with natural
persons or other systems. The robustness of high-risk AI systems may be
achieved through technical redundancy solutions, which may include backup or
fail-safe plans. High-risk AI systems that continue to learn after being
placed on the market or put into service shall be developed in such a way to
ensure that possibly biased outputs due to outputs used as an input for future
operations (‘feedback loops’) are duly addressed with appropriate mitigation
measures. …_

As CL will mostly be considered High-risk, these rules will apply directly.
The definition of resilience here excludes what is considered robustness, but
in a combined understanding, the Article covers what resilience traditionally
is. All in all, these are clear and well aligned with best practice, and even
include considerations which we did earlier regarding machine-learning. The
main problem then becomes enforcement, and whether those that use or develop
the systems will actually adhere to the rules.

####  4.2.2 Human Oversight

_Secondarily_ , the AI Act also considers human oversight in Article 14, which
from a literal reading does come very close to the kind of rules needed to
regulate CL adequately:

_1\. High-risk AI systems shall be designed and developed in such a way,
including with appropriate human-machine interface tools, that they can be
effectively overseen by natural persons during the period in which the AI
system is in use._

_2\. Human oversight shall aim at preventing or minimising the risks to
health, safety or fundamental rights that may emerge when a high-risk AI
system is used in accordance with its intended purpose or under conditions of
reasonably foreseeable misuse, in particular when such risks persist
notwithstanding the application of other requirements set out in this Chapter.
…_

Article 14(1) and 14(2) entail effective oversight, a rather strong and non-
negotiable position, which warrants transparency and ease of use, perhaps an
issue for CL systems which make use of machine-learning. Minimizing risk via
human oversight is very sound, especially regarding misuse606060Which could be
adversarial failures.. Continuing this, Article 14(3) requires built and
implemented or identified human oversight by the provider. This is a logical
extension of the ideas above. Article 14(4) specifies exactly what the human
oversight should be capable off, effectively professional or system
requirements, including full understanding, tendency and bias awareness,
correct interpretation, knowing when to disregard the AI, and being able to
intervene or stop the system.

Overall, Article 14 has taken all of the best elements of how humans in-the-
loop should be implemented, and expressed it in a fairly comprehensive manner.
Regarding CL, the complexity will exist in how the system handles and
understands the law, and the Article does not resolve this issue, but
definitely leads the way. However, like above, we are left without
opportunities for sanctioning, and there are no concrete definitions to find
here or in the Annexes as to what exact behaviour we are looking for, as this
will be fairly complicated when actually defined narrowly in internal rules or
guidance. This is unlike cybersecurity, where techniques and concepts like
requiring encryption and air gaps are fairly technical, but very possible to
require in hard law.

####  4.2.3 Enforcement

The enforcement structure is quite important to consider when discussing CL,
as there will be a certain overlap between it and AI. Ideally, the AI Act ends
up in a form or is accompanied by additional regulation which enables it to
handle the specific consequences of CL as AI rather well616161If Annex I stays
in its current shape, all CL will be considered AI unless they provide
assistance, but even some of these may pass., but in its current form, this is
not the case. However, the AI Act is equipped with obligations and
requirements, and some means to enforce them, albeit not as harshly or
directly as many had wanted626262See critical analysis in [Raposo2022] for
more.. The obligations are found in Articles 16 - 24 and 61 - 62, specifically
for providers636363Additional obligations for importers and distributors exist
as well, and uniquely, some obligations for users, which is extremely
important for CL systems. This is because unintended use or unexpected
consequences, in litigation, may hinge on who caused it or additionally,
product liability rules.. There are additional enforcement measures in
Articles 63 - 68, which may partially rely on the obligations above. Like
other product legislation, the authority is (usually) not the body which will
technically test whether the product conforms with the rules. Instead, this is
done by Notified Bodies646464Art 33, but be aware of the role which
subsidiaries will play, see Art 34 for this., which will likely be private
organisations, again a parallel to existing systems in, e.g., the medical
device world in the EU. Secondarily, you have the aforementioned National
Competent Authorities656565Art 59. who also act as Notifying
Authorities666666Art 30.. Most of the duty of care or fulfillment of
responsibilities, and even reporting, are put on the shoulders of the
providers. In a CL context, this can be rather dangerous, as the 15 day notice
in Article 62, by virtue of how long it is, can cause massive damage to
individuals wrongfully decided on or assisted by the CL system. The
enforcement mechanisms themselves rely on Article 64, which should give the
authorities access to pretty much everything regarding the AI, but for them to
independently step in and investigate, there must be a risk at a national
level676767Art 65(2).. CL will not fulfill the requirements to be considered a
risk on this stage, so Article 67 or 68 must be used instead. Using the same
evaluation as in Article 65, the authority can force the provider of AI to
withdraw and, if possible, repair and prevent the risk which the AI
poses686868Art 67(1).. Worth noting is the definition of risk or breach of
obligations in Article 67(1):

_… it presents a risk to the health or safety of persons, to the compliance
with obligations under Union or national law intended to protect fundamental
rights or to other aspects of public interest protection …_

For CL, the latter part is intrinsically relevant, and if literally and
loyally followed, could be the supporting stone which CL needs in its proper
deployment. The problem then becomes the authorities themselves, _whether they
will realise and act on this in time_ and other general enforcement issues.
The first is (uniquely) answered in Article 59(4), but this can be abused or
deliberately not followed, causing the authority to become inactive or at
least barely functioning. For the second part, the wording in Article 65(2) is
ambiguous, and can be interpreted in a variety of ways. If we follow the logic
and ideas from existing product legislation696969And since there is nothing
explicitly stating when, where, and how market surveillance could ideally be
done on AI in the AI Act., it must rely on self-reporting707070Art 61 and 62.
or information from other national authorities, or worse, through disclosures
from journalists or researchers in an informal manner. Lastly, the AI Act
delegates the means which the authorities can withdraw or otherwise sanction
the providers to the Member States, which sadly gives little certainty going
forward. In a CL context, this means that ongoing injustice or financial
damage could take a long time be discovered and resolved, and in this sense,
an _ex ante_ review mechanism of the AI with CL features would be much more
adequate.

##  5 Recommendations

If we consider the clash that CL and cybersecurity bring, we can combine it
with common sense arguments from both safety and security engineering, and the
ideas found in the AI Act, forming three recommendations:

  1. 1.

CL systems must accommodate the cybersecurity threats which they can include,
at a design, deployment and post-deployment level (life-cycle), considering
all levels of weakness.

  2. 2.

CL systems should only be used when adequate, as it is not a silver bullet for
every problem it is applied to. Adequate analysis and predictions must be
independently made, with the caveats which this brings through likelihood and
assumptions.

  3. 3.

Humans must still play a role in CL, as they represent plasticity,
adaptability and arbitration, values which no CL system can possess. This is
the only way which the systems can handle unexpected occurrences, systemic
injustice and other difficulties.

###  5.1 Comments

While these are more general and will apply to a general comprehension of CL
and cybersecurity in general, details and the understanding of each must be
commented on briefly.

##### First recommendation.

This concerns itself with defences and mitigation measures towards adversarial
threats. Many of these techniques or measures may follow from existing best
practice, rules or certifications, but for CL, the consequences may be as
severe as when cybersecurity is breached in medical devices717171We discuss
the potential consequences and attacks on medical devices like surgical robots
in [Ludvigsen2022]. or other areas which can damage individuals or human
rights. What follows from this, is taking it gravely seriously, not only
because of the consequences, but also the procedural risk that comes with it,
as failures will warrant lawsuits of various kinds depending on the
jurisdiction. On the other hand, if deployed outside of liberal democratic
states, these considerations can be removed and CL can be used with impunity,
as is seen in China[Liang2022], even if this is not advisable. Note that
wording is soft, accommodate only indicates what is reasonable to expect, and
does not mean that it cannot be merely resilient to the failures. As we noted
earlier, resilience includes recovery, making a broader and inclusive wording
more fitting.

##### Second recommendation.

There are times where machine-learning[Chien2022] or other types of CL may not
be adequate. _Firstly_ , because manual decision-making or the existing
judicial or public legal system is enough to handle to problem, or
_secondarily_ , because the proportionality or the consequences on rights or
finances on individuals may weigh greater than implementing the system.
Regardless of this, choosing to use CL is a political decision, but this
recommendation then warrants _ex post_ criticism and eventual dismantlement.
There is little appreciation of the conservative aspect of non-choice or
denial of certain uses of technology in existing rules, but this is
traditional and well known in safety engineering. There will simply be times
where safety risks outweigh the eventual advantages a system may bring, which
means it should not be used, regardless of the financial or personal interests
in it. This should in particular apply to CL because of its potentially
systemic damage capabilities.

##### Third Recommendation.

Mandating humans in systems is not in any way new, but we ask for it for a
more traditional reason: The qualities which only humans have727272But it may
have negative consequences, if used in a manner to circumvent or otherwise
abuse the good ideas behind human oversight, see [Green2022].. This is, again,
a reference to the role which humans have in safety engineering, where our
ability to adapt in emergencies make us invaluable[Leveson1995, 101 - 102]. Of
course, the caveat to this is where adversarial behaviour or
sludging737373Much great literature exists on this, see [Paul2022], for a
recent overview and example. causes us to not react appropriately. Still,
humans must act as both emergency measures and redundancies, the latter
specifically referring to humans stepping in and overtaking the role which CL
had so far. This could be in decision-making or automated assistance, both
fields where humans used to sit, which means we still have experience and
infrastructure to provide it, albeit much slower than what CL can provide.
Article 14 in EU’s proposed AI Act is a great example of how such a rule could
be designed.

##  6 Future Work

Continuous in-depth analysis of existing CL systems, and acknowledging what
constitutes CL to include as much as possible is, in our view, warranted. This
should include interdisciplinary aspects, as understanding one side or another
is not sufficient to actually comprehend the system as a whole, consequences,
constraints and all.

The role which economics play in both enforcement or lobbying by powerful
corporations or other states should be considered regarding CL too.
Essentially, economic studies which critically consider the potential
loopholes and adversarial actions that powerful actors can use against CL
systems, is wanted on the basis of this paper, perhaps extending existing
economics of security ideas.

Based on our very informal recommendations, expanded analysis on what
constitutes the right situation to deploy CL, a condensed analysis of which
threats for which types of CL exist, and the necessary human roles needed in
CL would be adequate.

And finally, fusing CL research with existing but much older digitalised
law747474Such as the works by Jon Bing. and legal informatics is suitable, as
both fields have common agendas and perceptions of digital law, and would
benefit from the expertise and alternative starting points which each bring.

##  7 Conclusion

As it is clear, CL presents a situation where law gains both the advantages
and disadvantages of the other discipline which inspired its
existence757575Not unlike the issues which philosophy or political directions
has caused of systemic damage to law in the past. An example of this could be
“communist law”, see e.g., [Priban2009].. In this paper, we firstly made a
conceptualisation of cybersecurity in CL specifically, which consists of five
levels. Now that CL absorbs cybersecurity weaknesses which occur in each
layer, we show that there exists mitigation measures for the problems. But we
know from safety and security engineering, that an ongoing arms race is and
will be needed going forward, a sort of constant ongoing development of
defences and considerations, which attempts to mitigate or lessen the damage
that attacks can cause. Building these into CL will be vital, just as it is
with cybersecurity everywhere else.

We find that CL is vulnerable to the same issues which both AI and neural
networks face, if the systems make use of the technology767676This may seem
rather redundant to mention, but these tend to be overlooked in favour of what
advantages the systems may bring. In this sense, making sure that all aspects
of new technology are discussed and understood is a core role of researchers
and the public alike.. The logic and practical implementation in which CL is
implemented, presents a danger which very few other legal disciplines contain.
Thereafter, we show how fragmented and rather softly cybersecurity is
regulated in the EU, which in relation to CL, presents practical and real
problems. But, we then analyse the AI Act, and it provides sound measures
known from elsewhere regarding cybersecurity and human oversight. Regardless
of issues with enforcement, this illustrates promise and understanding for a
future where CL will play a central role in the legal system and in public
consciousness in general. Clearly, the AI Act here combines CL (as AI) and
cybersecurity, and regulates them together in a legal sense. Finally, we
provide three recommendations for CL related to cybersecurity, which
summarised are: Accommodation of threats, adequate use of CL, maintenance of
the human role.

These are general, but from a EU perspective, the AI Act already does a good
job in implementing them all, giving us hope for a future where CL and
cybersecurity can safely melt together.

Figure 1: Visualisation of the 5 weakness levels and their interactions.


Content Production sec:content-production In the terminology used in this
paper, content production refers to the method of establishing all the content
that could potentially appear in a discourse. As described in section 1,
FASTFOOD represents discourse content using descriptive strings and processes;
thus the result of the work described in this chapter is the creation of a
database described in &can_make ← set of all descriptive strings recognised by
FASTFOOD
skills ← the set of all processes recognised by FASTFOOD . This database can
then be accessed by the algorithms described in the following Sections
whenever a user input is made. Section  gives an overview of what is required
of the productive method. Sections  and  introduce food classes, actions and
synonyms, the three types of structure that form the essential components of
the production of $can\\_make$ and $skills$.

##  Overview of Productive Method

One possible way of producing these sets would be to encode them
extensionally. A simple hypothetical extensional definition, termed $skills$*
and $can\\_make$* is given in Figure , along with a demonstration of how it
would be used to represent a recipe for ‘avocado on toast’.

[trim=0.1cm 0 0 0.1cm,clip, totalheight=0.8]scrot˙figures/wrong˙defs.png

Figure : Example extensional definition.

Extensional definition works in this example, but extending the domain to
include multiple recipes drastically increases the size of $can\\_make$ and
$skills$. Due to the specificity of each descriptive string –‘chopped carrot’
describes a separate state of affairs to ‘chopped peeled carrot’–and because
processes are defined on specification of both an input and an output string,
a large number of similar processes would need to be written separately, as
e.g. in Figure .

[trim=0.1cm 0 0 0.1cm,clip,
totalheight=0.4]scrot˙figures/excessive˙extensional˙defs.png

Figure : Example of the many redundant processes that extensional definition
requires.

In its current version FASTFOOD recognises over 400 processes ($can\\_make$
has order greater than 400) and any expansion would further increase this
number. Therefore manual encoding is not a feasible approach. Instead, the
discourse content is produced intensionally by the interaction of several
types of structure, as described in the following two sections.

##  Food Classes and Cooking Actions

This section discusses two types of structure used as part of this intensional
production of $can\\_make$ and $skills$. These structures are cooking actions
and food classes. Intuitively, cooking actions can be thought of as verbs and
food classes as nouns. The current version of FASTFOOD contains 8 cooking
actions, including chop, boil, mash and soak; and a number111This number is
different at different points in the production phase, see Figure 2.2. of food
classes including broccoli, coconut milk, potato and pasta. The formation of
$skills$ requires a specification of how these interact to obtain meaningful
processes. The term ‘meaningful’ is used to describe a process that it is
pertinent for FASTFOOD to consider; this excludes those that may be called
physically impossible, such as chopping coconut milk along with those that may
be possible but are highly impractical, such as soaking pasta. A food class
and cooking action are called compatible if they can together produce a
meaningful process. Syntactically, each action is a set of processes of a
similar form. For example, part of the form for chop is that the output is
‘chopped’ concatenated with the input. These sets are formed separately and
then their elements are all added to $skills$. There are some differences
between actions, such as the inclusion of ‘oil’ in the input of frying and
‘boiling water’ in the input of boiling, however the structure is essentially
the same for each. The general form for chop is given in (2.1). ¡chop-process¿
::= ¡chop-input-pair¿ ¡chop-output-pair¿ ¡chop-time-pair¿ ¡chop-ftime-pair¿
¡chop-direction-pair¿ ¡chop-input-pair¿ ::= ‘input’ ¡choppable-food¿ ¡chop-
output-pair¿ ::= ‘output’ ‘chopped’ ¡choppable-food¿ ¡chop-time¿ ::= ‘time’
integer ¡chop-ftime¿ ::= ‘f time’ 0 ¡chop-direction¿ ::= ‘direction’ ‘chop
the’ ¡choppable-food¿ (2.1) The element referred to as ‘choppable-food’ must,
as per the syntax of a process in (1.1), be a descriptive string. However,
some descriptive strings would produce non-meaningful processes. Thus, it is
specified which descriptive strings are compatible with which actions. Rather
than define a separate set of suitable descriptive strings for each cooking
action, a single set of descriptive strings is used, each accompanied by an
indication as to which cooking actions it is compatible with. For example, the
descriptive string ’avocado’ is compatible with chop and mash, but not with
boil. This information is encoded by a tuple containing a number of Booleans.
Each Boolean indicates compatibility with each of the cooking actions; the
tuple containing ‘avocado’ would have elements indicating compatibility with
chop and mash, and others indicating incompatibility with soak and boil. These
tuples are termed food classes. Based on what has been said so far, food
classes are a descriptive string and 8 Booleans (indicating True or False for
each of the 8 cooking action). However, two modifications to this
characterisation are necessary. Firstly, a binary True/False as to whether a
given descriptive string is compatible with a given cooking action misses the
fact that the same cooking action may take more or less time depending on the
descriptive string. Having True for boil in both ‘lentils’ and ‘spinach’
doesn’t allow for the fact that boiling spinach is considerably quicker than
boiling lentils. So, instead of True or False, the element is either False or
an integer, the latter specifying the ‘time’ value of the process that will be
produced. This element is referred to as an indicator. Secondly, some sets of
food classes are similar, both in their descriptive string and in their
indicators, and it is therefore efficient to define them off the back of one
another. For example the food classes for ‘peeled carrot’ can be defined from
that for ‘raw carrot’ by replacing ‘raw’ with ‘peeled‘ in the descriptive
string, and changing the peel indicator to False (to encode the fact that you
cannot peel a carrot once it has already been peeled). This is done by
splitting the descriptive string into two parts, called the root and the
state. The full syntax for food classes is shown in (2.7). ¡foodclass¿ ::=
¡root¿ ¡state¿ ¡Indicator¿ ¡Indicator¿ ¡Indicator¿ ¡Indicator¿ ¡Indicator¿
¡Indicator¿ ¡Indicator¿ ¡Indicator¿ ¡indicator¿ ::= False — integer (2.7) For
each cooking action, a function is defined that acts on a food class, modifies
the state and certain indicators and produces a new food class. So the
function for peel changes the state to ‘peeled’ and changes the peel indicator
to False. The formal description of these functions is given in Algorithm 2.
The descriptive string of a food class formed by combining its root and state
will be termed its ‘description’.

$f[\text{description}]\leftarrow\text{setofalldescriptivestringsrecognisedbyFASTFOOD}$
(1)

Then, an initial set of food classes, foods, is specified extensionally (20 in
the present version of FASTFOOD) and Algorithm 2, which produces descriptive
strings to be placed in $can\\_make$ and processes to be placed in $skills$,
adds many more food classes to foods along the way. First, the algorithm adds
the descriptive strings for each of these food classes to $can\\_make$. Then
foods is iterated through and for each food class f: the set of cooking
actions is iterated through, and if f is compatible with a food class c, then
a process is produced based on properties of both f and c. The output of each
new process forms the basis for the production of a new food class, which is
then added to foods. Thus, there is a recursive production where food classes
combine with cooking actions to produce processes and also new food
classes.222In theory this recursion could run forever. However, the changes in
indicators are designed in such a way that this never happens. Specifically,
every function sets the Boolean corresponding to itself as False (Chop sets
choppable = False, Boil sets boilable = False etc.) and once it is changed it
is never changed back. Thus the number of iterations for each food class is
limited by the number of actions, in this case 8. This is done by restraint,
not by constraint; it is a factor which must be borne in mind when defining
the effects of the cooking actions. Finally, the output of every element of
$skills$ is added to $can\\_make$. At this point in the production phase,
$skills$ includes all processes formed from the interaction of the appropriate
foods classes and cooking actions, and $can\\_make$ contains all descriptive
strings that are inputs or outputs to these processes. These are then added to
by the structures discussed in Section . Formal description of functions used
to create new food classes, f[choppable], f[boilable] and f[fryable] refer
respectively to f’s indicators for chopping, boiling and frying. chop$x$
$x\leftarrow$ a food class $chopped\\_x\leftarrow x$ will be updated to show
effects of chopping $chopped\\_x[\text{state}]\leftarrow$ ‘chopped’
$chopped\\_x[\text{choppable]}\leftarrow False$ chopped_x boilx $x\leftarrow$a
food class $boiled\\_x\leftarrow x$ will be updated to show effects of boiling
$boiled\\_x[\text{state]}\leftarrow$ ‘boiled’
$boiled\\_x[\text{boilable]}\leftarrow False$ boiled_x fryx $x\leftarrow$a
food class $fried\\_x\leftarrow x$ will be updated to show effects of frying
$fried\\_x[\text{state]}\leftarrow$‘fried’
$fried\\_x[\text{fryable}]\leftarrow False$
$fried\\_x[\text{boilable}]\leftarrow False$ fried_x Formal description of the
algorithm used to produce many descriptive strings and processes, which are
then stored in can_make and skills, f[choppable[, f[boilable] and f[fryable]
refer respectively to f’s indicators for chopping, boiling and frying.
produce_contentfoods $foods\leftarrow$ predefined set of food classes
$chop\leftarrow\emptyset$ , will be updated to contain all chopping-related
processes $boil\leftarrow\emptyset$, will be updated to contain all boiling-
related processes $fry\leftarrow\emptyset$, will be updated to contain all
frying-related processes $skills\leftarrow\emptyset$, will be updated to
contain all chopping- boiling- and frying-related processes
$can\\_make\leftarrow\emptyset$, will be updated to contain recognised
descriptive strings each $f$ in foods add $f[\text{description}]$ to
$can\\_make$ each f in foods $f[\text{choppable}$] is an integer
$chopped\\_f\leftarrow chop(f)$
$chop\\_f\leftarrow(\text{input},f[\text{description}]),(\text{output},chopped\\_f[\text{description}),$
$(\text{time},f[\text{choppable}),(\text{f\text{time}},0),(direction,\text{‘chopthe`$[\text{root})$}$
add $chopped\\_f$ to foods add $chop\\_f$ to $chop$ $f[\text{boilable}]$ is an
integer $boiled\\_f\leftarrow boil(f)$
$boil\\_f\leftarrow(\text{input},{f[\text{description}],\text{‘boilingwater’}}),(\text{output},boiled\\_f[\text{description}]),$
$(\text{time},f[\text{boilable}]),(\text{f\text{time}},f[\text{boilable}]-30),(direction,\text{‘boilthe$f[\text{root}]$for$f[\text{boilable}]$`)}$
add $boiled\\_f$ to foods add $boil\\_f$ to $boil$ $f[\text{fryable}]$ is an
integer $fried\\_f\leftarrow fry(f)$
$boil\\_f\leftarrow(\text{input},f[\text{description}]),(\text{output},fried\\_f[\text{description}]),$
$(\text{time},f[\text{fryable}]),(\text{f\text{time}},f[\text{fryable}]-120),(direction,\text{‘frythe$f[\text{root}]$for$f[\text{fryable}]$`})$
add $fried\\_f$ to foods add $fry\\_f$ to $fry$ $skills\leftarrow chop\cup
boil\cup fry$

##  Synonyms and Ghosts

The ultimate goal of FASTFOOD is to receive a description of a dish as input
and then output a recipe for that dish. In order to accomplish this, it must
be able to understand the input description, i.e. the input description must
an element of $can\\_make$. The production method described in Section
produces only those descriptive strings constructed from combinations of the
given cooking actions and food classes, e.g. ‘chopped carrot’, ‘mashed
chickpeas’; and this excludes the likes of ‘hummus’, ‘ratatouille’ or ‘chips’.
These latter sorts of description, unlike ‘chopped carrot’ have nothing in
their linguistic structure to indicate their physical structure. The
production methods of Section  would apply the cooking action for frying to
the descriptive string ‘sliced potato’ to produce ‘fried sliced potato’; but a
human would instead use the description ‘chips’. Based on Section  FASTFOOD
would be unable to understand this human description. The solution taken is to
encode these anomalous descriptions, like ‘chips’, along with a corresponding
one that could be understood by FASTFOOD, ‘fried sliced potato’. Such a
relation between two strings, one of which is a descriptive string produced by
the algorithm in figure 2.2, is defined syntactically in (2.8). ¡synonym¿ ::=
¡descriptive-string¿ ¡descriptive-string¿ (2.8) For each synonym $(x,y)$ a
process is produced which has $x$ as input, $y$ as output, zero time, zero f
time and empty direction. These processes are termed ghost processes and are
defined syntactically in (2.9). ¡ghost-process¿ ::= ¡input-pair¿¡output-pair¿
¡ghost direction¿ ¡ghost-time¿ ¡ghost-f-time¿ ¡input-pair¿ ::= ‘input’
¡descriptive-string¿ ¡output-pair¿ ::= ‘output’ ¡descriptive-string¿ ¡ghost-
direction¿ ::= ‘direction’ ‘’ ¡ghost-time¿ ::= ‘time’ 0 ¡ghost-f-time¿ ::= ‘f
time’ 0 (2.9) In the case of ‘chips’, this results in Figure .

[trim=0.1cm 0 0 0.1cm,clip,
totalheight=0.23]scrot˙figures/chips˙synonym˙ghost.png

Figure : Example of a synonym and the corresponding ghost process, for ‘chips’
and ‘fried sliced potato’.

A ghost process with input x and output y, allows FASTFOOD to simplify the
search for a recipe for y to the search for a recipe for x. On a request for a
recipe for chips FASTFOOD would, in virtue of G, interpret ‘chips’ as being
capable of being made from ‘fried sliced potatoes’, and then begin searching
for a recipe for the latter. 333A more precise function of these ghosts can be
understood in the context of the content selection algorithm (Section
LABEL:sec:content-selection and the worked example (Section LABEL:sec:worked-
example). G would initially be included in the discourse content, but would be
removed after the content selection stage (see Section LABEL:sec:content-
selection) and would make no appearance in the final generated output. Ghosts
also provide a way of allowing for multiple methods of making the same dish,
though this is an underdeveloped aspect of the current version of FASTFOOD.
Chips may also be made by baking sliced potatoes, and so the synonym S’ and
the ghost G’ are also defined.

[trim=0.1cm 0 0 0.1cm,clip,
totalheight=0.29]scrot˙figures/chips˙synonym˙ghost2.png

Figure : Example of how synonyms and ghosts can allow for multiple ways of
making the same dish: baking or frying potatoes to make chips.

If the content selection algorithm (Section LABEL:sec:content-selection) were
to select $G$ then the final generated discourse would be a recipe for chips
by frying, whereas if it were to select $G’$ the recipe would be by baking. A
possible extension of FASTFOOD would be to choose the most appropriate recipe
by requesting additional user input, i.e. asking the question ‘fried or
baked?’. However, in the current version the selection is made at random. The
existence of both Figure  and Figure  gives rise to the question of whether
‘chips’ and ‘fried sliced potato’ are in fact synonymous after all. If
synonymy is an equivalence relation, and both $S$ and $S’$ both describe such
relations, then this would seem to imply the spurious conclusion that ‘fried
sliced potato’ is a synonym of ‘baked sliced potato’.444This concern is purely
semantic. Synonyms are ordered pairs—ghost processes do not exist in both
directions—and so it would not be possible to use ‘chips’ as an intermediary
and employ both S and S’ to conclude that one can make ‘fried sliced potato’
from ‘baked sliced potato’ or vice versa. 13 See [cruse2002hyponymy] for a
detailed analysis of the hyponym-hypernym relationship. From one perspective,
the semantic relation between two elements of a synonym, is not one of
synonymy at all, but rather hyponymy-hypernymy. Two words that share the same
or a very similar meaning are said to be synonymous, e.g. ‘coriander’ and
‘cilantro’; whereas if the meaning of $A$ is a subset of the meaning of $B$,
e.g. ‘coriander’ and ‘herb’, then $A$ is said to be a hyponym of $B$, and $B$
a hypernym of $A$. Viewed this way, synonyms and ghosts are a method of
expanding the vocabulary of FASTFOOD, e.g. to include ‘chips’, by specifying
‘chips’ as being a hypernym of an already understood description.
Interestingly, there has been theoretical work expounding the importance of
syntagmatic relationships such as hypernyms and hyponyms in the teaching of
new vocabulary (Gao and Xu, 2013; Changhong, 2010); however discussion of such
work is outside the scope of the present paper. Treating ghosts as a method of
vocabulary teaching and weakening the corresponding syntagmatic relation from
synonymy to hyponymy-hypernymy is one feasible interpretation which avoids the
spurious conclusion that ‘fried sliced potato’ is synonymous with ‘baked
sliced potato’. Another feasible interpretation is to view ghosts as
instructions specifying which descriptive strings can be replaced with which
others—as will be seen in Section 3.1, a ghost process is selected for a
recipe when the algorithm replaces a search for its output with a search for
its input: if asked for a recipe for ‘chips’, G would allow FASTFOOD to
replace ‘chips’ with ‘fried sliced potato’, and instead search for a recipe
for the latter. This amounts to adopting a practical understanding of synonymy
based on the ability to substitute one term for another, and touches on some
prior work based off such a definition. By employing various paraphrase
extraction techniques to a number of online corpora, ganitkevitch2013ppdb
(ganitkevitch2013ppdb) have constructed a large database of paraphrases. This
database has allowed a practical investigation of synonymous pairs based on
the assumption that synonyms are words that can act as paraphrases for each
other ([preotiuc2016discovering, pavlick2015inducing, koeva2015paraphrasing].
However, as noted by koeva2015paraphrasing (koeva2015paraphrasing) synonymy
understood in terms of paraphrasing doesn’t always bear the formal properties
of an equivalence relation—for example ‘brain’ and ‘encephalon’ may be thought
of as synonymous, and ‘brain’ could likely be a substitute for ‘encephalon’ in
the medical literature, but the reverse substitution could be incongruous in
less formal discourse, and so the relation does not bear the symmetrical
property. By understanding synonymy in terms of paraphrasing, ‘chips’ could be
interpreted as synonymous with the two descriptive strings above, without
facilitating the spurious conclusion that these two strings are synonymous to
each other. Certain synonyms are built in to FASTFOOD from the
beginning—‘hummus’, ‘chips’,’guacamole’,’pancakes’ etc., but often such a
synonym may be required of the user, for example if a recipe for ‘butter bean
soup’ were requested, then the user would be required to add a synonym like
that in Figure .

[trim=0.1cm 0 0 0.1cm,clip,
totalheight=0.18]scrot˙figures/butterbean˙soup˙synonym.png

Figure : Example of how specifying a synonym to describe the details of the
dish you want to make.

It may seem a shortcoming of the system that a user would have to specify much
of the essential information, but there are two points in response to this.
Firstly, as discussed, synonyms can be interpreted as teaching FASTFOOD new
vocabulary, so repeated use would improve its function: a more experienced
version of the program would already know several synonyms for (and hence,
recipes for) ‘butter bean soup’555FASTFOOD currently chooses randomly between
multiple recipes for the same dish, but an extension of the system would allow
a more intelligent choice, such as one guided by requesting further user
input.. Secondly, the question ‘what exactly do you mean by butter bean soup?’
is exactly the sort of question a menu reader, or personal chef, might ask.
The description ‘butter bean soup’, like ‘chips’ is ambiguous in natural
language, and the hesitation of FASTFOOD is an appropriate response to this
ambiguity. The production method of Section  produces a large number of
descriptive strings of various precise forms. However, the input made by a
user may not be of such a form, and it is essential that FASTFOOD understand
the user input. Two types of structures, synonyms and ghost processes, are
defined to help ensure this understanding. These structures touch on two
theoretical issues; one relating to the role of syntagmatic relations in
vocabulary learning, and the other to differing understandings of the synonymy
relation. After the production of the ghost processes, they are added to
$skills$, and $can\\_make$ is updated accordingly. The result of the methods
of this chapter are the creation of a database consisting of a set of
processes, called $skills$, and a set of descriptive strings, called
$can\\_make$. This database can be stored and then accessed by the algorithms
described in the following chapters whenever a user input is made. These
content production methods need only be run when FASTFOOD is updated; i.e. on
the addition of new food classes, cooking actions, or synonyms.