# ”Forgetting” in Machine Learning and Beyond: A Survey

Alyssa Sha<EMAIL_ADDRESS>1234-5678-9012 Australian National
UniversityCanberraACTAustralia ,  Bernardo Pereira Nunes
<EMAIL_ADDRESS>Australian National UniversityCanberraACTAustralia
and  Armin Haller<EMAIL_ADDRESS>Australian National
UniversityCanberraACTAustralia

(2018)

###### Abstract.

This survey investigates the multifaceted nature of forgetting in machine
learning, drawing insights from neuroscientific research that posits
forgetting as an adaptive function rather than a defect, enhancing the
learning process and preventing overfitting. This survey focuses on the
benefits of forgetting and its applications across various machine learning
sub-fields that can help improve model performance and enhance data privacy.
Moreover, the paper discusses current challenges, future directions, and
ethical considerations regarding the integration of forgetting mechanisms into
machine learning models.

Forgetting, Survey, Machine Learning, Selective forgetting, Reinforcement
Learning, Learning enhancement, Regularisation

††copyright: acmcopyright††journalyear: 2018††doi: XXXXXXX.XXXXXXX††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Applied computing Computers in
other domains††ccs: General and reference Reliability

##  1\. Introduction

Human brain is a complex system, where forgetting serves as a dynamic nature
that allows us to avoid cognitive overload, update information to adapt to
changing environments (Gravitz, 2019), and can potentially enhance our
learning capabilities (Bjork and Bjork, 2019). The advantages of forgetting
have been investigated in various research fields, including education,
philosophy, ecology and linguistics, where forgetting has been found to
contribute significantly to the enhancement of humans’ decision-making,
creativity, and diversity from multiple perspectives.

Forgetting, an intrinsic aspect of human memory, does not naturally occur in
machines, highlighting a fundamental distinction between humans and artificial
systems. In the context of the human brain, overfitting arises when we simply
memorise specific examples rather than generalise patterns from them (Hoel,
2021). This narrow focus can cause inflexibility in our thinking and problem-
solving abilities, as well as lead to erroneous predictions or assumptions
when confronted with unfamiliar situations. Overfitting is also a challenge in
machine learning (ML) (Dietterich, 1995). By mimicking the human brain,
incorporating a forget-and-relearn function into machines has been proposed to
be a powerful paradigm for shaping the learning trajectories of artificial
neural networks (Zhou et al., 2022), as not all content in the past is equally
important for models to remember (Sukhbaatar et al., 2021).

There are different types of forgetting. Selective forgetting involves
selectively ignoring irrelevant or noisy data. This form of forgetting aids in
optimising the model’s memory utilisation (Sukhbaatar et al., 2021), improving
their generalisation ability (Zhou et al., 2022), and enhancing their
adaptability to different datasets and tasks (Zhang et al., 2023a). Besides
model performance, selective forgetting is reinforced by the principles of the
General Data Protection Regulation (GDPR) in Europe and the California
Consumer Privacy Act (CCPA) (Bourtoule et al., 2021) to address the critical
need for compliance with privacy laws and the ethical handling of personal
data without the need for resource-intensive retraining (Zhang et al., 2023b).
On the other hand, detrimental forgetting occurs when a model loses previously
learned information upon learning new information, a phenomenon known as
catastrophic forgetting. This is particularly problematic in continuous
learning environments, where a model needs to retain knowledge over time.
Detrimental forgetting undermines the model’s ability to build upon existing
knowledge, including poor performance on previously learned tasks (French,
1999), the loss of information that could be useful for future tasks, and
biased decision-making.

This survey article only focuses on selective forgetting, which acknowledges
that forgetting certain information can be beneficial by allowing the model to
prioritise and retain more important or relevant information, and protect user
privacy. We then explore how theories and research findings on selective
forgetting from other disciplines can be applied to the field of machine
learning to enhance models’ performance and address data privacy concerns.

This paper surveys the forgetting in machine learning literature to answer the
following research questions:

[RQ1]: How is forgetting manifested in different knowledge areas (e.g.,
psychology, philosophy, neuroscience)? This research question explores the
multifaceted nature of forgetting to inspire the development of new forgetting
models in machine learning approaches.

[RQ2]: How can forgetting be used to comply with data privacy laws, reduce
bias and prioritise relevant information in Machine Learning approaches?

[RQ3]: What are the future research opportunities and challenges associated
with implementing forgetting mechanisms in machine learning? Here, we explore
the current research gaps to advance this field.

Figure 1. Scope and structure of the survey

In the subsequent sections of this article, we aim to systematically address
the aforementioned research questions. Figure 1 presents a conceptual map that
serves as a guide to understand the structure and logical flow of our
discussion throughout various sections in this survey.

We followed a search strategy to thoroughly retrieve and filter the
literature, similar to the method in (Abdelrahman et al., 2023). First, we
curated the search queries [(forgetting OR ”memory loss” OR ”knowledge decay”)
AND machine learning] to match the survey topic and executed them using well-
known scientific search engines including Google Scholar, IEEE Xplore,
ScienceDirect, SpringerLink, and ACM Digital Library. After collecting around
6,500 articles, we then performed a series of inclusion and exclusion criteria
to ensure the relevancy and quality of the collected literature. We filtered
out those that were published before 2015 and were not peer-reviewed to focus
on recent advances. Subsequently, we further filtered the articles by reading
their abstracts to only retain those that discuss the positive side of
forgetting in machine learning instead of purely investigating catastrophic
forgetting. This reduced the pool to 535 articles. We then sorted the results
by multiple factors, including the novelty of the contribution (e.g.,
proposing a new technique or concept versus extending an existing one), the
relevance of the contributions (i.e., the significance of the forgetting
approach in the research pipeline), the quality of the publication venue, and
the depth of evaluation (e.g., the number and sise of evaluated datasets,
performance compared to baselines). Finally, we selected the top 100 articles
from the sorted list to be included in this survey. We further performed
backward snowballing to find historical methods using references from the
included articles and added extra keywords that are not directly related to
forgetting but are very related to the topic, such as ”machine unlearning,”
”adaptive learning,” and ”mitigating negative transfer.” This process added
approximately 50 additional papers to the set of articles covered by this
survey. Data from the selected publications were then extracted using a
standardised form, capturing authors, year of publication, research
objectives, methodology, key findings, and conclusions.

Following the comprehensive search and filtering of the literature on
forgetting in machine learning, we organise the key areas into a structured
taxonomy as attributes. This scheme was devised to systematically categorise
the various research findings, methodologies, and research gaps uncovered
during the review process. Through this taxonomy, we aim to provide a coherent
framework that aids in navigating the complexity of the field, fostering a
deeper comprehension of how forgetting phenomena are tackled across different
machine learning approaches.

The contributions of this research are summarised as follows:

  * •

A comprehensive overview of current research in machine learning where the
concept of ”forgetting” is integrated into model design along with an in-depth
discussion of different types of selective forgetting.

  * •

Bridge the gap between concepts of forgetting from various knowledge areas and
their application in machine learning, deepening our understanding of how
mechanisms of human forgetting can inform the design of machine learning
models.

  * •

Identify the challenges in existing research and ways to enhance model
learning by incorporating forgetting, thereby suggesting directions for future
research.

  * •

Discuss potential biases and ethical issues that may arise with the
implementation of forgetting in machine learning, underscoring the importance
of responsible research in this area.

The remainder of this paper is structured as follows. Section 2 reviews the
literature on the role of forgetting in various disciplines including computer
science, thereby addressing RQ1. In Section 3, the connections of forgetting
across multiple knowledge areas and machine learning are established. Section
4 brings up a taxonomy to summarise different types of forgetting dimensions
and approaches in machine learning, aiming to answer RQ2. The specific
approaches are summarised in Section 5. Finally, Section 6 summarises the
applications and challenges of integrating forgetting into machine learning,
while also highlighting potential biases and risks, leading up to the
exploration of future research directions in Section 7 to address RQ3.

##  2\. Background of Forgetting in different knowledge areas

This section discusses research that focuses on the beneficial aspects of
forgetting such as to decision-making, adaptation, creativity, mental health
of human beings across numerous domains (Gravitz, 2019; Ryan and Frankland,
2022), as well as the generalisation power and evolution of animals (Rodriguez
et al., 1993; Gagliano et al., 2014).

###  2.1. Forgetting in Psychology: ensure effective memorisation and
emotional regulation

The forgetting curve, proposed by Ebbinghaus (Ebbinghaus, 2013), is the most
prominent theory in psychology. It shows a rapid decrease in memory retention
within the first hour after learning new information, which then slows down
over time. Since then, much research has been focused on exploring and
adjusting factors to mitigate forgetting (MacLeod et al., 2009; Averell and
Heathcote, 2011). However, forgetting is also posed in psychology as a natural
and necessary process that allows humans and animals to prioritise information
(Roediger III et al., 2010; Storm, 2011). Jorge Luis Borges illustrated the
role of forgetting and memory for the human experience in ”Funes the
Memorious” (Borges and Hurley, 1999). As Funes is unable to forget anything,
leading to a state where the abundance of detail in his memories makes it
difficult to generalise, abstract, or prioritise information. The story
reflects on the psychological concept of ”cognitive overload” (Sweller, 2011),
where too much information can overwhelm the processing capacity of the
working memory, leading to difficulties in comprehension, decision-making, and
learning.

Emotion regulation, the attempts to control when and how people experience and
express emotions, also involves selective forgetting (Nørby, 2018). Such
forgetting may allow for a focus on positive memories and thereby help form a
mnemonic basis for optimism as well as active and explorative approach
behaviour. Relevant studies in clinical psychology also proposed that deficits
in active forgetting function may be a causative factor in psychopathologies,
like post-traumatic stress disorder, depression, schizophrenia, and obsessive-
compulsive disorder (Hertel et al., 2021; Costanzi et al., 2021). ”Forgetting
and reconstruction hypothesis” (Lee and Magill, 1983) suggests that
”reloading” previously learned knowledge helps improve long-term learning
(Bjork and Bjork, 2019). Human memory is composed by two components:
forgetting and learning (Bjork and Bjork, 2019; Woodward and Bjork, 1971). The
forgetting component is about the loss of retrieval strength and the learning
component relates to the gain in storage strength. Both of them need to be
combined to enable efficient knowledge updates.

In summary, research in psychology emphasises that forgetting is not just a
failure of memory, but a natural and necessary process that helps prioritise
relevant information, which ensures effective memorisation and potentially
contributes to psychological health.

###  2.2. Forgetting in Neuroscience: the magic of engram cells

In neuroscience, forgetting is seen as a form of neuroplasticity that alters
engram cell (cells related to memory association (Van der Linden and Glas,
2000)) accessibility in a manner that is sensitive to mismatches between
expectations and the environment (Ryan and Frankland, 2022). Animals perceive
the world by encoding information through a process of learning, forming
memories that enable adaptive behaviour and promote survival (James, 2007).
However, not all memories are maintained equally, with many forgotten after
learning (Davis2017). Summarised by recent neuroscience research (Ryan and
Frankland, 2022), five biological mechanisms of natural forgetting naturally
occur in our brain: receptor trafficking; spine instability; inhibition;
synapse elimination; and, neurogenesis. The prevalence of forgetting in the
healthy brain suggests that it represents a dynamic nature rather than a flaw
of normal memory functioning (Richards and Frankland, 2017; Nørby, 2020). This
dynamic nature is driven by the process of expectancy violation (Ryan and
Frankland, 2022; Gershman and Uchida, 2019). According to this hypothesis,
when expectancies are reinforced, or new salient information is linked to an
existing memory, positive prediction errors drive reconsolidation and memory
updating. When expectations are violated, negative predictions drive the
forgetting of seemingly inconsistent information (Miller, 2021).

Research also shows that forgetting, as a form of adaptive learning, helps us
to change and adapt in response to experiences, thus preventing experience
overfitting (Gravitz, 2019), which refers to the tendency to rely too heavily
on past experiences, leading to a failure to adapt to new situations. In the
knowledge updating process, where neuroplasticity occurs, forgetting can help
the brain reorganise itself and prune unnecessary connections, making space
for new learning and memory. These findings suggest that it is helpful to
consider learning and forgetting as different aspects of a cognitive process
of belief updating, where an organism is constantly changing how to respond to
environmental affordances (Pignatelli et al., 2019; Ryan and Frankland, 2022).

###  2.3. Forgetting in Education: a form to promote long-term learning

In the field of education, forgetting and learning are not opposing forces but
interconnected processes that shape a student’s educational journey (Bjork and
Bjork, 2019). A common misconception equates immediate performance with
learning, overlooking that true learning can only be assessed after a delay,
whereas performance reflects the short-term recall of knowledge or skills
(Soderstrom and Bjork, 2015). This critical distinction, recognised since the
1930s (Tolman and Honzik, 1930), underscores the need to differentiate between
short-term performance and long-term learning.

The factors that produce forgetting but enhance learning are driven by a few
effects in education. The Context Effect suggests that changing study contexts
can introduce desirable difficulties (Smith et al., 1978), improving recall by
blending subtopics rather than tackling them sequentially (Bjork et al., 2014;
Cooper et al., 2018). The Spacing Effect demonstrates that spaced learning, as
opposed to cramming, leads to better long-term retention by revisiting
material over time (Bjork and Allen, 1970; Teninbaum, 2016). Lastly, the
Generation Effect shows that actively generating information enhances recall
more than passive study (Slamecka and Graf, 1978), although this effect
requires effective feedback to prevent the downsides of forgetting (Kornell et
al., 2009). This is consistent with the principles of ”flow theory” (Sharek
and Wiebe, 2011), where the state of flow occurs when the learning content is
neither too easy nor too frustrating, allowing learners to fully engage in the
learning practice.

The literature in education suggests that forgetting can aid in long-term
learning, but it is essential to regulate the extent of forgetting to help
students stay in the flow channel.

###  2.4. Forgetting in Philosophy: multidimensional self-awareness and moral
duty in humanhood

In philosophy, forgetting is recognised as an essential element of human
existence, contributing to the development of personal identity and the
construction of self-narratives. As stated by Krondofer (Krondorfer, 2008),
”Without forgetting, the human species would have to continuously relive the
past and never fully experience the present moment. Without forgetting, there
would be no future.” Forgetting plays a crucial role in shaping what we
reasonably believe by losing not just evidence but also counter-evidence,
making it significant in evaluating epistemological theories (Frise, 2018;
Dougherty, 2011; Frise, 2017). Normal human forgetting “approximates a virtue”
rather than a vice, which represents a balanced approach between excessive
forgetting and excessive remembering, thus, underlies a normative epistemic
status (Michaelian, 2011). Furthermore, forgetting can be associated with a
gradual wear of memory connections, which helps shape personal identity
(Behan, 1979), yet excessive forgetting might undermine it. Thus,
understanding the specifics of what and when to forget is crucial for
examining these impacts.

Various memory types, such as semantic, episodic and procedural, bring
multiple dimensions of forgetting (Michaelian, 2011; Tulving, 1983).
Philosophically, forgetting is seen both as a mental state and a process over
time. Theories suggest that forgetting occurs either as permanent memory loss
or temporary inaccessibility (Frise, 2018; Harris et al., 2021). Matthew’s
Reductionist Process Theory (Frise, 2018) views forgetting as evolving through
stages, indicating that the depth of forgetting progresses with time, which is
similar to Ebbinghaus’s forgetting curve (Ebbinghaus, 2013). This concept
highlights the dynamic nature of forgetting, emphasising the importance of
time and gradual change in understanding memory loss, rather than viewing it
as a static state. Furthermore, the LEAD theory (Frise, 2018) suggests that
forgetting and relearning are interconnected, allowing for knowledge retention
alongside memory loss (Michaelian, 2016; Bernecker, 2008). It views memory as
an active process that reconstructs past information (Caravà, 2021), where
difficulty accessing memories is seen more as a generative challenge than mere
storage failure.

Feedback mechanisms unconsciously regulate memory retrieval, similar to
control loops in engineering systems that adapt to changing operational
conditions. This generates a metacognitive feeling that acts as feedback on
our cognitive process (Arango-Muñoz, 2013; Arango-Muñoz and Michaelian, 2014),
serving as evidence of forgetting in philosophical views (Halamish et al.,
2011; Michaelian, 2016).

Just as forgetting influences our beliefs, it also allows us to manage aspects
of our identity and facilitate the process of letting go (Bernecker, 2018;
Murray et al., 2019; Broome, 2013; Williamson, 2002). (Basu, 2022) introduced
”Siloing forgetting”, which involves segregating undesirable memories. The
”Fruit of the Poisonous Tree”111The phrase, “fruit of the poisonous tree,” was
first used in 1939 in Nardone v. United States, 308 U.S. 338. theory, from a
moral perspective, discusses the importance of not only believing what is true
but also obtaining truth in the appropriate manner (Basu, 2022). Similarly,
what should be forgotten is often determined not only by the content of the
information but also by how it was acquired in the first place (Basu, 2022;
Munch, [n. d.]).

In summary, research about forgetting in philosophy suggests the existence of
multiple forgetting dimensions. Moreover, the discussions on moral duties of
forgetting inspired us to consider the ethical manner of the collection and
use of data for training machine learning models.

###  2.5. Forgetting in Ecology: trade-offs in animal behaviour and plant
habituation

Forgetting plays a pivotal role in striking a balance between flexibility and
the associated costs incurred by both animals and plants in their behavioural
adaptations (Tello-Ramos et al., 2019). Psychologists have defined cognitive
flexibility as the ability to adjust and reverse contingencies while acquiring
new information (Badre and Wagner, 2006), contributing to enhanced survival
and reproductive success (Snell-Rood, 2013; Dukas, 2004; Lefebvre et al.,
2004). However, cognitive flexibility is not a random occurrence; it is
influenced by environmental conditions, affecting animals’ learning and
forgetting rates. In places with seasonal food changes, animals develop better
spatial memory to survive (Hampton et al., 1998), known as the harsh
environment hypothesis (Hermer et al., 2021; Sonnenberg et al., 2019;
Pravosudov and Clayton, 2002). A potential cost of being cognitively flexible
is handling proactive interference, due to the cue-overload principle (Watkins
and Watkins, 1976). Experiments with mice suggest that this interference can
be reduced by distinctly varying the context of tasks (Rodriguez et al.,
1993), similar to the ”context effect” (Smith et al., 1978).

Similar to animals, plants also adapt to changing environments through a form
of learning and forgetting, as seen in Mimosa pudica’s leaf-folding
habituation to repeated disturbances, indicating their capability to process
and respond to stimuli (Gagliano et al., 2014). These adaptive responses are
essential for their survival, allowing plants to selectively remember and
utilise information to adjust their behavior in response to external cues
(Hemmi and Merkle, 2009; Giles and Rankin, 2009; Eisenstein et al., 2001;
Thorpe, 1956). Nevertheless, plants’ development of adaptive abilities
involves a trade-off between energy expenditure and survival benefits, such as
balancing the predation risk (displaying visible leaves) and energy gain
(opportunity for photosynthesis with open leaves) (Jensen et al., 2011).

Research in ecology suggests the existence of trade-offs: between memory
retention and the acquisition of new memories in animals, and between energy
allocation for survival and growth in plants. Forgetting can be seen as a
mechanism allowing these trade-offs, thus, increasing the adaptability of some
animals and plants.

###  2.6. Forgetting in Linguistics: shaping language evolution and historical
narratives

_”Memories are crafted by oblivion as the outlines of the shore are created by
the sea.”_ One of Auge’s (Augé et al., 2004) most memorable portraits vividly
implies the power of forgetting. Forgetting has been a subject of extensive
investigation within the linguistics, specifically focusing on the evolution
of language itself and the forgetting of its application.

Previous research has identified seven theories of forgetting in language
attrition: repression/suppression, distortion, decay, interference, retrieval
failure, cue dependency and dynamic systems theory (Ecke, 2004; Köpke, 2007).
These theories shed light on the underlying regularity and rationality of the
forgetting process in language attrition. Similar to language acquisition,
speech can also be forgotten. Daniel Heller-Roazen’s ”Echolalias” (Heller-
Roazen, 2005) examines the complex nature of linguistic forgetfulness, where
language forgetting often reflects demographic shifts and changes in
expression. A prime example is the extinction of the Ubykh language in 1992,
marked by the death of its last native speaker (Nettle and Romaine, 2000). At
the meanwhile, the end of one language can lead to the birth of new forms of
expression and creativity, both communally and individually, as illustrated
through Jakobson’s Regression Hypothesis and the transformation of Hebrew in
Jewish scriptures (Keijzer, 2004; Jakobson and Prieto, 1941; Heller-Roazen,
2005). This process underscores the essence of the speaking animal who, in the
act of forgetting, discovers new realms of inspiration.

In linguistic discussions, the concept of forgetting also has implications for
our understanding of history, politics and lifestyle. Early thinkers such as
Plato (Plato, 1962), Augustine (Augustine, 1991) and Shakespeare (Shakespeare,
2019) viewed forgetting as a negative phenomenon, representing a loss of
wisdom. However, contemporary perspectives acknowledge that forgetting can
have positive effects on shaping public memory and rhetoric, offering
political and moral advantages (Margalit, 2002; Volf, 2021; Ricoeur, 2004;
Augé et al., 2004). The process of collective forgetting, which refers to the
public’s selective memory, influences individuals’ interpretations of the past
and future (Margalit, 2002). Additionally, discussions on forgetting prompt
considerations regarding the political biases inherent in training machine
learning models and the potential danger signals they may carry (Muller and
Strohmayer, 2022).

The merit of forgetting is also linked to forgiveness in linguistics,
suggesting that forgiveness comes from intentional and redemptive remembering
(Hawhee and Vivian, 2009). This process, seen as a form of emotional
expression, bridges confession and forgiveness, promoting spiritual equality
as per Kierkegaard’s concept of ”true love” (Volf, 2021; Kierkegaard, 1946).
Additionally, different dimensions of forgetting have been proposed in
rhetoric, namely, Return; Suspense; and Rebeginning, with each representing a
particular temporal duration (Augé et al., 2004).

In conclusion, the research about forgetting language itself in linguistics
showcases the continuity and creativity of expression. And the forgetting of
language use, at the same time, shapes our perspectives on history, politics
and the emotion of forgiveness.

###  2.7. The review of forgetting in machine learning

Forgetting in machine learning refers to the intentional process of removing
specific data or influences from a model’s training history, often to update
its knowledge for better performance (Zhou et al., 2022), comply with privacy
requirements (Cao and Yang, 2015) or correct biases (Peng et al., 2021b).
There are review studies that summarise machine learning approaches associated
with ”forgetting” in several different ways. For example, (Nguyen et al.,
2022a; Zhang et al., 2023b) summarise forgetting approaches in machine
learning under the machine unlearning domain, which is driven by user privacy
concerns; (Muller and Strohmayer, 2022) provides a comprehensive analysis of
how forgetting has been applied in HCI studies; (Wang et al., 2023) reviews
forgetting approaches from the deep learning perspective, while others touch
on ”forgetting” indirectly by summarising downstream machine learning tasks
that are highly relevant to forgetting (Zhang et al., 2023a; Pan and Yang,
2009; Mehrabi et al., 2021).

However, these existing reviews on forgetting in ML fall short in the
following gaps: 1) The discussion from many of them only stays on the
technical level (Nguyen et al., 2022a; Wang et al., 2023), thus lacking the
linkage with forgetting theories embedded in other disciplines. 2) Some
machine learning reviews only cover forgetting approaches for one narrow
problem (Zhang et al., 2023b, a; Muller and Strohmayer, 2022), which misses a
complete perspective to summarise the findings. To address the above research
gaps, this survey aims to establish the linkage from forgetting methodologies
in machine learning to forgetting theories in other domains and propose a
comprehensive taxonomy for a more complete summary of forgetting involving
multiple ML tasks.

##  3\. Expand The linkage of forgetting across knowledge areas to machine
learning

We adopted a qualitative data analysis approach rooted in the grounded theory
to helps us to extract insights about forgetting across multiple domains
systematically and build the linkage among them. The principles of grounded
theory enable us to explore a range of themes and patterns in the data,
identify underlying relationships among entities, and categorise them into a
theoretical framework that explains the forgetting phenomena (Corbin and
Strauss, 1990), which helps to ensure the research is grounded in the data and
thus can generate valid and reliable findings (Walker and Myrick, 2006; Noble
and Mitchell, 2016). Following the grounded theory, our qualitative data
analysis is broken down into three stages: open coding, axial coding and
selective coding (Oktay, 2012). The overall workflow of qualitative data
analysis using grounded theory is shown in Figure 2.

Figure 2. The workflow of qualitative data analysis using grounded theory

We began by gathering qualitative data (peer-reviewed research papers) on how
forgetting occurs in fields such as cognitive psychology, neuroscience, and
linguistics (Noble and Mitchell, 2016; Dickersin et al., 1994). During the
open coding stage, we identify and label concepts, debates, and themes within
the papers, breaking down the data into discrete parts for comparison. In the
axial coding stage, we reassemble the data by making connections among the
fragmented ideas, building categories that relate the labelled codes through a
mix of inductive and deductive reasoning. This process reveals patterns and
similar views across different fields, such as the emphasis on feedback
signals in both neuroscience and philosophy (Arango-Muñoz, 2013; Ryan and
Frankland, 2022), converging on themes like the generative power and
adaptability of memory systems (Gravitz, 2019; Caravà, 2021). During the
selective coding stage, we integrate and refine these categories to construct
coherent themes representing our findings on forgetting. For example, we
identify topics such as the dimensions of forgetting and inherent biases are
frequently discussed across disciplines during this stage. We continuously
incorporate new data to refine our framework, iterating until we achieve a
comprehensive understanding of forgetting from various perspectives.

Building on selective coding, we extend the key categories of research on
forgetting from other domains to the field of machine learning, as presented
in Figure 3. The highlights from background review have been categorised into
four sections below to expand our discussion, which inspired us to establish
linkage and develop more concrete ideas in the exploration of the role of
forgetting in machine learning.

Figure 3. Quantitative analysis on forgetting across different domains using
grounded theory

###  3.1. Actual forgetting is rational forgetting

The first common denominator in the literature is the assessment of the
phenomenon of the presence of forgetting. Hegel’s famous statement (Hegel,
1998), ”What is actual is rational”, reflects the recognition of forgetting as
a significant aspect of human cognition. Rather than being seen as a flaw,
forgetting is viewed as a legitimate and normal functionality. Neuroscientific
research supports this perspective by attributing forgetting to the natural
process of engram cell plasticity in the brain (Ryan and Frankland, 2022).
Philosophical theories also contribute to our understanding of forgetting,
presenting it as a mean to shape personal identity and establish a normative
epistemic status (Frise, 2018). Additionally, in rhetoric, forgetting is
recognised as a normal process that allows us to mirror past traces and serves
as a fundamental condition for constructing historical narratives (Hawhee and
Vivian, 2009). Considering these insights, it becomes evident that forgetting
in computational models must also have its place. This hypothesis encourages
us to take a positive stance on the important role of forgetting not only in
human learning but also in machine learning.

###  3.2. Unlocking Potential: The Role of Forgetting in Adaptability,
Generalisation, and Creativity

The rationality of forgetting lies in its positive impact on the
sustainability of species, both physiologically and psychologically,
particularly regarding its contributions to adaptability, generalisation, and
creativity.

Adaptability is emphasised in animal behaviour research, where forgetting is
proven that can help decrease proactive interference and thus help animals to
make flexible reactions to the changing environment according to the cue-
overload principle (Tello-Ramos et al., 2019). Similarly, plants also exhibit
forgetting behaviour, which aids in the storage and retrieval of biological
information, allowing them to provide adaptive responses to their surroundings
(Gagliano et al., 2014). The same purpose is proposed in neuroscience as well,
where forgetting, as a form of adaptive learning, prevents individuals from
experiencing overfitting (Gravitz, 2019). Notably, the challenge of
overfitting is not limited to natural organisms but also pertains to machine
learning. The observations from ecology and neuroscience highlight the broader
significance of forgetting in promoting adaptability and addressing challenges
in different domains, contributing to the sustainability and efficiency of
computational systems.

The facilitation of generalisation ability and creativity through forgetting
is emphasised across various disciplines. In cognitive psychology, forgetting
is characterised as being linked to retrieval rather than storage capabilities
(Bjork and Bjork, 2020). From educational perspectives, a certain degree of
forgetting can compel us to form new connections between new cues and existing
memories, thereby enhancing generalisation and summarisation abilities (Bjork
and Bjork, 2019). Moreover, at an individual level, the forgetting of speech
can serve as a catalyst for literary, philosophical, and artistic creations
from linguistic perspective (Heller-Roazen, 2005). Leveraging the concept of
forgetting may thus enable the development of more advanced generalisation and
innovation capabilities for machine learning.

###  3.3. How the process of forgetting is shaped?

Once the benefits of forgetting have been elucidated, various disciplines have
offered insights into effective strategies for forgetting. One crucial
approach is the establishment of feedback mechanisms or control and monitoring
loops. Neuroscience research emphasizes that forgetting is a dynamic system
that relies on feedback signals, which represent our expectations of the
environment (Ryan and Frankland, 2022). In philosophy, these signals manifest
as cognitive feelings, serving as phenomenological feedback that guides us in
what and how to forget (Arango-Muñoz, 2013; Arango-Muñoz and Michaelian,
2014).

The process of forgetting is intricately linked to different dimensions, as
emphasised in philosophy and linguistics. Philosophical theories classify
forgetting based on its temporal dimensions, distinguishing between states and
processes (Frise, 2018). Moreover, forgetting can be categorised based on the
way it occurs, encompassing temporary loss, retrieval failure, or permanent
forgetting (Michaelian, 2011; Tulving, 1983). Additionally, the dimensions of
forgetting intersect with moral duty, giving rise to classifications such as
No-trace forgetting, archival forgetting, and siloing forgetting (Basu, 2022).
The definition and classification of these diverse dimensions of forgetting
inspire us to contemplate the myriad modes of archiving within the field of
machine learning and the consequential implications they entail.

###  3.4. Challenges in forgetting

Although our focus primarily revolves around the benefits of forgetting, it is
crucial to acknowledge the accompanying issues and challenges. One notable
concern arises from the ethical implications discussed in philosophy and
linguistics. Forgetting, as a reflection of remembrance, significantly shapes
our perception of ethics, moral duty, history, politics, and the past and
future (Basu, 2022; Frise, 2018). Given this influence, it becomes imperative
to exercise caution when dealing with the forgetting behavior of machine
learning systems. Different types of data and their weighting during model
training can impact the reasoning and decision-making abilities of the model.
Hence, it is essential to carefully consider the ethical concerns and biases
associated with forgetting in this context. Another significant aspect to
consider is the boundary of fortuitous forgetting, as not all instances of
forgetting yield positive outcomes. In educational theories, excessive
forgetting can render problem-solving overly challenging, potentially blocking
students’ flow channel of learning thus creating negative experiences (Vann
and Tawfik, 2020). As mentioned before, forgetting is a method for animals to
achieve trade-off between memory retention and cognitive flexibility to adapt
to changing environments. However, excessive forgetfulness in a stable
environment can lead animals to forget vital information such as their homes
or the location of stored food, making them vulnerable to predation (Tello-
Ramos et al., 2019). Similarly, unfit archival practices may result in
catastrophic forgetting and nonsensical decision-making for machine learning
models, which presents us with the new challenge of determining suitable
boundaries for forgetting in different types of training tasks, ensuring that
forgetting is employed effectively and appropriately.

##  4\. A taxonomy of forgetting in machine learning

Built upon the expanded linkage from forgetting in other knowledge areas, we
present a taxonomy (Figure 4) with the main aspects of forgetting in machine
learning. This taxonomy serves as a framework for understanding the different
dimensions and approaches through which machine learning models can ’forget’
or ’unlearn’ information. This exploration will not only clarify the concepts
but also highlight the practical implications and challenges associated with
each. By doing so, we aim to provide clarity and structure to the complex
process of forgetting in machine learning, setting the stage for further
investigation and application in this field.

Figure 4. A taxonomy of forgetting in machine learning

###  4.1. The dimensions of forgetting in Machine Learning

Drawing upon prior research from different domains, this paper summarise the
dimensions of forgetting in machine learning into three categories, delineated
by the content, recoverability, and extent of forgetting as shown in Figure 4.

####  4.1.1. Content of forgetting

The content of forgetting in machine learning models is influenced by various
removal requests, which in turn guide the strategies for forgetting. The types
of content that can be removed span multiple levels, each representing a
distinct dimension of forgetting. Item-level Forgetting is concerned with the
exclusion of specific data instances from the training dataset, addressing the
most common requests for forgetting. Removal requests may arise not just for
individual data items, but for clusters of data sharing common features or
labels, leading to feature-level forgetting (Warnecke et al., 2021). This is
particularly critical when data attributes become outdated or misleading.
Class-level forgetting encompasses the targeted removal of entire classes from
the model, a process that can be intricate due to computational requirements
and its potential effects on the model’s efficacy (Tanha et al., 2020). For
example, in face recognition systems, each class represents an individual’s
facial data, and opting out necessitates the elimination of that data class.
Task-level forgetting is essential in continual or lifelong learning, where
there might be requests to remove data linked to a particular task. Consider a
robot trained for home patient care which must later discard that task from
its memory after the patient’s recovery (Liu et al., 2022b). Lastly, Stream-
level forgetting addresses the challenge of handling vast volumes of online
data with the constraint of limited storage (Nguyen et al., 2017), which
involves managing a sequence of data removal requests.

####  4.1.2. Recoverability of forgetting

The categorisation of dimensions based on approaches to forgetting finds its
origins in the field of philosophy, where forgetting is classified into
distinct dimensions such as archival forgetting, no-trace forgetting, and
siloing forgetting (Basu, 2022). Each of these categories is associated with
unique moral duties, as discussed in 3.3. In the context of machine learning,
a similar categorisation of dimensions can be observed concerning the
approaches to processing forgetting. These dimensions are typically
distinguished as recoverable forgetting and irrecoverable forgetting.

Irrecoverable forgetting, often referred to as non-trace forgetting, involves
the deliberate removal or deletion of specific knowledge stored within the
model. This action is typically undertaken for reasons related to privacy or
copyright concerns. It is essential to note that once this knowledge is
expunged, it cannot be recovered or restored. The elimination of extraneous or
irrelevant information further enhances the scalability of transformers,
enabling them to allocate resources more efficiently to attend to a broader
array of memories (Sukhbaatar et al., 2021).

On the other hand, recoverable forgetting, also known as archival forgetting,
pertains to a scenario in which temporarily forgotten information or knowledge
can be restored or retrieved through appropriate methods or processes. In
essence, there exists a means to recover the previously forgotten information.
This approach explicitly allows for the extracted knowledge from pre-trained
models to be temporarily set aside and subsequently reintroduced into the
model whenever required. The overarching objective is to equip models with a
flexible learning strategy that encompasses forgetting as well. This
flexibility grants users a high degree of control over task-specific or
sample-specific knowledge while simultaneously ensuring intellectual property
protection, as discussed by (Ye et al., 2022). Recoverable forgetting
encompasses several subdomains including ephemeral, inhibitory and gradient
forgetting. Ephemeral Forgetting pertains to transient or temporary
forgetting, where the model’s memory of specific items diminishes over time
but can potentially be restored. Inhibitory Forgetting involves situations
where the model puts a block on certain memories on purpose, making it hard to
recall those specific details unless the block is removed. Lastly, gradient
forgetting refers to the model forgetting different things at different rates
because of how it learns or optimises its knowledge which could be adjusted.

####  4.1.3. Extent of forgetting

The dimensions of machine forgetting can be delineated into two categories
based on the extent of forgetting: exact forgetting and approximate
forgetting. Exact forgetting entails that the model’s outputs, after the
removal of a sample, remain the same as those of a model that was never
trained on the removed sample. In essence, after exact forgetting, not only is
the specified data eliminated, but also its influence on the model is entirely
eradicated. From a model-building standpoint, a reasonable criterion for exact
forgetting is that the system’s state is adjusted to replicate the state it
would have in the complete absence of the forgotten data (Yan et al., 2022).

Nevertheless, approximate forgetting involves adjustments to the model and
dataset that eliminate the need to retrain from scratch. This forgetting
method entails diminishing the influence or significance of specific
information within a machine learning model, rather than erasing it entirely
(Izzo et al., 2021). Within this approach, the model retains a downscaled or
less potent version of the forgotten knowledge. The boundaries and extent of
forgetting are less stringent in approximate forgetting as compared to exact
forgetting. Approximate forgetting is often used when one wants to maintain a
certain level of adaptability or the ability to revisit and fine-tune
previously learned information while reducing its current influence on the
model.

###  4.2. The types of forgetting approaches in Machine Learning

This survey summarise the forgetting approaches in machine learning into two
branches based on their motivations: active and passive forgetting approaches.
The first branch, where machine learning models are designed to actively
perform forgetting with the primary goal of updating information to improve
performance, is classified as active forgetting approaches. The second branch,
where forgetting occurs only in response to specific requests to protect user
privacy and enhance security, is categorised as passive forgetting approaches.
The different workflows of each are shown in Figure 5. To provide clearer
narratives, the following sections will follow the taxonomy to summarise the
categories of active forgetting approaches and passive forgetting categories
separately.

Figure 5. The workflow of active and passive forgetting approaches

####  4.2.1. Active forgetting

One purpose of using an active forgetting approach to improve model
performance is to enhance the adaptability of models when switching tasks.
This requires the models to continue performing well on different or unseen
data. This is a central task in transfer learning, which enhances machine
learning models by utilising data from different domains, aiming to overcome
the limitations due to disparities between training and testing data (Pan and
Yang, 2009). However, this approach faces the challenge of Negative Transfer
(NT), where leveraging data from the source domain negatively impacts
performance in the target domain (Rosenstein et al., 2005). This issue, which
is also rooted in educational psychology (Chen and Daehler, 1989; Watson,
1938; Chen, 2020), underscores the importance of carefully selecting related
source data to avoid detrimental effects on learning. Recent research has
focused on the concept of ’forgetting’ irrelevant information as a strategy to
mitigate NT and improve knowledge transfer (Wang et al., 2019a; Zhang et al.,
2023a; Xie et al., 2017), indicating a promising research direction. This
involves aligning TL techniques with broader theories of forgetting to improve
model adaptability. Such approaches include estimating knowledge domain
similarity, enhancing data and model transferability, and refining the
training process to mitigate negative transfer.

One objective for ML models is not just to perform well on the training data
but to develop generalisation ability and apply that to unseen data. The ideal
model with generalisation ability would extract the underlying patterns or
concepts that are universally true, rather than memorising specific details or
statistical quirks in the training data. Active forgetting approach plays a
critical role in achieving such optimal generalisation. More information does
not equal to more learning. Instead of retaining all information, a model can
employ selective forgetting to discard noise and irrelevant details. This
strategy helps prevent the model from overfitting to the training data by
”forgetting” information that limits its understanding of the problem space.
Employing an iterative process of forgetting and relearning enables the model
to refine its focus on significant patterns and relationships, thereby
improving its performance on a variety of unfamiliar datasets.

When comparing human memory capacity to that of machines, it is often argued
that humans forget due to intrinsic limitations in brain capacity, whereas
machines, equipped with much larger memory capacities, supposedly have no need
to forget (Korteling et al., 2021; Moravec, 1988). This viewpoint, however,
simplifies the complex reality. Given that computational resources for ML
models are finite, it becomes crucial for them to forget in order to allocate
resources more efficiently. This is particularly true for Transformer
architectures, such as those used in the GPT series, BERT, and other NLP
models, which, despite their high performance across tasks, face significant
memory constraints. Their quadratic complexity in relation to sequence length,
due to the self-attention mechanism, makes processing long sequences
challenging without segmenting inputs or employing approximations. Considering
the attention context as a form of random-access memory where each token
occupies a slot, the memory size and associated overhead increase linearly
with sequence length. To overcome these memory challenges in Transformer
architectures, active forgetting emerges as a vital strategy. It allows for
scaling transformers to achieve lossless compression through improved
attention efficiency and strategies to reduce memory length.

####  4.2.2. Passive forgetting

Different from active forgetting which is performance-driven, passive
forgetting process is triggered by removal requests due to data privacy,
security or usability concerns (Nguyen et al., 2022a). To comply with a
removal request, the computer system must eliminate all user data and ’forget’
any impact on the models that were trained using that data. Completely
eliminating the impact of data marked for deletion is complex. Beyond just
removing it from the databases, its impact on artefacts like trained machine
learning models must also be nullified. Machine unlearning, as a passive
forgetting approach, has been introduced to address the data removal challenge
(Zhang et al., 2023b). Google’s first machine unlearning challenge highlighted
that the ideal unlearning algorithm should not only eradicate the influence of
a specific dataset but also retain the model’s performance (Pedregosa and
Triantafillou, 2023).

The unlearning problem is firstly formalised as a game between two entities,
the service provider and the user population by Bourtoule et al (Bourtoule et
al., 2021). The service provider $S$ can be an organisation that can collect
various users’ information and the collected information are stored in the
form of a dataset $D$. The service provider $S$ uses these data to train a
machine learning model through $A(\cdot)$ as a way to provide an intelligent
service to the user $U$. Then, according to the GDPR (Mantelero, 2013), any
user $u$ in $U$ has the right to request the removal of part of the data
$D_{f}$ from $D$, and the service provider $S$ must execute it. Thus, the
service provider $S$ must modify the model parameters $w=A(D)$ to generate a
new model with parameters $w_{u}=U(\cdot)$, which represents a model without
trained data $D_{f}$.

Currently, there are two strategies to perform unlearning on machine learning
models (Zhang et al., 2023b; Nguyen et al., 2022a; Wang et al., 2023). A
straightforward way to produce this unlearned model is to retrain the model on
an adjusted training set that excludes the samples from the forget set,
commonly termed ”exact unlearning” (complete, perfect) (Nguyen et al., 2022a).
However, this is not always a viable option, as retraining deep models can be
computationally expensive. An ideal unlearning algorithm would instead use the
already-trained model as a starting point and efficiently make adjustments to
remove the influence of the requested data. Therefore, a more efficient way is
to modify the machine learning model and dataset to achieve an ”approximate
unlearning” (certified, bounded) effect (Pedregosa and Triantafillou, 2023).
Current definitions of machine unlearning seek to make the output of
approximate unlearning as close as possible to the output of exact unlearning
(Zhang et al., 2023b; Pedregosa and Triantafillou, 2023). The differences
between exact unlearning (green line) and approximate unlearning (orange line)
are presented in Figure 6 which consists of the primary stages of machine
unlearning.

Figure 6. Machine Unlearning Pipeline: An unlearning algorithm takes as input
a pre-trained model and one or more samples from the train set to unlearn (the
”forget set”). From the model, forget set, and retain set, the unlearning
algorithm produces an updated model. An ideal unlearning algorithm produces a
model that is indistinguishable from the model trained without the forget set.

##  5\. Forgetting Approaches

This section summarises specific existing approaches following the taxonomy
proposed in section 4. The approaches are mapped into the approach categories
and the dimensions we proposed, as shown in Table 1. Note that many approaches
could be applied to various dimensions depending on the task requirement and
operating ways. Here, we only tick dimensions that it most possibly could be
applied to. It only serves as an overview for readers to navigate literature
more easily.

|  |  |  | Content of forgetting | Recoverability of forgetting | Extent of forgetting |
---|---|---|---|---|---|---|---
Taxonomy-parent | Taxonomy-node | Approach | Sub-approach | Item removal | Feature removal | Class removal | Task removal | Stream removal | Recoverable | Irrecoverable | Exact | Approximate | Related work
|  |  | Feature Statistics-Based |  | $\checkmark$ |  |  |  | NA | NA | NA | NA | (Gretton et al., 2012; Bernico et al., 2019; Sun et al., 2021; Zhang et al., 2022; Lin and Jung, 2017; Ying et al., 2018; Gong et al., 2012; Zhang and Wu, 2020)
|  |  | Test Prediction-Based |  |  |  | $\checkmark$ |  | NA | NA | NA | NA | (Xie et al., 2017; Sun et al., 2018; Ben-David et al., 2010; Xu et al., 2018; Wu and He, 2020; Feng et al., 2021)
|  | Domain Similarity Estimation Approaches | Fine-Tuning-Based | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |  | NA | NA | NA | NA | (Tran et al., 2019; Nguyen et al., 2020a; Afridi et al., 2018; Huang et al., 2022; Bao et al., 2019; Madasu and Vijjini, 2021)
3-40.2 |  |  | Data Transferability Enhancement | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | (Ahmed et al., 2021; Turrisi et al., 2022; Kim and Kim, 2020; Zhu et al., 2020; Seah et al., 2012; Wu, 2018; Xu et al., 2020; Yang et al., 2022; de Mathelin et al., 2021; Li et al., 2022a; Chen et al., 2019; Yosinski et al., 2014; Lipton et al., 2018; You et al., 2019)
|  |  | Model Transferability Enhancement |  |  |  | $\checkmark$ |  |  | $\checkmark$ |  | $\checkmark$ | (Madry et al., 2017; Zhang et al., 2021; Liu et al., 2021; Han et al., 2022; You et al., 2020; Ishii and Sugiyama, 2021; Gao et al., 2021)
|  |  | Training Process Enhancement |  |  |  | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ | (Long et al., 2013, 2015; Yoon and Li, 2018; Xiao and Zhang, 2021; Saito et al., 2021; Liang et al., 2021; Wan et al., 2019; Yoon and Li, 2018; Wan et al., 2019; Xiao and Zhang, 2021; Liang et al., 2021)
| Adaptability | NT mitigation approaches | Target Prediction Enhancement |  | $\checkmark$ | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ |  | $\checkmark$ | (Pei et al., 2018; Gui et al., 2018; Wang and Breckon, 2020; Liang et al., 2020; Tian et al., 2020; Jin et al., 2020)
|  |  | Knowledge Evolution (KE) | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | (Taha et al., 2021)
|  |  | Iterative Magnitude Pruning (IMP) | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | (Frankle and Carbin, 2018; Frankle et al., 2019)
|  |  | Reinitializing Final Layer (RIFLE) | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | (Li et al., 2020a)
|  |  | Later-Layer Forgetting (LLF) | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | $\checkmark$ |  | (Zhou et al., 2022)
|  |  | Iterated Learning | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | (Kirby, 2001; Kirby et al., 2014; Hoang et al., 2018; Ren et al., 2020a)
| Generalisation | Iterative training | Partial Balanced Forgetting (PBF) | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | $\checkmark$ |  | (Li and Bowling, 2019; Zhou et al., 2022)
|  |  | Dynamic attention spans |  | $\checkmark$ |  | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | (Sukhbaatar et al., 2019; Correia et al., 2019; Wu et al., 2022b)
|  | Improve attention’s efficiency | Attention with Bounded-Memory (ABC) |  | $\checkmark$ |  |  |  | $\checkmark$ | $\checkmark$ |  | $\checkmark$ | (Peng et al., 2021b, a)
3-40.2 |  |  | Expire-span | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | $\checkmark$ |  | (Sukhbaatar et al., 2021)
|  |  | Compressor (RECOMP, TRIME) | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | (Xu et al., 2023; Zhong et al., 2022)
Active Forgetting | Compression | Lossless compression | Neural Attentive Filtering (AI2V++) | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | (Gaiger et al., 2023)
|  | Statistical Query Learning | SQ learning | $\checkmark$ |  |  |  |  |  |  | $\checkmark$ |  | (Cao and Yang, 2015)
3-40.2 |  |  | SISA | $\checkmark$ |  |  |  |  | $\checkmark$ |  | $\checkmark$ |  | (Bourtoule et al., 2021; Aldaghri et al., 2021; He et al., 2021)
|  |  | Graph Partition | $\checkmark$ |  |  |  |  | $\checkmark$ |  | $\checkmark$ |  | (Chen et al., 2022b)
|  | Data Partitioning | Data partition in Recommender systems | $\checkmark$ |  |  |  |  | $\checkmark$ |  | $\checkmark$ |  | (Chen et al., 2022a)
3-40.2 |  |  | Deltagrad |  |  |  | $\checkmark$ |  | $\checkmark$ |  | $\checkmark$ |  | (Wu et al., 2020a)
|  | Rapid Retraining | ARCANE |  |  |  | $\checkmark$ |  | $\checkmark$ |  | $\checkmark$ |  | (Yan et al., 2022)
3-40.2 |  |  | Certified removal | $\checkmark$ |  |  |  | $\checkmark$ |  |  | $\checkmark$ |  | (Ullah et al., 2021)
|  |  | Generalisation theory |  |  |  |  |  |  |  | $\checkmark$ |  | (Sekhari et al., 2021)
|  |  | Bayesian Inferrence | $\checkmark$ |  |  |  |  |  |  | $\checkmark$ |  | (Fu et al., 2021; Nguyen et al., 2022b)
| Exact machine unlearning | Others | Linear GNN |  |  |  |  |  |  |  | $\checkmark$ |  | (Cong and Mahdavi, 2023)
|  |  | Differencial Privacy | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ | (Golatkar et al., 2020a, b)
|  |  | Mixed-privacy setting | $\checkmark$ |  |  |  |  | $\checkmark$ |  |  | $\checkmark$ | (Golatkar et al., 2021)
|  | Weight Scrubbing | Fisher Information Matrix |  |  |  |  |  | $\checkmark$ |  |  | $\checkmark$ | (Martens, 2020)
3-40.2 |  |  | Influence method | $\checkmark$ |  |  |  |  | $\checkmark$ |  |  | $\checkmark$ | (Guo et al., 2019; Izzo et al., 2021; Chien et al., 2022; Wu et al., 2022a; Warnecke et al., 2021)
|  |  | Adding regularization terms |  |  |  |  |  | $\checkmark$ |  |  | $\checkmark$ | (Zeng et al., 2021)
|  | Data Influence | Fisher-based unlearning method |  |  |  |  |  | $\checkmark$ |  |  | $\checkmark$ | (Peste et al., 2021)
3-40.2 |  |  | Gradient based method | $\checkmark$ |  |  |  | $\checkmark$ | $\checkmark$ |  |  | $\checkmark$ | (Neel et al., 2021)
|  |  | Incremental learning | $\checkmark$ |  |  |  |  | $\checkmark$ |  |  | $\checkmark$ | (Wu et al., 2020b)
Passive Forgetting | Approximate machine unlearning | Updates Control | Amnesiac unlearning |  | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | $\checkmark$ |  |  | $\checkmark$ | (Graves et al., 2021)

Table 1. Summary of forgetting approaches in Machine Learning (NA means the
field is not applicable)

###  5.1. Active forgetting approaches

According to their purposes, existing studies about active forgetting could be
categorised as forgetting to mitigate negative transfer; forgetting to enhance
the generalisation ability to unseen data; and forgetting to free up storage
space thus enabling the transformer to scale. This survey will expand the
discussion about active forgetting approaches in machine learning according to
the above distinct scenarios. The structure of this chapter with some
corresponding relevant work is presented in Table 1.

####  5.1.1. Domain similarity estimation

One major reason for NT to happen is substantial domain divergence (Zhang et
al., 2023a), which refers to significant differences in data distributions,
feature spaces, and inherent characteristics between the source domain (where
the model is initially trained) and the target domain (where the model is
subsequently applied or fine-tuned). Therefore, before implementing an active
forgetting strategy to mitigate negative transfer learning, it’s essential to
first estimate domain similarity. As summarised by (Zhang et al., 2023a),
existing estimation methods encompass feature statistics-based approaches,
test prediction-based methods and fine-tuning-based techniques.

Feature statistics-based approaches in transfer learning analyse data
distributions and statistics (mean, variance) in source and target domains to
estimate domain similarity. Techniques include histogram comparisons,
correlation coefficients, and covariance matrix analyses to measure
distribution discrepancies, with popular methods includes Earth Mover’s
Distance, Maximum Mean Discrepancy, and KL-divergence.

Test-prediction-based approaches evaluate model performance from source to
target domain, utilising strategies like target performance, domain
classifier, and consensus focus. These assess domain similarity through model
accuracy on labelled target data, classification error from a domain
classifier, and agreement among various source models on target predictions.

Fine-tuning-based approaches adjust a pre-trained model to a new target domain
by modifying higher layer parameters while retaining lower layer weights.
Methods include label-based estimation, assessing domain similarity by
comparing labels, and combined label-feature representation estimation, which
also considers feature alignment post-fine-tuning. Techniques to prevent
catastrophic forgetting, like L1/L2 regularisation and Elastic Weight
Consolidation, optimise knowledge transfer by balancing source knowledge
retention and target domain adaptation. Fine-tuning-based approaches might
involve various aspects like ’Item removal’, ’Feature removal’, and ’Class
removal’, depending on the specifics of the fine-tuning process.

####  5.1.2. NT mitigation approaches

Negative Transfer (NT) mitigation focuses on minimising the detrimental
effects that occur when inappropriate knowledge is transferred between
domains. This process is effectively managed through active forgetting, where
specific knowledge is intentionally discarded or de-emphasised to enhance
outcomes in the target domain (Zhang et al., 2023a).

To address data transferability, several strategies across different
dimensions are utilised to mitigate NT. The domain dimension involves
strategies like domain selection and weighting via convex optimisation, which
determines the relevance of different domains to ensure the transfer of
pertinent information (Ahmed et al., 2021; Turrisi et al., 2022).
Additionally, domain decomposition helps by strategically segmenting a domain
into subdomains to facilitate more focused adaptation (Kim and Kim, 2020; Zhu
et al., 2020). Moving to the instance dimension, techniques such as Predictive
Distribution Matching (PDM) and active learning prioritise instances from the
source domain that are beneficial for the target domain, implicitly omitting
less salient instances (Seah et al., 2012; Wu, 2018). This selection process
helps to ”forget” non-representative data, streamlining the knowledge
transfer. The feature dimension introduces complexity by breaking down or
adjusting the feature space to emphasise features that are universally
applicable. Methods like Dual TL and dynamic auxiliary importance classifiers
are employed to highlight and retain beneficial features while sidelining
those that do not contribute positively (de Mathelin et al., 2021; Li et al.,
2022a). In the class dimension, shift detection and weighted adversarial
networks ensure that the classes align between the source and target domains.
Techniques like universal domain adaptation actively filter out unrelated
source data that does not conform to the target domain’s class structure,
effectively discarding non-aligning data (Lipton et al., 2018; You et al.,
2019).

For model transferability, enhancing source model training through methods
such as transferable batch normalization allows the model to prioritise more
adaptable features and ”forget” less relevant ones (Wang et al., 2019b).
Target model adaptation then focuses on selecting transferable parameters and
applying parameter regularization to further reduce the influence of
inappropriate data or features (Han et al., 2022).

Training process enhancement involves managing training dynamics, such as
varying step sizes and directions during optimization. This is often guided by
a hyper-parameter, $\lambda$, which helps balance between discriminability and
transferability (Yoon and Li, 2018; Xiao and Zhang, 2021).

Target prediction enhancement employs techniques like soft pseudo-labelling,
selective pseudo-labelling, clustering-enhanced pseudo-labelling, and entropy
regularization. These methods enable the model to adapt to new domains by
”forgetting” noisy or irrelevant labels and data, thus refining the prediction
accuracy in the target domain (Pei et al., 2018; Wang and Breckon, 2020; Gui
et al., 2018; Tian et al., 2020; Jin et al., 2020; Grandvalet and Bengio,
2004; Liang et al., 2020).

####  5.1.3. Iterative training

Recent training algorithms have highlighted the importance of iteratively
refining machine learning models to enhance their generalization ability
(Frankle and Carbin, 2018; Frankle et al., 2019; Zhou et al., 2022).
Techniques such as ”Knowledge Evolution” involve reinitializing parts of the
network while consistently training others to refine the learning process
(Taha et al., 2021). Similarly, ”Iterative Magnitude Pruning” involves a cycle
of pruning and retraining, which has shown to improve performance by
selectively removing weights (Frankle and Carbin, 2018; Frankle et al., 2019).

Li et al. introduced RIFLE, a method that periodically reinitializes the final
layer of the model to reduce dependency on specific features, thereby
encouraging the learning of broader patterns (Li et al., 2020a). This
approach, along with strategies utilizing synthetic machine translation
corpora and iterative self-distillation, effectively enhances learning by
leveraging back-translations and successive teaching iterations (Hoang et al.,
2018; Furlanello et al., 2018; Xie et al., 2020).

Particularly noteworthy is the ”Iterated Learning” approach, which has been
effective in enhancing compositionality and mitigating language drift in
emergent languages, as demonstrated in various studies (Kirby, 2001; Ren et
al., 2020a; Vani et al., 2021; Lu et al., 2020). Zhou et al. proposed a
”later-layer forgetting” (LLF) strategy, which involves reinitializing later
layers of the network to specifically remove information associated with
complex examples, thereby promoting a ”forget-and-relearn” paradigm. This
method encourages relearning difficult examples using simpler features from
early layers, which has proven to enhance generalization in image
classification and emergent communication tasks (Zhou et al., 2022).

Figure 7. Last-layer forgetting (LLF)

Figure 8. Partial balanced forgetting (PBF)

The iterative retraining process is vital for refining emergent languages,
where consistently understood features across training ”generations” become
reinforced. Unlike traditional training phases that focus on imitation,
iterated learning thrives on retraining interactions, promoting structure-
preserving mappings as seen in human linguistics (Kirby, 2001; Kirby et al.,
2014).

In emergent language research, the focus spans foundational concepts to the
dynamics of sender-receiver interactions. Central to this is selective
forgetting, which enhances generalization in language learning models. By
periodically reinitializing receivers, the model promotes compositionality, as
a fresh receiver is more likely to adapt to a structured language, leading to
improved communication efficiency (Li and Bowling, 2019; Zhou et al., 2022).

This approach aligns with the concept of ”Partial Balanced Forgetting” (PBF),
where both sender and receiver selectively forget non-compositional aspects of
the language to enhance overall structure and generalization. This strategic
forgetting helps tackle complex language tasks by pushing the model towards
summarizing and abstracting information, rather than getting overwhelmed by
complexities (Ren et al., 2020a).

####  5.1.4. Improve attention’s efficiency

Recent research has focused on optimising attention mechanisms to facilitate
more concentrated learning. (Bhattacharjee and Mahajan, 2022) proposed a
mathematical approach that uses a model where a learner decides which facts to
store, and the system measures performance based on how it matches up to
various experts with specific memory preferences. Dynamic attention spans,
such as Adaptive Span (Sukhbaatar et al., 2019) and Adaptive Sparse (Correia
et al., 2019) focus on learning which attention heads can have shorter spans.
Instead of attending to all tokens, the model dynamically chooses which tokens
to attend to, thereby reducing the computational overhead. This selective
attention mechanism can be seen as a form of ”forgetting,” where the model
disregards less relevant tokens in favour of more meaningful ones. Other
research focus on attenuating attention through employing fixed attention
masks, sliding windows, and hierarchical architectures. (Wu et al., 2022b)
offers an innovative approach to scale transformers by introducing kNN lookup
and non-differentiable external memory. These techniques allow the model to
efficiently manage its attention mechanism, focusing on essential details
while ”forgetting” unnecessary computations. The various control strategies
that are used to organise consolidated memories are summarised as a unified
abstraction: attention with bounded-memory control (ABC) (Peng et al., 2021b).
$ABC_{MLP}$ introduced in (Peng et al., 2021a) as an instance of ABC
introduces a neural network to determine how to store each token in the memory
to strike a better trade-off between accuracy and efficiency.

####  5.1.5. Lossless compression

The aforementioned methods primarily aim to make attention more efficient
without minimizing memory usage. To address this, Expire-span (Sukhbaatar et
al., 2021) was introduced, calculating a specific lifespan for each memory
(Figure 9). Once a memory surpasses its lifespan, it becomes inaccessible.
This strategy prunes distant tokens deemed ”unimportant”, allowing
transformers to handle memories tens of thousands in size. This temporal
forgetting means transformers retain only recent and pertinent data, thereby
cutting memory demands and computational costs.

Figure 9. Expire-Span (Sukhbaatar et al., 2021). For every memory $h_{i}$, we
compute an expire-span $e_{i}$ that determines how long it should stay in
memory. Here, memories $h_{2}$ and $h_{5}$ are already expired at time $t$, so
the query $q_{t}$ can only access $\\{h_{1},h_{3},h_{4}\\}$ in self-attention.

The principle of lossless compression resonates with the transformer’s goal of
maximizing information retention with minimal resources. Compression
techniques, similar to the act of forgetting, aspire to condense data without
significant information loss. In transformers, the attention mechanism,
optimised via methods like Adaptive-Span or Expire-span, is akin to data
compression, retaining only vital information.

The principle behind the compressor serves various applications, with its
primary use in language model training. RECOMP (Xu et al., 2023) is proposed
for optical language model performance and operates by first retrieving and
then compressing documents into summaries before assimilation. The model uses
two compressor types: an extractive one that selects key sentences and an
abstractive one that generates summaries from multiple sources. These
compressors, designed to enhance the language model’s task performance, can
also opt for ”selective augmentation” by returning an empty string if the
documents are irrelevant. This selective approach mirrors the ”forgetting”
concept in transformers, streamlining information input to ensure optimal
trade-off between model accuracy and efficiency. Meanwhile, TRIME (Zhong et
al., 2022) emphasizes memory type classification, introducing methods for
memory construction and data batching for different memory types during
testing. The attentive Item2Vec++ (AI2V++) model proposed a neural attentive
collaborative filtering approach in which the user representation adapts
dynamically in the presence of the recommended item (Gaiger et al., 2023).
Inspired by the dynamic nature of the human memory, in which different
memories become more accessible in different contexts, the model uses
attention mechanisms to dynamically adjust the importance of the historical
items based on the item being considered for recommendation.

###  5.2. Passive forgetting approaches

In this survey, unlearning is categorised as a subset of machine forgetting
because it only includes a passive forgetting process triggered by removal
requests due to privacy, security or usability concerns. Previous research in
machine unlearning has consistently classified the methods into ”Exact
(complete, perfect) unlearning” and ”Approximate (certified, bounded)
unlearning”, which aligns with our prior definitions of forgetting dimensions
4.1.3.

####  5.2.1. Exact machine unlearning

The most straightforward way to perform exact machine unlearning is by
removing the requested ”forget set” and retraining the model from scratch,
which is computationally demanding. Recent studies have explored various exact
unlearning strategies to make the retraining process both efficient and easy
to control. Statistical Query Learning, Data Partitioning, and Rapid
Retraining appear to be the most related approaches for exact unlearning
(Nguyen et al., 2022a; Zhang et al., 2023b; Cong and Mahdavi, 2023; Wang et
al., 2023). Therefore, we will introduce those methods in more detail and
cover other approaches more briefly (please refer to (Nguyen et al., 2022a)
for a more in-depth summarisation of unlearning methodologies.

##### Statistical Query Learning

Statistical Query (SQ) learning trains models using statistical data
properties instead of direct data access (Kearns, 1998). This method supports
efficient unlearning by updating statistical summaries without revisiting
individual data points. Cao and Yang (Cao and Yang, 2015) applied SQ learning
for machine unlearning by updating summations, reducing data dependencies.
However, this approach struggles with complex models like deep neural networks
due to the exponential increase in the required statistical queries (Bourtoule
et al., 2021), which impacts the efficiency of unlearning and relearning
processes.

##### Data Partition

To reduce computational costs, Bourtoule et al. (Bourtoule et al., 2021)
introduced the SISA training approach, which divides the dataset into multiple
shards, each trained independently and then aggregated during inference (see
Figure 10). This method allows for targeted re-training upon unlearning
requests, enhancing process efficiency. However, model performance may decline
due to reduced training data per shard and data heterogeneity. Complex
learning tasks may require integrating SISA with other methods like transfer
learning to maintain accuracy and processing speed.

Figure 10. SISA proposed by (Bourtoule et al., 2021). It splits the dataset
into multiple shards and train an independent model on each data shard, then
aggregate their prediction during inference.

##### Rapid Retraining

Innovations in exact unlearning focus on accelerating the retraining process.
Deltagrad (Wu et al., 2020a) caches model parameters and gradients to expedite
retraining after data deletion. ARCANE (Yan et al., 2022) uses ensemble
learning to transform full retraining into simpler classification tasks,
reducing computational demands while maintaining performance. Additionally,
Ullah et al. (Ullah et al., 2021) and others explore efficient retraining
strategies and theoretical unlearning approaches for various applications
including Bayesian inference (Fu et al., 2021; Nguyen et al., 2022b) and
algorithmic unlearning (Cong and Mahdavi, 2023).

####  5.2.2. Approximate machine unlearning

Approximate unlearning has been proposed to mitigate the computationally
intensive nature of exact unlearning. The central concept involves
approximating the model trained without the deleted data in the parameter
space. Mahadevan et al. (Mahadevan and Mathioudakis, 2021) and Zhang et al.
(Zhang et al., 2023b) summarise that approximate unlearning methods can be
broadly categorised into three groups. This subsection will introduce several
of the most relevant methods based on this classification.

##### Weight ’Scrubbing’

Differential privacy, based on the Fisher Information Matrix (FIM), is used in
approximate unlearning strategies to maintain certifiability when updating
machine learning models. This method, known as Fisher forgetting, involves
adjusting the weights to obscure specific training data influences, avoiding
the need for complete retraining (Martens, 2020; Golatkar et al., 2020b, a,
2021). Golatkar et al. extend this to a mixed-privacy setting, allowing some
training samples to remain unaltered, simplifying the unlearning process for
non-core data by setting certain weights to zero, which minimally impacts
performance.

##### Data Influence

Unlearning approaches often utilize influence functions to understand how
changes in training data affect model parameters. These functions help compute
the impact of deleted data to ensure it does not influence the model after its
removal. Techniques range from employing first-order Taylor approximations to
using influence functions for estimating training data impacts on parameters,
enabling updates without reprocessing the entire dataset (Chundawat et al.,
2023; Mahadevan and Mathioudakis, 2022; Guo et al., 2019; Izzo et al., 2021;
Chien et al., 2022; Wu et al., 2022a; Warnecke et al., 2021). However,
ensuring that data meant to be forgotten remains entirely unrepresented in the
model remains a challenge (Thudi et al., 2022a).

##### Updates control

Influencing model training via updates control involves tracking and
potentially reversing training steps to remove data influences. This method,
known as Amnesiac Unlearning, selectively undoes updates involving sensitive
data. Techniques involve saving intermediate training states like weight
parameters and gradients to efficiently recalibrate the model’s training path
upon data deletion requests. While effective, this approach does not guarantee
perfect data removal and often involves adding random noise to achieve
approximate unlearning (Graves et al., 2021; Neel et al., 2021; Wu et al.,
2020a, b; Thudi et al., 2022b).

##  6\. Discussion

###  6.1. Applications of Forgetting Approaches in ML

Active forgetting in machine learning involves selectively ignoring outdated
or misleading knowledge to enhance model performance across various fields. In
the medical and healthcare sector, this approach improves diagnostic accuracy
by disregarding non-relevant features from different demographics or imaging
devices, thus enhancing model transferability (Niu et al., 2021; Zhang and Wu,
2020; Peres da Silva et al., 2021; He and Wu, 2019). In sentiment analysis,
the mechanism prevents misinterpretation of sentiments by ignoring domain-
specific jargon that is unsuitable for different contexts (Liu et al., 2019).
For autonomous driving, it adapts vehicles to different driving conditions by
forgetting irrelevant driving patterns learned in other regions (Sorocky et
al., 2020; Corso and Kochenderfer, 2021). Similarly, in robotics, active
forgetting enhances robot adaptability to new tasks by discarding behaviors
specific to previous environments (Li et al., 2020b).

Passive forgetting, or machine unlearning, is critical for addressing
security, privacy, and biases in machine learning by removing specific data
influences from models. In healthcare, this ensures the accuracy and security
of models handling sensitive data, supports privacy compliance, and
facilitates federated learning without needing to share patient data directly
(Nguyen et al., 2022a; Ren et al., 2020b; Marchant et al., 2022; Wu et al.,
2022c). In cybersecurity, machine unlearning counteracts backdoor and
membership inference attacks, enhancing model integrity and trustworthiness
(Toreini et al., 2020; Wang et al., 2019c; Liu et al., 2022a; Chen et al.,
2021). Furthermore, machine unlearning plays a significant role in correcting
social biases by removing biased data, thereby promoting fairer outcomes in
sensitive applications such as hiring and law enforcement (Mehrabi et al.,
2021; Zou and Schiebinger, 2018; Osoba et al., 2017; Kumar et al., 2022).

###  6.2. Challenges

####  6.2.1. Model Training

Selecting data to forget, known as ”forget set” selection, is a pivotal
challenge in machine learning, particularly in active forgetting scenarios.
This process involves evaluating the utility of each data point to determine
their impact on the model’s parameters. Core-set selection simplifies this by
identifying a subset that yields comparable results to the full dataset, thus
reducing learning costs and enhancing efficiency in forgetting (Huang et al.,
2010; Sener and Savarese, 2017; Baykal et al., 2018). The main difficulty lies
in accurately defining this core-set to capture all essential information
without compromising model performance (Bourtoule et al., 2021). Additionally,
deciding the granularity of forgetting—whether whole categories or specific
instances—requires understanding data dependencies to avoid residual
distortions in model output.

Training models to forget involves significant complexity due to the
unpredictable nature of neural network training, which uses random mini-
batches and sequences. This randomness disperses the influence of individual
data points, making it hard to isolate their specific impacts. Forgetting
requires not just removing the data point but also negating its cumulative
effects to maintain consistent model performance. This process is
computationally intensive, particularly in exact unlearning scenarios where
parts of the model may need frequent retraining. These challenges are
exacerbated in resource-limited environments, such as in online continual
learning and meta-learning, where the retention of previously learned
information is further complicated by limited data exposure (Bourtoule et al.,
2021; Nguyen et al., 2022a; Wang et al., 2023).

Catastrophic forgetting is a significant challenge in continual learning. As
new data is introduced, the model’s ability to perform tasks related to older
data can substantially degrade. This problem is compounded in settings with
continual and federated learning due to potential task interference. Various
strategies, such as designing special loss functions, have been proposed to
mitigate catastrophic forgetting, though challenges remain in deciding which
data points to replicate for improving model accuracy without increasing the
overhead of forgetting (Nguyen et al., 2020b; Du et al., 2019; Golatkar et
al., 2020a; Settles, 2009).

####  6.2.2. Evaluation and monitoring

Evaluating machine forgetting involves key metrics such as accuracy,
completeness, and efficiency. Accuracy assesses how well the model performs
post-forgetting, aiming to match the precision of a freshly retrained model on
various datasets (Mahadevan and Mathioudakis, 2022; Nguyen et al., 2022a).
Completeness ensures that the forgotten data leaves no residual trace,
aligning the predictions of the unlearned model with those of a model trained
from scratch. This is crucial for privacy, as it ensures that data once
forgotten remains inaccessible, even with sophisticated Membership Inference
Attacks (MIAs) or model inversion techniques (Carlini et al., 2022; Graves et
al., 2021). Efficiency measures the resource consumption involved in the
forgetting process, considering factors like time and computational overhead.
Efficient unlearning is essential to manage costs and maintain the feasibility
of forgetting processes, especially in resource-limited settings (Izzo et al.,
2021; Nguyen et al., 2022a). Equally important is ensuring that the evaluation
metrics themselves are computed efficiently, not just the forgetting
algorithms. As this field is still emerging, there are many relevant facets
that could be incorporated into the forgetting evaluation metric, yet they
have not been explored by current research. These include the applicability of
the forgetting method to different models, the granularity of forgetting
depths, the rate at which forgetting occurs, and the time decay that accounts
for the fading of information over time, especially in models set for
continual learning. A more comprehensive, efficient, and reliable evaluation
metric for machine forgetting is needed.

Data lineage management tracks the flow and transformation of data throughout
its lifecycle in a system, from origin to end-use. It is vital for ensuring
accurate and compliant machine forgetting, especially under regulations like
GDPR which mandate the right to be forgotten. This process enables precise
identification and removal of data, fostering trust among users by
transparently showing that their data can be completely erased upon request.
Additionally, data lineage is crucial for troubleshooting and reproducibility,
helping to pinpoint issues or reproduce outcomes when models are retrained
without certain datasets (Li et al., 2022b; Zhang et al., 2023b; Baracaldo et
al., 2017).

Figure 11. Example of data lineage management for data flow recording and
machine unlearning updating (original image is from (Zhang et al., 2023b))

How machine unlearning can be applied in conjunction with data lineage
management systems is presented in (Zhang et al., 2023b) as shown in Figure
11. Beginning with the user’s interaction via a graphical interface, the user
can access and potentially request the removal of specific data. This data
then undergoes preprocessing before being input into the machine learning
model. Throughout each step, from data acquisition to model training and
result generation, there are records and updates maintained to capture data
flow characteristics and changes. This meticulous recording within the data
lineage management system allows developers to have granular control over the
model’s learning process. When a user requests data removal, the system enters
an ”unlearning” phase, adjusting the machine learning model and producing a
new output result. This comprehensive flow ensures that any alterations,
particularly in terms of data removal, are reflected in the decision-making
outcomes of the model. Addressing the biases that may arise during the
forgetting process, (Muller and Strohmayer, 2022) developed a forgetfulness
stack for data work. This stack operates on records and variables pertinent to
data science, aiming to alter human actions throughout the machine learning
lifecycle.

####  6.2.3. Trustworthiness of forgetting

Incorporating forgetting mechanisms can potentially mitigate social biases in
datasets. However, this process introduces human interpretations that may
create new biases, termed extrinsic and intrinsic biases. Extrinsic bias
emerges from external perceptions of a biased dataset, whereas intrinsic bias
arises from modifications within the data itself (Muller et al., 2019;
Sambasivan et al., 2021; Selbst et al., 2019; Muller and Strohmayer, 2022).
Forgetting can inadvertently exclude certain groups or perspectives, such as
LGBTQIA2S+ individuals or indigenous names, due to system limitations like
ASCII character constraints (Costanza-Chock, 2020; Muller and Strohmayer,
2022). Moreover, the selective representation in data due to internet access
discrepancies further complicates this issue. It’s suggested that
reintegrating data from underrepresented sources might alleviate these
problems (Wegner and Lawless, 2021; Vigil-Hayes et al., 2017).

The complexity of machine learning models, especially deep neural networks,
makes the interpretation of forgetting mechanisms challenging. These models
are often ”black boxes” with numerous parameters that obscure how data is
forgotten. Adjustments to model architecture, training data modifications, and
application of forgetting techniques obscure interpretability further, making
it difficult to trace how changes influence the model’s predictions (Zhang et
al., 2023a).

Originally, machine unlearning aimed to enhance privacy by erasing traces of
training data, reducing the risk of privacy leaks. However, recent critiques
suggest that machine unlearning could still be vulnerable to attacks such as
membership inference attacks, which can potentially reveal sensitive
information (Chen et al., 2021; Shokri et al., 2017; Yeom et al., 2018;
Sablayrolles et al., 2019; Hayes et al., 2017). These challenges underscore
the need for robust unlearning methods that can prevent data reconstruction
and secure privacy effectively.

##  7\. Future Directions

##### Cross-disciplinary Forgetting

Cross-disciplinary research is pivotal in advancing machine learning, applying
diverse methods to understand forgetting mechanisms and adopting ML techniques
to address challenges in other fields. Examples include leveraging insights
from neural plasticity and cognitive theories like interference or decay
theory (Ebbinghaus, 2013) to refine ML algorithms. Active collaboration across
various fields is essential for innovative breakthroughs.

##### Forgetting Verification

Despite various approaches to forgetting in ML, standardised methods for
evaluating and verifying their effectiveness are lacking. Developing robust
forgetting verification methodologies is critical, especially for scenarios
requiring data removal for privacy or compliance reasons. This includes
creating algorithms to confirm complete data removal without impacting model
performance and ensuring that forgotten data does not indirectly affect future
predictions (Nguyen et al., 2022a).

##### Interpretable Forgetting

The interpretability of forgetting, particularly in neural networks, is
challenging. Research could focus on understanding factors affecting transfer
learning performance and developing interpretable machine unlearning methods
to verify the unlearning process (Basu et al., 2020; Koh and Liang, 2017).

##### Hybrid Forgetting

A promising research direction is hybrid forgetting, which combines active and
passive forgetting. This approach would allow models to dynamically decide
when to update or forget information based on performance, privacy, and
compliance, using algorithms that balance data removal’s impact on model
effectiveness and privacy (Guo et al., 2022).

##### Source-Free Forgetting

Source-free forgetting represents a future direction in machine unlearning,
moving away from traditional models that require access to all training data
for data removal. In stringent privacy settings like the zero-glance setting
described by Tarun et al. (Tarun et al., 2023), organisations must erase user
data, such as facial recognition details, without reusing those samples even
for model adjustments. This necessitates innovative methods to modify model
weights and parameters to eliminate data influence without revisiting the
data. In transfer learning, similar privacy and regulatory challenges require
performing tasks without accessing original source data, a process parallel to
zero-glance learning but applied across different domains. Despite progress in
reducing negative transfer in source-free transfer learning (Feng et al.,
2021; Turrisi et al., 2022; Liang et al., 2020; Ishii and Sugiyama, 2021),
there remains a critical need for algorithms with stronger theoretical
underpinnings.

##### ”Goldilocks Zone” of Forgetting

Identifying the ”Goldilocks zone” 222The ’Goldilocks Zone,’ or habitable zone,
is the area around a star where it is not too hot and not too cold for liquid
water to exist on the surface of surrounding planets (NASA). Inspired by (Zhou
et al., 2022), we use this term as a metaphor to describe the boundaries and
trade-offs involved in forgetting. of forgetting—finding an optimal balance
where a model retains enough information to maintain generalizability while
discarding outdated data—is crucial in continual learning. This involves
determining the precise threshold for effective learning without catastrophic
forgetting.

##### Trade-offs of forgetting applications

In industrial applications, implementing forgetting mechanisms involves
balancing efficiency, compliance, and historical knowledge retention against
challenges like data management complexity and computational costs. Until
fully efficient forgetting algorithms are developed, alternative measures like
data masking (Archana et al., 2018) may be necessary.

##### Forgetting Regularisation

Designing forgetting mechanisms involves not only technical challenges but
also profound social, political, and ethical considerations (Domínguez
Hernández et al., 2023). Collaboration with experts across various fields will
be crucial to ensure these mechanisms are transparent, fair, and effective,
shaping how machine learning impacts our digital society.

##  8\. Conclusion

The research outcomes on “Forgetting” from other disciplines explain the
theoretical foundation and motivation for forgetting in machine learning which
tells us why to forget. The process of forgetting in other disciplines, such
as cognitive psychology or neuroscience, has been studied extensively, and
insights from these fields can be applied to machine learning. For instance,
the principles of regularisation, which involve removing or modifying
information in machine learning models, are inspired by the concept of
forgetting in cognitive psychology. Similarly, the concept of transfer
learning, where knowledge from one task is used to improve performance on a
different but related task, can be viewed as a form of selective forgetting,
where irrelevant information is removed or modified to improve learning
efficiency. On the other hand, research from other fields sheds light on How
to forget in machine learning. The factors that influence human forgetting
behaviour, such as the passage of time, interference, and relevance, can also
be implemented in machine learning models. For instance, in machine learning,
the decay rate of learned information can be modelled as a function of time,
and the degree of interference can be controlled through regularisation
techniques.

Addressing [RQ1], we see that forgetting manifests differently across
knowledge areas, each offering unique perspectives that are vital for
developing advanced machine learning models. In response to [RQ2], this
research classifies and organises the dimensions and approaches of forgetting
in ML into a taxonomy to provide clearer literature navigation in this field.
Regarding [RQ3], the future of forgetting in machine learning poses both
opportunities and challenges, this survey discussed the applications,
challenges and concerns and future directions about forgetting in ML.

In conclusion, this survey highlights the critical role of selective
forgetting in enhancing machine learning models by enabling them to prioritise
relevant information, comply with privacy laws, and mitigate bias. By drawing
parallels between human cognitive processes and artificial systems, we open
avenues for novel machine learning methodologies that can dynamically adapt
and evolve. The exploration of forgetting not only enriches our understanding
of machine learning paradigms but also sets the stage for future research that
will further refine these concepts, ensuring both ethical and practical
advancements in the field.

## References

  * (1)
  * Abdelrahman et al. (2023) Ghodai Abdelrahman, Qing Wang, and Bernardo Nunes. 2023.  Knowledge Tracing: A Survey.  _ACM Comput. Surv._ 55, 11 (2 2023).  https://doi.org/10.1145/3569576
  * Afridi et al. (2018) Muhammad Jamal Afridi, Arun Ross, and Erik M Shapiro. 2018.  On automated source selection for transfer learning in convolutional neural networks.  _Pattern recognition_ 73 (2018), 65–75.
  * Ahmed et al. (2021) Sk Miraj Ahmed, Dripta S Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K Roy-Chowdhury. 2021.  Unsupervised multi-source domain adaptation without access to source data. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10103–10112.
  * Aldaghri et al. (2021) Nasser Aldaghri, Hessam Mahdavifar, and Ahmad Beirami. 2021.  Coded machine unlearning.  _IEEE Access_ 9 (2021), 88137–88150.
  * Arango-Muñoz (2013) Santiago Arango-Muñoz. 2013.  Scaffolded memory and metacognitive feelings.  _Review of Philosophy and Psychology_ 4 (2013), 135–152.
  * Arango-Muñoz and Michaelian (2014) Santiago Arango-Muñoz and Kourken Michaelian. 2014.  Epistemic Feelings and Epistemic Emotions (Focus Section).  (2014).
  * Archana et al. (2018) R A Archana, Ravindra S Hegadi, and T N Manjunath. 2018.  A study on big data privacy protection models using data masking methods.  _International Journal of Electrical and Computer Engineering_ 8, 5 (2018), 3976.
  * Augé et al. (2004) Marc Augé, Marjolijn De Jager, and James Edward Young. 2004.  _Oblivion_.
  * Augustine (1991) Saint Augustine. 1991.  Confessions, trans. Henry Chadwick.  _Oxford/New Youk: Oxford University Press Ayer, AJ (1946). Language Bernet, R., Kern I., and Mar Evanston_ 111 (1991), 265–280.
  * Averell and Heathcote (2011) Lee Averell and Andrew Heathcote. 2011.  The form of the forgetting curve and the fate of memories.  _Journal of mathematical psychology_ 55, 1 (2011), 25–35.
  * Badre and Wagner (2006) David Badre and Anthony D Wagner. 2006.  Computational and neurobiological mechanisms underlying cognitive flexibility.  _Proceedings of the National Academy of Sciences_ 103, 18 (2006), 7186–7191.
  * Bao et al. (2019) Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. 2019.  An information-theoretic approach to transferability in task transfer learning. In _2019 IEEE international conference on image processing (ICIP)_. IEEE, 2309–2313.
  * Baracaldo et al. (2017) Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, and Jaehoon Amir Safavi. 2017.  Mitigating poisoning attacks on machine learning models: A data provenance based approach. In _Proceedings of the 10th ACM workshop on artificial intelligence and security_. 103–110.
  * Basu (2022) Rima Basu. 2022.  The Importance of Forgetting.  _Episteme_ 19, 4 (2022), 471–490.  https://doi.org/10.1017/epi.2022.36
  * Basu et al. (2020) Samyadeep Basu, Philip Pope, and Soheil Feizi. 2020.  Influence functions in deep learning are fragile.  _arXiv preprint arXiv:2006.14651_ (2020).
  * Baykal et al. (2018) Cenk Baykal, Lucas Liebenwein, Igor Gilitschenski, Dan Feldman, and Daniela Rus. 2018.  Data-dependent coresets for compressing neural networks with applications to generalization bounds.  _arXiv preprint arXiv:1804.05345_ (2018).
  * Behan (1979) David P Behan. 1979.  Locke on persons and personal identity.  _Canadian Journal of Philosophy_ 9, 1 (1979), 53–75.
  * Ben-David et al. (2010) Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010.  A theory of learning from different domains.  _Machine learning_ 79 (2010), 151–175.
  * Bernecker (2008) Sven Bernecker. 2008.  _The metaphysics of memory_. Vol. 111.  Springer Science & Business Media.
  * Bernecker (2018) Sven Bernecker. 2018.  12 On the Blameworthiness of Forgetting.  _New directions in the philosophy of memory_ (2018).
  * Bernico et al. (2019) Michael Bernico, Yuntao Li, and Dingchao Zhang. 2019.  Investigating the impact of data volume and domain similarity on transfer learning applications. In _Proceedings of the Future Technologies Conference (FTC) 2018: Volume 2_. Springer, 53–62.
  * Bhattacharjee and Mahajan (2022) Robi Bhattacharjee and Gaurav Mahajan. 2022.  Learning what to remember. In _International Conference on Algorithmic Learning Theory_. PMLR, 70–89.
  * Bjork et al. (2014) Elizabeth Ligon Bjork, Jeri L Little, and Benjamin C Storm. 2014.  Multiple-choice testing as a desirable difficulty in the classroom.  _Journal of Applied Research in Memory and Cognition_ 3, 3 (2014), 165–170.
  * Bjork and Allen (1970) Robert A Bjork and Ted W Allen. 1970.  The spacing effect: Consolidation or differential encoding?  _Journal of Verbal Learning and Verbal Behavior_ 9, 5 (1970), 567–572.
  * Bjork and Bjork (2019) Robert A Bjork and Elizabeth L Bjork. 2019.  Forgetting as the friend of learning: implications for teaching and self-regulated learning.  _Advances in Physiology Education_ 43 (2019), 164–167.  https://doi.org/10.1152/advan.00001.2019.-One
  * Bjork and Bjork (2020) Robert A. Bjork and Elizabeth L. Bjork. 2020.  Desirable Difficulties in Theory and Practice.  , 475–479 pages.  https://doi.org/10.1016/j.jarmac.2020.09.003
  * Borges and Hurley (1999) J L Borges and A Hurley. 1999.  _Collected Fictions_.  Allen Lane The Penguin Press.  https://books.google.com.au/books?id=fVVAAQAAIAAJ
  * Bourtoule et al. (2021) Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021.  Machine unlearning.  _Proceedings - IEEE Symposium on Security and Privacy_ 2021-May (2021), 141–159.  https://doi.org/10.1109/SP40001.2021.00019
  * Broome (2013) John Broome. 2013.  _Rationality through reasoning_.  John Wiley & Sons.
  * Cao and Yang (2015) Yinzhi Cao and Junfeng Yang. 2015.  Towards making systems forget with machine unlearning.  _Proceedings - IEEE Symposium on Security and Privacy_ 2015-July (2015), 463–480.  https://doi.org/10.1109/SP.2015.35
  * Caravà (2021) Marta Caravà. 2021.  An exploration into enactive forms of forgetting.  _Phenomenology and the Cognitive Sciences_ 20, 4 (2021), 703–722.  https://doi.org/10.1007/s11097-020-09670-6
  * Carlini et al. (2022) Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022.  Membership inference attacks from first principles. In _2022 IEEE Symposium on Security and Privacy (SP)_. IEEE, 1897–1914.
  * Chen (2020) Chen Chen. 2020.  A study on positive transfer of native language and second language teaching methods.  _Theory and Practice in Language Studies_ 10, 3 (2020), 306–312.
  * Chen et al. (2022a) Chong Chen, Fei Sun, Min Zhang, and Bolin Ding. 2022a.  Recommendation unlearning. In _Proceedings of the ACM Web Conference 2022_. 2768–2777.
  * Chen et al. (2021) Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. 2021.  When machine unlearning jeopardizes privacy. In _Proceedings of the 2021 ACM SIGSAC conference on computer and communications security_. 896–911.
  * Chen et al. (2022b) Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. 2022b.  Graph unlearning. In _Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security_. 499–513.
  * Chen et al. (2019) Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. 2019.  Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Chen and Daehler (1989) Zhe Chen and Marvin W Daehler. 1989.  Positive and negative transfer in analogical problem solving by 6-year-old children.  _Cognitive Development_ 4, 4 (1989), 327–344.
  * Chien et al. (2022) Eli Chien, Chao Pan, and Olgica Milenkovic. 2022.  Certified graph unlearning.  _arXiv preprint arXiv:2206.09140_ (2022).
  * Chundawat et al. (2023) Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. 2023.  Zero-shot machine unlearning.  _IEEE Transactions on Information Forensics and Security_ (2023).
  * Cong and Mahdavi (2023) Weilin Cong and Mehrdad Mahdavi. 2023.  Efficiently Forgetting What You Have Learned in Graph Representation Learning via Projection. In _Proceedings of International Conference on Artificial Intelligence and Statistics (AISTATS)_ , Vol. 206. 6674–6703.
  * Cooper et al. (2018) Jeffrey Cooper, Brett O’Connor, Ronald Watanabe, Randall Fuerst, Sharon Berger, Nadine Eisenberg, and Sally M Dillehay. 2018.  Case series analysis of myopic progression control with a unique extended depth of focus multifocal contact lens.  _Eye & Contact Lens_ 44, 5 (2018), e16–e24.
  * Corbin and Strauss (1990) Juliet M Corbin and Anselm Strauss. 1990.  Grounded theory research: Procedures, canons, and evaluative criteria.  _Qualitative sociology_ 13, 1 (1990), 3–21.
  * Correia et al. (2019) Gonçalo M Correia, Vlad Niculae, and André F T Martins. 2019.  Adaptively sparse transformers.  _arXiv preprint arXiv:1909.00015_ (2019).
  * Corso and Kochenderfer (2021) Anthony Corso and Mykel J Kochenderfer. 2021.  Transfer learning for efficient iterative safety validation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 7125–7132.
  * Costanza-Chock (2020) Sasha Costanza-Chock. 2020.  _Design justice: Community-led practices to build the worlds we need_.  The MIT Press.
  * Costanzi et al. (2021) Marco Costanzi, Beatrice Cianfanelli, Alessandro Santirocchi, Stefano Lasaponara, Pietro Spataro, Clelia Rossi-Arnaud, and Vincenzo Cestari. 2021.  Forgetting Unwanted Memories: Active Forgetting and Implications for the Development of Psychological Disorders.  https://doi.org/10.3390/jpm11040241
  * de Mathelin et al. (2021) Antoine de Mathelin, Guillaume Richard, François Deheeger, Mathilde Mougeot, and Nicolas Vayatis. 2021.  Adversarial weighting for domain adaptation in regression. In _2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI)_. IEEE, 49–56.
  * Dickersin et al. (1994) Kay Dickersin, Roberta Scherer, and Carol Lefebvre. 1994.  Systematic reviews: identifying relevant studies for systematic reviews.  _Bmj_ 309, 6964 (1994), 1286–1291.
  * Dietterich (1995) Tom Dietterich. 1995.  Overfitting and undercomputing in machine learning.  _ACM computing surveys (CSUR)_ 27, 3 (1995), 326–327.
  * Domínguez Hernández et al. (2023) Andrés Domínguez Hernández, Richard Owen, Dan Saattrup Nielsen, and Ryan McConville. 2023.  Ethical, political and epistemic implications of machine learning (mis) information classification: insights from an interdisciplinary collaboration between social and data scientists.  _Journal of Responsible Innovation_ 10, 1 (2023), 2222514.
  * Dougherty (2011) Trent Dougherty. 2011.  _Evidentialism and its Discontents_.  Oxford University Press.
  * Du et al. (2019) Min Du, Zhi Chen, Chang Liu, Rajvardhan Oak, and Dawn Song. 2019.  Lifelong anomaly detection through unlearning. In _Proceedings of the 2019 ACM SIGSAC conference on computer and communications security_. 1283–1297.
  * Dukas (2004) Reuven Dukas. 2004.  Evolutionary biology of animal cognition.  _Annu. Rev. Ecol. Evol. Syst._ 35 (2004), 347–374.
  * Ebbinghaus (2013) Hermann Ebbinghaus. 2013.  Memory: A contribution to experimental psychology.  _Annals of neurosciences_ 20, 4 (2013), 155.
  * Ecke (2004) Peter Ecke. 2004.  Language attrition and theories of forgetting: A cross-disciplinary review.  _International Journal of Bilingualism_ 8, 3 (2004), 321–354.  https://doi.org/10.1177/13670069040080030901
  * Eisenstein et al. (2001) E M Eisenstein, D Eisenstein, and James C Smith. 2001.  The evolutionary significance of habituation and sensitization across phylogeny: A behavioral homeostasis model.  _Integrative Physiological & Behavioral Science_ 36 (2001), 251–265.
  * Feng et al. (2021) Haozhe Feng, Zhaoyang You, Minghao Chen, Tianye Zhang, Minfeng Zhu, Fei Wu, Chao Wu, and Wei Chen. 2021.  KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation.. In _ICML_. 3274–3283.
  * Frankle and Carbin (2018) Jonathan Frankle and Michael Carbin. 2018.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  _arXiv preprint arXiv:1803.03635_ (2018).
  * Frankle et al. (2019) Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. 2019.  Stabilizing the lottery ticket hypothesis.  _arXiv preprint arXiv:1903.01611_ (2019).
  * French (1999) Robert M French. 1999.  Catastrophic forgetting in connectionist networks.  _Trends in cognitive sciences_ 3, 4 (1999), 128–135.
  * Frise (2017) Matthew Frise. 2017.  Preservationism in the epistemology of memory.  _The Philosophical Quarterly_ 67, 268 (2017), 486–507.
  * Frise (2018) Matthew Frise. 2018.  11 Forgetting.  (2018), 223–240.
  * Fu et al. (2021) Shaopeng Fu, Fengxiang He, and Dacheng Tao. 2021.  Knowledge Removal in Sampling-based Bayesian Inference. In _International Conference on Learning Representations_.
  * Furlanello et al. (2018) Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018.  Born again neural networks. In _International Conference on Machine Learning_. PMLR, 1607–1616.
  * Gagliano et al. (2014) Monica Gagliano, Michael Renton, Martial Depczynski, and Stefano Mancuso. 2014.  Experience teaches plants to learn faster and forget slower in environments where it matters.  _Oecologia_ 175, 1 (2014), 63–72.  https://doi.org/10.1007/s00442-013-2873-7
  * Gaiger et al. (2023) Keren Gaiger, Oren Barkan, Shir Tsipory-Samuel, and Noam Koenigstein. 2023.  Not All Memories Created Equal: Dynamic User Representations for Collaborative Filtering.  _IEEE Access_ 11 (2023), 34746–34763.
  * Gao et al. (2021) Zhiqiang Gao, Shufei Zhang, Kaizhu Huang, Qiufeng Wang, and Chaoliang Zhong. 2021.  Gradient distribution alignment certificates better adversarial domain adaptation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 8937–8946.
  * Gershman and Uchida (2019) Samuel J Gershman and Naoshige Uchida. 2019.  Believing in dopamine.  _Nature reviews. Neuroscience_ 20, 11 (11 2019), 703–714.  https://doi.org/10.1038/s41583-019-0220-7
  * Giles and Rankin (2009) Andrew C Giles and Catharine H Rankin. 2009.  Behavioral and genetic characterization of habituation using Caenorhabditis elegans.  _Neurobiology of learning and memory_ 92, 2 (2009), 139–146.
  * Golatkar et al. (2021) Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. 2021.  Mixed-privacy forgetting in deep networks. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 792–801.
  * Golatkar et al. (2020a) Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020a.  Eternal sunshine of the spotless net: Selective forgetting in deep networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 9304–9312.
  * Golatkar et al. (2020b) Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020b.  Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16_. Springer, 383–398.
  * Gong et al. (2012) Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. 2012.  Geodesic flow kernel for unsupervised domain adaptation. In _2012 IEEE conference on computer vision and pattern recognition_. IEEE, 2066–2073.
  * Grandvalet and Bengio (2004) Yves Grandvalet and Yoshua Bengio. 2004.  Semi-supervised learning by entropy minimization.  _Advances in neural information processing systems_ 17 (2004).
  * Graves et al. (2021) Laura Graves, Vineel Nagisetty, and Vijay Ganesh. 2021.  Amnesiac Machine Learning.  _35th AAAI Conference on Artificial Intelligence, AAAI 2021_ 13A (2021), 11516–11524.  https://doi.org/10.1609/aaai.v35i13.17371
  * Gravitz (2019) Lauren Gravitz. 2019.  The importance of forgetting.  _Nature_ 571, July (2019), S12–S14.
  * Gretton et al. (2012) Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. 2012.  A kernel two-sample test.  _The Journal of Machine Learning Research_ 13, 1 (2012), 723–773.
  * Gui et al. (2018) Lin Gui, Ruifeng Xu, Qin Lu, Jiachen Du, and Yu Zhou. 2018.  Negative transfer detection in transductive transfer learning.  _International Journal of Machine Learning and Cybernetics_ 9 (2018), 185–197.
  * Guo et al. (2019) Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. 2019.  Certified data removal from machine learning models.  _arXiv preprint arXiv:1911.03030_ (2019).
  * Guo et al. (2022) Tao Guo, Song Guo, Jiewei Zhang, Wenchao Xu, and Junxiao Wang. 2022.  Efficient Attribute Unlearning: Towards Selective Removal of Input Attributes from Feature Representations.  _arXiv preprint arXiv:2202.13295_ (2022).
  * Halamish et al. (2011) Vered Halamish, Shannon McGillivray, and Alan D Castel. 2011.  Monitoring one’s own forgetting in younger and older adults.  _Psychology and aging_ 26, 3 (2011), 631.
  * Hampton et al. (1998) Robert R Hampton, Sara J Shettleworth, and Richard P Westwood. 1998.  Proactive interference, recency, and associative strength: Comparisons of black-capped chickadees and dark-eyed juncos.  _Animal Learning & Behavior_ 26 (1998), 475–485.
  * Han et al. (2022) Zhongyi Han, Haoliang Sun, and Yilong Yin. 2022.  Learning transferable parameters for unsupervised domain adaptation.  _IEEE Transactions on Image Processing_ 31 (2022), 6424–6439.
  * Harris et al. (2021) Celia B Harris, John Sutton, and Amanda J Barnier. 2021.  Autobiographical forgetting, social forgetting, and situated forgetting*: Forgetting in context.  In _Current Issues in Memory_. Routledge, 144–178.
  * Hawhee and Vivian (2009) Debra Hawhee and Bradford Vivian. 2009.  Review Essay: On the Language of Forgetting.  _Quarterly Journal of Speech_ 95, 1 (2009), 89–104.  https://doi.org/10.1080/00335630802621094
  * Hayes et al. (2017) Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. 2017.  Logan: Membership inference attacks against generative models.  _arXiv preprint arXiv:1705.07663_ (2017).
  * He and Wu (2019) He He and Dongrui Wu. 2019.  Transfer learning for brain–computer interfaces: A Euclidean space data alignment approach.  _IEEE Transactions on Biomedical Engineering_ 67, 2 (2019), 399–410.
  * He et al. (2021) Yingzhe He, Guozhu Meng, Kai Chen, Jinwen He, and Xingbo Hu. 2021.  Deepobliviate: a powerful charm for erasing data residual memory in deep neural networks.  _arXiv preprint arXiv:2105.06209_ (2021).
  * Hegel (1998) Georg Wilhelm Friedrich Hegel. 1998.  Philosophy of fine art.  (1998).
  * Heller-Roazen (2005) Daniel Heller-Roazen. 2005.  Echolalias: On the forgetting of language.  (2005).
  * Hemmi and Merkle (2009) Jan M Hemmi and Tobias Merkle. 2009.  High stimulus specificity characterizes anti-predator habituation under natural conditions.  _Proceedings of the Royal Society B: Biological Sciences_ 276, 1677 (2009), 4381–4388.
  * Hermer et al. (2021) Ethan Hermer, Ben Murphy, Alexis S. Chaine, and Julie Morand-Ferron. 2021.  Great tits who remember more accurately have difficulty forgetting, but variation is not driven by environmental harshness.  _Scientific Reports_ 11, 1 (2021), 1–11.  https://doi.org/10.1038/s41598-021-89125-3
  * Hertel et al. (2021) Paula T Hertel, Melissa Gerstle, Marco Costanzi, Beatrice Cianfanelli, Alessandro Santirocchi, Stefano Lasaponara, Pietro Spataro, Clelia Rossi-Arnaud, and Vincenzo Cestari. 2021.  Depressive Deficits in Forgetting.  _Psychological Science_ 14, 6 (11 2021), 573–578.  https://doi.org/10.3390/jpm11040241
  * Hoang et al. (2018) Cong Duy Vu Hoang, Philipp Koehn, Gholamreza Haffari, and Trevor Cohn. 2018.  Iterative back-translation for neural machine translation. In _2nd Workshop on Neural Machine Translation and Generation_. Association for Computational Linguistics, 18–24.
  * Hoel (2021) Erik Hoel. 2021.  The overfitted brain: Dreams evolved to assist generalization.  _Patterns_ 2, 5 (2021).
  * Huang et al. (2022) Long-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. 2022.  Frustratingly easy transferability estimation. In _International Conference on Machine Learning_. PMLR, 9201–9225.
  * Huang et al. (2010) Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou. 2010.  Active learning by querying informative and representative examples.  _Advances in neural information processing systems_ 23 (2010).
  * Ishii and Sugiyama (2021) Masato Ishii and Masashi Sugiyama. 2021.  Source-free domain adaptation via distributional alignment by matching batch normalization statistics.  _arXiv preprint arXiv:2101.10842_ (2021).
  * Izzo et al. (2021) Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. 2021.  Approximate data deletion from machine learning models. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 2008–2016.
  * Jakobson and Prieto (1941) Roman Jakobson and Luis Jorge Prieto. 1941.  Kindersprache, aphasie und allgemeine lautgesetze.  (1941).
  * James (2007) William James. 2007.  _The principles of psychology_. Vol. 1.  Cosimo, Inc.
  * Jensen et al. (2011) Evelyn L Jensen, Lawrence M Dill, and James F Cahill Jr. 2011.  Applying behavioral-ecological theory to plant defense: light-dependent movement in Mimosa pudica suggests a trade-off between predation risk and energetic reward.  _The American Naturalist_ 177, 3 (2011), 377–381.
  * Jin et al. (2020) Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. 2020.  Minimum class confusion for versatile domain adaptation. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16_. Springer, 464–480.
  * Kearns (1998) Michael Kearns. 1998.  Efficient noise-tolerant learning from statistical queries.  _Journal of the ACM (JACM)_ 45, 6 (1998), 983–1006.
  * Keijzer (2004) Merel Keijzer. 2004.  First language attrition: A cross-linguistic investigation of Jakobson’s regression hypothesis.  _International Journal of Bilingualism_ 8, 3 (2004), 389–393.
  * Kierkegaard (1946) Søren Kierkegaard. 1946.  _Kierkegaard anthology_.  Princeton University Press.
  * Kim and Kim (2020) Taekyung Kim and Changick Kim. 2020.  Attract, perturb, and explore: Learning a feature alignment network for semi-supervised domain adaptation. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16_. Springer, 591–607.
  * Kirby (2001) Simon Kirby. 2001.  Spontaneous evolution of linguistic structure-an iterated learning model of the emergence of regularity and irregularity.  _IEEE Transactions on Evolutionary Computation_ 5, 2 (2001), 102–110.
  * Kirby et al. (2014) Simon Kirby, Tom Griffiths, and Kenny Smith. 2014.  Iterated learning and the evolution of language.  _Current opinion in neurobiology_ 28 (2014), 108–114.
  * Koh and Liang (2017) Pang Wei Koh and Percy Liang. 2017.  Understanding black-box predictions via influence functions. In _International conference on machine learning_. PMLR, 1885–1894.
  * Köpke (2007) Barbara Köpke. 2007.  Language attrition at the crossroads of brain, mind, and society.  (2007), 9–37.  https://doi.org/10.1075/sibil.33.03kop
  * Kornell et al. (2009) Nate Kornell, Matthew Jensen Hays, and Robert A Bjork. 2009.  Unsuccessful retrieval attempts enhance subsequent learning.  _Journal of Experimental Psychology: Learning, Memory, and Cognition_ 35, 4 (2009), 989.
  * Korteling et al. (2021) J E (Hans) Korteling, Gillian C van de Boer-Visschedijk, Romy A M Blankendaal, Rudy C Boonekamp, and Aletta R Eikelboom. 2021.  Human-versus artificial intelligence.  _Frontiers in artificial intelligence_ 4 (2021), 622364.
  * Krondorfer (2008) Björn Krondorfer. 2008.  Is forgetting reprehensible? Holocaust remembrance and the task of oblivion.  _Journal of Religious Ethics_ 36, 2 (2008), 233–267.
  * Kumar et al. (2022) I Elizabeth Kumar, Keegan E Hines, and John P Dickerson. 2022.  Equalizing credit opportunity in algorithms: Aligning algorithmic fairness research with us fair lending regulation. In _Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society_. 357–368.
  * Lee and Magill (1983) Timothy D Lee and Richard A Magill. 1983.  The locus of contextual interference in motor-skill acquisition.  _Journal of Experimental Psychology: Learning, Memory, and Cognition_ 9 (1983), 730–746.  https://doi.org/10.1037/0278-7393.9.4.730
  * Lefebvre et al. (2004) Louis Lefebvre, Simon M Reader, and Daniel Sol. 2004.  Brains, innovations and evolution in birds and primates.  _Brain Behavior and Evolution_ 63, 4 (2004), 233–246.
  * Li and Bowling (2019) Fushan Li and Michael Bowling. 2019.  Ease-of-teaching and language structure from emergent communication.  _Advances in neural information processing systems_ 32 (2019).
  * Li et al. (2022a) Keqiuyin Li, Jie Lu, Hua Zuo, and Guangquan Zhang. 2022a.  Dynamic classifier alignment for unsupervised multi-source domain adaptation.  _IEEE Transactions on Knowledge and Data Engineering_ 35, 5 (2022), 4727–4740.
  * Li et al. (2020a) Xingjian Li, Haoyi Xiong, Haozhe An, Cheng-Zhong Xu, and Dejing Dou. 2020a.  Rifle: Backpropagation in depth for deep transfer learning through re-initializing the fully-connected layer. In _International Conference on Machine Learning_. PMLR, 6010–6019.
  * Li et al. (2020b) Xiang Li, Wei Zhang, Hui Ma, Zhong Luo, and Xu Li. 2020b.  Partial transfer learning in machinery cross-domain fault diagnostics using class-weighted adversarial networks.  _Neural Networks_ 129 (2020), 313–322.
  * Li et al. (2022b) Yuyuan Li, Xiaolin Zheng, Chaochao Chen, and Junlin Liu. 2022b.  Making recommender systems forget: Learning and unlearning for erasable recommendation.  _arXiv preprint arXiv:2203.11491_ (2022).
  * Liang et al. (2021) Jian Liang, Kaixiong Gong, Shuang Li, Chi Harold Liu, Han Li, Di Liu, and Guoren Wang. 2021.  Pareto domain adaptation.  _Advances in Neural Information Processing Systems_ 34 (2021), 12917–12929.
  * Liang et al. (2020) Jian Liang, Dapeng Hu, and Jiashi Feng. 2020.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In _International conference on machine learning_. PMLR, 6028–6039.
  * Lin and Jung (2017) Yuan-Pin Lin and Tzyy-Ping Jung. 2017.  Improving EEG-based emotion classification using conditional transfer learning.  _Frontiers in human neuroscience_ 11 (2017), 334.
  * Lipton et al. (2018) Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. 2018.  Detecting and correcting for label shift with black box predictors. In _International conference on machine learning_. PMLR, 3122–3130.
  * Liu et al. (2022b) Bo Liu, Qiang Liu, and Peter Stone. 2022b.  Continual learning and private unlearning. In _Conference on Lifelong Learning Agents_. PMLR, 243–254.
  * Liu et al. (2019) Ruijun Liu, Yuqian Shi, Changjiang Ji, and Ming Jia. 2019.  A survey of sentiment analysis based on transfer learning.  _IEEE access_ 7 (2019), 85401–85412.
  * Liu et al. (2022a) Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang, and Jianfeng Ma. 2022a.  Backdoor defense with machine unlearning. In _IEEE INFOCOM 2022-IEEE Conference on Computer Communications_. IEEE, 280–289.
  * Liu et al. (2021) Zihan Liu, Lubin Meng, Xiao Zhang, Weili Fang, and Dongrui Wu. 2021.  Universal adversarial perturbations for CNN classifiers in EEG-based BCIs.  _Journal of Neural Engineering_ 18, 4 (2021), 0460a4.
  * Long et al. (2015) Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. 2015.  Learning transferable features with deep adaptation networks. In _International conference on machine learning_. PMLR, 97–105.
  * Long et al. (2013) Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. 2013.  Transfer feature learning with joint distribution adaptation. In _Proceedings of the IEEE international conference on computer vision_. 2200–2207.
  * Lu et al. (2020) Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin. 2020.  Countering language drift with seeded iterated learning. In _International Conference on Machine Learning_. PMLR, 6437–6447.
  * MacLeod et al. (2009) Malcolm D MacLeod, Jo Saunders, and Laura Chalmers. 2009.  Retrieval-induced forgetting: The unintended consequences of unintended forgetting.  In _Current issues in applied memory research_. Psychology Press, 64–85.
  * Madasu and Vijjini (2021) Avinash Madasu and Anvesh Rao Vijjini. 2021.  Sequential domain adaptation through elastic weight consolidation for sentiment analysis. In _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 4879–4886.
  * Madry et al. (2017) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_ (2017).
  * Mahadevan and Mathioudakis (2021) Ananth Mahadevan and Michael Mathioudakis. 2021.  Certifiable machine unlearning for linear models. (2021).
  * Mahadevan and Mathioudakis (2022) Ananth Mahadevan and Michael Mathioudakis. 2022.  Certifiable unlearning pipelines for logistic regression: An experimental study.  _Machine Learning and Knowledge Extraction_ 4, 3 (2022), 591–620.
  * Mantelero (2013) Alessandro Mantelero. 2013.  The EU Proposal for a General Data Protection Regulation and the roots of the ‘right to be forgotten’.  _Computer Law & Security Review_ 29, 3 (2013), 229–235.
  * Marchant et al. (2022) Neil G Marchant, Benjamin I P Rubinstein, and Scott Alfeld. 2022.  Hard to forget: Poisoning attacks on certified machine unlearning. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 7691–7700.
  * Margalit (2002) Avishai Margalit. 2002.  _The ethics of memory_.  Harvard University Press.
  * Martens (2020) James Martens. 2020.  New insights and perspectives on the natural gradient method.  _The Journal of Machine Learning Research_ 21, 1 (2020), 5776–5851.