11institutetext: Masaryk University, Faculty of Informatics
Brno, Czech Republic
11email<EMAIL_ADDRESS>
# Trust Management in the Internet of Everything

Barbora Buhnova

###### Abstract

Digitalization is leading us towards a future where people, processes, data
and things are not only interacting with each other, but might start forming
societies on their own. In these dynamic systems enhanced by artificial
intelligence, trust management on the level of human-to-machine as well as
machine-to-machine interaction becomes an essential ingredient in supervising
safe and secure progress of our digitalized future. This tutorial paper
discusses the essential elements of trust management in complex digital
ecosystems, guiding the reader through the definitions and core concepts of
trust management. Furthermore, it explains how trust-building can be leveraged
to support people in safe interaction with other (possibly autonomous) digital
agents, as trust governance may allow the ecosystem to trigger an auto-immune
response towards untrusted digital agents, protecting human safety.

###### Keywords:

T

rust Management, Internet of Everything, Autonomous Ecosystems, Software
Architecture

##  1 Introduction

The Internet of Everything (IoE), which is constructed by networking people,
processes, data and things (e.g., devices, appliances, vehicles) [8], is
reshaping the vision of how humans and autonomous systems could engage in
common collaborative goals. While this advanced digitalization opens up
numerous opportunities for our sustainable future, it also imposes new safety
concerns regarding the undesirable behavior of autonomous digital actors in
these ecosystems (e.g., a driverless car intentionally causing dangerous road
consequences).

#### Trustworthiness does not guarantee trust.

Although approaches exist to ensure trustworthiness of the individual
ecosystem components, via improving their security, reliability, availability,
etc., trust is difficult to get addressed via such solutions due to the fact
that trust is conceptually a belief about a system that is out of our control.
Therefore, although the system might declare its trustworthiness, this does
not give us any guarantee that it can be trusted. This is an effect of the
fact that malicious agents can enter the ecosystem with the intention to
disrupt the basic functionality of a network for malicious purposes while
keeping their malicious intentions hidden behind announced collaborative goals
[7, 28].

#### Immune-response capabilities of trust-based ecosystems.

One promising strategy in supporting the safe progress of human-machine
partnership in the presence of malicious entities within IoE is based on
involving the intelligence inside the individual ecosystem agents, which could
be directed towards real-time detection of trust-breaking behaviour in other
agents [15] so that problematic agents are detected and isolated before they
can engage in harmful behaviour. An effective trust-management system is,
therefore, imperative for detecting and dealing with misbehaving agents before
they jeopardize the ecosystem functionality [28].

#### Scope of this paper.

This paper serves as supporting material for a tutorial that gives an
introduction to the main elements of trust management in complex digital
ecosystems. We discuss the understanding of trust in traditional domains, such
as psychology, sociology or economics, leading us to the emerging debates on
how trust can be understood in the context of the Internet of Everything. We
go through the individual components of trust management, including trust
evaluation and its influencing factors (e.g., feeling vulnerable), trust
propagation (e.g., via reputation), and trust updates (e.g., via dynamic
certification schemes). We give an overview of the current open discussions
about the centralized/global vs decentralized/local trust management
strategies and discuss their risks and related trust attacks (e.g, self-
promotion or whitewashing).

As the topic of trust governance in the context of dynamic autonomous
ecosystems involving human-machine interplay is very recent, the paper also
presents some insights from the interactive part of the tutorial, which was
organized as a working session around the most pressing research questions on
the topic, touching e.g. the topics of higher values in digital ecosystems
(such as ethics or fairness), privacy concerns and human-inherent subjectivity
of trust.

##  2 Understanding Trust

While the research on trust in the context of Internet of Everything is quite
recent, substantial body of knowledge exists in various research areas, namely
discussing the definition of trust and approaches to building and managing
trust. The main concepts relevant in the area of the Internet of Everything
are outlined in this section.

Figure 1: Trust Definition for the Context of Internet of Everything (inspired
from [22]).

###  2.1 Definitions of Trust

Trust is a complex phenomenon that has been studied in different contexts.
This section outlines the definitions of trust in different domains.

  * •

_Trust in Sociology:_ Subjective probability that another party will perform
an action that will not hurt my interest under uncertainty or ignorance [9].

  * •

_Trust in Philosophy:_ Risky action deriving from personal, moral
relationships between two entities [21].

  * •

_Trust in Economics:_ Expectation upon a risky action under uncertainty and
ignorance based on the calculated incentives for the action [17].

  * •

_Trust in Psychology:_ Cognitive learning process obtained from social
experiences based on the consequences of trusting behaviors [27].

  * •

_Trust in International Relations:_ Belief that the other party is trustworthy
with the willingness to reciprocate cooperation [20].

  * •

_Trust in Automation:_ Attitude or belief that an agent will help achieve
another agent’s goal in a situation characterized by uncertainty and
vulnerability [22].

In what follows, i.e. in IoE ecosystems, we adopt the definition inspired from
the context of automation, extending the understanding of the agents to
intelligent systems as well as humans and things, i.e. understanding trust as
_”the attitude or belief of an agent (trustor) to achieve a specific goal in
interaction with another agent (trustee) under uncertainty and vulnerability”_
(Figure 1).

###  2.2 Characteristics of Trust

Trust in general is characterized by the following three characteristics [10,
28]:

  * •

_Subjective:_ Trust is viewed using the centrality of an agent, wherein the
trust is computed based on trustor’s observation (i.e., direct trust) as well
as the opinion (i.e., feedback or indirect trust) of the other agents.

  * •

_Asymmetric:_ Trust is an asymmetric property, i.e., if an agent A trusts
another agent B, it does not guarantee that B also trusts A.

  * •

_Transitive:_ System agent A is more likely to develop trust towards an agent
B if A trust agent C that trusts agent B.

###  2.3 Scope of Trust Evaluation

When evaluating trust, the scope of the considered context makes a difference.
The scope that is typically considered is [10, 28]:

  * •

_Local:_ It represents the trust based on an agent-agent relationship, wherein
an agent evaluates the trustworthiness of another agent using local
information such as its current observation and past experience.

  * •

_Global:_ In comparison to the local trust, the global trust is considered as
the reputation of an agent within the ecosystem, wherein the reputation each
agent might be influenced by the local trust score of each of the other agents
in the ecosystem.

  * •

_Context-specific:_ Trust of an agent towards another agent varies with
context. A trust relation between the agents is usually dynamic and depends on
multiple factors such as temporal factors or location.

Figure 2: Trust Management Components.

###  2.4 Trust Components

Trust of one agent (trustor) in another (trustee) is being built via combining
the mechanisms of direct trust (based on local and highly context-specific
experience with the other agent) and indirect trust (possibly global
reputation of the agent) [29]. The two are later combined via the component of
trust decision (see Figure 2).

  * •

_Direct Trust_ represents an individual judgment by the trustor from its
direct interaction with the trustee. Specifically, it is being based on a
combination of present and past experience (direct observation) of the trustor
with the trustee. In this regard, mechanisms need to be in place to evaluate
the experience during runtime interaction of the trustor with the trustee
(possibly mimicking human cognitive processes) without exposing its
vulnerabilities.

  * •

_Indirect Trust_ is being inferred based on propagated opinions from different
trust paths to judge the reputation of a trustee. The primary sources are thus
the trusted peers of the trustor (their opinions and recommendation) and an
authority overseeing the reputation of the trustee within the ecosystem. In
this regard, mechanisms need to be in place to update the reputation and
propagate it through the ecosystem.

  * •

_Context Information_ influences the trust decision to reflect on the current
situation of the trustor in which the trust decision is being made (e.g. how
vulnerable it feels in interaction with the trustee, what risks it takes,
whether the trustee will be held accountable in case of malicious behaviour,
whether the possible harm is reversible or would be compensated, i.e.
reparability is ensured). Besides, broader contextual information also
influences the direct and indirect trust computation.

##  3 Trust Management Activities

The individual trust management activities are connected to the evaluation of
the direct trust, management of the indirect trust, and the combination of the
two in a trust decision [28].

###  3.1 Direct-Trust Evaluation

The assessment of direct trust involves the computation of trust metrics
characterizing the present/past experience together with aggregating the
results into a trust score via trust formation.

#### Trust Metrics

refer to the features that are chosen and combined in trust computation. These
features can refer to:

  * •

_QoS Metrics_ , which represent the confidence that an agent is able to offer
high quality of the delivered service, e.g. in terms of reliability,
availability, security or accuracy [30].

  * •

_Social Metrics_ , which represent the social relationships among ecosystem
agents, which can include integrity, benevolence, honesty, friendship,
openness, altruism, or unselfishness [26, 16].

#### Trust Formation

forms the trust based on one of the following strategies [28]:

  * •

_Single aspect_ , e.g., in terms of a positive or negative threshold for a
specific trust metric.

  * •

_Multiple aspects_ , i.e., more sophisticated trust model that includes
various trust metrics [3].

  * •

_Aspect aggregation_ , i.e. multiple aspects aggregated in a single trust
score, for instance via a Weighted Sum, Belief Theory, Bayesian System, Fuzzy
Logic, Regression Analysis, or Machine Learning [28].

###  3.2 Indirect-Trust Management

The management of the indirect trust involves trust updates and trust
propagation to ensure up-to-date opinion and reputation information that can
feed trust decisions.

#### Trust Update.

At the end of a transaction or at any specified time interval, trust
reputation of a trustee needs to be updated, which typically happens in one of
the following ways:

  * •

_Event-driven:_ trust is updated after each transaction or once an event has
occurred [4], which however increases the traffic overhead in the network.

  * •

_Time-driven:_ trust is collected and updated periodically after a given
interval of time [25].

  * •

_Hybrid:_ trust is updated periodically and/or in case of an event (after an
interaction) [30].

#### Trust Propagation.

Trust propagation facilitates the understanding of how the trust propagates in
the ecosystem and is generally categorized in [28]:

  * •

_Centralized schemes_ rely on a centralized entity that is primarily
responsible for (a) gathering trust-related information for the purpose of
trust computation and (b) propagating it in the ecosystem [26]. Centralized
controlled frameworks are vulnerable to a single point of failure, which can
cause the entire ecosystem trust to collapse.

  * •

_Distributed schemes_ rely on the individual agents being responsible for both
trust computation and propagation within the ecosystem without any central
authority [4]. This scheme faces other inherent challenges, like honest trust
computation, managing computational capabilities and unbiased trust
propagation within the ecosystem.

  * •

_Hybrid schemes_ are aimed at mitigating the challenges of the two earlier
schemes, dividing the propagation into two categories, i.e., (1) locally
distributed and globally centralized, and (2) locally centralized and globally
distributed [26].

###  3.3 Trust Decision

After computing the direct-trust score of a trustee and understanding its
indirect-trust reputation (from the peers and/or central authority), the main
purpose of a trust management system is to identify whether the trustee is to
be considered trustworthy or untrustworthy by means of [28]:

  * •

_Threshold-based decision:_ where the decision is taken on the basis of either
a rank-based function or a (possibly dynamically changing) threshold value.

  * •

_Policy-based decision:_ where more complex policies are used to identify and
decide whether an agent is classified as trustworthy or not by using
contextual information, e.g. in terms of location or temporal factors.

Table 1: Social Values in terms of Ethical Principles [18] 1 |  Transparency |  Transparency, explainability, explicability, understandability, interpretability, communication, disclosure, showing
---|---|---
2 |  Justice and fairness |  Justice, fairness, consistency, inclusion, equality, equity, (non-)bias, (non-)discrimination, diversity, plurality, accessibility, reversibility, remedy, redress, challenge, access and distribution
3 |  Non-maleficence |  Non-maleficence, security, safety, harm, protection, precaution, prevention, integrity (bodily or mental), non-subversion
4 |  Responsibility |  Responsibility, accountability, liability, acting with integrity
5 |  Privacy |  Privacy, personal or private information
6 |  Beneficence |  Benefits, beneficence, well-being, peace, social good, common good
7 |  Freedom and autonomy |  Freedom, autonomy, consent, choice, self-determination, liberty, empowerment
8 |  Trust |  Trust
9 |  Sustainability |  Sustainability, environment (nature), energy, resources (energy)
10 |  Dignity |  Dignity
11 |  Solidarity |  Solidarity, social security, cohesion

##  4 Trust-Based Mechanisms for Ecosystem Wellbeing

Once we have a way to assess the trustworthiness of the individual ecosystem
members, we can employ the information in the actual support of the wellbeing
of the ecosystem, e.g. in terms of its safety, fairness, solidarity, or other
values (see Table 1) [12, 18]. Some examples of wellbeing-promoting mechanisms
follow.

#### Incentives in Terms of Reward and Punishment.

One of the essential mechanisms for promoting ecosystem wellbeing is the
ability of the ecosystem to incentivize behaviours that contribute to
promoting the agreed values in the ecosystem. This is being done via the
mechanisms of reward and punishment, which can be based e.g. on the actual
trust score of the agent, or on the relative change (recent increase/decrease)
of the trust score [13, 14].

#### Evidence Collection for Justification and Reparation.

To ensure fairness in the ecosystem, mechanisms need to be in place that not
only help us see there might be discrimination or unfairness happening in the
system (e.g. newly joining agents having a hard time gaining sufficient trust
needed to be allowed to participate fully in the ecosystem), but also help us
justify decisions that might be opposed by certain ecosystem agents, or to
correct trust misjudgment [13]. Besides, evidence collection is a promising
tool to detect trust attacks (discussed in Section 5) and hidden malicious
intentions of ecosystem agents before they get fully revealed (discussed in
Section 7).

#### Safety Assurance in the Face of Untrusted Agents.

One of the essential ingredients of the _immune-response capabilities of
trust-based ecosystems_ envisioned in Section 1 is the ability to isolate the
misbehaving agents to protect the safety of the ecosystem. While this might be
easy to do in the scenarios where trust navigates spreading of information or
services (the trustor simply avoids using the knowledge and services by
untrustworthy agents), it becomes challenging when trust shall navigate
physical safety (e.g. to avoid collision with a malicious vehicle). In the
latter case, an action by the authority overseeing agents’ reputation might be
needed, e.g., imposing restrictions of the agent function in effect of the
reputation falling below certain threshold. In such cases, it is crucial to
apply the mechanisms of adaptive function restriction to mitigate the risk of
misjudging the trustworthiness of an agent [11].

Figure 3: Trust Attacks [28].

##  5 Trust Attacks

Given the fact that trust is largely influenced by the belief and perception
of the individual ecosystem agents, there is a high risk of trust attacks,
which can take different forms (see Figure 3), summarized below.

#### Individual Attacks:

refer to the attacks launched by an individual agent, which can take form of
[28, 2]:

  * •

_Self-Promoting Attacks:_ In this type of attack, an agent promotes its
significance by providing good recommendation for itself so as to be selected
as a service provider, and then acts maliciously.

  * •

_Whitewashing Attacks:_ In this attack, an agent exits and re-joins the
ecosystem to recover its reputation and to wash-away its own bad reputation.

  * •

_Discriminatory Attacks:_ In this type of attack, an agent explicitly attacks
other agents that do not have common friends with it, i.e. it performs well
for a particular service/agent and badly for some other services/agents.

  * •

_Opportunistic Service Attacks:_ In this type of attack, an agent might offer
a great service to improve its reputation when its reputation falls too low.

  * •

_On-Off Attacks:_ In this type of attack, an agent provides good and bad
services on and off (randomly) to avoid being labeled as a low-reputed agent.

#### Collusion-based Attacks:

represent the attacks launched by a group of agents to either provide a high
rating or low rating to a particular agent, such as [24]:

  * •

_Bad-Mouthing Attacks:_ In this type of attack, a group of agents diminishes
the reputation of a trustworthy agent within the ecosystem by providing bad
recommendations about it.

  * •

_Ballot-Stuffing Attacks:_ In this type of attack, a group of agents boosts
the reputation of bad agents within the ecosystem by providing good
recommendations for them.

##  6 Scenarios

This section discusses examples of trust-management scenarios in the context
of vehicular ecosystems, where trust management is crucial to avoid safety
consequences of misbehaviour of individual smart agents.

Figure 4: Interaction Scenario [15].

###  6.1 Collision Avoidance with Misbehaving Vehicle

Consider a situation of a vehicle (trustor) getting in the proximity of
another vehicle (trustee), which could possibly be misbehaving (either due to
a fault or being intentionally malicious) with an increased risk of causing a
collision. In such a scenario, the possibility to collect present experience
influencing the direct-trust evaluation is limited due to the risk of
collision in case of very close interaction. A way to support the experience
building in such a case, which we have explored in [15], is illustrated in
Figure 4 on an example of two drones. Consider Drone 1 (trustor) evaluating
its direct trust in Drone 2 (trustee). The present-experience evaluation
supporting the trust building in this scenario is formed in the following
steps: First, Drone 1 asks Drone 2 to declare its intended behaviour in form
of a Digital Twin (declared behaviour) that can be used by Drone 1 within the
direct-trust evaluation process. In response, Drone 2 shares its Digital Twin
(DT), which is checked by Drone 1 for its trustworthiness (absence of
malicious actions) and then (if harmless), Drone 1 employs the DT in runtime
compliance checking of the actual behaviour of Drone 2 against its Digital
Twin (D2-DT). If at runtime Drone 2 deviates from its declared behaviour,
Drone 1 can report Drone 2 to the ecosystem governance authority, which can
internally decrease its reputation and start closer surveillance of Drone 2,
possibly even employing some level of safeguarding of its operation (if the
reputation falls too low or the associated threat gets too high).

###  6.2 Trust in Run-time Update Downloaded to a Vehicle

The evolution in smart driving functions is pushing for frequent runtime
updates being downloaded to the vehicles to improve their functionality,
quality (e.g. vulnerability correction) or A/B test future software evolution.
While numerous procedures are in place to ensure the trustworthiness of the
software updates being downloaded to the vehicle, the risk is still there that
these updates can contain vulnerabilities or hidden intended faults
manifesting into malicious behavior [6]. This can be caused even
intentionally, with the help of an insider attacker engaged in the software
update development. In this scenario, the vehicle shall have mechanisms in
place to gradually build trust in the new software update before it allows it
to control its critical driving functions, where both direct-trust evaluation
and indirect-trust management can be integrated into the trust-building
process. In [6], we have explored the potential of the Digital Twin of the
update also in this scenario, to facilitate the experience gaining via
predictive simulation of the update with the help of the Digital Twin.

###  6.3 Trust-based Vehicle Admission in a Platoon

In the future automotive ecosystems, the formation of vehicle platoons is
envisioned to support the traffic flow and reduced fuel consumption for more
sustainable smart mobility. Consider a scenario where at the entry point of a
highway, a vehicle can download a software solution that would assist it in
platooning coordination [1]. When driving in a platoon, vehicles benefit from
reduced fuel consumption due to reduced air friction, but the complex dynamics
of collaborative and competitive forces could raise trust concerns about the
safety of riding in such close proximity to other vehicles [7]. In this
scenario, a vehicle in the platoon ecosystem can suddenly accelerate or
decelerate and cause multiple car crashes within the platoon. Such a behavior
can be caused by a fault or malicious intentions of the individual vehicle, or
by a logic bomb in the downloaded software solution responsible for the
platooning coordination, caused by a vulnerability inserted e.g. by a
malicious insider working for the software provider. In this way, the solution
can combine the two approaches outlined in the previous two scenarios.

###  6.4 Human-to-Vehicle Trust Building in Autonomous Driving

The future automotive systems will be subject to a gradual process towards
fully autonomous driving. During the process, human responsibilities will be
gradually replaced by autonomous driving functions, at a speed governed by the
level of trust of the humans in the autonomous driving capabilities of the
vehicles. The acceptance of any new level of autonomy shall be built up
gradually and governed by trust-management mechanisms, to ensure acceptance by
both drivers and pedestrians. In [5], we envision the incremental development
of autonomy on the road as a win-win situation for instrumenting the building
of trust on both sides. In order to build a trusting relationship, the driver
and the system shall in our view evolve together in a process that creates
sustainable trust and supports the interchange between the human trustor with
a smart software trustee.

###  6.5 Information Exchange in Coordinated Ecosystem Moves

An interesting scenario that came up in the tutorial discussion is linked to
the high reliance on information exchange in coordinated ecosystem moves, e.g.
when one vehicle is overtaking another one (whether driver-less or with a
driver, possibly a cyclist) and needs to know its planned speed, when an
emergency vehicle needs to be given priority when passing through a crossing,
or when a human needs to be let to cross the street. Trust in this case needs
to be built not only in the other agent in the ecosystem, but also in the
actual information being exchanged, which could be influenced e.g. by a faulty
sensor. In this regard, an interesting trust scenario is also between a
vehicle and its own sensor, which might be faulty. And a great comment was
given also about the individual pieces of information (e.g. that there is ice
on the road in a particular location), which could act as the first-class IoE
ecosystem entities with their own trust score governed by indirect trust
management.

##  7 Challenges of Trust Management in IoE

Besides the trust attacks, which bring a considerable challenge in designing
robust trust management systems, numerous other challenges of trust management
in IoE have been discussed with the tutorial participants, and also outlined
in literature [28]. We summarize the key challenges below.

#### Situational Scope of Trust.

There is high context-dependence of trust, namely in terms of the location and
time in their broader understanding [19]. Besides, the peer network can change
depending on the context of trust evaluation, similarly to the human context,
where a person who can be trusted in the context of their own expertise, might
be still untrustworthy in a field of knowledge where the person relies on the
opinion of peers influenced by misinformation. This is also a reason why
reputation shall be contextualized, as the reputation of an agent might be
influenced by various dimensions of the context in which the trustworthiness
is judged.

#### Subjectivity of Trust.

Trust is a highly subjective concept, being influenced not only by the context
of the trustor, but also the individual trust goal of the trustor (the reason
for which the trust is being established) and the factors inherent to the
individual trustor (e.g. the willingness to take risk and trust despite the
uncertainty, or willingness to rely on the peers recommendation). Furthermore,
the peer network that influences the trust decision might itself be subjective
or biased, which causes the propagation of subjectivity and bias in the
ecosystem.

#### Default Trust Score of New Agents.

The current trust management solutions presume the initial trust score of a
newly joined agent to be within the range $<0;0.5>$ [28]. Choosing the initial
trust score (so-called cold start problem) is challenging on both sides of the
interval. When the initial trust score is close to 0.5 (i.e. neutral, neither
trustworthy nor untrustworthy) [3], this might lead to a malicious agent
abusing its trust level to cause harm before it identifies as untrustworthy,
or to white-wash its reputation by leaving and re-joining the ecosystem under
new identity. On the other hand, when the initial trust score is close to 0,
the new agent might never get the chance to engage with the rest of the
ecosystem (i.e., not being trusted).

#### Trust Erosion.

Besides the fact that the trust score of any agent in the IoE ecosystem varies
with time, its trust score (or reputation) is also subject to decay if there
are no or too few interactions the agent is involved in [29]. It is therefore
important to consider the trust values lifespan where the trust score of
inactive agents shall be subject to decay after a particular duration of time
[28].

#### Detection of Hidden Malicious Intentions.

As the overview of trust attacks above has shown, the malicious intentions of
the IoE ecosystem agents can be very well hidden and hard to detect.
Furthermore, when we employ very sensitive methods of detection, e.g. the
discussed runtime compliance checking of the declared and actual trustee
behaviour [15], we run into the risk of misjudging the trustworthiness of the
agent, possibly reporting it as suspicious due to a slight deviation in the
declared-actual behaviour although it might be well-behaving (with a deviation
related to the accuracy of the declared behaviour). In that regard, it is
crucial to have mechanisms in place for long-term evidence collection that can
support richer reasoning on the behavioral patterns of the agents in the
ecosystem, indicating e.g. the patterns resembling trust attacks or hiding the
attempts of misbehaviour.

#### Building Trust in the Trustworthy.

Although much of the text has been centered around trust management directed
towards detecting untrustworthy agents, it is equally important to explore
methods supporting trust building in the trustworthy agents. Indeed, the fact
that an agent is trustworthy does not guarantee it will be trusted by others,
especially in face of trust attacks or biased peer opinion that might be
directed against it. One of the promising techniques in promoting trust in
this context is based on the ability of an agent to give evidence of its
trustworthiness. In the context of AI-enhanced agents, for instance, this
might be based on the explanation of agents’ actions, which is shown to have
positive influence on human participation and willingness to engage with the
digital agents [23].

#### High Degree of Dynamism and Uncertainty in IoE.

As illustrated by different examples in this section, the dynamism and
uncertainty inherent to the Internet of Everything is enormous. Agents can
join and leave the ecosystem anytime, and they have very limited information
about each other, facing high unpredictability of other agents and their
environment. In such conditions, crucial information that is needed to make a
trust decision might be missing, pushing trust models to include built-in
mechanisms to deal with uncertainty and bias in the decisions.

##  8 Conclusion

In the context of Internet of Everything (IoE), human and digital agents form
coalitions for achieving collaborative or individual goals. In a constant risk
of malicious attacks, ecosystem participants ask for strong guarantees of
their collaborators’ trustworthiness, which can be achieved via the mechanisms
of trust management. In this paper, we have presented the emerging research
field of trust management in the context of IoE, discussing the definitions,
concepts, examples of scenarios and research challenges relevant to the
community of software architecture, which undergoes the transition from rather
stable systems of systems towards dynamic autonomous ecosystems [1].

## Acknowledgement

The work was supported by ERDF ”CyberSecurity, CyberCrime and Critical
Information Infrastructures Center of Excellence” (No.
CZ.02.1.01/0.0/0.0/16_019/ 0000822).

## References

  * [1] Capilla, R., Cioroaica, E., Buhnova, B., Bosch, J.: On autonomous dynamic software ecosystems. IEEE Transactions on Engineering Management 69(6), 3633–3647 (2022)
  * [2] Chahal, R.K., Kumar, N., Batra, S.: Trust management in social internet of things: A taxonomy, open issues, and challenges. Computer Communications 150, 13–46 (2020)
  * [3] Chen, R., Bao, F., Guo, J.: Trust-based service management for social internet of things systems. IEEE transactions on dependable and secure computing 13(6), 684–696 (2015)
  * [4] Chen, R., Guo, J., Bao, F.: Trust management for soa-based iot and its application to service composition. IEEE Transactions on Services Computing 9(3), 482–495 (2014)
  * [5] Cioroaica, E., Buhnova, B., Kuhn, T., Schneider, D.: Building trust in the untrustable. In: 2020 IEEE/ACM 42nd International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS). pp. 21–24. IEEE (2020)
  * [6] Cioroaica, E., Kuhn, T., Buhnova, B.: (Do not) trust in ecosystems. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). pp. 9–12. IEEE (2019)
  * [7] Cioroaica, E., Purohit, A., Buhnova, B., Schneider, D.: Goals within trust-based digital ecosystems. In: 2021 IEEE/ACM Joint 9th International Workshop on Software Engineering for Systems-of-Systems and 15th Workshop on Distributed Software Development, Software Ecosystems and Systems-of-Systems (SESoS/WDES). pp. 1–7. IEEE (2021)
  * [8] Farias da Costa, V.C., Oliveira, L., de Souza, J.: Internet of everything (ioe) taxonomies: A survey and a novel knowledge-based taxonomy. Sensors 21(2), 568 (2021)
  * [9] Gambetta, D., et al.: Can we trust trust. Trust: Making and breaking cooperative relations 13, 213–237 (2000)
  * [10] Ghafari, S.M., Beheshti, A., Joshi, A., Paris, C., Mahmood, A., Yakhchi, S., Orgun, M.A.: A survey on trust prediction in online social networks. IEEE Access 8, 144292–144309 (2020)
  * [11] Halasz, D., Buhnova, B.: Rethinking safety in autonomous ecosystems. In: 2022 17th Conference on Computer Science and Intelligence Systems (FedCSIS). IEEE (2022)
  * [12] Hind Bangui, B.B., Ge, M.: Social internet of things: Ethical ai principles in trust management. In: 14th International Conference on Ambient Systems, Networks and Technologies (ANT-2023). Procedia Computer Science (2023)
  * [13] Hind Bangui, M.G., Buhnova, B.: Deep-learning based reputation model for indirect trust management. In: 14th International Conference on Ambient Systems, Networks and Technologies (ANT-2023). Procedia Computer Science (2023)
  * [14] Hind Bangui, Emilia Cioroaica, M.G., Buhnova, B.: Deep-learning based trust management with self-adaptation in the internet of behavior. In: Proceedings of the 38th Annual ACM Symposium on Applied Computing. ACM (2023)
  * [15] Iqbal, D., Buhnova, B.: Model-based approach for building trust in autonomous drones through digital twins. In: 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC). pp. 656–662. IEEE (2022)
  * [16] Iqbal, D., Buhnova, B., Cioroaica, E.: Digital twins for trust building in autonomous drones through dynamic safety evaluation. In: Proceedings of the 18th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE). Scitepress (2023)
  * [17] James Jr, H.S.: The trust paradox: a survey of economic inquiries into the nature of trust and trustworthiness. Journal of Economic Behavior & Organization 47(3), 291–307 (2002)
  * [18] Jobin, A., Ienca, M., Vayena, E.: The global landscape of ai ethics guidelines. Nature Machine Intelligence 1(9), 389–399 (2019)
  * [19] Khan, W.Z., Hakak, S., Khan, M.K., et al.: Trust management in social internet of things: Architectures, recent advancements, and future challenges. IEEE Internet of Things Journal 8(10), 7768–7788 (2020)
  * [20] Kydd, A.H.: Trust and mistrust in international relations. Princeton University Press (2007)
  * [21] Lagerspetz, O.: Trust: The tacit demand. Kluwer Academic Publishers (1998)
  * [22] Lee, J.D., See, K.A.: Trust in automation: Designing for appropriate reliance. Human factors 46(1), 50–80 (2004)
  * [23] Li, N., Cámara, J., Garlan, D., Schmerl, B.: Reasoning about when to provide explanation for human-involved self-adaptive systems. In: 2020 IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS). pp. 195–204. IEEE (2020)
  * [24] Marche, C., Nitti, M.: Trust-related attacks and their detection: A trust management model for the social iot. IEEE Transactions on Network and Service Management 18(3), 3297–3308 (2020)
  * [25] Namal, S., Gamaarachchi, H., MyoungLee, G., Um, T.W.: Autonomic trust management in cloud-based and highly dynamic iot applications. In: 2015 ITU Kaleidoscope: Trust in the Information Society (K-2015). pp. 1–8. IEEE (2015)
  * [26] Nitti, M., Girau, R., Atzori, L.: Trustworthiness management in the social internet of things. IEEE Transactions on knowledge and data engineering 26(5), 1253–1266 (2013)
  * [27] Rotter, J.B.: Interpersonal trust, trustworthiness, and gullibility. American psychologist 35(1), 1 (1980)
  * [28] Sagar, S., Mahmood, A., Sheng, Q.Z., Pabani, J.K., Zhang, W.E.: Understanding the trustworthiness management in the social internet of things: A survey. arXiv preprint arXiv:2202.03624 (2022)
  * [29] Truong, N.B., Um, T.W., Zhou, B., Lee, G.M.: From personal experience to global reputation for trust evaluation in the social internet of things. In: GLOBECOM 2017-2017 IEEE Global Communications Conference. pp. 1–7. IEEE (2017)
  * [30] Xiao, H., Sidhu, N., Christianson, B.: Guarantor and reputation based trust model for social internet of things. In: 2015 International wireless communications and mobile computing conference (IWCMC). pp. 600–605. IEEE (2015)


# Exploring Counterfactual Alignment Loss towards Human-centered AI

Mingzhou Liu, Xinwei Sun ✉, Ching-Wen Lee, Yu Qiao, Yizhou Wang
liumingzhou<EMAIL_ADDRESS>
###### Abstract

Deep neural networks have demonstrated impressive accuracy in supervised
learning tasks. However, their lack of transparency makes it hard for humans
to trust their results, especially in safe-critic domains such as healthcare.
To address this issue, recent explanation-guided learning approaches proposed
to align the gradient-based attention map to image regions annotated by human
experts, thereby obtaining an intrinsically human-centered model. However, the
attention map these methods are based on may fail to _causally_ attribute the
model predictions, thus compromising their validity for alignment. To address
this issue, we propose a novel human-centered framework based on
counterfactual generation. In particular, we utilize the counterfactual
generation’s ability for causal attribution to introduce a novel loss called
the CounterFactual Alignment (CF-Align) loss. This loss guarantees that the
features attributed by the counterfactual generation for the classifier align
with the human annotations. To optimize the proposed loss that entails a
counterfactual generation with an implicit function form, we leverage the
implicit function theorem for backpropagation. Our method is architecture-
agnostic and, therefore can be applied to any neural network. We demonstrate
the effectiveness of our method on a lung cancer diagnosis dataset, showcasing
faithful alignment to humans.

##  1 Introduction

Current deep learning (DL) models, though have reached remarkable performance,
are often biased towards human non-explainable features in the decision-making
process Najafabadi et al. (2015); Geirhos et al. (2018). Such biases can raise
concerns about trustworthiness, impeding these models from being deployed in
safety-critic domains such as healthcare. To address this issue, apart from
prediction accuracy, it is desirable for DL models to improve their alignment
with human decision-making.

######  Example 1.1.

Take the diagnosis of lung cancer as an example. The standard protocol
followed by radiologists is first to identify the region of interest (i.e.,
nodule), analyze its attributes (e.g., size, margin), and then assess its
risks of developing into cancers Gould et al. (2013). Nevertheless, this
protocol is not adhered to DL models, which primarily rely on data-driven
approaches to learn label-related, but unnecessarily explainable features. As
shown in Fig. 1, deep neural networks mainly utilize background features, such
as those of the rib and spine, for prediction. Though these nodule-irrelevant
features may endow networks with beyond human performance, they are not
understandable for radiologists, thus raising severe safety concerns.

To address this issue, many studies attempted to learn human explainable
features for prediction. Examples include Liu et al. (2021b); Wang et al.
(2022) that incorporated human knowledge with specialized neural network
architectures; and Zheng et al. (2017); Sreedevi et al. (2022) that proposed
decision-making systems based on the human cognitive model. In particular,
explanation-guided learning approaches Ross et al. (2017); Ismail et al.
(2021); Gao et al. (2022); Fei (2022) proposed to align the attention map of
DL models to image regions annotated by human experts, thereby enforcing an
intrinsically human-centered model. However, the attention map these methods
are based on is obtained in a data-driven manner and not necessarily provides
a causal attribution for the model’s decision, which largely compromises their
validity for alignment.

In this paper, we propose a novel human-centered framework based on
counterfactual generation. Specifically, we first leverage the counterfactual
generation to identify features that _causally_ attribute to the model’s
prediction. Then, to align these features with human annotations, we propose a
novel CounterFactual Alignment (CF-Align) loss, which punishes the network
once the identified causal factors do not align with human annotations. To
optimize the proposed loss that involves counterfactual generation utilizing
implicit function forms, we propose an implicit gradient solver rooted in the
implicit function theorem. Returning to the lung cancer example, Fig. 1 shows
that our method can learn explainable features; as a contrast, features
utilized by the baseline DL model are difficult to interpret.

Contributions. To summarize, our contributions are:

  1. 1.

We propose a counterfactual alignment loss for human-centered AI.

  2. 2.

We propose to leverage the implicit function theorem for optimization.

  3. 3.

We achieve more accurate and explainable results than existing methods on
real-world data.

Figure 1: Saliency maps of a baseline DL model Ronneberger et al. (2015) and
our method in lung cancer prediction. Lung nodules are marked with red
bounding boxes.

##  2 Related works

Existing works on explainable AI (XAI) can be classified in various ways
Speith (2022). One such classification distinguishes between human-centered
and technology-centered approaches, with the former focusing on the needs of
specific end-users and the latter on those of AI professionals. Another
classification is based on whether the method is ante-hoc or post-hoc
explainable, which respectively refers to instinct explainability and
explanation after the model has been trained.

Our method belongs to human-centered, ante-hoc XAI. Therefore, we will provide
a detailed discussion of these two branches of works below, leaving the
comprehensive review of other XAI methods to Arrieta et al. (2020) and Ali et
al. (2023).

Human-centered XAI aims to develop AI models that behave in a human-
understandable manner. To achieve this goal, existing methods have proposed
incorporating various human priors, such as knowledge Liu et al. (2021b); Wang
et al. (2022), memory Fu et al. (2014), and cognitive models (Zheng et al.
(2017); Sreedevi et al. (2022), into the design of AI models. By doing so,
their models could behave in a certain way that is similar to humans, thereby
gaining their trust. However, most of these methods require building model-
specific architectures, which makes them difficult to adapt to different
settings. In contrast, we propose a model-agnostic loss for human-centered
learning, which can be easily adapted to various task settings and neural
network architectures.

Ante-hoc XAI refers to the design of AI models that are transparent and
intrinsically explainable. Examples of such models include white-box models,
such as linear regression and decision trees; hybrid models that combine a
white-box model with a black-box model for improved performance Nauta et al.
(2021); Wang & Lin (2021); joint prediction-explanation models that were
trained to provide both predictions and explanations Hind et al. (2019);
Rieger et al. (2020); and methods that achieved explanation through
architecture adjustment Zhang et al. (2018); Chen et al. (2019).

Of particular relevance to our work are explanation-guided learning approaches
Ross et al. (2017); Ismail et al. (2021); Gao et al. (2022); Fei (2022), which
proposed to align model attention or gradient with human-annotated areas.
Nevertheless, these attention regions might not align with the causal factors
influencing the decision-making process, potentially undermining their
appropriateness for alignment purposes. In contrast, our method is based on
the counterfactual generation that intrinsically corresponds to causal
attributions, which thus ensures the alignment of the decision process with
that of domain experts.

Counterfactual explanation. Our work is closely related to the counterfactual
explanation methods Verma et al. (2020); Balasubramanian et al. (2020); Lang
et al. (2021), which aimed at answering what could the outcome have changed to
had input to a model had been changed in a particular way. To implement, they
proposed to minimize alterations that changed the prediction. Such a modified
region can be taken as the causal factor to determine the model’s prediction
Parafita & Vitrià (2019). Compared to the attention-based methods, this method
can causally attribute the model’s decision Parafita & Vitrià (2019). However,
it is important to note that existing counterfactual explanation methods were
designed to explain trained black-box models, while our method aims to design
an intrinsically explainable model (_i.e._ , ante-hoc explainability).

##  3 Preliminary

In this section, we introduce the problem setting, followed by an overview of
the counterfactual explanation that our method builds upon.

Problem setup & notations. We consider the classification scenario, where the
system includes an image $\boldsymbol{x}\in\mathcal{X}\in\mathbb{R}^{p}$ and a
label $y\in\mathcal{Y}$ from an expert annotator. In addition to $y$, we
assume the annotator also provides an explanation $\boldsymbol{e}$ that
explains his/her decision for each image.

In practice, the explanation mainly refers to the region of central interest.
For example, in a radiology report, the radiologist often explains his/her
decision by annotating disease-related/lesion areas. Motivated by this, we
assume that for each sample $(\boldsymbol{x},y)$, the explanation can be
formed as the mask $\boldsymbol{r}\in[0,1]^{p}$, where $r_{i}=1$ meaning that
the feature $i$ belongs to the region of interest. In this regard, the
training data we collect can be denoted as
$\\{\boldsymbol{x}_{i},y_{i},\boldsymbol{r}_{i}\\}_{i=1}^{n}$. With this data,
our objective is to learn a classification model
$f_{\boldsymbol{\theta}}:\mathcal{X}\mapsto\mathcal{Y}$ that i) predicts $y$
accurately, and ii) makes its prediction based on the region $\boldsymbol{r}$.

###  3.1 Counterfactual explanation

We first provide an overview of the counterfactual explanation method Verma et
al. (2020) that our method is based on. By generating a modified image and
observing its difference from the original image, this framework intends to
answer the following counterfactual question: what could have the model made a
different prediction had the input $X$ changed in a particular way?

For this purpose, for the classifier $f_{\boldsymbol{\theta}}$ and each sample
$(\boldsymbol{x},y)$, this framework generates a _counterfactual image_
$\boldsymbol{x}^{*}$ with respect to the counterfactual class $y^{*}\neq y$:

$\boldsymbol{x}^{*}:=\arg\min_{\boldsymbol{x}^{\prime}}\mathrm{CE}(f_{\boldsymbol{\theta}}(\boldsymbol{x}^{\prime}),y^{*})+\lambda
d(\boldsymbol{x},\boldsymbol{x}^{\prime}),$ (1)

where $\mathrm{CE}$ is the cross-entropy loss, $d(\cdot)$ is a distance
measure that constrains the modification to be sparse Verma et al. (2020), and
$\lambda$ is regularization hyperparameter.

With such a modified image $\boldsymbol{x}^{*}$, we can then say the modified
region $\mathbbm{1}(|\boldsymbol{x}^{*}-\boldsymbol{x}|)$ to be the
explanation of $f_{\boldsymbol{\theta}}$’s prediction, in the sense that if
the pixels in this region had taken values from $\boldsymbol{x}^{*}$, then the
prediction would have been $y^{*}$. Indeed, this means for a human-centered
model, the counterfactual modification should belong to the annotated region
of interest. This motivates our _counterfactual alignment loss_ , as
introduced in the next section.

Counterfactual Explanation (CE) vs Adversarial Attack (AA). These two methods
are similar in the optimization form Szegedy et al. (2013); Wachter et al.
(2017), but are intrinsically different in terms of objectives and distance
measures Freiesleben (2022). Specifically, the CE hopes to identify only
decision-relevant features. Therefore, the
$d(\boldsymbol{x},\boldsymbol{x}^{*})$ should take into account the sparsity,
which can be achieved by $\ell_{0}$ or $\ell_{1}$ norm. On the other hand, the
AA hopes to generate samples whose modifications are imperceptible, for which
one typically adopts $\ell_{2}$ or $\ell_{\infty}$ norm.

##  4 Method

In this section, we propose to align the decision basis of the model with
regions provided by human experts. To this end, we propose a regularity term
dubbed as the CounterFactual Alignment loss (CF-Align loss). The goal of this
loss is to enforce the network’s decision basis (identified with Eq. (1)) to
align with the region of interest $\boldsymbol{r}$. In this regard, the
classifier will be forced to make predictions based on human-explainable
features. The overall pipeline of our method is illustrated in Fig. 2.

Figure 2: Overview of our method. The overall objective consists of two parts:
the cross-entropy loss $\mathcal{L}_{cls}$ and the CF-Align loss
$\mathcal{L}_{align}$. In the forward pass, the counterfactual image
$\boldsymbol{x}^{*}(\boldsymbol{\theta})$ is used to compute the
$\mathcal{L}_{align}$, where $\boldsymbol{x}^{*}(\boldsymbol{\theta})$ takes
$\boldsymbol{x}$ and $\boldsymbol{\theta}$ as inputs. In the backward pass, we
compute the gradient of $\mathcal{L}_{align}$ w.r.t. $\theta$ with an implicit
gradient solver, then optimize the $\boldsymbol{\theta}$ such that the
counterfactual modification is within the human annotations boundary
$\boldsymbol{r}$.

Formally speaking, for a prediction model $f_{\boldsymbol{\theta}}$, we define
the CF-Align loss as:

$\mathcal{L}_{align}(\boldsymbol{\theta}):=\frac{1}{n}\sum_{i=1}^{n}\left\|(1-\boldsymbol{r}_{i})\odot(\boldsymbol{x}^{*}(\boldsymbol{\theta})-\boldsymbol{x})\right\|_{1},$
(2)

where $\boldsymbol{x}^{*}(\boldsymbol{\theta})$ obtained from Eq. 1 is a
function of the model parameter $\boldsymbol{\theta}$, $\boldsymbol{r}_{i}$ is
a binary mask that represents the region of interest, and $\odot$ denotes the
element-wise Hadamard product between matrices.

By combining with this loss to the cross-entropy loss
$\mathcal{L}_{cls}(\boldsymbol{\theta})$, the overall training objective for
the prediction model $f_{\boldsymbol{\theta}}$ is:

$\mathcal{L}(\boldsymbol{\theta}):=\mathcal{L}_{cls}(\boldsymbol{\theta})+\alpha\mathcal{L}_{align}(\boldsymbol{\theta}),$
(3)

where the hyper-parameter $\alpha$ is the trade-off between classification
accuracy and alignment to experts’ annotation boundary. A higher value of
$\alpha$ could potentially lead to improved alignment, but it might also
compromise prediction accuracy because the model could leverage other features
that may not be understandable to domain knowledge for making predictions.
Specifically, it was well established that deep learning methods mainly
focused on textual features Geirhos et al. (2018). Besides, existing works in
medical imaging have also shown that microscopic features (_e.g._ , contour,
curvature) Wang et al. (2021) or contextual features Liu et al. (2021a) can
improve the prediction power. Moreover, apart from the lack of explainability,
such features may fall outside the realm of causal relationships grasped by
domain experts and may pertain to spurious correlations. These correlations
could potentially lead to non-robustness when faced with perturbations or out-
of-distribution scenarios.

Optimization. To optimize Eq. (3), we need to compute the gradient
$\nabla_{\boldsymbol{\theta}}\mathcal{L}_{a}(\boldsymbol{\theta})$, which
involves the term
$\nabla_{\boldsymbol{\theta}}\boldsymbol{x}^{*}(\boldsymbol{\theta})$. The
main challenge lies in that the functional form of
$\boldsymbol{x}^{*}(\boldsymbol{\theta})$ is not explicit, making it hard to
derive the gradient using chain rules.

To compute this implicit gradient, we employ the Implicit Function Theorem
(IFT). The idea is to note the fact that
$\nabla_{\boldsymbol{x}}\arg\min_{\boldsymbol{x}}g(\boldsymbol{x},\boldsymbol{\theta})\equiv\boldsymbol{0}$
for any differential function $g$, which allows us to write
$\nabla_{\boldsymbol{\theta}}\boldsymbol{x}^{*}(\boldsymbol{\theta})$ as the
product of the inverse Hessian $H_{g}[\boldsymbol{x}]^{-1}$ and the mixed
derivative $\nabla_{\boldsymbol{\theta}}(\nabla_{\boldsymbol{x}}g)$. Formally,
we have:

######  Theorem 4.1 (Inverse Function Theorem (IFT)).

Consider two vectors
$\boldsymbol{x}\in\mathbb{R}^{p},\boldsymbol{\theta}\in\mathbb{R}^{d}$, and a
function
$g(\boldsymbol{x},\boldsymbol{\theta}):\mathbb{R}^{p}\times\mathbb{R}^{d}\mapsto\mathbb{R}$.
Let
$\boldsymbol{x}^{*}(\boldsymbol{\theta}):=\arg\min_{\boldsymbol{x}}g(\boldsymbol{x},\boldsymbol{\theta})$.
In addition, suppose that the following conditions hold: i) $g$ is
differentiable, ii) the $\mathrm{argmin}$ is unique for each
$\boldsymbol{\theta}$, and iii) the Hessian matrix $H_{g}[\boldsymbol{x}]$ is
invertible. Then, we have:

$\nabla_{\boldsymbol{\theta}}\boldsymbol{x}^{*}(\boldsymbol{\theta})=-H_{g}\left[\boldsymbol{x}\right]^{-1}\nabla_{\boldsymbol{\theta}}(\nabla_{\boldsymbol{x}}g)|_{\boldsymbol{x}^{*}(\boldsymbol{\theta}),\boldsymbol{\theta}}.$

######  Remark 4.2.

For modern neural networks, the inverse Hessian is generally intractable to
compute. For this case, to acquire
$\nabla_{\boldsymbol{\theta}}\boldsymbol{x}^{*}(\boldsymbol{\theta})$, we can
use linear system solvers such as the conjugate gradient solver Hestenes et
al. (1952) to solve the linear equation
$H_{g}\left[\boldsymbol{x}\right]\nabla_{\boldsymbol{\theta}}\boldsymbol{x}^{*}(\boldsymbol{\theta})+\nabla_{\boldsymbol{\theta}}(\nabla_{\boldsymbol{x}}g)|_{\boldsymbol{x}^{*}(\boldsymbol{\theta}),\boldsymbol{\theta}}=\boldsymbol{0}$.

By setting
$g(\boldsymbol{x},\boldsymbol{\theta}):=\mathrm{CE}(f_{\boldsymbol{\theta}}(\boldsymbol{x}),y^{*})$,
we can compute the implicit gradient
$\nabla_{\boldsymbol{\theta}}\boldsymbol{x}^{*}(\boldsymbol{\theta})$ and
thereby the gradient $\nabla_{\boldsymbol{\theta}}\mathcal{L}_{a}$ with the
chain rule. Besides, it is important to note that our proposed CF-align loss
is model-agnostic. Therefore, it can be easily plugged into the training of
various neural networks for human-centered learning.

Extension to attributes $\boldsymbol{a}$. In addition to the class label $y$,
Eq. 2 also applies when the annotated regions $\boldsymbol{r}$ explains its
decision to attributes annotations $\boldsymbol{a}$, which will subsequently
affect the label $y$. For instance, in the context of lung cancer diagnosis,
$\boldsymbol{a}$ pertains to clinical attributes related to nodules, which
include factors such as size, margin, spiculation, and more. During the
diagnostic process, clinicians commonly annotate the nodule’s region and
provide assessments of these attributes, which serve as the basis for their
diagnostic decisions.

If these attributes are provided during training, then we can leverage them
into our framework. In this case, the overall classifier can be written as
$g_{\boldsymbol{\gamma}}\circ f_{\theta}$, where
$f_{\boldsymbol{\theta}}:\mathcal{X}\mapsto\mathcal{A}$ and
$g_{\boldsymbol{\gamma}}:\mathcal{A}\mapsto\mathcal{Y}$. To determine which
attributes to modify, we first employ Zhao et al. (2023) to identify key
attributes $\boldsymbol{\bar{a}}$ that have causal influences on $y$, and then
generate the counterfactual image by:

$\boldsymbol{x}^{*}(\theta):=\arg\min_{\boldsymbol{x}^{\prime}}\mathrm{CE}(f_{\boldsymbol{\theta}}(\boldsymbol{x}^{\prime}),\boldsymbol{\bar{a}}^{*})+\lambda
d(\boldsymbol{x},\boldsymbol{x}^{\prime}).$ (4)

To train $(\boldsymbol{\theta},\boldsymbol{\gamma})$, we can optimize the
objective function Eq. 3 such that the CE loss is replaced with
$\mathcal{L}_{cls}(\boldsymbol{\theta},\boldsymbol{\gamma})$ to account for
the classifier $g_{\boldsymbol{\gamma}}$ and that the
$\boldsymbol{x}^{*}(\boldsymbol{\theta})$ is generated through Eq. 4. For a
new sample $\boldsymbol{x}$ to predict, we first use
$f_{\boldsymbol{\theta(x)}}$ to predict attributes $\boldsymbol{a}$ based on
explainable features, then predict $y$ based on attributes $\boldsymbol{a}$.

##  5 Experiment

To demonstrate the practicability of enhancing the explainability within our
method, we apply our model to the pulmonary nodule benign/malignant
classification. It’s important to clarify that our objective is not to attain
state-of-the-art performance on this particular task but rather to leverage it
as an illustrative example to emphasize our key point.

###  5.1 Dataset & implementation

We consider the LIDC-IDRI dataset Armato III et al. (2011), which contains
imaging data obtained from clinical thoracic CT scans, as well as fine-grained
annotations (nodule bounding boxes, malignancy scores, and attributes
information) provided by experienced physicians. Prior to analysis, we
preprocess the images by resampling the pixel space to a resolution of
$1\times 1\times 1\mathrm{mm}^{3}$ and normalizing the intensity based on a
window center of $\mathrm{HU}=-600$ and a window width of $\mathrm{HU}=1600$.
In order to ascertain whether our method can learn the same features as those
used by physicians, we opt to take the entire 2D slice as input, without
explicitly cropping out the nodules. Regarding the classification labels, each
sample was scored by physicians, with $3$ or lower meaning benign ($y=0$),
while those with a score of $4$ or higher are considered malignant ($y=1$).
Besides malignancy labels, each sample is accompanied by six clinical
attributes: subtlety, calcification, margin, spiculation, lobulation, and
texture. A comprehensive description of these attributes and their influence
on the disease label can be found in Appx. B.1. In addition, physicians also
provide annotations for the nodule bounding box for each sample, which serves
as the foundation for annotating attributes and determining the disease label.
We divide the dataset into three subsets: training ($n=731$ nodules),
validation ($n=244$ nodules), and test ($n=238$ nodules). To avoid possible
label leaking, this partition is based on individual patients.

Figure 3: Comparison with baselines on the LIDC-IDRI dataset. We use (a)
saliency map precision to evaluate model explainability and (b) accuracy to
evaluate classification performance.

We implement a seven-layer convolutional neural network (Fig. 7) to
parameterize $f_{\boldsymbol{\theta}}$. We use the Adam optimizer to train our
model, with the learning rate set to $0.001$, batch size set to $128$, and
epochs set to $300$. During the training, we iteratively optimize over the
classification loss $\mathcal{L}_{cls}$ and the alignment loss
$\mathcal{L}_{align}$. To generate the counterfactual image, we adopt the
LatentCF method Balasubramanian et al. (2020), which first encodes the image
into the latent space and then modifies the latent code for generation. To
estimate the implicit gradient
$\nabla_{{\boldsymbol{\theta}}}\mathcal{L}_{align}$, we use the conjugate
gradient method implemented in the $\mathrm{TorchOpt}$ package Ren et al.
(2022). We use Grad-Cam Selvaraju et al. (2017) to compute the saliency map
for each method. For implementation of compared baselines, we directly load
their published codes. To remove the effect of randomness, we repeat all
experiments using three different seeds.

###  5.2 Comparison with baselines

We evaluate the classification accuracy and explainability of our method and
baselines.

Compared baselines. Firstly, we compare with several ante-hoc explainable AI
baselines, namely i) BagNets Brendel & Bethge (2019) that integrated the
white-box bag-of-features model with a deep neural network to achieve both
explainability and performance; ii) CDEP Rieger et al. (2020) that required
the model to produce a prediction as well as an explanation (_i.e._ , multi-
tasks learning); iii) ICNN Zhang et al. (2018) that modified the architecture
of the neural network to achieve object-centered explainability; iv) RRR Ross
et al. (2017) and v) CIDG Chang et al. (2021) that constrains the gradient of
the input image to be aligned with annotations from radiologists.

Besides, to achieve a comprehensive comparison, we also include the vi)
Vanilla method that optimizes only the classification loss and the vii) Oracle
method that takes only features annotated by the physicians (_i.e._ , the
cropped-out nodule regions) as input.

Metrics. To evaluate model explainability, we employ the saliency map
precision, which is defined as the ratio of the saliency map area within the
nodule bounding box to the total area of the saliency map. To evaluate the
classification performance, we report the accuracy metric.

Comparison over explainability. We report the saliency map precision of our
method and baselines in Fig. 3 (a). Firstly, we can observe that our method
reaches a high saliency precision of $81\%$, which means that the decision
basis of our model is well aligned with that of the physicians. Such a
promising result can be attributed to the fact that our counterfactual
generation method can accurately identify the causes for model decision, and
that Thm. A.1 provides an efficient way to estimate the implicit gradient and
makes the alignment loss easy to optimize.

Figure 4: Visualization of the counterfactual images of six examples. For each
example, the left column ($x_{0}$), the middle column ($x^{*}$), and the right
column ($x^{*}-x_{0}$) respectively represent the original image, the
counterfactual image, and the modified regions. Regions with positive
modifications are highlighted in red, while those with negative modifications
are marked in blue. The corresponding attribute changes are noted in the
title.

Furthermore, we have also noticed that other baseline methods struggle to
discern features within nodule bounding boxes when making predictions. This
implies that their diagnostic decisions lack explainability for physicians. To
comprehend such a result, one should note that BagNets Brendel & Bethge
(2019), ICNN Zhang et al. (2018), and CDEP Rieger et al. (2020) proposed no
explicit constraint to learn explainable features. Therefore, for input with
complicated backgrounds such as the thoracic CT, these methods can be easily
biased by the contextual features in the image. Meanwhile, though methods such
as RRR Ross et al. (2017) and CIDG Chang et al. (2021) explicitly constrained
the gradient of the model to human decision areas, the gradient method itself
has been found unable to fully represent model decision boundary Jain &
Wallace (2019); Grimsley et al. (2020), making their identified areas failed
to align with ground-truth annotations.

Comparison over classification. We report the classification accuracy in Fig.
3 (b). As shown, our method significantly outperforms the baselines. This
outcome is a product of our method’s ability to harness expert-explainable
features that can be consistently applied to test data. In contrast, the
baselines may utilize other contextual features for decisions, which are
mainly pseudo-correlation and are hard to transfer.

Besides, it is also worth noting that in our results, even the oracle method
can only achieve a classification accuracy of $72\%$. In contrast, some work
Shen et al. (2017); Xie et al. (2018); Wu et al. (2018) on pulmonary nodule
classification claimed a classification accuracy over $99\%$. This gap can be
mainly explained by the Shen et al. (2017); Xie et al. (2018); Wu et al.
(2018) elimination of uncertain or challenging samples (those with a score of
3 in the data) from the test dataset in Shen et al. (2017); Xie et al. (2018);
Wu et al. (2018). Once again, we would like to emphasize that our experiment
is not primarily aimed at surpassing existing state-of-the-art classification
methods. Instead, our primary goal is to showcase the model’s explainability
and its potential benefits for classification tasks.

Visualization of counterfactual images. The counterfactual images generated
are presented in Fig. 4. For each nodule, the left, middle, and right columns
correspond to the original image, the counterfactual image, and the modified
regions, respectively. As we can see, the counterfactual modifications
generally correspond to one or more attributional features. For instance, the
two nodules in the first row were transformed from benign to malignant by
introducing spiculation around the nodule. Similarly, the two benign nodules
in the 2nd row are classified to be malignant after modifying the original
sharp margin to poorly defined ones111Please see
https://pylidc.github.io/annotation.html. These results demonstrate that our
counterfactual generation process can effectively identify fine-grained visual
features, thereby enabling the diagnosis decision of our model to align more
closely with those made by humans.

###  5.3 Visualization of saliency map

To further evaluate the explainability of our method and the ability of
counterfactual generation to localize fine-grained nodule features, we
visualize the saliency maps (Fig. 5) and the generated counterfactual images
(Fig. 4).

Figure 5: Saliency map visualization of our method and compared baselines.
Nodules are marked by red bounding boxes. More examples can be found in Fig.
8.

Visualization of saliency maps. The saliency maps of our method and compared
baselines are shown in Fig. 5. As we can see, our method can accurately
identify nodule-related features, while baseline methods mostly focus on areas
irrelevant to the nodule, such as the rib, pleural, and spine. These results
verify that our method can effectively constrain the neural network to learn
human-centered features, making our diagnosis process intrinsically
explainable.

###  5.4 Radiology report generation

To present our explanations to human end-users in a clear and accessible
manner, we use a Large Language Model (LLM)-based interface to generate a
structured report. Specifically, we employ GPT-4 as a backend through
prompting Brown et al. (2020). The prompt we provide to the language model
contains three parts, namely a task description, the predictions, and an
explanation that is composed of the disease label, attributes, and saliency
map from our classifier.

To ensure the report is well-organized and maintains a professional
appearance, we present GPT-4 with various standard examples provided by
radiologists from Irvin et al. (2019) and instruct it to generate a report
that adheres to the format of these examples. Specifically, the prompt we
provide to GPT-4 is formated as follows:

  1. 1.

Task description: You are a machine that explains the decision of a deep
learning model to human end-users.

  2. 2.

Predictions and explanations: [size][attributes] lesion found in [position].
(if malignant) These features may suggest a malignant etiology. (if benign)
These features suggest the nodule is stable and highly likely benign. [follow-
up schedule suggestions].

  3. 3.

Canonical examples: [reports by physicians]222Please refer to Appx. A.3 for
the examples we use..

To illustrate the fidelity of the generated reports, we provide two examples
in Fig. 6. In the 1st example, the nodule exhibits malignant features such as
poorly defined margin, spiculation, and lobulation, while in the 2nd example,
the nodule displays benign features such as solid texture and sharp margin.
Our evaluation focuses on the accuracy and comprehensiveness of the generated
reports. As shown, the findings section of the report clearly states the
position and attributes of the nodules. Based on such findings, the
impressions section provides a correct diagnosis and a detailed explanation of
the underlying reasons for such a diagnosis. Further, physicians offer
suggestions on follow-up examination schedules based on the disease severity.
The generated reports summarize these explanations, which can effectively
communicate the diagnosis results to human readers.

Figure 6: Radiology reports generated by GPT-4 that summarize our explanations
composed of the saliency map (attached to the original image in the left
corner, findings regarding attributes, and the final prediction and
suggestions.

##  6 Conclusions

In this paper, we present a counterfactual-based framework aimed at aligning
the model with human-explainable features for prediction. To this end, a
counterfactual alignment loss is introduced to ensure that the model only
modifies regions within human annotations during counterfactual generation. To
optimize this loss, we leverage the implicit function theorem to compute the
gradient of the alignment loss, which involves implicit forms in the
counterfactual generation process. Our framework’s effectiveness is
demonstrated by its ability to guide the prediction model in leveraging
nodule-related features for the diagnosis of pulmonary nodule malignancy.

## References

  * Ali et al. (2023) Sajid Ali, Tamer Abuhmed, Shaker El-Sappagh, Khan Muhammad, Jose M Alonso-Moral, Roberto Confalonieri, Riccardo Guidotti, Javier Del Ser, Natalia Díaz-Rodríguez, and Francisco Herrera.  Explainable artificial intelligence (xai): What we know and what is left to attain trustworthy artificial intelligence.  _Information Fusion_ , 99:101805, 2023.
  * Armato III et al. (2011) Samuel G Armato III, Geoffrey McLennan, Luc Bidaut, Michael F McNitt-Gray, Charles R Meyer, Anthony P Reeves, Binsheng Zhao, Denise R Aberle, Claudia I Henschke, Eric A Hoffman, et al.  The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans.  _Medical physics_ , 38(2):915–931, 2011.
  * Arrieta et al. (2020) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  _Information fusion_ , 58:82–115, 2020.
  * Balasubramanian et al. (2020) Rachana Balasubramanian, Samuel Sharpe, Brian Barr, Jason Wittenbach, and C Bayan Bruss.  Latent-cf: a simple baseline for reverse counterfactual explanations.  _arXiv preprint arXiv:2012.09301_ , 2020.
  * Brendel & Bethge (2019) Wieland Brendel and Matthias Bethge.  Approximating cnns with bag-of-local-features models works surprisingly well on imagenet.  In _International Conference on Learning Representations_ , 2019.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chang et al. (2021) Chun-Hao Chang, George Alexandru Adam, and Anna Goldenberg.  Towards robust classification model by counterfactual and invariant data generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15212–15221, 2021.
  * Chen et al. (2019) Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su.  This looks like that: deep learning for interpretable image recognition.  _Advances in neural information processing systems_ , 32, 2019.
  * Fei (2022) Zhengcong Fei.  Attention-aligned transformer for image captioning.  In _proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pp. 607–615, 2022.
  * Freiesleben (2022) Timo Freiesleben.  The intriguing relation between counterfactual explanations and adversarial examples.  _Minds and Machines_ , 32(1):77–109, 2022.
  * Fu et al. (2014) XiaoLan Fu, LianHong Cai, Ye Liu, Jia Jia, WenFeng Chen, Zhang Yi, GuoZhen Zhao, YongJin Liu, and ChangXu Wu.  A computational cognition model of perception, memory, and judgment.  _Science China Information Sciences_ , 57:1–15, 2014.
  * Gao et al. (2022) Yuyang Gao, Tong Steven Sun, Liang Zhao, and Sungsoo Ray Hong.  Aligning eyes between humans and deep neural network through interactive attention alignment.  _Proceedings of the ACM on Human-Computer Interaction_ , 6(CSCW2):1–28, 2022.
  * Geirhos et al. (2018) Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  _arXiv preprint arXiv:1811.12231_ , 2018.
  * Gould et al. (2013) Michael K Gould, Jessica Donington, William R Lynch, Peter J Mazzone, David E Midthun, David P Naidich, and Renda Soylemez Wiener.  Evaluation of individuals with pulmonary nodules: When is it lung cancer?: Diagnosis and management of lung cancer: American college of chest physicians evidence-based clinical practice guidelines.  _Chest_ , 143(5):e93S–e120S, 2013.
  * Grimsley et al. (2020) Christopher Grimsley, Elijah Mayfield, and Julia Bursten.  Why attention is not explanation: Surgical intervention and causal reasoning about neural models.  2020\.
  * Hestenes et al. (1952) Magnus R Hestenes, Eduard Stiefel, et al.  Methods of conjugate gradients for solving linear systems.  _Journal of research of the National Bureau of Standards_ , 49(6):409–436, 1952.
  * Hind et al. (2019) Michael Hind, Dennis Wei, Murray Campbell, Noel CF Codella, Amit Dhurandhar, Aleksandra Mojsilović, Karthikeyan Natesan Ramamurthy, and Kush R Varshney.  Ted: Teaching ai to explain its decisions.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , pp. 123–129, 2019.
  * Irvin et al. (2019) Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pp. 590–597, 2019.
  * Ismail et al. (2021) Aya Abdelsalam Ismail, Hector Corrada Bravo, and Soheil Feizi.  Improving deep learning interpretability by saliency guided training.  _Advances in Neural Information Processing Systems_ , 34:26726–26739, 2021.
  * Jain & Wallace (2019) Sarthak Jain and Byron C Wallace.  Attention is not explanation.  _arXiv preprint arXiv:1902.10186_ , 2019.
  * Lang et al. (2021) Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William T Freeman, Phillip Isola, Amir Globerson, Michal Irani, et al.  Explaining in style: Training a gan to explain a classifier in stylespace.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 693–702, 2021.
  * Liu et al. (2021a) Mingzhou Liu, Fandong Zhang, Xinwei Sun, Yizhou Yu, and Yizhou Wang.  Ca-net: Leveraging contextual features for lung cancer prediction.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24_ , pp. 23–32. Springer, 2021a.
  * Liu et al. (2021b) Yuhang Liu, Fandong Zhang, Chaoqi Chen, Siwen Wang, Yizhou Wang, and Yizhou Yu.  Act like a radiologist: towards reliable multi-view correspondence reasoning for mammogram mass detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(10):5947–5961, 2021b.
  * Najafabadi et al. (2015) Maryam M Najafabadi, Flavio Villanustre, Taghi M Khoshgoftaar, Naeem Seliya, Randall Wald, and Edin Muharemagic.  Deep learning applications and challenges in big data analytics.  _Journal of big data_ , 2(1):1–21, 2015.
  * Nauta et al. (2021) Meike Nauta, Ron Van Bree, and Christin Seifert.  Neural prototype trees for interpretable fine-grained image recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 14933–14943, 2021.
  * Parafita & Vitrià (2019) Álvaro Parafita and Jordi Vitrià.  Explaining visual models by causal attribution.  In _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , pp. 4167–4175. IEEE, 2019.
  * Ren et al. (2022) Jie Ren, Xidong Feng, Bo Liu, Xuehai Pan, Yao Fu, Luo Mai, and Yaodong Yang.  Torchopt: An efficient library for differentiable optimization.  _arXiv preprint arXiv:2211.06934_ , 2022.
  * Rieger et al. (2020) Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu.  Interpretations are useful: penalizing explanations to align neural networks with prior knowledge.  In _International conference on machine learning_ , pp. 8116–8126. PMLR, 2020.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pp. 234–241. Springer, 2015.
  * Ross et al. (2017) Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez.  Right for the right reasons: Training differentiable models by constraining their explanations.  _arXiv preprint arXiv:1703.03717_ , 2017.
  * Selvaraju et al. (2017) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 618–626, 2017.
  * Shen et al. (2017) Wei Shen, Mu Zhou, Feng Yang, Dongdong Yu, Di Dong, Caiyun Yang, Yali Zang, and Jie Tian.  Multi-crop convolutional neural networks for lung nodule malignancy suspiciousness classification.  _Pattern Recognition_ , 61:663–673, 2017.
  * Speith (2022) Timo Speith.  A review of taxonomies of explainable artificial intelligence (xai) methods.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , pp. 2239–2250, 2022.
  * Sreedevi et al. (2022) AG Sreedevi, T Nitya Harshitha, Vijayan Sugumaran, and P Shankar.  Application of cognitive computing in healthcare, cybersecurity, big data and iot: A literature review.  _Information Processing & Management_, 59(2):102888, 2022.
  * Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ , 2013.
  * Verma et al. (2020) Sahil Verma, Varich Boonsanong, Minh Hoang, Keegan E Hines, John P Dickerson, and Chirag Shah.  Counterfactual explanations and algorithmic recourses for machine learning: A review.  _arXiv preprint arXiv:2010.10596_ , 2020.
  * Wachter et al. (2017) Sandra Wachter, Brent Mittelstadt, and Chris Russell.  Counterfactual explanations without opening the black box: Automated decisions and the gdpr.  _Harv. JL & Tech._, 31:841, 2017.
  * Wang et al. (2021) Churan Wang, Xinwei Sun, Fandong Zhang, Yizhou Yu, and Yizhou Wang.  Dae-gcn: Identifying disease-related features for disease prediction.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24_ , pp. 43–52. Springer, 2021.
  * Wang et al. (2022) Dongsheng Wang, Yi Xu, Miaoge Li, Zhibin Duan, Chaojie Wang, Bo Chen, Mingyuan Zhou, et al.  Knowledge-aware bayesian deep topic model.  _Advances in Neural Information Processing Systems_ , 35:14331–14344, 2022.
  * Wang & Lin (2021) Tong Wang and Qihang Lin.  Hybrid predictive models: When an interpretable model collaborates with a black-box model.  _The Journal of Machine Learning Research_ , 22(1):6085–6122, 2021.
  * Wu et al. (2018) Botong Wu, Zhen Zhou, Jianwei Wang, and Yizhou Wang.  Joint learning for pulmonary nodule segmentation, attributes and malignancy prediction.  In _2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)_ , pp. 1109–1113. IEEE, 2018.
  * Xie et al. (2018) Yutong Xie, Yong Xia, Jianpeng Zhang, Yang Song, Dagan Feng, Michael Fulham, and Weidong Cai.  Knowledge-based collaborative deep learning for benign-malignant lung nodule classification on chest ct.  _IEEE transactions on medical imaging_ , 38(4):991–1004, 2018.
  * Zhang et al. (2018) Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu.  Interpretable convolutional neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 8827–8836, 2018.
  * Zhao et al. (2023) Ruiqi Zhao, Lei Zhang, Shengyu Zhu, Zitong Lu, Zhenhua Dong, Chaoliang Zhang, Jun Xu, Zhi Geng, and Yangbo He.  Conditional counterfactual causal effect for individual attribution.  In _Uncertainty in Artificial Intelligence_ , pp. 2519–2528. PMLR, 2023.
  * Zheng et al. (2017) Nan-ning Zheng, Zi-yi Liu, Peng-ju Ren, Yong-qiang Ma, Shi-tao Chen, Si-yu Yu, Jian-ru Xue, Ba-dong Chen, and Fei-yue Wang.  Hybrid-augmented intelligence: collaboration and cognition.  _Frontiers of Information Technology & Electronic Engineering_, 18(2):153–179, 2017.

Appendix

##  Appendix A Method

###  A.1 Preliminary on matrix derivative

For $y=f(\boldsymbol{x})$, where $f:\mathbb{R}^{p}\mapsto\mathbb{R}$ and
$x\in\mathbb{R}^{p}$, the gradient vector is defined as:

$\nabla_{x}f:=\left[\frac{\partial f}{\partial x_{1}},\cdots,\frac{\partial
f}{\partial x_{p}}\right]^{T}.$

For $\boldsymbol{x}=g(\boldsymbol{\theta})$, where
$g:\mathbb{R}^{m}\mapsto\mathbb{R}^{p}$,
$\boldsymbol{\theta}\in\mathbb{R}^{m}$, and $\boldsymbol{x}\in\mathbb{R}^{p}$,
the Jacobian matrix is defined as:

$\nabla_{\boldsymbol{\theta}}\boldsymbol{x}:=\begin{bmatrix}\frac{\partial
x_{1}}{\partial\theta_{1}}&\cdots&\frac{\partial x_{1}}{\partial\theta_{m}}\\\
\vdots&\ddots&\vdots\\\ \frac{\partial
x_{p}}{\partial\theta_{1}}&\cdots&\frac{\partial
x_{p}}{\partial\theta_{m}}\end{bmatrix}_{p\times m}.$

For $y=f(\boldsymbol{x})$, the Hessian matrix is defined as the Jacobian of
the gradient vector. That is:

$H_{f}[\boldsymbol{x}]:=\nabla_{\boldsymbol{x}}(\nabla_{\boldsymbol{x}f})=\begin{bmatrix}\frac{\partial^{2}f}{\partial
x_{1}^{2}}&\cdots&\frac{\partial^{2}f}{\partial x_{1}\partial x_{p}}\\\
\vdots&\ddots&\vdots\\\ \frac{\partial^{2}f}{\partial x_{p}\partial
x_{1}}&\cdots&\frac{\partial^{2}f}{\partial x_{p}^{2}}\end{bmatrix}_{p\times
p}$

###  A.2 Proof of the Inverse Function Theorem (IFT)

######  Theorem A.1 (IFT).

Consider two vectors
$\boldsymbol{x}\in\mathbb{R}^{p},\boldsymbol{\theta}\in\mathbb{R}^{m}$, and a
function
$f(\boldsymbol{x},\boldsymbol{\theta}):\mathbb{R}^{p}\times\mathbb{R}^{m}\mapsto\mathbb{R}$.
Let
$\boldsymbol{x}^{*}(\boldsymbol{\theta}):=\arg\min_{\boldsymbol{x}}f(\boldsymbol{x},\boldsymbol{\theta})$.
Suppose that the following regularities hold:

  1. 1.

$f$ is continuous and differentiable,

  2. 2.

the $\mathrm{argmin}$ is unique for each $\boldsymbol{\theta}$,

  3. 3.

the Hessian matrix $H_{f}[x]$ is invertible.

Then, we have:

$\nabla_{\boldsymbol{\theta}}\boldsymbol{x}^{*}(\boldsymbol{\theta})=-H_{f}\left[\boldsymbol{x}\right]^{-1}\cdot\nabla_{\boldsymbol{\theta}}(\nabla_{\boldsymbol{x}}f)|_{\boldsymbol{x}^{*}(\boldsymbol{\theta}),\boldsymbol{\theta}}.$

###### Proof.

Since the $\boldsymbol{x}^{*}(\boldsymbol{\theta})$ is defined by
$\arg\min_{\boldsymbol{x}}f(\boldsymbol{x},\boldsymbol{\theta})$, we have:

$\nabla_{\boldsymbol{x}}f|_{\boldsymbol{x}^{*}(\boldsymbol{\theta}),\boldsymbol{\theta}}=[\boldsymbol{0}]_{p\times
1}.$

Then, we have:

$\nabla_{\boldsymbol{\theta}}(\nabla_{\boldsymbol{x}}f|_{\boldsymbol{x}^{*}(\boldsymbol{\theta}),\boldsymbol{\theta}})=\nabla_{\boldsymbol{\theta}}([\boldsymbol{0}]_{p\times
1})=[\boldsymbol{0}]_{p\times m}.$

Now, applying the law of total derivation, we have:

$\nabla_{\boldsymbol{\theta}}(\nabla_{\boldsymbol{x}}f|_{\boldsymbol{x}^{*}(\boldsymbol{\theta}),\boldsymbol{\theta}})=\left\\{\nabla_{\boldsymbol{x}}(\nabla_{\boldsymbol{x}}f)\cdot\nabla_{\boldsymbol{\theta}}\boldsymbol{x}^{*}+\nabla_{\boldsymbol{\theta}}(\nabla_{\boldsymbol{x}}f)\right\\}|_{\boldsymbol{x}^{*}(\boldsymbol{\theta}),\boldsymbol{\theta}}$

Denote
$H_{f}[\boldsymbol{x}]:=\nabla_{\boldsymbol{x}}(\nabla_{\boldsymbol{x}}f)$ as
the Hessian matrix. The above two equations together mean:

$\left\\{H_{f}[\boldsymbol{x}]\cdot\nabla_{\boldsymbol{\theta}}\boldsymbol{x}^{*}(\boldsymbol{\theta})+\nabla_{\boldsymbol{\theta}}(\nabla_{\boldsymbol{x}}f)\right\\}|_{\boldsymbol{x}^{*}(\boldsymbol{\theta}),\boldsymbol{\theta}}=[\boldsymbol{0}]_{p\times
m},$

which shows the statement in the theorem. ∎

###  A.3 Prompt examples

##  Appendix B Experiment

###  B.1 Description of nodule attributes

Table 1: Attributes of pulmonary nodules

Name | Description | How to convert to a binary label
---|---|---
Subtlety | Size of the nodule | Subtle: score $\leq 4$; Obvious: score $=5$
Calcification | Whether calcification is present | Absent: score $=6$; Present: score $\leq 5$
Margin | Whether the margin is well-defined | Sharp: score $=5$; Pooly-defined: score $\leq 4$
Spiculation | Whether spiculation is present | Absent: score $=1$; Present: score $\geq 2$
Lobulation | Whether lobulation is present | Absent: score $=1$; Present: score $\geq 2$
Texture | Radiographic solidity | Solid: score $=5$; GGO/Mixed: score $\leq 4$

###  B.2 Network architecture

Figure 7: Architecture of the convolutional neural network used in the
experiment.

###  B.3 Extra visualization results

Figure 8: Visualization of the saliency maps of different methods. Each row
indicates an instance.