# What to tell when? – Information Provision as a Game

Astrid Rakow Mehrnoush Hajnorouzi Inst. of Systems Engineering for Future
Mobility,
German Aerospace Center (DLR) e.V<EMAIL_ADDRESS><EMAIL_ADDRESS>Dept. of Computing Science,
Carl von Ossietzky University Akhila Bairy Dept. of Computing Science,
Carl von Ossietzky University<EMAIL_ADDRESS>
###### Abstract

_Constantly informing systems_ (CIS), which provide us with information over a
long period of time, face a particular challenge in providing useful
information: Not only does a person’s information base change but also their
mood or abilities may change. An information provision strategy should hence
take these aspects into account. In this paper, we describe our vision for
supporting the design of CIS. We envisage using psychological models of the
human mind and emotions in a game of information provision. The analysis of
these games will give comparative insights into design variants.

##  1 Introduction

Complex technical systems are deeply integrated into our daily lives. They
provide us with various services ranging from entertainment to safety-critical
services. We use many of these systems over long periods of time and they even
adapt to our needs. Such systems and their users exchange information: On one
hand, the system acquires user information to adapt. On the other hand, users
often need information from the system e.g. some explanation of the system’s
behaviour. When such an interaction between the system and its user extends
over a longer period of time, an important question therefore arises: what
content should be communicated to the user and when? A few brief examples will
illustrate the challenges of this question:  (1) An autonomous vehicle AV
initiates emergency braking due to a deer suddenly crossing the road.  (2) The
AV slows down on the way to work due to a dumpy road.  (3) The AV slows down
due to a carnival procession, the user is accompanied by his excited kids.  In
these scenarios, the AV faces the challenge of determining what content helps
the user in the current context. It should factor in that the time span is
very short in (1), that the driver has prior knowledge in (2), and the right
level of detail is especially important, if the driver is occupied otherwise
as in (3).

In this paper, we argue that the design of a _constantly informing system
(CIS)_ 111Self-explaining systems are also CIS. A CIS is not necessarily self-
explaining.can profit from a holistic game theoretic analysis that places a
strong emphasis on the human aspect. We propose to analyze the challenge of
information provision (What content should be communicated to the user when?)
by methods of economics to determine which game is to be played (Sect. 2). In
the resulting game, we consider the human as a system of limited cognitive
resources (Sect. 4). In order to do so, we propose to use models of the human
mind and emotions from psychology, which we briefly survey in Sect. 3. We
outline how we envision integrating these models into games between a CIS and
its user (cf. Fig. 1). Their analysis results will allow designers comparing
variants of a CIS under development.

[capbesideposition=right,top,capbesidewidth=9cm]figure[]

Figure 1: CIS observes its environment and provides information to a human
model (HM). HM perceives this information and observes its surrounding Env to
choose its actions. The goal of CIS is to enable HM achieve its goals.

##  2 The Challenge of Information Provision Seen as a Game

Game theory, developed by von Neumann and Morgenstern [22] in the mid-20th
century, studies interactive decision-making where the outcome for each player
depends on the actions of the other players [21]. It has been used in various
domains such as economics, political science or computer science. In the
following, we discuss how the challenge of information provision can be
considered a game. By reinterpreting the economic analysis method PARTS [7]
for information provision, we outline the broader scale of this challenge and
then narrow the focus down to the next steps of our research.

PARTS defines a list of guiding questions to analyze a situation as a game:
the players (who is active? who has goals?), added value (what does player X
gain by Y’s participation?), rules (what rules do constrain the players? who
can change the rules?), tactics (how can the players’ perception of the game
be changed?) and scope (what are the boundaries of the game?). PARTS is only a
first step towards the definition of a game. The concrete game depends on the
characteristics of the application domain, the information available to build
a model of the application domain, and the goal of the analysis.

We want to analyze how the _“usefulness of a CIS ”_ can be optimized. _Useful_
information is relevant information222 Here, relevant information for the
CIS’s customer is the minimal information that they need to accomplish their
goal [27].  that the customer can _process and use_ to accomplish their goals,
i.e. the right content at the right time. It is part of our future work to
concretize this notion. Table 1 summarizes how we interpret the players’ roles
in an information provision setting. While the added values that the players
will bring to the game vary depending on the system considered, we can
identify two main types of rules as shown in Table 2. Tactics refer to actions
that a player can take to change how other players perceive the game, i.e. to
change their belief in who the players are, what their added values and what
the rules are. The scope of _our_ game is a CIS that is a subsystem of an AV
and serves a single human (the driver) who is captured via a human model HM.
For a start, we use a game of this very limited scope to investigate how to
formally include human capabilities in the design of a CIS.

Customers |  all recipients of information (humans or technical systems)
---|---
Suppliers |  suppliers of information (sensors and other CIS)
|  suppliers of computing services (hardware resources)
Competitors |  systems that render the CIS’s information redundant (other CIS, human perception)
|  processes that use the same computing services or human resources (e.g.
music system)
Complementors |  other CIS whose information enhances the CIS’ information (e.g. traffic warnings plus detour)
Table 1: Interpretation of player roles in the “information provision” game guiding rules |  a player is obliged to follow these rules, but can break them at will (e.g. ethical, legal rules)
---|---
constraint rules |  a player cannot break these rules; the rules can only be changed by a third party (resulting from the technical implementation)
Table 2: Types of rules in the “information provision” game

As the above exposé shows, information provision is an interdisciplinary
challenge where humans and technical systems interact – many CIS are embedded
into a human cyber-physical system of systems. Legal and ethical aspects, as
well as human capabilities, influence what a CIS should provide as useful
information. Whether information is useful strongly depends on whether the
information is relevant _and_ on whether humans can make use of it.

While the above was concerned with analyzing the application domain, we will
now discuss what game G in the “catalog” of well-studied games and respective
analysis methods is appropriate for our analysis goal. We will take a closer
look at the characteristics of the application domain and the goal of the
analysis – keeping in mind what aspects are most important and which aspects
might be omitted.

In a nutshell, the goal of G’s analysis is to determine which content our CIS
should provide to a HM in what context. This analysis is meant to be done at
design time in order to derive requirements on the information provision and
to compare different design alternatives. Our CIS controls what information
content it provides when to HM. We discuss what kinds of games are appropriate
using the following guiding questions based on the taxonomy of games as
developed in [20]: _Is CIS acting strategically? What is the relation between
HM and CIS? Who knows what?_

Is CIS acting strategically? We will study a CIS, that _chooses rationally_
the content that seems the most useful. We hence will develop a classical game
rather than an evolutionary game.

What is the relation between HM and CIS? Should we use cooperative or non-
cooperative game theory? In cooperative game theory (CGT), the players
cooperate with each other and form coalitions to obtain maximum reward/payoff.

In non-cooperative game theory (NCGT), the players make decisions
independently and each player aims to maximise their own payoff. We assume
that HM has a prioritized list of goals. CIS has only the goal to be helpful
within its constraints thus acting subordinately 333It is out of this paper’s
scope to discuss the case when goals of HM contradict the constraints of CIS..

In such a subordinate-dominant relationship, both CGT as well as NCGT could be
a fit. In order to apply NCGT we envision to model HM as always greedily
accepting all information that it can in its current _mode_ ; E.g. if HM is
tired, less information per time is accepted. We thereby encode a kind of
cooperation into HM’s modes, so that it is not playing against CIS by
maliciously rejecting all offered information. To derive these modes and the
dynamics of mode changes we plan to use psychological models of the human mind
and emotions. We will give a brief overview of these in Sect. 3. In Sect. 4 we
will give technical details on how we envision using such psychological models
to derive the HM for our game.

Who knows what? Both CIS and HM receive information from the environment. In
particular, HM receives information from our CIS within its car but also via
its five senses that are competing with CIS. CIS also does not exactly know
how HM reacts, as it has no precise means to derive the state and reactions of
HM. From CIS’s point of view HM’s reactions expose some degree of randomness.
HM is a system of bounded resources and will usually not (have the resources
to) think much about the beliefs of any support systems in its car. We hence
will define an epistemic game, where CIS has a belief on HM’s belief but HM
does not build beliefs on CIS beliefs.

CIS knows what the possible inputs of HM are, but assumes that these channels
are unreliable. That is, we assume that CIS can roughly predict what
information HM is able to receive via its senses. CIS moreover knows what
actions HM can choose (what a driver can do).

To us, the central challenge in the information provision game is capturing
the human factors. We envision to derive instantiations of HM from models of
the human mind and emotions. These instantiations are then “plugged” into our
game. We then check whether our CIS can inform HM, i.e. whether a strategy
exists, such that HM achieves its goals.

##  3 Models of Human Mind

For an automated system to provide useful information, it needs a human
behaviour model to determine the impact of the content and the optimal time
for providing the content. Well-crafted explanations can reassure users and
increase alertness [13]. The impact of percepts and its timing –or more
generally the dynamics of the human mind– is captured by cognitive
architectures. These are well-established frameworks developed in the field of
cognitive psychology [23, 3]. They simulate how knowledge is processed,
stored, and utilized in response to external stimuli, perception, and
knowledge [26]. There are many cognitive architectures with different foci,
like ICON FLUX[24], CASCaS [33], PRIM[29], but none of them captures the human
holistically.

In addition to modelling core cognitive functions (such as perception,
learning, memory, etc.), a few cognitive architectures can also capture the
interplay of emotions and behaviour. These architectures generally adopt two
primary approaches to model emotion-induced behaviours: altering architecture
structures (e.g. modifying memory modules), or modifying the processing
parameters (e.g. adjusting retrieval delay). Emotions act as conduits to human
behaviour, significantly impacting decisions and influencing the daily choices
of individuals [5]. A change in the person-environment relationship [16] may
trigger emotional shifts, which may change this relationship in future.

Over time, different theories of emotion have evolved to explain the
functioning of human emotions [25, 15, 12]. The cognitive appraisal theory is
one such theory. According to this theory, emotions are influenced by
individuals’ cognitive evaluations and interpretations of situations rather
than by the situations themselves. Many computational models are built on this
theory [18, 6, 17]. Implementation of the appraisal process within the context
of cognitive architectures is an ongoing effort. MAMID [11] is one example. It
processes incoming stimuli through various modules. The affect appraiser
module derives emotional states. The resulting affective state influences the
cognition processes by adjusting the rules related to goal selection, action
determination, module speed and capacity, and mental construct prioritization.

The appraisal process provides an assessment of the current situation,
influenced by various parameters, such as individual history, personality, and
current affective state. These parameters can be described by modelling the
effects of transient states (emotions) and personality traits
(characteristics).

To depict how the current situation affects human behavior, [34] incorporated
a situation assessor module into their proposed SAMPLE444Situation Awareness
Model for Pilot-in-the-Loop Evaluation pilot model architecture. This module
translates situation descriptions into types that trigger specific responses.
The module employs belief nets to hold required knowledge, with nodes denoting
particular features and events, and links illustrating causal and
correlational connections between them. Furthermore, weighted links denote
transition probabilities.

Appraisal theory is also utilised in the field of affective computing to
enhance systems’ abilities to perceive and generate human emotions. Affective
computing seeks to make computers more human-like by enabling them to
recognise, interpret, predict, and respond to human emotions. While few models
effectively integrate emotion generation and its impact on cognitive
processes, EmoCog [17] stands out. This work adeptly fuses diverse models of
computational emotions and cognitive architectures into a unified model. In
cognitive architectures, emotions can be divided into two parts: emotion
effects and emotion generation [17]. Emotion effects consider how emotions
might influence cognitive processing, while emotion generation explores how
cognitive processes contribute to the generation of emotion. EmoCog, developed
by Lin et al., also has primary and secondary appraisals similar to WASABI,
developed by Becker-Asano and Wachsmuth [6]. However, EmoCog’s secondary
appraisal is more versatile, explaining a broader range of emotions than
WASABI. The model includes an appraisal module that adjusts arousal and
valence values in short-term memory nodes.

A psychological model of a human can also be achieved by their attention
assessment. Wickens et al. developed SEEV 555Salience, Effort, Expectancy, and
Value model to quantify and predict a person’s attention level across various
areas of interest in a given situation [31]. Initially intended for pilots’
attention prediction, [10] and [32] later adapted SEEV to gauge driver
attention in road traffic. Leveraging SEEV, Bairy and Fränzle crafted a model
that predicts the optimal time to deliver explanations to drivers based on
their attention level [4].

##  4 Psychological Models in A Game

In a nutshell, we want to define a light-weight game as sketched in Fig. 1.
CIS plays against its environment env(CIS), which contains the human captured
via HM. As discussed in Sect. 2 we encode into HM how a human reacts so that
we do not need to consider the human as a separate player. CIS’s actions are
providing information to HM. The CIS model can reflect that information can
only be provided if CIS has the information in its information base and if it
has the required computing resources and time to retrieve it. Strategy
synthesis will then tell a designer whether CIS can provide information such
that HM achieves its goals 666This list of goals varies with the considered
scenario but we assume that the goal of being safe has highest priority while
the goal of feeling well has less priority.. A designer can plug in different
variants of CIS and compare how well they perform in terms of the goals that
can be achieved.

We aim for a HM model that encodes how a human reacts to the provided
information or other environmental perceptions. We assume that HM reacts
internally, i.e. emotionally, by consuming cognitive resources or by changing
its information base. This internal change might cause a mode change of HM. In
the new mode HM chooses a different strategy to reach its goals. We thus think
of HM as sketched in Fig. 2. We want to apply strategy synthesis to examine
the information provision game of CIS (cf. Fig. 1) that uses such a HM as part
of its environment model. There, a dominant strategy would be the best
strategy of CIS to provide the HM with the necessary information for HM to
achieves its goals.

[capbesideposition=right,top,capbesidewidth=8cm]figure[]

Figure 2: HM perceives its environment and chooses actions. During a scenario
HM changes its mode, reflecting a change in its cognitive resources, emotions,
and information.

A central idea of our approach is to consider HM as a resource-bounded system
that reacts mode-specifically to the provided information content. How HM
reacts will be derived from psychological models that are instantiated for the
considered scenario. We consider the central challenges of making these models
usable in our game to be: _attaining a formal representation of psychological
models_ , _reducing their complexity_ , _combining different psychological
theories_ , and _interpreting the results_.

As outlined in Sect. 3, there is a broad foundation of psychological theories
that are shedding light on different aspects of how humans react to provided
information. The level of precision of these theories varies heavily as well
as their level of formalization, ranging from informal over computational to
formal. As Reisenzein puts it “Theoretical fragmentation and a comparative
lack of precision are […] characteristic of psychology in general” [28,
p.248].

We focus on executable cognitive architectures as a starting point for our
work, as they have an appropriate level of precision. But to use such
architectures within a formal game, we need a formal representation of them.
The approach of Langenfeld et al. [14] for deriving timed automata for an
ACT-R instance promises a detailed study of the human reactions as captured in
ACT-R. Since our precursor work [4, 9] showed that the resulting models easily
get too complex for our needs, we want to derive simplified human models
$\textit{HM}_{i}$ from cognitive architectures by automata learning techniques
[2]. Automata learning techniques have been applied in many areas, such as
speech recognition, software development, and computational biology [30]. In
passive learning, the learning algorithm observes the inputs and the
corresponding outputs of the system that is to be learned. Active learning, on
the other hand, adopts an interactive approach where the learner also actively
asks strategic queries and observes the response. We want to use modes as a
means for guiding how the human models get simplified. Our idea is that a mode
roughly specifies how HM reacts, i.e. what kind of strategy HM will apply. We
think this might be done in a similar way to [8]. There Gosh and Verbrugge
captured the kinds of reasoning strategies of player in a marble drop game as
logical formulas, translated them into cognitive models and executed them in
the PRIMs architecture [29]. Instead of player types as in [8] we think of
player modes along the line “A tired driver reacts slowly. A driver gets tired
when they are bored over a period of time.” or “A driver that already has to
keep seven facts in mind, will not easily remember five more facts.”[19] We
then want to apply automata learning to the cognitive architecture to derive a
simplified model HM. There the choice of observable propositions will be an
important influence on whether meaningful modes can be derived. Since the
different architectures have different foci, we plan to use several
architectures.

Given we would have $n$ appropriately light-weight models $\textit{HM}_{i}$
–derived from different cognitive architectures or manually designed–, we
build $n$ games $\textit{G}_{i}$, $1\leq i\leq n$ (cf. Fig. 3).

Figure 3: From cognitive architectures to a comparison of CIS variants.

The performance scores of the same CIS $C_{j}$ in the different games
$\textit{G}_{i}$ –measured e.g. in terms of HM reaching its goals– will lead
to a $n$-tuple $t_{j}$ of scores for $C_{j}$. The comparison of two CIS
variations, $C_{j}$ with performance $t_{j}$ and $C_{k}$ with performance
$t_{k}$, will help the designer to get early on insight into how well the CIS
is tailored to human needs. But the comparison of $C_{j}$ and $C_{k}$ has to
be done with care, as the meaning of the $\textit{HM}_{i}$ models has to be
taken into account. This comparison gets especially difficult if the scores
relate to interdependent aspects. What if $\textit{HM}_{h}$ suffers from a
high workload while $\textit{HM}_{i}$ is in a good mood? Due to the coarse
granularity of models that we want to consider, we expect that such cases will
often occur, so that plausibility checks and if necessary refining analysis
has to be done. To this end, we think of spotlight approaches where the
learner gets more fine-grained observations where necessary. So, if this
approach is in use, a collection of appropriate $\textit{HM}_{i}$ models could
be evolved over time.

##  5 Conclusion

In this paper, we outlined our vision of how game theory can be used to
support the development of constantly informing systems (CIS), taking into
account the human factor. Our vision is to use models of the human mind and
emotions from psychology to derive coarse formal human models,
$\textit{HM}_{i}$ that can be used in a game between the CIS and the human
captured by $\textit{HM}_{i}$. The main challenges of our vision are attaining
a formal representation of psychological models, reducing their complexity,
combining different psychological theories, and interpreting the results.
Nevertheless, we hope to obtain at least comparative results that can guide
the early design of CIS in tailoring to the human needs.

##  Appendix A Acknowledgements

This research has been supported by the German Research Council (DFG) in the
PIRE Projects SD-SSCPS and ISCE-ACPS under grant no. DA 206/11-1, by RTG SEAS,
and by the German Federal Ministry of Education and Research (BMBF) within the
project ”ASIMOV-D” under grant agreement No. 01IS21022G, based on a decision
of the German Bundestag.

##  Appendix B Abbreviations

AV | Autonomous Vehicle
---|---
CIS | Constantly Informing System
CGT | Cooperative Game Theory
G | Game
HM | Human Model
NCGT | Non-Cooperative Game Theory

## References

  * [1]
  * [2] Bernhard K. Aichernig, Wojciech Mostowski, Mohammad Reza Mousavi, Martin Tappler & Masoumeh Taromirad (2018): _Model Learning and Model-Based Testing_ , pp. 74–100.  Springer International Publishing, Cham, 10.1007/978-3-319-96562-8_3.
  * [3] John R. Anderson & Christian Lebiere (1998): _The Atomic Components of Thought_.  Lawrence Erlbaum associates, 10.4324/9781315805696.
  * [4] Akhila Bairy & Martin Fränzle (2023): _Optimal Explanation Generation Using Attention Distribution Model_.  Human Interaction and Emerging Technologies (IHIET-AI 2023): Artificial Intelligence and Future Applications 70(70), 10.54941/ahfe1002928.
  * [5] Antoine Bechara, Hannah Damasio & Antonio Damasio (2000): _Emotion, Decision Making and the Orbitofrontal Cortex_.  Cerebral cortex (New York, N.Y. : 1991) 10, pp. 295–307, 10.1093/cercor/10.3.295.
  * [6] Christian Becker-Asano & Ipke Wachsmuth (2010): _Affective computing with primary and secondary emotions in a virtual human_.  Autonomous Agents and Multi-Agent Systems 20, pp. 32–49, 10.1007/s10458-009-9094-9.
  * [7] Adam M. Brandenburger & Barry J. Nalebuff (1995): _The Right Game: Use Game Theory to Shape Strategy. (Cover story)_.  Harvard Business Review 73(4), pp. 57 – 71.
  * [8] Sujata Ghosh & Rineke Verbrugge (2018): _Studying strategies and types of players: experiments, logics and cognitive models_.  Synthese 195(10), pp. 4265–4307, 10.1007/s11229-017-1338-7.
  * [9] Mehrnoush Hajnorouzi & Martin Fränzle: _Model-based Control for Human-Centered Systems_.  Unpublished.
  * [10] William J Horrey, Christopher D Wickens & Kyle P Consalus (2006): _Modeling Drivers’ Visual Attention Allocation While Interacting With In-Vehicle Technologies_.  Journal of experimental psychology. Applied 12(2), pp. 67–78, 10.1037/1076-898X.12.2.67.
  * [11] Eva Hudlicka (2002): _This Time with Feeling: Integrated Model of Trait and State Effects on Cognition and Behavior._ Applied Artificial Intelligence 16, pp. 1–31, 10.1080/08339510290030417.
  * [12] Carroll E Izard (1992): _Basic emotions, relations among emotions, and emotion-cognition relations._ 10.1037/0033-295x.99.3.561.
  * [13] Jeamin Koo, Dongjun Shin, Martin Steinert & Larry Leifer (2016): _Understanding driver responses to voice alerts of autonomous car operations_.  International Journal of Vehicle Design 70, p. 377, 10.1504/IJVD.2016.076740.
  * [14] Vincent Langenfeld, Bernd Westphal & Andreas Podelski (2019): _On Formal Verification of ACT-R Architectures and Models_.  In: CogSci, pp. 618–624.
  * [15] Richard S Lazarus (1982): _Thoughts on the relations between emotion and cognition._ American psychologist 37(9), p. 1019, 10.1037/0003-066X.37.9.1019.
  * [16] Richard S Lazarus (1991): _Emotion and adaptation_.  Oxford University Press.  Available at https://books.google.de/books?id=tTdIlwpxtWsC.
  * [17] Jerry Lin, Marc Spraragen, Jim Blythe & Michael Zyda (2011): _EmoCog: Computational Integration of Emotion and Cognitive Architecture_.  In: The Florida AI Research Society.  Available at http://aaai.org/ocs/index.php/FLAIRS/FLAIRS11/paper/view/2625.
  * [18] Stacy Marsella & Jonathan Gratch (2009): _EMA: A process model of appraisal dynamics_.  Cognitive Systems Research 10, pp. 70–90, 10.1016/j.cogsys.2008.03.005.
  * [19] George A Miller (1956): _The magical number seven, plus or minus two: Some limits on our capacity for processing information._ Psychological review 63(2), p. 81, 10.1037/h0043158.
  * [20] J.A. Moura & D.H. Hutchison (2019): _Game Theory for Multi-Access Edge Computing: Survey, Use Cases, and Future Trends_.  IEEE Communications Surveys and Tutorials 21(1), pp. 260–288, 10.1109/COMST.2018.2863030.
  * [21] Roger B. Myerson (1991): _Game Theory: Analysis of Conflict_.  Harvard University Press.  Available at http://www.jstor.org/stable/j.ctvjsf522.
  * [22] John von Neumann & Oskar Morgenstern (2007): _Theory of Games and Economic Behavior (60th-Anniversary Edition)_.  Princeton University Press, 10.1515/9781400829460.  Available at http://www.jstor.org/stable/j.ctt1r2gkx.
  * [23] Allen Newell (1990): _Unified Theories of Cognition_.  Harvard University Press, USA.
  * [24] Eric Nivel (2007): _Ikon flux 2.0_.  Available at https://api.semanticscholar.org/CorpusID:60599731.
  * [25] Keith Oatley & P. N. Johnson-laird (1987): _Towards a Cognitive Theory of Emotions_.  Cognition and Emotion 1(1), pp. 29–50, 10.1080/02699938708408362.
  * [26] David Peebles & Adrian Banks (2010): _Modelling Dynamic Decision Making with the ACT-R Cognitive Architecture_.  Journal of Artificial General Intelligence 2, pp. 52–68, 10.2478/v10229-011-0009-1.
  * [27] Astrid Rakow (2023): _Framing Relevance for Safety-Critical Autonomous Systems_ , 10.48550/arXiv.2307.14355.  arXiv:https://arxiv.org/abs/2307.14355.  Technical Report.
  * [28] Rainer Reisenzein, Eva Hudlicka, Mehdi Dastani, Jonathan Gratch, Koen Hindriks, Emiliano Lorini & John-Jules Meyer (2013): _Computational Modeling of Emotion: Toward Improving the Inter- and Intradisciplinary Exchange_.  IEEE Trans. Affect. Comput. 4(3), p. 246–266, 10.1109/T-AFFC.2013.14.
  * [29] Niels Taatgen (2013): _The Nature and Transfer of Cognitive Skills_.  Psychological review 120, pp. 439–471, 10.1037/a0033138.
  * [30] Frits Vaandrager, Bharat Garhewal, Jurriaan Rot & Thorsten Wißmann (2022): _A New Approach for Active Automata Learning Based on Apartness_.  In: Tools and Algorithms for the Construction and Analysis of Systems, Springer International Publishing, pp. 223–243, 10.48550/arXiv.2107.05419.
  * [31] Christopher Wickens, John Helleberg, Juliana Goh, Xidong Xu & William Horrey (2001): _Pilot Task Management: Testing an Attentional Expected Value Model of Visual Scanning_.  Savoy, IL, UIUC Institute of Aviation Technical Report.
  * [32] Bertram Wortelen (2014): _Das Adaptive-Information-Expectancy-Modell zur Aufmerksamkeitssimulation eines kognitiven Fahrermodells_.  Ph.D. thesis, Carl von Ossietzky Universität, Oldenburg, Germany.  Available at https://oops.uni-oldenburg.de/id/eprint/1970.
  * [33] Bertram Wortelen, Andreas Lüdtke & Martin Baumann (2013): _Integrated Simulation of Attention Distribution and Driving Behavior_.
  * [34] Greg L Zacharias, Adam X Miao, Christine Illgen, Jake M Yara & GM Siouris (1996): _SAMPLE: Situation awareness model for pilot in-the-loop evaluation_.  In: Proceedings of the 1st Annual Conference on Situation Awareness in the Tactical Air Environment, Citeseer.


# Segmentation of Non-Small Cell Lung Carcinomas: Introducing DRU-Net and
Multi-Lens Distortion

Soroush Oskouei  Norwegian University of Science and Technology (NTNU),
Department of Circulation and Medical Imaging, Trondheim, NO-7491, Norway
Levanger hospital, Nord-Trøndelag Health Trust, Clinic of Medicine, Levanger,
NO-7600, Norway<EMAIL_ADDRESS>Marit Valla  Norwegian University of
Science and Technology (NTNU), Department of Clinical and Molecular Medicine,
Trondheim, NO-7491, Norway  St. Olavs hospital, Trondheim University Hospital,
Clinic of Laboratory Medicine, Trondheim, NO-7030, Norway  St. Olavs hospital,
Trondheim University Hospital, Department of Pathology, Trondheim, NO-7030,
Norway  St. Olavs hospital, Trondheim University Hospital, Clinic of Surgery,
Trondheim, NO-7030, Norway  André Pedersen  Norwegian University of Science
and Technology (NTNU), Department of Clinical and Molecular Medicine,
Trondheim, NO-7491, Norway  St. Olavs hospital, Trondheim University Hospital,
Clinic of Surgery, Trondheim, NO-7030, Norway  Sopra Steria, Application
Solutions, Trondheim, NO-7010, Norway  Erik Smistad  Norwegian University of
Science and Technology (NTNU), Department of Circulation and Medical Imaging,
Trondheim, NO-7491, Norway  SINTEF Digital, Department of Health Research,
Trondheim, NO-7465, Norway  Vibeke Grotnes Dale  Norwegian University of
Science and Technology (NTNU), Department of Clinical and Molecular Medicine,
Trondheim, NO-7491, Norway  St. Olavs hospital, Trondheim University Hospital,
Department of Pathology, Trondheim, NO-7030, Norway  Maren Høibø  Norwegian
University of Science and Technology (NTNU), Department of Clinical and
Molecular Medicine, Trondheim, NO-7491, Norway  St. Olavs hospital, Trondheim
University Hospital, Clinic of Laboratory Medicine, Trondheim, NO-7030, Norway
Sissel Gyrid Freim Wahl  St. Olavs hospital, Trondheim University Hospital,
Department of Pathology, Trondheim, NO-7030, Norway  Mats Dehli Haugum  St.
Olavs hospital, Trondheim University Hospital, Department of Pathology,
Trondheim, NO-7030, Norway  Thomas Langø  SINTEF Digital, Department of Health
Research, Trondheim, NO-7465, Norway  St. Olavs hospital, Trondheim University
Hospital, Research Department, Center for Medical equipment, Technology, and
Innovation, Trondheim, NO-7491, Norway  Maria Paula Ramnefjell  University of
Bergen, Department of Clinical Medicine Centre for Cancer Biomarkers (CCBIO),
Bergen, NO-5007, Norway  Haukeland University Hospital, Department of
Pathology, Bergen, NO-5020, Norway  Lars Andreas Akslen  University of Bergen,
Department of Clinical Medicine Centre for Cancer Biomarkers (CCBIO), Bergen,
NO-5007, Norway  Haukeland University Hospital, Department of Pathology,
Bergen, NO-5020, Norway  Gabriel Kiss  St. Olavs hospital, Trondheim
University Hospital, Research Department, Center for Medical equipment,
Technology, and Innovation, Trondheim, NO-7491, Norway  Norwegian University
of Science and Technology (NTNU), Department of Computer Science, Trondheim,
NO-7491, Norway  Hanne Sorger  Norwegian University of Science and Technology
(NTNU), Department of Circulation and Medical Imaging, Trondheim, NO-7491,
Norway  Levanger hospital, Nord-Trøndelag Health Trust, Clinic of Medicine,
Levanger, NO-7600, Norway

###### Abstract

Considering the increased workload in pathology laboratories today, automated
tools such as artificial intelligence models can help pathologists with their
tasks and ease the workload. In this paper, we are proposing a segmentation
model (DRU-Net) that can provide a delineation of human non-small cell lung
carcinomas and an augmentation method that can improve classification results.
The proposed model is a fused combination of truncated pre-trained DenseNet201
and ResNet101V2 as a patch-wise classifier followed by a lightweight U-Net as
a refinement model. We have used two datasets (Norwegian Lung Cancer Biobank
and Haukeland University Hospital lung cancer cohort) to create our proposed
model. The DRU-Net model achieves an average of 0.91 Dice similarity
coefficient. The proposed spatial augmentation method (multi-lens distortion)
improved the network performance by 3%. Our findings show that choosing image
patches that specifically include regions of interest leads to better results
for the patch-wise classifier compared to other sampling methods. The
qualitative analysis showed that the DRU-Net model is generally successful in
detecting the tumor. On the test set, some of the cases showed areas of false
positive and false negative segmentation in the periphery, particularly in
tumors with inflammatory and reactive changes.

###### keywords:

Lung carcinoma, Digital pathology, Tumor segmentation, Deep learning, Data
augmentation

## Introduction

Early diagnosis of lung cancer is crucial for patient survival [1]. Although
physical examinations and medical imaging are included in the diagnostic work-
up, tissue samples are needed to establish a cancer diagnosis. The
histopathological diagnosis including analysis of tumor biomarkers influences
therapeutic decisions and should therefore be assessed as accurately and as
early as possible [2, 3].

By scanning tissue sections, the resultant whole slide images (WSIs) can be
assessed on a computer screen instead of with a regular microscope.
Digitization of histopathological sections may increase efficiency compared to
conventional microscopy assessment and it allows the use of artificial
intelligence (AI) in the analysis of WSIs [4]. AI methods may increase
accuracy and speed of image interpretation and have the potential to reduce
inter-observer variability and refine clinical decision-making [5, 6, 7]. AI
can be used for automated tissue classification, identification, and
segmentation [8]. Correct segmentation of the tumor is a necessary step
towards computer-assisted tumor analysis and lung cancer diagnosis [9, 10, 11,
12, 13, 14].

When working with WSIs, the application of AI models is complicated due to the
large size of the images. One approach involves separating the images into
several small squares, called patches. A patch-based analysis may, however,
lead to loss of broader spatial relationships. Alternatively, the image can be
down-sampled, or a hybrid strategy that combines both methods can be used to
optimize the analytical balance between detailed resolution and global
context. Some of the best-performing AI methods in the analysis of WSIs are
deep neural networks [15, 14]. The state-of-the-art in image segmentation
tasks is the use of complex neural network architectures such as vision
transformers and InternImage [16, 17]. However, these methods require a
relatively large amount of data [18]. Transfer learning techniques may also be
used to train or fine-tune pre-trained models on new data [19]. Patch-wise
classification (PWC) or segmentation approaches may outperform direct
segmentation of the tumor in a down-sampled image without dividing it into
patches [20].

Several models have been proposed for tumor segmentation in WSIs [11, 21, 22,
23, 24, 25, 26, 27, 28, 29]. Zhao et al. proposed a novel hybrid deep learning
framework for colorectal cancer that uses a U-Net architecture. This model
features innovative residual ghost blocks, which include switchable
normalization and bottleneck transformers for extracting features. [11].

The MAMC-Net model introduced a multi-resolution attention module that
utilizes pyramid inputs for broader feature information and detail capture
[21]. An attention mechanism refines features for segmentation, while a multi-
scale convolution module integrates semantic and high-resolution details.
Finally, a connected conditional random field ensures accurate segmentation by
addressing discontinuities [21]. The authors showcased superior performance of
their model on breast cancer metastases and gastric cancer [21].

DHU-Net combines Swin Transformer and ConvNeXt within a dual-branch
hierarchical U-shaped architecture [22, 30, 31]. This method effectively fuses
global and local features by processing WSI patches through parallel encoders,
utilizing global-local fusion modules and skip connections for detailed
feature integration [22]. The Cross-scale Expand Layer aids in resolution
recovery across different scales. The network was evaluated on datasets
covering different tumor features and cancer types and achieved higher
segmentation results than other tested methods [22].

Pedersen et al. introduced H2G-Net, a cascaded convolutional neural network
(CNN) architecture for segmenting breast cancer regions from gigapixel
histopathological images [23]. It employs a patch-wise detection stage and a
convolutional autoencoder for refinement, demonstrating significant
improvements in tumor segmentation. The approach outperformed single-
resolution methods, achieving a Dice similarity coefficient (DSC) of
(0.933±0.069) [23]. Its efficiency is underscored by fast processing times and
the ability to train deep neural networks without needing to store patches on
disk.

One of the most significant challenges in using WSIs for tumor segmentation is
still the scarcity of labeled data. The marking of tumor tissue in WSIs by
pathology experts is time-consuming, and may be a bottle neck in research. The
limited availability of such annotated datasets poses a significant hurdle for
the development and application of supervised learning algorithms in tumor
segmentation. Given this constraint, alternative computational strategies such
as unsupervised or semi-supervised learning methods should be explored.
Clustering emerges as a potent tool in this context, allowing for segmentation
of tumor regions with little or no need for predefined labels. [24, 25].

Yan et al. presented a self-supervised learning method using contrastive
learning to process WSIs for tissue clustering [26]. This approach generates
discriminative embeddings for initial clustering, refined by a silhouette-
based scheme, and extracts features using a multi-scale encoder [26]. It
achieved high accuracy in identifying tissues without annotations. Their
results show an area under the curve (AUC) of 0.99 and accuracy of
approximately 0.93 for distinguishing benign from malignant polyps in a cohort
of 20 patients [26].

Few-shot learning presents a promising way of handling scarcity of labeled
data [27, 28]. By design, few-shot learning algorithms can learn from a very
limited number of labeled examples. This capability is particularly relevant
for the classification of small patches, where a small set of labeled examples
can guide the learning process. Few-shot learning techniques can generalize
from these examples to classify new, unseen patches, facilitating the
identification and segmentation of tumor regions [27, 28]. Titoriya et al.
explored few-shot learning to enhance dataset generalization and manageability
by utilizing prototypical networks and model agnostic meta learning across
four datasets [29]. The design achieved 85% accuracy in a 2-way 2-shot 2-query
mode [29].

In this paper, we propose a new AI model (DRU-Net) for segmenting non-small
cell lung carcinomas (NSCLCs). It is an end-to-end approach consisting of a
dual head for feature extraction and patch classification followed by a U-Net
for refining the segmentation result. The method is trained and tested on a
novel in-house dataset of 97 annotated NSCLC WSIs. To increase model
performance, we adopted a few shot learning approach during training and added
a multi-lens distortion augmentation technique to WSI images.

## Methods

### Cohorts

In this study, two different collections of NSCLCs were used; the Norwegian
Lung Cancer Biobank (NLCB) cohort and the Bergen cohort [32, 33]. The NLCB
cohort includes histopathological, cytological, biomarker, and clinical
follow-up data from patients with suspected lung cancer diagnosed in Central
Norway after 2006 [34]. Both diagnostic tumor biopsies and sections from
surgical lung cancer specimen are available. The distribution of histological
subtypes in each dataset is listed in Table 1 [35, 36].

The Bergen Cohort comprises 438 surgically treated NSCLC patients diagnosed at
Haukeland University Hospital, Bergen, Norway from 1993–2010. In this study,
97 NSCLC cases from the Bergen cohort were included. From both cohorts, 4µm
tissue sections were stained with hematoxylin and eosin (HE) and scanned using
Olympus BX61VS VS120S5 at x40 magnification.

The sections were deparaffinized, rehydrated in ethanol, and immersed in tap
water. Hematoxylin staining was applied and the sections were rinsed in water,
and then in ethanol. Sections were then stained with alcoholic eosin. Post-
staining, the slides were dehydrated in ethanol, placed in TissueClear, and
air-dried [37].

Table 1: Histological subtypes of non-small cell lung carcinoma cases in the Norwegian lung cancer biobank (NLCB) and the Bergen cohort. Histological subtype |  NLCB (n) |  Bergen cohort - training and validation (n) |  Bergen cohort - test (n)
---|---|---|---
Adenocarcinoma |  16 |  38 |  7
Squamous cell carcinoma |  15 |  32 |  10
Other non-small cell carcinoma |  11 |  7 |  3
Total number of whole slide images |  42 |  77 |  20

To conduct a broader study of the proposed augmentation’s effect, we utilized
the following open datasets: MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100
[38, 39, 40].

### Ethical aspects

All methods were carried out in accordance with relevant guidelines and
regulations, and the experimental protocols were approved by the Regional
Committee for Medical and Health Sciences Research Ethics (REK) Norway
(2013/529, 2016/1156 and 257624). Informed consent was obtained from all
subjects and/or their legal guardian(s) for NLCB in accordance with REK
2016/1156. For subjects in the Bergen cohort, exempt from consent was
ethically approved by REK (2013/529).

### Annotations and dataset creations

We used two annotation approaches on WSIs; whole tumor annotation (WTA) and
partial selective annotation (PSA). In the WTA approach, pathologists marked
the tumor outline in 97 WSIs from the Bergen cohort. Of these WSIs, 51 were
used for training, 26 were used for validation, and 20 were used for testing.
All WSIs with tissue microarray (TMA) holes were placed in the test set to
prevent potential biased training. The remaining WSIs were randomly separated
into the training, validation, and test sets.

To reduce time spent by pathologists in making the WTA annotations, initial
annotations were first made in 72 cases using two different AI-based
segmentation models, i) the H2G-Net model developed for breast cancer
segmentation (n=25) and ii) a customized early-stage clustering model based on
the corrected annotations from the H2G-Net model (n=47) [23]. Pathologists
then manually adjusted the annotated tumor region using the QuPath software
[41]. The remaining 25 cases were annotated manually without AI-based
segmentation models. A second pathologist reviewed the annotations, and in
case of discrepancy, consensus was reached after discussion. The final
annotations were exported as binary masks, serving as ground truth.

In the PSA approach, pathologists marked small regions of interest in 42 WSIs
from the NLCB cohort. These WSIs were used for training and validation. The
marked areas included parts of the invasive tumor, normal alveolar tissue,
stromal tissue, immune cells, areas of necrosis, and other non-tumor tissue
such as respiratory epithelium, reactive alveolar tissue, cartilage, blood
vessels, glands, lymph nodes, and macrophages. The purpose of marking these
regions was to save time spent on manual annotations of the whole tumor
regions, and to guide the selection of patches intended for use in the patch-
wise model’s training.

### Proposed method

The pipeline of the proposed model (DRU-Net) has two distinct stages, a patch-
wise classification (PWC) stage and a refinement stage. The PWC model was
trained on the NLCB cohort using the many-shot method, and the refinement
U-Net was trained on a set of down-sampled WSIs from the Bergen cohort. In the
PWC stage, the model assigns probabilities to each patch of the WSIs
(excluding the glass) indicating whether the patch contains tumor tissue or
non-tumor tissue. The classifier’s output provides a preliminary assessment of
each patch’s nature, based on local features within the patch. The patches are
then stitched together to produce a heatmap of the same size as the down-
sampled WSIs.

Figure 1: Illustration of the proposed DRU-Net model. The patched image is fed
into the classifier part. The output of the classifier is combined with a
down-sampled WSI as an input for the refinement head.

#### Patch-wise classifier

The PWC was constructed by fusing truncated backbones of two architectures,
DenseNet201 and ResNet101V2, pre-trained on ImageNet [42]. These networks are
used for parallel processing of the input and feature generation (we refer to
this PWC model as DR-fused). In our proposed architecture, both DenseNet201
and ResNet101V2 receive the same input, which is the image patch. Each network
processes this input concurrently, and after feature extraction, the outputs
from both DenseNet201 and ResNet101V2 pass through their respective global
average pooling layers. This step compresses the feature representation and is
used to prevent overfitting. The compressed features from both networks are
then concatenated and fed through the classifier head (Figure 1).

#### Refinement network

Heatmaps are generated from applying the PWC across the WSIs. The resultant
heatmaps are then resized and concatenated with a down-sampled version of the
WSI ($1120\times 1120$ pixels). The fused inputs are then fed to a refinement
network, similarly, as proposed in H2G-Net [23]. Using a refinement network
allows for adjusting the initial patch-wise predictions based on global WSI-
level information.

The proposed refinement network is a simple, lightweight U-Net architecture,
specifically tailored to process two image inputs (Figure 1). In this model,
the two inputs (down-sampled RGB WSI and the heatmap) are concatenated into a
4-channel image and then processed through multiple convolutional layers with
ReLU activation function. The architecture includes standard components such
as Conv2D layers, batch normalization, spatial dropout, skip connections, max
pooling for down-sampling, and upsampling layers (nearest-neighbor
interpolation). The network concludes with a softmax activation function.

#### Data augmentation

To improve model robustness, data augmentation is commonly performed. Data
augmentation generates artificial copies of the training data through some
predefined algorithm. This allows the training data to better cover the
expected data variation. Data augmentation was integrated in the data
generation process of the training and the following methods were applied
randomly: flipping, rotations (multiples of $90^{\circ}$), multiplicative
contrast adjustment, hue, and brightness, and the proposed multi-lens
distortion augmentation method. During the many-shot learning using PSA, we
extracted patches by randomly cropping a $224\times 224$-pixel section from
each image. Each image appeared only once per epoch, where an epoch is defined
as one iteration of all the training data.

#### Multi-lens distortion augmentation

A novel data augmentation method, multi-lens distortion, was developed to
simulate several local random lens distortions. This technique aims to allow
the model to recognize the important features of the images under a wider
range of cell/tissue shapes.

The algorithm uses a fixed number of lenses. For each lens, a random position
in the image is selected. A random distortion radius and strength is then used
to apply the barrel and/or pincushion distortion effect at the position
(Algorithm 1). An example of this augmentation is shown in Figure 2.

Algorithm 1 Multi-Lens Distortion Effect on an Image.

Input: $image$ \- array (Input image of shape $H\times W\times C$)

Input: $num\\_lenses$ \- int (Number of lenses to apply)

Input: $radius\\_range$ \- tuple $(min\\_radius,max\\_radius)$

Input: $strength\\_range$ \- tuple $(min\\_strength,max\\_strength)$

Output: $distorted\\_image$ \- array (Image with lens effects applied)

$H,W,C\leftarrow\text{shape of }image$

$distorted\\_image\leftarrow\text{copy of }image$

Generate grid of pixel indices $yidx,xidx$ for $image$

Generate $num\\_lenses$ random centers $(cx,cy)$ within
$[image\\_size[0]-radius,image\\_size[1]-radius]$

for $i\leftarrow 0$ to $num\\_lenses-1$ do

$radius\leftarrow$ random value within $radius\\_range$

$strength\leftarrow$ random value within $strength\\_range$

Calculate the Euclidean distance $r$ from each pixel to the lens center
$(cx[i],cy[i])$

Normalize distances: $normalized\\_r\leftarrow\frac{r}{radius}$

Calculate scaling factor: $scaling\\_factor\leftarrow\max(1-normalized\\_r,0)$

Apply distortion to calculate new pixel positions:

$distorted\\_y\leftarrow(yidx-cy[i])\cdot(1-strength\cdot
scaling\\_factor)+cy[i]$

$distorted\\_x\leftarrow(xidx-cx[i])\cdot(1-strength\cdot
scaling\\_factor)+cx[i]$

Clip $distorted\\_y,distorted\\_x$ to image bounds

Update $distorted\\_image$ with pixels from original image at new positions

end for

return $distorted\\_image$

Figure 2: Sample effect of the novel augmentation on a patch with overlaid
grids to illustrate the effect. a) Original image showing epithelial cells. b)
Augmented image with parameters set too high, cell size variation and
deformation are visible. c) Augmented image with a medium setting of the
parameters.

#### Model training

The PWC network was fine-tuned to adapt to the specific task by freezing the
initial layers. The training parameters included: optimizer—Adamax with a
learning rate of $1\times 10^{-4}$; loss function—Categorical Crossentropy;
metrics—F1-score; batch size—dynamically determined based on the training
generator configuration; epochs—up to 200 with early stopping based on
validation loss to prevent overfitting.

The refinement network training involved: optimizer—Adam with a learning rate
of $1\times 10^{-4}$; loss function—Dice loss function, optimized for
segmentation tasks; metrics—Thresholded Dice score; batch size—2; epochs—up to
300 with early stopping based on validation loss to prevent overfitting;
training environment—utilization of GPU and memory growth settings to optimize
hardware usage.

In the WTA method, the same set of slides was used for both PWC and
segmentation models’ training. From the 97 slides, 77 slides were randomly
chosen and divided into training and validation sets by a ratio of 1/3 (51 and
26 slides, respectively), while 20 slides (including those with TMA holes)
were used for testing.

WSIs in the dataset from the Bergen cohort were divided into tiles (patches)
and each tile was fed into the neural network along with the non-tumor/tumor
label based on the provided annotation. To create the annotation labels for
patches, non-tumor and tumor tiles were assigned the values 0 and 1,
respectively. We first used a threshold on color gradients to separate the
tissue from the background glass. Any tile that did not include more than 25%
tissue was disregarded, meaning that all the input tiles contained less than
75% background glass. Also, a minimum of 5% tumor area was required for a tile
to be classified as tumor; and for the non-tumor regions, only tiles with no
tumor were assigned. Tiles with less than 5% area of tumor were disregarded.

Using the annotated WSI regions in the NLCB dataset, 40 areas were appointed
to the tumor class (labeled as 1) and 50 areas to the non-tumor class (labeled
as 0). The selected areas led to the generation of patches in subsequent
steps. Specifically, out of 50 areas categorized as non-tumor, 40 clearly
lacked tumor characteristics, and 10 showed features slightly above the
initially-achieved threshold, as shown in Supplementary Fig. S2. This
threshold was established through model training before intentionally creating
an imbalance in the dataset. The imbalance was introduced after unsuccessful
attempts to enhance model generalizability through various methods, including
weighted loss functions, focal loss, threshold adjustment, and sampling
strategies.

#### Post-processing

After the segmentation results were received, two post-processing steps were
performed. First, small fragments were removed by converting images into
grayscale and then to binary format to identify and eliminate fragments
smaller than certain size threshold. The threshold was set as the smallest
segmentation area annotated in our ground truth. In the second step, an edge
smoothing algorithm was applied to enhance image quality. This improvement was
achieved through mathematical techniques known as morphological operations,
which are commonly used in digital image processing to modify the geometrical
structure of images. Specifically, we used a process called morphological
opening, which involves an erosion operation followed by a dilation. This
sequence helps reduce jagged edges and smooths the boundaries of objects
within the image. The operations were performed using a kernel size of
$7\times 7$. Additionally, a median blur with a kernel size of $11\times 11$
was applied to further smooth the edges. It’s important to note that these
morphological operations are purely computational methods used to process the
digital images and should not be confused with the morphological study of
biological tissues.

### Implementation

Implementation was done in Python 3.8.10. TensorFlow (v2.13.1) was used for
model architecture implementation and training [43]. These additional
libraries were used for the experiments: pyFAST, OpenCV, NumPy, Pillow, SciPy,
scikit-learn, and Matplotlib [44, 45, 46, 47, 48, 49, 50, 51]. Trained models
were converted to the ONNX format using the tf2onnx library [52]. Converted
models were then integrated into FastPathology for deployment [53].
FastPathology is an open-source, user-friendly software developed for deep
learning-based digital pathology that offers tools for processing and
visualizing WSIs. The source code used to conduct the experiments is made
openly available at https://github.com/AICAN-Research/DRU-Net.

### Experiments

To compare the proposed model (DRU-Net) with other models, the following
experiments were carried out: modifications of the previously introduced
H2G-Net model on both datasets, DRU-Net with the backbone trained on the
Bergen cohort and NLCB, and applying the few-shot and many-shot learning
techniques along with clustering (Table 2) [23].

H2G-Net could be tested as is, and be fine-tuned with five different
modifications [23]. First, H2G-Net was tested without any modification, fine-
tuning, or additional training, to see whether a model trained for breast
cancer tumor delineation can also work for lung cancer. Second, the PWC of the
H2G-Net was fine-tuned on annotated NSCLCs from the Bergen cohort, and the
original U-Net of H2G-Net was applied on top of the PWC results. Third, the
whole model (PWC and U-Net) was fine-tuned on the training data. Then, the
same three methods were tested, but with the PWC trained on NLCB instead of
the Bergen cohort.

Table 2: Methods and experiments carried out with various models on the same 20 WSIs of the test set from the Bergen cohort. Abbreviations: PWC: patch-wise classifier; NLCB: Norwegian Lung Cancer Biobank; FSC: few-shot (with a pre-trained MobileNetV2 [54] model) + clustering; MSC: many-shot (with a pre-trained MobileNetV2 [54] model) + clustering. | Models | Modifications | Training dataset(s)
---|---|---|---
(i@) | H2G-Net | — | —
(ii@) | H2G-Net | Fine-tuned PWC | Bergen Cohort
(iii@) | H2G-Net | Fine-tuned U-Net | Bergen Cohort
(iv@) | H2G-Net | Fine-tuned PWC and original U-Net | Bergen Cohort
(v@) | DRU-Net | — | Bergen Cohort
(vi@) | H2G-Net | Fine-tuned PWC | NLCB
(vii@) | H2G-Net | Fine-tuned PWC and U-Net | PWC trained on NLCB, U-Net trained on Bergen Cohort
(viii@) | FSC | — | NLCB
(ix@) | MSC | — | NLCB
(x@) | DRU-Net | — | PWC trained on NLCB, U-Net trained on Bergen Cohort

An ablation study was performed to evaluate the effect of the proposed multi-
lense distortion augmentation. A pre-trained DenseNet121 was tested on four
open datasets: MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100 [38, 39, 40].
Experiments were repeated with and without this augmentation on the mentioned
open datasets by randomly selecting 10% of the training data and the results
were compared using Wilcoxon test (Table 4). Both control and test groups
included other augmentation techniques such as color adjustments, flipping,
rotation, brightness, and contrast augmentations. The effect of this
augmentation on the training time was measured using the integrated TensorFlow
functions by comparing the time with and without the augmentation and the
results were averaged on WSIs and compared between the two [43].

We also investigated the effect of removing the top-most skip connection of
the U-Net refinement model and we calculated the average Hausdorff distances
(HDs) for two sets of final segmentation predictions in comparison to a ground
truth set. This was done to quantify the effect of removing that skip
connection, which was done to reduce the small fragments around the
segmentation perimeter.

### Model evaluation

#### Quantitative model assessment

To quantitatively validate the patch-wise classification performance,
precision, recall, and F1-score were used [55]. The validation of the final
segmentation on WSI-level was performed using DSC and HD [56].

#### Qualitative model assessment

The qualitative assessment of the segmentation results was conducted by two
pathologists using the scoring system described in Table 3. Qualitative
assessment was done on the same 20 WSIs of the test set from the Bergen
cohort.

Table 3: Qualitative evaluation scoring system. 0 |  1 |  2 |  3 |  4 |  5
---|---|---|---|---|---
No tumor tissue in image or segmentation, or image not suitable for analysis |  Completely wrong segmentation of tumor, tumor tissue not segmented |  A large part of the tumor is not segmented |  Most of the tumor is correctly segmented, but some false positive or false negative areas |  Most of the tumor is correctly segmented, only sparse false positive or false negative areas |  The whole or almost the whole tumor correctly segmented

#### Saliency maps

To survey the model’s decision-making process and the areas of patches that
were most relevant for predicting the tumor class, we employed a method known
as gradient-based saliency maps [57, 58, 59, 60]. This approach operates by
computing the gradient of the output class (the class for which we want to
understand model sensitivity) with respect to the input image. These gradients
indicate the sensitivity of the output to each pixel in the input image. By
highlighting the pixels with the highest gradients, we can visualize the areas
that most strongly influenced the model’s classification decision. We used six
different patches for this test selected from six different WSIs from the
Bergen cohort. Patches were chosen to represent true and false positive
predictions. Patches with true positive predictions were selected to include
various histological features and cell types in each patch to better assess
the model’s decision process.

## Results

Highest DSC in average on the 20 WSIs of the test set from Bergen cohort was
achieved by DRU-Net followed by the H2G-Net with fine-tuned PWC on the Bergen
cohort (Figure 3). Similar differences in DSC were observed for the models
without the refinement networks (Figure 4).

Figure 3: Boxplots of the Dice similarity coefficients (DSCs) of the
experiments shown Table 2 on the 20 WSIs of the test set. I) original H2G-Net,
II) H2G-Net with fine-tuned PWC on Bergen cohort, III) H2G-Net with fine-tuned
U-Net on Bergen cohort, IV) H2G-Net with fine-tuned PWC and U-Net on Bergen
cohort, V) DRU-Net trained on Bergen Cohort, VI) H2G-Net with fine-tuned PWC
on NLCB, VII) H2G-Net with fine-tuned PWC on NLCB and fine-tuned U-Net on
Bergen Cohort, VIII) FSC, IX) MSC, X) DRU-Net with PWC trained on NLCB and
U-Net trained on Bergen Cohort. Figure 4: Boxplot of the Dice similarity
coefficients (DSCs) of the PWC models in experiments listed in Table 2 without
the refinement network, only the patch-wise classifier is used to produce
these results. I) original H2G-Net, II) H2G-Net with fine-tuned PWC on Bergen
cohort, V) DR-fused trained on Bergen Cohort, VI) H2G-Net with fine-tuned PWC
on NLCB, X) DR-fused trained on NLCB and U-Net trained on Bergen Cohort.

Proposed multi-lens distortion augmentation applied to various datasets
resulted in increased F1-score overall, this change was statistically
significant when applied to our dataset from the NLCB (Table 4). Applying this
augmentation technique increased training time by an average of 8%. DSC and
patch-wice accuracy were increased when the multi-lens distortion augmentation
was used with a strength of magnitude in the range [0.2, 0.4], but higher
magnitudes caused a decrease in the performance (Figure 5).

Table 4: The impact of the multi-lens distortion augmentation technique using different architectures on different datasets with randomly selecting 10% of the training data. Pairwise tests were performed using Wilcoxon signed-rank tests. The augmentation design with the highest F1-scores row-wise are highlighted in bold. |  | F1-score |
---|---|---|---
Model | Dataset | wo/aug | w/aug |  $p$-value
DenseNet121 | MNIST | 0.9893 | 0.9894 | 0.2311
DenseNet121 | Fashion-MNIST | 0.9043 | 0.9208 | <0.001
DenseNet121 | CIFAR-10 | 0.8086 | 0.8235 | <0.001
DenseNet121 | CIFAR-100 | 0.5199 | 0.5581 | 0.0502
H2G-Net | NLCB | 0.8299 | 0.8341 | 0.0701
DRU-Net | NLCB | 0.8868 | 0.9025 | 0.0241
Figure 5: The impact of the multi-lens distortion augmentation technique using
the DRU-Net model. DSC: Dice similarity coefficient. The highlighted regions
indicate the variance, and the mean values are shown on the curve.

The original H2G-Net resulted in an average of 0.76 DSC (Figure 3) and 0.66
intersection over union (IOU) scores. On average, 25% of the non-tumor regions
around the true tumor outlines were falsely labeled as tumor. When the PWC
part of the model was used without refinement, the predictions resulted in
0.64 DSC and 0.61 IOU, showing that the refinement improved the predictions
significantly.

Fine-tuned PWC trained and validated on 77 WSIs from the Bergen cohort with
direct implementation of pre-trained U-Net from H2G-Net was tested on 20 WSIs
from the Bergen cohort and resulted in an average of 0.83 DSC (median 0.91)
(Figure 3) and an average 0.74 IOU scores. Scores were reduced to an average
of 0.77 DSC (median of 0.87) and an average of 0.69 IOU when both the U-Net
and the PWC were fine-tuned.

The proposed model (DRU-Net) tested on the same 20 WSIs resulted in an average
of 0.91 DSC (median 0.93) and 0.81 IOU. Also removing the top skip connection
in our U-Net model (DRU-Net) resulted in the average reduction of HD by 4.8%.
Figure 6 shows a comparison of the results from various models. Table 5
summarizes various backbones’ performance in the patch-wise classifier part of
the model.

Table 5: Comparison of different backbone architectures for patch-wise classification of lung cancer tissue using the many-shot method. The best-performing architecture per metric is highlighted in bold. Abbreviations: DR: fusion of DenseNet201 (D) and ResNet101V2 (R). Architecture | F1-score | Precision | Recall
---|---|---|---
VGG19 [61] | 0.87 | 0.86 | 0.87
ResNet101V2 [62] | 0.89 | 0.89 | 0.89
MobileNetV2 [54] | 0.86 | 0.86 | 0.86
EfficientNetV2 [63] | 0.89 | 0.89 | 0.89
InceptionV3 [64] | 0.90 | 0.89 | 0.91
DenseNet201 [65] | 0.91 | 0.91 | 0.91
Proposed DR-fused | 0.94 | 0.94 | 0.93

We compared the performance of several models on processing a set of 20 WSIs
with average dimensions being approximately 108 640 pixels in width and 129
835 pixels in height. H2G-Net and its fine-tuned versions were the fastest
models during inference (62 seconds). Although the many-shot and few-shot
models had faster training they exhibited slower runtimes, with MSC taking the
longest at 167 seconds and DRU-Net at 152 seconds.

Figure 6: Sample results of three tested networks. First row: original whole
slide images (WSIs), second row: DRU-Net, third row: FSC (Few-shot learning +
clustering), fourth row: H2G-Net with fine-tuned patch-wise classifier and
original U-Net. Green pixels indicate true positives, White pixels indicate
false positives and red pixels indicate false negatives.

Results of saliency map analysis on 6 patches are shown in Figure 7. False
positive areas in the saliency maps were partly explained by areas with
reactive pneumocytes, macrophages, and reactive pneumocyte hyperplasia.

The qualitative assessment resulted in an average score of 3.95 out of 5. In 9
of the assessed cases, there were sparse areas in the periphery that the model
misclassified.

Figure 7: Sample patches (top row) and their overlaid saliency maps (bottom
row) where only the patches were given to the PWC model. Note that the
saliency map does not indicate malignancy; instead, it shows how different
regions of the image influence the classification decision. The colors on the
map range from blue, indicating the least influence, to red, which indicates
the most influence. Figures a, b, and c show false positive tumor detection.
Figures d, e, and f show true positive tumor detection.

## Discussion

In this paper, we introduce a novel AI-based model to segment the outline of
NSCLCs. We have incorporated a patch-wise classifier, synergistically
integrating truncated DenseNet201 and ResNet101V2 architectures, which is
enhanced by a segmentation refinement model adopting a streamlined U-Net
framework. This composite model demonstrated superior performance over other
tested backbones. This study also resulted in a novel dataset comprising
annotated NSCLCs and marked regions of interest in WSIs from NSCLCs, including
different tissue types. Our results also indicate that the PSA approach
yielded more effective training outcomes for the patch-wise classifier than
WTA techniques with and without class balancing using tissue clustering.

Our study also demonstrated that the implementation of the multi-lens
distortion augmentation technique enhanced classification outcomes across
diverse datasets with a limited volume of training data. However, it is
important to acknowledge that the effect of this augmentation can vary
depending on the data itself. We investigated the effect of the augmentation’s
strength range on the patch-wise accuracy and refinement network’s DSC on WSI-
level, concluding that the degree of augmentation applied plays a pivotal role
in its impact on the training process. Excessively strong distortion of images
could obstruct the model’s ability to learn relevant patterns as shown in the
impact of the multi-lens distortion augmentation with various strength ranges.

The RGSB-UNet model features a unique hybrid design that combines residual
ghost blocks with switchable normalization and a bottleneck transformer [11].
This design focuses on extracting refined features through its complex
structure. However, in our study, we found that simpler and more synergistic
designs can also effectively extract reliable features.

The MAMC-Net model improves tumor boundary detection by using a conditional
random field layer [21], whereas the DRU-Net model enhances segmentation by
fine-tuning a U-Net on a down-sampled image. Both methods achieved good
results, but our approach using a U-Net on down-sampled images is faster and
still highly efficient. Compared to the MAMC-Net study, our PSA approach used
a much smaller dataset and achieved similar results.

Similar to H2G-Net, our proposed model, DRU-Net, also utilizes a cascaded
design with the two stages of PWC and refinement and has achieved comparable
results [23]. Although H2G-Net uses a lightweight PWC and a relatively heavier
U-Net for refinement, DRU-Net performed better with a heavier PWC and a
lightweight U-Net. This could be due to less training data available in our
case which can require a more complex feature extraction process. Pedersen et
al. also introduced a balancing technique to ensure a balanced representation
of the available categories, which helps minimizing bias toward any specific
tissue type or tumor characteristic. In our study, we have also used a
clustering-based balancing method to reduce bias during sampling from our WTA.
However, we have also benefited from our PSA approach with the induced
imbalance in generated samples to minimize the bias toward tumor labels, which
proved to work better in our test dataset.

The decrease in performance after fine-tuning the U-Net layers of the H2G-Net
may be due to the relatively small number of annotated WSIs available in our
study. Conversely, the DRU-Net network’s superior performance under similar
conditions suggests the efficacy of the DR-fused network accompanied by a
relatively lightweight U-Net architecture in data-scarce scenarios.

The relatively low performance of the original H2G-Net on NSCLCs can be
explained by different tissue morphology, growth pattern, and stromal
invasions, which can cause misleading during inference [66, 67, 68, 69, 70,
71, 72, 36].

In this study, we encountered challenges due to significant class imbalance
between the patches derived from the WTA approach. Addressing the resultant
low precision, a comprehensive strategy was implemented to improve model
accuracy. Key interventions included resampling techniques, both under- and
over-sampling, as well as the incorporation of focal loss, which specifically
helps to address class imbalance by modulating the loss function to focus on
harder-to-classify examples [73]. Furthermore, we explored the clustering of
similar tissue types before sampling, the use of a weighted loss function, and
adjustments to the decision threshold.

Additionally, in the training phase of the many-shot model using the samples
derived from PSA approach, to maximize the model’s performance, we
deliberately introduced a controlled imbalance which was aimed at optimizing
the threshold settings. The deliberate construction of an imbalanced dataset
resulted in improved performance. This approach outperformed resampling,
under- and over-sampling, focal loss, sampling from clustered tissue types,
weighted loss function, and threshold tuning [73]. However, the induced class
imbalance, while promising, carries the risk of significant bias. It
necessitates careful calibration and continuous monitoring to ensure that the
model does not disproportionately favor certain classes or features, leading
to skewed results. In our case, the performance was ascertained by testing on
an external dataset (the PWC trained using the many-shot method was trained on
the NLCB dataset and tested on the 20 slides that were selected for testing
from the Bergen cohort).

Our results indicate that refining the PWC heatmap with the suggested
refinement networks improves the accuracy of the tested models. However, the
main strengths and weaknesses of the models compared to each other directly
stem from the training method used for the PWC models. Additionally, combining
the two processes seems to improve and reduce the variance in the segmentation
DSC values, indicating that the refinement models have learned to understand
overall patterns and connections leading to a better segmentation.

The difference observed in the average DSCs between PWC models indicates that
PSA can outperform WTA approaches when the amount of data is relatively
limited. This is because of the inadequate separability of the features
distributions between tumor and non-tumor. In the WTA approach, the method
involved annotating entire tumor regions, which often included patches where
the feature distributions of tumor and non-tumor tissues overlapped
significantly. This overlap resulted in inadequate separability, thereby
reducing the discriminatory power of the classification models trained using
this approach. Consequently, the distinction between tumor and non-tumor
features in these patches became less pronounced, leading to potential
misclassifications.

Conversely, the PSA method adopted a more selective approach by targeting
patches for annotation based on their discriminative morphology. By focusing
on patches where the features of each class (tumor and non-tumor) were
distinctly separable, PSA enhanced the model’s ability to accurately classify
these features. This selective annotation process, effectively increased the
inter-class variance while reducing the intra-class variance, thus
significantly improving the overall performance of the classification models
in distinguishing between tumor and non-tumor tissues under conditions of
limited data. In the WTA approach, the mentioned inseparable feature
distribution affected the loss function negatively resulting in lower
accuracy. This was most likely rooted in the fact that the tumor regions also
include other cell types than the invasive epithelial cells. By using
histopathological knowledge for selecting areas with the most relevant
features in PSA, variation of the features between the two classes could be
increased.

Additionally, the study presents evidence that employing few-shot learning in
conjunction with a clustering approach can achieve accuracy levels comparable
to those reliant on extensive datasets, thereby potentially mitigating the
need for large-scale data collection. The few-shot learning approach is
preferable when there is a high degree of similarity within each class of
tissue types and a clear distinction between the classes in the feature space
[74].

In our few-shot learning method, we introduced a novelty by utilizing an
evolutionary optimization technique to determine the optimal number of
clusters (classes) to minimize intra-cluster variance and maximize inter-
cluster variance prior to training. This method ensures that clusters are
optimally configured to reflect the most coherent and meaningful class
structures, which is crucial when the available training data is scarce. By
focusing on minimizing intra-cluster variance and minimizing inter-cluster
similarity, the approach enhances the model’s ability to generalize from
limited examples, a critical aspect in few-shot scenarios where the risk of
overfitting is high. Furthermore, evolutionary algorithms offer robust
adaptability and flexibility, enabling the model to effectively handle
different types and distributions of data. This pre-training optimization led
to more efficient training and improved model performance by grouping patches
into different classes.

Qualitative assessment of our results suggests that the DRU-Net model shows
limitations in accurately delineating the tumor periphery. This challenge was
particularly evident in regions with fibrosis, reactive tissue, or
inflammation, where the model tends to produce false positive and false
negative segmentations.

In future work, we suggest reducing the model size, incorporating advanced
attention-focusing mechanisms, and using a multi-scale patch-wise classifier
to better incorporate information at different scales. Employing anomaly
detection algorithms might help identify reactive tissue outliers that
contribute to false positive classifications. Additionally, Mask R-CNN
architectures are highly effective in distinguishing complex patterns that can
be used for a better tumor border delineation. Also, implementing Bayesian
neural networks can help in predicting tumor boundaries while quantifying the
uncertainty of the predictions. To better use the global information, one can
also integrate Markov random fields or conditional random fields along with
PWC or transformer architectures to help ensure that the segmented areas are
not only based on local pixel values. We also suggest Neuro-Fuzzy Systems,
which leverage the learning capabilities of neural networks with the reasoning
capabilities of fuzzy logic to improve the differentiation between the two
classes. To overcome the limited data problem, we can suggest using
unsupervised domain adaptation algorithms to leverage annotated data from
other histopathology source domains.

## Conclusion

In conclusion, we have introduced DRU-Net for non-small cell lung cancer tumor
delineation in WSIs. Our novel AI-based model, which synergistically
integrates truncated DenseNet201 and ResNet101V2 with a U-Net based refinement
framework, has demonstrated high performance in NSCLCs over various tested
methods. The effectiveness of our patch-wise classifier has been significantly
enhanced through the incorporation of the multi-lens distortion augmentation
technique and PSA strategy. Additionally, employing few-shot learning with
optimized clustering could mitigate the need for large datasets.

## References

  * [1] Rami-Porta, R.  Future perspectives on the TNM staging for lung cancer.  _Cancers_ 13, 1940 (2021).
  * [2] Lim, C. _et al._ Biomarker testing and time to treatment decision in patients with advanced nonsmall-cell lung cancer.  _Annals of Oncology_ 26, 1415–1421 (2015).
  * [3] Woodard, G. A., Jones, K. D. & Jablons, D. M.  Lung cancer staging and prognosis.  _Lung cancer: treatment and research_ 47–75 (2016).
  * [4] Hanna, M. G. _et al._ Implementation of digital pathology offers clinical and operational increase in efficiency and cost savings.  _Archives of pathology & laboratory medicine_ 143, 1545–1555 (2019).
  * [5] Bera, K., Schalper, K. A., Rimm, D. L., Velcheti, V. & Madabhushi, A.  Artificial intelligence in digital pathology—new tools for diagnosis and precision oncology.  _Nature reviews Clinical oncology_ 16, 703–715 (2019).
  * [6] Sakamoto, T. _et al._ A narrative review of digital pathology and artificial intelligence: focusing on lung cancer.  _Translational Lung Cancer Research_ 9, 2255 (2020).
  * [7] Niazi, M. K. K., Parwani, A. V. & Gurcan, M. N.  Digital pathology and artificial intelligence.  _The lancet oncology_ 20, e253–e261 (2019).
  * [8] Kurc, T. _et al._ Segmentation and classification in digital pathology for glioma research: challenges and deep learning approaches.  _Frontiers in neuroscience_ 14, 27 (2020).
  * [9] Ho, D. J. _et al._ Deep multi-magnification networks for multi-class breast cancer image segmentation.  _Computerized Medical Imaging and Graphics_ 88, 101866 (2021).
  * [10] Qaiser, T. _et al._ Fast and accurate tumor segmentation of histology images using persistent homology and deep convolutional features.  _Medical image analysis_ 55, 1–14 (2019).
  * [11] Zhao, T., Fu, C., Tie, M., Sham, C.-W. & Ma, H.  RGSB-UNet: Hybrid Deep Learning Framework for Tumour Segmentation in Digital Pathology Images.  _Bioengineering_ 10, 957 (2023).
  * [12] Viswanathan, V. S., Toro, P., Corredor, G., Mukhopadhyay, S. & Madabhushi, A.  The state of the art for artificial intelligence in lung digital pathology.  _The Journal of pathology_ 257, 413–429 (2022).
  * [13] Wang, S. _et al._ Artificial intelligence in lung cancer pathology image analysis.  _Cancers_ 11, 1673 (2019).
  * [14] Davri, A. _et al._ Deep Learning for Lung Cancer Diagnosis, Prognosis and Prediction Using Histological and Cytological Images: A Systematic Review.  _Cancers_ 15, 3981 (2023).
  * [15] Cheng, J., Huang, K. & Xu, J.  Computational pathology for precision diagnosis, treatment, and prognosis of cancer.  _Frontiers in Medicine_ 10, 1209666 (2023).
  * [16] Ranftl, R., Bochkovskiy, A. & Koltun, V.  Vision transformers for dense prediction.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 12179–12188 (2021).
  * [17] Wang, W. _et al._ Internimage: Exploring large-scale vision foundation models with deformable convolutions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 14408–14419 (2023).
  * [18] Park, N. & Kim, S.  How do vision transformers work?  _arXiv preprint arXiv:2202.06709_ (2022).
  * [19] Kassani, S. H., Kassani, P. H., Wesolowski, M. J., Schneider, K. A. & Deters, R.  Deep transfer learning based model for colorectal cancer histopathology segmentation: A comparative study of deep pre-trained models.  _International Journal of Medical Informatics_ 159, 104669 (2022).
  * [20] Lin, H. _et al._ ScanNet: A Fast and Dense Scanning Framework for Metastastic Breast Cancer Detection from Whole-Slide Image.  In _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 539–546, DOI: 10.1109/WACV.2018.00065 (2018).
  * [21] Zeng, L. _et al._ MAMC-Net: an effective deep learning framework for whole-slide image tumor segmentation.  _Multimedia Tools and Applications_ 1–21 (2023).
  * [22] Wang, L. _et al._ DHUnet: Dual-branch hierarchical global–local fusion network for whole slide image segmentation.  _Biomedical Signal Processing and Control_ 85, 104976 (2023).
  * [23] Pedersen, A. _et al._ H2G-Net: A multi-resolution refinement approach for segmentation of breast cancer region in gigapixel histopathological images.  _Frontiers in Medicine_ 9, 971873 (2022).
  * [24] Albusayli, R. _et al._ Simple non-iterative clustering and CNNs for coarse segmentation of breast cancer whole-slide images.  In _Medical Imaging 2021: Digital Pathology_ , vol. 11603, 100–108 (SPIE, 2021).
  * [25] Chelebian, E., Avenel, C., Ciompi, F. & Wählby, C.  DEPICTER: Deep representation clustering for histology annotation.  _Computers in Biology and Medicine_ 108026 (2024).
  * [26] Yan, J., Chen, H., Li, X. & Yao, J.  Deep contrastive learning based tissue clustering for annotation-free histopathology image analysis.  _Computerized Medical Imaging and Graphics_ 97, 102053 (2022).
  * [27] Deuschel, J. _et al._ Multi-prototype few-shot learning in histopathology.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 620–628 (2021).
  * [28] Shakeri, F. _et al._ FHIST: a benchmark for few-shot classification of histological images.  _arXiv preprint arXiv:2206.00092_ (2022).
  * [29] Titoriya, A. K. & Singh, M. P.  Few-Shot Learning on Histopathology Image Classification.  In _2022 International Conference on Computational Science and Computational Intelligence (CSCI)_ , 251–256 (IEEE, 2022).
  * [30] Liu, Z. _et al._ Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 10012–10022 (2021).
  * [31] Liu, Z. _et al._ A ConvNet for the 2020s.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 11976–11986 (2022).
  * [32] Hatlen, P.  Lung cancer-influence of comorbidity on incidence and survival: The Nord-Trøndelag Health study (2014).
  * [33] Ramnefjell, M., Aamelfot, C., Helgeland, L. & Akslen, L. A.  Vascular invasion is an adverse prognostic factor in resected non–small-cell lung cancer.  _Apmis_ 125, 197–206 (2017).
  * [34] Hatlen, P., Grønberg, B. H., Langhammer, A., Carlsen, S. M. & Amundsen, T.  Prolonged survival in patients with lung cancer with diabetes mellitus.  _Journal of Thoracic Oncology_ 6, 1810–1817 (2011).
  * [35] Yoh Watanabe, M.  TNM classification for lung cancer.  _Ann Thorac Cardiovasc Surg_ 9 (2003).
  * [36] Travis, W.  The 2015 WHO classification of lung tumors.  _Der Pathologe_ 35, 188–188 (2014).
  * [37] Valla, M. _et al._ Molecular subtypes of breast cancer: long-term incidence trends and prognostic differences.  _Cancer Epidemiology, Biomarkers & Prevention_ 25, 1625–1634 (2016).
  * [38] Deng, L.  The MNIST Database of Handwritten Digit Images for Machine Learning Research.  _IEEE Signal Processing Magazine_ 29, 141–142 (2012).
  * [39] Xiao, H., Rasul, K. & Vollgraf, R.  Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms.  _arXiv preprint arXiv:1708.07747_ (2017).
  * [40] Krizhevsky, A., Hinton, G. _et al._ Learning Multiple Layers of Features from Tiny Images (2009).
  * [41] Bankhead, P. _et al._ QuPath: Open source software for digital pathology image analysis.  _Scientific reports_ 7, 1–7 (2017).
  * [42] Deng, J. _et al._ ImageNet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255 (Ieee, 2009).
  * [43] Abadi, M. _et al._ TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems (2015).  Software available from tensorflow.org.
  * [44] Smistad, E., Bozorgi, M. & Lindseth, F.  FAST: framework for heterogeneous medical image computing and visualization.  _International Journal of computer assisted radiology and surgery_ 10, 1811–1822 (2015).
  * [45] Smistad, E., Østvik, A. & Pedersen, A.  High performance neural network inference, streaming, and visualization of medical images using FAST.  _IEEE Access_ 7, 136310–136321 (2019).
  * [46] Bradski, G.  The OpenCV Library.  _Dr. Dobb’s Journal of Software Tools_ (2000).
  * [47] Clark, A.  Pillow (pil fork) documentation (2015).
  * [48] Harris, C. R. _et al._ Array programming with NumPy.  _Nature_ 585, 357–362, DOI: 10.1038/s41586-020-2649-2 (2020).
  * [49] Virtanen, P. _et al._ SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  _Nature Methods_ 17, 261–272, DOI: 10.1038/s41592-019-0686-2 (2020).
  * [50] Pedregosa, F. _et al._ Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ 12, 2825–2830 (2011).
  * [51] Hunter, J. D.  Matplotlib: A 2d graphics environment.  _Computing in Science & Engineering_ 9, 90–95, DOI: 10.1109/MCSE.2007.55 (2007).
  * [52] ONNX.  Convert TensorFlow, Keras, Tensorflow.js and Tflite models to ONNX (2024).
  * [53] Pedersen, A. _et al._ FastPathology: An open-source platform for deep learning-based research and decision support in digital pathology.  _IEEE Access_ 9, 58216–58229 (2021).
  * [54] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. & Chen, L.-C.  MobileNetV2: Inverted Residuals and Linear Bottlenecks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4510–4520 (2018).
  * [55] Goutte, C. & Gaussier, E.  A probabilistic interpretation of precision, recall and F-score, with implication for evaluation.  In _European conference on information retrieval_ , 345–359 (Springer, 2005).
  * [56] Kim, H. _et al._ Quantitative evaluation of image segmentation incorporating medical consideration functions.  _Medical physics_ 42, 3013–3023 (2015).
  * [57] Patro, B. N., Lunayach, M., Patel, S. & Namboodiri, V. P.  U-CAM: Visual Explanation using Uncertainty based Class Activation Maps.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 7444–7453 (2019).
  * [58] Simonyan, K., Vedaldi, A. & Zisserman, A.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  _arXiv preprint arXiv:1312.6034_ (2013).
  * [59] Zeiler, M. D. & Fergus, R.  Visualizing and understanding convolutional networks.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13_ , 818–833 (Springer, 2014).
  * [60] Sundararajan, M., Taly, A. & Yan, Q.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , 3319–3328 (PMLR, 2017).
  * [61] Simonyan, K. & Zisserman, A.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * [62] He, K., Zhang, X., Ren, S. & Sun, J.  Deep Residual Learning for Image Recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778 (2016).
  * [63] Tan, M. & Le, Q.  EfficientNetV2: Smaller Models and Faster Training.  In _International conference on machine learning_ , 10096–10106 (PMLR, 2021).
  * [64] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z.  Rethinking the Inception Architecture for Computer Vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2818–2826 (2016).
  * [65] Huang, G., Liu, Z., Van Der Maaten, L. & Weinberger, K. Q.  Densely Connected Convolutional Networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4700–4708 (2017).
  * [66] Menon, A., Singh, P., Vinod, P. & Jawahar, C.  Exploring Histological Similarities Across Cancers From a Deep Learning Perspective.  _Frontiers in Oncology_ 12, 842759 (2022).
  * [67] Kashima, J., Kitadai, R. & Okuma, Y.  Molecular and Morphological Profiling of Lung Cancer: A Foundation for "Next-Generation" Pathologists and Oncologists.  _Cancers_ 11, 599 (2019).
  * [68] Petersen, I.  The morphological and molecular diagnosis of lung cancer.  _Deutsches Ärzteblatt International_ 108, 525 (2011).
  * [69] Inamura, K.  Lung cancer: understanding its molecular pathology and the 2015 WHO classification.  _Frontiers in oncology_ 7, 193 (2017).
  * [70] Zhao, S. _et al._ Single-cell morphological and topological atlas reveals the ecosystem diversity of human breast cancer.  _Nature Communications_ 14, 6796 (2023).
  * [71] Binder, A. _et al._ Morphological and molecular breast cancer profiling through explainable machine learning.  _Nature Machine Intelligence_ 3, 355–366 (2021).
  * [72] Tan, P. H. _et al._ The 2019 WHO classification of tumours of the breast.  _Histopathology_ (2020).
  * [73] Lin, T.-Y., Goyal, P., Girshick, R., He, K. & Dollár, P.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , 2980–2988 (2017).
  * [74] Qi, Y., Sun, H., Liu, N. & Zhou, H.  A Task-Aware Dual Similarity Network for Fine-Grained Few-Shot Learning.  In _Pacific Rim International Conference on Artificial Intelligence_ , 606–618 (Springer, 2022).

## Acknowledgements

We extend our gratitude to Borgny Ytterhus for her contributions to this
project.

## Author contributions statement

SO created the proposed model and augmentation, conducted the experiments, and
wrote the main text. MV prepared the data, checked and confirmed the
annotations, conducted the qualitative assessments, and helped with the
histopathological aspects of the experiments. AP assisted with the technical
and programming aspects of the work. ES provided guidelines for programming
and technical aspects of the work. SGFD collected and quality-checked all
histopathology samples from the NLCB archive. VGD annotated the slides, helped
with the histopathological aspects of the experiments, and conducted the
qualitative assessments. MH reviewed and provided feedback on the paper. MDH
annotated the slides and helped with the histopathological aspects of the
experiments. TL Provided general guidelines for the experiments and methods.
MPR contributed in the preparation of the Bergen cohort, reviewed and provided
feedback on the paper. LAA contributed in the preparation of the Bergen
cohort, reviewed and provided feedback on the paper. GK provided guidelines
for experiments and technical aspects of the work. HS was project leader,
provided funding and supervision over the whole work and guidelines for the
clinical aspects of the work. All authors reviewed, revised, and approved the
manuscript.

## Data availability

The datasets generated and/or analysed during the current study are not
publicly available due to the sensitive nature of personal medical data from
patients who may still be alive.

## Conflict of Interest

There is no conflict of interest to report.