# A customizable approach to assess software quality through Multi-Criteria
Decision Making

Francesco Basciani  University of L’Aquila
L’Aquila, Italy
<EMAIL_ADDRESS>Daniele Di Pompeo  University of L’Aquila
L’Aquila, Italy
<EMAIL_ADDRESS>Juri Di Rocco  University of L’Aquila
L’Aquila, Italy
<EMAIL_ADDRESS>Alfonso Pierantonio  University of L’Aquila
L’Aquila, Italy
<EMAIL_ADDRESS>
###### Abstract

Over the years, Software Quality Engineering has increased interest,
demonstrated by significant research papers published in this area.
Determining when a software artifact is qualitatively valid is tricky, given
the impossibility of providing an objective definition valid for any
perspective, context, or stakeholder. Many quality model solutions have been
proposed that reference specific quality attributes in this context. However,
these approaches do not consider the context in which the artifacts will
operate and the stakeholder’s perspective who evaluate its validity.
Furthermore, these solutions suffer from the limitations of being artifact-
specific and not extensible.

In this paper, we provide a generic and extensible mechanism that makes it
possible to aggregate and prioritize quality attributes. The user, taking into
account his perspective and the context in which the software artifact will
operate, is guided in defining all the criteria for his quality model. The
management of these criteria is then facilitated through Multi-Criteria
Decision Making (MCDM). In addition, we present the PRETTEF model, a concrete
instance of the proposed approach for assessing and selecting MVC frameworks.

###### Index Terms:

Quality Model, Quality Assessment, Multi-Criteria Decision Making (MCDM)

##  I Introduction

In modern software products, the definition of quality is a critical and
complex process due to several properties that can be considered. Furthermore,
the selection and prioritization of these properties to measure are strictly
related to the context to which the software refers. In addition, stakeholders
can decide on preferences for these properties based on their feelings and
expertise.

Over the years, many studies have been conducted to provide valuable tools to
define the quality of software products [1, 2, 3, 4, 5]. Still, this multitude
of studies demonstrates how much the debate on establishing software quality
is a hot topic. Many of these studies offer reasonable solutions for some
stakeholders or specific contexts without considering the context in which the
stakeholders find themselves can change and their point of view, effectively
making these approaches useless due to their specificity.

This paper aims to provide a generic and extensible mechanism by which
assertions about the quality of software artifacts can be made. It extends [6,
7] by providing a tool to define a quality model for Multi-Criteria Decision
Making (MCDM) and managing criteria involved in the quality assessment of
software products. Furthermore, the expressiveness provided by the MCDM
approach allows for establishing priorities in a set of established criteria.

Indeed, introducing the concept of MCDM in a generic and extensible system,
such as the Quality Metamodel [7], provides a highly customizable tool that
aims to determine the quality of a software product according to stakeholders’
point of view.

We validate our framework on Model-View-Control (MVC) framework selection by
introducing the MCDM quality model extension. It is worth to remark technology
investment is crucial to keep IT teams updated with current trends. Therefore,
choosing the right software technologies depends on various quantitative and
qualitative factors. Indeed, our framework helps IT teams select the right MVC
framework by defining and extracting quality metrics. PREsentation, Trend,
TEchnology, and Features (PRETTEF) quality attributes have been combined to
select a suitable MVC framework that reflects the users’ needs.

In this respect, the contributions of this work are as follows:

  * •

an extension of the tool-supported approach defined in [6, 7] to support
(MCDM) [8, 9] assessments;

  * •

the PRETTEF model, a concrete instance of the extended approach for selecting
the suitable MVC web developing framework.

The structure of the paper is as follows: Section II presents some backgrounds
for our work. The proposed approach is illustrated in Section III. The next
section explains the evaluation settings used to study the approach’s
performance. Section IV presents the experimental results of applying our
approach to assess the best suitable web developing framework. Section V
reviews the related work and associates them with our approach. Finally,
Section VI outlines possible future work and draws some conclusions.

##  II Background

This section describes how we dealt whit the quality assessment of a software
artifact in our previous work [7]. Also, it provides a brief introduction to
MCDM approaches for selecting the most suitable alternative whit respect to a
set of criteria by highlighting the most relevant aspects.

###  II-A Quality models

Assessing the quality of software systems is underpinned by the definition of
several metrics that, once evaluated and properly aggregated, can give
insights into the quality of the analyzed system. One way to formulate a
software quality model is to identify a small set of high-level attributes and
decompose them into sets of subordinate attributes [6]. That approach is
generic and extensible, with a tool-chain enabling users to specify custom
quality models whose quality attributes are hierarchically aggregated. Their
evaluation depends on metrics specifically conceived and applied to the
artifacts to be analyzed and combined.

For this purpose, using a Quality Metamodel plays a key role since it enables
the specification of quality models according to user requirements. We based
our work on enhancing the approach proposed in [6, 7] and Fig. 1 shows how we
extended the Quality Metamodel. Specifically, the elements already present in
the starting Quality Metamodel are in the green area while all other elements
represent the extension provided.

Figure 1: The MCDM Quality Metamodel

As can be seen in the figure, the QualityModel represents the root element,
which consists of QualityAttributes, Values,ValueTypes, and MetricProviders. A
QualityAttribute represents a quality aspect considered relevant for
contributing to the quality assessment of a given artifact and it can be an
aggregation of other quality attributes. Thus, each quality attribute
specifies how to aggregate the contained attributes to provide overall quality
value for the considered attribute. Each quality attribute has a Value
representing its calculated Value that is abstract, and it can be SingleValue
or AggregatedValue. The metaclass SingleValue represents the Value obtained by
applying a given MetricProvider, which refers to the software component’s
ability to calculate specific metrics. AggregatedValue indicates a composition
of different values specified through an
OCL111https://projects.eclipse.org/projects/modeling.mdt.ocl expression
contained in the attribute body of the Operation element. Each Value has a
reference with a ValueType element, which is abstract and defines its type.
Several specializations are provided to enable the specifications of integer,
boolean or ranged values (e.g., from 0 to 5) for example (the other
specializations have been omitted but can be viewed in [6, 7]).

###  II-B MCDM

In different application domains, Multi-Criteria Decision-Making (MCDM) [8, 9]
approaches are used to make decisions when heterogeneous factors are involved
in the evaluation of many alternatives. Just to mention a few, economics [10,
11], medical [12, 13], and non-functional analysis [14, 15, 16] domains took
benefits by applying MCDM approaches.

The purpose of an MCDM is to assist the decision-maker in mastering the
different involved criteria and advancing toward selecting a possible
alternative. Among the factor that impacts the selection of suitable
alternatives, the definition of the priorities/weights on a set of criteria is
a key aspect of the MCDM approaches that strongly rely on subjective
evaluation. In a generic decision-making process, the following entities can
be identified:

  * •

_Objective_ : it defines the purpose for which the artifact assessment is
carried out; therefore, the result will have to get as close as possible to
the set goal.

  * •

_Decision-maker (DM)_ : he is the actor who has the task of defining the
selection criteria and the method of evaluating alternatives. In most cases,
it is not a single person but a group of people;

  * •

_Alternatives_ : the alternatives represent the space of solutions;

  * •

_Criteria_ : they represent the attributes on which the assessment is
performed. The criteria describe measurable characteristics;

  * •

_Assessment_ : it computes the ranked list of the alternatives consistent with
DM value judgments.

  * •

_Solution_ : it represents the alternative that achieves the best compromise
in relation to the objective.

Over the years, several approaches [17] have been developed to deal with MCDM
assessments, e.g., the SMARTER and SMARTS ones [18, 19]. Moreover, many hybrid
approaches [20, 21, 22] have been proposed to combine two or more existing
ones. The goal of hybrid MCDM approaches is to use different MCDM techniques
to combine the evaluation of different criteria. In the remainder of the
section, we focus on SMARTER and SMARTS since they underpin the approach
proposed in the next section where such approaches are combined with the
quality model proposed in the previous section.

####  II-B1 SMARTER

The Simple Multi-Attribute RaTing Exploiting Ranks (SMARTER) [23] technique
consists of a combined approach between the subjective elicitation of
preferences with an objective distribution of weights. This approach relies on
the following rationale: “The DM usually can rank items much more easily than
give weight to them”. Therefore, the elicitation phase does not provide for
the assignment of weights by the decision maker. In this approach, the
decision maker’s task is limited to ranking the criteria according to her
preferences. In contrast, the weight of the individual criteria are assigned
through predefined methods [24], i.e., Rank Order Centroid (ROC), Rank Sum
(RS), Rank Reciprocal (RR). These methods help the DM develop more consistent
and transparent weights according to their rank assignments. This method takes
those ranks as inputs and converts them to weights for each item.

Given a list $n$ of prioritized ranked criteria, where each criterion $j$ has
a rank $r_{j}$, $j\in(1,2,...,n)$ For each such method, a criterion weight
assessment $w_{j}$ is defined in the following:

$w_{j}(ROC)=\frac{1}{n}\sum\limits_{k=j}^{n}\frac{1}{r_{k}},j=1,2,...n$ (1)

$w_{j}(RR)=\frac{\frac{1}{r_{j}}}{\sum\limits_{k=1}^{n}\frac{1}{r_{k}}},j=1,2,...n$
(2)

$w_{j}(RS)=\frac{n-r_{j}+1}{\sum\limits_{k=1}^{n}n-r_{k}+1}=\frac{2(n+1-r_{j})}{n(n+1)},j=1,2,...n$
(3)

Once the weights have been assigned by one of the previous techniques, the
utilities of alternatives $A=\\{a_{1},a_{1},...,a_{m}\\}$ are evaluated by the
following formula:

$u(a_{i})=\sum\limits_{j=1}^{n}w_{j}*v_{j}(a_{i})$ (4)

where $n$ are the number of criteria, $w_{j}$ is the assigned weight for the
criterion $j$, and $v_{j}(a_{i})$ is the values of criteria $j$ in the
alternative $i$.

####  II-B2 SMARTS

The Simple Multi-Attribute Rating Swing Weight (SMARTS) [23] method drives the
DM in selecting an alternative. Unlike the SMARTER approach, the DM directly
assigns ratings to the criteria on a natural scale. Then, SMARTS uses swing-
weight techniques to select the most appropriate alternative. In particular,
given a $W=\\{w_{1},w_{2},...,w_{n}\\}$ DM ratings of the criterion, the
following formula normalizes those values:

$w_{j(norm)}=\frac{w_{j}}{\sum\limits_{k=1}^{n}w_{k}}$ (5)

Holding the weighting of criteria and rating of alternatives as separate as
possible, the different scales of criteria are converted to a common internal
scale. Then, for each alternative $a_{i}$ the utilities $u(a_{i})$ is computed
as following:

$u(a_{i})=\sum\limits_{j=1}^{n}w_{j(norm)}*sv_{j}(a_{i})$ (6)

where $sv_{j}(a_{i})$ is the scaled value of the criteria $j$ in the
alternative $i$.

##  III Proposed approach

This section presents a process for solving MCDM decision problems using an
extension of the quality model defined in [7, 6]. In the MDCM approaches, the
DM performs subjective (e.g., the selection of criteria and their weight) and
objective choices (i.e., the assessment of the alternatives) to achieve
predefined goals. MCDM can be seen as a tool for evaluating alternatives when
several criteria are considered to produce a common output. In previous
studies [7, 6], we presented a generic approach aiming at the custom
definition of quality models where hierarchically organized quality
attributes, whose evaluation depends on measurable aspects and applied on the
software artifacts to be analyzed, are therefore combined.

Both domains (i.e., quality models and MCDM) strongly rely on subjective
choices that the user underpins to perform qualitative assessments and
alternative evaluations. On the one hand, the quality models allow the modeler
to define their choices by a generic, extensible, and self-contained language.
On the other hand, the MCDM approach supports the DM in assessing alternatives
by well-defined and formal steps.

###  III-A Extended quality model notation

We extended the quality model notation and tools to native support MCDM
reasoning. First, we investigate possible mappings between the concepts
belonging to the two domains. The quality model lets the user define
single/aggregated attributes and the DM aggregates criteria. For instance,
hybrid MCDM approaches aggregate criterion by involving different MCDM
approaches to perform the alternative assessments, and the quality model
aggregates quality attributes during the model actualization. As defined in
Section II, the quality values are aggregated by user-defined operations, and
the quality model doesn’t provide any support in the definition of the
aggregation formula.

The DM defines the objectives and the list of alternatives, whereas the
quality model defines the criteria and the assessment settings. Then, the
quality model evaluation engine assesses the alternatives. Given an
alternative and an input quality model, the automated assessment step applies
the MCDM approaches to produce the score of the alternative. Such a score is
used to rank the alternatives. Finally, the ranked list of alternatives is
provided to the DM.

In this context, as we introduced in Sec. II, MCDM approaches have been
proposed (e.g., SMARTER, SMARTS, etc.) to support the DM with predefined
formulas in the alternative assessment.

Given that the Quality Model allows the user to define hierarchical structures
of quality attributes and how single measurement values have to be combined to
achieve overall quality statements, we aim to extend the quality model to
support hybrid MCDM approaches, where different criteria can be aggregated at
different levels.

The quality evaluation engine computes and aggregates the criteria in a
bottom-up manner. Figure 2 shows how different aggregating criteria can be
expressed with the quality model. In particular, the alternatives assessment
takes a bottom-up approach, starting from the elements at the bottom of the
hierarchy, where a leaf represents a measurable value. The hierarchy is
climbed up by aggregating the values with MCDM techniques. This process ends
when the root is reached. The root of the hierarchy stores and represents the
score assigned to an alternative.

Figure 2: Combining MCDM approaches with a hierarchical quality model
assessment.

To ensure that aggregate values could be defined through multiple criteria, as
you can see in Fig. 1 outside the gree area, the Quality Metamodel was
enhanced by introducing the MCDMAggregateValue metaclass as an extension of
Value.

This metaclass allows the introduction of the calculation of the MCDM. Unlike
the AggregatedValue, it calculates the aggregation of different Values through
MCDM techniques.

We, therefore, introduced SMARTS and SMARTER metaclasses as extensions to the
abstract MCDM metaclass:

  * •

in the SMARTER metaclass, it is possible to indicate the type of algorithm to
be used in the calculation through an enumerator, SMARTERAlgorithm), with the
values: RS, RR, and ROC. (see Sec. II-B1). The chosen algorithm will be
applied to the QualityAttributes to which this metaclass refers.

  * •

the algorithm used for SMARTS is unique and is always SWING (see Sec. II-B2).
As anticipated in Section II-B2 the SMARTER approach requires the DM directly
assigns ratings to the criteria in a natural scale. For this reason, an
instance of SMARTSWeight allows the DM define the ratings for each quality
attribute involved in the assessment. It is worth noting that the DM ratings
should map the involved quality attributes. For this reason, the metamodel
invariant (defined as an OCL statement in Figure 1) ensures that the size of
the score list is equal to the size of quality attributes.

Both approaches are represented by a metaclass (nominally SMARTER and SMARTS)
calculating the aggregate value, each through a specific process. More
precisely, SMARTER’s use requires indicating the algorithm type used for the
aggregation (SMARTERAlgorithm): RS, RR, or ROC. As a result, it returns the
QualityAttributes aggregation as an ordered list. With SMARTS, on the
contrary, there is no need to indicate the algorithm to be used as this is
implicit (SWING). Still, it is also necessary to provide at least one
initializer, represented by the metaclass SMARTInitializer to return the final
value given by the aggregation of at least one QualityAttribute.

###  III-B Extending the Quality model engine to support MCDM analysis

The support engine for multi-criteria analysis has been implemented by
providing built-in formulas defined in OCL. Therefore, this engine can
interpret these formulas without further modifications. In other words, the
system takes as input the quality model defined by the user, considers the
possible implementations of MCDM (i.e., SMART and SMARTER), and instantiates
them on the artifacts object of evaluation. This is possible thanks to the
fact that the quality model engine supports the computation of aggregated
values written using OCL. Consequently, it was possible to reuse the existing
calculation engine by focusing on the representation of SMARTER or SMART
formulas via OCL.

Figure 2 shows the hierarchical organization of a model using the MCDM
techniques. The engine, in a bottom-up manner, therefore, starting from the
Single Values represented by the leaves of the tree structure, aggregates all
the values up to providing a series of alternatives from which the decision
maker can then determine the solution.

##  IV Automated MCDM reasoning for selecting web development frameworks

This section presents an application of our approach in the selection process
of an MVC framework. In particular, we instantiate the quality model on a
concrete example of a user/company evaluating which MVC framework deserves
more attention. PREsentation, Trend, TEchnology, and Features (PRETTEF)
quality attributes have been combined to select a suitable MVC framework that
reflects the users’ needs.

Table I lists the 17 selected MVC frameworks that play as alternatives in the
MCDM analysis. Those frameworks are written in 7 programming languages (i.e.,
Java, PHP, JavaScript, Python, Ruby, and C#), where JavaScript is the most
used language. All the considered frameworks have more than 3,000 forks and
more than 900 pull requests. Figure 3 shows how the quality attributes and
metrics are hierarchically organized in the PRETTEF model to evaluate the
framework described above. Each node is a quality attribute with a
corresponding value type in the model. The PRETTEF model uses predefined value
types provided by the quality model, e.g., numerical, categorical, and boolean
values. For instance, Stack and Design Pattern are defined as categorical
values, Stars and Forks as numerical data, and Documentation as a boolean
value. To enable the assessment, the user/modeler has to assign numerical
scores to non-numeric values (e.g., categorical and boolean). For instance,
the current PRETTEF model uses 0.5, 0.5, and 1 for the back end, front end,
and full-stack, respectively.

Once the quality attributes are defined with their type, we apply SMARTER and
SMARTS MCDM techniques for each aggregation layer by instantiating the
corresponding SMARTER and SMARTS metaclasses. Both instances include the list
of quality attributes involved in the assessment. SMARTER includes the weight
specification algorithm, while SMARTS the list of weights. For instance, Table
II lists the adopted rank and scores for SMARTER and SMARTS approaches,
respectively. It is worth noting that we defined our weights and ranks in the
PRETTEF models. All the parameters, as well as the aggregation items, can be
changed according to the modeler’s needs. Once the modelers complete the
specification of the enhanced quality, each alternative is evaluated among all
the configurations.

TABLE I: Selected alternatives. Data are updated to Dec 2020 Framework | URL | language | Licence | #Forks | #Pull-Requests
---|---|---|---|---|---
Spring MVC | https://spring.io/projects/spring-framework | Java | Apache 2.0 | 20,550 | 3,393
Laravel | https://laravel.com | PHP | MIT | 16,685 | 3,433
Symfony | https://symfony.com | PHP | MIT | 7,105 | 20,470
Cake PHP | https://cakephp.org | PHP | MIT | 3,406 | 8,134
Code Igniter | https://codeigniter.com | PHP | MIT | 7,731 | 2777
AngularJs | https://angularjs.org | JavaScript | MIT | 28,888 | 7,881
React.JS | https://reactjs.org | JavaScript | MIT | 25,206 | 8,459
Express.js | https://expressjs.com | JavaScript | MIT | 7,576 | 904
Node.js | https://nodejs.org | JavaScript | ISC | 14,420 | 18,559
Ember.Js | https://emberjs.com | JavaScript | MIT | 4,169 | 8,631
Backbone.js | https://backbonejs.org | JavaScript | MIT | 5,687 | 1,831
Vue.js | https://vuejs.org | JavaScript | MIT | 21,547 | 1,597
Django | https://www.djangoproject.com | Python | Django o.s. | 18,870 | 11,713
Flask | https://flask.palletsprojects.com | Python | BSD-3C | 12,850 | 1,565
Rails | https://rubyonrails.org | Ruby | MIT | 17,751 | 24,042
AspNetCore | https://learn.microsoft.com/en-us/aspnet/core/ | C# | Apache 2.0 | 3,186 | 3,476
Bootstrap | https://getbootstrap.com | Javascript | MIT | 66,668 | 10,068
TABLE II: Rank and weights for Trend quality attribute aggregation QualityAttribute | Rank | Weight
---|---|---
GitHub Contributors | 3 | 0,111
GitHub Stars | 3 | 0,111
GitHub Pull Request | 2 | 0,166
Forks | 1 | 0,335
Release per Year | 3 | 0,111
Language | 2 | 0,167

Figure 4 shows how the RR, ROC, and RS algorithms have very similar trends:
there are differences in evaluation only for the last items in the alternative
rank. The SMART results are more subjective, and there is a more progressive
trend among the alternatives (that depends on the DM weights). It can also be
seen that the result of the first 8 alternatives coincides for all four
methods. At the same time, there is a different alternative rank between the
swing technique and the RR on the AngularJs and Laravel frameworks. In the
last 6 alternatives, ROC, RS, and RR trends fluctuate and correspond to
different alternatives assessments. Figure 5 shows that the evaluation of the
solutions is equivalent even using different techniques for calculating the
weights. The ranking of the first five alternatives is comparable, even using
various techniques. This evidence allows us to state that the methods based on
the ordering are similar and can be used according to the decision maker’s
preference. At the same time, the Swing Weights technique can radically change
the evaluation result. Therefore it is advisable to use the SMARTS method in
the presence of experts in the field of application of the investigation
context.

Figure 3: The PRETTEF model Figure 4: The alternative assessment Figure 5: Top
4 alternatives

It is worth noting that this section aims at providing a

##  V Related work

Baroni et al.222member of the QUASAR research team
https://sites.google.com/site/quasarresearchgroup/ introduced the Meta-Model
Driven Measurement (M2DM) that allows the specification of specific object-
oriented design metrics through OCL constructs [25]. The M2MD can be applied
to formalize the MOOD2 metrics over the GOODLY design language metamodel [25],
define object-oriented design metrics [26], express relational database schema
metrics [27], and assess components composition using the CORBA Components
Metamodel [28].

Abandah and Alsmadi showed that software metrics could be used in deficiency
prediction [29]. For example, software call graph models were extracted and
used to evaluate their ability to predict quality-related attributes in
developed software products. Specifically, the generated call graph models
represent and reflect the degree of complexity, especially in
understandability, testability, and product maintenance efforts. The extracted
metrics are investigated in relation to or in correlation with bugs collected
from reports. They are used as input to a data miner for classification,
prediction, and association analysis.

The ISO/IEC 9126 standard defines a quality model consisting of _factors_ ,
_criteria_ , and _metrics_. A factor (e.g., reliability or functional
capacity) is decomposed into _criteria_ (e.g., safety or modularity). Criteria
are decomposed in terms of _metrics_ computed on the source code.

The SQUALE project [30] extended the ISO/IEC 9126 standard by the _practice_
concept, which is an intermediate level between criteria and metrics. This
additional concept is exploited to assess if the considered project respects
technical principles, e.g., the fact that complex classes should be more
documented than trivial ones. Furthermore, the _practices_ concept defines new
practices (and eventually composes them) to assess the quality of a software
system by directly analyzing the source code through a supporting tool.
Finally, the SQUALE project introduced more than 50 practices collected in a
list that is not intended to be closed.

Siavvas et al. [31] presented the QATCH tool-chain to generate software
quality models that reflect user specification through static analysis.
Furthermore, QATCH generates quality models that exploit a weighted average
aggregation scheme to calculate the overall quality. Expert judgments can
define these weights.

A quality model specifically conceived to measure the quality of metamodels is
introduced in [1]. It allows for defining quality attributes, e.g.,
maintainability, portability, and usability, along with sub-characteristics,
e.g., analyzability, adaptability, and understandability.

EMF Metrics [32] supports the specification and calculation of metrics for
models developed atop the Eclipse Modeling Platform (EMF). It focuses on
metrics explicitly defined to measure the model’s structural characteristics.
For example, it supports the number of attributes, classes, and associations.

In this paper, we introduced a quality computational framework that helps
users compute their quality metrics on sever artifacts. Furthermore, we differ
from the above methods because we introduced an extensible framework, which is
also agnostic to the used modeling language.

##  VI Conclusion

In this work, we have defined a quality model (i.e., PRETTEF) integrated with
Multiple Criteria Decision Making (MCDM) for the assessment of selecting Web
Development frameworks based on software qualities. Furthermore, PRETTEF
allows a decision-maker to define and modify criteria and metrics to make the
analysis flexible. In fact, a decision maker can modify: i) the Attributes,
Metrics, and Weights calculation, ii) the elicitation techniques, and iii) the
data aggregation method.

Our approach eases the decision-making process and regulates the decision-
makers subjectivity. The proposed quality model combines classic MCDM methods,
e.g., SMARTER and SMARTS techniques. The quality model effectively formalizes
and structures the evaluation criteria or attributes.

## Acknologement

The authors would like to thank Andrea Serafini for his hard work on
supporting the PRETTEF use case.

## References

  * [1] F. Bertoa Manuel and A. Vallecillo, “Quality attributes for software metamodels,” in _QAAOSE@TOOLS_ , 2010.
  * [2] J. J. López-Fernández, E. Guerra, and J. De Lara, “Assessing the quality of meta-models.” in _MoDeVVa@ MoDELS_. Citeseer, 2014, pp. 3–12.
  * [3] V. Cortellessa, D. Di Pompeo, R. Eramo, and M. Tucci, “A model-driven approach for continuous performance engineering in microservice-based systems,” _J. Syst. Softw._ , vol. 183, p. 111084, 2022. [Online]. Available: https://doi.org/10.1016/j.jss.2021.111084
  * [4] Z. Ma, X. He, and C. Liu, “Assessing the quality of metamodels,” _Frontiers of Computer Science_ , vol. 7, no. 4, pp. 558–570, 2013.
  * [5] V. Cortellessa, D. Di Pompeo, V. Stoico, and M. Tucci, “On the impact of performance antipatterns in multi-objective software model refactoring optimization,” in _2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)_. IEEE, 2021, p. 224–233. [Online]. Available: https://ieeexplore.ieee.org/document/9582578/
  * [6] F. Basciani, J. d. Rocco, D. d. Ruscio, L. Iovino, and A. Pierantonio, “A customizable approach for the automated quality assessment of modelling artifacts,” in _2016 10th International Conference on the Quality of Information and Communications Technology (QUATIC)_ , Sept 2016, pp. 88–93.
  * [7] F. Basciani, J. Di Rocco, D. Di Ruscio, L. Iovino, and A. Pierantonio, “A tool-supported approach for assessing the quality of modeling artifacts,” _Journal of Computer Languages_ , vol. 51, pp. 173–192, 2019.
  * [8] U. e Habiba and S. Asghar, “A survey on multi-criteria decision making approaches,” in _2009 International Conference on Emerging Technologies_ , Oct. 2009, pp. 321–325.
  * [9] M. Aruldoss, T. M. Lakshmi, and V. P. Venkatesan, “A Survey on Multi Criteria Decision Making Methods and Its Applications,” 2013.
  * [10] E. K. Zavadskas and Z. Turskis, “Multiple criteria decision making (mcdm) methods in economics: an overview,” _Technological and economic development of economy_ , vol. 17, no. 2, pp. 397–427, 2011.
  * [11] G. Kou, Y. Peng, and G. Wang, “Evaluation of clustering algorithms for financial risk analysis using mcdm methods,” _Information Sciences_ , vol. 275, pp. 1–12, 2014.
  * [12] W. Sałabun and A. Piegat, “Comparative analysis of mcdm methods for the assessment of mortality in patients with acute coronary syndrome,” _Artificial Intelligence Review_ , vol. 48, no. 4, pp. 557–571, 2017.
  * [13] P. Ren, Z. Xu, H. Liao, and X.-J. Zeng, “A thermodynamic method of intuitionistic fuzzy mcdm to assist the hierarchical medical system in china,” _Information Sciences_ , vol. 420, pp. 490–504, 2017.
  * [14] A. Kumar, B. Sah, A. R. Singh, Y. Deng, X. He, P. Kumar, and R. Bansal, “A review of multi criteria decision making (mcdm) towards sustainable renewable energy development,” _Renewable and Sustainable Energy Reviews_ , vol. 69, pp. 596–609, 2017.
  * [15] M. Kabak and M. Dağdeviren, “Prioritization of renewable energy sources for turkey by using a hybrid mcdm methodology,” _Energy Conversion and Management_ , vol. 79, pp. 25–33, 2014.
  * [16] D. Di Pompeo and M. Tucci, “Search budget in multi-objective refactoring optimization: a model-based empirical study,” in _48th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2022_. IEEE, 2022, pp. 406–413. [Online]. Available: https://doi.org/10.1109/SEAA56994.2022.00070
  * [17] M. Aruldoss, T. M. Lakshmi, and V. P. Venkatesan, “A survey on multi criteria decision making methods and its applications,” _American Journal of Information Systems_ , vol. 1, no. 1, pp. 31–43, 2013.
  * [18] E. Triantaphyllou, “Multi-Criteria Decision Making Methods,” in _Multi-criteria Decision Making Methods: A Comparative Study_. Boston, MA: Springer US, 2000, pp. 5–21. [Online]. Available: https://doi.org/10.1007/978-1-4757-3157-6_2
  * [19] M. Velasquez and P. T. Hester, “An analysis of multi-criteria decision making methods,” _International journal of operations research_ , vol. 10, no. 2, pp. 56–66, 2013.
  * [20] S.-K. Hu, M.-T. Lu, and G.-H. Tzeng, “Exploring smart phone improvements based on a hybrid mcdm model,” _Expert Systems with Applications_ , vol. 41, no. 9, pp. 4401–4413, 2014.
  * [21] H.-Y. Wu, J.-K. Chen, I.-S. Chen, and H.-H. Zhuo, “Ranking universities based on performance evaluation by a hybrid mcdm model,” _Measurement_ , vol. 45, no. 5, pp. 856–880, 2012.
  * [22] G.-H. Tzeng, C.-H. Chiang, and C.-W. Li, “Evaluating intertwined effects in e-learning programs: A novel hybrid mcdm model based on factor analysis and dematel,” _Expert systems with Applications_ , vol. 32, no. 4, pp. 1028–1044, 2007.
  * [23] W. Edwards and F. H. Barron, “Smarts and smarter: Improved simple methods for multiattribute utility measurement,” _Organizational behavior and human decision processes_ , vol. 60, no. 3, pp. 306–325, 1994.
  * [24] E. Roszkowska, “Rank ordering criteria weighting methods – a comparative overview,” _Optimum.Studia Ekonomiczne_ , vol. Nr 5 (65, pp. 14–33, 01 2013\.
  * [25] A. L. Baroni, S. Braz, and F. B. e Abreu, “Using ocl to formalize object-oriented design metrics definitions,” in _ECOOP’02 Workshop on Quantitative Approaches in OO Software Engineering_ , 2002.
  * [26] A. Baroni, S. Braz, F. Brito, and Abreu, “A formal library for aiding metrics extraction,” in _Proceedings of ECOOP Workshop on Object-Oriented Re-Engineering_ , 2003.
  * [27] A. L. Baroni, C. Calero, F. B. e Abreu, and M. Piattini, “Object-relational database metrics formalization,” in _Sixth International Conference on Quality Software (QSIC 2006), 26-28 October 2006, Beijing, China_ , 2006, pp. 30–37.
  * [28] M. Goulão and F. B. e Abreu, “Formal definition of metrics upon the CORBA component model,” in _Quality of Software Architectures and Software Quality, First International Conference on the Quality of Software Architectures, QoSA 2005 and Second International Workshop on Software Quality, SOQUA 2005, Erfurt, Germany, September 20-22, 2005, Proceedings_ , 2005, pp. 88–105.
  * [29] H. Abandah and I. Alsmadi, “Call graph based metrics to evaluate software design quality,” _Int. J. of Software Engineering and Applications_ , vol. 7, no. 1, 2013.
  * [30] K. Mordal-Manet, F. Balmas, S. Denier, S. Ducasse, H. Wertz, J. Laval, F. Bellingard, and P. Vaillergues, “The Squale Model - A Practice-based Industrial Quality Model,” in _International conference on Software maintenance_ , Edmonton, Canada, Nov. 2009. [Online]. Available: https://hal.inria.fr/inria-00637364
  * [31] M. G. Siavvas, K. C. Chatzidimitriou, and A. L. Symeonidis, “QATCH – An adaptive framework for software product quality assessment,” _Expert Systems with Applications_ , vol. 86, pp. 350–366, 2017.
  * [32] T. Arendt, P. Stepien, and G. Taentzer, “EMF metrics: Specification and calculation of model metrics within the eclipse modeling framework,” in _of the BENEVOL workshop_ , 2010.


* Mahapatra et al. [2022] Mahapatra, D., Poellinger, A., Reyes, M., 2022.  Interpretability-guided inductive bias for deep learning based medical image.  Medical Image Analysis 81, 102551.  URL: https://www.sciencedirect.com/science/article/pii/S1361841522001980, doi:https://doi.org/10.1016/j.media.2022.102551.
  * Majurski et al. [2019] Majurski, M., Manescu, P., Padi, S., Schaub, N., Hotaling, N., Simon, C., Bajcsy, P., 2019.  Cell image segmentation using generative adversarial networks, transfer learning, and augmentations, in: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1114--1122.  doi:10.1109/CVPRW.2019.00145.
  * Marini et al. [2022] Marini, N., Marchesin, S., Otálora, S., Wodzinski, M., Caputo, A., van Rijthoven, M., Aswolinskiy, W., Bokhorst, J.M., Podareanu, D., Petters, E., Boytcheva, S., Buttafuoco, G., Vatrano, S., Fraggetta, F., van der Laak, J., Agosti, M., Ciompi, F., Silvello, G., Muller, H., Atzori, M., 2022.  Unleashing the potential of digital pathology data by training computer-aided diagnosis models without human annotations.  npj Digital Medicine 5, 102.  URL: https://doi.org/10.1038/s41746-022-00635-4, doi:10.1038/s41746-022-00635-4.
  * Mehta and Rastegari [2021] Mehta, S., Rastegari, M., 2021.  Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer.  arXiv preprint arXiv:2110.02178 .
  * Min et al. [2022] Min, K., Lee, G.H., Lee, S.W., 2022.  Attentional feature pyramid network for small object detection.  Neural Networks 155, 439--450.  URL: https://www.sciencedirect.com/science/article/pii/S089360802200329X, doi:https://doi.org/10.1016/j.neunet.2022.08.029.
  * Misra and van der Maaten [2020] Misra, I., van der Maaten, L., 2020.  Self-supervised learning of pretext-invariant representations, in: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6706--6716.  doi:10.1109/CVPR42600.2020.00674.
  * Mongan et al. [2020] Mongan, J., Moy, L., Kahn, C.E., 2020.  Checklist for artificial intelligence in medical imaging (claim): A guide for authors and reviewers.  Radiology: Artificial Intelligence 2, e200029.  URL: https://doi.org/10.1148/ryai.2020200029, doi:10.1148/ryai.2020200029, arXiv:https://doi.org/10.1148/ryai.2020200029.
  * Montezuma et al. [2023] Montezuma, D., Oliveira, S.P., Neto, P.C., Oliveira, D., Monteiro, A., Cardoso, J.S., Macedo-Pinto, I., 2023.  Annotating for artificial intelligence applications in digital pathology: A practical guide for pathologists and researchers.  Modern Pathology , 100086URL: https://www.sciencedirect.com/science/article/pii/S0893395222055260, doi:https://doi.org/10.1016/j.modpat.2022.100086.
  * Murtaza Dogar et al. [2021] Murtaza Dogar, G., Fraz, M., Javed, S., 2021.  Feature attention network for simultaneous nuclei instance segmentation and classification in histology images, Institute of Electrical and Electronics Engineers Inc.  URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85107670840&doi=10.1109%2fICoDT252288.2021.9441474&partnerID=40&md5=465c65c8c9c304f4861036ae250d82ab, doi:10.1109/ICoDT252288.2021.9441474.
  * Myronenko [2019] Myronenko, A., 2019.  3d mri brain tumor segmentation using autoencoder regularization, in: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (Eds.), Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, Springer International Publishing, Cham. pp. 311--320.  doi:https://doi.org/10.1007/978-3-030-11726-9_28.
  * Naseem et al. [2023] Naseem, U., Khushi, M., Kim, J., 2023.  Vision-language transformer for interpretable pathology visual question answering.  IEEE Journal of Biomedical and Health Informatics 27, 1681--1690.  doi:10.1109/JBHI.2022.3163751.
  * Naseer et al. [2021] Naseer, M.M., Ranasinghe, K., Khan, S.H., Hayat, M., Shahbaz Khan, F., Yang, M.H., 2021.  Intriguing properties of vision transformers, in: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 23296--23308.  URL: https://proceedings.neurips.cc/paper_files/paper/2021/file/c404a5adbf90e09631678b13b05d9d7a-Paper.pdf.
  * Nasir et al. [2022] Nasir, E.S., Parvaiz, A., Fraz, M.M., 2022.  Nuclei and glands instance segmentation in histology images: a narrative review.  Artificial Intelligence Review URL: https://doi.org/10.1007/s10462-022-10372-5, doi:10.1007/s10462-022-10372-5.
  * Nema et al. [2020] Nema, S., Dudhane, A., Murala, S., Naidu, S., 2020.  Rescuenet: An unpaired gan for brain tumor segmentation.  Biomedical Signal Processing and Control 55, 101641.  URL: https://www.sciencedirect.com/science/article/pii/S1746809419302228, doi:https://doi.org/10.1016/j.bspc.2019.101641.
  * Neto et al. [2023] Neto, P.C., Montezuma, D., Oliveira, S.P., Oliveira, D., Fraga, J., Monteiro, A., Monteiro, J., Ribeiro, L., Gonçalves, S., Reinhard, S., et al., 2023.  A cad system for colorectal cancer from wsi: A clinically validated interpretable ml-based prototype.  arXiv preprint arXiv:2301.02608 .
  * Neto et al. [2022] Neto, P.C., Oliveira, S.P., Montezuma, D., Fraga, J., Monteiro, A., Ribeiro, L., Gonçalves, S., Pinto, I.M., Cardoso, J.S., 2022.  imil4path: A semi-supervised interpretable approach for colorectal whole-slide images.  Cancers 14.  URL: https://www.mdpi.com/2072-6694/14/10/2489, doi:10.3390/cancers14102489.
  * Niazi et al. [2019] Niazi, M.K.K., Parwani, A.V., Gurcan, M.N., 2019.  Digital pathology and artificial intelligence.  The Lancet Oncology 20, e253--e261.  URL: https://www.sciencedirect.com/science/article/pii/S1470204519301548, doi:https://doi.org/10.1016/S1470-2045(19)30154-8.
  * Northcutt et al. [2021] Northcutt, C., Jiang, L., Chuang, I., 2021.  Confident learning: Estimating uncertainty in dataset labels.  J. Artif. Int. Res. 70, 1373–1411.  URL: https://doi.org/10.1613/jair.1.12125, doi:10.1613/jair.1.12125.
  * Oakden-Rayner et al. [2020] Oakden-Rayner, L., Dunnmon, J., Carneiro, G., Re, C., 2020.  Hidden stratification causes clinically meaningful failures in machine learning for medical imaging, in: Proceedings of the ACM Conference on Health, Inference, and Learning, Association for Computing Machinery, New York, NY, USA. p. 151–159.  URL: https://doi.org/10.1145/3368555.3384468, doi:10.1145/3368555.3384468.
  * Obeid et al. [2022] Obeid, A., Mahbub, T., Javed, S., Dias, J., Werghi, N., 2022.  Nucdetr: End-to-end transformer for nucleus detection in histopathology images, in: Qin, W., Zaki, N., Zhang, F., Wu, J., Yang, F. (Eds.), Computational Mathematics Modeling in Cancer Analysis, Springer Nature Switzerland, Cham. pp. 47--57.  doi:https://doi.org/10.1007/978-3-031-17266-3_5.
  * Ouyang et al. [2023] Ouyang, C., Chen, C., Li, S., Li, Z., Qin, C., Bai, W., Rueckert, D., 2023.  Causality-inspired single-source domain generalization for medical image segmentation.  IEEE Transactions on Medical Imaging 42, 1095--1106.  doi:10.1109/TMI.2022.3224067.
  * Pan et al. [2023] Pan, X., Cheng, J., Hou, F., Lan, R., Lu, C., Li, L., Feng, Z., Wang, H., Liang, C., Liu, Z., Chen, X., Han, C., Liu, Z., 2023.  Smile: Cost-sensitive multi-task learning for nuclear segmentation and classification with imbalanced annotations.  Medical Image Analysis 88, 102867.  URL: https://www.sciencedirect.com/science/article/pii/S1361841523001275, doi:https://doi.org/10.1016/j.media.2023.102867.
  * [209] Papers with Code: Cityscapes, .  Cityscapes test benchmark (instance segmentation).  URL: https://paperswithcode.com/sota/instance-segmentation-on-cityscapes. accessed: 01-08-2023.
  * [210] Papers With Code: COCO, .  Coco test-dev benchmark (instance segmentation).  URL: https://paperswithcode.com/sota/instance-segmentation-on-coco. accessed: 01-08-2023.
  * Pati et al. [2022] Pati, P., Jaume, G., Foncubierta-Rodríguez, A., Feroce, F., Anniciello, A.M., Scognamiglio, G., Brancati, N., Fiche, M., Dubruc, E., Riccio, D., Di Bonito, M., De Pietro, G., Botti, G., Thiran, J.P., Frucci, M., Goksel, O., Gabrani, M., 2022.  Hierarchical graph representations in digital pathology.  Medical Image Analysis 75, 102264.  URL: https://www.sciencedirect.com/science/article/pii/S1361841521003091, doi:https://doi.org/10.1016/j.media.2021.102264.
  * Paul and Chen [2022] Paul, S., Chen, P.Y., 2022.  Vision transformers are robust learners, in: Proceedings of the AAAI conference on Artificial Intelligence, pp. 2071--2081.
  * Pearl [2009] Pearl, J., 2009.  Causal inference in statistics: An overview .
  * Peters et al. [2016] Peters, J., Bühlmann, P., Meinshausen, N., 2016.  Causal inference by using invariant prediction: identification and confidence intervals.  Journal of the Royal Statistical Society Series B: Statistical Methodology 78, 947--1012.
  * Peters et al. [2017] Peters, J., Janzing, D., Schölkopf, B., 2017.  Elements of causal inference: foundations and learning algorithms.  The MIT Press.
  * Pineau et al. [2020] Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d’Alché Buc, F., Fox, E., Larochelle, H., 2020.  Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program).  J. Mach. Learn. Res. 22.
  * Pinto et al. [2022] Pinto, F., Torr, P.H.S., K. Dokania, P., 2022.  An impartial take to the cnn vs transformer robustness contest, in: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (Eds.), Computer Vision -- ECCV 2022, Springer Nature Switzerland, Cham. pp. 466--480.  doi:https://doi.org/10.1007/978-3-031-19778-9_27.
  * Prangemeier et al. [2020] Prangemeier, T., Reich, C., Koeppl, H., 2020.  Attention-based transformers for instance segmentation of cells in microstructures, in: 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 700--707.  doi:10.1109/BIBM49941.2020.9313305.
  * Priego-Torres et al. [2022] Priego-Torres, B.M., Lobato-Delgado, B., Atienza-Cuevas, L., Sanchez-Morillo, D., 2022.  Deep learning-based instance segmentation for the precise automated quantification of digital breast cancer immunohistochemistry images.  Expert Systems with Applications 193, 116471.  URL: https://www.sciencedirect.com/science/article/pii/S0957417421017516, doi:https://doi.org/10.1016/j.eswa.2021.116471.
  * Qin et al. [2022] Qin, J., He, Y., Zhou, Y., Zhao, J., Ding, B., 2022.  Reu-net: Region-enhanced nuclei segmentation network.  Computers in Biology and Medicine 146.  URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85129478540&doi=10.1016%2fj.compbiomed.2022.105546&partnerID=40&md5=646f80b2044c8775277691fc93c1bedd, doi:10.1016/j.compbiomed.2022.105546.
  * Qu et al. [2019] Qu, H., Yan, Z., Riedlinger, G.M., De, S., Metaxas, D.N., 2019.  Improving nuclei/gland instance segmentation in histopathology images by full resolution neural network and spatial constrained loss, in: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan, A. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2019, Springer International Publishing, Cham. pp. 378--386.  doi:https://doi.org/10.1007/978-3-030-32239-7_42.
  * Raghu et al. [2021] Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., Dosovitskiy, A., 2021.  Do vision transformers see like convolutional neural networks?, in: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 12116--12128.  URL: https://proceedings.neurips.cc/paper_files/paper/2021/file/652cf38361a209088302ba2b8b7f51e0-Paper.pdf.
  * Rao et al. [2021] Rao, A., Park, J., Woo, S., Lee, J.Y., Aalami, O., 2021.  Studying the effects of self-attention for medical image analysis, in: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 3409--3418.  doi:10.1109/ICCVW54120.2021.00381.
  * Ravin et al. [2008] Ravin, R., Hoeppner, D.J., Munno, D.M., Carmel, L., Sullivan, J., Levitt, D.L., Miller, J.L., Athaide, C., Panchision, D.M., McKay, R.D., 2008.  Potency and fate specification in cns stem cell populations in vitro.  Cell Stem Cell 3, 670--680.  URL: https://www.sciencedirect.com/science/article/pii/S1934590908004827, doi:https://doi.org/10.1016/j.stem.2008.09.012.
  * Raza et al. [2019] Raza, S.E.A., Cheung, L., Shaban, M., Graham, S., Epstein, D., Pelengaris, S., Khan, M., Rajpoot, N.M., 2019.  Micro-net: A unified model for segmentation of various objects in microscopy images.  Medical Image Analysis 52, 160--173.  URL: https://www.sciencedirect.com/science/article/pii/S1361841518300628, doi:https://doi.org/10.1016/j.media.2018.12.003.
  * Redmon et al. [2016] Redmon, J., Divvala, S., Girshick, R., Farhadi, A., 2016.  You only look once: Unified, real-time object detection, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779--788.  doi:10.1109/CVPR.2016.91.
  * Reinke et al. [2021] Reinke, A., Maier-Hein, L., Müller, H., 2021.  Common limitations of performance metrics in biomedical image analysis.  Proceedings of the Medical Imaging with Deep Learning (MIDL 2021) .
  * Reinke et al. [2023] Reinke, A., Tizabi, M.D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A.E., Rädsch, T., Sudre, C.H., Acion, L., Antonelli, M., Arbel, T., Bakas, S., Benis, A., Blaschko, M.B., Buettner, F., Cardoso, M.J., Cheplygina, V., Chen, J., Christodoulou, E., Cimini, B.A., Collins, G.S., Farahani, K., Ferrer, L., Galdran, A., VAN Ginneken, B., Glocker, B., Godau, P., Haase, R., Hashimoto, D.A., Hoffman, M.M., Huisman, M., Isensee, F., Jannin, P., Kahn, C.E., Kainmueller, D., Kainz, B., Karargyris, A., Karthikesalingam, A., Kenngott, H., Kleesiek, J., Kofler, F., Kooi, T., Kopp-Schneider, A., Kozubek, M., Kreshuk, A., Kurc, T., Landman, B.A., Litjens, G., Madani, A., Maier-Hein, K., Martel, A.L., Mattson, P., Meijering, E., Menze, B., Moons, K.G.M., Müller, H., Nichyporuk, B., Nickel, F., Petersen, J., Rafelski, S.M., Rajpoot, N., Reyes, M., Riegler, M.A., Rieke, N., Saez-Rodriguez, J., Sánchez, C.I., Shetty, S., Summers, R.M., Taha, A.A., Tiulpin, A., Tsaftaris, S.A., VAN Calster, B., Varoquaux, G., Yaniv, Z.R., Jäger, P.F., Maier-Hein, L., 2023.  Understanding metric-related pitfalls in image analysis validation.
  * Ren et al. [2015] Ren, S., He, K., Girshick, R., Sun, J., 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks, in: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.  URL: https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf.
  * Rolls and Farmer [2008] Rolls, G., Farmer, N., 2008.  Artifacts in Histological and Cytological Preparations.  Scientia leica Microsystems education series , 1--112URL: http://www.leica-microsystems.com/fileadmin/academy/2011/Artifacts{_}Handbook.pdf.
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., Brox, T., 2015.  U-net: Convolutional networks for biomedical image segmentation, in: Medical Image Computing and Computer-Assisted Intervention--MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, Springer. pp. 234--241.  doi:https://doi.org/10.1007/978-3-319-24574-4_28.
  * Ryu et al. [2023] Ryu, J., Puche, A.V., Shin, J., Park, S., Brattoli, B., Lee, J., Jung, W., Cho, S.I., Paeng, K., Ock, C.Y., et al., 2023.  Ocelot: Overlapped cell on tissue dataset for histopathology, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23902--23912.
  * Sahasrabudhe et al. [2020] Sahasrabudhe, M., Christodoulidis, S., Salgado, R., Loi, S., André, F., Paragios, N., Vakalopoulou, M., 2020.  Self-supervised nuclei segmentation in histopathological images using attention, in: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2020, Springer International Publishing, Cham. pp. 393--402.  doi:https://doi.org/10.1007/978-3-030-59722-1_38.
  * Salgado et al. [2015] Salgado, R., Denkert, C., Demaria, S., Sirtaine, N., Klauschen, F., Pruneri, G., Wienert, S., Van den Eynden, G., Baehner, F., Penault-Llorca, F., Perez, E., Thompson, E., Symmans, W., Richardson, A., Brock, J., Criscitiello, C., Bailey, H., Ignatiadis, M., Floris, G., Sparano, J., Kos, Z., Nielsen, T., Rimm, D., Allison, K., Reis-Filho, J., Loibl, S., Sotiriou, C., Viale, G., Badve, S., Adams, S., Willard-Gallo, K., Loi, S., 2015\.  The evaluation of tumor-infiltrating lymphocytes (tils) in breast cancer: recommendations by an international tils working group 2014.  Annals of Oncology 26, 259--271.  URL: https://www.sciencedirect.com/science/article/pii/S0923753419313699, doi:https://doi.org/10.1093/annonc/mdu450.
  * Schmidt et al. [2018] Schmidt, U., Weigert, M., Broaddus, C., Myers, G., 2018.  Cell detection with star-convex polygons, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer. pp. 265--273.  doi:https://doi.org/10.1007/978-3-030-00934-2_30.
  * Schölkopf et al. [2021] Schölkopf, B., Locatello, F., Bauer, S., Ke, N.R., Kalchbrenner, N., Goyal, A., Bengio, Y., 2021.  Toward causal representation learning.  Proceedings of the IEEE 109, 612--634.  doi:10.1109/JPROC.2021.3058954.
  * Shan et al. [2023] Shan, T., Yan, J., Cui, X., Xie, L., 2023.  Dsca-net: A depthwise separable convolutional neural network with attention mechanism for medical image segmentation.  Mathematical Biosciences and Engineering 20, 365--382.  URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85140914178&doi=10.3934%2fmbe.2023017&partnerID=40&md5=60a766589358704279749a6898d4d1b9, doi:10.3934/mbe.2023017.
  * Shao et al. [2021] Shao, Z., Bian, H., Chen, Y., Wang, Y., Zhang, J., Ji, X., et al., 2021.  Transmil: Transformer based correlated multiple instance learning for whole slide image classification.  Advances in neural information processing systems 34, 2136--2147.
  * Sheikh et al. [2022] Sheikh, T.S., Shim, J., Cho, M., 2022.  A multi-class nuclei segmentation model using attention modules, in: 2022 7th International Conference on Intelligent Informatics and Biomedical Science (ICIIBMS), pp. 213--217.  doi:10.1109/ICIIBMS55689.2022.9971499.
  * Shi et al. [2023] Shi, Y., Daunhawer, I., Vogt, J.E., Torr, P.H., Sanyal, A., 2023.  How robust is unsupervised representation learning to distribution shift?, in: The Eleventh International Conference on Learning Representations, OpenReview.
  * Sirinukunwattana et al. [2017] Sirinukunwattana, K., Pluim, J.P., Chen, H., Qi, X., Heng, P.A., Guo, Y.B., Wang, L.Y., Matuszewski, B.J., Bruni, E., Sanchez, U., Böhm, A., Ronneberger, O., Cheikh, B.B., Racoceanu, D., Kainz, P., Pfeiffer, M., Urschler, M., Snead, D.R., Rajpoot, N.M., 2017.  Gland segmentation in colon histology images: The glas challenge contest.  Medical Image Analysis 35, 489--502.  URL: https://www.sciencedirect.com/science/article/pii/S1361841516301542, doi:https://doi.org/10.1016/j.media.2016.08.008.
  * Sirinukunwattana et al. [2016] Sirinukunwattana, K., Raza, S.E.A., Tsang, Y.W., Snead, D.R.J., Cree, I.A., Rajpoot, N.M., 2016.  Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images.  IEEE Transactions on Medical Imaging 35, 1196--1206.  doi:10.1109/TMI.2016.2525803.
  * Soenksen et al. [2021] Soenksen, L.R., Kassis, T., Conover, S.T., Marti-Fuster, B., Birkenfeld, J.S., Tucker-Schwartz, J., Naseem, A., Stavert, R.R., Kim, C.C., Senna, M.M., Avilés-Izquierdo, J., Collins, J.J., Barzilay, R., Gray, M.L., 2021.  Using deep learning for dermatologist-level detection of suspicious pigmented skin lesions from wide-field images.  Science Translational Medicine 13, eabb3652.  doi:10.1126/scitranslmed.abb3652.
  * Song et al. [2023] Song, Z., Du, P., Yan, J., Li, K., Shou, J., Lai, M., Fan, Y., Xu, Y., 2023.  Nucleus-aware self-supervised pretraining using unpaired image-to-image translation for histopathology images.  IEEE Transactions on Medical Imaging , 1--1doi:10.1109/TMI.2023.3309971.
  * Srinidhi et al. [2021] Srinidhi, C.L., Ciga, O., Martel, A.L., 2021.  Deep neural network models for computational histopathology: A survey.  Medical Image Analysis 67, 101813.  URL: https://www.sciencedirect.com/science/article/pii/S1361841520301778, doi:https://doi.org/10.1016/j.media.2020.101813.
  * Srinidhi and Martel [2021] Srinidhi, C.L., Martel, A.L., 2021.  Improving self-supervised learning with hardness-aware dynamic curriculum learning: An application to digital pathology, in: 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 562--571.  doi:10.1109/ICCVW54120.2021.00069.
  * Srivastava et al. [2014] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., 2014.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research 15, 1929--1958.  URL: http://jmlr.org/papers/v15/srivastava14a.html.
  * Steiner et al. [2022] Steiner, A.P., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., Beyer, L., 2022.  How to train your vit? data, augmentation, and regularization in vision transformers.  Transactions on Machine Learning Research URL: https://openreview.net/forum?id=4nPswr1KcP.
  * Sugimoto et al. [2022] Sugimoto, T., Ito, H., Teramoto, Y., Yoshizawa, A., Bise, R., 2022.  Multi-class cell detection using modified self-attention, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1854--1862.  doi:10.1109/CVPRW56347.2022.00202.
  * Sun et al. [2021] Sun, P., He, J., Chao, X., Chen, K., Xu, Y., Huang, Q., Yun, J., Li, M., Luo, R., Kuang, J., et al., 2021.  A computational tumor-infiltrating lymphocyte assessment method comparable with visual reporting guidelines for triple-negative breast cancer.  EBioMedicine 70.
  * Tan and Le [2019] Tan, M., Le, Q., 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks, in: International conference on machine learning, PMLR. pp. 6105--6114.
  * Tellez et al. [2018] Tellez, D., Balkenhol, M., Karssemeijer, N., Litjens, G., van der Laak, J., Ciompi, F., 2018.  H and E stain augmentation improves generalization of convolutional networks for histopathological mitosis detection, in: Tomaszewski, J.E., Gurcan, M.N. (Eds.), Medical Imaging 2018: Digital Pathology, International Society for Optics and Photonics. SPIE. p. 105810Z.  URL: https://doi.org/10.1117/12.2293048, doi:10.1117/12.2293048.
  * Thiagarajan et al. [2019] Thiagarajan, J.J., Kashyap, S., Karargyris, A., 2019.  Distill-to-label: Weakly supervised instance labeling using knowledge distillation, in: 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pp. 902--907.  doi:10.1109/ICMLA.2019.00156.
  * Tourniaire et al. [2023] Tourniaire, P., Ilie, M., Hofman, P., Ayache, N., Delingette, H., 2023.  Ms-clam: Mixed supervision for the classification and localization of tumors in whole slide images.  Medical Image Analysis 85, 102763.  URL: https://www.sciencedirect.com/science/article/pii/S1361841523000245, doi:https://doi.org/10.1016/j.media.2023.102763.
  * Touvron et al. [2021a] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jegou, H., 2021a.  Training data-efficient image transformers & distillation through attention, in: International Conference on Machine Learning, pp. 10347--10357.
  * Touvron et al. [2021b] Touvron, H., Cord, M., El-Nouby, A., Bojanowski, P., Joulin, A., Synnaeve, G., Jégou, H., 2021b.  Augmenting convolutional networks with attention-based aggregation.  arXiv preprint arXiv:2112.13692 .
  * of Tumours Editorial Board [2019] of Tumours Editorial Board, W.C., 2019.  WHO Classification of Tumours, 5th Edition, Volume 2: Breast Tumours.  IARC Publications.
  * Ullah et al. [2023] Ullah, Z., Usman, M., Gwak, J., 2023.  Mtss-aae: Multi-task semi-supervised adversarial autoencoding for covid-19 detection based on chest x-ray images.  Expert Systems with Applications 216, 119475.  URL: https://www.sciencedirect.com/science/article/pii/S0957417422024940, doi:https://doi.org/10.1016/j.eswa.2022.119475.
  * Vahadane et al. [2021] Vahadane, A., B, A., Majumdar, S., 2021.  Dual encoder attention u-net for nuclei segmentation, in: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 3205--3208.  doi:10.1109/EMBC46164.2021.9630037.
  * Valanarasu et al. [2021] Valanarasu, J.M.J., Oza, P., Hacihaliloglu, I., Patel, V.M., 2021.  Medical transformer: Gated axial-attention for medical image segmentation, in: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2021, Springer International Publishing, Cham. pp. 36--46.  doi:https://doi.org/10.1007/978-3-030-87193-2_4.
  * van der Velden et al. [2022] van der Velden, B.H., Kuijf, H.J., Gilhuijs, K.G., Viergever, M.A., 2022.  Explainable artificial intelligence (xai) in deep learning-based medical image analysis.  Medical Image Analysis 79, 102470.  URL: https://www.sciencedirect.com/science/article/pii/S1361841522001177, doi:https://doi.org/10.1016/j.media.2022.102470.
  * Vanguri et al. [2022] Vanguri, R.S., Luo, J., Aukerman, A.T., Egger, J.V., Fong, C.J., Horvat, N., Pagano, A., Araujo-Filho, J.d.A.B., Geneslaw, L., Rizvi, H., Sosa, R., Boehm, K.M., Yang, S.R., Bodd, F.M., Ventura, K., Hollmann, T.J., Ginsberg, M.S., Gao, J., Vanguri, R., Hellmann, M.D., Sauter, J.L., Shah, S.P., Consortium, M.S.K.M.I.N.D., 2022.  Multimodal integration of radiology, pathology and genomics for prediction of response to pd-(l)1 blockade in patients with non-small cell lung cancer.  Nature Cancer 3, 1151--1164.  URL: https://doi.org/10.1038/s43018-022-00416-8, doi:10.1038/s43018-022-00416-8.
  * Varoquaux and Cheplygina [2022] Varoquaux, G., Cheplygina, V., 2022.  Machine learning for medical imaging: methodological failures and recommendations for the future.  npj Digital Medicine 5, 48.  URL: https://doi.org/10.1038/s41746-022-00592-y, doi:10.1038/s41746-022-00592-y.
  * Veličković [2023] Veličković, P., 2023.  Everything is connected: Graph neural networks.  Current Opinion in Structural Biology 79, 102538.
  * Verma et al. [2021] Verma, R., Kumar, N., Patil, A., Kurian, N.C., Rane, S., Graham, S., Vu, Q.D., Zwager, M., et al., 2021.  Monusac2020: A multi-organ nuclei segmentation and classification challenge.  IEEE Transactions on Medical Imaging 40, 3413--3423.  doi:10.1109/TMI.2021.3085712.
  * Vo and Kim [2023] Vo, V.T., Kim, S.H., 2023.  Mulvernet: Nucleus segmentation and classification of pathology images using the hover-net and multiple filter units.  Electronics (Switzerland) 12.  URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85146793207&doi=10.3390%2felectronics12020355&partnerID=40&md5=6258cac2926aa5af9a6711c8b2b81923, doi:10.3390/electronics12020355.
  * Vu et al. [2019] Vu, Q.D., Graham, S., Kurc, T., To, M.N.N., Shaban, M., Qaiser, T., Koohbanani, N.A., Khurram, S.A., Kalpathy-Cramer, J., Zhao, T., Gupta, R., Kwak, J.T., Rajpoot, N., Saltz, J., Farahani, K., 2019.  Methods for segmentation and classification of digital microscopy tissue images.  Frontiers in Bioengineering and Biotechnology 7.  URL: https://www.frontiersin.org/articles/10.3389/fbioe.2019.00053, doi:10.3389/fbioe.2019.00053.
  * Wagner and Rohr [2022] Wagner, R., Rohr, K., 2022.  Cellcentroidformer: Combining self-attention and convolution for cell detection, in: Yang, G., Aviles-Rivero, A., Roberts, M., Schönlieb, C.B. (Eds.), Medical Image Understanding and Analysis, Springer International Publishing, Cham. pp. 212--222.  doi:https://doi.org/10.1007/978-3-031-12053-4_16.
  * Wahab et al. [2022] Wahab, N., Miligy, I.M., Dodd, K., Sahota, H., Toss, M., Lu, W., Jahanifar, M., Bilal, M., Graham, S., Park, Y., Hadjigeorghiou, G., Bhalerao, A., Lashen, A.G., Ibrahim, A.Y., Katayama, A., Ebili, H.O., Parkin, M., Sorell, T., Raza, S.E.A., Hero, E., Eldaly, H., Tsang, Y.W., Gopalakrishnan, K., Snead, D., Rakha, E., Rajpoot, N., Minhas, F., 2022.  Semantic annotation for computational pathology: multidisciplinary experience and best practice recommendations.  The Journal of Pathology: Clinical Research 8, 116--128.  URL: https://pathsocjournals.onlinelibrary.wiley.com/doi/abs/10.1002/cjp2.256, doi:https://doi.org/10.1002/cjp2.256, arXiv:https://pathsocjournals.onlinelibrary.wiley.com/doi/pdf/10.1002/cjp2.256.
  * Wan et al. [2020] Wan, T., Zhao, L., Feng, H., Li, D., Tong, C., Qin, Z., 2020.  Robust nuclei segmentation in histopathology using asppu-net and boundary refinement.  Neurocomputing 408, 144--156.  URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85082497177&doi=10.1016%2fj.neucom.2019.08.103&partnerID=40&md5=e4205e8105f57102df65d0c900a6f138, doi:10.1016/j.neucom.2019.08.103.
  * Wan et al. [2023] Wan, Z., Li, M., Wang, Z., Tan, H., Li, W., Yu, L., Samuel, D.J., 2023.  Cellt-net: A composite transformer method for 2-d cell instance segmentation.  IEEE Journal of Biomedical and Health Informatics , 1--15doi:10.1109/JBHI.2023.3265006.
  * Wang et al. [2019] Wang, H., Wang, G., Sheng, Z., Zhang, S., 2019.  Automated segmentation of skin lesion based on pyramid attention network, in: Suk, H.I., Liu, M., Yan, P., Lian, C. (Eds.), Machine Learning in Medical Imaging, Springer International Publishing, Cham. pp. 435--443.  doi:https://doi.org/10.1007/978-3-030-32692-0_50.
  * Wang et al. [2022a] Wang, H., Xu, G., Pan, X., Liu, Z., Lan, R., Luo, X., 2022a.  Multi-task generative adversarial learning for nuclei segmentation with dual attention and recurrent convolution.  Biomedical Signal Processing and Control 75.  URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85124284705&doi=10.1016%2fj.bspc.2022.103558&partnerID=40&md5=79452a04cb5fc5cad2c21cb87949b1da, doi:10.1016/j.bspc.2022.103558.
  * Wang et al. [2023] Wang, J., Zhang, Z., Wu, M., Ye, Y., Wang, S., Cao, Y., Yang, H., 2023.  Nuclei instance segmentation using a transformer-based graph convolutional network and contextual information augmentation.  Computers in Biology and Medicine 167, 107622.  URL: https://www.sciencedirect.com/science/article/pii/S0010482523010879, doi:https://doi.org/10.1016/j.compbiomed.2023.107622.
  * Wang and Quan [2021] Wang, N., Quan, H., 2021.  Glunet: Global-local fusion u-net for 2d medical image segmentation, in: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (Eds.), Artificial Neural Networks and Machine Learning -- ICANN 2021, Springer International Publishing, Cham. pp. 74--85.  doi:https://doi.org/10.1007/978-3-030-86380-7_7.
  * Wang [2022] Wang, X., 2022.  Context understanding in computer vision: A survey.  SSRN Electronic Journal doi:10.2139/ssrn.4081162.
  * Wang et al. [2022b] Wang, X., Yang, S., Zhang, J., Wang, M., Zhang, J., Yang, W., Huang, J., Han, X., 2022b.  Transformer-based unsupervised contrastive learning for histopathological image classification.  Medical Image Analysis 81, 102559.  URL: https://www.sciencedirect.com/science/article/pii/S1361841522002043, doi:https://doi.org/10.1016/j.media.2022.102559.
  * Wazir and Fraz [2022] Wazir, S., Fraz, M.M., 2022.  Histoseg: Quick attention with multi-loss function for multi-structure segmentation in digital histology images, in: 2022 12th International Conference on Pattern Recognition Systems (ICPRS), pp. 1--7.  doi:10.1109/ICPRS54038.2022.9854067.
  * Wei et al. [2022] Wei, X., Bai, H., Zhang, X.S., Li, Y., 2022.  Deep pneumonia: Attention-based contrastive learning for class-imbalanced pneumonia lesion recognition in chest x-rays.  ArXiv abs/2207.11393.
  * Weigert and Schmidt [2022] Weigert, M., Schmidt, U., 2022.  Nuclei instance segmentation and classification in histopathology images with stardist, in: 2022 IEEE International Symposium on Biomedical Imaging Challenges (ISBIC), IEEE. pp. 1--4.
  * Wilkinson [2016] Wilkinson, M.D.e.a., 2016.  The fair guiding principles for scientific data management and stewardship.  Scientific Data 3, 160018.  URL: https://doi.org/10.1038/sdata.2016.18, doi:10.1038/sdata.2016.18.
  * Woo et al. [2018] Woo, S., Park, J., Lee, J.Y., Kweon, I.S., 2018.  Cbam: Convolutional block attention module, in: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (Eds.), Computer Vision -- ECCV 2018, Springer International Publishing, Cham. pp. 3--19.  doi:https://doi.org/10.1007/978-3-030-01234-2_1.
  * Wu et al. [2021] Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., Zhang, L., 2021.  Cvt: Introducing convolutions to vision transformers, in: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22--31.  doi:10.1109/ICCV48922.2021.00009.
  * Wu et al. [2023] Wu, Y., Zhou, Y., Saiyin, J., Wei, B., Lai, M., Shou, J., Fan, Y., Xu, Y., 2023.  Zero-shot nuclei detection via visual-language pre-trained models, in: Greenspan, H., Madabhushi, A., Mousavi, P., Salcudean, S., Duncan, J., Syeda-Mahmood, T., Taylor, R. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2023, Springer Nature Switzerland, Cham. pp. 693--703.
  * Xiao et al. [2021] Xiao, T., Singh, M., Mintun, E., Darrell, T., Dollar, P., Girshick, R., 2021.  Early convolutions help transformers see better, in: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 30392--30400.  URL: https://proceedings.neurips.cc/paper_files/paper/2021/file/ff1418e8cc993fe8abcfe3ce2003e5c5-Paper.pdf.
  * Xie et al. [2022a] Xie, C., Mousavian, A., Xiang, Y., Fox, D., 2022a.  Rice: Refining instance masks in cluttered environments with graph neural networks, in: Conference on Robot Learning, PMLR. pp. 1655--1665.
  * Xie et al. [2022b] Xie, C., Mousavian, A., Xiang, Y., Fox, D., 2022b.  Rice: Refining instance masks in cluttered environments with graph neural networks, in: Conference on Robot Learning, PMLR. pp. 1655--1665.
  * Xie et al. [2017] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K., 2017.  Aggregated residual transformations for deep neural networks, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987--5995.  doi:10.1109/CVPR.2017.634.
  * Xu et al. [2024] Xu, K., Goetz, L., Rajpoot, N., 2024.  On generalisability of segment anything model for nuclear instance segmentation in histology images.  arXiv preprint arXiv:2401.14248 .
  * Xu et al. [2022] Xu, R., Li, Y., Wang, C., Xu, S., Meng, W., Zhang, X., 2022.  Instance segmentation of biological images using graph convolutional network.  Engineering Applications of Artificial Intelligence 110, 104739.  URL: https://www.sciencedirect.com/science/article/pii/S0952197622000446, doi:https://doi.org/10.1016/j.engappai.2022.104739.
  * Xu et al. [2021] Xu, R., Wang, C., Xu, S., Meng, W., Zhang, X., 2021.  Dc-net: Dual context network for 2d medical image segmentation, in: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2021, Springer International Publishing, Cham. pp. 503--513.  doi:https://doi.org/10.1007/978-3-030-87193-2_48.
  * Yan et al. [2022] Yan, R., Ren, F., Li, J., Rao, X., Lv, Z., Zheng, C., Zhang, F., 2022.  Nuclei-guided network for breast cancer grading in he-stained pathological images.  Sensors 22.  URL: https://www.mdpi.com/1424-8220/22/11/4061, doi:10.3390/s22114061.
  * Yang et al. [2021] Yang, L., Zhang, R.Y., Li, L., Xie, X., 2021.  Simam: A simple, parameter-free attention module for convolutional neural networks, in: International conference on machine learning, PMLR. pp. 11863--11874.
  * Yang et al. [2020] Yang, Y., Li, H., Li, X., Zhao, Q., Wu, J., Lin, Z., 2020.  Sognet: Scene overlap graph network for panoptic segmentation.  Proceedings of the AAAI Conference on Artificial Intelligence 34, 12637--12644.  URL: https://ojs.aaai.org/index.php/AAAI/article/view/6955, doi:10.1609/aaai.v34i07.6955.
  * Yi et al. [2020] Yi, C., Chen, X., Quan, L., Lu, C., 2020.  Attentional dilated convolution neural network for nuclei segmentation in histopathology images, Institute of Electrical and Electronics Engineers Inc.. pp. 6737--6740.  URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85100947100&doi=10.1109%2fCAC51589.2020.9326630&partnerID=40&md5=6c25b5590a2604953244f41396514a8e, doi:10.1109/CAC51589.2020.9326630.
  * Yi et al. [2019] Yi, J., Wu, P., Jiang, M., Huang, Q., Hoeppner, D.J., Metaxas, D.N., 2019.  Attentive neural cell instance segmentation.  Medical Image Analysis 55, 228--240.  URL: https://www.sciencedirect.com/science/article/pii/S1361841518308442, doi:https://doi.org/10.1016/j.media.2019.05.004.
  * Yoo et al. [2019] Yoo, I., Yoo, D., Paeng, K., 2019.  Pseudoedgenet: Nuclei segmentation only with point annotations, in: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan, A. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2019, Springer International Publishing, Cham. pp. 731--739.  doi:https://doi.org/10.1007/978-3-030-32239-7_81.
  * Yoo et al. [2022] Yoo, J.J., Namdar, K., Wagner, M.W., Nobre, L., Tabori, U., Hawkins, C., Ertl-Wagner, B.B., Khalvati, F., 2022.  A novel gan-based paradigm for weakly supervised brain tumor segmentation of mr images.  arXiv preprint arXiv:2211.05269 .
  * Zbontar et al. [2021] Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S., 2021.  Barlow twins: Self-supervised learning via redundancy reduction, in: International Conference on Machine Learning, PMLR. pp. 12310--12320.
  * Zeng et al. [2019] Zeng, Z., Xie, W., Zhang, Y., Lu, Y., 2019.  Ric-unet: An improved neural network based on unet for nuclei segmentation in histology images.  IEEE Access 7, 21420--21428.  doi:10.1109/ACCESS.2019.2896920.
  * Zhang et al. [2022a] Zhang, C., Zhang, M., Zhang, S., Jin, D., Zhou, Q., Cai, Z., Zhao, H., Liu, X., Liu, Z., 2022a.  Delving deep into the generalization of vision transformers under distribution shifts, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7267--7276.  doi:10.1109/CVPR52688.2022.00713.
  * Zhang et al. [2022b] Zhang, D., Han, J., Cheng, G., Yang, M.H., 2022b.  Weakly supervised object localization and detection: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 5866--5885.  doi:10.1109/TPAMI.2021.3074313.
  * Zhang et al. [2020] Zhang, D., Zhang, H., Tang, J., Hua, X.S., Sun, Q., 2020.  Causal intervention for weakly-supervised semantic segmentation, in: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.. pp. 655--666.  URL: https://proceedings.neurips.cc/paper_files/paper/2020/file/07211688a0869d995947a8fb11b215d6-Paper.pdf.
  * Zhang et al. [2023a] Zhang, H., Liang, P., Sun, Z., Song, B., Cheng, E., 2023a.  Circleformer: Circular nuclei detection in whole slide images with circle queries and attention, in: Greenspan, H., Madabhushi, A., Mousavi, P., Salcudean, S., Duncan, J., Syeda-Mahmood, T., Taylor, R. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2023, Springer Nature Switzerland, Cham. pp. 493--502.
  * Zhang et al. [2021] Zhang, H., Liu, J., Yu, Z., Wang, P., 2021.  Masg-gan: A multi-view attention superpixel-guided generative adversarial network for efficient and simultaneous histopathology image segmentation and classification.  Neurocomputing 463, 275--291.  URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85113589295&doi=10.1016%2fj.neucom.2021.08.039&partnerID=40&md5=204acbc88155f422da17451e87962550, doi:10.1016/j.neucom.2021.08.039.
  * Zhang et al. [2023b] Zhang, J., Ma, K., Kapse, S., Saltz, J., Vakalopoulou, M., Prasanna, P., Samaras, D., 2023b.  Sam-path: A segment anything model for semantic segmentation in digital pathology, in: Celebi, M.E., Salekin, M.S., Kim, H., Albarqouni, S., Barata, C., Halpern, A., Tschandl, P., Combalia, M., Liu, Y., Zamzmi, G., Levy, J., Rangwala, H., Reinke, A., Wynn, D., Landman, B., Jeong, W.K., Shen, Y., Deng, Z., Bakas, S., Li, X., Qin, C., Rieke, N., Roth, H., Xu, D. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2023 Workshops, Springer Nature Switzerland, Cham. pp. 161--170.
  * Zhang et al. [2022c] Zhang, X., Zhu, X., Tang, K., Zhao, Y., Lu, Z., Feng, Q., 2022c.  Ddtnet: A dense dual-task network for tumor-infiltrating lymphocyte detection and segmentation in histopathological images of breast cancer.  Medical image analysis 78, 102415.
  * Zhang et al. [2023c] Zhang, Y., Zhou, T., Wang, S., Liang, P., Zhang, Y., Chen, D.Z., 2023c.  Input augmentation with sam: Boosting medical image segmentation with segmentation foundation model, in: Celebi, M.E., Salekin, M.S., Kim, H., Albarqouni, S., Barata, C., Halpern, A., Tschandl, P., Combalia, M., Liu, Y., Zamzmi, G., Levy, J., Rangwala, H., Reinke, A., Wynn, D., Landman, B., Jeong, W.K., Shen, Y., Deng, Z., Bakas, S., Li, X., Qin, C., Rieke, N., Roth, H., Xu, D. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2023 Workshops, Springer Nature Switzerland, Cham. pp. 129--139.
  * Zhao et al. [2022a] Zhao, J., He, Y.J., Zhao, S.Q., Huang, J.J., Zuo, W.M., 2022a.  Al-net: Attention learning network based on multi-task learning for cervical nucleus segmentation.  IEEE Journal of Biomedical and Health Informatics 26, 2693--2702.  URL: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85122078106&doi=10.1109%2fJBHI.2021.3136568&partnerID=40&md5=57fd0211fc1dfbb00ea6d47b2f7c2559, doi:10.1109/JBHI.2021.3136568.
  * Zhao et al. [2022b] Zhao, J., Hou, X., Pan, M., Zhang, H., 2022b.  Attention-based generative adversarial network in medical imaging: A narrative review.  Computers in Biology and Medicine 149, 105948.  URL: https://www.sciencedirect.com/science/article/pii/S0010482522006837, doi:https://doi.org/10.1016/j.compbiomed.2022.105948.
  * Zhao et al. [2023] Zhao, S., Chen, D.P., Fu, T., Yang, J.C., Ma, D., Zhu, X.Z., Wang, X.X., Jiao, Y.P., Jin, X., Xiao, Y., Xiao, W.X., Zhang, H.Y., Lv, H., Madabhushi, A., Yang, W.T., Jiang, Y.Z., Xu, J., Shao, Z.M., 2023.  Single-cell morphological and topological atlas reveals the ecosystem diversity of human breast cancer.  Nature Communications 14, 6796.  URL: https://doi.org/10.1038/s41467-023-42504-y, doi:10.1038/s41467-023-42504-y.
  * Zhou et al. [2022] Zhou, K., Liu, Z., Qiao, Y., Xiang, T., Loy, C.C., 2022.  Domain generalization: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence , 1--20doi:10.1109/TPAMI.2022.3195549.
  * Zhou et al. [2020] Zhou, X., Cheng, Z., Gu, M., Chang, F., 2020.  Lirnet: Local integral regression network for both strongly and weakly supervised nuclei detection, in: 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 945--951.  doi:10.1109/BIBM49941.2020.9313265.
  * Zhou et al. [2019a] Zhou, Y., Chen, H., Xu, J., Dou, Q., Heng, P.A., 2019a.  Irnet: Instance relation network for overlapping cervical cell segmentation, in: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.T., Khan, A. (Eds.), Medical Image Computing and Computer Assisted Intervention -- MICCAI 2019, Springer International Publishing, Cham. pp. 640--648.  doi:https://doi.org/10.1007/978-3-030-32239-7_71.
  * Zhou et al. [2019b] Zhou, Y., Graham, S., Alemi Koohbanani, N., Shaban, M., Heng, P.A., Rajpoot, N., 2019b.  Cgc-net: Cell graph convolutional network for grading of colorectal cancer histology images, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops.
  * Zhou et al. [2023] Zhou, Y., Li, W., Yang, G., 2023.  Scts: Instance segmentation of single cells using a transformer-based semantic-aware model and space-filling augmentation, in: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5933--5942.  doi:10.1109/WACV56688.2023.00589.
  * Zhou et al. [2019c] Zhou, Y., Onder, O.F., Dou, Q., Tsougenis, E., Chen, H., Heng, P.A., 2019c.  Cia-net: Robust nuclei instance segmentation with contour-aware information aggregation, in: Chung, A.C.S., Gee, J.C., Yushkevich, P.A., Bao, S. (Eds.), Information Processing in Medical Imaging, Springer International Publishing, Cham. pp. 682--693.  doi:https://doi.org/10.1007/978-3-030-20351-1_53.
  * Zhou et al. [2018] Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N., Liang, J., 2018.  Unet++: A nested u-net architecture for medical image segmentation, in: Stoyanov, D., Taylor, Z., Carneiro, G., Syeda-Mahmood, T., Martel, A., Maier-Hein, L., Tavares, J.M.R., Bradley, A., Papa, J.P., Belagiannis, V., Nascimento, J.C., Lu, Z., Conjeti, S., Moradi, M., Greenspan, H., Madabhushi, A. (Eds.), Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer International Publishing, Cham. pp. 3--11.  doi:https://doi.org/10.1007/978-3-030-00889-5_1.
  * Zhou [2018] Zhou, Z.H., 2018.  A brief introduction to weakly supervised learning.  National science review 5, 44--53.
  * Zhu et al. [2016] Zhu, Z., Liang, D., Zhang, S., Huang, X., Li, B., Hu, S., 2016.  Traffic-sign detection and classification in the wild, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2110--2118.  doi:10.1109/CVPR.2016.232.
  * Zimmerer et al. [2018] Zimmerer, D., Kohl, S.A., Petersen, J., Isensee, F., Maier-Hein, K.H., 2018.  Context-encoding variational autoencoder for unsupervised anomaly detection.  arXiv preprint arXiv:1812.05941 .
  * Zunair and Ben Hamza [2021] Zunair, H., Ben Hamza, A., 2021.  Sharp u-net: Depthwise convolutional network for biomedical image segmentation.  Computers in Biology and Medicine 136, 104699.  URL: https://www.sciencedirect.com/science/article/pii/S0010482521004935, doi:https://doi.org/10.1016/j.compbiomed.2021.104699.

  *[H&E]: Haematoxylin and Eosin
  *[WSI]: Whole Slide Image
  *[WSIs]: Whole Slide Image
  *[AI]: Artificial Intelligence
  *[CPath]: computational pathology
  *[CV]: Computer Vision
  *[ML]: Machine Learning
  *[DL]: Deep Learning
  *[SSL]: Self-supervised Learning
  *[WSL]: Weakly Supervised Learning
  *[SmSL]: Semi-supervised Learning
  *[RoIs]: Region of Interest
  *[ViT]: Vision Transformer
  *[ViTs]: Vision Transformer
  *[TILs]: Tumour-Infiltrating Lymphocyte
  *[TIL]: Tumour-Infiltrating Lymphocyte
  *[IHC]: Immunohistochemical
  *[DA]: Domain Adaptation
  *[DG]: Domain Generalization
  *[OOD]: out of distribution
  *[IID]: independent and identically distributed
  *[IoU]: Intersection over Union
  *[AJI]: Aggregated Jaccard Index
  *[PQ]: Panoptic Quality
  *[mAP]: Mean Average Precision
  *[AUC]: Area Under the Curve
  *[DC]: Dice Similarity Coefficient
  *[HD]: Hausdorff Distance
  *[dH]: directed Hausdorff Distance
  *[ConvNets]: Convolutional Neural Network
  *[NC]: Nuclear Classification
  *[HoVer]: Horizontal-Vertical
  *[NP]: Nuclear Prediction
  *[NF]: Nuclear Foreground
  *[NT]: Nuclear Type
  *[NO]: Nuclear Ordinal Regression
  *[DDD]: Decreasing Distance Discretization
  *[SPN]: Semantic Proposal Network
  *[IEN]: Instance Encoding Network
  *[NMS]: Non-Maximum Suppression
  *[IAM]: Information Aggregation Module
  *[MSDU]: Multi-scale Dense Units
  *[CRC]: Colorectal Cancer
  *[SE]: Squeeze and Excitation
  *[MLP]: Multilayer Perceptron
  *[ReLU]: Rectified Linear Unit
  *[CBAM]: Convolutional Block Attention Module
  *[FPN]: Feature Pyramid Network
  *[AFPN]: Attention Feature Pyramid Network
  *[DTA]: Dynamic Texture Attention
  *[FACA]: Foreground Aware Co-Attention
  *[DCA]: Detail Context Attention
  *[RoI]: Region of Interest
  *[NLP]: Natural Language Processing
  *[DeiT]: Data Efficient Image Transformer
  *[FPNs]: Feature Pyramid Network
  *[DETR]: Detection Transformer
  *[ConvNet]: Convolutional Neural Network
  *[RPN]: Region Proposal Network
  *[DC-Net]: Dual Context Network
  *[ACF]: Adaptive Context Fusion
  *[FTN]: Fully Transformer Network
  *[GAN]: Generative Adversarial Network
  *[CT]: Computerized Tomography
  *[MRI]: Magnetic Resonance Imaging
  *[IRNet]: Instance Relation Network
  *[CL]: Curriculum Learning
  *[CEM]: Context Enhancement Module
  *[CWM]: Confidence-based Weighting Module
  *[KD]: Knowledge Distillation
  *[MIL]: Multiple Instance Learning
  *[SL]: Strongly Supervised Learning
  *[RCCA]: Recurrent Criss-cross Attention
  *[LSTM]: Long Short-term Memory
  *[SCM]: Structural Causal Model
  *[MLP-Head]: Multi-layer Perceptron Head
  *[GNNs]: Graph Neural Network
  *[GNN]: Graph Neural Network
  *[MPN]: Message Passing Network
  *[GANs]: Generative Adversarial Network
  *[VAE]: Variational Autoencoder
  *[SAM]: Segment Anything
  *[SRM]: Semantic Relation Module
  *[ANNs]: Artificial Neural Network