#  The Evolution of Astronomical Observatory Design

Miguel Ángel Castro Tirado  Instituto de Astrofísica de Andalucía (IAA-CSIC),
Glorieta de la Astronomía s/n, 18008 Granada, Spain;  Unidad Asociada al CSIC
Departamento de Ingeniería de Sistemas y Automática, Escuela de Ingeniería
Industrial, Universidad de Málaga, Arquitecto Francisco Peñalosa, 29071
Málaga, Spain;  Alberto J. Castro-Tirado  Instituto de Astrofísica de
Andalucía (IAA-CSIC), Glorieta de la Astronomía s/n, 18008 Granada, Spain;
Unidad Asociada al CSIC Departamento de Ingeniería de Sistemas y Automática,
Escuela de Ingeniería Industrial, Universidad de Málaga, Arquitecto Francisco
Peñalosa, 29071 Málaga, Spain<EMAIL_ADDRESS><EMAIL_ADDRESS>
(Received May 4, 2019; accepted June 11, 2019)

##  1 Introduction

The theoretical approach to astronomical observatories has its origin in the
roots of the modern observatory (Brahe, 1602; Caramuel de Lobkowitz, 1678).
However, historical-descriptive works that consider a particular project are
the most common during the next centuries. In addition, some studies appeared
focused on specific periods (Sayili, 1960; Kwan, 2012; Leverington, 2017),
others with a more general retrospective nature (Donnelly, 1971; Krisciunas,
1999; Wolfschmidt, 2008) and only a few works focused on the architectural-
constructive aspects of the observatories (Dumitrache & Dumitrache, 2009;
Waumans, 2013).

The concept of astronomical observatory itself lacks a precise and globally
accepted definition given that historically there has not been a critical-
intellectual movement that theorized about its nature or conception. Although
it may seem obvious, what is understood by an observatory has changed over the
years, and, in addition, it depends on the perspective from which it is
valued. In this paper, the intention is to present an approach from the point
of view of the architecture of the building itself, so that other aspects such
as its astronomical equipment or its possible scientific usage will be set
aside. That is to say, we will study the particularities related to the
construction in which the instruments are located and whose purpose is the
astronomical function. The analysis will also be limited to optical
observatories.

Talking about evolution necessarily implies the extension of the study over
time to analyze and assess how, when and why the changes that have defined the
development of these astronomical centers have occurred. For this reason,
following an earlier work (Castro Tirado, 2019a), starting from the
examination of plans, ruins and the buildings themselves, this study goes back
to the genesis of the observatories trying to illustrate the reciprocal
relationship that exists between the design strategies of the building and the
practice of its astronomical function, with the goal of theorizing about their
future development. We have focused on 32 observatories which we consider to
be representative ones in the period 1259–2007, deeply analyzing the
architectural drafts.

##  2 Origins

Astronomy is one of the oldest scientific disciplines of mankind. Since man
raised his gaze to the sky and contemplated the stars with his own eyes, a
relationship of fascination and devotion was established. This would be
reflected in one way or another in different cultures that would arise across
the planet. And even without having tools of precision or a basic
understanding of the physical laws that rule the functioning of the universe,
the first civilizations already had certain notions of the main astronomical
facts. All based, always, on mere observations.

In any case, it is not possible to determine the date on which astronomy
arises, although its roots probably extend to the origins of the human
societies. What does seem clear is that, regardless of a spiritual or pseudo-
religious sense, the beginnings of this science point towards an
instrumentalization of the sky as a system for measuring periods of time and
as a mean of orientation. Beyond the numerous artistic representations of the
Sun, the Moon and the stars that have been found (Pannekoek, 1961; Pasztor &
Roslund, 2007), sometimes more realistic and others more allegorical or
figurative, or of the written testimonies that have lasted over time, this
astronomical predilection remains through the incidence of some celestial
phenomena in the disposition of certain archaeological sets.

Although certain remains of ancient civilizations show an impact of astronomy
on their culture, it will be the proto-observatories that stand out as points
of interest to know the way in which these peoples approached the practice of
this science. In these spaces the astronomical observatories have their
deepest antecedents and roots. Although the astronomical principles that
underlie the constitution of these assemblies are undoubted, there is no
evidence to support the existence of research or scientific work in these
places.

The oldest site that has been recorded so far is the Goseck Circle, in
Germany, a solar proto-observatory with an annular plan that consists of a
Neolithic structure dating back almost 7000 years (Brown, 2016; Scherrer,
2018). Of a similar period is the megalithic group of Nabta Playa, in Egypt.
There are many other cases of lesser antiquity, among which the temple of
Mnajdra, in Malta, or the famed Stonehenge, in England (Scherrer, 2018). All
these sites belong to ancient societies, with a low degree of technology and
even, frequently, of nomadic habits. These facts propitiate relatively simple
identifications of their constructions. In addition, the scale and materials
of these structures favor their durability up to the present, allowing them to
be studied and identified, where relevant, as proto-observatories.

However, the evolution towards more advanced civilizations tend to create
settlements of multiple buildings that do not reach the precise degree of
specialization for the observation function as such. Since no scientific
instruments had been developed that required demanding conditions, systematic
surveying of the sky could be carried out with the naked eye in almost any
construction or, even, in the vicinity of these out in the open.

The oldest building strictly dedicated to astronomy of which there is evidence
is Cheomseongdae, in South Korea. It was designed as a small bottle-shaped
hollow tower made of masonry. Its function was to enable the use of
astronomical instruments from an elevated position to avoid surrounding
obstacles. It dates from the seventh century and remains in a great state of
preservation (Park, 2010) and its square top and round body reflect the
astronomical concept of the time (i.e. the Earth was considered a square and
the sky was considered to have a round shape) with the 29 layers of stone from
the bottom to the top corresponding to the 29.5 days of full lunation (or
synodic month, i.e. the time from one new moon to the next) and the 27 layers
of the round body representing the 27.3-day sidereal month, amongst other
astronomical symbolism (Park, 2008).

Anyway, although references to the Tower of Babel or to some type of
astronomical institution directed by Ptolemy in Alexandria can be found
(Mundt, 1927; George, 2005), there is no evidence to support the existence as
such of pre-medieval astronomical observatories.

##  3 The Medieval Islamic Observatory

Coinciding with the beginning of the Middle Ages, a period of cultural
darkness began in the Western world from which astronomy would not escape.
During this period, Islamic culture came to play an essential role in the
history of astronomy by preserving the vast Greek knowledge of this subject
and incorporating its own findings. All this would be recovered for the West
by the School of Translators of Toledo of Alfonso X, during the thirteenth
century (Gargatagli, 1999).

The medieval Islamic observing sites can indeed be called observatories
because of two main reasons: i) the fact that they were spaces dedicated
specifically to the collective and prolonged study of celestial phenomena and
ii) to the exchange of scientific knowledge. It can be said that the
observatory, as such, appears for the first time in this civilization. This is
not a coincidence; the features of the Islamic culture as a society link with
the observatory as a scientific institution (Sayili, 1960). However, although
this permanence is implemented in the Islamic observatory, they would have a
reduced life (Sayili, 1960).

Two singularities that would ultimately be decisive for the relevance of the
Islamic observatory as an institution were the size of the observing tools and
its patronage. The fact that the tools of observation and study reached a
dimension such as to prevent their portability led to a necessary settlement.
Royal or state support meant the formation of institutions larger and longer
lived than an individual scientist (Sayili, 1980).

Although several observing posts, more or less temporary, had previously been
established, it is difficult to speak of observatories as an institution
before Al Mamûn (ninth century). The numerous observations collected by
different authors allow us to recognize centers established in Shammasiya,
Baghdad and Mount Qasiyun, Damascus (Mujani et al., 2012). However, the fact
that there are no remains, descriptions or illustrations of observatories
could mean that no specific space was devoted to this use. This lack of
material legacy makes it extremely difficult to distinguish between the
observatories of this period and the specific observing posts established for
an ephemeral study. The main differences amongst medieval islamic
observatories were in the number of personnel dedicated to them, the quality
of observing equipment and the life span of these centers. That is to say, at
this initial moment the observatories were not a clearly defined institution
and it is complicated to indicate which observation sites transcended the
barrier between the most advanced temporary work post and the embryonic
scientific center. It would be from this stage onward that the Islamic
observatory began to develop as a specialized astronomical institution linked
to a scientific team and a specific location. It then begins to be structured
with its own administration around a work program and its instruments become
more precise, larger and heavier.

All these circumstances suppose a change in the paradigm of the Islamic
observatory that, in a hesitant process over several centuries, transcends
from a semi-improvised observation post to the scientific reference center of
its time. The construction of the observatory of Maragheh, in northwest Iran,
was completed in 1264 and is part of the period that represents the climax of
the evolution of the Islamic observatory (Saliba, 1987). In fact, the main
progress and features of this institution in the Islamic world appear with
this observatory, which became the paradigm to imitate of its time.

Although only the ruins of the construction remain, these together with the
existing testimonies (Sayili, 1960) and the subsequent observatories built in
its image, allow a description of the building to be developed. The
observatory was a circular wall erected around a large central space occupied
by the large quadrant for meridian transit observations. On both sides of this
area there would be spaces to house the library, workshops, stores and other
functions distinct from mere observation. Even though the date of the fall of
the observatory at Maragheh is not exactly known, it still became the model to
imitate by the great Islamic observatories that would arise during the
following centuries, such as those of Samarkand and Istanbul.

Figure 1: Lower or access levels of the case studies considered for the
analysis of the programmatic-functional configuration in its evolution from
the Islamic-medieval observatory to the contemporary one. The scale is the
same in order to allow direct comparison. (Drawings by M. A. Castro-Tirado)

##  4 The Modern Observatory

Although some references attribute the first European observatory to either
Regiomontanus (Dreyer, 1890; Gunther, 1932) or Copernicus (Zinner, 1943),
there is no evidence of a construction or of fixed instruments that support
these claims (Zinner, 1943). There is a consensus to place the first modern
observatory in the palace of Wilhelm IV of Hesse-Kassel (this is recognized by
Todd 1922; Pannekoek 1961 or Pantin 1999). However, a platform built as a
lookout point in the palace where portable instruments are taken out (Gaulke,
2009) does not represent an advance from the perspective of architecture. This
does not invalidate its importance since it was a visit to these facilities
that inspired Tycho Brahe, a few months later, to build the first two modern
observatories (Zinner, 1956) in Hven characterized by having a certain degree
of intention and architectural specialization for astronomy.

The first project of Tycho Brahe would be Uraniborg (1576), a palace in which
the study of alchemy and astronomy would coexist with his own home. A dialogue
between science and architecture in which the spatial configuration would
adapt to the needs of research resulted in a solution that relegated alchemy
to the basement and elevated astronomy to the upper floor, leaving the work
and residence areas in the middle floor. Elevated platforms were built on
slender poles from which it was possible to look at the sky without obstacles
with covers that protected both the instruments and observers from poor
weather conditions.

After several years Tycho Brahe noticed that the vibration of the platforms
was being transmitted to the instruments and damaging the research
unacceptably, so he planned a second center in which neither structural nor
thermal instability would compromise astronomy. Following this idea
Stjerneborg (1584) arose as a building excavated in the earth in which the
observation tools would be fixed on pillars founded on the bedrock itself.

The appearance of the telescope in the early seventeenth century and its early
connection with astronomy would condition the future design of observatories,
in which the increase in accuracy would require improvements to avoid
architecture from becoming a brake on scientific research. However, since the
first telescopes were small and manageable, many astronomers could make their
observations from the windows, balconies, terraces or gardens of their own
residence, or from public spaces. Possibly, this is why, during this period,
the development of astronomical centers tends towards simple structures or,
directly, to adding platforms to existing buildings from which to get better
views of the sky, as in the universities of Leiden (1632) and Ingolstadt
(1637) or in the Danish Rundetårn (1637) (Donnelly, 1971). Therefore, many
years would pass between the appearance of the first modern observatory and
the consolidation of the idea that a center for scientific observation
required a building expressly designed for that purpose.

The observatory of Paris (1667) was created with the support of Louis XIV as a
national institution in a context of increasing interest for culture and
science. In addition to its scientific purpose, the project was to accommodate
the headquarters of the Academy of Sciences and to enhance the prestige of its
promoter. The resulting building is a palace where the institutional function
shares space with the residences and the areas dedicated to astronomy:
observation rooms, a zenith study room in the basement and a flat walkway for
instruments. For more than a century, observatories followed the example of
the French, limited to more or less contrived buildings that combined
residence, work area and spaces where telescopes could operate. This is how
the high observation rooms with vertical windows were reproduced, such as at
the Greenwich Observatory (1675) or the openings from north to south were
implemented at the Radcliffe Observatory (1772).

It was not until the construction of the Dunsink Observatory (1785) that
significant changes were assimilated, where the architectural configuration
was defined by the astronomical instruments that were to be installed. The
person responsible for this project defined the key elements of the building
as ”situation, foundation and soil” (Ussher, 1787). So the location was
selected for its good visibility and accessibility, and the importance of
stability meant the construction of foundations for the instruments to become
structurally independent of the rest of the building. Systems were also
developed to promote thermal stability in observation rooms. The design even
reserves a privileged position to the telescopes, placing them under movable
domes. The modern observatory is designed and planned to respond to a specific
astronomical use during the period between the seventeenth and eighteenth
centuries. The new observatories implement changes derived from the
development of astronomy itself as a science (especially in relation to its
instruments) and improve its performance for the sake of an optimal
performance of the research activity without underestimating the welfare of
its occupants.

Sometimes for strategic purposes (because of its importance for maritime
navigation), sometimes for cultural ambition and in others, perhaps, by
imitation, these buildings began to emerge throughout Europe in the eighteenth
century, from where they would end up spreading to the rest of the world.
Spaces for observation, characterized by the instruments that occupied them,
coexisted with spaces in which the activities derived from these observations
would take place (calculation, data analysis, discussion of results …),
secondary ones (preparation of instruments, offices, classrooms, libraries …),
and others corresponding to service uses (residences, storage areas …) that,
on occasion, moved to one or more independent buildings. Likewise, the
scientific and technological progress of this period had an impact on the
observatories: on the one hand the buildings incorporated new functions such
as physics or chemistry laboratories, photographic work rooms or optical and
mechanical workshops, while on the other hand the work of astronomers was
facilitated by mechanizing the movement of domes and telescopes. In addition,
technical advances allowed a growth of the refractor that led to a race to
having the largest telescope in the world. This struggle reached its peak at
the Yerkes Observatory (1897) where the size of the refractor (1.02 m
diameter) entailed an increase in the architectural scale of the building
(Hussey, 1897).

Although in this period the trend led to the observatories to move away from
the city and, therefore, the town, some centers would still emerge, such as
the Allegheny (1912) or the Griffith (1925), which would open to the public
offering the use of some instruments and other activities to disseminate and
promote astronomy.

Table 1: Percentages of surface areas of observatory buildings dedicated to different functions. Observatory | Area (m2) | Observation (%) | Derivatives (%) | Accessories (%)
---|---|---|---|---
Maragheh (1259) | 1164 | 4 | 96 | 0
Samarkand (1424) | 3082 | 3 | 97 | 0
Uraniborg (1576) | 1180 | 9 | 22 | 69
Stjerneborg (1584) | 91 | 71 | 29 | 0
Paris (1667) | 3152 | 48 | 8 | 44
Greenwich (1675) | 501 | 25 | 4 | 71
Cádiz (1753) | 1364 | 33 | 39 | 28
Madrid (1790) | 618 | 27 | 14 | 59
Radcliffe (1772) | 643 | 36 | 32 | 32
Dusink (1783) | 802 | 21 | 27 | 52
Seeberg (1790) | 473 | 13 | 4 | 83
Göttingen (1816) | 1092 | 16 | 16 | 68
Hamburgo (1825) | 627 | 18 | 19 | 63
Cape of Good Hope (1828) | 355 | 26 | 3 | 61
Helsinki (1834) | 1097 | 23 | 16 | 61
Harvard (1839) | 744 | 29 | 20 | 51
Pulkovo (1839) | 1429 | 44 | 18 | 38
U.S. Naval in Washington (1844) | 367 | 26 | 40 | 34
Lisboa (1861) | 893 | 40 | 10 | 50
Yerkes (1897) | 2396 | 28 | 40 | 32
Allegheny (1912) | 969 | 44 | 26 | 30
Lick (great refractor, 1881) | 957 | 43 | 32 | 25
Niza (large equatorial, 1881) | 1040 | 43 | 43 | 14
Mount Wilson (60-inch, 1908) | 975 | 52 | 48 | 0
Mount Wilson (100-inch, 1917) | 2509 | 51 | 49 | 0
Hamburg-Bergedorf (large refractor, 1912) | 562 | 32 | 56 | 12
Palomar (Hale telescope, 1948) | 1443 | 28 | 63 | 9
Lick (Shane reflector, 1959) | 2108 | 33 | 60 | 7
SAO (BTA-6m, 1975) | 4480 | 33 | 60 | 7
Calar Alto (3.5m, 1975) | 3709 | 18 | 74 | 8
Keck (1993) | 3873 | 49 | 47 | 4
Gran Telescopio Canarias (2007) | 3457 | 24 | 70 | 6

##  5 The Contemporary Observatory

The development of cities from the nineteenth century began to cause problems
in some nineteenth-century observatories, which not only began to withdraw
from large cities to smaller settlements but also to seek locations far from
urban areas. This withdrawal became increasingly necessary in the following
decades, underlining a physical distance from the ordinary citizen that also
led to an intellectual disconnection from astronomers until the press/media
started to recover the connection amongst them in the 1960s. Astronomical
centers with spaces such as conference halls, classrooms or exhibitions, open
to visitors, amateurs and experts, or even public observatories, lost
prominence in favor of highly specialized scientific complexes almost
exclusively for astronomy.

The search for the highest possible precision from the instruments resulted in
optimization of the observation conditions, which led to the establishment of
these institutions in areas not affected by anthropogenic (light) pollution
and with more favorable atmospheric conditions. The Lick Observatory (1881)
was the first continuously inhabited mountain observatory in the world
(Smiley, 1938; Misch & Stone, 1998). The location of the center was
fundamentally based on its astronomical advantages, without regard of the
distance to any city or university (Neubauer, 1950). The architectural
characteristics of the building are more like those of modern observatories
than other contemporary observatories, which is why the Lick Observatory
represents an intermediate, transitional case between the two epochs.

Almost simultaneously, the Nice Observatory (1881) emerged as a set of
independent structures scattered over the top of a mountain. Each building was
designed and located for a specific function, including the first building
devoted entirely to a telescope. In this way, it pioneered the contemporary
observatory layout of a set of specialized and independent buildings (Holden,
1891; Etienne et al., 2014; Hünsch et al., 2012).

The size of the new large reflectors, the differences in scale, the different
needs intrinsic to the multiple uses or the orography of the mountainous
terrain that makes it difficult to have wide level soils, are aspects that
drive the segregation of the initially unified observatory building into a
series of independent structures. This separation allows for a freedom of
disposition and orientation in the territory that will favor the conditions of
observation, minimizing interference (vibrations, smoke, light, heating) that
would compromise the quality of the astronomical observation.

Although mirror-based telescopes had been installed in observatories for
decades, their role had always been secondary to that of large refractors.
However, their importance increased thanks to technical advances that enabled
the manufacture of the required mirrors in parallel with the development of
astrophysics. In addition, the dissociation of the observatory buildings
allowed the adaptation of the constructions to the instruments they housed
without external constraints. An example of this is the Mount Wilson
Observatory (1904), initially harboring the large 60-inch reflector and,
later, in the 100-inch Hooker telescope (Adams, 1954).

Ignoring the heterogeneous large-scale layout of the modern observatory, the
contemporary design for an individual instrument is generally a base
architectural structure scaled to the instrument it contains, topped by a
movable cover, generally a dome. The size of telescopes reached such
magnitudes that it rivaled the very scale of the architecture that had to
contain it. This meant increasing the size of the buildings that housed the
telescopes, producing a large amount of contained space available for other
uses. Similarly, these large instruments require major structural supports, to
such an extent that spaces inside the supports themselves can accommodate some
functions.

Following the earlier Hooker telescope development, the Hale Telescope (1948)
ended up consolidating different functions and hosting almost a thousand
people during its opening ceremony. In this way, these large telescopes would
incorporate or could incorporate enough secondary functions (offices, work
canteens, meeting rooms, laboratories, warehouses, workshops, rest rooms,
toilets …) to be able to contain practically everything required for an entire
observatory. Thus, the construction planned for each instrument would have
sufficient autonomy to constitute an astronomical observatory by itself.

Figure 2: Fractions of the surface areas allocated to research support spaces
in the observatories in our sample.

From this point on, the most important observatories were constituted as sets
of independent structures where, habitually, they would include buildings for
some accessory uses (residences, visitor areas …), other structures allocated
to secondary instruments, and the main buildings housing the great telescopes
together with the primary functions (meeting rooms, work rooms, laboratories,
offices …), which would constitute the essence of the observatory.

The constant growth of the mirrors caused mechanical-structural problems in
the BTA-6 (1975) (with a 6-meter primary mirror) that forced to introduce some
fundamental changes in its design, adopting an alt-azimuth instead of an
equatorial mount. This variation complicated the tracking of celestial
objects, requiring a computerized control system. These two revolutionary
advances were consolidated and replicated in later observatories (Leverington,
2017). After the largest monolithic mirrors (8.2-meter diameter) were produced
for the European Southern Observatory’s Very Large Telescope (VLT, first light
in 1998) (Schilling, 1998) and Subaru (8.2 m), and the Large Binocular
Telescope (8.4 m) later on, the planners of the W. M. Keck Observatory
(1993/1996) decided to design the primary mirror as a combination of smaller
mirrors to make ever-increasing mirror sizes possible. Correctly aligning
segmented mirrors at that time represented a very hard problem. Because of
this, for VLT a single-mirror approach was selected. Keck did indeed have
control problems for years, vindicating the VLT decision. However, the
experience of fixing Keck’s problems paved the way for the future larger
segmented mirrors telescopes.

The alt-azimuth design allowed for a significant reduction of the structures,
which could be light, and the size of the domes, which would also reduce their
cost (Finn, 1985) compared to equatorially mounted telescopes. The race to the
largest reflector paved the way to the Gran Telescopio Canarias (GTC, first
light in 2007), with effective aperture of 10.4 m (composed of 36 segmented
mirrors of 1.8 m diameter each) and inspired by the Keck. However, other
projects are already underway that will exceed it in size but without
significant changes in its configuration: the GMT (24.5m), the TMT (30m) or
the ELT (39m).

Despite the changes experienced during this period, or even from the very
origin of this institution, the importance of technological development in
terms of computerization and communication systems must be highlighted as the
causes of the greatest observatory revolution. Both the remote control and the
remote access to the observations mark a decisive turning point regarding
these complexes, since it seems to offer the possibility of abandoning all
physical presence in the observatories. And although it is not necessary for
an astronomer to look through a lens to access an observation, there are still
a series of tasks derived from it that require a space in which scientists and
engineers can work.

##  6 Space-function Analysis

Beyond this summary of the development of the observatory up to the present,
anticipating the future evolution needs some analysis. Since it is difficult
to evaluate all the variables that affect the constitution of these
astronomical centers, this study focuses on the configuration of its
architecture in relation to its uses, considering the evolution of astronomy
and astronomical observing as a function of time. For this purpose, a
classification is established for the different activities for which the
different spaces of the buildings that house the instruments are intended.
Observation spaces are considered to be those places occupied by active
observation equipment, that is, they can be in operation if the environmental
conditions make it possible. Research support spaces are those space that,
though not directly participate in observations, provide functions necessary
for astronomical work. Included in this category are work rooms, workshops,
instrument warehouses, and so on. Finally, all of the dependencies that are
not essential for astronomical research, such as conference rooms, rest areas,
visitor galleries or exhibitions, are described as service spaces.

Based on this classification, we selected a representative sample (Figure 1)
of astronomical centers that had the greatest impact and influence on other
observatories of their time and afterwards. They are grouped according to the
periods described in the previous sections and ordered chronologically and
summarized in Table 1, which is based on the analysis of the plans and drafts
of the different observatories together with information on how various spaces
were used.

Figure 3: Fractions of the surface areas dedicated to service spaces in the
observatories in our sample.

Our work shows how the aspects defined in the different periods have their
corresponding manifestation in this classification. This highlights the weak
correlation between the Islamic observatory and the modern observatory. In
addition, during the beginnings of the modern observatory (between the
Uraniborg and Dunsink cases) there is a period of continuous changes in the
distribution of uses, probably related to the different types of
instrumentation and observing techniques used. However, since its
consolidation (since Seeberg) a tendency has emerged regarding the development
of its organization that extends to the current contemporary observatory.

On the one hand, Figure 2 shows a tendency toward an increase of the fraction
occupied by research support spaces over time. That means a higher proportion
of area in which astronomers and technicians can actually work (offices,
laboratories, workshops …), which suggests how the complexity of the
instruments attached to the telescopes has increased as time went on. It
underscores the growing importance of these functions for achieving results in
research.

On the other hand, as shown in Figure 3, the fraction occupied by service
spaces decreases with time. This can be due to a greater specialization of
these buildings as they suppress some functions not strictly astronomical
(rest areas, cafeterias, exhibition halls or visitors centers) which may be
relegated to secondary structures.

An important reason for the scattered or segregated pattern of modern
observatories is the addition of new projects after the initial foundation.
Good sites are rare, getting water, electricity and roads to that site may be
the single most expensive part of the project. By comparison, adding further
instruments to an established site is often a minor expense. Also, in terms of
project planning, existing observatories have a science-grade record of the
local weather, eliminating the need for possibly several years of site
monitoring to establish whether a new proposed site is good enough.

##  7 The Forthcoming Observatory

Since its inception, the astronomical observatory has reflected a continuous
architectural adaptation of the building to the needs of astronomy. Thus,
aspects such as structural stability, thermal stability, orientation of the
construction, the specific configuration of windows, the adaptation to the
different astronomical instruments, the generation of service and comfort
spaces for the users of the observatory, the incorporation of new functions
derived from the advances of science, the change of scale according to the
growth of the instruments, the adaptation to non-urban geographic environments
or the conciliation with new technologies have been addressed.

Obviously the current observatories, of the beginning of the 21st century, are
vastly different from their modern or Islamic-medieval ancestors. Nor are they
the same as those of the mid-twentieth century, even though the influence of
the Hale Telescope has spread through all the astronomical centers that
succeeded it. In any case, this evolution is not taking place in an
intentional or planned way, but the changes take place in an organic and
almost improvised way. This, which is not a problem in itself, does denote a
lack of deep understanding regarding observatories, understood as buildings in
which astronomical research and human activity coexist and relate through
architecture. All this means that occasionally some of the different functions
of the observatory come into conflict with each other, indicating a certain
lack of criticism and reflection. It is difficult to answer clearly and
unequivocally where astronomical observatories are heading, since there does
not seem to be a roadmap that points in a clear direction, due to the fact
that technological advances are unpredictable.

##  8 Conclusions

Although the selection of astronomical observatories analyzed in this work
could include a larger sample, it would not significantly change the result
since the architecture and configuration of the buildings was frequently
replicated with few variations.

In any case, our investigation has found evidence for two opposite directions
of evolution:

  * •

continuation of the dissociation of astronomical instruments from other
functions related to research;

  * •

return to heterogeneous uses of space in the same building such that the needs
of its users can match the astronomical activity.

The first of these possibilities would involve a split of what is now known as
an observatory. The result is a pair of buildings, one of which houses all the
spaces and dependencies necessary for the staff to do their research
(laboratories, offices, etc.), all the elements that could result in better
comfort for workers (cafeteria, dining room, rest areas, etc.). In addition,
this property should also house all kinds of technical rooms (control rooms,
workshops …) or accessories (facilities, warehouses …) that would serve the
astronomical function. The second building, separate and independent, would be
completely unrelated to human activity (for example, the electricity supply).

The second possibility would involve unifying in a single building both the
strictly intrinsic functions of astronomical research and the rest of the uses
and services associated with the workers of the center or even visitors. For
this, it would be necessary to carefully consider and correct the possible
interference and incompatibilities between the different uses. This would
result in a rather complex single construction.

When planning a new observatory in an area where other research centers are
concentrated, it would be reasonable to choose a structure for the new
telescope that would be separated from a place for workers and other users
that could even be shared with other facilities and neighbors. However, opting
for a unitary building would be adequate if the new observatory considers
aspects beyond pure research (such as teaching or public outreach) or if the
project were to be implemented in a remote and isolated place, saving
resources and optimizing supply lines.

Overall it seems reasonable that the two opposite directions of evolution can
coexist in the near future where they could represent the most appropriate
solutions in different scenarios; see also Castro Tirado (2019b).

###### Acknowledgements.

We have profited from valuable discussions with Ronan Cunniffe and Gemma
Olmedo Paradas. We are grateful to Luis Alberto Rodríguez and Antonio Cabrera-
Lavers (Gran Telescopio Canarias), Cristina Rodríguez (Observatorio de Sierra
Nevada), Santos Pedraz (Calar Alto Observatory) and Tatyana Sokolova (Special
Astrophysical Observatory) for providing the requested material and pictures
of the astronomical observatories they are related to. We also acknowledge
very useful comments from the two anonymous referees. M. A. C. T. has been
supported by a Junta de Andalucía predoctoral grant under project TIC-2839.
Finally, we also acknowledge financial support from the State Agency for
Research of the Spanish MCIU through the ”Center of Excellence Severo Ochoa”
award for the Instituto de Astrofísica de Andalucía (SEV-2017-0709).

## References

  * Adams (1954) Adams, W. S. 1954, The Founding of The Mount Wilson Observatory, PASP, 66, 267
  * Brahe (1602) Brahe, T. 1602, Astronomiae Instauratae Mechanica, Nuremberg
  * Brown (2016) Brown, D. 2016, An Introductory View on Archaeoastronomy, J. Phys. Conf. Ser., 685, 2
  * Caramuel de Lobkowitz (1678) Caramuel de Lobkowitz, J. 1678, Architectura Civil Recta y Obliqua (Vigevano: Imprenta Obispal)
  * Castro Tirado (2019a) Castro Tirado, M. A. 2019a, Astronomical Observatories: From the Prehistory to the XVIIIth Century, RMxAA Conf. Ser. 51, 1
  * Castro Tirado (2019b) Castro Tirado, M. A. 2019b, El Observatorio Astronómico: Un Diálogo entre Ciencia y Arquitectura, Ph.D. Thesis, University of Málaga
  * Donnelly (1971) Donnelly, M. C. 1971, A Short History of Observatories (Eugene: University of Oregon Books)
  * Dreyer (1890) Dreyer, J. L. E. 1890, Tycho Brahe (Edinburgh: A. & C. Black)
  * Dumitrache & Dumitrache (2009) Dumitrache, C. & Dumitrache, D. 2009, Architectural Evolution of Astronomical Observatories, Romanian Astron. J., 19, 1
  * Etienne et al. (2014) Etienne, C. et al. 2014, L’Observatoire de la Côte d’Azur et sa Mission de Diffusion de la Culture Scientifique et Technique, infos-patrimoinespaca.org
  * Finn (1985) Finn, R. 1985, The WM Keck Observatory, Engin. Sci., 48, 5
  * Gargatagli (1999) Gargatagli, M. 1999, La Escuela de Traductores de Toledo, Quaderns. Revista de Traducció, 4, 9
  * Gaulke (2009) Gaulke, K. 2009, The first European Observatory of the Sixteenth Century, as Founded by Landgrave Wilhelm IV of Hesse-Kassel: a Serious Historiographic Category or a Misleading Marketing Device?, in: Strano, G. et al. (eds.), European Collection of Scientific Instruments 1550--1750 (Leiden: Brill)
  * George (2005) George, A. 2005, The Tower of Babel: Archaeology, History and Cuneiform Texts, Archiv für Orientforschung, 51, 75
  * Gunther (1932) Gunther, R. T. 1932, Early Science in Oxford (Oxford: Oxford University Press)
  * Holden (1891) Holden, E. S. 1891, The Observatory of Nice, PASP, 3, 123
  * Hünsch et al. (2012) Hünsch, M., Seemann, A. & Wolfschmidt, G. 2012, The Hamburg Observatory. Justification of Outstanding Universal Value, Freie und Hansestadt Hamburg / Kulturbehörde / Denkmalschutzamt
  * Hussey (1897) Hussey, W. J. 1897, The Yerkes Observatory, PASP, 59, 220
  * Krisciunas (1999) Krisciunas, K., 1999, Observatories, Cornell University Library (arXiv:astro-ph/9902030)
  * Kwan (2012) Kwan, A. M. 2012, Architectures of Astronomical Observation: from Sternwarte Kassel (circa 1560) to Radcliffe Observatory (1772), PhD Thesis, Yale University
  * Leverington (2017) Leverington, D. 2017, Observatories and Telescopes of Modern Times (Cambridge: Cambridge University Press)
  * Misch & Stone (1998) Misch, A. & Stone, R. 1998, Building the Observatory, Web of The Lick Observatory Historical Collections, collections.ucolick.org
  * Mujani et al. (2012) Mujani, W. K. et al. 2012, Observatories in Islamic History, Adv. Nat. Appl. Sci., 6, 1370
  * Mundt (1927) Mundt, C. S. 1927, Notable Astronomers of Past Ages. II -- Ptolemy, PASP, 39, 122
  * Neubauer (1950) Neubauer, F. J. 1950, A Short History of the Lick Observatory, Pop. Astron., 58, 200
  * Pannekoek (1961) Pannekoek, A. 1961, A history of Astronomy (New York: Dover)
  * Pantin (1999) Pantin, I. 1999, New Philosophy and Old Prejudices: Aspects of the Reception of Copernicanism in a Divided Europe, Studies Hist. Philos. Sci., 30, 237
  * Park (2008) Park, C. 2008, Astronomy: Traditional Korean Science (Seoul: Ewha Womans University)
  * Park (2010) Park, J. E. 2010, Cheomseongdae Observatory, Republic of Korea, in: Ruggles, C. & Cotte, M. (eds.), Heritage Sites of Astronomy and Archaeoastronomy in the Context of the UNESCO World Heritage Convention (Paris: ICOMOS/IAU), 96, whc.unesco.org
  * Pasztor & Roslund (2007) Pasztor, E. & Roslund, R. 2007, An Interpretation of the Nebra Disc, Antiquity, 81, 267
  * Saliba (1987) Saliba, G. 1987, The Role of Maragha Observatory in the Development of Islamic Astronomy: a Scientific Revolution before the Renaissance, Revue de Synthèse, 108, 361
  * Sayili (1960) Sayili, A. 1960, The Observatory in Islam and its Place in the General History of Astronomy (Ankara: Türk Tarih Kurumu Basimevi)
  * Sayili (1980) Sayili, A. 1980, The International Symposium on the Observatories in Islam, 19--23 Sep 1977 (Istanbul, Ed. Dizer)
  * Scherrer (2018) Scherrer, D. 2018, Ancient Observatories -- Timeless Knowledge, Stanford University, solar-center.stanford.edu
  * Schilling (1998) Schilling, G. 1998, World’s Biggest Telescope Is About to See the Light, Science, 280, 670
  * Smiley (1938) Smiley, C. H. 1938, The History of the Lick Observatory, Sci. Monthly, 47, 128
  * Todd (1922) Todd, D. (1922): Astronomy: the Science of the Heavenly Bodies (London: Harper & Brothers)
  * Ussher (1787) Ussher, H. 1787, The Transactions of the Royal Irish Academy (Dublin: Royal Irish Academy), 1, 3
  * Waumans (2013) Waumans, A. A. 2013, The Typology of Astronomical Observatories, Ph.D. Thesis, Delft University of Technology
  * Wolfschmidt (2008) Wolfschmidt, G. (ed.) 2008, Cultural Heritage of Astronomical Observatories. From Classical Astronomy to Modern Astrophysics (Hamburg: ICOMOS)
  * Zinner (1943) Zinner, E. 1943a, Entstehung und Ausbreitung der Copernikanischen Lehre (Erlangen: Max Mencke)
  * Zinner (1956) Zinner, E. 1956, Astronomische Instrumente des 11. bis 18. Jahrhunderts (Munich: C. H. Beck)


# ProMIL: Probabilistic Multiple Instance Learning for Medical Imaging

Łukasz Struski Corresponding Author. Email<EMAIL_ADDRESS>Dawid
Rymarczyk  Arkadiusz Lewicki  Robert Sabiniewicz  Jacek Tabor  Bartosz
Zieliński  Faculty of Mathematics and Computer Science, Jagiellonian
University  Faculty of Applied Computer Science, University of Information
Technology and Management in Rzeszow  Department of Pediatric Cardiology and
Congenital Heart Diseases, Medical University of Gdansk  UES Ltd.

###### Abstract

Multiple Instance Learning (MIL) is a weakly-supervised problem in which one
label is assigned to the whole bag of instances. An important class of MIL
models is instance-based, where we first classify instances and then aggregate
those predictions to obtain a bag label. The most common MIL model is when we
consider a bag as positive if at least one of its instances has a positive
label. However, this reasoning does not hold in many real-life scenarios,
where the positive bag label is often a consequence of a certain percentage of
positive instances. To address this issue, we introduce a dedicated instance-
based method called ProMIL, based on deep neural networks and Bernstein
polynomial estimation. An important advantage of ProMIL is that it can
automatically detect the optimal percentage level for decision-making. We show
that ProMIL outperforms standard instance-based MIL in real-world medical
applications. We make the code available.

##  1 Introduction

Classification-oriented machine learning typically assumes that each example
in a training set has a unique label assigned to it. However, in many
practical situations, it is not feasible to label each instance individually.
This results in a challenge known as Multiple Instance Learning (MIL) [9],
where a bag of instances is associated with a single label, and it is assumed
that some instances in the bag are relevant to that label.

MIL represents a powerful approach used in different application fields,
mostly to solve problems where instances are naturally arranged in sets or to
leverage weakly annotated data [4]. Its numerous applications include molecule
classification task [9], predicting gene functions [11], content-based image
retrieval [42], object location [17] and segmentation [31], computer-aided
diagnosis and detection [2], document classification [14], and web mining
[50].

Multitude applications of MIL contribute to introducing many new methods,
described in the surveys [4, 49]. They generally divide into two types,
instance- and representation-based. The former classifies instances and then
aggregates those predictions to obtain a bag label. The latter aggregates
instance representations and predicts a bag label directly. Therefore,
instance-based approaches are more interpretable but usually less accurate
than representation-based methods. Most methods are representation-based and
work only for the standard assumption, where a bag of instances is considered
positive if it contains at least one positive instance. Only a few consider
other assumptions described in [12], like presence-, threshold-, or count-
based assumptions.

Figure 1: In the percentage-based MIL assumption, each bag has a label
indicating whether it is positive or negative based. It is based on the
percentage of positive instances, but the information about the percentage
threshold and the instance labels are unavailable during training. This
example includes six various-sized bags containing positive and negative
instances (green pluses and red minuses, respectively), and bags with a
percentage of positive instances over threshold $0.3$ are positive. In this
paper, we define this assumption and introduce a dedicated method to solve it.
Figure 2: ProMIL model architecture consists of two trainable parameters (here
marked as blue boxes), an instance classification network $f$ and parameters
$q$. The model takes a bag of instances (e.g. video frames) and passes them
through network $f$, obtaining instances predictions
$c_{0},c_{1},\ldots,c_{4}$. These predictions are then used with the trainable
parameter $q$ in the differentiable estimation of the $q$-th quantile
($\mathbf{c}_{q}$). If the quantile exceeds $0.5$, then we conclude that the
bag is positive (e.g. correspond to a heart defect).

In this paper, we consider a specific type of MIL assumption, which we have
named the percentage-based assumption. It is motivated by real-world medical
problems, such as predicting neutrophils infiltration level in the Geboes
scoring system on biopsy images [13], identifying bacteria strains on
microscopic images [2], recording myocardial ischemia on ECG [40], or
detecting congenital heart defects in newborns on ultrasound videos [21]. In
percentage-based assumption, a bag is positive if a certain percentage of its
instances is positive (however, this threshold is unknown during training).
For instance, consider the evaluation of digestive tract health using the NHI
scoring system [27], where a biopsy is given a score of $2$ if more than 50%
of crypts are infiltrated with neutrophils and there is no damage or
ulceration in the epithelium.

Besides providing a novel MIL assumption, we introduce a dedicated method
ProMIL based on the Bernstein polynomial estimation that can be applied to any
deep architecture. It classifies each instance separately and analyzes whether
the appropriate percentage of those predictions corresponds to a positive
label (is above the specific threshold). Importantly, ProMIL determines this
threshold automatically during training.

We compared our approach with the baseline methods for artificial and real-
world datasets, including our database with ultrasound videos of congenital
heart defects in newborns that are among the most common developmental
abnormalities and often lead to serious health consequences, including
increased mortality and complications later in life. Finally, we particularly
focus on the interpretable aspect of our method, bearing in mind its potential
medical applications.

Our contributions can be summarized as follows:

  * •

We define a novel percentage-based MIL assumption motivated by real-work
medical problems.

  * •

We introduce a method dedicated to this assumption based on the Bernstein
polynomial estimator, which automatically discovers the threshold and can be
applied to any deep architecture.

  * •

Our method overpasses existing instance-based solutions, obtaining high
performance while maintaining well-interpretable instance-level predictions.

##  2 Related Works

#### Instance-based MIL.

Initial instance-based MIL methods trained a classifier by assigning pseudo
labels (typically a bag label) to each instance. This approach was introduced
in [30] and has been widely applied, e.g. in microscopy image classification
[23] and drug activity prediction [48]. However, it can result in noisy
labels. Therefore, recent studies focus on enhancing instance-based
classifiers with dedicated MIL residual connections [43] or dynamic pooling
[44]. Other approaches choose only crucial instances for training. Crucial
instances can be selected based on classifier outputs [3], centered loss and
centroids [6], or cluster conditioned distribution [33]. ProMIL is an
instance-based MIL method, which, in contrast to existing solutions, is
dedicated to percentage-based assumption and can automatically estimate the
required threshold.

#### Representation-based MIL.

Representation-based methods differ from instance-based approaches because
they extract instance-level representations, aggregate them to a bag-level
embedding, and use it to classify a bag directly. Most methods use the
attention mechanism to assign importance to each instance during
representation aggregation. AbMILP [18], introducing attention-based pooling,
is one of the most popular approaches. It was extended by many mechanisms,
including self-attention to detect intra-bag dependencies between instances
[35], adversarial training to improve robustness [15], cluster assignments
[51], siamese networks [45], attention loss [38], feature distillation [47],
pyramidal fusion [26], prototypical parts for global interpretability [36,
46], multi-attention [22], and clustering constrained attention [29]. While
there have been attempts to classify single instances while obtaining the bag-
level embedding [32], the predictions obtained through this approach are still
noisy. Hence, from the interpretability point of view, instance-based
approaches, such as ProMIL, are preferable.

##  3 Preliminaries

#### Multiple Instance Learning (MIL).

Multiple Instance Learning (MIL) is a weakly-supervised problem in which one
label $\mathbf{y}\in\mathbb{R}$ is assigned to a bag of instances
$\mathbf{x}=\\{x_{i}\\}_{i=0}^{n}$, where $n$ varies between bags [12].
Moreover, in the _standard MIL assumption_ , the label of bag
$\mathbf{y}\in\\{0,1\\}$, each instance $x_{i}$ has a hidden binary label
$y_{i}\in\\{0,1\\}$ (unknown during training), and a bag is positive if at
least one of its instances is positive

$\centering\mathbf{y}=\begin{cases}0,&\text{ iff
}\sum\limits_{i=0}^{n}y_{i}=0,\\\ 1,&\text{
otherwise.}\end{cases}\@add@centering$ (1)

#### Bernstein polynomial estimator.

One of the main advantages of our approach is the ability to automatically
discover the threshold in the percentage-based assumption. For this purpose,
we use Bernstein polynomial estimator [5, 24, 52]. Assuming that we have a set
$\mathbf{p}$ with sorted elements $p_{0}\leq p_{1}\leq\ldots\leq p_{n}$, its
$q$-quantile estimator is calculated as

$\mathbf{p}_{q}=\sum_{k=0}^{n}\binom{n}{k}q^{n-k}(1-q)^{k}\cdot p_{k},$ (2)

where $\binom{n}{k}$ is a binomial coefficient.

Compared to the competitive solutions [19, 34, 41], Bernstein polynomial
estimator provides more accurate estimates of quantiles for datasets with
complex or non-uniform distributions and is computationally efficient, even
when implemented using standard numerical methods.

##  4 ProMIL

Probabilistic Multiple Instance Learning (ProMIL) was created with the
percentage-based assumption in mind that occurs in many real-world biomedical
applications. It assumes that a positive bag is associated with a certain
percentage of abnormal instances. Therefore, we classify those instances and
then calculate the $q$-quantile of their predictions (in the range $[0,1]$).
The bag is considered abnormal if the $q$-quantile is larger than $0.5$ and
normal otherwise. From this perspective, a method for quantile estimation
should be differentiable to train the model and the optimal value of
additional parameter $q$. Therefore, we use Bernstein polynomial estimation
Eq. 2.

To formalize our approach, let us assume that
$\mathbf{x}=\\{x_{i}\\}_{i=0}^{n}$ is a bag of instances from the training set
with the corresponding label $\mathbf{y}$. Moreover, let $f$ be the
classification network run for each instance $x_{i}$ separately (see Figure
2).

During training, presented in Algorithm 1, we take random bag $\mathbf{x}$
from a training set and pass its instances through network $f$. As a result,
we obtain a set of predictions $\mathbf{c}=\\{c_{0},c_{1},\ldots,c_{n}\\}$. We
sort this set so that $c_{0}\leq c_{1}\leq\ldots\leq c_{n}$, and then we
compute their $q$\- and $1-q$-quantile ($\mathbf{c}_{q}$ and
$\mathbf{c}_{1-q}$) using Eq. 2. Because $c_{i}\in[0,1]$ then, according to
properties of Bernstein polynomials [28] $\mathbf{c}_{q}\in[0,1]$. Finally, we
apply BCE loss to train our model

$\mathrm{cost}(\mathbf{x},\mathbf{y})=-\mathbf{y}\log\mathbf{c}_{q}-(1-\mathbf{y})\log\mathbf{c}_{1-q},$
(3)

where $q$ is a trainable parameter.

Data: $\mathbf{X}$, $\mathbf{Y}$ – bags from training set and their labels

Parameter: $f$ – instance classification network; $q$ – quantile (threshold)

Result: trained $f$ and $q$

1

2randomly initialize $q$;

3 while  _no converngence_ do

4 take random bag $\mathbf{x}=\\{x_{0},x_{1},\ldots,x_{n}\\}$ from
$\mathbf{X}$;

5 take bag label $\mathbf{y}$ from $\mathbf{Y}$;

6 calculate $\mathbf{c}=\\{c_{i}=f(x_{i})\\}_{i=0}^{n}$;

7 sort $\mathbf{c}$ in ascending order;

8 calculate $\mathbf{c}_{q}$ ($q$-quantile of $\mathbf{c}$) using Eq. 2;

9 calculate $\mathrm{cost}$ by applying $\mathbf{c}_{q}$ and $\mathbf{y}$ to
Eq. 3;

10 update $f$ and $q$ using $\mathrm{cost}$ back-propagation

Algorithm 1 ProMIL training

Most of the calculations described in this algorithm are performed on a
logarithmic scale for numerical stability, including logsumexp and the
binomial coefficient calculated as

${n\choose k}=\exp(\log\Gamma(n+1)-\log\Gamma(k+1)-\log\Gamma(n-k+1)),$

where $\log\Gamma$ denotes the natural logarithm of the gamma function. Both
logsumexp and lgamma functions are available in the
PyTorch111https://pytorch.org framework.

##  5 Experimental Setup

###  5.1 Datasets

#### MNIST-bag

To evaluate the effectiveness of our ProMIL approach for percentage-based MIL
assumption, we use the artificial MIL dataset MNIST-bag. The dataset consists
of $1000$ bags, each containing instances randomly sampled from a normal
distribution with mean of $30$ instances with variance equals to $5$. The
percantage of numbers of interests within a bag is taken randomly from a
uniform distribution to ensure the balance within a dataset. We permute the
order of instances within a bag to maintain permutation invariance. The
examples of created bags from the dataset are presented in Figure 3.

The dataset is an enhancement of the dataset introduced in [18] with a
different definition of a positive bag to fulfill the percentage-based
assumption. We define the bag as positive if at least $q$ fraction of
instances from the bag are $9$s. We maintain MNIST division into training and
test sets.

Figure 3: Four examples from the MNIST-bag dataset where the number of
instances within a bag is sampled from $\mathcal{N}(5,\,2)$, and the quantile
$q=0.4$. In red boxes, we highlight $9$’s corresponding to the positive
instances.

#### Histopathology datasets

Here we present details on three standard MIL datasets that represent MIL
problems in real life, classification of histopathology images: Colon Cancer
[39], Camelyon16 [10], and TCGA-NSCLC [1]. They consist of 100, 399, and 956
H&E stained whole slide images, respectively. For Colon Cancer, we use patches
of the size $32\times 32$ while for other datasets we use $224\times 224$.
Colon Cancer has $100$ bags with $2,244$ patches, Camelyon has $399$ bags with
a mean of $8,871$ patches, and TCGA-NSCLC has $1,016$ bags with a mean of
$3,961$ patches. From a medical perspective, the Colon Cancer dataset focuses
on detecting epithelial nuclei which are often symptoms of developing disease,
Camelyon16 is about the detection of micro- and macro-metastases in lymph
nodes which is a prognostic factor for breast cancer growth. While the TCGA-
NSCLC dataset is the identification of a cancer subtype which conditions which
treatment should be applied. Each of those tasks is of high relevance from a
clinical perspective.

#### Doppler Ultrasound database

Our proposed approach holds significant potential in analyzing ultrasound
videos (USG), which is currently one of the primary diagnostic tools for
detecting congenital heart defects in newborns. These defects are among the
most common developmental abnormalities and can lead to serious health
consequences, including increased mortality and complications later in life.

The traditional approach to analyzing USG images involves individual
examination and evaluation by qualified physicians specializing in pediatric
cardiology. However, this method is time-consuming, requires highly
specialized knowledge, and is susceptible to human errors. It is especially
problematic for challenging cases, such as Tetralogy of Fallot, hypoplastic
left heart syndrome, or defects associated with the pulmonary vein (see Figure
4).

Figure 4: Ultrasound video frames corresponding to congenital heart defects in
newborns, such as structural abnormalities and defects in large blood vessels.

We performed the Doppler ultrasound on over 250 patients to record blood flow
in the heart and detect abnormalities. For some patients, the study was
repeated, resulting in a dataset with over 1000 films of varying lengths
(ranging from 10 frames to over 240 frames), with over 40% of the films being
without defects. Heart defects specialists labeled each film. Before training
the network, the data were randomly split into training and test sets while
maintaining class proportions and ensuring that studies from the same patient
were not included in both sets.

The training set constituted 85% of the entire dataset, while the remaining
15% was reserved for testing. Each frame was scaled to $200\times 200$ pixels
during network training, and a center crop of 172 pixels was used. During an
evaluation, the frames were scaled to $172\times 172$ pixels.

Figure 5: The effectiveness of various MIL methods - AbMILP, Instance-MAX,
Instance-MEAN, and ProMIL (our proposed method) on the MNIST-bag dataset was
compared. The performance of these methods was analyzed under different
scenarios, and the results were plotted. The left image shows the effect of
the percentage of positive instances within a bag on the performance of the
methods. The middle plot depicts how the bag size influences the effectiveness
of MIL methods. The right plot shows the impact of the training dataset size
on the model’s performance. It can be observed that our proposed ProMIL method
consistently outperformed other methods in all scenarios. Moreover, AbMILP was
found to be unstable due to overfitting when the number of instances within a
bag increased. Lastly, the other methods showed higher variance in the results
as compared to our ProMIL method.

###  5.2 Implementation details

For the MNIST-bag dataset, we use LeNet-5 [25] network to encode the images.
It is learned for 100 epochs with early stopping after 15 epochs without a
change on a validation set. Training for each scenario was repeated five
times. We use Adam optimizer [20] with learning rate $10^{-4}$ with
$\beta_{1}=0.99,\beta_{2}=0.999$, weight decay equals $10^{-5}$, and batch
size 1. AbMILP used as a baseline method also uses LeNet5 backbone, consisting
of one attention head with embedding size equal to $500$ and classifier hidden
space $128$. We use a binary cross entropy as a loss function.

For Colon Cancer dataset we use backbone from [39], Adam optimizer with
learning rate $10^{-3}$ with $\beta_{1}=0.99,\beta_{2}=0.999$, weight decay
equals $10^{-3}$, and batch size $1$. We train a model for $200$ epochs with
an early stopping window equal to $25$ epochs. We split the data into ten
folds and repeat the training 5 times, similarly as it is done in [18].

When it comes to Camelyon16 and TCGA-NSCLC datasets, we use pretrained in a
self-supervised way ResNet18 [7] as a backbone and train a model for $200$
epochs with an early stopping window equal to $40$ epochs. Also, we use Adam
optimizer with learning rate $5\cdot 10^{-4}$ with
$\beta_{1}=0.99,\beta_{2}=0.999$, weight decay equals $10^{-3}$, and batch
size $1$.

For all the aforementioned datasets, we performed a hyperparameter random grid
search with the following sets of parameters: learning rate in the range
$[10^{-6},10^{-2}]$, weight decay in range $[0,1]$, $q$ in the range
$[0.1,0.5]$.

To compute frame-based predictions for ultrasound videos, we utilized
ResNet-18 [16], which was pretrained on the ImageNet [8] dataset. We replaced
the last layer with two fully-connected layers, each having dimensions of
$128$ and $64$, respectively. We employed grid search techniques to fine-tune
hyperparameters, such as the learning rate (with possible values of
$10^{-6},10^{-5},10^{-4},10^{-3}$), dropout rate (searched over values of 0.1,
0.2, 0.3, 0.4), and L2 regularization (with possible values of
$0,10^{-6},10^{-5},10^{-4}$). Moreover, we experimented with different
learning rate schedulers, including CyclicLR, LambdaLR, and without any
scheduler.

We trained the models in two stages. In the first stage, we froze the feature
extraction part of the pre-trained ResNet-18 model and only trained the two
additional linear layers for 25 epochs. In the second stage, we fine-tuned the
entire network for 100 epochs. To train the models, we used the Adam optimizer
[20] with default values, similar to other datasets. To accommodate the
varying lengths of videos in our dataset, we utilized a batch size of 1.

##  6 Results

Table 1: Results for Colon Cancer dataset, where ProMIL outperforms the baseline methods (instance+max and instance+mean). However, it achieves worse results compared to more advanced techniques that employ attention mechanisms such as SA-AbMILP*. Notice that values for comparison are indicated with “*” from [36]. | Colon Cancer
---|---
Method | Accuracy | AUC
embedding+max* | $82.4\%\pm 1.5\%$ | $0.918\pm 0.010$
embedding+mean* | $86.0\%\pm 1.4\%$ | $0.940\pm 0.010$
AbMILP* | $88.4\%\pm 1.4\%$ | $0.973\pm 0.007$
SA-AbMILP* |  90.8% $\pm$ 1.3% |  0.981 $\pm$ 0.007
ProtoMIL* | $81.3\%\pm 1.9\%$ | $0.932\pm 0.014$
instance+max* | $84.2\%\pm 2.1\%$ | $0.914\pm 0.010$
instance+mean* | $77.2\%\pm 1.2\%$ | $0.866\pm 0.008$
ProMIL (our) | $\textbf{88.1}\%\pm\textbf{1.7\%}$ | $\textbf{0.969}\pm\textbf{0.010}$

This section provides the results of experiments conducted on five different
datasets. We begin with MNIST-gab, which is a toy dataset that enables a more
thorough understanding and analysis of our ProMIL method. Next, we present our
findings on three histopathological datasets, which are commonly used
benchmarks for MIL algorithms, demonstrating the versatility of our approach.
Finally, we introduce a novel heart ultrasound dataset, where we demonstrate
the suitability of our ProMIL method for analyzing medical videos. We also
include a detailed interpretability analysis of the method conducted with
specialized medical doctors to confirm that our models accurately recognize
important features.

Table 2: Our ProMIL approach achieves comparable or better results than baseline approaches, specifically instance+max and instance+mean, for the Camelyon16 and TCGA-NSCLC datasets. However, it falls short of matching the performance of more complex models such as TransMIL, which are based on transformer architectures. It is worth noting that ProMIL matches the performance of the instance+max approach. This may be related to the characterization of lymph metastasis (Camelyon16 dataset), which typically covers a small tissue portion. As a result, the ProMIL learns to estimate a small value for $q$ and achieves similar results to the instance+max approach. Notice that values for comparison marked with “*” are taken from [36]. | Camelyon16 | TCGA-NSCLC
---|---|---
Method | Accuracy | AUC | Accuracy | AUC
MILRNN* | 80.62% | 0.807 | 86.19% | 0.910
AbMILP* | 84.50% | 0.865 | 77.19% | 0.865
DSMIL* | 86.82% | 0.894 | 80.58% | 0.892
CLAM-SB* | 87.60% | 0.881 | 81.80% | 0.881
CLAM-MB* | 83.72% | 0.868 | 84.22% | 0.937
TransMIL* | 88.37% | 0.931 | 88.35% | 0.960
ProtoMIL* | 87.29% | 0.935 | 83.66% | 0.918
instance+mean* | 79.84% | 0.762 | 72.82% | 0.840
instance+max* | 82.95% | 0.864 | 85.93% | 0.946
ProMIL (our) | 82.95% | 0.864 | 86.71% | 0.940
Table 3: Our ProMIL method achieves superior results for the Doppler ultrasound database. Other methods such as instance+max, and instance+mean obtained inferior results compared to ProMIL. Moreover, the performance of AbMILP for this task is reduced showing that our ProMIL which is an instance-based approach is more suitable for this task. By ’Accuracy’ we mean balanced accuracy. | Doppler Ultrasound
---|---
Method | Accuracy | AUC
AbMILP | $79.77\%$ | $0.85$
instance+max | $70.09\%$ | $0.71$
instance+mean | $75.67\%$ | $0.79$
ProMIL (our) | $\textbf{86.36}\%$ | 0.92

#### MNIST-bag results.

We conducted a comparative analysis of our ProMIL approach with the instance-
max and instance-mean methods, as well as with Attention MIL Pooling [18]. As
illustrated in Figure 5, our ProMIL outperforms all other methods in terms of
AUC. Additionally, we found that percentage-based assumption is difficult for
AbMILP and can lead to random performance on the test dataset, even if the
model converges during training. Moreover, we observed that Instance-MAX and
Instance-MEAN struggle to distinguish between positive and negative bags when
the ground truth $q$ is in the range of $[0.1,0.5]$. With an increase in bag
size or the number of training bags, all models achieve better performance.
However, our ProMIL still outperforms them, making it suitable for both small
and large datasets with small or large bags.

#### Histopathological results.

We evaluated the versatility of our ProMIL approach on three real-life
datasets commonly used as standard benchmarks in MIL: Colon Cancer,
Camelyon16, and TCGA-NSCLC. The results for Colon Cancer are presented in
Table 1, while those for Camelyon16 and TCGA-NSCLC are in Table 2. Our ProMIL
approach outperformed other Instance-based methods for Colon Cancer and TCGA-
NSCLC and matched the performance of the Instance-MAX approach for Camelyon16.
This similarity in performance can be attributed to the small amount of
metastatic tissue required for a positive label in Camelyon16 [10], leading
our ProMIL to learn a small $q$ what is making it similar to Instance-MAX.
When compared to other MIL approaches that operate on the bag level, such as
TransMIL and ProtoMIL, our ProMIL outperformed some of them (e.g., MILRNN for
TCGA-NSCLC and AbMILP for Colon Cancer) and was slightly worse than recent
approaches using transformer-based architectures such as TransMIL. These
results demonstrate the adaptability of ProMIL to a variety of MIL problems in
the digital pathology field.

#### Doppler Ultrasound database results.

We conducted an evaluation of our approach on a dataset of ultrasound videos
with varying numbers of frames. Similar to the previous experiment, we
compared our method with the baseline (instance-based) approaches. The results
of this evaluation are presented in Table 3.

ProMIL utilizes a unique strategy for selecting the most significant frames in
each video using the $q=38.56\%$-quantile criterion. This approach contrasts
with the competing methods, instance+max and instance+mean, which rely on
single-frame and all-frame analyses, respectively. In our evaluation, we found
that choosing the dominant prediction did not necessarily lead to high-
performance scores. Furthermore, using all frames, as done in the
instance+mean method, produced better results than the instance+max method,
but still failed to achieve significantly better results, indicating
overfitting to the noise.

Our approach’s superiority over the baseline methods is further highlighted by
the effectiveness of selecting the most significant frames in each video,
allowing us to mitigate the effects of noise and achieve superior performance.

Furthermore, Figure 6 provides insights into the distribution of the
predictions made by the $f$ classification component of the ProMIL model. This
distribution is based on the selection of significant frames, which our model
determined to be 38.56% for X data. To illustrate this, we only considered
true negative and positive bags and presented their prediction distributions
separately. Our results show that for true positive bags, all significant
frames have predictions above 0.8, indicating high confidence in their
classification. Conversely, for true negative bags, up to the 50th percentile,
instances with predictions close to zero dominate the model’s inference.

Figure 6: The illustration depicts the classification predictions generated by
the ProMIL $f$ classification component. This component relies on the
selection of significant frames, representing 38.56% of the frames with the
highest prediction values, that impacted its decision-making process. The
figure displays the 1st, 25th, 50th, 75th, and 99th percentiles of these
significant frames in each row. The left column exhibits the prediction
distribution for true negative bags, while the right column shows the
distribution for true positive bags. Note that frames with predictions below
threshold 0.5 are absent from the right column, which is not apparent from the
left column. Even among the 99th percentile, there are frames with predictions
approaching zero.

In addition to analyzing the number and selection of frames, our study also
focused on identifying the specific objects and regions of interest in
individual frames that our model considers when making predictions. To achieve
this, we utilized the widely recognized Grad-CAM technique [37]. This approach
generates a heatmap highlighting the critical regions of an input image that
the model focused on to arrive at its decision.

Figure 7 presents several results that demonstrate the heart areas our model
emphasized when examining ultrasound images of a child with a congenital heart
defect that involves a septal defect. After consulting with clinical experts
in the field of pediatric cardiology, we confirmed that the regions marked on
the heatmaps align with the areas typically identified by experts.
Consequently, our approach could assist medical practitioners in comprehending
the outcomes and recognizing heart defects in newborns.

Figure 7: An ultrasonographic image of a newborn’s heart with a ventricular
septal defect and the corresponding heat map. This map allows to understand
which features of the image were important for the model during the prediction
process, taking into account the differences between successive frames of the
processed and analyzed ultrasound recording. Areas with larger changes on the
heat map are marked in yellow, red and burgundy as areas with higher heat
intensity. Consultations with medical experts in the field of cardiology
confirmed that such areas in the presented image indicate the locations of the
ventricular septal defect.

##  7 Conclusions

In this paper, we propose a novel instance-based approach ProMIL to Multiple
Instance Learning (MIL), where bag classification is obtained by aggregating
instance-level predictions. It solves the percentage-based assumption, which
occurs in many real-world medical applications, using Bernstein polynomial
estimation.

ProMIL outperforms other instance-based approaches and obtains results
comparable to less interpretable representation-based models. Therefore, it
can be easily applied to critical domains, such as computer-assisted
interventions.

##  8 Acknowledgements

This research was partially funded by the National Science Centre, Poland,
grants no. 2020/39/D/ST6/01332 (work by Łukasz Struski), 2022/47/B/ST6/03397
(work by Bartosz Zieliński), and 2022/45/N/ST6/04147 (work by Dawid
Rymarczyk). Moreover, Dawid Rymarczyk received an incentive scholarship from
the funds of the program Excellence Initiative – Research University at the
Jagiellonian University in Kraków. Works of Jacek Tabor, Arkadiusz Lewicki,
and Robert Sabiniewicz were supported by company the UES Ltd. Some experiments
were performed on servers purchased with funds from a grant from the Priority
Research Area (Artificial Intelligence Computing Center Core Facility) under
the Strategic Programme Excellence Initiative at Jagiellonian University.

## References

  * [1] Shaimaa Bakr, Olivier Gevaert, et al., ‘A radiogenomic dataset of non-small cell lung cancer’, Scientific data, 5(1), 1–9, (2018).
  * [2] Adriana Borowa, Dawid Rymarczyk, et al., ‘Identifying bacteria species on microscopic polyculture images using deep learning’, IEEE Journal of Biomedical and Health Informatics, 27(1), 121–130, (2023).
  * [3] Gabriele Campanella, Matthew G Hanna, et al., ‘Clinical-grade computational pathology using weakly supervised deep learning on whole slide images’, Nature medicine, 25(8), 1301–1309, (2019).
  * [4] Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon, ‘Multiple instance learning: A survey of problem characteristics and applications’, Pattern Recognition, 77, 329–353, (2018).
  * [5] Cheng Cheng, ‘The bernstein polynomial estimator of a smooth quantile function’, Statistics & probability letters, 24(4), 321–330, (1995).
  * [6] Philip Chikontwe, Meejeong Kim, Soo Jeong Nam, Heounjeong Go, and Sang Hyun Park, ‘Multiple instance learning with center embeddings for histopathology classification’, in MICCAI, (2020).
  * [7] Ozan Ciga, Anne L Martel, and Tony Xu, ‘Self supervised contrastive learning for digital histopathology’, preprint arXiv:2011.13971, (2020).
  * [8] Jia Deng, Wei Dong, et al., ‘Imagenet: A large-scale hierarchical image database’, in CVPR, (2009).
  * [9] Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez, ‘Solving the multiple instance problem with axis-parallel rectangles’, Artificial intelligence, 89(1-2), 31–71, (1997).
  * [10] Babak Ehteshami Bejnordi, Mitko Veta, et al., ‘Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer’, JAMA, 318(22), 2199–2210, (12 2017).
  * [11] Ridvan Eksi, Hong-Dong Li, et al., ‘Systematically differentiating functions for alternatively spliced isoforms through integrating rna-seq data’, PLoS computational biology, 9(11), e1003314, (2013).
  * [12] James Foulds and Eibe Frank, ‘A review of multi-instance learning assumptions’, The knowledge engineering review, 25(1), 1–25, (2010).
  * [13] Karel Geboes, R Riddell, A Öst, B Jensfelt, T Persson, and R Löfberg, ‘A reproducible grading scale for histological assessment of inflammation in ulcerative colitis’, Gut, 47(3), 404–409, (2000).
  * [14] Zellig S Harris, ‘Distributional structure’, Word, 10(2-3), 146–162, (1954).
  * [15] Noriaki Hashimoto, Daisuke Fukushima, et al., ‘Multi-scale domain-adversarial multiple-instance cnn for cancer subtype classification with unannotated histopathological images’, in CVPR, (2020).
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, ‘Deep residual learning for image recognition’, in CVPR, (2016).
  * [17] Judy Hoffman, Deepak Pathak, Trevor Darrell, and Kate Saenko, ‘Detector discovery in the wild: Joint multiple instance and representation learning’, in CVPR, (2015).
  * [18] Maximilian Ilse, Jakub Tomczak, and Max Welling, ‘Attention-based deep multiple instance learning’, in ICML, (2018).
  * [19] Maurice George Kendall et al., ‘The advanced theory of statistics.’, The advanced theory of statistics., (2nd Ed), (1946).
  * [20] Diederik P Kingma and Jimmy Ba, ‘Adam: A method for stochastic optimization’, preprint arXiv:1412.6980, (2014).
  * [21] Masaaki Komatsu, Akira Sakai, Reina Komatsu, et al., ‘Detection of cardiac structural abnormalities in fetal ultrasound videos using deep learning’, Applied Sciences, 11(1), 371, (2021).
  * [22] Andrei V Konstantinov and Lev V Utkin, ‘Multi-attention multiple instance learning’, Neural Computing and Applications, 34(16), 14029–14051, (2022).
  * [23] Oren Z Kraus, Jimmy Lei Ba, and Brendan J Frey, ‘Classifying and segmenting microscopy images with deep multiple instance learning’, Bioinformatics, 32(12), i52–i59, (2016).
  * [24] Alexandre Leblanc, ‘On estimating distribution functions using bernstein polynomials’, Annals of the Institute of Statistical Mathematics, 64(5), 919–943, (2012).
  * [25] Yann LeCun, Bernhard Boser, et al., ‘Handwritten digit recognition with a back-propagation network’, NeurIPS, (1989).
  * [26] Bin Li, Yin Li, and Kevin W Eliceiri, ‘Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning’, in CVPR, (2021).
  * [27] Katherine Li, Richard Strauss, Colleen Marano, et al., ‘A simplified definition of histologic improvement in ulcerative colitis and its association with disease outcomes up to 30 weeks from initiation of therapy: Post hoc analysis of three clinical trials’, Journal of Crohn’s and Colitis, 13(8), 1025–1035, (2019).
  * [28] G. G. Lorentz.  Bernstein polynomials, 1953.
  * [29] Ming Y Lu, Drew FK Williamson, et al., ‘Data-efficient and weakly supervised computational pathology on whole-slide images’, Nature biomedical engineering, 5(6), 555–570, (2021).
  * [30] Oded Maron and Tomás Lozano-Pérez, ‘A framework for multiple-instance learning’, NeurIPS, (1997).
  * [31] Andreas Müller and Sven Behnke, ‘Multi-instance methods for partially supervised image segmentation’, in Partially Supervised Learning: First IAPR TC3 Workshop, PSL 2011, Ulm, Germany, September 15-16, 2011, Revised Selected Papers 1, pp. 110–119. Springer, (2012).
  * [32] Andriy Myronenko, Ziyue Xu, et al., ‘Accounting for dependencies in deep learning based multiple instance learning for whole slide imaging’, in MICCAI, (2021).
  * [33] Linhao Qu, Xiaoyuan Luo, Shaolei Liu, Manning Wang, and Zhijian Song, ‘Dgmil: Distribution guided multiple instance learning for whole slide image classification’, in MICCAI, (2022).
  * [34] Murray Rosenblatt, ‘Remarks on some nonparametric estimates of a density function’, The annals of mathematical statistics, 832–837, (1956).
  * [35] Dawid Rymarczyk, Adriana Borowa, Jacek Tabor, and Bartosz Zielinski, ‘Kernel self-attention for weakly-supervised image classification using deep multiple instance learning’, in WACV, (2021).
  * [36] Dawid Rymarczyk, Adam Pardyl, et al., ‘Protomil: Multiple instance learning with prototypical parts for whole-slide image classification’, in ECML PKDD, (2023).
  * [37] Ramprasaath R Selvaraju, Michael Cogswell, et al., ‘Grad-cam: Visual explanations from deep networks via gradient-based localization’, in ICCV, (2017).
  * [38] Xiaoshuang Shi, Fuyong Xing, Yuanpu Xie, Zizhao Zhang, Lei Cui, and Lin Yang, ‘Loss-based attention for deep multiple instance learning’, in AAAI, (2020).
  * [39] Korsuk Sirinukunwattana, Shan E Ahmed Raza, et al., ‘Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images’, IEEE transactions on medical imaging, 35(5), 1196–1206, (2016).
  * [40] Li Sun, Yanping Lu, Kaitao Yang, and Shaozi Li, ‘Ecg analysis using multiple instance learning for myocardial infarction detection’, IEEE transactions on biomedical engineering, 59(12), 3348–3356, (2012).
  * [41] Aad W Van der Vaart, Asymptotic statistics, volume 3, Cambridge university press, 2000.
  * [42] Sudheendra Vijayanarasimhan and Kristen Grauman, ‘Keywords to visual categories: Multiple-instance learning forweakly supervised object categorization’, in CVPR, (2008).
  * [43] Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu, ‘Revisiting multiple instance neural networks’, Pattern Recognition, 74, 15–24, (2018).
  * [44] Yongluan Yan, Xinggang Wang, Xiaojie Guo, Jiemin Fang, Wenyu Liu, and Junzhou Huang, ‘Deep multi-instance learning with dynamic pooling’, in ACML, (2018).
  * [45] Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang, ‘Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks’, Medical Image Analysis, 65, 101789, (2020).
  * [46] Jin-Gang Yu, Zihao Wu, et al., ‘Prototypical multiple instance learning for predicting lymph node metastasis of breast cancer from whole-slide pathological images’, Medical Image Analysis, 102748, (2023).
  * [47] Hongrun Zhang, Yanda Meng, et al., ‘Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification’, in CVPR, (2022).
  * [48] Zhendong Zhao, Gang Fu, et al., ‘Drug activity prediction using multiple-instance learning via joint instance and feature selection’, in BMC bioinformatics, volume 14, pp. 1–12. BioMed Central, (2013).
  * [49] Zhi-Hua Zhou, ‘Multi-instance learning: A survey’, Department of Computer Science & Technology, Nanjing University, Tech. Rep, 1, (2004).
  * [50] Zhi-Hua Zhou, Kai Jiang, and Ming Li, ‘Multi-instance learning based web mining’, Applied intelligence, 22, 135–147, (2005).
  * [51] Xinliang Zhu, Jiawen Yao, Feiyun Zhu, and Junzhou Huang, ‘Wsisa: Making survival prediction from whole slide histopathological images’, in CVPR, (2017).
  * [52] Ryszard Zielinski, Optimal Quantile Estimators Small Sample Approach, Polish Academy of Sciences. Institute of Mathematics, 2004.