|  Leveraging high-resolution spatial features in mid-infrared spectroscopic
imaging to classify tissue subtypes in ovarian cancer†
---|---
, , , , , , , , , , * |  Chalapathi Charan Gajjela,a Matthew Brun,b Rupali Mankar,a Noah Kennedy,c Sara Corvigno,d Yanping Zhong,d Jinsong Liu,d Anil K. Sood,d David Mayerich,a Sebastian Berisha,c and Rohith Reddy∗a
|  Mid-infrared spectroscopic imaging (MIRSI) is an emerging class of label-
free techniques being leveraged for digital histopathology. Optical
photothermal infrared (O-PTIR) is based on vibrational absorbance imaging
using a pump-probe architecture capable of a 10$\times$ enhancement in spatial
resolution relative to FTIR imaging. This allows truly sub-cellular
spectroscopic investigation of tissue at biochemically important fingerprint
wavelengths. Modern histopathologic identification of ovarian cancer involves
tissue staining followed by morphological pattern recognition. This process is
time-consuming, subjective, and requires extensive expertise. In this paper,
we present the first label-free automated histological classification of
ovarian tissue sub-types using MIRSI. We demonstrate that enhanced resolution
of sub-cellular features, combined with spectroscopic information, enables
reliable classification ($0.98$ AUC) of ovarian cell sub-types. Moreover, we
present statistically robust validation from $74$ patient samples with over
$60$ million data points. This demonstrates that sub-cellular resolution from
five wavenumbers is sufficient to outperform state-of-the-art diffraction-
limited techniques from up to $374$ different wavenumbers. O-PTIR also
performs measurements in back-reflection geometry, opening the door to future
in vivo studies on glass slides.

##

††footnotetext: a Address, 4226 Martin Luther King Boulevard, N308 Engineering
Building 1, Houston TX, 77584, USA; E-mail<EMAIL_ADDRESS>b
Address, Address, Town, Country. ††footnotetext: † Electronic Supplementary
Information (ESI) available: [details of any supplementary information
available should be included here]. See DOI: 00.0000/00000000.††footnotetext:
‡ Additional footnotes to the title and authors can be included e.g. ‘Present
address:’ or ‘These authors contributed equally to this work’ as above using
the symbols: ‡, §, and ¶. Please place the appropriate symbol next to the
author’s name and include a \footnotetext entry in the the correct place in
the list.

##  1 Introduction

Epithelial ovarian cancer is the leading cause of death among gynecological
malignancies in the United States. Serous ovarian cancer, its most common
subtype, is often diagnosed at late stage (III or IV) where 5-year survival is
51% and 29% respectively.1 Standard treatments involve surgery and at least
six courses of chemotherapy containing platinum.2 Several novel compounds have
been studied and approved over the past 20 years, however none substantially
modify overall survival.3 The strongest prognostic factor remains complete
eradication of neoplastic tissue through radical surgery.4, 5, 6 This is
affected by (1) late diagnosis resulting in unresectable disease, and (2)
unclear identification of neoplastic margins. Objective and early
identification of neoplastic tissue is therefore essential for optimal
surgical attempts.

Recent advances reveal the complex organization of the ovarian tumor
microenvironment, highlighting inter-cellular pathways 7 as potential
treatment targets. New methods quantifying biomolecular characteristics reveal
detailed structural and molecular changes that may reveal novel therapeutic
targets. The current standard for ovarian cancer diagnosis uses contrast-
inducing stains on biopsy sections followed by microscopic examination by a
pathologist. Hematoxylin and eosin (H&E) is widely used to identify cellular
and extracellular components. Epithelial carcinoma is the most common
histologic type, accounting for about 90 percent of cancers of the ovary,
fallopian tube, and peritoneum. 8, 9 In high-grade serous carcinoma (HGSC), a
pathologist identifies a variety of architectural patterns including complex
papillary, glandular, microcystic, and solid patterns. HGSC infiltrates,
destroys, and/or replaces the normal stroma. Therefore, histological
identification of cellular sub-types is an important step 10 in ovarian cancer
diagnosis and prognosis.

Inter-pathologist variability is an important challenge 11 and grading schemes
have been proposed to reduce this variability.12, 13, 14 However, these
methods have only been successful in resource-rich hospitals with
comprehensive training.15 Automated and semi-automated techniques to aid
reduce inter-pathologist variability, especially in lower-resource settings,
is critical for equitable care.16, 17, 18

Automated tissue classification into epithelium and stroma subtypes is
challenging, and several techniques have been proposed. Most use H&E 19, 20,
21 and immunohistochemical staining 22 combined with machine learning (ML).
Staining quality and variability can confound ML and lead to inconsistent
results.17 Our goal is to perform label-free recognition of tissue sub-types
without the use of chemical contrast agents. Moreover, we obtain intrinsic
quantitative and repeatable biochemical measurements that are independent of
operator tissue processing.

Spectroscopic techniques are used widely in chemical and biochemical analysis
23 to identify molecules with excellent sensitivity and specificity.
Vibrational spectroscopy is used routinely in the identification of organic
biomolecules and commercial spectral libraries that identify 24 over 260,000
spectra are now available.

Prior work on ovarian tissue analysis using label-free technologies has
utilized spectroscopy without obtaining hyperspectral tissue images. For
example, Raman spectroscopy 25, 26, 27, conventional Fourier Transform
Infrared (FTIR) spectroscopy 28, 29, 30 and attenuated total reflection (ATR)
FTIR spectroscopy 31, 32 have been used for the detection and diagnosis of
ovarian cancer. MALDI imaging 33 has also been used to analyze ovarian
histotypes but is tissue destructive. Second-harmonic generation (SHG) has
been used to identify collagen in stroma 34, 35, 36 and multi-photon
microscopy 37 is used on murine tissue. Raman imaging has been used for
ovarian cancer diagnosis and tissue analysis, often with added nanoparticles
38, 39 to obtain reliable signals. However, these techniques cannot provide
classified images of ovarian tissue sub-types in an automated, label-free,
quantitative, and non-destructive manner.

Mid-infrared spectroscopic imaging (MIRSI) can extract spectral and spatial
information from tissue samples, without the need for staining or external
contrast agents. It utilizes intrinsic biochemical information for
identification and classification of tissue. Furthermore, the technology is
non-destructive and the tissue can be used for independent analysis using
complimentary technologies. 40. Fourier transform infrared (FTIR)
spectroscopic imaging, the best known MIRSI technology has been used
successfully to classify cell subtypes in a variety of diseases including
breast, 41, lung 42, prostate 43, and colon 44 cancers. We hypothesize that it
is also useful to automatically segment and classify ovarian cancer tissue.
HGSC is the marked cytologic atypia with prominent mitotic activity in ovarian
tissue. The atypical nuclei are hyperchromatic with a threefold or greater
variation in nuclear size, and tumor giant cells are common. The phosphate
spectroscopic peaks (1080, 1201, 1236, 1262 cm-1) corresponding to nucleic
acids have a strong correlation to mitotic activity and spectroscopic imaging
has been shown to recognize increases mitoctic activity in a variety of
cancers.45, 46, 47.

FTIR imaging provides a spatial resolution limited by the diffraction-limit of
mid-infrared light ($2.5\text{\,}\mathrm{\SIUnitSymbolMicro m}$ \-
$11\text{\,}\mathrm{\SIUnitSymbolMicro m}$). Since typical cells are of
$\leavevmode\nobreak\ 5\mu m$ in size, FTIR imaging cannot provide sub-
cellular information that is potentially important to tissue analysis. Recent
MIRSI advances have resulted in the development of optical-Photothermal
infrared (O-PTIR) imaging that overcome this resolution limitation. This
technique combines a visible laser beam and a mid-infrared beam, in a pump-
probe architecture and estimates the sample’s absorbance by measuring the
change in intensity of the visible laser caused by the photothermal effect.
Therefore the image resolution is determined by the wavelength of visible
light ($0.5\text{\,}\mathrm{\SIUnitSymbolMicro m}$), which is much shorter
than the wavelength of IR light incident on the sample, allowing $5$x to $22$x
improvement in spatial resolution 48. The improved spatial resolution is
comparable to the optical microscope image as demonstrated in Figure 1. This
technology has been used previously for studying the chemistry of inorganic 2D
pervoskite, and allowed us to understand the edge emission phenomenon in
inorganic 2D pervoskite. 49 O-PTIR has also been successfully used in
ultrafast chemical imaging of live human ovarian cancer cells 50, where high
resolution is needed to analyze sub-cellular structures in a small sample.
However, our work is the first large scale (74 cancer patient) study of
clinical ovarian tissue biopsy samples each with a large sample areas (1 mm
diameter each).

Conventional machine learning algorithms, including random forest classifiers
(RF) and Bayesian classifiers, use per-pixel spectral data to classify tissue
regions. These approaches do not involve the use of spatial information
contained in the images. However, MIRSI provides spatial and spectroscopic
information simultaneously and requires innovations in classification
algorithms for optimal tissue segmentation. Convolutional neural networks
(CNNs) are deep learning models that take advantage of the local spatial
organization inside the images’ to compute their analysis. CNNs have
previously demonstrated to be successful in analysing hyperspectral imaging 51
and in mid-infrared spectroscopic tissue classification of breast cancer
tissues 52. Traditional network structures consist of alternating convolution
and pooling layers, followed by fully connected layers. In this work, we use
CNNs to determine what effect the improved resolution of O-PTIR imaging has on
tissue classification algorithms. We report the first application of the
O-PTIR technology to tissue classificaion. Furthermore, IR spectroscopic
imaging and CNNs have not previously been applied to tissue histology in
ovarian cancer. Previous work in spectroscopic analysis of ovarian cancer has
relied on point spectral data for identification of malignant tissue or
categorization of cancer grading 31, 27. To the best of our knowledge, this is
also the first application of MIRSI to automated histopathologic
classification of ovarian cancer tissue.

Fig. 1: Two tissue cores from ovarian cancer patients is stained with H&E and
presented in (a). Corresponding tissue images at
$1664\text{\,}{\mathrm{cm}}^{-1}$ from FT-IR is presented in (b) and data from
O-PTIR is in (c). The figure demonstrates a good correspondence between H&E
and spectroscopic imaging data. Moreover, the O-PTIR image has a significantly
higher resolution and finer tissue details are visible in (c) relative to
images in (b).

##  2 Materials and methods

A tissue microarray (TMA) consisting of ovarian tissue biopsies from different
patients, obtained from Biomax US, Rockville, MD (TMA ID: BC11115c), and was
imaged using O-PTIR spectroscopy. The dataset contained 100 cores from
separate patients with cases of normal, hyperplastic, dysplastic, and
malignant tumors tissues. The patient cohort was composed of women aged 29 to
69 years; ovarian tumor stages varied between stage I to stage IIIC;
histological subtypes included clear cell carcinoma, high-grade serous
carcinoma, and Mucinous adenocarcinoma. The paraffin-embedded samples were
deparaffinized before undergoing O-PTIR imaging. The data was collected using
both FTIR and O-PTIR techniques from the eighty deparaffinized tissue
biopsies. The corresponding adjacent histological section, stained with H&E,
was examined by pathologist expert in the field of ovarian cancer. Cell types
were identified from different disease stages. We trained a random forest
classifier, Support Vector Machine (SVM), K-Nearest Neighbor (KNN) and a CNN
model on each randomly selected pixels from left half of TMA and compared the
accuracies of the models by testing it on left half of the TMA.

###  2.1 O-PTIR imaging

Photothermal microscopy obtains a measurement of sample absorbance by
estimating the thermal expansion caused by absorption of infrared light using
a co-localized visible laser beam. The visible and IR laser are incident on
the sample collinearly as shown in Figure 2. The thermal expansion caused in
the sample due to IR absorbance causes variation in the refractive index due
to the photothermal effect. This change in refractive index is detected by
measuring the change in intensity of the back-reflected green laser (visible
laser) using a point detector. For the current experiment the optimal settings
of 80 percent of maximum IR laser power and 40 percent of maximum green laser
power with detector gain of 10x were used to get best signal to noise ratio
(SNR) without burning the sample.

Fig. 2: Schematic of the optical path of IR and green laser (532 nm) in our
O-PTIR instrument. A pulsed Quantum Cascade Laser (QCL) shown in (a) is the
source of mid-IR light which acts as a pump causing photothermal expansion at
sample. A Continuous Wave (CW) green laser shown in (b) is incident
collinearly on the sample and acts as a probe beam. A dichroic mirror (c)
combines the green and QCL light and focuses them on the sample (e) using a
reflective Cassagrain objective (d). The modulation in intensity of the green
light (f) that is scattered back from the sample enables measurement of IR
absorbance of the sample.

###  2.2 Improved Spatial Resolution

FTIR has been the standard spectroscopic imaging technique for characterizing
the material’s chemistry, while the H&E staining is the clinical gold standard
in disease diagnosis. Both of these techniques have their deficiencies, with
H&E staining being more qualitative and subjected to possible ambiguity in
interpretation. While FTIR provides information from the full range of near
and mid-infrared wavenumbers, it is bound by low spatial resolution caused by
the diffraction limit. The new O-PTIR technique gets a much higher spatial
resolution image comparable to that of the H&E stained image. Figure 3
demonstrates the increase in spatial resolution of O-PTIR image as compared to
the FTIR image of the same cancer core and comparable to that of the stained
image of the adjacent section of the corresponding cancer core. The spatial
differences in different classes of cells are evident in O-PTIR image as shown
in Figure 3.

Fig. 3: Spatial differences between different cell types in FT-IR image on the
left, H & E image in the center and O-PTIR image on the right. Cropped regions
around pixels from the same core in FT-IR and O-PTIR images collected at 1650
$cm^{-1}$.

###  2.3 Data acquisition

FTIR chemical images for the HD dataset of the TMA were acquired using an
Agilent Stingray imaging system consisting of a $680$-IR spectrometer
connected to a $620$-IR imaging microscope with a numerical aperture of
$0.62$. We performed the imaging of each core in the TMA at $16$ co-additions
in transmission mode. The spectral resolution was
$8\text{\,}{\mathrm{cm}}^{-1}$ with a pixel size of
$1.1\text{\,}\mathrm{\SIUnitSymbolMicro m}$ and a truncated spectral range of
$902\text{\,}{\mathrm{cm}}^{-1}$ to $3892\text{\,}{\mathrm{cm}}^{-1}$.We
collected the background scan at $128$ co-additions and ratioed to the single
beam data to remove spectral contributions from the substrate, atmosphere, and
globar source.

The O-PTIR dataset was acquired with a Photothermal mIRage microscope with a
silicon photodiode, a pixel size of $0.5\text{\,}\mathrm{\SIUnitSymbolMicro
m}$$\times$$0.5\text{\,}\mathrm{\SIUnitSymbolMicro m}$ and a $0.65$ numerical
aperture. A Quantum Cascade Laser (QCL) is the source of IR which sweeps
through the range of $902\text{\,}{\mathrm{cm}}^{-1}$ to
$1898\text{\,}{\mathrm{cm}}^{-1}$ emitting discreet wavenumbers. Each core was
imaged at five selected discrete wavenumbers. The system’s focus position does
not change during imaging, and the O-PTIR system is highly sensitive to focus
position; hence, any tilt in the sample can cause a gradient in data
collected. To avoid such issues, we calculated each core’s mean focus by
taking the mean of the focal position at $4$ corners of the core and the
center. We used 40% probe laser power and 80% infrared laser power as optimal
settings in order to get the highest signal to noise ratio (SNR) without
damaging the tissue. An image of the entire TMA acquired at the Amide I band
is shown in Figure 4.

We used an Aperio Scanscope system to acquire the Light microscope images of
the whole slide’s H&E chemically stained sections.

Fig. 4: O-PTIR microarray (8$\times$8) shown at band
$1664\text{\,}{\mathrm{cm}}^{-1}$. Date from 70 ovarian cancer patients is
shown. The biochemical variations in tissue is evident from differences in
color in the figure. Spectral bands provide biochemical information, which, in
combination with machine learning enable tissue-subtype identification and
disease diagnosis.

###  2.4 Feature Selection

Since the O-PTIR signal is detected using a point detector, the time taken to
collect an image of a single core varies between 90 to 100 minutes per
wavenumber. Hence collecting the full image of core at all wavenumbers in
$900\text{\,}{\mathrm{cm}}^{-1}$ to $1900\text{\,}{\mathrm{cm}}^{-1}$ at
$2\text{\,}{\mathrm{cm}}^{-1}$ spacing will take approximately 32 to 35 days.
To reduce the time of imaging, we need to collect fewer band images with
important biochemical information. These wavenumbers are determined by
analyzing FTIR spectra of ovarian tissue to determine absorbance peaks
corresponding to major functional groups. Based on this analysis we decided to
acquire O-PTIR data at wavenumbers $1162\text{\,}{\mathrm{cm}}^{-1}$,
$1234\text{\,}{\mathrm{cm}}^{-1}$, $1396\text{\,}{\mathrm{cm}}^{-1}$,
$1540\text{\,}{\mathrm{cm}}^{-1}$, and $1661\text{\,}{\mathrm{cm}}^{-1}$
(Figure 5), which correspond biochemically to glycogen, amide III, nucleic
acids and lipids, amide II, and amide I, respectively 53, 45.

Fig. 5: The figure shows absorption spectrum of ovarian tissue collected using
O-PTIR technique. The spectrum shows the IR absorption values (Y-axis) for the
wavenumbers (X-axis) in the fingerprint region. The marked peaks on the
spectrum correspond to biochemically relevant functional groups of glycogen at
$1162\text{\,}{\mathrm{cm}}^{-1}$, amide III at
$1234\text{\,}{\mathrm{cm}}^{-1}$, nucleic acids and lipids at
$1396\text{\,}{\mathrm{cm}}^{-1}$, amide II at
$1540\text{\,}{\mathrm{cm}}^{-1}$, and amide I at
$1661\text{\,}{\mathrm{cm}}^{-1}$ 53.

###  2.5 Data pre-processing

The standard preprocessing steps used in FTIR are not used in O-PTIR because
the data collected in FTIR is a direct measurement of IR absorption by the
sample ratioed to the incident source power while raw data obtained from
O-PTIR is an indirect measure of absorption of the IR wavenumber by the sample
using a visible laser. Since the power emitted by QCL is not uniform over all
the wavenumbers, the background spectra is collected with 10 co-additions to
measure the power of different wavenumbers in QCL. The raw data is then
normalized with the power spectra value corresponding to its wavenumber from
the background spectra. The range of values of raw O-PTIR signal typically
vary between $0$mV to $1.5m$V for the current set of parameters mentioned. But
there are always few outlier pixels which give very high signal which could be
happening due to high reflectivity of green laser from those pixels. This
leads to different range of O-PTIR signal values in different cores of TMA.
Normalizing all the bands with Amide I band
($1664\text{\,}{\mathrm{cm}}^{-1}$) will bring the range between $0$ and $1$
for all the cores significantly reducing the variation in range caused by
those few outlying pixels.

Note that some tissue biopsies are missing because of the collected data was
bad and larger areas of epithelium and stroma were intermixed so much that
presence of subtypes of stroma or other cells being present in annotations is
causing errors.

###  2.6 Data annotation

The tissue cores were labeled by two pathologists on adjacent tissue slices
stained with H&E as stroma or epithelium. The provided H&E annotations were
used to annotate O-PTIR data by manually aligning the chemically stained
annotated image with IR images. To generate training and testing data sets,
the left half of TMA was used for training and the right half of the TMA was
used for testing to maximize the variety of tissue the classifiers were
exposed to while maintaining independent training and testing sets.

###  2.7 Classification Models

As a baseline comparison, an SVM classifier is trained on $10000$ randomly
selected pixels from training dataset summarized in Table 1. 10,000 samples
per class are used to train the classifier to balance the number of samples
for each class. Similarly, we trained a random forest classifier with $100$
trees using 10,000 samples per class. The inputs to the classifiers were five
element vectors containing the IR absorption values at each pixel and the
corresponding pixel label. Further increasing the number of samples per class
did not show any significant increases in the overall accuracy of these
classifiers.

We designed a CNN model based on structure and hyperparameters which have
demonstrated success in breast cancer histology 52. As inputs to the networks,
we cropped a $32\times 32$ region around the center pixel in O-PTIR data to
leverage the local spatial information. The base network structure used
consists of a convolution layer with $32$-feature maps, followed by a $2\times
2$ max-pooling layer, followed by a convolution layer with $32$-feature maps
and another convolution layer with $64$-feature maps, then ends with a fully
connected layer with $64$ nodes and a softmax layer to output classification
probabilities. The network for O-PTIR data contains an extra $32$-feature
convolution layer and $2\times 2$ max pooling layer before the $64$-feature
convolution layers to adjust to the larger input size, ensuring that the
feature size in the final layers are equivalent in both models (Figure 6).

Fig. 6: Schematic presentation of the CNN architecture used for classification
of O-PTIR data. A spatial region of size $33\times 33$ is cropped around each
pixel. Data cubes of size $33\times 33\times\times 5$ are fed into the first
convolution layer. Each input is convolved with filters of size $3\times 3$
outputting 32 feature maps. The following layer is a max pooling layer, which
reduces the spatial dimensions by half. Feature extraction continues with
three more convolution layers consisting of one $32$ and two $64$ feature maps
consecutively. The extracted features are then flattened and fed to a fully
connected layer with $128$ units. The last layer, softmax, consisting of $2$
units (number of classes) outputs a vector of class probabilities. At the end,
maximum probability is used to map each input pixel to its corresponding class
labels.

###  2.8 Hyperparameters

We chose the following hyperparameters for the Random Forest model: The random
forest classifier is trained on $10000$ randomly selected pixels from training
dataset summarized in Table 1 with $100$ trees. The inputs to the classifiers
were five element vectors containing the IR absorption values at each pixel
and the corresponding pixel label.

We chose the following hyperparameters for the CNN model:

  1. 1.

Optimization: The Adam optimizer was used 54.

  2. 2.

Dropout: The networks had a dropout layer before the fully connected layer
with a keep probability of $0.5$. This layer aids regularization and prevents
overfitting by randomly disabling nodes in the first and last layers in
training. Dropout layer were included before the softmax layer and before the
first $64$-feature convolution layer.

  3. 3.

Non-linearity: Rectified Linear Unit (relu) activation function is used as
activation function for each layer.

  4. 4.

Weight initialization: The initial weights of keras layers are initialized
using Randomnormal class from the built-in initializer in the module. This
initializer generates tensors with a normal distribution with mean at 0.0 and
standard deviation of 0.05.

  5. 5.

Batch Size: The networks trained with a batch size of $128$ images.

  6. 6.

Epochs: The networks were trained for $8$ epochs. Data were randomly shuffled
between epochs.

###  2.9 Implementation

All data pre-processing was performed using our open-source SIproc software
55, implemented in C++ and CUDA. Training and testing was performed in Python
using open-source software packages. The CNNs were implemented in python with
the Keras library 56, built on TensorFlow 57. Random forest classifiers and
accuracy scores were computed using the Scikit-learn library 58. The
experiments were run on a NVIDIA Tesla K40m GPU.

##  3 Results

Considering our primary goal to leverage improved spatial resolution in O-PTIR
imaging to classify cell types in O-PTIR images, we used overall accuracy (OA)
and receiver operating characteristic (ROC) curves as evaluation metrics. The
overall accuracy (OA) would be useful for both binary, and multi-class
classification as it represents the percentage of pixels that were mapped
correctly to the appropriate class. The ROC curves delineate the correlation
between specificity and sensitivity to ascertain acceptable false positive and
true positive indicators. We tested classification algorithms such as RF and
SVM, which are spectral-based, and CNN classifier which uses spectral and
spatial features on the O-PTIR data. The overall accuracy of these classifiers
is compared in Table 2 to evaluate the highest performing classifier. CNN
outperforms RF and SVM classifiers in terms of overall and classwise accuracy.

The per class accuracy values and the overall accuracy obtained after
classification of testing dataset is summerized in Table 2. The ROC curves and
AUC values for all the classifiers and datasets used is shown in Figure 8
where it can be seen that CNN overcomes SVM and RF in AUC values for all the
datasets. The overall accuracy with CNN shows a very high improvement of
approximately 40-50% in comparision with RF and SVM. The low overall accuracy
scores for RF ($53.21\%$) and SVM ($45.57\%$) can be attributed to using
spectral information from only 5 wavenumbers instead of all the wavenumbers in
the spectrum from 900-1900. Meanwhile the high overall accuracy achieved by
CNN ($94.61\%$) is due to utilization of both spectral and spatial features.

Table 1: The table shows the number of pixels available for training and testing from O-PTIR data, separated by class. Testing and training sets were built by splitting the entire TMA in half and assigning one half to each set. To prevent class bias in training, equivalent numbers of pixels were selected from each class, $10,000$ per class in random forest classifiers and, in CNNs, $400,000$ pixels per class from O-PTIR data. Class | Training | Testing
---|---|---
Epithelium | $22,766,257$ | $19,249,625$
Stroma | $11,001,575$ | $8,719,719$
Total | $33,767,832$ | $27,969,344$
Table 2: Accuracy scores of random forest models by data type across 80 experiments, trained with $10,000$ randomly selected samples per class and in CNNs, $400,000$ pixels per class. The random forest classifier is much less accurate than the CNN, likely because the increased resolution allows an increase in within-class spectral diversity, making classification more difficult without use of spatial features. Class | SVM | Random Forest | CNN
---|---|---|---
Epithelium | $80.31\pm 0.18$ | $60.18\pm 0.29$ | $95.33\pm 1.52$
Stroma | $29.84\pm 0.26$ | $44.27\pm 0.21$ | $93.00\pm 1.97$
Total | $45.57\pm 0.3$ | $53.21\pm 0.05$ | $94.61\pm 0.82$

The classification model obtained from CNN was used to classify the entire TMA
of ovarian tissue punches and not not only the annotated areas. Figure 7 shows
the classification results which are consistent with the H&E stained image
dataset. The results show that epithelium and stroma are well classified using
the CNN classifier and the distinction in these classes is comparable to that
of the visible stained images. In order to quantitate the efficiency of
different classifiers, we used Area Under the Curve (AUC) and Receiver
Operating Characteristic plot (ROC) curves as metrics. The CNN classifier
demonstrates high value with AUC of $0.98$ and $0.58$ for the RF classifier
and the visual representation of ROC is demonstrated in Figure 8.

Fig. 7: O-PTIR classification using a convolutional neural network with both
spectral and spatial features. The upper row of the image shows the H&E
stained adjacent sections and the lower row shows the Individual classified
cores in false-color. Fig. 8: ROC curves and associated AUC values for binary
classification of tissue type, separated by classifier type and datasets used.
Due to the use of two-class models, each tissue class curve is a reflection of
the curve from the other class, and thus the AUC values are equal across
tissue class. CNN classifiers exhibit superior results to the random forest
(RF) classifiers, indicating that spatial information is important in
distinguishing tissue types.

Fig. 9: Stromal and Epithelial Ratios are plotted as a function of
pathological FIGO stage of ovarian cancer.Stromal ratios in panels a-b are
calculated by dividing the number of pixels classified as stroma by the total
number of pixels, while epithelial ratios in panels c-d are calculated by
dividing the number of pixels classified as epithelial by the number of total
pixels.The (a) and (c) graph shows a non-linear fit for stromal ratio and
epithelial ratio to cancer stage. There is a substantial decrease in stromal
ratio and a dramatic increase in epithelial ratio during the early stage of
cancer from normal tissue until stage II, which is plotted with the mean and
error bands in (b) and (d). P values calculated in figure (b) show that
stromal and epithelial amounts significantly differ in normal ovary versus
early stages of cancer.Since this is a binary classification, p values remain
the same for the plots in panel b and d.

We calculated the ratio of Stroma to Stroma and Epithelium and plotted it
against the stage of cancer as shown in Figure 9. Since the stage of cancer is
qualitative, we assigned a quantitative variable, and the curve is fitted
using non-linear regression to observe the trend in the ratio as a function of
the stage of cancer which can be observed from the plot in figure 8 (a). The
plot in figure 8 (b) shows the trendline with error bands for each stage,
demonstrating the dramatic reduction of stroma during the early stages of
cancer relative to normal healthy tissue.

All data pre-processing was performed using our open-source SIproc software
55, implemented in C++ and CUDA. Training and testing was performed in Python
using open-source software packages. The CNNs were implemented in python with
the Keras library 56, built on TensorFlow 57. Random forest and SVM
classifiers and accuracy scores were computed using the Scikit-learn library
58. The CNN classifier’s performance was calculated by testing the classifiers
on ten different sets of randomly selected training pixels and averaging the
overall accuracy run on a NVIDIA Tesla K40m GPU.

##  4 Discussion

FTIR imaging captures an absorption spectrum across a range of wavenumbers at
each point, while O-PTIR allows us to obtain high-resolution absorption images
of the sample at discrete wavenumbers. O-PTIR techniques also require more
time to scan equivalent areas relative to FTIR, limiting the amount of data
that can be reasonably collected. The effect of the improvement in resolution
on tissue classification has not yet been studied. The classification accuracy
of the CNN model ($\approx 90$%–$94$%) demonstrate that ovarian cancer tissue
can be classified using spectral and spatial data. While this result has been
found in other tissue types, this work extends the technique’s application to
ovarian tissue. The improvement seen in CNNs over random forest classifiers
demonstrates that the spatial data hid in the images contains essential
information about the tissue type. CNNs take advantage of this information by
extracting spectral-spatial features characteristic of tissue types in the
Area surrounding a pixel, thus achieving improved accuracy. Additionally, as
the accuracy of random forest classifiers trained on O-PTIR data is much lower
than CNN, the greater resolution in O-PTIR imaging likely causes an increase
in within-class spectral diversity, meaning the classes are broader and more
difficult for a non-spatial model to distinguish. This effect is overcome by
the convolutional models, which use spatial features in classification. The
comparison of networks trained on O-PTIR data demonstrated a statistically
significant improvement in classification accuracy. Therefore, this work shows
that the higher resolution of O-PTIR imaging provides additional spectral
information, which aids in tissue classification. This result is promising for
using IR spectroscopic imaging techniques with improved resolution in refining
the accuracy of tissue classification algorithms.

There are multiple ways of improving classification accuracy, and the quality
of Annotations has a significant effect on training and testing the accuracy
of networks. By improving the accuracy of our epithelial and stromal tissue
annotations, network performance would likely improve. Additionally, as
annotations are drawn on adjacent tissue sections and manually aligned, there
is an inherent error in the ground truth classes, decreasing classifier
performance. We also selected five wavenumbers from the mid-infrared spectrum
to image with the O-PTIR system and similarly limited the FTIR data to the
same wavenumbers as O-PTIR data collection is time-intensive, and the time
required to image is directly proportional to the number of wavenumbers.
Additional spectral information to distinguish tissue classes could be
achieved by adding additional IR wavenumbers to the data. Our method for
wavenumber selection was based on tissue qualities and spectral appearance.
More rigorous methods for selection which analyze the deterministic power of a
wavenumber could contribute to classification improvement. Further research is
necessary for optimal wavenumber selection for O-PTIR imaging.

The effect of improved spatial resolution may be more profound when
considering other tissue types in classification, such as blood, adipocytes,
or myofibroblasts. Designing classifiers with these tissue types could provide
more insight into the magnitude of improvement afforded by additional
resolution. Further investigation is necessary in these areas to more fully
explore the benefits of O-PTIR spectroscopic imaging.

Stroma to Epithelium ratio is used as one of the important clinical markers to
determine the stage of cancer, and from Figure 9 we can demonstrate that there
is a significant decrease in the stroma percentage as there is the progression
in stages of cancer relative to normal healthy tissue. The Stroma ratio is
evaluated per core for each stage of cancer after the CNN model has done
complete classification on the entire core. Since CNN output is seen as
probabilities, we set a threshold of 80 percent to determine whether a given
pixel belongs to epithelium or stroma and calculated the ratio of stroma to
the epithelium and stroma pixels while marking the pixels with probabilities
less than 80 percent as other class which could correspond to the presence of
necrotic tissue, lymphocytes and blood vessels in the tissue.

##  5 Conclusion

The introduction of the O-PTIR spectroscopic imaging technique has
significantly improved spatial resolution over the FTIR imaging technique. It
is thus essential to consider whether this improvement is consequential in the
application. This study is the first analysis of the effect of resolution
improvement in IR spectroscopic image data in classification of human tissue
to the best of our knowledge. O-PTIR imaging with CNN classification can
provide a meaningful improvement in computational and quantitative
histopathology, and preliminary results in ovarian tissue histopathology using
infrared spectroscopy are promising. This project demonstrates the utility of
O-PTIR in ovarian tissue analysis and opens the door for further ovarian
cancer studies using O-PTIR.

## Author Contributions

We strongly encourage authors to include author contributions and recommend
using CRediT for standardised contribution descriptions. Please refer to our
general author guidelines for more information about authorship.

## Conflicts of interest

There are no conflicts to declare.

## Acknowledgements

This work is supported in part by the NLM Training Program in Biomedical
Informatics and Data Science T15LM007093 (RM, RR), the Cancer Prevention and
Research Institute of Texas (CPRIT) #RR170075 (RR), National Institutes of
Health #R01HL146745 (DM), the National Science Foundation CAREER Award
#1943455, NLM Training Program in Biomedical Informatics and Data Science
#T15LM007093 SB), the NLM 2019 Data Science and Biomedical Informatics
Undergraduate Summer Research Program (MB) grant 3T15LM007093-27S1, National
Institutes of Health P50 CA217685 (AKS), P30CA016672 (AKS), American Cancer
Society Research Professor Award (AKS), Frank McGraw Memorial Chair in Cancer
Research (AKS), Moon Shot in Ovarian Cancer (AKS).

## Notes and references

  * Torre _et al._ 2018 L. A. Torre, B. Trabert, C. E. DeSantis, K. D. Miller, G. Samimi, C. D. Runowicz, M. M. Gaudet, A. Jemal and R. L. Siegel, _CA: a cancer journal for clinicians_ , 2018, 68, 284.
  * Lheureux _et al._ 2019 S. Lheureux, C. Gourley, I. Vergote and A. Oza, _Lancet (London, England)_ , 2019, 393, 1240.
  * Lee _et al._ 2019 J.-M. Lee, L. Minasian and E. C. Kohn, _Cancer_ , 2019, 125, 4623\.
  * du Bois _et al._ 2009 A. du Bois, A. Reuss, E. Pujade-Lauraine, P. Harter, I. Ray-Coquard and J. Pfisterer, _Cancer_ , 2009, 115, 1234–1244.
  * Kehoe _et al._ 2015 S. Kehoe, J. Hook, M. Nankivell, G. C. Jayson, H. Kitchener, T. Lopes, D. Luesley, T. Perren, S. Bannoo, M. Mascarenhas _et al._ , _Lancet (London, England)_ , 2015, 386, 249–257.
  * Vergote _et al._ 2010 I. Vergote, C. G. Tropé, F. Amant, G. B. Kristensen, T. Ehlen, N. Johnson, R. H. Verheijen, M. E. van der Burg, A. J. Lacave, P. B. Panici _et al._ , _The New England journal of medicine_ , 2010, 363, 943–953.
  * Luo _et al._ 2016 Z. Luo, Q. Wang, W. Lau, B. Lau, L. Xu, L. Zhao, H. Yang, M. Feng, Y. Xuan, Y. Yang _et al._ , _Cancer letters_ , 2016, 377, 174.
  * Banks 2001 E. Banks, _Methods in Molecular Medicine_ , 2001, 39, 3–11.
  * Heintz _et al._ 2001 A. P. Heintz, F. Odicino, P. Maisonneuve, U. Beller, J. L. Benedet, W. T. Creasman, H. Y. Ngan, M. Sideri and S. Pecorelli, _Journal of Epidemiology and Biostatistics_ , 2001, 6, 107–138.
  * Malpica 2008 A. Malpica, _International journal of gynecological pathology: official journal of the International Society of Gynecological Pathologists_ , 2008, 27, 175–181.
  * Hernandez _et al._ 1984 E. Hernandez, B. Bhagavan, T. Parmley and N. Rosenshein, _Gynecologic oncology_ , 1984, 17, 117–123.
  * Malpica _et al._ 2004 A. Malpica, M. T. Deavers, K. Lu, D. C. Bodurka, E. N. Atkinson, D. M. Gershenson and E. G. Silva, _The American journal of surgical pathology_ , 2004, 28, 496–504.
  * Taylor _et al._ 1999 R. Taylor, J. Zeller, R. Lieberman and D. O’Connor, _Gynecologic oncology_ , 1999, 74, 3–6.
  * Zeppernick and Meinhold-Heerlein 2014 F. Zeppernick and I. Meinhold-Heerlein, _Archives of gynecology and obstetrics_ , 2014, 290, 839–842.
  * Malpica _et al._ 2007 A. Malpica, M. T. Deavers, C. Tornos, R. J. Kurman, R. Soslow, J. D. Seidman, M. F. Munsell, E. Gaertner, D. Frishberg and E. G. Silva, _The American journal of surgical pathology_ , 2007, 31, 1168–1174.
  * Bera _et al._ 2019 K. Bera, K. A. Schalper, D. L. Rimm, V. Velcheti and A. Madabhushi, _Nature reviews. Clinical oncology_ , 2019, 16, 703–715.
  * Wu _et al._ 2018 M. Wu, C. Yan, H. Liu and Q. Liu, _Bioscience Reports_ , 2018, 38, .
  * Komura and Ishikawa 2018 D. Komura and S. Ishikawa, _Computational and Structural Biotechnology Journal_ , 2018, 16, 34.
  * Du _et al._ 2018 Y. Du, R. Zhang, A. Zargari, T. C. Thai, C. C. Gunderson, K. M. Moxley, H. Liu, B. Zheng and Y. Qiu, _Annals of biomedical engineering_ , 2018, 46, 1988–1999.
  * Chen _et al._ 2020 X. Chen, R. Zhang, K.-M. Fung, H. Liu, B. Zheng and Y. Qiu, Biophotonics and Immune Responses XV, 2020, p. 112410F.
  * Xu _et al._ 2017 J. Xu, C. Zhou, B. Lang and Q. Liu, _Deep Learning and Convolutional Neural Networks for Medical Image Computing_ , Springer, 2017, pp. 73–95.
  * Fiore _et al._ 2012 C. Fiore, D. Bailey, N. Conlon, X. Wu, N. Martin, M. Fiorentino, S. Finn, K. Fall, S.-O. Andersson, O. Andren _et al._ , _Journal of clinical pathology_ , 2012, 65, 496–502.
  * ur Rehman _et al._ 2012 I. ur Rehman, Z. Movasaghi and S. Rehman, _Vibrational spectroscopy for tissue analysis_ , CRC press, 2012.
  * D’Souza and Koyoshi 2009 M. J. D’Souza and F. Koyoshi, _Pharmaceutical reviews_ , 2009, 7, http–www.
  * Morais _et al._ 2019 C. L. Morais, P. L. Martin-Hirsch and F. L. Martin, _Analyst_ , 2019, 144, 2312–2319.
  * Paraskevaidi _et al._ 2018 M. Paraskevaidi, K. M. Ashton, H. F. Stringfellow, N. J. Wood, P. J. Keating, A. W. Rowbottom, P. L. Martin-Hirsch and F. L. Martin, _Talanta_ , 2018, 189, 281–288.
  * Maheedhar _et al._ 2008 K. Maheedhar, R. A. Bhat, R. Malini, N. Prathima, P. Keerthi, P. Kushtagi and C. M. Krishna, _Photomedicine and Laser Surgery_ , 2008, 26, 83–90.
  * Krishna _et al._ 2007 C. M. Krishna, G. Sockalingum, R. A. Bhat, L. Venteo, P. Kushtagi, M. Pluot and M. Manfait, _Analytical and bioanalytical chemistry_ , 2007, 387, 1649–1656.
  * Flower _et al._ 2011 K. R. Flower, I. Khalifa, P. Bassan, D. Démoulin, E. Jackson, N. P. Lockyer, A. T. McGown, P. Miles, L. Vaccari and P. Gardner, _Analyst_ , 2011, 136, 498–507.
  * Gajjar _et al._ 2013 K. Gajjar, J. Trevisan, G. Owens, P. J. Keating, N. J. Wood, H. F. Stringfellow, P. L. Martin-Hirsch and F. L. Martin, _Analyst_ , 2013, 138, 3917–3926.
  * Theophilou _et al._ 2016 G. Theophilou, K. M. Lima, P. L. Martin-Hirsch, H. F. Stringfellow and F. L. Martin, _Analyst_ , 2016, 141, 585–594.
  * Lima _et al._ 2015 K. M. Lima, K. B. Gajjar, P. L. Martin-Hirsch and F. L. Martin, _Biotechnology progress_ , 2015, 31, 832–839.
  * Klein _et al._ 2019 O. Klein, F. Kanter, H. Kulbe, P. Jank, C. Denkert, G. Nebrich, W. D. Schmitt, Z. Wu, C. A. Kunze, J. Sehouli _et al._ , _PROTEOMICS–Clinical Applications_ , 2019, 13, 1700181.
  * Zeitoune _et al._ 2017 A. A. Zeitoune, J. S. Luna, K. Sanchez Salas, L. Erbes, C. L. Cesar, L. A. Andrade, H. F. Carvahlo, F. Bottcher-Luiz, V. H. Casco and J. Adur, _Cancer informatics_ , 2017, 16, 1176935117690162.
  * Pouli _et al._ 2019 D. Pouli, E. M. Genega, T. B. Sullivan, K. M. Rieger-Christ, V. Wright, I. Georgakoudi and T. Schnelldorfer, _Biomedical Optics Express_ , 2019, 10, 4479.
  * Tilbury and Campagnola 2015 K. Tilbury and P. J. Campagnola, _Perspectives in medicinal chemistry_ , 2015, 7, PMC–S13214.
  * Huttunen _et al._ 2018 M. J. Huttunen, A. Hassan, C. W. McCloskey, S. Fasih, J. Upham, B. C. Vanderhyden, R. W. Boyd and S. Murugkar, _Journal of biomedical optics_ , 2018, 23, 066002.
  * Jokerst _et al._ 2012 J. V. Jokerst, A. J. Cole, D. Van de Sompel and S. S. Gambhir, _ACS nano_ , 2012, 6, 10366–10377.
  * Oseledchyk _et al._ 2017 A. Oseledchyk, C. Andreou, M. A. Wall and M. F. Kircher, _ACS nano_ , 2017, 11, 1488–1497.
  * Petibois and Déléris 2006 C. Petibois and G. Déléris, _TRENDS in Biotechnology_ , 2006, 24, 455–462.
  * Benard _et al._ 2014 A. Benard, C. Desmedt, M. Smolina, P. Szternfeld, M. Verdonck, G. Rouas, N. Kheddoumi, F. Rothé, D. Larsimont, C. Sotiriou _et al._ , _Analyst_ , 2014, 139, 1044–1056.
  * Großerueschkamp _et al._ 2015 F. Großerueschkamp, A. Kallenbach-Thieltges, T. Behrens, T. Brüning, M. Altmayer, G. Stamatis, D. Theegarten and K. Gerwert, _Analyst_ , 2015, 140, 2114–2120.
  * Baker _et al._ 2009 M. J. Baker, E. Gazi, M. D. Brown, J. H. Shanks, N. W. Clarke and P. Gardner, _Journal of biophotonics_ , 2009, 2, 104–113.
  * Krafft _et al._ 2008 C. Krafft, D. Codrich, G. Pelizzo and V. Sergo, _Journal of biophotonics_ , 2008, 1, 154–169.
  * Baker _et al._ 2014 M. J. Baker, J. Trevisan, P. Bassan, R. Bhargava, H. J. Butler, K. M. Dorling, P. R. Fielden, S. W. Fogarty, N. J. Fullwood, K. A. Heys _et al._ , _Nature protocols_ , 2014, 9, 1771.
  * Pounder _et al._ 2016 F. N. Pounder, R. K. Reddy and R. Bhargava, _Faraday Discussions_ , 2016, 187, 43–68.
  * Pahlow _et al._ 2018 S. Pahlow, K. Weber, J. Popp, B. R. Wood, K. Kochan, A. Rüther, D. Perez-Guaita, P. Heraud, N. Stone, A. Dudgeon, B. Gardner, R. Reddy, D. Mayerich and R. Bhargava, _Applied spectroscopy_ , 2018, 72, 52–84.
  * Zhang _et al._ 2016 D. Zhang, C. Li, C. Zhang, M. N. Slipchenko, G. Eakins and J.-X. Cheng, _Science advances_ , 2016, 2, e1600521.
  * Qin _et al._ 2020 Z. Qin, S. Dai, C. C. Gajjela, C. Wang, V. G. Hadjiev, G. Yang, J. Li, X. Zhong, Z. Tang, Y. Yao _et al._ , _Chemistry of Materials_ , 2020\.
  * Bai _et al._ 2019 Y. Bai, D. Zhang, L. Lan, Y. Huang, K. Maize, A. Shakouri and J.-X. Cheng, _Science advances_ , 2019, 5, eaav7127.
  * Hu _et al._ 2015 W. Hu, Y. Huang, L. Wei, F. Zhang and H. Li, _Journal of Sensors_ , 2015, 2015, .
  * Berisha _et al._ 2019 S. Berisha, M. Lotfollahi, J. Jahanipour, I. Gurcan, M. Walsh, R. Bhargava, H. Van Nguyen and D. Mayerich, _Analyst_ , 2019, 144, 1642–1653.
  * Zohdi _et al._ 2015 V. Zohdi, D. R. Whelan, B. R. Wood, J. T. Pearson, K. R. Bambery and M. J. Black, _PLoS One_ , 2015, 10, e0116491.
  * Kingma and Ba 2019 D. P. Kingma and J. A. Ba, _arXiv preprint arXiv:1412.6980_ , 2019, 434, .
  * Berisha _et al._ 2017 S. Berisha, S. Chang, S. Saki, D. Daeinejad, Z. He, R. Mankar and D. Mayerich, _Analyst_ , 2017, 142, 1350–1357.
  * Chollet _et al._ 2015 F. Chollet _et al._ , _Keras_ , https://keras.io, 2015.
  * Abadi _et al._ 2015 M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu and X. Zheng, _TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems_ , 2015, http://tensorflow.org/, Software available from tensorflow.org.
  * Pedregosa _et al._ 2011 F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot and E. Duchesnay, _Journal of Machine Learning Research_ , 2011, 12, 2825–2830.