# PCA: Semi-supervised Segmentation with
Patch Confidence Adversarial Training

Zihang Xu, Zhenghua Xu, , Shuo Zhang, Thomas Lukasiewicz This work was
supported by the National Natural Science Foundation of China under the grant
61906063, by the Natural Science Foundation of Hebei Province, China, under
the grant F2021202064, by the “100 Talents Plan” of Hebei Province, China,
under the grant E2019050017, by the Natural Science Foundation of Tianjin
City, China, under the grant 19JCQNJC00400, and by the Key Research and
Development Project of Hainan Province, China, under the grant
ZDYF2022SHFZ015. This work was also partially supported by the AXA Research
Fund. (Corresponding author: Zhenghua Xu, e-mail:
zhenghua.xu@hebut.edu.cn.)Zihang Xu, Shuo Zhang and Zhenghua Xu are with State
Key Laboratory of Reliability and Intelligence of Electrical Equipment and
Tianjin Key Laboratory of Bioelectromagnetic Technology and Intelligent
Health, Hebei University of Technology, China.Thomas Lukasiewicz is with
Department of Computer Science, University of Oxford, United Kingdom.

###### Abstract

Deep learning based semi-supervised learning (SSL) methods have achieved
strong performance in medical image segmentation, which can alleviate doctors’
expensive annotation by utilizing a large amount of unlabeled data. Unlike
most existing semi-supervised learning methods, adversarial training based
methods distinguish samples from different sources by learning the data
distribution of the segmentation map, leading the segmenter to generate more
accurate predictions. We argue that the current performance restrictions for
such approaches are the problems of feature extraction and learning
preference. In this paper, we propose a new semi-supervised adversarial method
called Patch Confidence Adversarial Training (PCA) for medical image
segmentation. Rather than single scalar classification results or pixel-level
confidence maps, our proposed discriminator creates patch confidence maps and
classifies them at the scale of the patches. The prediction of unlabeled data
learns the pixel structure and context information in each patch to get enough
gradient feedback, which aids the discriminator in convergent to an optimal
state and improves semi-supervised segmentation performance. Furthermore, at
the discriminator’s input, we supplement semantic information constraints on
images, making it simpler for unlabeled data to fit the expected data
distribution. Extensive experiments on the Automated Cardiac Diagnosis
Challenge (ACDC) 2017 dataset and the Brain Tumor Segmentation (BraTS) 2019
challenge dataset show that our method outperforms the state-of-the-art semi-
supervised methods, which demonstrates its effectiveness for medical image
segmentation.

###### Index Terms:

Semi-Supervised learning, adversarial learning, medical image segmentation.

##  I Introduction

Segmentation, identifying interesting regions with anatomical or pathological
structures from medical images, is the basic task of medical image analysis,
which is of great significance for computer-assisted diagnosis, surgery
simulation, and treatment planning. Recently, deep learning methods [1, 2]
have achieved excellent results in various fields of medical applications,
which are trained with various typical segmentation networks in a fully
supervised way, (e.g., FCNs [3], U-Nets [4] and GAN [5]). The success of the
deep network model is due to the deep and wide network, which is highly
dependent on large-scale and high-quality pixel annotation data. However, the
lack of sufficient labeled data has always been a major challenge for medical
image segmentation.

Figure 1: Different output styles of general adversarial training (ImageGAN),
our method (PatchGAN), and confidence map based adversarial training
(PixelGAN).

To reduce the annotation burden on doctors, many methods have been proposed
for medical image segmentation applications. Considering that unlabeled data
is usually abundant and easily available, many researchers focused on
implementing segmentation tasks in a semi-supervised learning fashion. The
main idea of semi-supervised segmentation is to learn from a limited amount of
labeled data and a large amount of unlabeled data to improve the accuracy of
segmentation. According to the training manner, common semi-supervised
segmentation methods can be divided into self-training [11], consistency
regularization [12], co-training [13] and adversarial training [14].

Recently, many researchers have focused on semi-supervised segmentation
methods based on adversarial training [15, 16, 18, 21, 28, 29], which shows
great potential for improving semantic segmentation. Specifically, these
methods are inspired by the Generative Adversarial Network (GAN) [5], which
consists of a generator and a discriminator. The generator network uses a
semantic segmentation network to generate the probability maps of the semantic
labels. The discriminator network generates the probability that the input is
real by distinguishing generated samples from target ones. Through the
discriminator, the quality of the predicted segmentation map can be
effectively assessed. However, the model lacks stability due to the
insufficient supervision of the discriminator, which brings challenges to
semi-supervised adversarial training. Some recent studies have attempted to
improve the performance of the discriminator [15, 16, 18]. For example, Son et
al. [15] connected prediction, inverse prediction, and image, forcing the
discriminator to learn the mapping relationship between image and prediction,
thus alleviating this problem. Zhang et al. [16] multiplied the prediction
with the image to obtain an image containing only foreground information.
Inspired by the attention mechanism [17], Han et al. [18] proposed a dual-
attentive-fusion block that has two independent spatial attention paths on the
predicted segmentation map and leverages the corresponding original image. Nie
et al. [28] introduced the concept of a confidence map to supervise the
learning of unannotated data.

However, all of the above methods have the following two shortcomings: (i)
Feature mining problem: For the discriminator, its purpose is to distinguish
whether the generated sample is real or not, and its output can represent the
probability that the input is real. However, regardless of whether the
segmentations generated from the unlabeled data are accurate or not, it is
unreasonable to regard all pixels as a negative class, which makes the
discriminator difficult to dig out the features that distinguish the
distribution of the two data domains. When its output is not a scalar result
but a pixel-level confidence map, i.e. classifying each pixel, unlabeled data
can use the pseudo label mask generated based on the confidence map to train
in a self-learning manner. However, the generator can easily deceive the
discriminator by reducing the information entropy of segmentation maps. That
means the discriminator doesn’t dig out enough features to generate reliable
confidence maps to reflect the correct probability of the segmentation result.
This motivates us to consider how to generate more reliable evaluation
results. (ii) Learning preference problem: The original image directly
concatenated or multiplied with the segmentation map as the input of the
discriminator not only introduces the relevant information of the segmentation
map and the input image but also makes the discriminator generate learning
preferences. This causes the discriminator to change the optimization goal of
training, which affects the accuracy of the segmentation network. From the
results, not only the segmentation accuracy has decreased, but also the
problem of under-fitting, which is unacceptable in semi-supervised medical
image segmentation and may be difficult to apply to clinical practice.

Inspired by these, we propose a patch confidence adversarial training
framework for semi-supervised medical image segmentation. We improve the
original adversarial training strategy from the input and output of the
discriminator, addressing the two problems mentioned above, respectively.
First, to better evaluate segmentation, we introduce the idea of PatchGAN [30]
. The discriminator classifies each image patch independently, so that it
generates a patch confidence map instead of a scalar classification result or
pixel-level confidence map. The discriminator penalizes structure at the scale
of patches, guiding the generator to optimize for different patch areas. At
the same time, with the help of the context information between pixels, the
generator is forced to focus on high-frequency features, and it is no longer
easy to fool the discriminator, which helps the discriminator converge to an
ideal state. In addition, to utilize the image information more effectively,
we also add the weighted image and the segmentation map. This operation
balances the information contained in the image and segmentation maps. In that
case, the discriminator will focus on the relationship between the image and
the segmentation, learning how to map unlabeled data to an expected
distribution. In adversarial training, improving the performance of the
discriminator will help the deep model achieve better segmentation
performance.

The major contributions can be summarized as follows:

  * •

We propose a novel and universal semi-supervised method, namely PCA, for semi-
supervised medical image segmentation tasks. The proposed method effectively
addresses two important shortfalls, including the feature mining problem and
the learning preference problem.

  * •

Inspired by CGAN [42] and PatchGAN, we propose the patch confidence map and
pixel additive blending, making the discriminator easily converge to a
desirable status and guiding the prediction of the unlabeled data to fit the
expected data distribution, which can significantly improve the performance of
semi-supervised segmentation.

  * •

A large number of experiments on the ACDC 2017 dataset and BraTS 2019 dataset
prove that our semi-supervised method is efficient. We found that the designed
patch confidence map is more effective than the most advanced adversarial
training strategy. In addition, we achieve the best results compared to other
state-of-the-art methods in semi-supervised segmentation.

Figure 2: a) The schematic view of our proposed patch confidence adversarial
training model (take cardiac image as an example). b) Each value of the output
matrix from PatchGAN discriminator represents the probability of whether the
corresponding segmentation patch is real or fake. c) The process of pixel
additive blending calculation.

##  II RELATED WORKS

As a basic task, medical image segmentation is of great significance in many
biomedical applications. At present, almost all segmentation frameworks are
based on deep learning, which has achieved impressive performance improvements
on various medical image segmentation tasks and set the new state-of-the-art.
However, there is a lack of a large amount of data annotation in the medical
field, which limits the performance of the model. In this section, we will
focus on reviewing related methods and the latest developments in semi-
supervised learning, and then discuss semi-supervised methods based on
adversarial training, which are most relevant to our work.

###  II-A Semi-supervised segmentation for medical images

In semi-supervised medical image segmentation tasks, it is usually assumed
that only a small part of the training images have complete pixel-level
annotations, but there are also a large number of unlabeled images that can be
used to improve the performance of the model. Since unlabeled data does not
need to be manually labeled by the doctor, unlabeled data can be used at a low
cost to improve performance. The main challenge in this scenario is how to
effectively use a large amount of unlabeled data, which is also the main
difference between different methods.

Recently, almost all semi-supervised medical image segmentation frameworks are
based on deep learning. For example, Bai et al. [6] proposed an iterative
method for heart segmentation of MR images, using pseudo-labels generated by
network prediction to update network parameters. Feng et al. [7] improved the
training strategy based on the work of Bai et al. [6]. Only part of the
reasonable segmentation maps predicted from the unannotated samples was
progressively combined with the annotated samples to improve the training
procedure. Li et al. [8] proposed a transformation consistent self-integration
model for semi-supervised skin lesion segmentation based on the $\pi$ model.
Inspired by the Mean Teacher model, Yu et al. [9] proposed a semi-supervised
framework for uncertainty perception to segment the left atrium from 3D MR
images. Cao et al. [10] extended the uncertain time model for semi-supervised
ABUS quality segmentation. However, such methods do not consider the quality
of the predicted segmentation map, which may introduce misinformation into the
segmentation network. Adversarial training introduces a discrimination network
to evaluate the predicted segmentation map, which is simple and effective for
semi-supervised medical image segmentation.

###  II-B Adversarial training for Semi-Supervised Segmentation

The Generative Adversarial Network (GAN) is proposed by Goodfellow et al. [5].
It generates samples by optimizing the adversarial game between the
discriminator and the generator. When applying adversarial learning to semi-
supervised segmentation tasks, the model usually uses two networks: a
segmentation network (generator) for image segmentation and a discriminant
network (discriminator) to identify whether the sample is extracted from real
data or generated by the generator. Generally, there are three popular
strategies in the medical image analysis community based on adversarial
training and semi-supervised methods, including generative models, confidence
maps, and segmentation evaluation. For instance, Sedai et al. [19] introduced
a generative model (VAE [20]) for optic cup (OC) segmentation from retinal
fundus images. They used VAE to learn feature embedding from unlabeled images
and then combined the feature embedding with a segmentation autoencoder
trained on labeled images to perform pixel segmentation on the cup region.
Lahiri et al. [29] deployed unannotated images to generate the fake samples to
increase the amount of the training dataset. However, the model lacks
stability due to the insufficient supervision of the discriminator.

To improve the performance of the discriminator, a simple strategy is inspired
by conditional GAN, where the discriminator is conditioned on the input image
to classify whether the segmentation map is real or fake. Son et al. [15]
connected prediction, inverse prediction, and image, and distinguished good or
poor segmentation results by finding the mapping relationship between image
and prediction. Zhang et al. [16] multiplied the prediction with the image to
judge directly whether the target is accurately segmented. Han et al. [18]
introduced a dual attention fusion block based on connection, extracting
geometric level and intensity level information, thus digging for more
relevant features. Such works explore the correlations between the
segmentation and input image for evaluating the segmentation quality. However,
regardless of whether the segmentation and the corresponding image are related
or not, it is unreasonable to regard all pixels in the generated samples as
negative samples, which is too abstract for the discriminator. As a
consequence, the generated evaluation score does not reflect the correct
probability of the segmentation result, and the discriminator cannot
contribute to instructing the segmentation task.

Nie et al. [21] adopted another strategy, using a discriminator to output the
confidence map and choose high-confidence regions to obtain the ground truth,
then updating the segmentation network in a self-learning manner. The main
limitation of this method is that a confidence threshold must be provided, the
value of which will affect performance. What’s more, if the discriminator
cannot distinguish between good and bad segmentation, poor confidence maps may
reduce the performance of the entire adversarial training. In particular, for
medical images, the presence of speckle noise may affect the confidence map.
The unstable confidence map may also affect the entire learning process and
lead to unsatisfactory segmentation results for unlabeled data.

At the input of the discriminator, we design a new combination of conditional
GAN, called pixel additive blending. At the output of the discriminator,
different from the confidence map and single scalar output, we use PatchGAN to
output an $N\times N$ array to evaluate the good or bad segmentation results
in a region, rather than the entire image or individual pixels. Such
improvements effectively solve the feature mining and learning preference
problems that exist in other methods.

##  III Method

Fig. 2 shows a schematic illustration of our patch confidence adversarial
training model (PCA) for semi-supervised segmentation. The PCA framework
consists of a segmentation network and a discrimination network. First, the
segmentation network takes input data and produces the corresponding
segmentation probability maps. Then, the segmentation probability map and the
image weighted combination are used as the inputs of the conditional
discriminator. Subsequently, the discriminator distinguishes the data
distribution of new images generated based on different maps, including the
ground truth from the annotation data and the segmentation probability maps
from all the data. The discriminator tries to classify if each $N\times N$
patch in a segmentation map is real or fake.

###  III-A General training strategy

To define the loss function, the symbols utilized is first enlisted. The
training set consists of $M+N$ inputs, including $M$ labeled inputs and $N$
unlabeled inputs. Let $A=\\{(x_{i}^{a},y_{i}^{a})\\}_{i=1}^{M}$ be a labeled
set with $M$ samples and $U=\\{x_{i}^{u}\\}_{i=1}^{N}$ be a set with $N$
samples. The segmentation network and the discrimination network are
represented by $S(\cdot)$ and $D(\cdot)$.

In our proposed PCA, the segmentation network is trained by minimizing the
following loss function $L_{S}$:

$L_{S}(A,U;\theta_{S})=L_{seg}+\lambda_{adv}L_{adv},$ (1)

where $\theta_{S}$ represents the parameters for the segmentation network,
$L_{seg}$ and $L_{adv}$ represent the supervised segmentation and adversarial
loss, respectively. $\lambda_{adv}$ refers to the weight of adversarial
learning.

The loss function $L_{seg}$ is used to determine whether the segmentation
probability map generated by the input annotation data is close to ground
truth, which is expressed as:

$L_{seg}(S(x_{i}^{a}),y_{i}^{a})=0.5\ast L_{bce}+0.5\ast L_{dice},$ (2)
$\begin{split}L_{bce}(S(x_{i}^{a}),y_{i}^{a})=&-y_{i}^{a}\cdot{\rm
log}(S(x_{i}^{a}))\\\ &-(1-y_{i}^{a})\cdot{\rm
log}(1-S(x_{i}^{a}))),\end{split}$ (3)
$L_{dice}(S(x_{i}^{a}),y_{i}^{a})=1-\frac{2\ast\sum S(x_{i}^{a})\ast
y_{i}^{a}}{\sum S(x_{i}^{a})+\sum y_{i}^{a}},$ (4)

where $L_{bce}$ is constrained by the standard Binary Cross Entropy (BCE) loss
function, $L_{dice}$ is constrained by the standard DICE loss function.

The general semi-supervised adversarial training objective loss function $L$
is defined as:

$\begin{split}\min\limits_{\theta_{S}}\max\limits_{\theta_{D}}L(\theta_{S},\theta_{D})={\rm
E}_{x_{i}\sim P_{A}}[{-\rm log}D(S(x_{i}))]&\\\ +{\rm E}_{x_{i}\sim
P_{U}}[{-\rm log}(1-D(S(x_{i})))]&,\end{split}$ (5)

where $x_{i}$ is the input image from data distribution $P_{A+U}$, and
$\theta_{D}$ represents the parameters for the discrimination network. As in
GAN [5], when updating the segmentation network, we replace the ${\rm
log}(1-D(x_{i}))$ by ${-\rm log}D(x_{i})$.

###  III-B Patch confidence map

The choice of the discriminator differs due to the different output sizes at
which the decision is made. In this work, we explored several models for the
discriminators with various output sizes, as done in the previous work,
corresponding to several adversarial training strategies. Notably, our
$D(\cdot)$ produces a $N\times N$ matrix as a result.

According to the general adversarial training strategy, the image-level
discriminator’s purpose is to determine whether the input sample is real. The
last layer of the general discriminator outputs a scalar value, which is a
weighted value of the whole image but cannot reflect the local features of the
image. It is hard to train for tasks requiring high precision, such as image
segmentation. Especially in semi-supervised segmentation tasks, if the
discriminator treats predictions from unlabeled data as negative samples and
only learns global features, it won’t be easy to mine enough features to guide
the segmenter to generate more accurate predictions.

The pixel-level discriminator generates the confidence map to determine
whether each pixel in the input sample is real or not. Afterwards, unlabeled
data is trained by generating reliable pseudo-label masks from the confidence
map. This requires the confidence map to infer sufficiently close regions from
the ground truth distribution to find more confident pixels. However, the
confidence map generated by the current method cannot reflect the correct
probability of the segmentation result, which will affect the performance of
the segmentation network and even cause a negative transfer.

Between the extremes, it is also possible to set the receptive field to a
$K\times K$ patch where the decision can be given at the patch-level
(PatchGAN). The final output of the patch-based discriminator is a matrix with
a size of $N\times N$. Each of the items in the matrix represents a local
region in the image. The patch-based discriminator tries to classify whether
each $N\times N$ patch in the image is real or fake, which achieves the
extraction and characterization of local image features and is conducive to
the generation of high-precision images. Compared with image-level (ImageGAN)
or pixel-level (PixelGAN) adversarial learning, PatchGAN has the ability to
capture the local statistics of the output space and guide the segmentation
network to focus on the local structure similarity in the image patches. We
achieve this variation in patch size by adjusting the depth of the GAN
discriminator.

In the early stages of the training process, the positive and negative sample
distributions may not overlap each other, where the discriminator network can
easily distinguish them. To stabilize the training of the semi-supervised
framework, we directly align the features learned by the discriminator network
to the segmentation network and introduce the feature matching [43] loss
$L_{fm}$. It aims to minimize the discrepancy in feature statistics between
the generating samples and the ground truth. It is calculated by a mean
squared error (MSE) loss:

$L_{fm}(x_{i},y_{i})={\rm E}_{x_{i},y_{i}\sim
P_{A}}||D(S(x_{i}))-D(y_{i})||^{2},$ (6)

where $x_{i}$ and $y_{i}$ are sampled from the labeled set $A$.

For unlabeled data, it is impossible to align ground truth features directly.
Inspired by the integrated probability metric [44], we choose the intermediate
layer of the discriminative network as the mapping function and calculate the
mean value to measure the distance between the distributions of positive and
negative samples. We believe the two distributions are the same when the
feature centres match (i.e. the difference in means is minimal). $L_{ipm}$ is
used for unlabeled prediction, forcing the segmentation network to use a
reasonable solution without label information. $L_{ipm}$ displays in Eq. (7):

$L_{ipm}(x_{i},y_{i})=||{\rm E}_{x_{i}\sim P_{U}}D(S(x_{i}))-{\rm
E}_{y_{i}\sim P_{A}}D(y_{i})||^{2},$ (7)

where $x_{i}$ represents unlabeled images sampled from unlabeled set $U$, and
$y_{i}$ represents the ground truth sampled from labeled set $A$. The final
training objective $L_{S}$ is as follows:

$L_{S}=L_{seg}+\lambda_{adv}L_{adv}+\lambda_{fea}L_{fea}+\lambda_{ipm}L_{ipm},$
(8)

where $\lambda_{fea}$ and $\lambda_{ipm}$ are the corresponding weights.

###  III-C Pixel additive blending

The simplest input form can be just the segmentation probability maps [24, 25,
26], which allow the discriminator to learn useful shape properties of the
objects, thereby evaluating the segmentation result quality. However, in this
form, the discriminator has weaker discriminability, which can easily cause an
over-fitting problem.

In conditional GAN (CGAN), the discrimination model explores the relationship
between the segmentation probability map and the image, to enhance the
discriminative ability. In that case, the discriminator might generate the
probability values by distinguishing between labeled and unlabeled images
without learning segmentation information. So the key is how to encode the
correlation information to construct the effective input of the discriminator.
Below, we will discuss several methods for combining the original image
enhancement segmentation results.

The segmentation result can be directly concatenated to the original image
[15]. Since the discrimination network has separate model parameters for
handling information from the segmentation maps and from the original image,
the discriminator may focus on learning the features of the original image to
give a judgment score, which leads to the learning preference problem.

Element-wise multiplication [16] forces the discriminator to learn about the
association information between the segmentation maps and the original images,
which ensures that the segmentation maps are meaningful in adversarial
training. However, element-wise multiplication ignores the correlation between
the background and the target. The discriminator will focus on the distinction
of some uncomplicated categories, which is not conducive to improving its
ability.

The attention block [18] is utilized to generate the attentive rated maps from
the segmentation maps and the corresponding input images. The image’s features
are multiplied by the attentive rating map in an element-by-element manner to
generate the attentive feature map. However, the attention method does not
deal with all the details of the image accurately, introducing a lot of noisy
information, which is usually fatal for medical image segmentation tasks.

Figure 3: Using the condition that the labeled data and unlabeled data are
identically distributed, the input and output are coupled, and the obtained
joint probability is approximately identically distributed.

To alleviate the influence of learning preference on network optimization and
improve feature mining ability, we introduce a novel conditional GAN, called
pixel additive blending, where the discriminator is conditioned on the input
image $x$. The main idea is to fit the joint probability of the image and the
segmentation to narrow the gap between the labeled data pair and the unlabeled
data pair, thereby helping the discriminator to better learn the correlation
between the image and the segmentation. This is because semi-supervised
learning assumes that the labeled data and unlabeled data samples belong to
the same distribution, but there is a gap between their corresponding
segmentations. If the joint probability distribution of the image and the
segmentation can be fitted, then the joint probability distribution of the
unlabeled data will be the same as that of the labeled data. close. Figure 3
shows the process information.

Our solution is to assign a Gaussian noise with a small value to the image and
add it to the segmentation map pixel by pixel. It is worth mentioning that
here we use the segmenter’s predictions as negative samples and the ground
truth of the labeled data as positive samples. It is to avoid the
discriminator to complete the classification task by distinguishing between
labeled and unlabeled images, which will deviate from the original purpose.
Thus, the objective function becomes:

$\begin{split}\min\limits_{\theta_{S}}\max\limits_{\theta_{D}}L(\theta_{S},\theta_{D})={\rm
E}_{x_{i},y_{i}\sim P_{A}}[{-\rm log}D(Z(x_{i},y_{i}))]&\\\ +{\rm
E}_{x_{i}\sim P_{A+U}}[{-\rm log}(1-D(Z(x_{i},S(x_{i}))))]&.\end{split}$ (9)

Particularly, the pixel additive blending $Z(\cdot)$ is expressed as follows:

$Z(x_{i},y_{i})=y_{i}+\lambda_{noise}x_{i}\cdot noise,$ (10)

where $\lambda_{noise}$ and $noise$ represent the weight coefficient of the
image and Gaussian noise, respectively. All $D(x)$ are replaced by
$D(Z(x_{i},y_{i}))$, when introducing the pixel additive blending.

Here are three reasons: Firstly, pixel-addition ensures both the segmentation
map and the image are used in the discriminator’s decision-making and in the
adversarial training process. Also, the randomness of Gaussian noise reduces
the correlation between image pixels, forcing the discriminator to learn the
correlation between segmentation and image. In addition, we believe that the
output is less complex than the input in the semi-supervised segmentation
task. That means the segmentation map may have very little influence on both
the decision-making process of the discriminator and the parameter updates of
the segmenter. In order to reduce this bias, we must limit the influence of
image information. Here we assign a small weight to the image.

###  III-D Network architecture

For the segmentation network, we follow the spirit of U-Net where skip
connections are added between the down-sampling path and the up-sampling path
to save low-level information. This skip-connection is crucial to segmentation
tasks as the initial feature maps maintain low-level features such as edges
and blobs that can be properly exploited for accurate segmentation. In
addition, the U-Net model utilizes bi-linear interpolation to expand the
feature maps.

The proposed discrimination network generally follows the architecture of
PatchGAN [30]. The network contains three convolutional layers with a kernel
size of $4\times 4$ and a stride of $2\times 2$. The channel numbers of the
three convolutional layers are 32, 64, and 1, respectively. The activation
function following each convolutional layer is LeakyReLU with an alpha value
of 0.2, except for the last one using the Sigmoid function. The output size
($m\times n$) the patch-based discriminator is $32\times 32$, in which one
pixel corresponds to a patch of size $22\times 22$ in the input probability
maps. Each patch is classified into real (1) or fake (0) through the
discriminator. We employ this adversarial learning strategy to force each
generated patch in the prediction of unlabeled data to be similar to the patch
of labeled data.

##  IV Experiments

###  IV-A Dataset

We evaluated our proposed semi-supervised segmentation method on two typical
medical images, including cardiac images and brain tumor images.

Cardiac image dataset. This dataset consists of 200 MRI scans from 100
patients [38] for training and 50 patients for testing. Three cardiac regions
are labeled in the ground truth: left ventricle (LV), right ventricle (RV),
and myocardium (Myo). For a fair comparison, we only selected the training set
in our experiments and divided the data set into the training, validation, and
testing sets, respectively, containing 70, 10, and 20 patients’ data. Slices
within 3DMRI scans were considered 2D images, which were fed as input to the
network.

Brain tumor image dataset. The brain tumor image dataset comes from the Brain
Tumor Segmentation (BraTS) 2019 challenge. The released BraTs dataset contains
335 3D cases. The experiment of whole brain tumor segmentation was performed
using the multi-modal MRI data from the BraTS 2019 challenge [39, 40, 41]. The
entire dataset contains multi-institutional preoperative MRI of 335 glioma
patients (259 HGG and 76 LGG), where each patient has four modalities of MRI
scans with neuroradiologist-examined pixel-wise labels. Here, we use the T1-CE
modal of HGG for whole tumor segmentation since this modality can better
manifest malignant tumors. In our experiments, the MRI scans are normalized to
zero mean and unit variance. We randomly selected 207 samples for training and
52 for validation. We slice 3D imags into 2D images and crop them to
$160\times 160$.

TABLE I: Comparisons with state-of-the-art methods on the ACDC2017 dataset

Method | Scans used | Area Overlap | Boundary Error
---|---|---|---
An. | Un. | DSC(%)$\uparrow$ | Imp. | JA(%)$\uparrow$ | Imp. | 95HD(voxel)$\downarrow$ | Imp. | ASD(voxel)$\downarrow$ | Imp.
Baseline | 7(10%) | 0 | 77.34 | - | 66.20 | - | 9.18 | - | 2.45 | -
GAN [15] | 7(10%) | 63(90%) | 83.28 | 5.94% | 72.84 | 6.64% | 9.84 | -0.66 | 2.78 | -0.33
DAN [16] | 79.33 | 1.99% | 67.93 | 1.73% | 11.85 | -2.67 | 3.08 | -0.63
BUS-GAN [34] | 81.80 | 4.46% | 71.20 | 5.00% | 6.11 | 3.07 | 2.21 | 0.24
ASDNet [28] | 82.89 | 5.55% | 71.98 | 5.78% | 16.50 | -7.32 | 4.13 | -1.68
SASSNet [27] | 84.14 | 6.80% | 74.09 | 7.89% | 5.03 | 4.15 | 1.40 | 1.05
MT [23] | 80.40 | 3.06% | 69.28 | 3.08% | 10.05 | -0.87 | 2.65 | -0.20
ICT [22] | 83.54 | 6.20% | 72.84 | 6.64% | 7.58 | 1.60 | 2.27 | 0.18
UAMT [9] | 81.58 | 4.24% | 70.48 | 4.28% | 12.35 | -3.17 | 3.62 | -1.17
CCT [35] | 83.34 | 6.00% | 72.84 | 6.64% | 7.07 | 2.11 | 2.18 | 0.27
DTC [33] | 82.71 | 5.37% | 72.14 | 5.94% | 11.31 | -2.13 | 2.99 | -0.54
CPS [36] | 85.32 | 7.98% | 75.42 | 9.22% | 6.64 | 2.54 | 1.98 | 0.47
UPRC [32] | 81.77 | 4.43% | 70.85 | 4.65% | 5.04 | 4.14 | 1.41 | 1.04
PCA(w/o CGAN) | 86.39 | 9.05% | 76.91 | 10.71% | 4.63 | 4.55 | 1.20 | 1.25
PCA(w/ CGAN) | 87.33 | 9.99% | 78.31 | 12.11% | 3.10 | 6.08 | 1.35 | 1.10
Baseline | 14(20%) | 0 | 83.69 | - | 74.00 | - | 6.63 | - | 1.74 | -
GAN [15] | 14(20%) | 56(80%) | 84.52 | 0.83% | 74.92 | 0.92% | 10.51 | -3.20 | 2.64 | -0.29
DAN [16] | 86.18 | 2.49% | 76.71 | 2.71% | 9.23 | -1.92 | 2.38 | -0.03
BUS-GAN [34] | 85.01 | 1.32% | 75.83 | 1.83% | 6.64 | 0.67 | 1.76 | 0.59
ASDNet [28] | 84.18 | 0.49% | 74.01 | 0.01% | 6.68 | 0.63 | 2.10 | 0.25
SASSNet [27] | 85.99 | 2.30% | 76.63 | 2.63% | 5.32 | 1.99 | 1.47 | 0.88
MT[23] | 85.58 | 1.89% | 76.38 | 2.38% | 4.89 | 2.42 | 1.60 | 0.75
ICT [22] | 85.25 | 1.56% | 75.71 | 1.71% | 7.66 | -0.35 | 2.32 | 0.03
UAMT [9] | 85.87 | 0.72% | 76.78 | 1.30% | 5.06 | 1.14 | 1.54 | 0.58
CCT [35] | 86.09 | 2.40% | 77.05 | 3.05% | 7.01 | 0.30 | 1.98 | 0.37
DTC [33] | 86.28 | 1.13% | 77.03 | 1.55% | 6.14 | 0.06 | 2.11 | 0.01
CPS [36] | 87.38 | 3.69% | 78.61 | 4.61% | 6.06 | 1.25 | 1.69 | 0.66
UPRC [32] | 85.07 | -0.08% | 75.61 | 0.13% | 6.26 | -0.06 | 1.77 | 0.35
PCA(w/o CGAN) | 87.86 | 4.17% | 79.13 | 5.13% | 5.10 | 2.21 | 1.51 | 0.84
PCA(w/ CGAN) | 88.09 | 4.40% | 79.44 | 5.44% | 2.91 | 4.40 | 0.98 | 1.37
Upper bound | 70(100%) | 0 | 91.65 | - | 84.93 | - | 1.89 | - | 0.56 | -

###  IV-B Implementation Details

We implement our framework in PyTorch with 2 Nvidia 2080Ti GPUs. On the ACDC
dataset, all settings followed the public benchmark Luo et al. [31] for fair
comparisons. We normalized the samples as a 0–1 range and used data
augmentaition for the random rotation and flip operations. All the 2D patches
were interpolated to a size of $256\times 256$ and randomly extracted. The
batch size was set at 24, including 12 labeled samples and 12 unlabeled
samples. The model was trained via 30K iterations. To train the segmentation
network and discrimination network, an SGD and an Adam optimizer were employed
to minimize $l_{s}$ and $l_{d}$. The initial learning rates were set to 1e-2
and 1e-4, respectively, and decayed according to the equation
$lr=(1-\frac{iterations}{iterartions_{total}})^{0.9}$. During the testing
time, we also interpolated the output results to $256\times 256$ as input and
then restored them to their original size. For the brain tumor image dataset,
we normalized the samples as zero mean and unit variance. The batch size was
set at 30. The initial learning rates were set at 2e-2 and 1e-4. The model was
trained over 100 epochs. After obtaining the segmentation probability map from
the segmentation network, we apply thresholding with 0.5 to generate a binary
segmentation result. $\lambda_{adv}$,$\lambda_{fea}$,$\lambda_{ipm}$ and
$\lambda_{noise}$ were set to 0.1, 1, 0.1 and 0.001.

###  IV-C Evaluation metrics

Following Luo et al. [31], we adopt four metrics, including dice similarity
coefficient (DSC), Jaccard (JA), the 95% Hausdorff Distance (95HD), and the
average surface distance (ASD). DSC and JA were employed to examine the
overlap areas between the two comparisons. 95HD and ASD were exploited to
measure the euclidean distance between a computer-identified lesion boundary
and the boundary determined by physicians. Higher DSC and JA, along with lower
95HD and ASD, correspond to the higher similarity between the two compared
regions.

###  IV-D Experiments on Cardiac Image Data Set

TABLE II: Discussion of different proporation of data to train our model on the ACDC2017 dataset Label/Unlabel | Model | Area Overlap | Boundary Error
---|---|---|---
DSC(%)$\uparrow$ | Imp. | JA(%)$\uparrow$ | Imp. | 95HD(voxel)$\downarrow$ | Imp. | ASD(voxel)$\downarrow$ | Imp.
3/67 | Baseline | 48.09 | - | 36.54 | - | 50.04 | - | 21.03 | -
PCA(w/o CGAN) | 56.98 | 8.89% | 46.43 | 9.89% | 14.15 | 35.89 | 7.51 | 13.52
PCA(w/ CGAN) | 66.15 | 18.06% | 54.70 | 18.16% | 15.22 | 22.74 | 5.42 | 15.61
7/63 | Baseline | 77.34 | - | 66.20 | - | 9.18 | - | 2.45 | -
PCA(w/o CGAN) | 86.39 | 9.05% | 76.91 | 10.71% | 4.63 | 4.55 | 1.20 | 1.25
PCA(w/ CGAN) | 87.33 | 9.99% | 78.31 | 12.11% | 3.10 | 6.08 | 1.35 | 1.10
14/56 | Baseline | 83.69 | - | 74.00 | - | 7.31 | - | 2.35 | -
PCA(w/o CGAN) | 87.86 | 4.17% | 79.13 | 5.13% | 5.10 | 2.21 | 1.51 | 0.84
PCA(w/ CGAN) | 88.09 | 4.40% | 79.44 | 5.44% | 2.91 | 4.40 | 0.98 | 1.37
70/0 | Baseline | 91.65 | - | 84.93 | - | 1.89 | - | 0.56 | -

####  IV-D1 Comparison with other semi-supervised methods

We compare our method with the latest semi-supervised learning methods,
including GAN-based methods [15, 16, 34, 28, 27] and current state-of-the-art
semi-supervised methods [9, 35, 33, 36, 32, 22, 23]. We implemented these
methods and conducted comparative experiments on the public dataset ACDC 2017.
The reported results in Table I are the average performance of four classes on
the test set. In each semi-supervised setting, we list the performance of the
fully-supervised baseline, adversarial training methods, the latest semi-
supervised methods, and our methods in turn. We train U-Net with 100%, 20% and
10% of training data as upper bounds and the two baselines. The GAN-based
methods perform better compared to the baseline overall, showing the
effectiveness of GAN-based methods for semi-supervised segmentation. However,
we can notice that the performance of ASDNet and BUS-GAN is similar to that of
GAN, indicating that the application of the confidence map and segmentation
evaluation are still challenging for semi-supervised segmentation. Compared to
GAN-based methods, consistency-based methods achieve comparable performance,
demonstrating them effectively utilizing unlabeled data. For example, under
the 10% setting, UAMT achieved a 4.24% improvement in the DSC indicator.
Obviously, compared with other methods, the model we proposed obtained the
best quantitative results for DSC and JA performance in each semi-supervised
setting. Specifically, we achieves an average DSC and JA improvement of 9.99%
and 12.11% or 4.40% and 5.44% than the fully-supervised baseline trained with
10% or 20% labeled data. In addition, by exploiting the unlabeled data
effectively, our model almost always obtains the lowest standard values of
boundary error (i.e., 95HD and ASD). Fig. 4 shows the visualization of all
methods. We can see that these methods often perform well for lesions.
However, for the blurry lesions, our method can predict them better than the
other methods. Compared with the most advanced semi-supervised methods, our
results can describe the target boundary more accurately and retain more
details, which also proves the effectiveness of our method.

####  IV-D2 Different ratio of labeled data

To further verify the influence of different percentages of labeled data on
performance for our method, a different number of labeled images were selected
from the training set. We chose three settings of 3, 7 and 14 cases, which are
4%, 10% and 20% of labeled training data. Table II shows the segmentation
results using different numbers of labeled and unlabeled images. All PCA
models perform better than the corresponding supervision methods, which shows
that our method effectively utilizes unlabeled data and promotes performance.
Our method can achieve a substantial surpass of 2.4% DSC with 10% labeled
scans compared with the baseline with 20% labeled scans, demonstrating the
significant advantage of our method under a small-scale labeled dataset. When
the number of labeled data is small (i.e. labeled data = 3), our method
obtains a substantial increase (18.06% DSC and 18.16% JA) in accuracy over the
fully supervised method (48.09% DSC and 36.54% JA). What‘s more, our method
achieves a large reduction (about 70% 95HD and 75% ASD) in boundary error.
Both improvements indicate that our proposed method has broad potential for
further clinical applications. It can also be noticed that the performance of
all methods increases slowly with the increase of labeled data, which
illustrates that the performance of models tends to converge as labeled data
increases.

####  IV-D3 Ablation study

We propose an ablation study to measure the contribution of different
components of the method, and the results are shown in Table III. The
abbreviations “Im.”, “Pa.” and “Pi.” stand for ImageGAN, PatchGAN and
PixelGAN. Our framework contains two main components: CGAN and PatchGAN. To
investigate the effectiveness of each component, we performed an ablation
study by adding the two components to the baseline one by one. In addition, we
also explore the impact of a variety of discriminators, including PixelGAN and
ImageGAN. The experiments are conducted in the setting of three labeled
datasets. The results of different settings are presented in Table III. The
first line is the supervised baseline model, which was trained with only 3
labels. First, we add the unlabeled data and the adversarial loss. Our method
achieves the best results (56.98% DSC) among the three, surpassing the
baseline by 8.89%. Other adversarial methods are also better than the
baseline, showing the effectiveness of the GAN framework. Then, we explore the
infulence of the CGAN strategy. From the results, we can observe that
conditonal image constraint significantly improves segmentation performance by
63.81%. We believe it is because the image-segmentation association plays a
key role in the few annotations. Finally, with the joint learning of CGAN and
PatchGAN, the performance of our framework is further promoted to the state-
of-the-art, surpassing the supervised baseline by 18.06% DSC. It is observed
that applying conditonal image constraint alone contributes more to the
model’s performance.

TABLE III: Ablation studies of our proposed methods on the ACDC2017 dataset Setting | Scans used | Metrics
---|---|---
UNet | Im. | Pa. | Pi. | CGAN | An. | Un. | DSC_Mean | Imp.
✓ |  |  |  |  | 3(4%) | 67(96%) | 48.09 | -
✓ | ✓ |  |  |  | 54.43 | 6.34%
✓ |  | ✓ |  |  | 56.98 | 8.89%
✓ |  |  | ✓ |  | 54.49 | 6.40%
✓ | ✓ |  |  | ✓ | 63.81 | 15.72%
✓ |  | ✓ |  | ✓ | 66.15 | 18.06%
TABLE IV: Comparisons with state-of-the-art methods on the BraTs2019 dataset Method | Scans used | Metrics
---|---|---
An. | Un. | DSC_WT(%) | Imp. | DSC_TC(%) | Imp. | DSC_ET(%) | Imp. | DSC_Mean(%) | Imp.
Baseline | 11(5%) | 0 | 38.84 | - | 35.69 | - | 37.47 | - | 37.33 | -
GAN[15] | 11(5%) | 196(95%) | 46.50 | 7.66% | 52.85 | 17.16% | 50.77 | 13.30% | 50.04 | 12.71%
DAN [16] | 42.50 | 3.66% | 47.92 | 12.23% | 47.07 | 9.60% | 45.83 | 8.50%
BUS-GAN [34] | 40.25 | 1.41% | 44.64 | 8.95% | 44.95 | 7.48% | 43.28 | 5.95%
ASDNet [28] | 39.22 | 0.38% | 42.61 | 6.92% | 41.98 | 4.51% | 41.27 | 3.94%
SSASNet [27] | 39.45 | 0.61% | 49.12 | 4.49% | 42.37 | 4.90% | 40.67 | 3.33%
MT[23] | 40.27 | 1.43% | 43.43 | 7.74% | 44.10 | 6.63% | 42.60 | 5.27%
ICT [22] | 41.64 | 2.80% | 45.32 | 9.63% | 46.13 | 8.66% | 44.36 | 7.03%
UAMT [9] | 39.62 | 0.78% | 43.77 | 8.08% | 44.53 | 7.06% | 42.64 | 5.31%
CCT [35] | 42.29 | 3.45% | 46.50 | 10.81% | 46.22 | 8.75% | 45.00 | 7.67%
DTC [33] | 36.72 | -2.12% | 42.73 | 7.04% | 42.98 | 5.51% | 40.81 | 3.48%
CPS [36] | 40.03 | 1.19% | 47.00 | 11.31% | 48.00 | 10.53% | 45.01 | 7.68%
UPRC [32] | 37.81 | -1.03% | 42.30 | 6.61% | 43.17 | 5.70% | 41.09 | 3.76%
PCA(w/o CGAN) | 49.80 | 10.96% | 57.61 | 21.92% | 55.40 | 17.93% | 54.27 | 16.94%
PCA(w/ CGAN) | 44.59 | 5.75% | 57.71 | 22.02% | 55.20 | 17.73% | 52.50 | 15.17%

###  IV-E Experiments on Brain Tumor Image Data Set

We further tested our proposed method on the brain tumor dataset from the
public 2019 Brain Tumor Segmentation Challenge. Since the test set and
validation set are not publicly available, we randomly divided the training
set into the training set and test set at a ratio of 8:2. The input is T1-CE
modal brain images, and the output result is the binarized segmentation of the
three target regions. On the test set, we performed a quantitative comparison
of all semi-supervised segmentation methods on the test set and trained the
model with 10% labeled data.

The results in Table IV show that among these semi-supervised segmentation
methods, our PCA (without CGAN) model achieves the best performance (Mean DSC
= 52.99%) in both settings. Compared with the supervised segmentation method
trained on only 10% of labeled data, the improvement is 15.66%. What’s more,
we find that unconditional GAN performs much better than conditional GAN. A
possible reason could be that small-amplitude jitter produces significant
changes in the gray value of brain tumor images. With the introduction of the
constraints of the original image information, our model also received noisy
information. We visualized the segmentation results in Fig. 5. Compared with
other methods, our results have a higher overlap rate with the ground truth,
produce fewer false positives, and remain more detailed. This part of the
experiment further demonstrates the effectiveness of our method.

##  V Conclusion

In this paper, we propose a novel semi-supervised learning model (PCA) for
medical image segmentation. Specifically, we built a PatchGAN and CGAN
framework based on GAN to mine segmentation contextual information and
stabilize discriminator training, thereby improving the accuracy of the
segmentation model. In addition, we propose a novel conditional input for the
GAN framework to alleviate the over-fitting of unlabeled data prediction.
Extensive experiments on two public data sets prove the effectiveness of the
method. Compared with the current state-of-the-art methods, our proposed model
shows superior performance. In the future, we will extend the segmentation
scheme to other medical image segmentation tasks that lack enough annotated
data and explore the potential of PCA in more visual tasks.

Figure 4: The visualization of different semi-supervised segmentation methods
under 10% labeled data. Figure 5: The visualization of different semi-
supervised segmentation methods under 10% labeled data. Black corresponds to
the background, the green area corresponds to the whole tumor region (wt), the
yellow area represents the core tumor region (ct), and the red area represents
the enhancing tumor region (et).

## References

  * [1] M. Mahmud, M. S. Kaiser, A. Hussain, and S. Vassanelli, “Applications of deep learning and reinforcement learning to biological data,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 29, no. 6, pp. 2063–2079, 2018.
  * [2] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” _Medical image analysis_ , vol. 42, pp. 60–88, 2017.
  * [3] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 4, pp. 640–651, 2015.
  * [4] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [6] W. Bai, O. Oktay, M. Sinclair, H. Suzuki, M. Rajchl, G. Tarroni, B. Glocker, A. King, P. M. Matthews, and D. Rueckert, “Semi-supervised learning for network-based cardiac mr image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2017, pp. 253–260.
  * [7] Z. Feng, D. Nie, L. Wang, and D. Shen, “Semi-supervised learning for pelvic mr image segmentation based on multi-task residual fully convolutional networks,” in _2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)_. IEEE, 2018, pp. 885–888.
  * [8] X. Li, L. Yu, H. Chen, C.-W. Fu, L. Xing, and P.-A. Heng, “Transformation-consistent self-ensembling model for semisupervised medical image segmentation,” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. 32, no. 2, pp. 523–534, 2020.
  * [9] L. Yu, S. Wang, X. Li, C.-W. Fu, and P.-A. Heng, “Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 605–613.
  * [10] X. Cao, H. Chen, Y. Li, Y. Peng, S. Wang, and L. Cheng, “Uncertainty aware temporal-ensembling model for semi-supervised abus mass segmentation,” _IEEE Transactions on Medical Imaging_ , vol. 40, no. 1, pp. 431–443, 2020\.
  * [11] W. Bai, O. Oktay, M. Sinclair, H. Suzuki, M. Rajchl, G. Tarroni, B. Glocker, A. King, P. M. Matthews, and D. Rueckert, “Semi-supervised learning for network-based cardiac mr image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2017, pp. 253–260.
  * [12] G. Bortsova, F. Dubost, L. Hogeweg, I. Katramados, and M. de Bruijne, “Semi-supervised medical image segmentation via learning consistency under transformations,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 810–818.
  * [13] Y. Xia, F. Liu, D. Yang, J. Cai, L. Yu, Z. Zhu, D. Xu, A. Yuille, and H. Roth, “3d semi-supervised learning with uncertainty-aware multi-view co-training,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 3646–3655.
  * [14] W.-C. Hung, Y.-H. Tsai, Y.-T. Liou, Y.-Y. Lin, and M.-H. Yang, “Adversarial learning for semi-supervised semantic segmentation,” _arXiv preprint arXiv:1802.07934_ , 2018.
  * [15] J. Son, S. J. Park, and K.-H. Jung, “Retinal vessel segmentation in fundoscopic images with generative adversarial networks,” _arXiv preprint arXiv:1706.09318_ , 2017.
  * [16] Y. Zhang, L. Yang, J. Chen, M. Fredericksen, D. P. Hughes, and D. Z. Chen, “Deep adversarial networks for biomedical image segmentation utilizing unannotated images,” in _International conference on medical image computing and computer-assisted intervention_. Springer, 2017, pp. 408–416.
  * [17] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in _International conference on machine learning_. PMLR, 2019, pp. 7354–7363.
  * [18] L. Han, Y. Huang, H. Dou, S. Wang, S. Ahamad, H. Luo, Q. Liu, J. Fan, and J. Zhang, “Semi-supervised segmentation of lesion from breast ultrasound images with attentional generative adversarial network,” _Computer methods and programs in biomedicine_ , vol. 189, p. 105275, 2020.
  * [19] S. Sedai, D. Mahapatra, S. Hewavitharanage, S. Maetschke, and R. Garnavi, “Semi-supervised segmentation of optic cup in retinal fundus images using variational autoencoder,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2017, pp. 75–82.
  * [20] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [21] S. Li, C. Zhang, and X. He, “Shape-aware semi-supervised 3d semantic segmentation for medical images,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 552–561.
  * [22] V. Verma, K. Kawaguchi, A. Lamb, J. Kannala, Y. Bengio, and D. Lopez-Paz, “Interpolation consistency training for semi-supervised learning,” _arXiv preprint arXiv:1903.03825_ , 2019.
  * [23] A. Tarvainen and H. Valpola, “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,” _arXiv preprint arXiv:1703.01780_ , 2017.
  * [24] Y. Dong, D. Xu, S. K. Zhou, B. Georgescu, M. Chen, S. Grbic, D. Metaxas, and D. Comaniciu, “Automatic liver segmentation using an adversarial image-to-image network,” _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 2017.
  * [25] Z. Han, B. Wei, M. Ashley, L. Stephanie, and S. Li, “Spine-gan: Semantic segmentation of multiple spinal structures,” _Medical Image Analysis_ , vol. 50, pp. S136 184 151 830 642X–, 2018.
  * [26] S. M. Shankaranarayana, K. Ram, K. Mitra, and M. Sivaprakasam, “Joint optic disc and cup segmentation using fully convolutional and adversarial networks,” 2017.
  * [27] S. Li, C. Zhang, and X. He, “Shape-aware semi-supervised 3d semantic segmentation for medical images,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 552–561.
  * [28] D. Nie, Y. Gao, L. Wang, and D. Shen, “Asdnet: attention based semi-supervised deep networks for medical image segmentation,” in _International conference on medical image computing and computer-assisted intervention_. Springer, 2018, pp. 370–378.
  * [29] A. Lahiri, K. Ayush, P. Kumar Biswas, and P. Mitra, “Generative adversarial learning for reducing manual annotation in semantic segmentation on large scale miscroscopy images: Automated vessel segmentation in retinal fundus image as test case,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2017, pp. 42–48.
  * [30] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [31] X. Luo, “SSL4MIS,” https://github.com/HiLab-git/SSL4MIS, 2020.
  * [32] X. Luo, W. Liao, J. Chen, T. Song, Y. Chen, S. Zhang, N. Chen, G. Wang, and S. Zhang, “Efficient semi-supervised gross target volume of nasopharyngeal carcinoma segmentation via uncertainty rectified pyramid consistency,” in _Medical Image Computing and Computer Assisted Intervention – MICCAI 2021_ , 2021, pp. 318–329.
  * [33] X. Luo, J. Chen, T. Song, and G. Wang, “Semi-supervised medical image segmentation through dual-task consistency,” 2021, pp. 8801–8809.
  * [34] L. Han, Y. Huang, H. Dou, S. Wang, S. Ahamad, H. Luo, Q. Liu, J. Fan, and J. Zhang, “Semi-supervised segmentation of lesion from breast ultrasound images with attentional generative adversarial network,” _Computer methods and programs in biomedicine_ , vol. 189, p. 105275, 2020.
  * [35] Y. Ouali, C. Hudelot, and M. Tami, “Semi-supervised semantic segmentation with cross-consistency training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 674–12 684.
  * [36] X. Chen, Y. Yuan, G. Zeng, and J. Wang, “Semi-supervised semantic segmentation with cross pseudo supervision,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2613–2622.
  * [37] J. Peng, G. Estrada, M. Pedersoli, and C. Desrosiers, “Deep co-training for semi-supervised image segmentation,” _Pattern Recognition_ , vol. 107, p. 107269, 2020.
  * [38] O. Bernard, A. Lalande, C. Zotti, F. Cervenansky, X. Yang, P.-A. Heng, I. Cetin, K. Lekadir, O. Camara, M. A. G. Ballester _et al._ , “Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved?” _IEEE transactions on medical imaging_ , vol. 37, no. 11, pp. 2514–2525, 2018.
  * [39] S. Bakas, M. Reyes, A. Jakab, S. Bauer, M. Rempfler, A. Crimi, R. T. Shinohara, C. Berger, S. M. Ha, M. Rozycki _et al._ , “Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge,” _arXiv preprint arXiv:1811.02629_ , 2018.
  * [40] S. Bakas, H. Akbari, A. Sotiras, M. Bilello, M. Rozycki, J. S. Kirby, J. B. Freymann, K. Farahani, and C. Davatzikos, “Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features,” _Scientific data_ , vol. 4, no. 1, pp. 1–13, 2017.
  * [41] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest _et al._ , “The multimodal brain tumor image segmentation benchmark (brats),” _IEEE transactions on medical imaging_ , vol. 34, no. 10, pp. 1993–2024, 2014.
  * [42] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [43] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [44] A. Jolicoeur-Martineau, “The relativistic discriminator: a key element missing from standard gan,” _arXiv preprint arXiv:1807.00734_ , 2018.


# Syntax Aware LSTM Model for Chinese Semantic Role Labeling

Feng Qian2, Lei Sha1, Baobao Chang1, Lu-chen Liu2, Ming Zhang2
1 Key Laboratory of Computational Linguistics, Ministry of Education, Peking
University
2 Institute of Network Computing and Information Systems, Peking University
{nickqian, shalei<EMAIL_ADDRESS>
{liuluchen292<EMAIL_ADDRESS>

###### Abstract

As for semantic role labeling (SRL) task, when it comes to utilizing parsing
information, both traditional methods and recent recurrent neural network
(RNN) based methods use the feature engineering way. In this paper, we propose
Syntax Aware Long Short Time Memory(SA-LSTM). The structure of SA-LSTM
modifies according to dependency parsing information in order to model parsing
information directly in an architecture engineering way instead of feature
engineering way. We experimentally demonstrate that SA-LSTM gains more
improvement from the model architecture. Furthermore, SA-LSTM outperforms the
state-of-the-art on CPB 1.0 significantly according to Student t-test
($p<0.05$).

##  1 Introduction

The task of SRL is to recognize arguments of a given predicate in a sentence
and assign semantic role labels. Since SRL can give a lot of semantic
information, and can help in sentence understanding, a lot of NLP works such
as machine translationXiong et al. (2012); Aziz et al. (2011) use SRL
information. Figure 1 shows an example of SRL task from Chinese Proposition
Bank $1.0$(CPB $1.0$)Xue and Palmer (2003).

Traditional methods on SRL use statistical classifiers such as CRF, MaxEntropy
and SVM Sun and Jurafsky (2004); Xue (2008); Ding and Chang (2008, 2009); Sun
(2010) to do classification according to manually designed features.

Recent works based on recurrent neural network Collobert and Weston (2008);
Zhou and Xu (2015); Wang et al. (2015) extract features automatically, and
outperform traditional methods significantly. However, RNN methods treat
language as sequence data, so most of them fail to take tree structured
parsing information into account, which is considered important for SRL task
Xue (2008); Punyakanok et al. (2008); Pradhan et al. (2005). Even though there
are some RNN based works trying to utilize parsing information, they still do
it in a feature-engineering way.

We propose Syntax Aware LSTM (SA-LSTM) to directly model complex dependency
parsing information in an architecture engineering way instead of feature
engineering way. For example, in Figure 1, the arrowed line stands for
dependency relationship, which is rich in syntactic information. Our SA-LSTM
architecture is shown in Figure 2. Compares to ordinary LSTM, We add
additional connections between dependency related words to capture and model
such rich syntactic information in architecture engineering way. Also, to take
dependency relationship type into account, we also introduce trainable weights
for different types of dependency relationship. The weights can be trained to
indicate importance of a dependency type.

Figure 1: A sentence from CPB with semantic role label and dependency parsing
information Figure 2: SA-LSTM architecture: A $\bigcirc$ stands for one word.
The dotted arrows stand for original neighbor connections of bi-LSTM. Solid
arrows stand for dependency relationship connections. Note that though
dependency parsing relationship is directed, we here treated them as
undirected. We only consider whether there is a connection, and the connection
type.

We experimentally demonstrate that SA-LSTM utilizes parsing information better
than traditional feature engineering way. Furthermore, SA-LSTM reaches
$79.64\%F_{1}$ score on CPB 1.0, outperforms the state-of-the-art
significantly based on Student’s t-test($p<0.05$).

##  2 Syntax Aware LSTM

Compares to traditional feature engineering method, RNN-LSTM alleviates the
burden of manual feature design and selection. However, most RNN-LSTM based
methods failed to utilize dependency parsing relationship. Based on bi-RNN-
LSTM, we propose SA-LSTM which keeps all the merit points of bi-RNN-LSTM, and
at the same time can model dependency parsing information directly.

###  2.1 Conventional bi-LSTM Model for SRL

In a sentence, each word $w_{t}$ has a feature representation $x_{t}$ which is
generated automatically as Wang et al. (2015) did. $z_{t}$ is feature
embedding for $w_{t}$, calculated as followed:

$z_{t}=f(W_{1}x_{t})$ (1)

where $W_{1}\in\mathbb{R}^{n_{1}\times n_{0}}$. $n_{0}$ is the length of word
feature representation.

In a sentence, each word $w_{t}$ has six internal vectors, $\widetilde{C}$,
$g_{i}$, $g_{f}$, $g_{o}$, $C_{t}$, and $h_{t}$, shown in Equation 2:

$\displaystyle\widetilde{C}$ $\displaystyle=f(W_{c}z_{t}+U_{c}h_{t-1}+b_{c})$
(2) $\displaystyle g_{j}$
$\displaystyle=\sigma(W_{j}z_{t}+U_{j}h_{t-1}+b_{j})\ \ j\in\\{i,f,o\\}$
$\displaystyle C_{t}$ $\displaystyle=g_{i}\odot\widetilde{C}+g_{f}\odot
C_{t-1}$ $\displaystyle h_{t}$ $\displaystyle=g_{o}\odot f(C_{t})$

where $\widetilde{C}$ is the candidate value of the current cell state. $g$
are gates used to control the flow of information. $C_{t}$ is the current cell
state. $h_{t}$ is hidden state of $w_{t}$. $W_{x}$ and $U_{x}$ are matrixs
used in linear transformation:

$\displaystyle W_{x},x\in\\{c,i,f,o\\}$
$\displaystyle\in\mathbb{R}^{n_{h}\times n_{1}}$ (3) $\displaystyle
U_{x},x\in\\{c,i,f,o\\}$ $\displaystyle\in\mathbb{R}^{n_{h}\times n_{h}}$

As convention, $f$ stands for $tanh$ and $\sigma$ stands for $sigmoid$.
$\odot$ means the element-wise multiplication.

In order to make use of bidirectional information, the forward
${\overrightarrow{h_{t}}}^{T}$ and backward ${\overleftarrow{h_{t}}}^{T}$ are
concatenated together, as shown in Equation 4:

$a_{t}=[{\overrightarrow{h_{t}}}^{T},{\overleftarrow{h_{t}}}^{T}]$ (4)

Finally, $o_{t}$ is the result vector with each dimension corresponding to the
score of each semantic role tag, and are calculated as shown in Equation 5:

$o_{t}=W_{3}f(W_{2}a_{t})$ (5)

where $W_{2}\in\mathbb{R}^{n_{3}\times n_{2}}$, $n_{2}$ is $2\times h_{t}$,
$W_{3}\in\mathbb{R}^{n_{4}\times n_{3}}$ and $n_{4}$ is the number of tags in
IOBES tagging schema.

###  2.2 Syntax Aware LSTM Model for SRL

Structure of our SA-LSTM is shown in Figure 3. The most significant change we
make to the original RNN-LSTM is shown in the shaded area.

Figure 3: Cell Structure of Syntax Aware LSTM

$S_{t}$ is the syntax information input into current cell, and is calculated
as shown in Equation 6:

$S_{t}=f(\sum_{i=0}^{t-1}\alpha\times h_{i})\\\ $ (6)
$\alpha=\begin{cases}1&\text{If there exists dependency }\\\
&\text{relationship between $w_{i}$ and $w_{t}$}\\\
0&\text{Otherwise}\end{cases}$ (7)

$S_{t}$ is the weighted sum of all hidden state vectors $h_{i}$ which come
from previous words $w_{i}$ . Note that, $\alpha\in\\{0,1\\}$ indicates
whether there is a dependency relationship between $w_{i}$ and $w_{t}$, only
dependency related $h_{i}$ can be input into current cell.

We add a gate $g_{s}$ to constrain information from $S_{t}$, as shown in
Equation 8. To protect the original sentence information from being dilutedWu
et al. (2016) by $S_{t}$, we add $S_{t}$ to hidden layer vector $h_{t}$
instead of adding to cell state $C_{t}$, as shown in Equation 9:

$g_{s}=\sigma(W_{s}z_{t}+U_{s}h_{t-1}+b_{s})$ (8)

So $h_{t}$ in our SA-LSTM cell is calculated as:

$h_{t}=g_{o}\odot f(C_{t})+g_{s}\odot S_{t}$ (9)

SA-LSTM changes structure by adding different connections according to
dependency parsing information. In this way, we consider the whole structure
of dependency tree into SA-LSTM in an architecture engineering way.

However, by using $\alpha$ in Equation LABEL:eq:alpha_1, we do not take
dependency type into account, so we further improve the way $\alpha$ is
calculated from Equation LABEL:eq:alpha_1 to Equation LABEL:eq:alpha_2. Each
$type_{m}$ of dependency relationship is assigned a trainable weight
$\alpha_{m}$. In this way SA-LSTM can model differences between types of
dependency relationship.

$\alpha=\begin{cases}\alpha_{m}&\text{If there exists $type_{m}$ dependency
}\\\ &\text{relationship between $w_{i}$ and $w_{t}$}\\\
0&\text{Otherwise}\end{cases}$ (10)

###  2.3 Training Criteria

We use maximum likelihood criterion to train our model. Stochastic gradient
ascent algorithm is used to optimize the parameters. Global normalization is
applied.

Given a training pair $T=(x,y)$ where $T$ is the current training pair, $x$
denotes current the training sentence, and $y$ is the corresponding correct
answer path. $y_{t}=k$ means that the $t$-th word has the $k$-th semantic role
label. The score of $o_{t}$ is calculated as:

$s(x,y,\theta)=\sum\limits_{t=1}^{N_{i}}o_{ty_{t}}$ (11)

where $N_{i}$ is the word number of the current sentence and $\theta$ stands
for all parameters. So the log likelihood of a single sentence is

$\displaystyle\log
p(y|x,\theta)=\log\frac{exp(s(x,y,\theta))}{\sum\nolimits_{y^{\prime}}exp(s(x,y^{\prime},\theta))}$
(12)
$\displaystyle=s(x,y,\theta)-log\sum\nolimits_{y^{\prime}}exp(s(x,y^{\prime},\theta)）$

where $y^{\prime}$ ranges from all valid paths of answers.

##  3 Experiment

###  3.1 Experiment setting

In order to compare with previous Chinese SRL works, we choose to do
experiment on CPB 1.0. We also follow the same data setting as previous
Chinese SRL workXue (2008); Sun et al. (2009) did. Pre-trained111Trained by
word2vec on Chinese Gigaword Corpus word embeddings are tested on SA-LSTM and
shows improvement.

We use Stanford ParserChen and Manning (2014) to get dependency parsing
information, which now supports Universal Dependency representation in
Chinese. Note that the train set of the parser overlaps a part of our test
set, so we retrained the parser to avoid overlap.

Dimension of our hyper parameters are tuned according to development set and
are shown in Table 1.222All experiment code and related files are available on
request

Hyper Params |  $n_{1}$ |  $n_{h}$ |  $n_{2}$ |  $n_{3}$ |  $learning-rate$
---|---|---|---|---|---
dim |  $200$ |  $100$ |  $200$ |  $100$ |  $0.001$
Table 1: Hyper parameter dimensions

###  3.2 Syntax Aware LSTM Performance

Figure 4: Visualization of trained weight $\alpha_{m}$. X axis is Universal Dependency type, Y axis is the weight. Method |  $F_{1}$%
---|---
Xue(2008) |  71.90
Sun et al.(2009) |  74.12
Yand and Zong(2014) |  75.31
Wang et al.(2015)(Random Initialized) |  77.09
Sha et al.(2016) |  77.69
Comparison Feature Engineering Way |  77.75
Our SA-LSTM(Random Initialized) |  79.56
Our SA-LSTM(Pre-trained Embedding) |  79.64
Table 2: Results comparison on CPB 1.0

To prove that SA-LSTM gains more improvement from the new SA-LSTM
architecture, than from the extra introduced parsing information, we design a
experiment in which dependency relationship is taken into account in
traditional feature engineering way.

Given a word $w_{t}$, $F_{t}$ is the average of all dependency related $x_{i}$
of previous words $w_{i}$ , as shown in Equation 13:

$F_{t}=\frac{1}{T}\sum_{i=0}^{t-1}\alpha\times x_{i}\\\ $ (13)

where T is the number of dependency related words and $\alpha$ is a 0-1
variable calculated as in Equation LABEL:eq:alpha_1.

Then $F_{t}$ is concatenated to $x_{t}$ to form a new feature representation.
In this way, we model dependency parsing information in a conventional feature
engineering way. After that, we feed these new feature representation into
ordinary bi-LSTM.

As shown in Table 2, SA-LSTM reaches $79.56\%F_{1}$ score with random
initialization and $79.64\%F_{1}$ score with pre-traind word embedding on
CPB1.0 dataset. Both of them are the best $F_{1}$ score ever published on CPB
1.0 dataset.

Wang et al. (2015) used bi-LSTM without parsing information and got
$77.09\%F_{1}$ score. “comparison feature engineering method” based on his
work reaches $77.75F_{1}$ score. This demonstrates the introduction of
dependency parsing information has impact on SRL job.

Compared with the “comparison feature engineering method” shown in table 2, it
is clear that SA-LSTM gain more improvement($77.75\%$ to $79.56\%$) from the
architecture of SA-LSTM than from the introduction of extra dependency parsing
information($77.09\%$ to $77.75\%$). Indeed, it is difficult to introduce the
whole tree structure into the model using the simple feature engineering way.
By building the dependency relationship directly into the structure of SA-LSTM
and changing the way information flows, SA-LSTM is able to consider whole tree
structure of dependency parsing information.

###  3.3 Visualization of Trained Weights

According to Equation LABEL:eq:alpha_2, influence from a single type of
dependency relationship will be multiplied with type weight $\alpha_{m}$. When
$\alpha_{m}$ is 0, the influence from this type of dependency relationship
will be ignored totally. When the weight is bigger, the type of dependency
relationship will have more influence on the whole system.

As shown in Figure 4, dependency relationship type $dobj$ receives the highest
weight after training, as shown by the red bar. According to grammar
knowledge, $dobj$ should be an informative relationship for SRL task, and our
system give $dobj$ the most influence automatically. This example further
demonstrate that the result of SA-LSTM is highly in accordance with grammar
knowledge, which further validates SA-LSTM.

##  4 Related works

Semantic role labeling (SRL) was first defined by Gildea and Jurafsky (2002).
Early worksGildea and Jurafsky (2002); Sun and Jurafsky (2004) on SRL got
promising result without large annotated SRL corpus. Xue and Palmer built the
Chinese Proposition BankXue and Palmer (2003) to standardize Chinese SRL
research.

Traditional works such as Xue and Palmer (2005); Xue (2008); Ding and Chang
(2009); Sun et al. (2009); Chen et al. (2006); Yang et al. (2014) use feature
engineering methods. Traditional methods can take parsing information into
account in feature engineering way, such as syntactic path feature. However,
they suffer from heavy manually feature design workload, and data sparsity
problem.

More recent SRL works often use neural network based methods. Collobert and
Weston (2008) proposed a convolutional neural network method for SRL. Zhou and
Xu (2015) proposed bidirectional RNN-LSTM method for English SRL, and Wang et
al. (2015) proposed a bi-RNN-LSTM method for Chinese SRL on which our method
is based. NN based methods extract features automatically and significantly
outperforms traditional methods. However, most NN based methods can not
utilize parsing information which is considered important for semantic related
NLP tasks Xue (2008); Punyakanok et al. (2008); Pradhan et al. (2005).

The work of Roth and Lapata (2016) and Sha et al. (2016) have the same
motivation as ours, but in feature engineering way. Roth and Lapata (2016)
embed dependency parsing path into feature representations using LSTM. Sha et
al. (2016) use dependency parsing information as feature to do argument
relationships classification. In contrast, LA-LSTM utilizes parsing
information in an architecture engineering way, by absorbing the parsing tree
structure into SA-LSTM structure.

##  5 Conclusion

We propose Syntax Aware LSTM model for Chinese semantic role labeling. SA-LSTM
is able to model dependency information directly in an architecture
engineering way. We experimentally testified that SA-LSTM gains more
improvement from the SA-LSTM architecture than from the input of extra
dependency parsing information. We push the state-of-the-art $F_{1}$ to
$79.64\%$, which outperforms the state-of-the-art significantly according to
Student t-test($p<0.05$).

## References

  * Aziz et al. (2011) Wilker Aziz, Miguel Rios, and Lucia Specia. 2011.  Shallow semantic trees for smt.  In Proceedings of the Sixth Workshop on Statistical Machine Translation. Association for Computational Linguistics, pages 316–322.
  * Chen and Manning (2014) Danqi Chen and Christopher D Manning. 2014.  A fast and accurate dependency parser using neural networks.  In EMNLP. pages 740–750.
  * Chen et al. (2006) Wenliang Chen, Yujie Zhang, and Hitoshi Isahara. 2006.  An empirical study of chinese chunking.  In Proceedings of the COLING/ACL on Main conference poster sessions. Association for Computational Linguistics, pages 97–104.
  * Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In Proceedings of the 25th international conference on Machine learning. ACM, pages 160–167.
  * Ding and Chang (2008) Weiwei Ding and Baobao Chang. 2008.  Improving chinese semantic role classification with hierarchical feature selection strategy.  In Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, pages 324–333.
  * Ding and Chang (2009) Weiwei Ding and Baobao Chang. 2009.  Word based chinese semantic role labeling with semantic chunking.  International Journal of Computer Processing Of Languages 22(02n03):133–154.
  * Gildea and Jurafsky (2002) Daniel Gildea and Daniel Jurafsky. 2002.  Automatic labeling of semantic roles.  Computational linguistics 28(3):245–288.
  * Pradhan et al. (2005) Sameer Pradhan, Kadri Hacioglu, Wayne Ward, James H Martin, and Daniel Jurafsky. 2005.  Semantic role chunking combining complementary syntactic views.  In Proceedings of the Ninth Conference on Computational Natural Language Learning. Association for Computational Linguistics, pages 217–220.
  * Punyakanok et al. (2008) Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008.  The importance of syntactic parsing and inference in semantic role labeling.  Computational Linguistics 34(2):257–287.
  * Roth and Lapata (2016) Michael Roth and Mirella Lapata. 2016.  Neural semantic role labeling with dependency path embeddings.  arXiv preprint arXiv:1605.07515 .
  * Sha et al. (2016) Lei Sha, Tingsong Jiang, Sujian Li, Baobao Chang, and Zhifang Sui. 2016.  Capturing argument relationships for chinese semantic role labeling.  In EMNLP. pages 2011–2016.
  * Sun and Jurafsky (2004) Honglin Sun and Daniel Jurafsky. 2004.  Shallow semantic parsing of chinese.  In Proceedings of NAACL 2004. pages 249–256.
  * Sun (2010) Weiwei Sun. 2010.  Improving chinese semantic role labeling with rich syntactic features.  In Proceedings of the ACL 2010 conference short papers. Association for Computational Linguistics, pages 168–172.
  * Sun et al. (2009) Weiwei Sun, Zhifang Sui, Meng Wang, and Xin Wang. 2009.  Chinese semantic role labeling with shallow parsing.  In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3-Volume 3. Association for Computational Linguistics, pages 1475–1483.
  * Wang et al. (2015) Zhen Wang, Tingsong Jiang, Baobao Chang, and Zhifang Sui. 2015.  Chinese semantic role labeling with bidirectional recurrent neural networks.  In EMNLP. pages 1626–1631.
  * Wu et al. (2016) Huijia Wu, Jiajun Zhang, and Chengqing Zong. 2016.  An empirical exploration of skip connections for sequential tagging.  arXiv preprint arXiv:1610.03167 .
  * Xiong et al. (2012) Deyi Xiong, Min Zhang, and Haizhou Li. 2012.  Modeling the translation of predicate-argument structure for smt.  In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, pages 902–911.
  * Xue (2008) Nianwen Xue. 2008.  Labeling chinese predicates with semantic roles.  Computational linguistics 34(2):225–255.
  * Xue and Palmer (2003) Nianwen Xue and Martha Palmer. 2003.  Annotating the propositions in the penn chinese treebank.  In Proceedings of the second SIGHAN workshop on Chinese language processing-Volume 17. Association for Computational Linguistics, pages 47–54.
  * Xue and Palmer (2005) Nianwen Xue and Martha Palmer. 2005.  Automatic semantic role labeling for chinese verbs.  In IJCAI. Citeseer, volume 5, pages 1160–1165.
  * Yang et al. (2014) Haitong Yang, Chengqing Zong, et al. 2014.  Multi-predicate semantic role labeling.  In EMNLP. pages 363–373.
  * Zhou and Xu (2015) Jie Zhou and Wei Xu. 2015.  End-to-end learning of semantic role labeling using recurrent neural networks.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics.