# SynergyNet: Bridging the Gap between Discrete and Continuous Representations
for Precise Medical Image Segmentation

Vandan Gorade1, Sparsh Mittal1, Debesh Jha2, Ulas Bagci2
1 Indian Institute of Technology Roorkee, India
2 Machine & Hybrid Intelligence Lab, Department of Radiology, Northwestern
University, USA

###### Abstract

In recent years, continuous latent space (CLS) and discrete latent space (DLS)
deep learning models have been proposed for medical image analysis for
improved performance. However, these models encounter distinct challenges. CLS
models capture intricate details but often lack interpretability in terms of
structural representation and robustness due to their emphasis on low-level
features. Conversely, DLS models offer interpretability, robustness, and the
ability to capture coarse-grained information thanks to their structured
latent space. However, DLS models have limited efficacy in capturing fine-
grained details. To address the limitations of both DLS and CLS models, we
propose SynergyNet, a novel bottleneck architecture designed to enhance
existing encoder-decoder segmentation frameworks. SynergyNet seamlessly
integrates discrete and continuous representations to harness complementary
information and successfully preserves both fine and coarse-grained details in
the learned representations. Our extensive experiment on multi-organ
segmentation and cardiac datasets demonstrates that SynergyNet outperforms
other state of the art methods including TransUNet: dice scores improving by
2.16%, and Hausdorff scores improving by 11.13%, respectively. When evaluating
skin lesion and brain tumor segmentation datasets, we observe a remarkable
improvements of 1.71% in Intersection-over-Union scores for skin lesion
segmentation and of 8.58% for brain tumor segmentation. Our innovative
approach paves the way for enhancing the overall performance and capabilities
of deep learning models in the critical domain of medical image analysis.

##  1 Introduction

Medical image segmentation, a key step in gaining vital anatomical insights,
assists clinicians in injury identification, disease monitoring, and treatment
planning. As reliance on medical image analysis grows, the demand for precise,
robust segmentation techniques rises. In this regard, deep learning has
greatly improved our ability to do this. Existing deep learning models can be
divided into continuous latent space (CLS) and discrete latent space (DLS)
models. The CLS models represent latent variables as continuous values,
enabling fine-grained representation.

CLS Models such as FCNs [17], UNet [23], and TransUNet [5] and others [19, 4,
14] have shown an ability to capture spatial relationships and fine-grained
details for medical image segmentation. However, these models offer limited
latent interpretable representations of structural information and robustness
[25] in terms of generalization. DLS methods employ discrete codes instead of
continuous values for latent variables. They use techniques such as vector
quantization to discretize the latent space into a finite set of elements
representing anatomical structures. This enables efficient and generalized
data representation. Approaches such as VQVAE [27] and VQGAN [10] have shown
promise in image generation, representation learning, and data compression.

Recent studies [20, 12, 25, 13, 15] highlight the effectiveness of DLS models
in achieving interpretable and robust medical segmentation, particularly for
organs like lungs, retinas, optic discs, and prostates. However, DLS models
struggle to capture fine-grained details and complex spatial relationships,
especially in multi-organ and cardiac segmentation tasks. Accurate modeling of
spatial interdependencies between organs is crucial for precisely segmenting
intricate boundaries and overlapping structures. Recent studies [21, 16, 18,
11, 5, 29, 22, 28] have highlighted the advantages of learning complementary
information across various domains, including medical imaging. Motivated by
this trend, our study aims to address the pivotal question: “How can we
effectively integrate complementary information from discrete and continuous
latent space models for improved medical image segmentation?”.

We present SynergyNet, a novel bottleneck architecture designed specifically
for encoder-decoder segmentation models, aiming to enhance medical image
segmentation results by integrating continuous and discrete latent spaces.
SynergyNet includes the Quantizer, DisConX, and Refinement modules. The
encoder extracts a detailed continuous representation, while the quantizer
module maps it to a compact discrete representation using vector quantization.
By reducing dimensionality, the quantizer module enables efficient, structured
representation while preserving essential information. The DisConX module
serves as a bridge, employing cross-attention to effectively combine the
discrete and continuous representations. Leveraging their complementary
information, the DisConX module enhances pattern capture and interpretation.
The refinement module further enhances the fused features, using hard
attention to emphasize essential elements and filter out noise. The refinement
module improves discriminative power and segmentation quality by focusing on
relevant features. Our contributions are as follows:

  * •

We propose SynergyNet, a novel method that integrates discrete and continuous
representations to enhance medical image segmentation performance. This
integration has not been explored in prior studies for medical image
segmentation tasks.

  * •

Our study demonstrates the effectiveness of combining CLS and DLS models in
improving model generalization across diverse datasets. By leveraging CLS
models for fine-grained detail capture and DLS models’ structured latent space
for encoding coarse-grained details, we observe notable enhancements in
learning and generalization. This integration effectively utilizes the
strengths of each approach, resulting in improved performance across various
datasets.

  * •

SynergyNet is extensively evaluated on four diverse datasets, including
Synapse multi-organ segmentation, ACDC dataset for cardiac segmentation, ISIC
2018 dataset for skin lesion segmentation, and brain tumor segmentation
dataset. Results show that SynergyNet outperforms both CLS [23, 19, 9, 5] and
DLS-based methods [27] across all evaluated datasets. Qualitative analysis
confirms the efficacy of SynergyNet in capturing intricate anatomical
structures and achieving more precise segmentation compared to existing
methods

Figure 1:  Figures (a) and (b) compare the bottleneck architecture of our
proposed SynergyNet with the existing work [5]. Figure (c) illustrates the
workflow of our architecture, where the input image is encoded to generate a
continuous representation $z_{con}$. This continuous representation is
quantized to obtain discrete codes $e_{k}$ and forms a discrete representation
$z_{q}$. The DisConX module (Section 2.2.1) is applied to both the continuous
and discrete representations, resulting in a synergized representation
$z^{attn}_{dc}$. The continuous representation $z_{con}$, discrete
representation $z_{q}$, and attended representation $z^{attn}_{dc}$ are then
fused and processed through the refinement module (Section 2.2.2). The output
of the refinement module is passed to the decoder to obtain the final result.

##  2 Proposed Method

We first discuss the preliminaries (Section 2.1) and then present our newly
proposed algorithm (SynergyNet) and its architecture (Section 2.2).

###  2.1 Preliminaries

####  2.1.1 Problem Statement

Medical image segmentation aims to automatically label anatomical structures
or pathological regions within medical images. Mathematically, this involves
finding a mapping function $f$ that assigns labels $y$ to pixels $x$ in the
input image domain $\mathcal{X}$. The goal is to maximize the conditional
probability of the ground truth segmentation labels $\hat{y}$ given the input
image $x$, i.e., $\hat{y}=\arg\max_{y}P(y|x)$. Learning the parameters of the
mapping function $f$ involves assigning the correct labels to each pixel using
training data. The learning process employs a loss function that usually
consists of Binary Cross Entropy (BCE) and Dice similarity coefficient. This
loss function can be defined as follows:

$L_{seg}={BCE}(y,\hat{y})+(1-{Dice}(y,\hat{y})),$ (1)

where ${BCE}(y,\hat{y})$ calculates the binary cross entropy loss between the
predicted labels $y$ and the ground truth segmentation $\hat{y}$, and
${Dice}(y,\hat{y})$ computes the dice similarity coefficient between $y$ and
$\hat{y}$.

####  2.1.2 Vector Quantization

Following VQVAE[27], Vector quantization (VQ) transforms continuous latent
space vectors $z_{con}\in\mathbb{R}^{dim}$ into discrete codes $e_{k}$ from a
predefined codebook $E\in\mathbb{R}^{K\times dim}$, where $K$ is the codebook
size. The objective of VQ is to find the code $e_{k}$ from the codebook that
minimizes the euclidean distance to the input vector $z_{con}$. This code
$e_{k}$ serves as the discrete representation $z_{dis}$ of $z_{con}$. During
training, the codebook $E$ and the mapping functions between the continuous
and discrete representations are learned by minimizing the quantization loss
$\mathcal{L}_{quant}=\lVert z_{con}-e_{k}\rVert_{2}^{2}$. The quantization
process efficiently encodes and decodes data while preserving important
information in discrete representations. We use the total loss function
$\mathcal{L}_{total}=\mathcal{L}_{seg}+\mathcal{L}_{quant}$ for end-to-end
model training.

####  2.1.3 Multi-head Cross-attention Mechanism

The multi-head cross-attention mechanism extends the cross-attention by
incorporating multiple attention heads. Each attention head attends to
different subspaces of queries and keys, capturing diverse relationships and
dependencies. Given a set of queries $Q$ and keys $K$, multiple sets of
attention weights are computed, one for each attention head. The relevance
scores between a query $q_{i}$ and a key $k_{j}$ are obtained using a
similarity function denoted as ${score}(q_{i},k_{j})={sim}(q_{i},k_{j})$. The
softmax function is applied to transform the relevance scores into attention
weights for each attention head:

$\mathcal{A}_{soft}^{(h)}(q_{i},k_{j})=\frac{\exp({score}^{(h)}(q_{i},k_{j}))}{\sum_{j^{\prime}}\exp({score}^{(h)}(q_{i},k_{j^{\prime}}))}$
(2)

The multi-head cross-attention mechanism then computes a weighted sum of the
values associated with the keys using the attention weights of each attention
head $h$:

$z^{attn}_{h}=\sum_{j}\mathcal{A}_{soft}^{(h)}(q_{i},k_{j})\cdot v_{j}.$ (3)

Here, $z^{attn}_{h}$ represents the aggregated result for the given query
$q_{i}$, considering the importance assigned by the attention weights of the
$h$-th attention head. The outputs from all the attention heads are
concatenated and linearly transformed to produce the final output:

$z^{attn}=Concat(z^{attn}_{1},z^{attn}_{2},\ldots,z^{attn}_{h})\cdot W^{O}.$
(4)

The multi-head cross-attention mechanism enables the model to capture various
interactions and dependencies between queries and keys, enhancing its
representation and information retrieval capabilities.

###  2.2 Proposed Architecture

Our proposed architecture, illustrated in Fig. 1(c), consists of three key
components: the encoder, bottleneck, and decoder. The bottleneck incorporates
the Quantizer, DisConX, and Refinement modules. Starting with an input image
$X$, the encoder function $f$ generates the continuous representation
$z_{con}$. The Quantizer module (Section 2.1.2) maps $z_{con}$ to a more
compact discrete representation $z_{dis}$, capturing essential information
efficiently. The DisConX module (Section 2.2.1) combines the discrete and
continuous representations through cross-attention, leveraging both benefits
to enhance data interpretation. The Refinement module (Section 2.2.2) further
improves the representation by emphasizing relevant features. This step
enhances the model’s discriminative power for the given task.

####  2.2.1 DisConX Module

The DisConX module integrates the discrete representation $z_{{dis}}$ and
continuous representation $z_{{con}}$ using cross-attention, as discussed in
Section 2.1.3. It calculates relevance scores between discrete queries $q_{i}$
and continuous keys $k_{j}$, and computes attention weights using a softmax
function. The module then performs a weighted sum of the continuous values
$v_{j}$ associated with the keys based on the attention weights. The
computation happens as:

$z^{attn}_{dc}=\sum_{j}\mathcal{A}^{(h_{s})}_{soft}(z_{{dis}},z_{{con}})\cdot
z_{{con}}.\vspace{-2.5mm}$ (5)

Here, $h_{s}$ represents the index of the attention heads, $z_{{dis}}$ denotes
the discrete query, and $z_{{con}}$ represents the continuous key/value. The
resulting $z^{attn}_{dc}$ is the aggregated representation, considering the
attention weights from all attention heads. This integration of discrete and
continuous representations enables the exchange of complementary information,
enhancing the model’s ability to capture complex patterns and improving
performance in tasks such as semantic segmentation. Next, the information of
$z_{{dis}}$, $z_{{con}}$, and $z_{{attn}}^{{dc}}$ is fused as
$z_{f}={Fusion}(z_{{dis}},z_{{con}},z_{{dc}}^{{attn}})$.

The fusion operation integrates complementary information from discrete and
continuous representations, enhancing the overall representation for
subsequent refinement modules. We empirically choose addition for fusion.

####  2.2.2 Refinement Module

The proposed refinement module incorporates a hardness-aware self-attention
mechanism, which captures the relevance and similarity between elements in the
fused representation. This mechanism enhances the overall representation
quality by emphasizing important elements and filtering out the noise. The
element with the highest relevance score is identified as the most important.
The attention weight for each element is determined by comparing its
similarity to other elements. The equation below represents this calculation:

$\mathcal{A}^{(h_{h})}_{hard}(z_{{f}_{i}})=\mathbb{I}({sim}^{(h_{h})}(z_{{f}_{i}},z_{{f}j})=\max_{j}{sim}^{(h_{h})}(z_{{f}_{i}},z_{{f}_{j}})).$
(6)

Here, the indicator function $\mathbb{I}$ checks if the similarity between
element $z_{{f}_{i}}$ and any other element $z_{{f}_{j}}$ is the maximum among
all similarities. $h_{h}$ is the index of attention head. Next, the self-
attention mechanism calculates a weighted sum of the values associated with
the selected elements using the attention weights for each attention head:

$z_{{ref}}=\sum_{j}\mathcal{A}^{(h_{h})}_{hard}(z_{{f}_{i}})\cdot(z_{{f}_{j}}).\vspace{-2.5mm}$
(7)

The refined information $z_{{ref}}$ represents the output of the self-
attention mechanism for the $h_{h}$-th attention head. This process is
repeated for all attention heads. The resulting refined information from all
the attention heads is then concatenated and linearly transformed to produce
the final refined representation. It highlights the most important elements
within the fused representation, considering multiple attention heads. This
refined representation enhances the discriminative power and overall quality
of the fused features. Finally, the fused representation $z_{\text{f}}$ is
added to $z_{ref}$ and then passed through the decoder.

Table 1: Quantitative results for multi-organ segmentation: Green \- best,
Blue \- second-best (R50=ResNet50).

Method Mean scores Organ-wise dice similarity coefficient (DSC) DSC HD Aorta
Gallbladder KidneyL KidneyR Liver Pancreas Spleen Stomach R50 UNet $74.68$
$36.87$ $87.74$ $63.66$ $80.60$ $78.19$ $93.74$ $56.90$ $85.87$ $74.16$
DeepLabV3+ $77.63$ $39.95$ 88.04 $66.51$ 82.76 $74.21$ $91.23$ $58.27$ $87.43$
$73.53$ R50 Att-UNet $75.57$ $36.97$ 55.92 $63.91$ $79.20$ $72.71$ $93.56$
$49.37$ $87.19$ $74.95$ R50 ViT $71.29$ $32.87$ $73.73$ $55.13$ $75.80$
$72.20$ $91.51$ $45.99$ $81.99$ $73.95$ TransUNet 77.48 31.69 87.23 63.13
$81.87$ $77.02$ $94.08$ $55.86$ $85.08$ $75.62$ VQUNet 63.44 68.79 $78.99$
$50.74$ 67.32 $61.91$ 89.94 33.96 73.83 50.87 TransVQUNet-2s2h $65.44$ $40.79$
$80.95$ 48.29 $71.42$ 61.89 $90.90$ $36.88$ $77.59$ $55.61$ TransVQUNet-8s2h
$68.41$ $35.05$ $83.57$ $54.53$ $73.73$ $66.21$ $92.39$ $39.45$ $80.26$
$57.10$ SynergyNet-2s2h 78.81 $26.19$ $85.31$ $61.14$ 81.89 79.75 94.42 56.66
89.81 81.51 SynergyNet-8s2h 79.65 23.29 86.10 65.49 82.78 79.23 95.06 58.28
88.95 81.30 SynergyNet-8s8h 77.33 20.56 $85.79$ $61.11$ $81.69$ $77.07$
$94.44$ 64.80 $86.40$ $75.28$

##  3 Experimental Platform

Datasets: We utilized four open-source medical segmentation datasets for our
experiments. The Synapse Multi-Organ Segmentation dataset [1] consists of 30
clinical CT cases with annotated segmentation masks for eight abdominal
organs. We followed the configuration described in [5], using 18 cases for
training and 12 cases for testing. The ACDC dataset [2] is a cardiac MRI
dataset with 100 exams, including labels for the left ventricle (LV), right
ventricle (RV), and myocardium (MYO). We divided the dataset into 70 training
samples, 10 validation samples, and 20 testing samples as per [5]. For skin
lesion segmentation, we adopted the ISIC 2018 dataset [8] and followed the
division into train, validation, and test sets as per previous work [3, 24].
The Brain Tumour Segmentation (BTS) dataset [7] comprises 233 volumetric
T1-weighted contrast-enhanced images from 233 patients (with a total of 3064
2D slices), including three types of brain tumors (meningioma, glioma, and
pituitary tumor) with corresponding binary masks. We maintained an approximate
80:20 ratio for the training and test sets.

Metrics: We utilize the Dice Similarity Coefficient (DSC) and the 95%
Hausdorff Distance (HD) metrics for the synapse and ACDC datasets to follow
the segmentation challange standards and benchmarking. For the ISIC-18 and BTS
datasets, we employ a more comprehensive range of metrics per segmentation
challenge benchmarking, including the Intersection over Union (IOU), DSC,
Specificity (SP), Sensitivity (SE), and Accuracy (ACC). For HD, lower is
better. For other metrics, higher is better.

Implementation Details: We use PyTorch framework and train the models on three
RTX 2080 GPUs, each with 11GB of memory. The input image size was set to 224 ×
224. During training, we used a batch size of 8 and a learning rate of 0.01.
We utilized the SGD optimizer with a momentum of 0.9 and weight decay of
0.0001. We employed data augmentations, such as flipping and rotating.

Architecture Configuration: SynergyNet employs a ResNet50 encoder pre-trained
on the ImageNet dataset, although we have no restriction on the choice of
architecture for encoder. The quantizer module utilizes a codebook size of
$K=1024$. The quantizer, DisConX and Refinement module maintain a hidden
dimension of $dim=512$. We evaluate multiple SynergyNet variants, for example,
SynergyNet-8s2h implies that $h_{s}=8$ and $h_{h}=2$, i.e., it has 8 DisConX
heads and 2 refinement heads. The pre-and post-quantization blocks consist of
two convolution block. The decoder has the same depth as the encoder.

Techniques for Comparison: We compare SynergyNet against four CLS methods,
i.e., UNet [23], Att-UNet [19], DeeplabV3+ [6] R50ViT [9, 5], TransUNet [5]
and two DLS methods, i.e.,VQUNet [27, 26, 25] and TransVQUNet. TransVQUNet
architecture is a combination of VQUNet and TransUNet. It consists of an
encoder followed by a quantizer module and a transformer bottleneck, similar
to the bottleneck of TransUNet. We kept the hyperparameters and architectural
design consistent across all the methods for consistency.

##  4 Experimental Results

###  4.1 Synapse multi-Organ segmentation

Table 1 compares SynergyNet with both CLS and DLS methods. SynergyNet
outperforms both CLS and DLS methods by a significant margin. Quantitatively,
SynergyNet-8s2h achieves a 2.17pp improvement in DSC and a 12.20pp
deterioration in HD compared to TransUNet, while showing an 11.24pp
improvement in DSC and an 11.16pp deterioration in HD compared to
TransVQUNet-8h (pp= percentage point). SynergyNet-8s8h variant shows the best
results in terms of HD metric. SynergyNet demonstrates superior accuracy in
delineating the organs and capturing the boundary between them. It outperforms
other methods in learning both coarse-grained anatomical structures (e.g.,
stomach and liver) and fine-grained anatomical structures (e.g., gallbladder
and spleen). TransUNet, a well-engineered CLS method, exhibits comparable
performance in learning fine-grained structures. On the other hand, DLS
methods can capture coarse anatomical structures but struggle to capture fine-
grained boundaries. SynergyNet benefits from the complementary information
extracted by continuous and discrete latent spaces. Fig. 3 further highlights
the effectiveness of SynergyNet in accurately segmenting fine/coarse and
complex structures. SynergyNet yields more robust and precise segmentation
results even in the presence of intricate variations.

Interpretability Analysis: Here, we analyze the bottleneck architecture to
evaluate learned representations. Fig. 2 visualizes the GradCAMs, revealing
that CLS methods excel in capturing fine organ boundaries, while DLS methods
excel in locating organs but struggle with fine boundary details. In contrast,
SynergyNet effectively captures both fine and coarse boundaries, emphasizing
the importance of leveraging complementary information. These findings further
support the significance of synergistic effects.

Figure 2: Grad-CAM visualization. Figure 3: Delineations on Synapse (first and
second row) and ACDC (third and fourth row) datasets are shown with color-
coded (First row, yellow: liver, blue: right kidney, green: left kidney, light
blue: pancreas. Second row, blue, purple, and yellow represent the RV, LV, and
MYO, respectively.). The overlapping white bounding box represents errors made
by models.

###  4.2 Cardiac Segmentation

From Table 2, we note that the proposed SynergyNet outperforms both continuous
and discrete baselines. SynergyNet can effectively capture complex
heterogeneous structures. Compared to TransUNet and TransVQUNet-8s2h,
SynergyNet-8s2h demonstrates 0.07pp and 11.61pp higher DSC and 0.06pp and
3.23pp lower HD. The qualitative results are shown in Fig. 3 further validate
effectiveness of our approach in delivering more accurate segmentation
results.

Table 2: Quantitaive results for cardiac segmentation

Method Mean Scores Class-wise DSC DSC HD RV Myo LV R50 UNet 87.94 2.01 84.62
84.52 93.68 DeepLabV3+ 88.35 4.45 85.65 85.55 93.85 R50 AttnUNet 86.90 2.10
83.27 84.33 93.53 R50 ViT 86.19 1.98 82.51 83.01 93.05 TransUNet 89.71 1.92
86.67 87.27 95.18 VQUNet 78.15 3.19 70.14 74.13 90.13 TransVQUNet-2s2h 74.40
4.27 64.69 72.75 85.77 TransVQUNet-8s2h 78.17 4.63 69.48 75.95 89.05
SynergyNet-2s2h 88.96 2.41 86.80 85.51 94.60 SynergyNet-8s2h 89.78 1.86 87.68
86.60 95.06 SynergyNet-8s8h 89.37 2.14 87.63 86.49 94.98

###  4.3 Skin Lesion Segmentation

Figure 4: Segmentation maps on BTD (first two rows) and ISIC-2018 (last two
rows) datasets. Actual and predicted pathological regions are shown in Red and
Green, respectively.

Table 3 demonstrates the quantitative results on the ISIC 2018 dataset.
Compared to CLS methods, DLS-based approaches can effectively capture shapes
like lesions, which typically exhibit less variability in terms of shape and
size compared to organs and cardiac structures. However, the proposed
SynergyNet method consistently outperforms both CLS and DLS-based methods,
showcasing its ability to generalize well across different scenarios. Fig. 4
further highlights SynergyNet’s ability to capture both coarse and fine-
grained structured skin lesions. CLS-based methods tend to over-segment non-
contour structures, while DLS-based methods such as VQUNet tend to under-
segment lesions. In contrast, SynergyNet successfully and accurately segments
lesions with smoother boundaries, demonstrating the importance of learning
synergistic representations.

Table 3: Quantitaive results for Skin lesion segmentation

IOU DSC ACC SP SE R50 UNet 77.86 87.55 94.05 96.69 85.86 DeepLabV3+ 78.52
87.59 94.29 95.97 86.46 R50 AttnUNet 78.43 87.91 94.13 96.23 87.60 R50 ViT
78.13 87.45 93.53 96.13 87.10 TransUNet 78.97 88.25 94.32 96.48 87.60 VQUNet
79.13 88.35 94.46 97.09 86.29 TransVQUNet-2s2h 79.83 88.78 94.58 96.62 88.21
TransVQUNet-8s2h 79.68 88.69 94.54 96.64 88.01 SynergyNet-2s2h 79.80 88.77
94.56 96.59 88.26 SynergyNet-8s2h 80.68 89.31 94.91 97.37 87.28
SynergyNet-8s8h 80.05 88.92 94.66 96.81 87.98

###  4.4 Brain Tumour Segmentation

SynergyNet achieves the best score on all metrics on the the BTS dataset
(Table 4) and outperforms the second-best method by a large margin. From Fig.
4, we note that DLS methods tend to lose boundary information, but they
segment regions of interest more accurately than CLS methods for this
particular case. On the other hand, SynergyNet consistently identifies regions
of interest with smoother boundaries, surpassing both CLS and DLS methods.
SynergyNet accurately predicts lesions, even in case of varying locations,
sizes, and modality views. It effectively suppresses irrelevant information,
such as the background.

Table 4: Quantitaive results for Brain tumour segmentation

Method IOU DSC ACC SP SE R50 UNet 63.90 78.00 99.33 99.77 77.22 DeepLabV3+
65.23 78.90 99.60 99.77 77.22 R50 AttnUNet 63.15 77.65 99.21 99.57 76.88 R50
ViT 62.13 76.35 99.18 99.44 76.62 TransUNet 62.36 76.82 99.24 99.62 76.90
VQUNet 58.00 73.40 99.11 99.66 72.66 TransVQUNet-2s2h 60.92 75.72 99.10 99.53
72.24 TransVQUNet-8s2h 61.22 75.94 99.25 99.73 72.44 SynergyNet-2s2h 70.64
82.81 99.52 99.66 80.25 SynergyNet-8s2h 70.94 83.00 99.55 99.86 80.45
SynergyNet-8s8h 69.88 82.27 99.43 99.76 79.85

##  5 Ablation Studies

Unless otherwise mentioned, we use $K=1024$, $dim=512$, $h_{h}=8$, $h_{s}=2$
and backbone as ResNet-50.

Table 5: Codebook size (${K}$) analysis ${K}$ | ${dim}$ | Synapse | ACDC
---|---|---|---
DSC | HD | DSC | HD
1024 | 512 | 79.65 | 23.29 | 89.78 | 1.86
512 | 512 | 79.61 | 23.89 | 88.89 | 2.42
256 | 512 | 79.21 | 29.47 | 89.29 | 1.62
128 | 512 | 78.48 | 30.97 | 89.01 | 1.79
64 | 512 | 77.29 | 88.67 | 88.79 | 1.98

###  5.1 Codebook analysis

From Table 5, we observe a direct relationship between $K$ and the performance
of SynergyNet on the Synapse dataset, where increasing $K$ leads to a notable
improvement in HD scores. On the ACDC dataset, the trend is different, such
that $K=256$ gives the best HD score, and $K=128$ and $K=1024$ give comparable
results. A smaller codebook size in the quantization module leads to higher
compression and more aggressive quantization, but it can result in the loss of
local information. This loss of fine-grained details and subtle variations can
negatively impact the segmentation model’s ability to capture intricate
boundaries, leading to lower HD scores. To achieve the best segmentation
performance, the codebook size needs to be chosen so as to balance compression
and preservation of local information.

###  5.2 Hidden Dimension ($dim$) Analysis

From Table 6, we observe that using a codebook size of $K$ = 1024 with $dim$
(hidden dimension size) greater than 512 or $dim$ less than 512 leads to a
deterioration in performance. Empirically, we found that setting $K$ to be
twice the value of $dim$ ($K$ = 2 * $dim$) yields the best performance. Thus,
multiple parameters, including the dataset characteristics, influence the
overall performance.

Table 6: Hidden Dimension ($dim$) Analysis ${K}$ | ${dim}$ | Synapse | ACDC
---|---|---|---
DSC | HD | DSC | HD
1024 | 2048 | 78.48 | 30.28 | 88.58 | 2.33
1024 | 1024 | 77.61 | 29.53 | 88.64 | 2.12
1024 | 512 | 79.65 | 23.29 | 89.78 | 1.86
1024 | 256 | 79.29 | 30.07 | 89.18 | 2.29
1024 | 128 | 78.98 | 35.81 | 88.87 | 2.60

###  5.3 Bottleneck Size Analysis

Table 7 presents the impact of the size of the DisconX module and the
Refinement Module of SynergyNet on the overall segmentation performance. The
combinations $h_{s}=8,h_{h}=0$ and $h_{s}=2,h_{h}=0$ denote the configurations
without the refinement module. We observe a significant deterioration in the
overall performance when the refinement module is not utilized. For the
Synapse dataset, the best value of DSC is obtained for $h_{s}=8,h_{h}=2$ and
the best value of HD is obtained for $h_{s}=8,h_{h}=8$. For the ACDC dataset,
the combination $h_{s}=8,h_{h}=2$ results in the best values of DSC and HD.
Overall, the optimal module size is dataset and task-dependent. It is crucial
to consider these factors when determining the optimal sizes for the DisConX
and refinement modules.

Table 7: Bottleneck Size Analysis $h_{s}$ | $h_{h}$ | Synapse | ACDC
---|---|---|---
DSC | HD | DSC | HD
8 | 0 | $78.62$ | $26.82$ | $87.95$ | $2.33$
8 | 2 | 79.65 | 23.29 | 89.78 | 1.86
2 | 8 | $78.45$ | $25.19$ | $88.96$ | $2.41$
8 | 8 | $77.33$ | 20.56 | 89.68 | 2.14
2 | 2 | $78.81$ | $26.19$ | $88.96$ | $2.41$
2 | 0 | $78.12$ | $27.71$ | $87.36$ | $3.11$

###  5.4 Contribution of DisConX Module

The DisConX module plays a crucial role in the SynergyNet’s ability to learn
fine-grained local features. To understand its contribution, we create a
variant SynergyNet(Fusion), which replaces the DisConX module with a simple
feature fusion approach. It combines discrete and continuous representations
and passes them through a refinement module. As shown in Table 8, this variant
attains lower performance, which clearly demonstrates that the DisConX module
is essential for learning fine-grained local features. Overall, results
indicate that selectively attending to complementary information preserves
higher-quality discriminative and semantic information.

Table 8: Contribution of DisConX module Backbone | Synapse | ACDC
---|---|---
DSC | HD | DSC | HD
SynergyNet(Fusion) | 78.79 | 26.91 | 88.20 | 2.57
SynergyNet | 79.65 | 23.29 | 89.03 | 2.17

###  5.5 Backbone Analysis

We evaluate SynergyNet with ResNet and EfficientNet backbones. For the Synapse
dataset, ResNet-50 achieved a DSC score of 79.65%, and EfficientNet-B7
achieved the lowest HD score of 21.53%. In the ACDC dataset, ResNet-101
performed the best on both metrics. EfficientNet-B0 exhibited remarkable
boundary delineation capabilities despite its shallower architecture. Please
Refer to the supplementary materials for parameters analysis.

Table 9: Backbone Analysis Backbone | Synapse | ACDC
---|---|---
DSC | HD | DSC | HD
ResNet-18 | 77.28 | 27.88 | 88.82 | 2.08
ResNet-34 | 78.35 | 25.58 | 89.42 | 1.93
ResNet-50 | 79.65 | 23.29 | 89.78 | 1.86
ResNet-101 | 78.66 | 28.50 | 91.49 | 0.91
EfficientNet-B0 | 78.05 | 27.77 | 91.13 | 1.29
EfficientNet-B7 | 78.70 | 21.53 | 90.81 | 2.90

Limitations: i) The quantizer’s reliance on selecting the most similar
codebook item for input representation may lead to difficulties in capturing
intricate patterns, potentially causing information loss. ii) Both CLS and DLS
struggle to effectively model inter-class relationships, resulting in
increased false negatives. SynergyNet reduces false negatives compared to CLS
and DLS but still can be further improved.

##  6 Conclusion and Future Work

We propose SynergyNet, a novel bottleneck architecture for learning
complementary information from CLS and DLS. Extensive experiments and ablation
studies confirm that SynergyNet captures both fine and coarse-grained details
in the learned representations and outperforms previous works. SynergyNet is a
promising model for medical image analysis that offers high performance.
Integrating SynergyNet with efficient architectures like Swin Transformer [4],
HiFormer [14] and others shows promise for further advancements. Exploring
SynergyNet’s performance with unsupervised models is an intriguing research
area that enables leveraging unlabeled data to enhance capabilities in medical
image analysis. This holds the potential to improve efficiency and performance
in this critical domain.

## References

  * [1] Multi-atlas abdomen labeling challenge. synapse multi-organ segmentation dataset.  Available at: https://www.synapse.org/#!Synapse:syn3193805/wiki/217789, 2015.
  * [2] Acdc (automated cardiac diagnosis challenge).  Available at: https://www.creatis.insa-lyon.fr/Challenge/acdc, 2017\.
  * [3] Reza Azad, Maryam Asadi-Aghbolaghi, Mahmood Fathy, and Sergio Escalera.  Bi-directional convlstm u-net with densley connected convolutions.  In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0, 2019.
  * [4] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  In Proceedings of the European Conference on Computer Vision, pages 205–218, 2022.
  * [5] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou.  Transunet: Transformers make strong encoders for medical image segmentation.  arXiv preprint arXiv:2102.04306, 2021.
  * [6] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [7] Jun Cheng.  Brain tumor dataset.  https://doi.org/10.6084/m9.figshare.1512427.v5, 2017.
  * [8] Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic).  In Proceedings of the 15th international symposium on biomedical imaging (ISBI 2018), pages 168–172, 2018.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [10] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021.
  * [11] Azade Farshad, Yousef Yeganeh, Peter Gehlbach, and Nassir Navab.  Y-net: A spatiospectral dual-encoder network for medical image segmentation.  In Proceedings of the 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), pages 582–592, 2022\.
  * [12] Hugo Gangloff, Minh-Tan Pham, Luc Courtrai, and Sébastien Lefèvre.  Leveraging vector-quantized variational autoencoder inner metrics for anomaly detection.  In Proceedings of the International Conference on Pattern Recognition (ICPR), pages 435–441, 2022.
  * [13] Mark S Graham, Petru-Daniel Tudosiu, Paul Wright, Walter Hugo Lopez Pinaya, U Jean-Marie, Yee H Mah, James T Teo, Rolf Jager, David Werring, Parashkev Nachev, et al.  Transformer-based out-of-distribution detection for clinically safe segmentation.  In International Conference on Medical Imaging with Deep Learning, pages 457–476, 2022.
  * [14] Moein Heidari, Amirhossein Kazerouni, Milad Soltany, Reza Azad, Ehsan Khodapanah Aghdam, Julien Cohen-Adad, and Dorit Merhof.  Hiformer: Hierarchical multi-scale representations using transformers for medical image segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6202–6212, 2023.
  * [15] Ge Jin, Xu Chen, and Long Ying.  Deep multi-task learning for an autoencoder-regularized semantic segmentation of fundus retina images.  Mathematics, 10(24):4798, 2022.
  * [16] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  Advances in neural information processing systems, 34:9694–9705, 2021.
  * [17] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [18] Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun.  Attention bottlenecks for multimodal fusion.  Advances in Neural Information Processing Systems, 34:14200–14213, 2021.
  * [19] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al.  Attention u-net: Learning where to look for the pancreas.  arXiv preprint arXiv:1804.03999, 2018.
  * [20] Walter HL Pinaya, Petru-Daniel Tudosiu, Robert Gray, Geraint Rees, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso.  Unsupervised brain imaging 3d anomaly detection and segmentation with transformers.  Medical Image Analysis, 79:102475, 2022.
  * [21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In Proceedings of the International conference on machine learning, pages 8748–8763, 2021.
  * [22] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.  Do vision transformers see like convolutional neural networks?  Advances in Neural Information Processing Systems, 34:12116–12128, 2021.
  * [23] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 234–241, 2015\.
  * [24] Jiacheng Ruan, Suncheng Xiang, Mingye Xie, Ting Liu, and Yuzhuo Fu.  Malunet: A multi-attention and light-weight unet for skin lesion segmentation.  In Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 1150–1156, 2022.
  * [25] Ainkaran Santhirasekaram, Avinash Kori, Mathias Winkler, Andrea Rockall, and Ben Glocker.  Vector quantisation for robust segmentation.  In Proceedings of the 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022), pages 663–672, 2022.
  * [26] Stanislaw Szymanowicz, James Charles, and Roberto Cipolla.  Discrete neural representations for explainable anomaly detection.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 148–156, 2022.
  * [27] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  Advances in neural information processing systems, 30, 2017.
  * [28] Haoqing Wang, Xun Guo, Zhi-Hong Deng, and Yan Lu.  Rethinking minimal sufficient representation in contrastive learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16041–16050, 2022.
  * [29] Feiniu Yuan, Zhengxiao Zhang, and Zhijun Fang.  An effective cnn and transformer complementary network for medical image segmentation.  Pattern Recognition, 136:109228, 2023.


# SEMIE: SEMantically Infused Embeddings with Enhanced Interpretability for
Domain-specific Small Corpus

Rishabh Gupta  Rajesh N Rao
Research and Technology Center, Robert Bosch, Bengaluru, India
{Gupta.Rishabh<EMAIL_ADDRESS>

###### Abstract

Word embeddings are a basic building block of modern NLP pipelines. Efforts
have been made to learn rich, efficient, and interpretable embeddings for
large generic datasets available in the public domain. However, these
embeddings have limited applicability for small corpora from specific domains
such as automotive, manufacturing, maintenance and support, etc. In this work,
we present a comprehensive notion of interpretability for word embeddings and
propose a novel method to generate highly interpretable and efficient
embeddings for a domain-specific small corpus. We report the evaluation
results of our resulting word embeddings and demonstrate their novel features
for enhanced interpretability.

##  1 Introduction

Distributed representations of words, also termed as word embeddings, have
been used extensively to excel at various applications such as parsing
Lazaridou et al. (2013); Bansal et al. (2014), named entity recognition Guo et
al. (2014), image captioning You et al. (2016) and sentiment analysis Socher
et al. (2013). They have also proven effective in modeling cognitive
operations such as the judgement of word similarity Turney and Pantel (2010);
Baroni and Lenci (2010), and the brain activity elicited by specific concepts
Mitchell et al. (2008). However, these representations contain mappings of
words to vectors of real numbers in dense and continuous space, and thus,
inherently difficult to interpret.

Recent studies Murphy et al. (2012); Fyshe et al. (2014) suggest that sparsity
and non-negativity of the word embeddings are two important characteristics
that make them interpretable. The sparsity makes each word vector contain a
small number of active (non-zero) dimensions Olshausen and Field (1997), which
helps in increasing their separability and stability in the presence of noise
Lewicki and Sejnowski (2000); Donoho et al. (2005). In addition, the studies
define the notion of interpretability in terms of the coherence of dimensions
of the word embeddings Faruqui et al. (2015); Lipton (2018); Subramanian et
al. (2018). In other words, the word embeddings are considered as
interpretable if their dimensions denote specific semantic concepts. However,
these studies are primarily focused on pre-trained word embeddings like GloVe
Pennington et al. (2014) and word2vec Mikolov et al. (2013). These pre-trained
embeddings are generated using millions of documents from generic public
domain datasets such as Wikipedia and Google News, which contain billions of
words. Also, to interpret the thousands of dimensions of the sparse non-
negative (hereafter ‘SNN’) word embeddings111Usually the SNN word embeddings
have at least 10x-times more dimensions as compared to the dense word
embeddings. Faruqui et al. (2015); Subramanian et al. (2018) and understand
the sense they correspond to, we still need human judges who put in manual
efforts and read these unlabeled dimensions.

In addition to the aforementioned semantic similarities, the recent studies
suggest that the interpretability of word embeddings should also consider the
semantic dissimilarities such as identification of discriminative word triples
Krebs and Paperno (2016); Krebs et al. (2018) . Given a triplet of words
($w_{1}$, $w_{2}$, $d$), the word embeddings should be able to determine
whether $d$ is a discriminative feature between two concepts $w_{1}$ and
$w_{2}$. For example, the word “buckle” is a discriminative feature in the
triplet (“seat belt”, “tires”, “buckle”) that characterizes the first concept
but not the second. Researchers have formulated this property as a binary
classification task and proposed machine learning and similarity-based methods
to evaluate the word embeddings Zhang and Carpuat (2018); Dumitru et al.
(2018); Grishin (2018). However, to perform these evaluations for a domain-
specific small corpus, we would need a manually curated set of discriminative
(positive) and non-discriminative (negative) triples, which can be costly and
time-consuming to curate.

_Comprehensive Notion of Interpretability_ : In conclusion, we can say that
the interpretability of word embeddings is expressed in terms of both semantic
similarities and dissimilarities while representing them in SNN embedding
space. Here, semantic similarities correspond to dimensional coherence,
whereas semantic dissimilarities refer to properties such as identification of
discriminative word triples. This raises the following question:

_How can we generate word embeddings for domain-specific small corpus, which
are interpretable in terms of both semantic similarities and dissimilarities,
when represented in SNN embedding space?_

To address the above-mentioned question, we present a novel method to generate
such word embeddings, which we name as SEMantically Infused Embeddings
(SEMIE). We showcase the enhanced interpretability (both in terms of semantic
similarities and dissimilarities) capabilities of SEMIE while representing
them in SNN embedding space. We also demonstrate the efficiency of SEMIE on
the downstream classification task, both in dense and SNN embedding space.

##  2 Preliminaries

In this section, we discuss following preliminaries based on which we build
our method to generate the SEMantically Infused Embeddings (SEMIE):

  1. 1.

A Semantic Infusion technique Gupta and Rao (2020), which helps in leveraging
the meta-data of the text corpus to bring in semantic knowledge within the
word embeddings.

  2. 2.

A mathematical framework Yin and Shen (2018), which helps in determining the
optimal dimension of the word embeddings for a given text corpus. Such that
these embeddings are expressive enough to capture all possible word relations
while avoiding over-fitting.

###  2.1 Semantic Infusion Technique

Semantic Infusion Gupta and Rao (2020) is an efficient technique to associate
meta-data within the text corpus. Using this, we can infuse special markers,
referred to as _Anchors_ , within each document of the corpus. Given a
document $d_{i}$ of length $l_{d_{i}}$ and belonging to a category class
$c_{j}$ in the corpus, an anchor term $A_{c_{j}}$ is infused at $I_{freq}$
random and non-consecutive positions within the document. Here, $I_{freq}$
denotes _Infusion Frequency_ and its value is computed as:

$I_{freq}=\left\lceil\frac{\log_{2}(l_{d_{i}})}{2}\right\rceil$ (1)

For e.g., a sentence _“new procurement scheme for farmers to focus on all
crops.”_ of a document class $c_{j}=$ _India_ will be processed as follows:

_“new A_India procurement scheme A_India for farmers to focus on all A_India
crops.”_

Intuitively, when an infused corpus like this is used to generate the word
embeddings, the anchor terms help in identifying the semantically significant
words based on the co-occurrence statistics.

###  2.2 Optimal Dimensionality

The selection of dimensionality hyper-parameter is critical for the efficiency
of any word embeddings. Researchers select dimensionality either in an ad hoc
manner or using a grid search. Otherwise, they choose $300$, which is the most
commonly used dimensionality while generating the word embeddings using
millions of documents Mikolov et al. (2013); Pennington et al. (2014);
Bojanowski et al. (2017). But, for a domain-specific small corpus, we need a
precise optimal dimensionality, otherwise it may lead to sub-optimal
performance.

To identify the optimal dimensionality, we leverage the mathematical framework
as proposed in Yin and Shen (2018). It determines the optimal dimensionality
$d$, where $d\leq k$, as the one which minimizes the Pairwise Inner Product
(PIP) loss between an oracle embeddings $E\in\mathbb{R}^{n\times k}$ and the
trained embeddings $\hat{E}\in\mathbb{R}^{n\times d}$ as defined by:

$\displaystyle\mathinner{\\!\left\lVert\text{PIP}(E)-\text{PIP}(\hat{E})\right\rVert}={}\mathinner{\\!\left\lVert
EE^{T}-\hat{E}\hat{E}^{T}\right\rVert}$ (2)
$\displaystyle=k-d+2\mathinner{\\!\left\lVert\hat{E}^{T}E^{\bot}\right\rVert}^{2}$

We use this framework as an intermediatory step to compute the optimal
dimensionality of the corpus while generating SEMIE, as shown in Figure 1.

Figure 1: Flowchart for the proposed method. Given a domain-specific
categorical corpus $\mathcal{D}$, we first generate the semantically infused
corpus using the semantic infusion technique as explained in the Section 2.1.
Next, using a mathematical framework as explained in the Section 2.2, we
compute the optimal dimensionality $d$ for the infused corpus and generate the
infused optimal dimensional embeddings $E^{Inf}_{OPT}$ using word2vec. Last,
we generate the semantically infused embeddings $E_{SEMIE}$ as described in
Section 3. In addition, we also generate the baseline optimal dimensional
embeddings $E_{OPT}$ which we use to evaluate $E_{SEMIE}$ in terms of
Interpretability and downstream classification task performance.

##  3 Methodology

In this section, we describe the proposed method used to generate SEMIE, as
illustrated in Figure 1. Given a domain-specific categorical corpus of $N$
documents, $\mathcal{D}=[d_{1}\dots d_{N}]$, where each document $d_{i}$
belongs to a category class $c_{j}$, in the set of $M$ classes,
$\mathcal{C}=[c_{1}\dots c_{M}]$. We generate the semantically infused
embeddings $E_{SEMIE}\in\mathbb{R}^{(V+M)\times d}$, where $V$ is the
vocabulary size of the corpus, $M$ are the anchor terms infused using the
semantic infusion technique, and $d$ is the computed optimal dimensionality of
word embeddings of the corpus. The key steps of the method are as follows:

  1. 1.

Semantically Infused Corpus

We take the domain-specific categorical corpus $\mathcal{D}$ and generate the
semantically infused corpus using the semantic infusion technique, as
explained in Section 2.1. After this step, the vocabulary size of the entire
corpus increases from $V$ to $(V+M)$, where $V$ is the vocabulary size of the
initial corpus, and $M$ are the infused additional anchor terms.

  2. 2.

Optimal Dimensional Embeddings

We take the semantically infused corpus, compute the optimal dimensionality
$d$, as explained in Section 2.2 and generate the infused optimal dimensional
embeddings $E^{Inf}_{OPT}\in\mathbb{R}^{(V+M)\times d}$ using the word2vec
method as demonstrated in the Figure 1.

  3. 3.

Semantically Infused Embeddings

We take the infused optimal dimensional embeddings
$E^{Inf}_{OPT}\in\mathbb{R}^{(V+M)\times d}$ and generate the semantically
infused embeddings $E_{SEMIE}\in\mathbb{R}^{(V+M)\times d}$ as follows:

For each column $C_{i}$ of the embeddings matrix $E^{Inf}_{OPT}$, we sort the
column in the increasing order of values and then select the values of $M$
anchor terms. For each anchor term $A_{C_{i}}$ and non-anchor word $w_{C_{i}}$
pair in the column $C_{i}$, we compute a semantic weight $w_{s}$, as given in
Equation 3 and add it to the value of non-anchor word $w_{C_{i}}$. This gives
us the semantically infused embeddings $E_{SEMIE}$.

$w_{s}=\frac{A_{C_{i}}}{||index(A_{C_{i}})-index(w_{C_{i}})||}$ (3)

Intuitively, this means that in each column (dimension) of the semantically
infused embeddings $E_{SEMIE}$, the words in the neighborhood of the anchors
will form semantically coherent groups such as top words in Table 4.

Table 1: Detailed statistics of the three datasets used in this work. For each dataset, the optimal embedding dimension $d$ is computed using the mathematical framework as described in Section 2.2. The SNN embedding dimension is taken as $10\times d$. The $\\#$ Anchors are equal to the $\\#$ classes in the dataset. | Dataset
---
| #
---
Docs
| #
---
Anchors / Vocab
($M$/ $V$)
| Class Names
---
| Optimal
---
Dimension
($d$)
| SNN
---
Dimension
($10\times d$)
AG News | 938,076 | 10 / 92,721 |  | Business, Sci/Tech,
---
Entertainment, Sports,
World, Top News, Europe,
Italia, U.S., Health
103 | 1030
NHTSA | 800,815 | 10 / 45,762 |  | Steering, Visibility,
---
Suspension, Structure,
Electrical System, Fuel System,
Air Bags, Seat Belts, Tires
Vehicle Speed Control
113 | 1130
| Yahoo
---
Answers
730,000 | 5 / 131,607 |  | Science & Mathematics, Health,
---
Sports, Business & Finance,
Politics & Government
139 | 1390

##  4 Experimental Setup

We study the efficacy of SEMIE in terms of performance in downstream
classification task both in dense and SNN embedding space. Furthermore, we
demonstrate the enhanced interpretability of SEMIE both in terms of semantic
similarities and dissimilarities when transformed in SNN embedding space.

In the following subsections, we describe the datasets used and the generation
of baseline optimal dimensional embeddings $E_{OPT}$ and semantically infused
embeddings $E_{SEMIE}$ for each dataset. In addition, we describe two state-
of-the-art methods namely SPINE Subramanian et al. (2018) and SPOWV Faruqui et
al. (2015) used to transform the $E_{OPT}$ and $E_{SEMIE}$ to SNN embedding
space. Later, for each dataset, we present the effectiveness of $E_{SEMIE}$ on
downstream classification task (Section 5) and interpretability in terms of
semantic similarities (Section 6) and dissimilarities (Section 6.3) as
compared to $E_{OPT}$.

###  4.1 Datasets

In our experiments, we use $3$ datasets: $(1)$ NHTSA (National Highway Traffic
Safety Administration), a web forum platform, which contains complaints of the
automobile domain registered by the consumers. $(2)$ Yahoo answers, a
question-answering platform, which contains answers to various questions
posted by online users. $(3)$ AG NEWS, a collection of news articles gathered
from more than $2000$ news sources. The statistics of datasets are summarized
in the Table 1.

###  4.2 $E_{OPT}$ and $E_{SEMIE}$ Embeddings

For each dataset of vocabulary size $V$, we compute the optimal dimensionality
hyper-parameter $d$ as described in the Section 2.2 and generate the baseline
optimal dimensional embeddings $E_{OPT}\in\mathbb{R}^{V\times d}$ using the
word2vec method. In addition, we generate the semantically infused corpus
while infusing $M$ anchor terms and generate the semantically infused
embeddings $E_{SEMIE}\in\mathbb{R}^{(V+M)\times d}$ for each dataset as
explained in the Section 3.

In our experiments, we compare the $E_{SEMIE}$ embeddings w.r.t. the baseline
$E_{OPT}$ embeddings for each dataset and demonstrate the effectiveness of
$E_{SEMIE}$ in terms of downstream classification task performance and
interpretability.

Table 2: The effectiveness of SEMIE in terms of the downstream classification accuracy both in dense and SNN embedding space. $E_{SEMIE}$ either improves the classification accuracy or it is comparable to baseline $E_{OPT}$ across all datasets for both dense and SNN embedding space. Dataset | Dense Embeddings | Sparse Embeddings
---|---|---
$E_{OPT}$ | $E_{SEMIE}$ |  |  $E_{OPT}$-
---
SPINE
|  $E_{SEMIE}$-
---
SPINE
|  $E_{OPT}$-
---
SPOWV
|  $E_{SEMIE}$-
---
SPOWV
AG News | 76.95% | 78.10% | 77.55% | 78.28% | 77.48% | 77.93%
NHTSA | 87.45% | 87.45% | 87.48% | 87.48% | 88.63% | 87.80%
| Yahoo
---
Answers
75.47% | 76.09% | 72.40% | 73.18% | 74.12% | 74.84%
Table 3: The precision score comparisons on the word intrusion detection tests where higher precision numbers indicate more interpretable dimensions. $E_{SEMIE}$ improves the interpretability as compared to $E_{OPT}$ for NHTSA and Yahoo Answers datasets in SNN embedding space. Dataset | Sparse Embeddings
---|---
|  $E_{OPT}$-
---
SPINE
|  $E_{SEMIE}$-
---
SPINE
|  $E_{OPT}$-
---
SPOWV
|  $E_{SEMIE}$-
---
SPOWV
AG News | 32.67 | 30.00 | 21.67 | 18.00
NHTSA | 25.67 | 30.00 | 29.67 | 30.00
| Yahoo
---
Answers
24.33 | 31.00 | 34.00 | 34.00

###  4.3 Sparse Non-negative Transformation

To demonstrate the enhanced interpretability of SEMIE, we transform both
$E_{OPT}$ and $E_{SEMIE}$ to SNN embeddings space for each dataset. We perform
the transformations using $2$ state-of-the-art methods: $(1)$ SPINE: SParse
Interpretable Neural Embeddings Subramanian et al. (2018) and $(2)$ Sparse
Overcomplete Word Vector Representations (SPOWV) Faruqui et al. (2015). Using
SPINE and SPOWV, we transform both $E_{OPT}$ and $E_{SEMIE}$ embeddings of
each dataset to a $\mathbb{R}^{10\times d}$ space such that ($10\times
d$)-dimensional embeddings are both sparse and non-negative. Here, $d$ is the
computed optimal dimension for the word embeddings of the dataset as mentioned
in the Table 1.

Hyper-parameters: For SPINE and SPOWV, we use hyper-parameters which are best
performing as per the authors’ recommendations. SPINE- No. of Epochs: $4000$,
Sparsity: $0.85$, Noise: $0.2$; SPOWV- l1_reg: $0.5$, l2_reg: $1e-5$,
num_cores: $1$.

After this step, we have baseline optimal dimensional embeddings: $E_{OPT}$,
$E_{OPT}$-SPINE and $E_{OPT}$-SPOWV and semantically infused embeddings:
$E_{SEMIE}$, $E_{SEMIE}$-SPINE and $E_{SEMIE}$-SPOWV for each dataset.

##  5 Downstream Classification Task

In this section, we evaluate the performance of SEMIE in the downstream
classification task both in dense and SNN embedding space. For each dataset,
we compare the classification accuracy while using $E_{SEMIE}$,
$E_{SEMIE}$-SPINE and $E_{SEMIE}$-SPOWV w.r.t. $E_{OPT}$, $E_{OPT}$-SPINE and
$E_{OPT}$-SPOWV respectively. We use the average of word vectors of the words
in a document as features and experiment with SVMs for the text classification
task. For each dataset, we take a balanced set of $1600$ documents as the
training set and $400$ documents as the test set from each class of the
dataset. We then combine the training sets and test sets and report the
accuracy on the combined test set as listed in the Table 2. Please note that
for this evaluation we removed the $M$ anchor terms from all $E_{SEMIE}$
embedding matrices.

Table 4: The Interpretation and labels of some sampled dimensions. For each dataset, we examine the top $5$ words and top-ranked anchor term for a few randomly chosen dimensions of $E_{SEMIE}$-SPOWV SNN embeddings. Clearly, the word groups are interpretable and semantically coherent with the anchor terms. Also, the anchor terms can be seen as the labels for the dimension as they are able to cater to the interpretations of the dimensions. Dataset |  | Top-ranked Anchor in
---
Sampled Dimension
| Top Words in
---
Sampled Dimension
AG News | A_Health | colon, murdoch, breast, smokers, older
A_Europe | holy, brighton, pope, xvi, benedict
A_Sports | touchdown, seconds, pointer, interceptions, tiebreaking
A_Top News | resumption, islamist, disarmament, zimbabwe, ceasefire
A_U.S. | eliot, blitz, attorney, spitzer, buskirk
NHTSA | A_Electrical System | speakers, audio, radio, buttons, stereo
A_Tires | durability, cupped, engulfed, disintegrated, worn
A_Structure | attachment, creeping, framing, section, weakened
A_Seat Belts | webbing, anchorage, limiter, waist, unbuckle
A_Visibility | obstructing, hybrid, draining, contact, interfering
Yahoo Answers | A_Health | stones, mothers, hypothyroidism, thyroid, nmyth
A_Sports | abuse, hyperactivity, psychiatric, drank, suffers
A_Business & Finance | declined, angle, defects, worker, uninsured
A_Politics & Government | peaceful, openly, dominion, altar, forefathers
A_Science & Mathematics | longitude, language, cartesian, sapiens, coordinates

##  6 Interpretability

We evaluate the interpretability (in terms of semantic similarities) of SEMIE
in SNN embedding space using the intrusion detection tests (Section 6.1).
These tests have widely been used in literature Chang et al. (2009); Murphy et
al. (2012); Fyshe et al. (2014); Subramanian et al. (2018); Faruqui et al.
(2015) to evaluate the interpretability of the word embeddings.

Additionally, in SEMIE, due to the Anchor term infusion, we can get labels for
the dimensions of SNN embeddings. This helps in better interpretation of the
SNN embedding dimensions as the labels are coherent with the semantic concepts
represented by the dimensions (Section 6.2).

###  6.1 Word Intrusion Detection Test

In a particular word intrusion detection test, a human judge is presented with
$5$ words and asked to select the “intruder” or semantically the “odd one
out”. The strategy to generate these tests: for a given dimension (column) of
the embeddings matrix, we select $4$ top-ranked words and a “intruder” word
from the bottom half of the ranked list, which is also present in the top $10$
percentile in at least one other dimension. An example of the test is:
“hyundai”, “crumbled”, “spectra”, “suzuki”, “kia”. Here, “crumbled” is the
intruder.

For each dataset, we compare the precision scores on the word intrusion
detection tests while using $E_{SEMIE}$, $E_{SEMIE}$-SPINE and
$E_{SEMIE}$-SPOWV w.r.t. $E_{OPT}$, $E_{OPT}$-SPINE and $E_{OPT}$-SPOWV
embeddings respectively. We randomly sample $100$ dimensions of all embedding
matrices and generate the intrusion detection tests. In total, we generate
$1800$ tests and get each test annotated independently by the $3$ judges. The
results of these intrusion detection tests are listed in Table 3. Please note
that for this evaluation we removed the $M$ anchor terms from all $E_{SEMIE}$
embedding matrices.

Figure 2: An illustration of the discriminative and non-discriminative
features as identified using SEMIE. We examine the embedding vectors for the
anchors “A_Seat Belts” and “A_Tires” in the $E_{SEMIE}$-SPINE embeddings of
the NHTSA dataset. Clearly, the top words in the dimensions when only one
anchor is active are discriminative features corresponding to the concepts
Seat Belts and Tires, for example, the word“punctured” (an issue specific to
tires). This helps us in creating the discriminative triple such as _(Tire,
Seat Belts, punctured)_. Whereas, the top words in dimensions when both
anchors are active (non-zero) are non-discriminative features such as the word
“crv” (a car model name common across both seat belts and tires). This helps
us in creating the non-discriminative triple such as _(Tire, Seat Belts,
crv)_.

###  6.2 Dimensions’ Labels

In this section, we demonstrate that SEMIE provides a novel mechanism to
automatically label SNN embedding dimensions. These labels are able to cater
to the interpretations of the top words within the dimensions and thus help in
a better understanding and interpretability of the dimensions.

We take the SNN semantically infused embeddings $E_{SEMIE}$-SPOWV for each
dataset and randomly sample a few the dimensions (columns) of the embeddings
matrix. For each sampled dimension, we sort the column in the decreasing order
of values and then select top $5$ words and the top-ranked anchor term in that
dimension as listed in the Table 4. Here, we observe that the semantic groups
formed by the top words in these sampled dimensions are highly coherent with
the top-ranked anchor term.

###  6.3 Semantic Dissimilarities

The interpretability of word embeddings is also expressed in terms of semantic
dissimilarities such as identification of discriminative and non-
discriminative word triples. A word triplet ($w_{1}$, $w_{2}$, $d$) is termed
as discriminative if the feature $d$ characterizes the first concept $w_{1}$
but not the second concept $w_{2}$ and thus act as a discriminative feature
between the two concepts as listed in Table 5.

SEMIE provides a novel mechanism to identify the discriminative and non-
discriminative triples in an unsupervised fashion. To showcase this, we take
the $E_{SEMIE}$-SPINE SNN embeddings of the NHTSA dataset and select the
vectors of “A_Seat Belts” and “A_Tires” anchor terms. For each dimension in
both vectors, we then identify which one is active (non-zero) and which one is
not active (zero), as illustrated in the Figure 2.

We observe that the top five words for the dimensions where both anchors are
active are non-discriminative features corresponding to the concepts “Tire”
and “Seat Belts” in the dataset. For example, a mix of seat belt and tire
issues or the common car model names. Thus, we can formulate the non-
discriminative triples such as _(Tire, Seat Belts, crv)_ and _(Tire, Seat
Belts, passport)_ related to the concepts “Tire” and “Seat Belts”.

Whereas, the top five words for the dimensions where only one anchor is active
are discriminative features such as specific tire manufacturing company names,
issues related specifically to either tires or seat belts, and injuries
related to seat belts. Thus, we can formulate the discriminative triples such
as _(Tire, Seat Belts, punctured)_ and _(Seat Belts, Tire, buckle)_ related to
the concepts “Tire” and “Seat Belts”. The identified discriminative and non-
discriminative triplets for the concepts “Tire” and “Seat Belts” are listed in
the Table 5.

Table 5: The discriminative and non-discriminative triples. A triplet (Concept 1, Concept 2, Feature) is termed as discriminative if the “Feature” characterizes the concept “Concept 1” but not the concept “Concept 2”. We examine the two concepts “Seat Belts” and “Tires” using $E_{SEMIE}$-SPINE embeddings of the NHTSA dataset as shown in Fig 2. Clearly, the “Feature” characterizes the “Concept 1” in the discriminative triples whereas in the non-discriminative triples the “Feature” are generic features corresponding to both “Concept 1” and “Concept 2”. Discriminative Triples | Non-discriminative Triples
---|---
(Tire, Seat Belts, sliced) | (Tire, Seat Belts, crv)
(Tire, Seat Belts, punctured) | (Tire, Seat Belts, exl)
(Tire, Seat Belts, groove) | (Tire, Seat Belts, passport)
(Tire, Seat Belts, gashed) | (Tire, Seat Belts, accord)
(Tire, Seat Belts, rubbed) | (Tire, Seat Belts, odyssey)
(Seat Belts, Tire, retract) | (Tire, Seat Belts, difficulties)
(Seat Belts, Tire, buckle ) | (Tire, Seat Belts, shutter)
(Seat Belts, Tire, anchorage ) | (Tire, Seat Belts, whip)
(Seat Belts, Tire, retractor ) | (Tire, Seat Belts, sag)
(Seat Belts, Tire, css ) | (Tire, Seat Belts, cupping)

##  7 Results and Discussions

In this section, we summarize the results of our experiments and discuss their
benefits and impacts.

Downstream Classification Task Performance

Based on the experiment performed in Section 5 and the results shown in Table
2, it is evident that SEMIE’s performance is competitive as compared to the
baseline $E_{OPT}$ across all domain-specific small datasets for both in dense
and SNN embedding space.

Interpretability: Semantic Similarities

Based on precision accuracies in the word intrusion detection tests, as shown
in Table 3, it is clear that SEMIE’s performance is comparable to the baseline
$E_{OPT}$ across majority of the datasets. This implies that, for domain-
specific small datasets, SEMIE is an interpretable word embedding
representation when transformed in SNN embedding space.

From the Table 4, it is evident that SEMIE provides a novel mechanism to
provide labels for SNN embedding dimensions. These labels are able to cater to
the interpretations of the embedding dimensions and thus help in quick and
easy understanding.

Interpretability: Semantic Dissimilarities

Based on the demonstration shown in Section 6.3 and Figure 2, we observe that,
given two concepts, SEMIE provides a novel mechanism to generate the
discriminative and non-discriminative words triples from SNN embeddings. These
triples can help in the better assessment of the embeddings’ interpretability
in terms of semantic dissimilarities.

##  8 Conclusion

We define a comprehensive notion of interpretability of word embeddings which
includes both semantic similarities and dissimilarities while representing
them in SNN embedding space. We present a novel method to generate efficient
and highly interpretable word embeddings (SEMIE) for a domain-specific small
corpus. Also, we demonstrate that SEMIE provides novel mechanisms to provide
labels for SNN embedding dimensions and to generate discriminative/non-
discriminative word triples in an unsupervised fashion.

## References

  * Bansal et al. (2014) Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014.  Tailoring continuous word representations for dependency parsing.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 809–815.
  * Baroni and Lenci (2010) Marco Baroni and Alessandro Lenci. 2010.  Distributional memory: A general framework for corpus-based semantics.  _Computational Linguistics_ , 36(4):673–721.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Chang et al. (2009) Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L Boyd-Graber, and David M Blei. 2009.  Reading tea leaves: How humans interpret topic models.  In _Advances in neural information processing systems_ , pages 288–296.
  * Donoho et al. (2005) David L Donoho, Michael Elad, and Vladimir N Temlyakov. 2005.  Stable recovery of sparse overcomplete representations in the presence of noise.  _IEEE Transactions on information theory_ , 52(1):6–18.
  * Dumitru et al. (2018) Bogdan Dumitru, Alina Maria Ciobanu, and Liviu P Dinu. 2018.  Alb at semeval-2018 task 10: A system for capturing discriminative attributes.  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 963–967.
  * Faruqui et al. (2015) Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A Smith. 2015\.  Sparse overcomplete word vector representations.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1491–1500.
  * Fyshe et al. (2014) Alona Fyshe, Partha P Talukdar, Brian Murphy, and Tom M Mitchell. 2014.  Interpretable semantic vectors from a joint model of brain-and text-based meaning.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2014, page 489. NIH Public Access.
  * Grishin (2018) Maxim Grishin. 2018.  Igevorse at semeval-2018 task 10: Exploring an impact of word embeddings concatenation for capturing discriminative attributes.  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 995–998.
  * Guo et al. (2014) Jiang Guo, Wanxiang Che, Haifeng Wang, and Ting Liu. 2014.  Revisiting embedding features for simple semi-supervised learning.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 110–120.
  * Gupta and Rao (2020) Rishabh Gupta and Rajesh N Rao. 2020.  Towards semantic noise cleansing of categorical data based on semantic infusion.  _arXiv preprint arXiv:2002.02238_.
  * Krebs et al. (2018) Alicia Krebs, Alessandro Lenci, and Denis Paperno. 2018.  Semeval-2018 task 10: Capturing discriminative attributes.  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 732–740.
  * Krebs and Paperno (2016) Alicia Krebs and Denis Paperno. 2016.  Capturing discriminative attributes in a distributional space: Task proposal.  In _Proceedings of the 1st workshop on evaluating vector-space representations for NLP_ , pages 51–54.
  * Lazaridou et al. (2013) Angeliki Lazaridou, Eva Maria Vecchi, and Marco Baroni. 2013.  Fish transporters and miracle homes: How compositional distributional semantics can help np parsing.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1908–1913.
  * Lewicki and Sejnowski (2000) Michael S Lewicki and Terrence J Sejnowski. 2000.  Learning overcomplete representations.  _Neural computation_ , 12(2):337–365.
  * Lipton (2018) Zachary C Lipton. 2018.  The mythos of model interpretability.  _Queue_ , 16(3):31–57.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  In _Advances in neural information processing systems_ , pages 3111–3119.
  * Mitchell et al. (2008) Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. 2008.  Predicting human brain activity associated with the meanings of nouns.  _science_ , 320(5880):1191–1195.
  * Murphy et al. (2012) Brian Murphy, Partha Talukdar, and Tom Mitchell. 2012.  Learning effective and interpretable semantic models using non-negative sparse embedding.  In _Proceedings of COLING 2012_ , pages 1933–1950.
  * Olshausen and Field (1997) Bruno A Olshausen and David J Field. 1997.  Sparse coding with an overcomplete basis set: A strategy employed by v1?  _Vision research_ , 37(23):3311–3325.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Subramanian et al. (2018) Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Eduard Hovy. 2018.  Spine: Sparse interpretable neural embeddings.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Turney and Pantel (2010) Peter D Turney and Patrick Pantel. 2010.  From frequency to meaning: Vector space models of semantics.  _Journal of artificial intelligence research_ , 37:141–188.
  * Yin and Shen (2018) Zi Yin and Yuanyuan Shen. 2018.  On the dimensionality of word embedding.  In _Advances in Neural Information Processing Systems_ , pages 887–898.
  * You et al. (2016) Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016.  Image captioning with semantic attention.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4651–4659.
  * Zhang and Carpuat (2018) Alexander Zhang and Marine Carpuat. 2018.  Umd at semeval-2018 task 10: Can word embeddings capture discriminative attributes?  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 1022–1026.