# A Comprehensive Survey of Deep Learning for Image Captioning

Md. Zakir Hossain  0000-0003-1212-4652 Murdoch UniversityAustralia
<EMAIL_ADDRESS>,  Ferdous Sohel  Murdoch UniversityAustralia
<EMAIL_ADDRESS>,  Mohd Fairuz Shiratuddin  Murdoch UniversityAustralia
<EMAIL_ADDRESS>and  Hamid Laga  Murdoch UniversityAustralia
<EMAIL_ADDRESS>
(2018; April 2018; October 2018; October 2018)

###### Abstract.

Generating a description of an image is called image captioning. Image
captioning requires to recognize the important objects, their attributes and
their relationships in an image. It also needs to generate syntactically and
semantically correct sentences. Deep learning-based techniques are capable of
handling the complexities and challenges of image captioning. In this survey
paper, we aim to present a comprehensive review of existing deep learning-
based image captioning techniques. We discuss the foundation of the techniques
to analyze their performances, strengths and limitations. We also discuss the
datasets and the evaluation metrics popularly used in deep learning based
automatic image captioning.

Image Captioning, Deep Learning, Computer Vision, Natural Language Processing,
CNN, LSTM.

††journal: CSUR††journalvolume: 0††journalnumber: 0††article: 0††copyright:
acmcopyright††doi: 0000001.0000001††ccs: Computing methodologies Machine
learning††ccs: Computing methodologies Supervised learning††ccs: Computing
methodologies Unsupervised learning††ccs: Computing methodologies
Reinforcement learning††ccs: Computing methodologies Neural networks

##  1\. Introduction

Every day, we encounter a large number of images from various sources such as
the internet, news articles, document diagrams and advertisements. These
sources contain images that viewers would have to interpret themselves. Most
images do not have a description, but the human can largely understand them
without their detailed captions. However, machine needs to interpret some form
of image captions if humans need automatic image captions from it.

Image captioning is important for many reasons. For example, they can be used
for automatic image indexing. Image indexing is important for Content-Based
Image Retrieval (CBIR) and therefore, it can be applied to many areas,
including biomedicine, commerce, the military, education, digital libraries,
and web searching. Social media platforms such as Facebook and Twitter can
directly generate descriptions from images. The descriptions can include where
we are (e.g., beach, cafe), what we wear and importantly what we are doing
there.

Image captioning is a popular research area of Artificial Intelligence (AI)
that deals with image understanding and a language description for that image.
Image understanding needs to detect and recognize objects. It also needs to
understand scene type or location, object properties and their interactions.
Generating well-formed sentences requires both syntactic and semantic
understanding of the language (Vinyals et al., 2017).

Understanding an image largely depends on obtaining image features. The
techniques used for this purpose can be broadly divided into two categories:
(1) Traditional machine learning based techniques and (2) Deep machine
learning based techniques.

In traditional machine learning, hand crafted features such as Local Binary
Patterns (LBP) (Ojala et al., 2000), Scale-Invariant Feature Transform (SIFT)
(Lowe, 2004), the Histogram of Oriented Gradients (HOG) (Dalal and Triggs,
2005), and a combination of such features are widely used. In these
techniques, features are extracted from input data. They are then passed to a
classifier such as Support Vector Machines (SVM) (Boser et al., 1992) in order
to classify an object. Since hand crafted features are task specific,
extracting features from a large and diverse set of data is not feasible.
Moreover, real world data such as images and video are complex and have
different semantic interpretations.

On the other hand, in deep machine learning based techniques, features are
learned automatically from training data and they can handle a large and
diverse set of images and videos. For example, Convolutional Neural Networks
(CNN) (LeCun et al., 1998) are widely used for feature learning, and a
classifier such as Softmax is used for classification. CNN is generally
followed by Recurrent Neural Networks (RNN) in order to generate captions.

In the last 5 years, a large number of articles have been published on image
captioning with deep machine learning being popularly used. Deep learning
algorithms can handle complexities and challenges of image captioning quite
well. So far, only three survey papers (Bernardi et al., 2016; Kumar and Goel,
2017; Bai and An, 2018) have been published on this research topic. Although
the papers have presented a good literature survey of image captioning, they
could only cover a few papers on deep learning because the bulk of them was
published after the survey papers. These survey papers mainly discussed
template based, retrieval based, and a very few deep learning-based novel
image caption generating models. However, a large number of works have been
done on deep learning-based image captioning. Moreover, the availability of
large and new datasets has made the learning-based image captioning an
interesting research area. To provide an abridged version of the literature,
we present a survey mainly focusing on the deep learning-based papers on image
captioning.

The main aim of this paper is to provide a comprehensive survey of deep
learning for image captioning. First, we group the existing image captioning
articles into three main categories: (1) Template-based Image captioning, (2)
Retrieval-based image captioning, and (3) Novel image caption generation. The
categories are discussed briefly in Section 2. Most deep learning based image
captioning methods fall into the category of novel caption generation.
Therefore, we focus only on novel caption generation with deep learning.
Second, we group the deep learning-based image captioning methods into
different categories namely (1) Visual space-based, (2) Multimodal space-
based, (3) Supervised learning, (4) Other deep learning, (5) Dense captioning,
(6) Whole scene-based, (7) Encoder-Decoder Architecture-based, (8)
Compositional Architecture-based, (9) LSTM (Long Short-Term Memory)
(Hochreiter and Schmidhuber, 1997) language model-based, (10) Others language
model-based, (11) Attention-Based, (12) Semantic concept-based, (13) Stylized
captions, and (12) Novel object-based image captioning. We discuss all the
categories in Section 3. We provide an overview of the datasets and commonly
used evaluation metrics for measuring the quality of image captions in Section
4. We also discuss and compare the results of different methods in Section 5.
Finally, we give a brief discussion and future research directions in Section
6 and then a conclusion in Section 7.

Figure 1. An overall taxonomy of deep learning-based image captioning.

##  2\.  Image Captioning Methods

In this section, we review and describe the main categories of existing image
captioning methods and they include template-based image captioning,
retrieval-based image captioning, and novel caption generation. Template-based
approaches have fixed templates with a number of blank slots to generate
captions. In these approaches, different objects, attributes, actions are
detected first and then the blank spaces in the templates are filled. For
example, Farhadi et al. (Farhadi et al., 2010) use a triplet of scene elements
to fill the template slots for generating image captions. Li et al. (Li et
al., 2011) extract the phrases related to detected objects, attributes and
their relationships for this purpose. A Conditional Random Field (CRF) is
adopted by Kulkarni et al. (Kulkarni et al., 2011) to infer the objects,
attributes, and prepositions before filling in the gaps. Template-based
methods can generate grammatically correct captions. However, templates are
predefined and cannot generate variable-length captions. Moreover, later on,
parsing based language models have been introduced in image captioning (Aker
and Gaizauskas, 2010; Elliott and Keller, 2013; Kuznetsova et al., 2012, 2014;
Mitchell et al., 2012) which are more powerful than fixed template-based
methods. Therefore, in this paper, we do not focus on these template based
methods.

Captions can be retrieved from visual space and multimodal space. In
retrieval-based approaches, captions are retrieved from a set of existing
captions. Retrieval based methods first find the visually similar images with
their captions from the training data set. These captions are called candidate
captions. The captions for the query image are selected from these captions
pool (Ordonez et al., 2011; Hodosh et al., 2013; Sun et al., 2015; Gong et
al., 2014). These methods produce general and syntactically correct captions.
However, they cannot generate image specific and semantically correct
captions.

Novel captions can be generated from both visual space and multimodal space. A
general approach of this category is to analyze the visual content of the
image first and then generate image captions from the visual content using a
language model (Kiros et al., 2014b; Xu et al., 2015; Yao et al., 2017b; You
et al., 2016). These methods can generate new captions for each image that are
semantically more accurate than previous approaches. Most novel caption
generation methods use deep machine learning based techniques. Therefore, deep
learning based novel image caption generating methods are our main focus in
this literature.

An overall taxonomy of deep learning-based image captioning methods is
depicted in Figure 1. The figure illustrates the comparisons of different
categories of image captioning methods. Novel caption generation-based image
caption methods mostly use visual space and deep machine learning based
techniques. Captions can also be generated from multimodal space. Deep
learning-based image captioning methods can also be categorized on learning
techniques: Supervised learning, Reinforcement learning, and Unsupervised
learning. We group the reinforcement learning and unsupervised learning into
Other Deep Learning. Usually captions are generated for a whole scene in the
image. However, captions can also be generated for different regions of an
image (Dense captioning). Image captioning methods can use either simple
Encoder-Decoder architecture or Compositional architecture. There are methods
that use attention mechanism, semantic concept, and different styles in image
descriptions. Some methods can also generate description for unseen objects.
We group them into one category as “Others”. Most of the image captioning
methods use LSTM as language model. However, there are a number of methods
that use other language models such as CNN and RNN. Therefore, we include a
language model-based category as “LSTM vs. Others”.

##  3\. Deep Learning Based Image Captioning Methods

We draw an overall taxonomy in Figure 1 for deep learning-based image
captioning methods. We discuss their similarities and dissimilarities by
grouping them into visual space vs. multimodal space, dense captioning vs.
captions for the whole scene, Supervised learning vs. Other deep learning,
Encoder-Decoder architecture vs. Compositional architecture, and one ‘Others’
group that contains Attention-Based, Semantic Concept-Based, Stylized
captions, and Novel Object-Based captioning. We also create a category named
LSTM vs. Others.

A brief overview of the deep learning-based image captioning methods is shown
in Table 1. Table 1 contains the name of the image captioning methods, the
type of deep neural networks used to encode image information, and the
language models used in describing the information. In the final column, we
give a category label to each captioning technique based on the taxonomy in
Figure 1.

###  3.1. Visual Space vs. Multimodal Space

Deep learning-based image captioning methods can generate captions from both
visual space and multimodal space. Understandably image captioning datasets
have the corresponding captions as text. In the visual space-based methods,
the image features and the corresponding captions are independently passed to
the language decoder. In contrast, in a multimodal space case, a shared
multimodal space is learned from the images and the corresponding caption-
text. This multimodal representation is then passed to the language decoder.

####  3.1.1. Visual Space

Bulk of the image captioning methods use visual space for generating captions.
These methods are discussed in Section 3.2 to Section 3.5.

Figure 2. A block diagram of multimodal space-based image captioning.

####  3.1.2. Multimodal Space

The architecture of a typical multimodal space-based method contains a
language Encoder part, a vision part, a multimodal space part, and a language
decoder part. A general diagram of multimodal space-based image captioning
methods is shown in Figure 2. The vision part uses a deep convolutional neural
network as a feature extractor to extract the image features. The language
encoder part extracts the word features and learns a dense feature embedding
for each word. It then forwards the semantic temporal context to the recurrent
layers. The multimodal space part maps the image features into a common space
with the word features. The resulting map is then passed to the language
decoder which generates captions by decoding the map.

The methods in this category follow the following steps:

  1. (1)

Deep neural networks and multimodal neural language model are used to learn
both image and text jointly in a multimodal space.

  2. (2)

The language generation part generates captions using the information from
Step 1 .

2*Reference |  2*Image Encoder |  2*  | Language Model
---
2*Category
Kiros et al. 2014 (Kiros et al., 2014a) | AlexNet | LBL | MS,SL,WS,EDA
Kiros et al. 2014 (Kiros et al., 2014b) | AlexNet, VGGNet |  | 1\. LSTM
---
2\. SC-NLM
MS,SL,WS,EDA
Mao et al. 2014 (Mao et al., 2014) | AlexNet | RNN | MS,SL,WS
Karpathy et al. 2014 (Karpathy et al., 2014) | AlexNet | DTR | MS,SL,WS,EDA
Mao et al. 2015 (Mao et al., 2015b) | AlexNet, VGGNet | RNN | MS,SL,WS
Chen et al. 2015 (Chen and Lawrence Zitnick, 2015) | VGGNet | RNN | VS,SL,WS,EDA
Fang et al. 2015 (Fang et al., 2015) | AlexNet, VGGNet | MELM | VS,SL,WS,CA
Jia et al. 2015 (Jia et al., 2015) | VGGNet | LSTM | VS,SL,WS,EDA
Karpathy et al. 2015 (Karpathy and Fei-Fei, 2015) | VGGNet | RNN | MS,SL,WS,EDA
Vinyals et al. 2015 (Vinyals et al., 2015) | GoogLeNet | LSTM | VS,SL,WS,EDA
Xu et al. 2015 (Xu et al., 2015) | AlexNet | LSTM | VS,SL,WS,EDA,AB
Jin et al. 2015 (Jin et al., 2015) | VGGNet | LSTM | VS,SL,WS,EDA,AB
Wu et al. 2016 (Wu and Cohen, 2016) | VGGNet | LSTM | VS,SL,WS,EDA,AB
Sugano et at. 2016 (Sugano and Bulling, 2016) | VGGNet | LSTM | VS,SL,WS,EDA,AB
Mathews et al. 2016 (Mathews et al., 2016) | GoogLeNet | LSTM | VS,SL,WS,EDA,SC
Wang et al. 2016 (Wang et al., 2016b) | AlexNet, VGGNet | LSTM | VS,SL,WS,EDA
Johnson et al. 2016 (Johnson et al., 2016) | VGGNet | LSTM | VS,SL,DC,EDA
Mao et al. 2016 (Mao et al., 2016) | VGGNet | LSTM | VS,SL,WS,EDA
Wang et al. 2016 (Wang et al., 2016a) | VGGNet | LSTM | VS,SL,WS,CA
Tran et al. 2016 (Tran et al., 2016) | ResNet | MELM | VS,SL,WS,CA
Ma et al. 2016 (Ma and Han, 2016) | AlexNet | LSTM | VS,SL,WS,CA
You et al. 2016 (You et al., 2016) | GoogLeNet | RNN | VS,SL,WS,EDA,SCB
Yang et al. 2016 (Yang et al., 2016) | VGGNet | LSTM | VS,SL,DC,EDA
Anne et al. 2016 (Anne Hendricks et al., 2016) | VGGNet | LSTM | VS,SL,WS,CA,NOB
Yao et al. 2017 (Yao et al., 2017b) | GoogLeNet | LSTM | VS,SL,WS,EDA,SCB
Lu et al. 2017 (Lu et al., 2017) | ResNet | LSTM | VS,SL,WS,EDA,AB
Chen et al. 2017 (Chen et al., 2017b) | VGGNet, ResNet | LSTM | VS,SL,WS,EDA,AB
Gan et al. 2017 (Gan et al., 2017b) | ResNet | LSTM | VS,SL,WS,CA,SCB
Pedersoli et al. 2017 (Pedersoli et al., 2017) | VGGNet | RNN | VS,SL,WS,EDA,AB
Ren et al. 2017 (Ren et al., 2017) | VGGNet | LSTM | VS,ODL,WS,EDA
Park et al. 2017 (Park et al., 2017) | ResNet | LSTM | VS,SL,WS,EDA,AB
Wang et al. 2017 (Wang et al., 2017) | ResNet | LSTM | VS,SL,WS,EDA
Tavakoli et al. 2017 (Tavakoli et al., 2017) | VGGNet | LSTM | VS,SL,WS,EDA,AB
Liu et al. 2017 (Liu et al., 2017a) | VGGNet | LSTM | VS,SL,WS,EDA,AB
Gan et al. 2017 (Gan et al., 2017a) | ResNet | LSTM | VS,SL,WS,EDA,SC
Dai et al. 2017 (Dai et al., 2017) | VGGNet | LSTM | VS,ODL,WS,EDA
Shetty et al. 2017 (Shetty et al., 2017) | GoogLeNet | LSTM | VS,ODL,WS,EDA
Liu et al. 2017 (Liu et al., 2017b) | Inception-V3 | LSTM | VS,ODL,WS,EDA
Gu et al. 2017 (Gu et al., 2017) | VGGNet |  | 1\. Language CNN
---
2\. LSTM
VS,SL,WS,EDA
Yao et al. 2017 (Yao et al., 2017a) | VGGNet | LSTM | VS,SL,WS,CA,NOB
Rennie et al. 2017 (Rennie et al., 2017) | ResNet | LSTM | VS,ODL,WS,EDA
Vsub et al. 2017 (Venugopalan et al., 2017) | VGGNet | LSTM | VS,SL,WS,CA,NOB
Zhang et al. 2017 (Zhang et al., 2017) | Inception-V3 | LSTM | VS,ODL,WS,EDA
Wu et al. 2018 (Wu et al., 2018) | VGGNet | LSTM | VS,SL,WS,EDA,SCB
Aneja et al. 2018 (Aneja et al., 2018) | VGGNet | Language CNN | VS,SL,WS,EDA
Wang et al. 2018 (Wang and Chan, 2018) | VGGNet | Language CNN | VS,SL,WS,EDA

Table 1. An overview of the deep learning-based approaches for image
captioning (VS=Visual Space, MS=Multimodal Space, SL=Supervised Learning,
ODL=Other Deep Learning, DC=Dense Captioning, WS=Whole Scene, EDA=Encoder-
Decoder Architecture, CA=Compositional Architecture, AB=Attention-Based,
SCB=Semantic Concept-Based, NOB=Novel Object-Based, SC=Stylized Caption).

An initial work in this area proposed by Kiros et al. (Kiros et al., 2014a).
The method applies a CNN for extracting image features in generating image
captions. It uses a multimodal space that represents both image and text
jointly for multimodal representation learning and image caption generation.
It also introduces the multimodal neural language models such as Modality-
Biased Log-Bilinear Model (MLBL-B) and the Factored 3-way Log-Bilinear Model
(MLBL-F) of (Mnih and Hinton, 2007a) followed by AlexNet (Krizhevsky et al.,
2012). Unlike most previous approaches, this method does not rely on any
additional templates, structures, or constraints. Instead it depends on the
high level image features and word representations learned from deep neural
networks and multimodal neural language models respectively. The neural
language models have limitations to handle a large amount of data and are
inefficient to work with long term memory (Jozefowicz et al., 2016).

Kiros et al. (Kiros et al., 2014a) extended their work in (Kiros et al.,
2014b) to learn a joint image sentence embedding where LSTM is used for
sentence encoding and a new neural language model called the structure-content
neural language model (SC-NLM) is used for image captions generations. The SC-
NLM has an advantage over existing methods in that it can extricate the
structure of the sentence to its content produced by the encoder. It also
helps them to achieve significant improvements in generating realistic image
captions over the approach proposed by (Kiros et al., 2014a)

Karpathy et al. (Karpathy et al., 2014) proposed a deep, multimodal model,
embedding of image and natural language data for the task of bidirectional
images and sentences retrieval. The previous multimodal-based methods use a
common, embedding space that directly maps images and sentences. However, this
method works at a finer level and embeds fragments of images and fragments of
sentences. This method breaks down the images into a number of objects and
sentences into a dependency tree relations (DTR) (De Marneffe et al., 2006)
and reasons about their latent, inter-modal alignment. It shows that the
method achieves significant improvements in the retrieval task compared to
other previous methods. This method has a few limitations as well. In terms of
modelling, the dependency tree can model relations easily but they are not
always appropriate. For example, a single visual entity might be described by
a single complex phrase that can be split into multiple sentence fragments.
The phrase “black and white dog” can be formed into two relations (CONJ,
black, white) and (AMOD, white, dog). Again, for many dependency relations we
do not find any clear mapping in the image (For example: “each other” cannot
be mapped to any object).

Mao et al. (Mao et al., 2015b) proposed a multimodal Recurrent Neural Network
(m-RNN) method for generating novel image captions. This method has two sub-
networks: a deep recurrent neural network for sentences and a deep
convolutional network for images. These two sub-networks interact with each
other in a multimodal layer to form the whole m-RNN model. Both image and
fragments of sentences are given as input in this method. It calculates the
probabilty distribution to generate the next word of captions. There are five
more layers in this model: Two-word embedding layers, a recurrent layer, a
multimodal layer and a SoftMax layer. Kiros et al. (Kiros et al., 2014a)
proposed a method that is built on a Log-Bilinear model and used AlexNet to
extract visual features. This multimodal recurrent neural network method is
closely related to the method of Kiros et al. (Kiros et al., 2014a). Kiros et
al. use a fixed length context (i.e. five words), whereas in this method, the
temporal context is stored in a recurrent architecture, which allows an
arbitrary context length. The two word embedding layers use one hot vector to
generate a dense word representation. It encodes both the syntactic and
semantic meaning of the words. The semantically relevant words can be found by
calculating the Euclidean distance between two dense word vectors in embedding
layers. Most sentence-image multimodal methods (Karpathy et al., 2014; Frome
et al., 2013; Socher et al., 2014; Kiros et al., 2014b) use pre-computed word
embedding vectors to initialize their model. In contrast, this method randomly
initializes word embedding layers and learn them from the training data. This
helps them to generate better image captions than the previous methods. Many
image captioning methods (Mao et al., 2014; Kiros et al., 2014a; Karpathy et
al., 2014) are built on recurrent neural networks at the contemporary times.
They use a recurrent layer for storing visual information. However, (m-RNN)
use both image representations and sentence fragments to generate captions. It
utilizes the capacity of the recurrent layer more efficiently that helps to
achieve a better performance using a relatively small dimensional recurrent
layer.

Chen et al. (Chen and Lawrence Zitnick, 2015) proposed another multimodal
space-based image captioning method. The method can generate novel captions
from image and restore visual features from the given description. It also can
describe a bidirectional mapping between images and their captions. Many of
the existing methods (Hodosh et al., 2013; Socher et al., 2014; Karpathy et
al., 2014) use joint embedding to generate image captions. However, they do
not use reverse projection that can generate visual features from captions. On
the other hand, this method dynamically updates the visual representations of
the image from the generated words. It has an additional recurrent visual
hidden layer with RNN that makes reverse projection.

###  3.2. Supervised Learning vs. Other Deep Learning

In supervised learning, training data come with desired output called label.
Unsupervised learning, on the other hand, deals with unlabeled data.
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are a type of
unsupervised learning techniques. Reinforcement learning is another type of
machine learning approach where the aims of an agent are to discover data
and/or labels through exploration and a reward signal. A number of image
captioning methods use reinforcement learning and GAN based approaches. These
methods sit in the category of “Other Deep Learning”.

####  3.2.1. Supervised Learning-Based Image Captioning

Supervised learning-based networks have successfully been used for many years
in image classification (Krizhevsky et al., 2012; He et al., 2016; Simonyan
and Zisserman, 2015; Szegedy et al., 2015), object detection (Girshick, 2015;
Girshick et al., 2014; Ren et al., 2015a), and attribute learning (Gan et al.,
2016). This progress makes researchers interested in using them in automatic
image captioning (Vinyals et al., 2015; Mao et al., 2015b; Karpathy and Fei-
Fei, 2015; Chen and Lawrence Zitnick, 2015). In this paper, we have identified
a large number of supervised learning-based image captioning methods. We
classify them into different categories: (i) Encoder-Decoder Architecture,
(ii) Compositional Architecture, (iii) Attention-based, (iv) Semantic concept-
based, (v) Stylized captions, (vi) Novel object-based, and (vii) Dense image
captioning.

Figure 3. A block diagram of other deep learning-based captioning.

####  3.2.2. Other Deep Learning-Based Image Captioning

In our day to day life, data are increasing with unlabled data because it is
often impractical to accurately annotate data. Therefore, recently,
researchers are focusing more on reinforcement learning and unsupervised
learning-based techniques for image captioning.

A reinforcement learning agent chooses an action, receives reward values, and
moves to a new state. The agent attempts to select the action with the
expectation of having a maximum long-term reward. It needs continuous state
and action information, to provide the guarantees of a value function.
Traditional reinforcement learning approaches face a number of limitations
such as the lack of guarantees of a value function and uncertain state-action
information. Policy gradient methods (Sutton et al., 2000) are a type of
reinforcement learning that can choose a specific policy for a specific action
using gradient descent and optimization techniques. The policy can incorporate
domain knowledge for the action that guarantees convergence. Thus, policy
gradient methods require fewer parameters than value-function based
approaches.
Existing deep learning-based image captioning methods use variants of image
encoders to extract image features. The features are then fed into the neural
network-based language decoders to generate captions. The methods have two
main issues: (i) They are trained using maximum likelihood estimation and
back-propagation (Ranzato et al., 2016) approaches. In this case, the next
word is predicted given the image and all the previously generated ground-
truth words. Therefore, the generated captions look-like ground-truth
captions. This phenomenon is called exposure bias (Bengio et al., 2015)
problem. (ii) Evaluation metrics at test time are non-differentiable. Ideally
sequence models for image captioning should be trained to avoid exposure bias
and directly optimise metrics for the test time. In actor-critic-based
reinforcement learning algorithm, critic can be used in estimating the
expected future reward to train the actor (captioning policy network).
Reinforcement learning-based image captioning methods sample the next token
from the model based on the rewards they receive in each state. Policy
gradient methods in reinforcement learning can optimize the gradient in order
to predict the cumulative long-term rewards. Therefore, it can solve the non-
differentiable problem of evaluation metrics.

The methods in this category follow the following steps:

  1. (1)

A CNN and RNN based combined network generates captions.

  2. (2)

Another CNN-RNN based network evaluates the captions and send feedback to the
first network to generate high quality captions.

A block diagram of a typical method of this category is shown in Figure 3.

Ren et al. 2017 (Ren et al., 2017) introduced a novel reinforcement learning
based image captioning method. The architecture of this method has two
networks that jointly compute the next best word at each time step. The
“policy network” works as local guidance and helps to predict next word based
on the current state. The “value network”’ works as global guidance and
evaluates the reward value considering all the possible extensions of the
current state. This mechanism is able to adjust the networks in predicting the
correct words. Therefore, it can generate good captions similar to ground
truth captions at the end. It uses an actor-critic reinforcement learning
model (Konda and Tsitsiklis, 2000) to train the whole network. Visual semantic
embedding (Ren et al., 2015b, 2016) is used to compute the actual reward value
in predicting the correct word. It also helps to measure the similarity
between images and sentences that can evaluate the correctness of generated
captions.

Rennie et al. (Rennie et al., 2017) proposed another reinforcement learning
based image captioning method. The method utilizes the test-time inference
algorithm to normalize the reward rather than estimating the reward signal and
normalization in training time. It shows that this test-time decoding is
highly effective for generating quality image captions.

Zhang et al. (Zhang et al., 2017) proposed an actor-critic reinforcement
learning-based image captioning method. The method can directly optimize non-
differentiable problems of the existing evaluation metrics. The architecture
of the actor-critic method consists of a policy network (actor) and a value
network (critic). The actor treats the job as sequential decision problem and
can predict the next token of the sequence. In each state of the sequence, the
network will receive a task-specific reward (in this case, it is evaluation
metrics score). The job of the critic is to predict the reward. If it can
predict the expected reward, the actor will continue to sample outputs
according to its probability distribution.

GAN based methods can learn deep features from unlabeled data. They achieve
this representations applying a competitive process between a pair of
networks: the Generator and the Discriminator. GANs have already been used
successfully in a variety of applications, including image captioning(Dai et
al., 2017; Shetty et al., 2017), image to image translation (Isola et al.,
2017), text to image synthesis (Reed et al., 2016; Bodnar, 2018), and text
generation (Fedus et al., 2018; Wang et al., 2018).

There are two issues with GAN. First, GAN can work well in generating natural
images from real images because GANs are proposed for real-valued data.
However, text processing is based on discrete numbers. Therefore, such
operations are non-differentiable, making it difficult to apply back-
propagation directly. Policy gradients apply a parametric function to allow
gradients to be back-propagated. Second, the evaluator faces problems in
vanishing gradients and error propagation for sequence generation. It needs a
probable future reward value for every partial description. Monte Carlo
rollouts (Yu et al., 2017) is used to compute this future reward value.

GAN based image captioning methods can generate a diverse set of image
captions in contrast to conventional deep convolutional network and deep
recurrent network based model. Dai et al. (Dai et al., 2017) also proposed a
GAN based image captioning method. However, they do not consider multiple
captions for a single image. Shetty et al. (Shetty et al., 2017) introduced a
new GAN based image captioning method. This method can generate multiple
captions for a single image and showed impressive improvements in generating
diverse captions. GANs have limitations in backpropagating the discrete data.
Gumbel sampler (Jang et al., 2017; Maddison et al., 2017) is used to overcome
the discrete data problem. The two main parts of this adversarial network are
the generator and the discriminator. During training, generator learns the
loss value provided by the discriminator instead of learning it from explicit
sources. Discriminator has true data distribution and can discriminate between
generator-generated samples and true data samples. This allows the network to
learn diverse data distribution. Moreover, the network classifies the
generated caption sets either real or fake. Thus, it can generate captions
similar to human generated one.

Figure 4. A block diagram of dense captioning.

###  3.3. Dense Captioning vs. Captions for the whole scene

In dense captioning, captions are generated for each region of the scene.
Other methods generate captions for the whole scene.

####  3.3.1. Dense Captioning

The previous image captioning methods can generate only one caption for the
whole image. They use different regions of the image to obtain information of
various objects. However, these methods do not generate region wise captions.

Johnson et al. (Johnson et al., 2016) proposed an image captioning method
called DenseCap. This method localizes all the salient regions of an image and
then it generates descriptions for those regions.

A typical method of this category has the following steps:

  1. (1)

Region proposals are generated for the different regions of the given image.

  2. (2)

CNN is used to obtain the region-based image features.

  3. (3)

The outputs of Step 2 are used by a language model to generate captions for
every region.

A block diagram of a typical dense captioning method is given in Figure 4.

Dense captioning (Johnson et al., 2016) proposes a fully convolutional
localization network architecture, which is composed of a convolutional
network, a dense localization layer, and an LSTM (Hochreiter and Schmidhuber,
1997) language model. The dense localization layer processes an image with a
single, efficient forward pass, which implicitly predicts a set of region of
interest in the image. Thereby, it requires no external region proposals
unlike to Fast R-CNN or a full network (i.e., RPN (Region Proposal Network
(Girshick, 2015))) of Faster R-CNN. The working principle of the localization
layer is related to the work of Faster R-CNN (Ren et al., 2015a). However,
Johnson et al. (Johnson et al., 2016) use a differential, spatial soft
attention mechanism (Gregor et al., 2015; Jaderberg et al., 2015) and bilinear
interpolation (Jaderberg et al., 2015) instead of ROI pooling mechanism
(Girshick, 2015). This modification helps the method to backpropagate through
the network and smoothly select the active regions. It uses Visual Genome
(Krishna et al., 2017) dataset for the experiments in generating region level
image captions.

One description of the entire visual scene is quite subjective and is not
enough to bring out the complete understanding. Region-based descriptions are
more objective and detailed than global image description. The region-based
description is known as dense captioning. There are some challenges in dense
captioning. As regions are dense, one object may have multiple overlapping
regions of interest. Moreover, it is very difficult to recognize each target
region for all the visual concepts. Yang et al. (Yang et al., 2016) proposed
another dense captioning method. This method can tackle these challenges.
First, it addresses an inference mechanism that jointly depends on the visual
features of the region and the predicted captions for that region. This allows
the model to find an appropriate position of the bounding box. Second, they
apply a context fusion that can combine context features with the visual
features of respective regions to provide a rich semantic description.

####  3.3.2. Captions for the whole scene

Encoder-Decoder architecture, Compositional architecture, attention-based,
semantic concept-based, stylized captions, Novel object-based image
captioning, and other deep learning networks-based image captioning methods
generate single or multiple captions for the whole scene.

###  3.4. Encoder-Decoder Architecture vs. Compositional Architecture

Some methods use just simple vanilla encoder and decoder to generate captions.
However, other methods use multiple networks for it.

####  3.4.1. Encoder-Decoder Architecture-Based Image captioning

The neural network-based image captioning methods work as just simple end to
end manner. These methods are very similar to the encoder-decoder framework-
based neural machine translation (Sutskever et al., 2014). In this network,
global image features are extracted from the hidden activations of CNN and
then fed them into an LSTM to generate a sequence of words.

Figure 5. A block diagram of simple Encoder-Decoder architecture-based image
captioning.

A typical method of this category has the following general steps:

  1. (1)

A vanilla CNN is used to obtain the scene type, to detect the objects and
their relationships.

  2. (2)

The output of Step 1 is used by a language model to convert them into words,
combined phrases that produce an image captions.

A simple block diagram of this category is given in Figure 5.

Vinyals et al. (Vinyals et al., 2015) proposed a method called Neural Image
Caption Generator (NIC). The method uses a CNN for image representations and
an LSTM for generating image captions. This special CNN uses a novel method
for batch normalization and the output of the last hidden layer of CNN is used
as an input to the LSTM decoder. This LSTM is capable of keeping track of the
objects that already have been described using text. NIC is trained based on
maximum likelihood estimation.

In generating image captions, image information is included to the initial
state of an LSTM. The next words are generated based on the current time step
and the previous hidden state. This process continues until it gets the end
token of the sentence. Since image information is fed only at the beginning of
the process, it may face vanishing gradient problems. The role of the words
generated at the beginning is also becoming weaker and weaker. Therefore, LSTM
is still facing challenges in generating long length sentences (Bahdanau et
al., 2015; Cho et al., 2014). Therefore, Jia et al. (Jia et al., 2015)
proposed an extension of LSTM called guided LSTM (gLSTM). This gLSTM can
generate long sentences. In this architecture, it adds global semantic
information to each gate and cell state of LSTM. It also considers different
length normalization strategies to control the length of captions. Semantic
information is extracted in different ways. First, it uses a cross-modal
retrieval task for retrieving image captions and then semantic information is
extracted from these captions. The semantic based information can also be
extracted using a multimodal embedding space.

Mao et al. (Mao et al., 2016) proposed a special type of text generation
method for images. This method can generate a description for an specific
object or region that is called referring expression (van Deemter et al.,
2006; Viethen and Dale, 2008; Mitchell et al., 2010, 2013; FitzGerald et al.,
2013; Golland et al., 2010; Kazemzadeh et al., 2014). Using this expression it
can then infer the object or region which is being described. Therefore,
generated description or expression is quite unambiguous. In order to address
the referring expression, this method uses a new dataset called ReferIt
dataset (Kazemzadeh et al., 2014) based on popular MS COCO dataset.

Previous CNN-RNN based image captioning methods use LSTM that are
unidirectional and relatively shallow in depth. In unidirectional language
generation techniques, the next word is predicted based on visual context and
all the previous textual contexts. Unidirectional LSTM cannot generate
contextually well formed captions. Moreover, recent object detection and
classification methods (Krizhevsky et al., 2012; Simonyan and Zisserman, 2015)
show that deep, hierarchical methods are better at learning than shallower
ones. Wang et al. (Wang et al., 2016b) proposed a deep bidirectional LSTM-
based method for image captioning. This method is capable of generating
contextually and semantically rich image captions. The proposed architecture
consists of a CNN and two separate LSTM networks. It can utilize both past and
future context information to learn long term visual language interactions.

####  3.4.2. Compositional Architecture-Based Image captioning

Compositional architecture-based methods composed of several independent
functional building blocks: First, a CNN is used to extract the semantic
concepts from the image. Then a language model is used to generate a set of
candidate captions. In generating the final caption, these candidate captions
are re-ranked using a deep multimodal similarity model.

Figure 6. A block diagram of a compositional network-based captioning.

A typical method of this category maintains the following steps:

  1. (1)

Image features are obtained using a CNN.

  2. (2)

Visual concepts (e.g. attributes) are obtained from visual features.

  3. (3)

Multiple captions are generated by a language model using the information of
Step 1 and Step 2.

  4. (4)

The generated captions are re-ranked using a deep multimodal similarity model
to select high quality image captions.

A common block diagram of compositional network-based image captioning methods
is given in Figure 6.

Fang et al.(Fang et al., 2015) introduced generation-based image captioning.
It uses visual detectors, a language model, and a multimodal similarity model
to train the model on an image captioning dataset. Image captions can contain
nouns, verbs, and adjectives. A vocabulary is formed using 1000 most common
words from the training captions. The system works with the image sub-regions
rather that the full image. Convolutional neural networks (both AlexNet
(Krizhevsky et al., 2012) and VGG16Net) are used for extracting features for
the sub-regions of an image. The features of sub-regions are mapped with the
words of the vocabulary that likely to be contained in the image captions.
Multiple instance learning (MIL) (Maron and Lozano-Pérez, 1998) is used to
train the model for learning discriminative visual signatures of each word. A
maximum entropy (ME) (Berger et al., 1996) language model is used for
generating image captions from these words. Generated captions are ranked by a
linear weighting of sentence features. Minimum Error rate training (MERT)
(Och, 2003) is used to learn these weights. Similarity between image and
sentence can be easily measured using a common vector representation. Image
and sentence fragments are mapped with the common vector representation by a
deep multimodal similarity model (DMSM). It achieves a significant improvement
in choosing high quality image captions.

Until now a significant number of methods have achieved satisfactory progress
in generating image captions. The methods use training and testing samples
from the same domain. Therefore, there is no certainty that these methods can
perform well in open-domain images. Moreover, they are only good at
recognizing generic visual content. There are certain key entities such as
celebrities and landmarks that are out of their scope. The generated captions
of these methods are evaluated on automatic metrics such as BLEU (Papineni et
al., 2002), METEOR (Agarwal and Lavie, 2008), and CIDEr (Vedantam et al.,
2015). These evaluation metrics have already shown good results on these
methods. However, in terms of performance there exists a large gap between the
evaluation of the metrics and human judgement of evaluation (Devlin et al.,
2015; Callison-Burch et al., 2006; Kulkarni et al., 2011). If it is considered
real life entity information, the performance could be weaker. However, Tran
et al. (Tran et al., 2016) introduced a different image captioning method.
This method is capable of generating image captions even for open domain
images. It can detect a diverse set of visual concepts and generate captions
for celebrities and landmarks. It uses an external knowledge base Freebase
(Bollacker et al., 2008) in recognizing a broad range of entities such as
celebrities and landmarks. A series of human judgments are applied for
evaluating the performances of generated captions. In experiments, it uses
three datasets: MS COCO, Adobe-MIT FiveK (Bychkovsky et al., 2011), and images
from Instagram. The images of MS COCO dataset were collected from the same
domain but the images of other datasets were chosen from an open domain. The
method achieves notable performances especially on the challenging Instagram
dataset.

Ma et al. (Ma and Han, 2016) proposed another compositional network-based
image captioning method. This method uses structural words $<$object,
attribute, activity, scene$>$ to generate semantically meaningful
descriptions. It also uses a multi-task method similar to multiple instance
learning method (Fang et al., 2015), and multi-layer optimization method (Han
and Li, 2015) to generate structural words. An LSTM encoder-decoder-based
machine translation method (Sutskever et al., 2014) is then used to translate
the structural words into image captions.

Wang et al. (Wang et al., 2016a) proposed a parallel-fusion RNN-LSTM
architecture for image caption generation. The architecture of the method
divides the hidden units of RNN and LSTM into a number of same-size parts. The
parts work in parallel with corresponding ratios to generate image captions.

###  3.5. Others

Attention-based, Semantic concept-based, Novel object-based methods, and
Stylized captions are put together into “Others” group because these
categories are independent to other methods.

####  3.5.1. Attention based Image Captioning

Neural encoder-decoder based approaches were mainly used in machine
translation (Sutskever et al., 2014). Following these trends, they have also
been used for the task of image captioning and found very effective. In image
captioning, a CNN is used as an encoder to extract the visual features from
the input image and an RNN is used as a decoder to convert this representation
word-by-word into natural language description of the image. However, these
methods are unable to analyze the image over time while they generate the
descriptions for the image. In addition to this, the methods do not consider
the spatial aspects of the image that is relevant to the parts of the image
captions. Instead, they generate captions considering the scene as a whole.
Attention based mechanisms are becoming increasingly popular in deep learning
because they can address these limitations. They can dynamically focus on the
various parts of the input image while the output sequences are being
produced.

Figure 7. A block diagram of a typical attention-based image captioning
technique.

A typical method of this category adopts the following steps:

  1. (1)

Image information is obtained based on the whole scene by a CNN.

  2. (2)

The language generation phase generates words or phrases based on the output
of Step 1.

  3. (3)

Salient regions of the given image are focused in each time step of the
language generation model based on generated words or phrases.

  4. (4)

Captions are updated dynamically until the end state of language generation
model.

A block diagram of the attention-based image captioning method is shown in
Figure 7.

Xu et al. (Xu et al., 2015) were the first to introduce an attention-based
image captioning method. The method describes the salient contents of an image
automatically. The main difference between the attention-based methods with
other methods is that they can concentrate on the salient parts of the image
and generate the corresponding words at the same time. This method applies two
different techniques: stochastic hard attention and deterministic soft
attention to generate attentions. Most CNN-based approaches use the top layer
of ConvNet for extracting information of the salient objects from the image. A
drawback of these techniques is that they may lose certain information which
is useful to generate detailed captions. In order to preserve the information,
the attention method uses features from the lower convolutional layer instead
of fully connected layer.

Jin et al. (Jin et al., 2015) proposed another attention-based image
captioning method. This method is capable to extract the flow of abstract
meaning based on the semantic relationship between visual information and
textual information. It can also obtain higher level semantic information by
proposing a scene specific context. The main difference between this method
with other attention-based methods is that it introduces multiple visual
regions of an image at multiple scales. This technique can extract proper
visual information of a particular object. For extracting scene specific
context, it first uses the Latent Dirichlet Allocation (LDA) (Blei et al.,
2003) for generating a dictionary from all the captions of the dataset. Then a
multilayer perceptron is used to predict a topic vector for every image. A
scene factored LSTM that has two stacked layers are used to generate a
description for the overall context of the image.

Wu et al. (Wu and Cohen, 2016) proposed a review-based attention method for
image captioning. It introduces a review model that can perform multiple
review steps with attention on CNN hidden states. The output of the CNN is a
number of fact vectors that can obtain the global facts of the image. The
vectors are given as input to the attention mechanism of the LSTM. For
example, a reviewer module can first review: What are the objects in the
image? Then it can review the relative positions of the objects and another
review can extract the information of the overall context of the image. These
information is passed to the decoder to generate image captions.

Pedersoli et al. (Pedersoli et al., 2017) proposed an area based attention
mechanism for image captioning. Previous attention based methods map image
regions only to the state of RNN language model. However, this approach
associates image regions with caption words given the RNN state. It can
predict the next caption word and corresponding image region in each time-step
of RNN. It is capable of predicting the next word as well as corresponding
image regions in each time-step of RNN for generating image captions. In order
to find the areas of attention, previous attention-based image caption methods
use either the position of CNN activation grid or object proposals. In
contrast, this method uses an end to end trainable convolutional spatial
transformer along with CNN activation gird and object proposal methods. A
combination of these techniques help this method to compute image adaptive
areas of attention. In experiments, the method shows that this new attention
mechanism together with the spatial transformer network can produce high
quality image captions.

Lu et al. (Lu et al., 2017) proposed another attention-based image captioning
method. The method is based on adaptive attention model with a visual
sentinel. Current attention-based image captioning methods focus on the image
in every time step of RNN. However, there are some words or phrase (for
example: a, of) that do not need to attend visual signals. Moreover, these
unnecessary visual signals could affect the caption generation process and
degrade the overall performance. Therefore, their proposed method can
determine when it will focus on image region and when it will just focus on
language generation model. Once it determines to look on the image then it
must have to choose the spatial location of the image. The first contribution
of this method is to introduce a novel spatial attention method that can
compute spatial features from the image. Then in their adaptive attention
method, they introduced a new LSTM extension. Generally, an LSTM works as a
decoder that can produce a hidden state at every time step. However, this
extension is capable of producing an additional visual sentinel that provides
a fallback option to the decoder. It also has a sentinel gate that can control
how much information the decoder will get from the image.

While attention-based methods look to find the different areas of the image at
the time of generating words or phrases for image captions, the attention maps
generated by these methods cannot always correspond to the proper region of
the image. It can affect the performance of image caption generation. Liu et
al. (Liu et al., 2017a) proposed a method for neural image captioning. This
method can evaluate and correct the attention map at time step. Correctness
means to make consistent map between image regions and generated words. In
order to achieve these goals, this method introduced a quantitative evaluation
metric to compute the attention maps. It uses Flickr30k entity dataset
(Plummer et al., 2015) and MS COCO (Lin et al., 2014) dataset for measuring
both ground truth attention map and semantic labelings of image regions. In
order to learn a better attention function, it proposed supervised attention
model. Two types of supervised attention models are used here: strong
supervision with alignment annotation and weak supervision with semantic
labelling. In strong supervision with alignment annotation model, it can
directly map ground truth word to a region. However, ground truth alignment is
not always possible because collecting and annotating data is often very
expensive. Weak supervision is performed to use bounding box or segmentation
masks on MS COCO dataset. In experiments, the method shows that supervised
attention model performs better in mapping attention as well as image
captioning.

Chen et al. (Chen et al., 2017b) proposed another attention-based image
captioning method. This method considers both spatial and channel wise
attentions to compute an attention map. The existing attention-based image
captioning methods only consider spatial information for generating an
attention map. A common drawback of these spatial attention methods are that
they compute weighted pooling only on attentive feature map. As a result,
these methods lose the spatial information gradually. Moreover, they use the
spatial information only from the last conv-layer of the CNN. The receptive
field regions of this layer are quite large that make the limited gap between
the regions. Therefore, they do not get significant spatial attentions for an
image. However, in this method, CNN features are extracted not only from
spatial locations but also from different channels and multiple layers.
Therefore, it gets significant spatial attention. In addition to this, in this
method, each filter of a convolutional layer acts as semantic detectors
(Zeiler and Fergus, 2014) while other methods use external sources for
obtaining semantic information.

In order to reduce the gap between human generated description and machine
generated description Tavakoli et al. (Tavakoli et al., 2017) introduced an
attention-based image captioning method. This is a bottom up saliency based
attention model that can take advantages for comparisons with other attention-
based image captioning methods. It found that humans first describe the more
important objects than less important ones. It also shows that the method
performs better on unseen data.

Most previous image captioning methods applied top-down approach for
constructing a visual attention map. These mechanisms typically focused on
some selective regions obtained from the output of one or two layers of a CNN.
The input regions are of the same size and have the same shape of receptive
field. This approach has a little consideration to the content of the image.
However, the method of Anderson et al. (Anderson et al., 2017) applied both
top down and bottom up approaches. The bottom up attention mechanism uses
Faster R-CNN (Ren et al., 2015a) for region proposals that can select salient
regions of an image . Therefore, this method can attend both object level
regions as well as other salient image regions.

Park et al. (Park et al., 2017) introduced a different type of attention-based
image captioning method. This method can generate image captions addressing
personal issues of an image. It mainly considers two tasks : hashtag
prediction and post generation. This method uses a Context Sequence Memory
Network (CSMN) to obtain the context information from the image. Description
of an image from personalized view has a lot of applications in social media
networks. For example, everyday people share a lot of images as posts in
Facebook, Instagram or other social media. Photo-taking or uploading is a very
easy task. However, describing them is not easy because it requires theme,
sentiment, and context of the image. Therefore, the method considers the past
knowledge about the user's vocabularies or writing styles from the prior
documents for generating image descriptions. In order to work with this new
type of image captioning, the CSMN method has three contributions: first, the
memory of this network can work as a repository and retain multiple types of
context information. Second, the memory is designed in such a way that it can
store all the previously generated words sequentially. As a result, it does
not suffer from vanishing gradient problem. Third, the proposed CNN can
correlate with multiple memory slots that is helpful for understanding
contextual concepts.

Attention-based methods have already shown good performance and efficiency in
image captioning as well as other computer vision tasks. However, attention
maps generated by these attention based methods are only machine dependent.
They do not consider any supervision from human attention. This creates the
necessity to think about the gaze information whether it can improve the
performance of these attention methods in image captioning. Gaze indicates the
cognition and perception of humans about a scene. Human gaze can identify the
important locations of objects in an image. Thus, gaze mechanisms have already
shown their potential performances in eye-based user modeling (Bulling et al.,
2011; Fathi et al., 2012; Papadopoulos et al., 2014; Sattar et al., 2015;
Shanmuga Vadivel et al., 2015), object localization (Mishra et al., 2012) or
recognition (Karthikeyan et al., 2013) and holistic scene understanding (Yun
et al., 2013; Zelinsky, 2013). However, Sugano et al. (Sugano and Bulling,
2016) claimed that gaze information has not yet been integrated in image
captioning methods. This method introduced human gaze with the attention
mechanism of deep neural networks in generating image captions. The method
incorporates human gaze information into an attention-based LSTM model (Xu et
al., 2015). For experiments, it uses SALICON dataset (Jiang et al., 2015) and
achieves good results.

Figure 8. A block diagram of a semantic concept-based image captioning.

####  3.5.2. Semantic Concept-Based Image Captioning

Semantic concept-based methods selectively attend to a set of semantic concept
proposals extracted from the image. These concepts are then combined into
hidden states and the outputs of recurrent neural networks.

The methods in this category follow the following steps:

  1. (1)

CNN based encoder is used to encode the image features and semantic concepts.

  2. (2)

Image features are fed into the input of language generation model.

  3. (3)

Semantic concepts are added to the different hidden states of the language
model.

  4. (4)

The language generation part produces captions with semantic concepts.

A typical block diagram of this category is shown in Figure 8.

Karpathy et al. extended their method (Karpathy et al., 2014) in (Karpathy and
Fei-Fei, 2015). The later method can generate natural language descriptions
for both images as well as for their regions. This method employs a novel
combination of CNN over the image regions, bidirectional Recurrent Neural
Networks over sentences, and a common multimodal embedding that associates the
two modalities. It also demonstrates a multimodal recurrent neural network
architecture that utilizes the resultant alignments to train the model for
generating novel descriptions of image regions. In this method, dependency
tree relations (DTR) are used to train to map the sentence segments with the
image regions that have a fixed window context. In contrast to their previous
method, this method uses a bidirectional neural network to obtain word
representations in the sentence. It considers contiguous fragments of
sentences to align in embedding space which is more meaningful, interpretable,
and not fixed in length. Generally an RNN considers the current word and the
contexts from all the previously generated words for estimating a probability
distribution of the next word in a sequence. However, this method extends it
for considering the generative process on the content of an input image. This
addition is simple but it makes it very effective for generating novel image
captions.

Attributes of an image are considered as rich semantic cues. The method of Yao
et al. (Yao et al., 2017b) has different architectures to incorporate
attributes with image representations. Mainly, two types of architectural
representations are introduced here. In the first group, it inserts only
attributes to the LSTM or image representations to the LSTM first and then
attributes and vice versa. In the second group, it can control the time step
of LSTM. It decides whether image representation and attributes will be
inputted once or every time step. These variants of architectures are tested
on MS COCO dataset and common evaluation metrics.

You et al. (You et al., 2016) proposed a semantic attention-based image
captioning method. The method provides a detailed, coherent description of
semantically important objects. The top-down paradigms (Chen and Lawrence
Zitnick, 2015; Vinyals et al., 2015; Mao et al., 2015b; Karpathy and Fei-Fei,
2015; Donahue et al., 2015; Xu et al., 2015; Mao et al., 2015a) are used for
extracting visual features first and then convert them into words. In bottom
up approaches, (Farhadi et al., 2010; Kulkarni et al., 2011; Li et al., 2011;
Elliott and Keller, 2013; Kuznetsova et al., 2012; Lebret et al., 2015) visual
concepts (e.g., regions, objects, and attributes) are extracted first from
various aspects of an image and then combine them. Fine details of an image
are often very important for generating a description of an image. Top- down
approaches have limitations in obtaining fine details of the image. Bottom up
approaches are capable of operating on any image resolution and therefore they
can do work on fine details of the image. However, they have problems in
formulating an end to end process. Therefore, semantic based attention model
applied both top-down and bottom up approaches for generating image captions.
In top-down approaches, the image features are obtained using the last
1024-dimensional convolutional layer of the GoogleNet (Szegedy et al., 2015)
CNN model. The visual concepts are collected using different non-parametric
and parametric method. Nearest neighbour image retrieval technique is used for
computing non-parametric visual concepts. Fully convolutional network (FCN)
(Long et al., 2015) is used to learn attribute from local patches for
parametric attribute prediction. Although Xu et al. (Xu et al., 2015)
considered attention-based captioning, it works on fixed and pre-defined
spatial location. However, this semantic attention-based method can work on
any resolution and any location of the image. Moreover, this method also
considers a feedback process that accelerates to generate better image
captions.

Previous image captioning methods do not include high level semantic concepts
explicitly. However, Wu et al. (Wu16W) proposed a high-level semantic concept-
based image captioning. It uses an intermediate attribute prediction layer in
a neural network-based CNN-LSTM framework. First, attributes are extracted by
a CNN-based classifier from training image captions. Then these attributes are
used as high level semantic concepts in generating semantically rich image
captions.

Figure 9. A block diagram of a typical novel object-based image captioning.

Recent semantic concept based image captioning methods (Wu et al., 2018; You
et al., 2016) applied semantic-concept-detection process (Gan et al., 2016)to
obtain explicit semantic concepts. They use these high level semantic concepts
in CNN-LSTM based encoder-decoder and achieves significant improvements in
image captioning. However, they have problems in generating semantically sound
captions. They cannot distribute semantic concepts evenly in the whole
sentence. For example, Wu et al. (Wu et al., 2018) consider the initial state
of the LSTM to add semantic concepts. Moreover, it encodes visual features
vector or an inferred scene vector from the CNN and then feeds them to LSTM
for generating captions. However, Gan et al. (Gan et al., 2017b) introduced a
Semantic Compositional Network (SCN) for image captioning. In this method, a
semantic concept vector is constructed from all the probable concepts (called
tags here) found in the image. This semantic vector has more potential than
visual feature vector and scene vector and can generate captions covering the
overall meaning of the image. This is called compositional network because it
can compose most semantic concepts.

Existing LSTM based image captioning methods have limitations in generating a
diverse set of captions because they have to predict the next word on a
predefined word by word format. However, a combination of attributes, subjects
and their relationship in a sentence irrespective of their location can
generate a broad range of image captions. Wang et al. (Wang et al., 2017)
proposed a method that locates the objects and their interactions first and
then identifies and extracts the relevant attributes to generate image
captions. The main aim of this method is to decompose the ground truth image
captions into two parts: Skeleton sentence and attribute phrases. The method
is also called Skeleton Key. The architecture of this method has ResNet (He et
al., 2016) and two LSTMs called Skel-LSTM and Attr-LSTM. During training,
skeleton sentences are trained by Skel-LSTM network and attribute phrases are
trained by the Attr-LSTM network. In the testing phase, skeleton sentences are
generated first that contain the words for main objects of the image and their
relationships. Then these objects look back through the image again to obtain
the relevant attributes. It is tested on MS COCO dataset and a new Stock3M
dataset and can generate more accurate and novel captions.

####  3.5.3. Novel Object-based Image Captioning

Despite recent deep learning-based image captioning methods have achieved
promising results, they largely depend on the paired image and sentence
caption datasets. These type of methods can only generate description of the
objects within the context. Therefore, the methods require a large set of
training image-sentence pairs. Novel object-based image captioning methods can
generate descriptions of novel objects which are not present in paired image-
captions datasets.

The methods of this category follow the following general steps:

  1. (1)

A separate lexical classifier and a language model are trained on unpaired
image data and unpaired text data.

  2. (2)

A deep caption model is trained on paired image caption data.

  3. (3)

Finally, both models are combined together to train jointly in that can
generate captions for novel object.

Figure 10. A block diagram of image captioning based on different styles.

A simple block diagram of a novel object-based image captioning method is
given in Figure 9.

Current image captioning methods are trained on image-captions paired
datasets. As a result, if they get unseen objects in the test images, they
cannot present them in their generated captions. Anne et al. (Anne Hendricks
et al., 2016) proposed a Deep Compositional Captioner (DCC) that can represent
the unseen objects in generated captions.

Yao et al. (Yao et al., 2017a) proposed a copying mechanism to generate
description for novel objects. This method uses a separate object recognition
dataset to develop classifiers for novel objects. It integrates the
appropriate words in the output captions by a decoder RNN with copying
mechanism. The architecture of the method adds a new network to recognize the
unseen objects from unpaired images and incorporate them with LSTM to generate
captions.

Generating captions for the unseen images is a challenging research problem.
Venugopalan et al. (Venugopalan et al., 2017) introduced a Novel Object
Captioner (NOC) for generating captions for unseen objects in the image. They
used external sources for recognizing unseen objects and learning semantic
knowledge.

####  3.5.4. Stylized Caption

Existing image captioning systems generate captions just based on only the
image content that can also be called factual description. They do not
consider the stylized part of the text separately from other linguistic
patterns. However, the stylized captions can be more expressive and attractive
than just only the flat description of an image.

The methods of this category follow the following general steps:

  1. (1)

CNN based image encoder is used to obtain the image information.

  2. (2)

A separate text corpus is prepared to extract various stylized concepts (For
example: romantic, humorous) from training data.

  3. (3)

The language generation part can generate stylized and attractive captions
using the information of Step 1 and Step 2.

A simple block diagram of stylized image captioning is given in Figure 10.

Such captions have become popular because they are particularly valuable for
many real-world applications. For example, everyday people are uploading a lot
of photos in different social media. The photos need stylized and attractive
descriptions. Gan et al. (Gan et al., 2017a) proposed a novel image captioning
system called StyleNet. This method can generate attractive captions adding
various styles. The architecture of this method consists of a CNN and a
factored LSTM that can separate factual and style factors from the captions.
It uses multitask sequence to sequence training (Luong et al., 2016) for
identifying the style factors and then add these factors at run time for
generating attractive captions. More interestingly, it uses an external
monolingual stylized language corpus for training instead of paired images.
However, it uses a new stylized image caption dataset called FlickrStyle10k
and can generate captions with different styles.

Existing image captioning methods consider the factual description about the
objects, scene, and their interactions of an image in generating image
captions. In our day to day conversations, communications, interpersonal
relationships, and decision making we use various stylized and non-factual
expressions such as emotions, pride, and shame. However, Mathews et al.
(Mathews et al., 2016) claimed that automatic image descriptions are missing
this non-factual aspects. Therefore, they proposed a method called SentiCap.
This method can generate image descriptions with positive or negative
sentiments. It introduces a novel switching RNN model that combines two
CNN+RNNs running in parallel. In each time step, this switching model
generates the probability of switching between two RNNs. One generates
captions considering the factual words and other considers the words with
sentiments. It then takes inputs from the hidden states of both two RNNs for
generating captions. This method can generate captions successfully given the
appropriate sentiments.

###  3.6. LSTM vs. Others

Image captioning intersects computer vision and natural language processing
(NLP) research. NLP tasks, in general, can be formulated as a sequence to
sequence learning. Several neural language models such as neural probabilistic
language model (Bengio et al., 2003), log-bilinear models (Mnih and Hinton,
2007b), skip-gram models (Mikolov et al., 2013), and recurrent neural networks
(RNNs) (Mikolov et al., 2010) have been proposed for learning sequence to
sequence tasks. RNNs have widely been used in various sequence learning tasks.
However, traditional RNNs suffer from vanishing and exploding gradient
problems and cannot adequately handle long-term temporal dependencies.

LSTM (Hochreiter and Schmidhuber, 1997) networks are a type of RNN that has
special units in addition to standard units. LSTM units use a memory cell that
can maintain information in memory for long periods of time. In recent years,
LSTM based models have dominantly been used in sequence to sequence learning
tasks. Another network, Gated Recurrent Unit (GRU) (Chung et al., 2014) has a
similar structure to LSTM but it does not use separate memory cells and uses
fewer gates to control the flow of information.

However, LSTMs ignore the underlying hierarchical structure of a sentence.
They also require significant storage due to long-term dependencies through a
memory cell. In contrast, CNNs can learn the internal hierarchical structure
of the sentences and they are faster in processing than LSTMs. Therefore,
recently, convolutional architectures are used in other sequence to sequence
tasks, e.g., conditional image generation (van den Oord et al., 2016) and
machine translation (Gehring et al., 2016; Gehring et al., 2017; Vaswani et
al., 2017).

Inspired by the above success of CNNs in sequence learning tasks, Gu et al.
(Gu et al., 2017) proposed a CNN language model-based image captioning method.
This method uses a language-CNN for statistical language modelling. However,
the method cannot model the dynamic temporal behaviour of the language model
only using a language-CNN. It combines a recurrent network with the language-
CNN to model the temporal dependencies properly. Aneja et al. (Aneja et al.,
2018) proposed a convolutional architecture for the task of image captioning.
They use a feed-forward network without any recurrent function. The
architecture of the method has four components: (i) input embedding layer (ii)
image embedding layer (iii) convolutional module, and (iv) output embedding
layer. It also uses an attention mechanism to leverage spatial image features.
They evaluate their architecture on the challenging MSCOCO dataset and shows
comparable performance to an LSTM based method on standard metrics.

Wang et al. (Wang and Chan, 2018) proposed another CNN+CNN based image
captioning method. It is similar to the method of Aneja et al. except that it
uses a hierarchical attention module to connect the vision-CNN with the
language-CNN. The authors of this method also investigate the use of various
hyperparameters, including the number of layers and the kernel width of the
language-CNN. They show that the influence of the hyperparameters can improve
the performance of the method in image captioning.

##  4\. Datasets and Evaluation Metrics

A number of datasets are used for training, testing, and evaluation of the
image captioning methods. The datasets differ in various perspective such as
the number of images, the number of captions per image, format of the
captions, and image size. Three datasets: Flickr8k (Hodosh et al., 2013),
Flickr30k (Plummer et al., 2015), and MS COCO Dataset (Lin et al., 2014) are
popularly used. These datasets together with others are described in Section
4.1. In this section, we show sample images with their captions generated by
image captioning methods on MS COCO, Flickr30k, and Flickr8k datasets. A
number of evaluation metrics are used to measure the quality of the generated
captions compared to the ground-truth. Each metric applies its own technique
for computation and has distinct advantages. The commonly used evaluation
metrics are discussed in Section 4.2. A summary of deep learning-based image
captioning methods with their datasets and evaluation metrics are listed in
Table 2.

Figure 11. Captions generated by Wu et al. (Wu et al., 2015) on some sample
images from the MS COCO dataset.

###  4.1.  Datasets

####  4.1.1. MS COCO Dataset

Microsoft COCO Dataset (Lin et al., 2014) is a very large dataset for image
recognition, segmentation, and captioning. There are various features of MS
COCO dataset such as object segmentation, recognition in context, multiple
objects per class, more than 300,000 images, more than 2 million instances, 80
object categories, and 5 captions per image. Many image captioning methods
(Jin et al., 2015; Wu and Cohen, 2016; Tran et al., 2016; Wang et al., 2016b;
You et al., 2016; Gan et al., 2017a; Pedersoli et al., 2017; Ren et al., 2017;
Dai et al., 2017; Shetty et al., 2017; Wu et al., 2015) use the dataset in
their experiments. For example, Wu et al. (Wu et al., 2015) use MS COCO
dataset in their method and the generated captions of two sample images are
shown in Figure 11.

####  4.1.2. Flickr30K Dataset

Flickr30K (Plummer et al., 2015) is a dataset for automatic image description
and grounded language understanding. It contains 30k images collected from
Flickr with 158k captions provided by human annotators. It does not provide
any fixed split of images for training, testing, and validation. Researchers
can choose their own choice of numbers for training, testing, and validation.
The dataset also contains detectors for common objects, a color classifier,
and a bias towards selecting larger objects. Image captioning methods such as
(Karpathy and Fei-Fei, 2015; Vinyals et al., 2015; Wang et al., 2016b; Wu et
al., 2018; Chen et al., 2017a) use this dataset for their experiments. For
example, performed their experiment on Flickr30k dataset. The generated
captions by Chen et al. (Chen et al., 2017a) of two sample images of the
dataset are shown in Figure 12.

2*Reference |  2*Datasets |  2*Evaluation Metrics
---|---|---
Kiros et al. 2014 (Kiros et al., 2014a) | IAPR TC-12,SBU | BLEU, PPLX
Kiros et al. 2014 (Kiros et al., 2014b) | Flickr 8K, Flickr 30K | R@K, mrank
Mao et al. 2014 (Mao et al., 2014) | IAPR TC-12, Flickr 8K/30K | BLEU, R@K, mrank
Karpathy et al. 2014 (Karpathy et al., 2014) | PASCAL1K, Flickr 8K/30K | R@K, mrank
Mao et al. 2015 (Mao et al., 2015b) |  | IAPR TC-12, Flickr 8K/30K,
---
MS COCO
BLEU, R@K, mrank
Chen et al. 2015 (Chen and Lawrence Zitnick, 2015) |  | PASCAL, Flickr 8K/30K,
---
MS COCO
| BLEU, METEOR,
---
CIDEr
Fang et al. 2015 (Fang et al., 2015) | PASCAL, MS COCO | BLEU, METEOR, PPLX
Jia et al. 2015 (Jia et al., 2015) | Flickr 8K/30K, MS COCO | BLEU, METEOR, CIDEr
Karpathy et al. 2015 (Karpathy and Fei-Fei, 2015) | Flickr 8K/30K, MS COCO | BLEU, METEOR, CIDEr
Vinyals et al. 2015 (Vinyals et al., 2015) | Flickr 8K/30K, MS COCO | BLEU, METEOR, CIDEr
Xu et al. 2015 (Xu et al., 2015) | Flickr 8K/30K, MS COCO | BLEU, METEOR
Jin et al. 2015 (Jin et al., 2015) | Flickr 8K/30K, MS COCO | BLEU, METEOR, ROUGE, CIDEr
Wu et al. 2016 (Wu and Cohen, 2016) | MS COCO | BLEU, METEOR, CIDEr
Sugano et at. 2016 (Sugano and Bulling, 2016) | MS COCO | BLEU, METEOR, ROUGE, CIDEr
Mathews et al. 2016 (Mathews et al., 2016) | MS COCO, SentiCap | BLEU, METEOR, ROUGE, CIDEr
Wang et al. 2016 (Wang et al., 2016b) | Flickr 8K/30K, MS COCO | BLEU, R@K
Johnson et al. 2016 (Johnson et al., 2016) | Visual Genome | METEOR, AP, IoU
Mao et al. 2016 (Mao et al., 2016) | ReferIt | BLEU, METEOR, CIDEr
Wang et al. 2016 (Wang et al., 2016a) | Flickr 8K | BLEU, PPL, METEOR
Tran et al. 2016 (Tran et al., 2016) |  | MS COCO, Adobe-MIT,
---
Instagram
Human Evaluation
Ma et al. 2016 (Ma and Han, 2016) | Flickr 8k, UIUC | BLEU, R@K
You et al. 2016 (You et al., 2016) | Flickr 30K, MS COCO | BLEU, METEOR, ROUGE, CIDEr
Yang et al. 2016 (Yang et al., 2016) | Visual Genome | METEOR, AP, IoU
Anne et al. 2016 (Anne Hendricks et al., 2016) | MS COCO, ImageNet | BLEU, METEOR
Yao et al. 2017 (Yao et al., 2017b) | MS COCO | BLEU, METEOR, ROUGE, CIDEr
Lu et al. 2017 (Lu et al., 2017) | Flickr 30K, MS COCO | BLEU, METEOR, CIDEr
Chen et al. 2017 (Chen et al., 2017b) | Flickr 8K/30K, MS COCO | BLEU, METEOR, ROUGE, CIDEr
Gan et al. 2017 (Gan et al., 2017b) | Flickr 30K, MS COCO | BLEU, METEOR, CIDEr
Pedersoli et al. 2017 (Pedersoli et al., 2017) | MS COCO | BLEU, METEOR, CIDEr
Ren et al. 2017 (Ren et al., 2017) | MS COCO | BLEU, METEOR, ROUGE, CIDEr
Park et al. 2017 (Park et al., 2017) | Instagram | BLEU, METEOR, ROUGE, CIDEr
Wang et al. 2017 (Wang et al., 2017) | MS COCO, Stock3M | SPICE, METEOR, ROUGE, CIDEr
Tavakoli et al. 2017 (Tavakoli et al., 2017) | MS COCO, PASCAL 50S | BLEU, METEOR, ROUGE, CIDEr
Liu et al. 2017 (Liu et al., 2017a) | Flickr 30K, MS COCO | BLEU, METEOR
Gan et al. 2017 (Gan et al., 2017a) | FlickrStyle10K | BLEU, METEOR, ROUGE, CIDEr
Dai et al. 2017 (Dai et al., 2017) | Flickr 30K, MS COCO | E-NGAN, E-GAN, SPICE, CIDEr
Shetty et al. 2017 (Shetty et al., 2017) | MS COCO |  | Human Evaluation,
---
SPICE, METEOR
Liu et al. 2017 (Liu et al., 2017b) | MS COCO | SPIDEr, Human Evaluation
Gu et al. 2017 (Gu et al., 2017) | Flickr 30K, MS COCO | BLEU, METEOR, CIDEr, SPICE
Yao et al. 2017 (Yao et al., 2017a) | MS COCO, ImageNet | METEOR
Rennie et al. 2017 (Rennie et al., 2017) | MS COCO | BLEU, METEOR, CIDEr, ROUGE
Vsub et al. 2017 (Venugopalan et al., 2017) | MS COCO, ImageNet | METEOR
Zhang et al. 2017 (Zhang et al., 2017) | MS COCO | BLEU, METEOR, ROUGE, CIDEr
Wu et al. 2018 (Wu et al., 2018) | Flickr 8K/30K, MS COCO | BLEU, METEOR, CIDEr
Aneja et al. 2018 (Aneja et al., 2018) | MS COCO | BLEU, METEOR, ROUGE, CIDEr
Wang et al. 2018 (Wang and Chan, 2018) | MS COCO | BLEU, METEOR, ROUGE, CIDEr

Table 2. An overview of methods, datasets, and evaluation metrics Figure 12.
Captions generated by Chen et al. (Chen et al., 2017a) on some sample images
from the Flickr30k dataset.

####  4.1.3. Flickr8K Dataset

Flickr8k (Hodosh et al., 2013) is a popular dataset and has 8000 images
collected from Flickr. The training data consists of 6000 images, the test and
development data, each consists of 1,000 images. Each image in the dataset has
5 reference captions annotated by humans. A number of image captioning methods
(Jia et al., 2015; Jin et al., 2015; Xu et al., 2015; Wang et al., 2016b; Wu
et al., 2018; Chen et al., 2017b) have performed experiments using the
dataset. Two sample results by Jia et al. (Jia et al., 2015) on this dataset
are shown in Figure 13.

####  4.1.4. Visual Genome Dataset

Visual Genome dataset (Krishna et al., 2017) is another dataset for image
captioning. Image captioning requires not only to recognise the objects of an
image but it also needs reasoning their interactions and attributes. Unlike
the first three datasets where a caption is given to the whole scene, Visual
Genome dataset has separate captions for multiple regions in an image. The
dataset has seven main parts: region descriptions, objects, attributes,
relationships, region graphs, scene graphs, and question answer pairs. The
dataset has more than 108k images. Each image contains an average of 35
objects, 26 attributes, and 21 pairwise relationships between objects.

####  4.1.5. Instagram Dataset

Tran et al. (Tran et al., 2016) and Park et al. (Park et al., 2017) created
two datasets using images from Instagram which is a photo-sharing social
networking services. The dataset of Tran et al. has about 10k images which are
mostly from celebrities. However, Park et al. used their dataset for hashtag
prediction and post-generation tasks in social media networks. This dataset
contains 1.1m posts on a wide range of topics and a long hashtag lists from
6.3k users.

Figure 13. Captions generated by Jia et al. (Jia et al., 2015) on some sample
images from the Flickr8k dataset.

####  4.1.6. IAPR TC-12 Dataset

IAPR TC-12 dataset (Grubinger et al., 2006) has 20k images. The images are
collected from various sources such as sports, photographs of people, animals,
landscapes and many other locations around the world. The images of this
dataset have captions in multiple languages. Images have multiple objects as
well.

####  4.1.7. Stock3M Dataset

Stock3M dataset has 3,217,654 images uploaded by users and it is 26 times
larger than MSCOCO dataset. The images of this dataset have a diversity of
content.

####  4.1.8. MIT-Adobe FiveK dataset

MIT-Adobe FiveK (Bychkovsky et al., 2011) dataset consists of 5,000 images.
These images contain a diverse set of scenes, subjects, and lighting
conditions and they are mainly about people, nature, and man-made objects.

####  4.1.9. FlickrStyle10k Dataset

FlickrStyle10k dataset has 10,000 Flickr images with stylized captions. The
training data consists of 7000 images. The validation and test data consists
of 2,000 and 1,000 images respectively. Each image contains romantic,
humorous, and factual captions.

###  4.2. Evaluation Metrics

####  4.2.1. BLEU

BLEU (Bilingual evaluation understudy) (Papineni et al., 2002) is a metric
that is used to measure the quality of machine generated text. Individual text
segments are compared with a set of reference texts and scores are computed
for each of them. In estimating the overall quality of the generated text, the
computed scores are averaged. However, syntactical correctness is not
considered here. The performance of the BLEU metric is varied depending on the
number of reference translations and the size of the generated text.
Subsequently, Papineni et al. introduced a modified precision metric. This
metrics uses n-grams. BLEU is popular because it is a pioneer in automatic
evaluation of machine translated text and has a reasonable correlation with
human judgements of quality (Denoual and Lepage, 2005; Callison-Burch et al.,
2006). However, it has a few limitations such as BLEU scores are good only if
the generated text is short (Callison-Burch et al., 2006). There are some
cases where an increase in BLEU score does not mean that the quality of the
generated text is good (Lin and Och, 2004).

####  4.2.2. ROUGE

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) (Lin, 2004) is a set
of metrics that are used for measuring the quality of text summary. It
compares word sequences, word pairs, and n-grams with a set of reference
summaries created by humans. Different types of ROUGE such as ROUGE-1, 2,
ROUGE-W, ROUGE-SU4 are used for different tasks. For example, ROUGE-1 and
ROUGE-W are appropriate for single document evaluation whereas ROUGE-2 and
ROUGE-SU4 have good performance in short summaries. However, ROUGE has
problems in evaluating multi-document text summary.

####  4.2.3. METEOR

METEOR (Metric for Evaluation of Translation with Explicit ORdering) (Banerjee
and Lavie, 2005) is another metric used to evaluate the machine translated
language. Standard word segments are compared with the reference texts. In
addition to this, stems of a sentence and synonyms of words are also
considered for matching. METEOR can make better correlation at the sentence or
the segment level.

####  4.2.4. CIDEr

CIDEr (Consensus-based Image Descripton Evaluation) (Vedantam et al., 2015) is
an automatic consensus metric for evaluating image descriptions. Most existing
datasets have only five captions per image. Previous evaluation metrics work
with these small number of sentences and are not enough to measure the
consensus between generated captions and human judgement. However, CIDEr
achieves human consensus using term frequency-inverse document frequency (TF-
IDF) (Robertson, 2004).

####  4.2.5. SPICE

SPICE (Semantic Propositional Image Caption Evaluation) (Anderson et al.,
2016) is a new caption evaluation metric based on semantic concept. It is
based on a graph-based semantic representation called scene-graph (Johnson et
al., 2015; Schuster et al., 2015). This graph can extract the information of
different objects, attributes and their relationships from the image
descriptions.

Existing image captioning methods compute log-likelihood scores to evaluate
their generated captions. They use BLEU, METEOR, ROUGE, SPICE, and CIDEr as
evaluation metrics. However, BLEU, METEOR, ROUGE are not well correlated with
human assessments of quality. SPICE and CIDEr have better correlation but they
are hard to optimize. Liu et al. (Liu et al., 2017b) introduced a new captions
evaluation metric that is a good choice by human raters. It is developed by a
combination of SPICE and CIDEr, and termed as SPIDEr. It uses a policy
gradient method to optimize the metrics.

The quality of image captioning depends on the assessment of two main aspects:
adequacy and fluency. An evaluation metric needs to focus on a diverse set of
linguistic features to achieve these aspects. However, commonly used
evaluation metrics consider only some specific features (e.g., lexical or
semantic) of languages. Sharif et al. (Sharif et al., 2018) proposed learning-
based composite metrics for evaluation of image captions. The composite metric
incorporates a set of linguistic features to achieve the two main aspects of
assessment and shows improved performances.

##  5\. Comparison on benchmark datasets and common evaluation metrics

While formal experimental evaluation was left out of the scope of this paper,
we present a brief analysis of the experimental results and the performance of
various techniques as reported. We cover three sets of results:

  1. (1)

We find a number of methods use the first three datasets listed in Section
4.1. and a number of commonly used evaluation metrics to present the results.
These results are shown in Table 3.

  2. (2)

A few methods fall into the following groups: Attention-based and Other deep
learning-based (Reinforcement learning and GAN-based methods) image
captioning. The results of such methods are shown in Tables 4 and 5,
respectively.

  3. (3)

We also list the methods that proivide top two results scored on each
evaluation metric on the MSCOCO dataset. These results are shown in Table 6.

As shown in Table 3, on Flickr8k, Mao et al. achieved 0.565, 0.386, 0.256, and
0.170 on BLEU-1, BLEU-2, BLEU-3, and BLEU-4 , respectively. For Flickr30k
dataset, the scores are 0.600, 0.410, 0.280, and 0.190, respectively which are
higher than the Flickr8k scores. The highest scores were achieved on the
MSCOCO dataset. The higher results on a larger dataset follows the fact that a
large dataset has more data, comprehensive representation of various scenes,
complexities, and their own natural context. The results of Jia et al. are
similar for Flickr8k and Flickr30k datasets but higher on MSCOCO dataset. The
method uses visual space for mapping image-features and text features. Mao et
al. use multimodal space for the mapping of image-features and text features.
On the other hand, Jia et al. use visual space for the mapping. Moreover, the
method uses an Encoder-Decoder architecture where it can guide the decoder
part dynamically. Consequently, this method performs better than Mao et al.

Xu et al. also perform better on MSCOCO dataset. This method outperformed both
Mao et al. and Jia et al. The main reason behind this is that it uses an
attention mechanism which focuses only on relevant objects of the image. The
semantic concept-based methods can generate semantically rich captions. Wu et
al. proposed a semantic concept-based image captioning method. This method
first predicts the attributes of different objects from the image and then
adds these attributes with the captions which are semantically meaningful. In
terms of performance, the method is superior to all the methods mentioned in
Table 3.

Dataset | Method | Category | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR
---|---|---|---|---|---|---|---
4*Flickr8k | Mao et al. 2015 (Mao et al., 2015b) | MS,SL,WS | 0.565 | 0.386 | 0.256 | 0.170 | -
| Jia et al. 2015 (Jia et al., 2015) | VS,SL,WS,EDA | 0.647 | 0.459 | 0.318 | 0.216 | 0.201
| Xu et al. 2015 (Xu et al., 2015) | VS,SL,WS,EDA,AB | 0.670 | 0.457 | 0.314 | 0.213 | 0.203
| Wu et al. 2018 (Wu et al., 2018) | VS,SL,WS,EDA,SCB | 0.740 | 0.540 | 0.380 | 0.270 | -
4*Flickr30k | Mao et al. 2015 (Mao et al., 2015b) | MS,SL,WS | 0.600 | 0.410 | 0.280 | 0.190 | -
| Jia et al. 2015 (Jia et al., 2015) | VS,SL,WS,EDA | 0.646 | 0.466 | 0.305 | 0.206 | 0.179
| Xu et al. 2015 (Xu et al., 2015) | VS,SL,WS,EDA,AB | 0.669 | 0.439 | 0.296 | 0.199 | 0.184
| Wu et al. 2018 (Wu et al., 2018) | VS,SL,WS,EDA,SCB | 0.730 | 0.550 | 0.400 | 0.280 | -
4*MSCOCO | Mao et al. 2015 (Mao et al., 2015b) | MS,SL,WS | 0.670 | 0.490 | 0.350 | 0.250 | -
| Jia et al. 2015 (Jia et al., 2015) | VS,SL,WS,EDA | 0.670 | 0.491 | 0.358 | 0.264 | 0.227
| Xu et al. 2015 (Xu et al., 2015) | VS,SL,WS,EDA,AB | 0.718 | 0.504 | 0.357 | 0.250 | 0.230
| Wu et al. 2018 (Wu et al., 2018) | VS,SL,WS,EDA,SCB | 0.740 | 0.560 | 0.420 | 0.310 | 0.260

Table 3. Performance of different image captioning methods on three benchmark
datasets and commonly used evaluation metrics.

Table 4 shows the results of attention-based based methods on MSCOCO dataset.
Xu et al.’s stochastic hard attention produced better results than
deterministic soft attention. However, these results were outperformed by Jin
et al. which can update its attention based on the scene-specific context.

Wu et al. 2016 and Pedersoli et al. 2017 only show BLEU-4 and METEOR scores
which are higher than the aforementioned methods. The method of Wu et al. uses
an attention mechanism with a review process. The review process checks the
focused attention in every time step and updates it if necessary. This
mechanism helps to achieve better results than the prior attention-based
methods. Pedersoli et al. propose a different attention mechanism that maps
the focused image regions directly with the caption words instead of LSTM
state. This behavior of the method drives it to achieve top performances among
the mentioned attention-based methods in Table 4.

Reinforcement learning-based (RL) and GAN-based methods are becoming
increasingly popular. We name them as “Other Deep Learning-based Image
Captioning”. The results of the methods of this group are shown in Table 5.
The methods do not have results on commonly used evaluation metrics. However,
they have their own potentials to generate the descriptions for the image.

Shetty et al. employed adversarial training in their image captioning method.
This method is capable to generate diverse captions. The captions are less-
biased with the ground-truth captions compared to the methods use maximum
likelihood estimation. To take the advantages of RL, Ren et al. proposed a
method that can predict all possible next words for the current word in
current time step. This mechanism helps them to generate contextually more
accurate captions. Actor-critic of RL are similar to the Generator and the
Discriminator of GAN. However, at the beginning of the training, both actor
and critic do not have any knowledge about data. Zhang et al. proposed an
actor-critic-based image captioning method. This method is capable of
predicting the ultimate captions at its early stage and can generate more
accurate captions than other reinforcement learning-based methods.

2*Method |  2*Category | MS COCO
---|---|---
|  | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | CIDEr
Xu et al. 2015 (Xu et al., 2015), soft | VS,SL,WS,EDA,VC | 0.707 | 0.492 | 0.344 | 0.243 | 0.239 | - | -
Xu et al. 2015 (Xu et al., 2015), hard | VS,SL,WS,EDA,VC | 0.718 | 0.504 | 0.357 | 0.250 | 0.230 | - | -
Jin et al. 2015 (Jin et al., 2015) | VS,SL,WS,EDA,VC | 0.697 | 0.519 | 0.381 | 0.282 | 0.235 | 0.509 | 0.838
Wu et al. 2016 (Wu and Cohen, 2016) | VS,SL,WS,EDA,VC | - | - | - | 0.290 | 0.237 | - | 0.886
Pedersoli et al. 2017 (Pedersoli et al., 2017) | VS,SL,WS,EDA,VC | - | - | - | 0.307 | 0.245 | - | 0.938

Table 4. Performance of attention-based image captioning methods on MSCOCO
dataset and commonly used evaluation metrics.

2*Method |  2*Category | MS COCO
---|---|---
|  | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE
Shetty et al. 2017GAN (Shetty et al., 2017) | VS,ODL,WS,EDA | - | - | - | - | 0.239 | - | - | 0.167
Ren et al. 2017RL (Ren et al., 2017) | VS,ODL,WS,EDA | 0.713 | 0.539 | 0.403 | 0.304 | 0.251 | 0.525 | 0.937 | -
Zhang et al. 2017RL (Zhang et al., 2017) | VS,ODL,WS,EDA | - | - | - | 0.344 | 0.267 | 0.558 | 1.162 | -

Table 5. Performance of Other Deep learning-based image captioning methods on
MSCOCO dataset and commonly used evaluation metrics.

2*Method |  2*Category | MSCOCO
---|---|---
|  | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | METEOR | ROUGE-L | CIDEr | SPICE
Lu et al. 2017 (Lu et al., 2017) | VS,SL,WS,EDA,AB | 0.742 | 0.580 | 0.439 | 0.332 | 0.266 | - | 1.085 | -
Gan et al. 2017 (Gan et al., 2017b) | VS,SL,WS,CA,SCB | 0.741 | 0.578 | 0.444 | 0.341 | 0.261 | - | 1.041 | -
Zhang et al. 2017 (Zhang et al., 2017) | VS,ODL,WS,EDA | - | - | - | 0.344 | 0.267 | 0.558 | 1.162 | -
Rennie et al. 2017 (Rennie et al., 2017) | VS,ODL,WS,EDA | - | - | - | .319 | 0.255 | 0.543 | 1.06 | -
Yao et al. 2017 (Yao et al., 2017b) | VS,SL,WS,EDA,SCB | 0.734 | 0.567 | 0.430 | 0.326 | 0.254 | 0.540 | 1.00 | 0.186
Gu et al. 2017 (Gu et al., 2017) | VS,SL,WS,EDA | 0.720 | 0.550 | 0.410 | 0.300 | 0.240 | - | 0.960 | 0.176

Table 6. Top two methods based on different evaluation metrics and MSCOCO
dataset (Bold and Italic indicates the best result; Bold indicates the second
best result).

We found that the performance of a technique can vary across different
metrics. Table 6 shows the methods based on the top two scores on every
individual evaluation metric. For example, Lu et al., Gan et al., and Zhang et
al. are within the top two methods based on the scores achieved on BLEU-n and
METEOR metrics. BLEU-n metrics use variable length phrases of generated
captions to match against ground-truth captions. METEOR (Banerjee and Lavie,
2005) considers the precision, recall, and the alignments of the matched
tokens. Therefore, the generated captions by these methods have good precision
and recall accuracy as well as the good similarity in word level. ROUGE-L
evaluates the adequacy and fluency of generated captions, whereas CIDEr
focuses on grammaticality and saliency. SPICE can analyse the semantics of the
generated captions. Zhang et al., Rennie et al., and Lu et al. can generate
captions, which have adequacy, fluency, saliency, and are grammaticality
correct than other methods in Table 6. Gu et al. and Yao et al. perform well
in generating semantically correct captions.

##  6\. Discussions and Future Research Directions

Many deep learning-based methods have been proposed for generating automatic
image captions in the recent years. Supervised learning, reinforcement
learning, and GAN based methods are commonly used in generating image
captions. Both visual space and multimodal space can be used in supervised
learning-based methods. The main difference between visual space and
multimodal space occurs in mapping. Visual space-based methods perform
explicit mapping from images to descriptions. In contrast, multimodal space-
based methods incorporate implicit vision and language models. Supervised
learning-based methods are further categorized into Encoder-Decoder
architecture-based, Compositional architecture-based, Attention-based,
Semantic concept-based, Stylized captions, Dense image captioning, and Novel
object-based image captioning.

Encoder-Decoder architecture-based methods use a simple CNN and a text
generator for generating image captions. Attention-based image captioning
methods focus on different salient parts of the image and achieve better
performance than encoder-decoder architecture-based methods. Semantic concept-
based image captioning methods selectively focus on different parts of the
image and can generate semantically rich captions. Dense image captioning
methods can generate region based image captions. Stylized image captions
express various emotions such as romance, pride, and shame. GAN and RL based
image captioning methods can generate diverse and multiple captions.

MSCOCO, Flickr30k and Flickr8k dataset are common and popular datasets used
for image captioning. MSCOCO dataset is very large dataset and all the images
in these datasets have multiple captions. Visual Genome dataset is mainly used
for region based image captioning. Different evaluation metrics are used for
measuring the performances of image captions. BLEU metric is good for small
sentence evaluation. ROUGE has different types and they can be used for
evaluating different types of texts. METEOR can perform an evaluation on
various segments of a caption. SPICE is better in understanding semantic
details of captions compared to other evaluation metrics.

Although success has been achieved in recent years, there is still a large
scope for improvement. Generation based methods can generate novel captions
for every image. However, these methods fail to detect prominent objects and
attributes and their relationships to some extent in generating accurate and
multiple captions. In addition to this, the accuracy of the generated captions
largely depends on syntactically correct and diverse captions which in turn
rely on powerful and sophisticated language generation model. Existing methods
show their performances on the datasets where images are collected from the
same domain. Therefore, working on open domain dataset will be an interesting
avenue for research in this area. Image-based factual descriptions are not
enough to generate high-quality captions. External knowledge can be added in
order to generate attractive image captions. Supervised learning needs a large
amount of labelled data for training. Therefore, unsupervised learning and
reinforcement learning will be more popular in future in image captioning.

##  7\. Conclusions

In this paper, we have reviewed deep learning-based image captioning methods.
We have given a taxonomy of image captioning techniques, shown generic block
diagram of the major groups and highlighted their pros and cons. We discussed
different evaluation metrics and datasets with their strengths and weaknesses.
A brief summary of experimental results is also given. We briefly outlined
potential research directions in this area. Although deep learning-based image
captioning methods have achieved a remarkable progress in recent years, a
robust image captioning method that is able to generate high quality captions
for nearly all images is yet to be achieved. With the advent of novel deep
learning network architectures, automatic image captioning will remain an
active research area for some time.

## Acknowledgements

This work was partially supported by an Australian Research Council grant
DE120102960.

## References

  * (1)
  * Agarwal and Lavie (2008) Abhaya Agarwal and Alon Lavie. 2008.  Meteor, m-bleu and m-ter: Evaluation metrics for high-correlation with human rankings of machine translation output. In _Proceedings of the Third Workshop on Statistical Machine Translation_. Association for Computational Linguistics, 115–118.
  * Aker and Gaizauskas (2010) Ahmet Aker and Robert Gaizauskas. 2010.  Generating image descriptions using dependency relational patterns. In _Proceedings of the 48th annual meeting of the association for computational linguistics_. Association for Computational Linguistics, 1250–1258.
  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016.  Spice: Semantic propositional image caption evaluation. In _European Conference on Computer Vision_. Springer, 382–398.
  * Anderson et al. (2017) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2017.  Bottom-up and top-down attention for image captioning and vqa.  _arXiv preprint arXiv:1707.07998_ (2017).
  * Aneja et al. (2018) Jyoti Aneja, Aditya Deshpande, and Alexander G Schwing. 2018\.  Convolutional image captioning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 5561–5570.
  * Anne Hendricks et al. (2016) Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, Trevor Darrell, Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, et al. 2016\.  Deep compositional captioning: Describing novel object categories without paired training data. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015\.  Neural machine translation by jointly learning to align and translate. In _International Conference on Learning Representations (ICLR)_.
  * Bai and An (2018) Shuang Bai and Shan An. 2018.  A Survey on Automatic Image Caption Generation.  _Neurocomputing_.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , Vol. 29. 65–72.
  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled sampling for sequence prediction with recurrent neural networks. In _Advances in Neural Information Processing Systems_. 1171–1179.
  * Bengio et al. (2003) Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003.  A neural probabilistic language model.  _Journal of machine learning research_ 3, Feb, 1137–1155.
  * Berger et al. (1996) Adam L Berger, Vincent J Della Pietra, and Stephen A Della Pietra. 1996\.  A maximum entropy approach to natural language processing.  _Computational linguistics_ 22, 1 (1996), 39–71.
  * Bernardi et al. (2016) Raffaella Bernardi, Ruket Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, Barbara Plank, et al. 2016\.  Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures.  _Journal of Artificial Intelligence Research (JAIR)_ 55, 409–442.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003\.  Latent dirichlet allocation.  _Journal of machine Learning research_ 3, Jan (2003), 993–1022.
  * Bodnar (2018) Cristian Bodnar. 2018\.  Text to Image Synthesis Using Generative Adversarial Networks.  _arXiv preprint arXiv:1805.00676_.
  * Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008.  Freebase: a collaboratively created graph database for structuring human knowledge. In _Proceedings of the 2008 ACM SIGMOD international conference on Management of data_. AcM, 1247–1250.
  * Boser et al. (1992) Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. 1992.  A training algorithm for optimal margin classifiers. In _Proceedings of the fifth annual workshop on Computational learning theory_. ACM, 144–152.
  * Bulling et al. (2011) Andreas Bulling, Jamie A Ward, Hans Gellersen, and Gerhard Troster. 2011.  Eye movement analysis for activity recognition using electrooculography.  _IEEE transactions on pattern analysis and machine intelligence_ 33, 4 (2011), 741–753.
  * Bychkovsky et al. (2011) Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. 2011.  Learning photographic global tonal adjustment with a database of input/output image pairs. In _Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on_. IEEE, 97–104.
  * Callison-Burch et al. (2006) Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006\.  Re-evaluation the Role of Bleu in Machine Translation Research.. In _EACL_ , Vol. 6. 249–256.
  * Chen et al. (2017b) Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, and Tat-Seng Chua. 2017b.  SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning. In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_. 6298–6306.
  * Chen et al. (2017a) Tseng-Hung Chen, Yuan-Hong Liao, Ching-Yao Chuang, Wan-Ting Hsu, Jianlong Fu, and Min Sun. 2017a.  Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner. In _The IEEE International Conference on Computer Vision (ICCV)_ , Vol. 2.
  * Chen and Lawrence Zitnick (2015) Xinlei Chen and C Lawrence Zitnick. 2015.  Mind’s eye: A recurrent visual representation for image caption generation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2422–2431.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014.  On the properties of neural machine translation: Encoder-decoder approaches. In _Association for Computational Linguistics_. 103–111.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_.
  * Dai et al. (2017) Bo Dai, Dahua Lin, Raquel Urtasun, and Sanja Fidler. 2017\.  Towards Diverse and Natural Image Descriptions via a Conditional GAN. In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_. 2989–2998.
  * Dalal and Triggs (2005) Navneet Dalal and Bill Triggs. 2005.  Histograms of oriented gradients for human detection. In _Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on_ , Vol. 1. IEEE, 886–893.
  * De Marneffe et al. (2006) Marie-Catherine De Marneffe, Bill MacCartney, and Christopher D Manning. 2006.  Generating typed dependency parses from phrase structure parses. In _Proceedings of LREC_ , Vol. 6. Genoa Italy, 449–454.
  * Denoual and Lepage (2005) Etienne Denoual and Yves Lepage. 2005.  BLEU in characters: towards automatic MT evaluation in languages without word delimiters. In _Companion Volume to the Proceedings of the Second International Joint Conference on Natural Language Processing_. 81–86.
  * Devlin et al. (2015) Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, and Margaret Mitchell. 2015\.  Language models for image captioning: The quirks and what works.  _arXiv preprint arXiv:1505.01809_ (2015).
  * Donahue et al. (2015) Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015\.  Long-term recurrent convolutional networks for visual recognition and description. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2625–2634.
  * Elliott and Keller (2013) Desmond Elliott and Frank Keller. 2013.  Image description using visual dependency representations. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. 1292–1302.
  * Fang et al. (2015) Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, and John C Platt. 2015\.  From captions to visual concepts and back. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1473–1482.
  * Farhadi et al. (2010) Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010\.  Every picture tells a story: Generating sentences from images. In _European conference on computer vision_. Springer, 15–29.
  * Fathi et al. (2012) Alireza Fathi, Yin Li, and James M Rehg. 2012.  Learning to recognize daily actions using gaze. In _European Conference on Computer Vision_. Springer, 314–327.
  * Fedus et al. (2018) William Fedus, Ian Goodfellow, and Andrew M Dai. 2018\.  Maskgan: Better text generation via filling in the _.  _arXiv preprint arXiv:1801.07736_.
  * FitzGerald et al. (2013) Nicholas FitzGerald, Yoav Artzi, and Luke Zettlemoyer. 2013\.  Learning distributions over logical forms for referring expression generation. In _Proceedings of the 2013 conference on empirical methods in natural language processing_. 1914–1925.
  * Frome et al. (2013) Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, and Tomas Mikolov. 2013\.  Devise: A deep visual-semantic embedding model. In _Advances in neural information processing systems_. 2121–2129.
  * Gan et al. (2017a) Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. 2017a.  Stylenet: Generating attractive visual captions with styles. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3137–3146.
  * Gan et al. (2016) Chuang Gan, Tianbao Yang, and Boqing Gong. 2016.  Learning attributes equals multi-source domain generalization. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 87–97.
  * Gan et al. (2017b) Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Li Deng. 2017b.  Semantic compositional networks for visual captioning. In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_. 1141–1150.
  * Gehring et al. (2016) Jonas Gehring, Michael Auli, David Grangier, and Yann N Dauphin. 2016.  A convolutional encoder model for neural machine translation.  _arXiv preprint arXiv:1611.02344_.
  * Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017.  Convolutional sequence to sequence learning.  _arXiv preprint arXiv:1705.03122_.
  * Girshick (2015) Ross Girshick. 2015\.  Fast r-cnn. In _Proceedings of the IEEE international conference on computer vision_. 1440–1448.
  * Girshick et al. (2014) Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014.  Rich feature hierarchies for accurate object detection and semantic segmentation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 580–587.
  * Golland et al. (2010) Dave Golland, Percy Liang, and Dan Klein. 2010.  A game-theoretic approach to generating spatial descriptions. In _Proceedings of the 2010 conference on empirical methods in natural language processing_. Association for Computational Linguistics, 410–419.
  * Gong et al. (2014) Yunchao Gong, Liwei Wang, Micah Hodosh, Julia Hockenmaier, and Svetlana Lazebnik. 2014.  Improving image-sentence embeddings using large weakly annotated photo collections. In _European Conference on Computer Vision_. Springer, 529–545.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial nets. In _Advances in neural information processing systems_. 2672–2680.
  * Gregor et al. (2015) Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. 2015\.  DRAW: A recurrent neural network for image generation. In _Proceedings of Machine Learning Research_. 1462–1471.
  * Grubinger et al. (2006) Michael Grubinger, Paul Clough, Henning Müller, and Thomas Deselaers. 2006.  The iapr tc-12 benchmark: A new evaluation resource for visual information systems. In _International workshop ontoImage_ , Vol. 5. 10.
  * Gu et al. (2017) Jiuxiang Gu, Gang Wang, Jianfei Cai, and Tsuhan Chen. 2017\.  An empirical study of language cnn for image captioning. In _Proceedings of the International Conference on Computer Vision (ICCV)_. 1231–1240.
  * Han and Li (2015) Yahong Han and Guang Li. 2015.  Describing images with hierarchical concepts and object class localization. In _Proceedings of the 5th ACM on International Conference on Multimedia Retrieval_. ACM, 251–258.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Hodosh et al. (2013) Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013\.  Framing image description as a ranking task: Data, models and evaluation metrics.  _Journal of Artificial Intelligence Research_ 47 (2013), 853–899.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017.  Image-to-image translation with conditional adversarial networks. In _Proceedings of the IEEE International Conference on Computer Vision (CVPR)_. 5967–5976.
  * Jaderberg et al. (2015) Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. 2015\.  Spatial transformer networks. In _Advances in Neural Information Processing Systems_. 2017–2025.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017.  Categorical reparameterization with gumbel-softmax. In _International Conference on Learning Representations (ICLR)_.
  * Jia et al. (2015) Xu Jia, Efstratios Gavves, Basura Fernando, and Tinne Tuytelaars. 2015.  Guiding the long-short term memory model for image caption generation. In _Proceedings of the IEEE International Conference on Computer Vision_. 2407–2415.
  * Jiang et al. (2015) Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. 2015\.  Salicon: Saliency in context. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1072–1080.
  * Jin et al. (2015) Junqi Jin, Kun Fu, Runpeng Cui, Fei Sha, and Changshui Zhang. 2015.  Aligning where to see and what to tell: image caption with region-based attention and scene factorization.  _arXiv preprint arXiv:1506.06272_.
  * Johnson et al. (2016) Justin Johnson, Andrej Karpathy, and Li Fei-Fei. 2016\.  Densecap: Fully convolutional localization networks for dense captioning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4565–4574.
  * Johnson et al. (2015) Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. 2015.  Image retrieval using scene graphs. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3668–3678.
  * Jozefowicz et al. (2016) Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016.  Exploring the limits of language modeling.  _arXiv preprint arXiv:1602.02410_ (2016).
  * Karpathy and Fei-Fei (2015) Andrej Karpathy and Li Fei-Fei. 2015.  Deep visual-semantic alignments for generating image descriptions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3128–3137.
  * Karpathy et al. (2014) Andrej Karpathy, Armand Joulin, and Fei Fei F Li. 2014\.  Deep fragment embeddings for bidirectional image sentence mapping. In _Advances in neural information processing systems_. 1889–1897.
  * Karthikeyan et al. (2013) S Karthikeyan, Vignesh Jagadeesh, Renuka Shenoy, Miguel Ecksteinz, and BS Manjunath. 2013\.  From where and how to what we see. In _Proceedings of the IEEE International Conference on Computer Vision_. 625–632.
  * Kazemzadeh et al. (2014) Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L Berg. 2014.  ReferItGame: Referring to Objects in Photographs of Natural Scenes.. In _EMNLP_. 787–798.
  * Kiros et al. (2014a) Ryan Kiros, Ruslan Salakhutdinov, and Rich Zemel. 2014a.  Multimodal neural language models. In _Proceedings of the 31st International Conference on Machine Learning (ICML-14)_. 595–603.
  * Kiros et al. (2014b) Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. 2014b.  Unifying visual-semantic embeddings with multimodal neural language models. In _Workshop on Neural Information Processing Systems (NIPS))_.
  * Konda and Tsitsiklis (2000) Vijay R Konda and John N Tsitsiklis. 2000.  Actor-critic algorithms. In _Advances in neural information processing systems_. 1008–1014.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017\.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International Journal of Computer Vision_ 123, 1 (2017), 32–73.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks. In _Advances in neural information processing systems_. 1097–1105.
  * Kulkarni et al. (2011) Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. 2011.  Baby talk: Understanding and generating image descriptions. In _Proceedings of the 24th CVPR_. Citeseer.
  * Kumar and Goel (2017) Akshi Kumar and Shivali Goel. 2017.  A survey of evolution of image captioning techniques.  _International Journal of Hybrid Intelligent Systems_ Preprint, 1–19.
  * Kuznetsova et al. (2012) Polina Kuznetsova, Vicente Ordonez, Alexander C Berg, Tamara L Berg, and Yejin Choi. 2012.  Collective generation of natural image descriptions. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1_. Association for Computational Linguistics, 359–368.
  * Kuznetsova et al. (2014) Polina Kuznetsova, Vicente Ordonez, Tamara L Berg, and Yejin Choi. 2014.  TREETALK: Composition and Compression of Trees for Image Descriptions.  _TACL_ 2, 10 (2014), 351–362.
  * Lebret et al. (2015) Rémi Lebret, Pedro O Pinheiro, and Ronan Collobert. 2015\.  Simple image description generator via a linear phrase-based approach.  _Workshop on International Conference on Learning Representations (ICLR)_.
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998\.  Gradient-based learning applied to document recognition.  _Proc. IEEE_ 86, 11 (1998), 2278–2324.
  * Li et al. (2011) Siming Li, Girish Kulkarni, Tamara L Berg, Alexander C Berg, and Yejin Choi. 2011.  Composing simple image descriptions using web-scale n-grams. In _Proceedings of the Fifteenth Conference on Computational Natural Language Learning_. Association for Computational Linguistics, 220–228.
  * Lin (2004) Chin-Yew Lin. 2004\.  Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out: Proceedings of the ACL-04 workshop_ , Vol. 8. Barcelona, Spain.
  * Lin and Och (2004) Chin-Yew Lin and Franz Josef Och. 2004.  Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In _Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics_. Association for Computational Linguistics, 605.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014\.  Microsoft coco: Common objects in context. In _European conference on computer vision_. Springer, 740–755.
  * Liu et al. (2017a) Chenxi Liu, Junhua Mao, Fei Sha, and Alan L Yuille. 2017a.  Attention Correctness in Neural Image Captioning. In _AAAI_. 4176–4182.
  * Liu et al. (2017b) Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. 2017b.  Improved image captioning via policy gradient optimization of spider. In _Proceedings of the IEEE Internationa Conference on Computer Vision (ICCV)_ , Vol. 3. 873–881.
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015\.  Fully convolutional networks for semantic segmentation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3431–3440.
  * Lowe (2004) David G Lowe. 2004\.  Distinctive image features from scale-invariant keypoints.  _International journal of computer vision_ 60, 2 (2004), 91–110.
  * Lu et al. (2017) Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017\.  Knowing when to look: Adaptive attention via A visual sentinel for image captioning. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 3242–3250.
  * Luong et al. (2016) Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016.  Multi-task sequence to sequence learning. In _International Conference on Learning Representations (ICLR)_.
  * Ma and Han (2016) Shubo Ma and Yahong Han. 2016.  Describing images by feeding LSTM with structural words. In _Multimedia and Expo (ICME), 2016 IEEE International Conference on_. IEEE, 1–6.
  * Maddison et al. (2017) Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2017\.  The concrete distribution: A continuous relaxation of discrete random variables. In _International Conference on Learning Representations (ICLR)_.
  * Mao et al. (2016) Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016\.  Generation and comprehension of unambiguous object descriptions. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 11–20.
  * Mao et al. (2015a) Junhua Mao, Xu Wei, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan L Yuille. 2015a.  Learning like a child: Fast novel visual concept learning from sentence descriptions of images. In _Proceedings of the IEEE International Conference on Computer Vision_. 2533–2541.
  * Mao et al. (2015b) Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. 2015b.  Deep captioning with multimodal recurrent neural networks (m-rnn). In _International Conference on Learning Representations (ICLR)_.
  * Mao et al. (2014) Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan L Yuille. 2014.  Explain images with multimodal recurrent neural networks.  _arXiv preprint arXiv:1410.1090_ (2014).
  * Maron and Lozano-Pérez (1998) Oded Maron and Tomás Lozano-Pérez. 1998.  A framework for multiple-instance learning. In _Advances in neural information processing systems_. 570–576.
  * Mathews et al. (2016) Alexander Patrick Mathews, Lexing Xie, and Xuming He. 2016\.  SentiCap: Generating Image Descriptions with Sentiments.. In _AAAI_. 3574–3580.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013\.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mikolov et al. (2010) Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. 2010.  Recurrent neural network based language model. In _Eleventh Annual Conference of the International Speech Communication Association_.
  * Mishra et al. (2012) Ajay K Mishra, Yiannis Aloimonos, Loong Fah Cheong, and Ashraf Kassim. 2012.  Active visual segmentation.  _IEEE transactions on pattern analysis and machine intelligence_ 34, 4 (2012), 639–653.
  * Mitchell et al. (2012) Margaret Mitchell, Xufeng Han, Jesse Dodge, Alyssa Mensch, Amit Goyal, Alex Berg, Kota Yamaguchi, Tamara Berg, Karl Stratos, and Hal Daumé III. 2012\.  Midge: Generating image descriptions from computer vision detections. In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_. Association for Computational Linguistics, 747–756.
  * Mitchell et al. (2010) Margaret Mitchell, Kees van Deemter, and Ehud Reiter. 2010\.  Natural reference to objects in a visual domain. In _Proceedings of the 6th international natural language generation conference_. Association for Computational Linguistics, 95–104.
  * Mitchell et al. (2013) Margaret Mitchell, Kees Van Deemter, and Ehud Reiter. 2013\.  Generating Expressions that Refer to Visible Objects.. In _HLT-NAACL_. 1174–1184.
  * Mnih and Hinton (2007a) Andriy Mnih and Geoffrey Hinton. 2007a.  Three new graphical models for statistical language modelling. In _Proceedings of the 24th international conference on Machine learning_. ACM, 641–648.