# PixLore: A Dataset-driven Approach to Rich Image Captioning

Diego Bonilla Salvador
Cognizant
Valencia, Spain
<EMAIL_ADDRESS>
Under review as a preprint. Do not distribute

###### Abstract

In the domain of vision-language integration, generating detailed image
captions poses a significant challenge due to the lack of a curated and rich
dataset. This study introduces PixLore, a novel method that leverages Querying
Transformers through the fine-tuning of the BLIP-2 model using the LoRa method
on a standard commercial GPU. Our approach, which involves training on a
carefully assembled dataset from state-of-the-art Computer Vision models
combined and augmented by ChatGPT, addresses the question of whether intricate
image understanding can be achieved with an ensemble of smaller-scale models.
Comparative evaluations against major models such as GPT-4 and Google Bard
demonstrate that PixLore-2.7B, despite having considerably fewer parameters,
is rated higher than the existing State-of-the-Art models in over half of the
assessments. This research not only presents a groundbreaking approach but
also highlights the importance of well-curated datasets in enhancing the
performance of smaller models. More details, fine-tuned model and dataset can
be found on our project page: https://diegobonilla98.github.io/PixLore/.

_K_ eywords Computer Vision $\cdot$ Image Captioning $\cdot$ Large Language
Models $\cdot$ ChatGPT $\cdot$ GPT-4

|  |  |
---|---|---|---
|  |  |
BLIP-2 Large [1] |  there is a stop sign on the corner of a road |  a painting of a square with many different colors and shapes |  flames are lit in the dark near a fire pit
Google Bard [2] |  The image shows a red stop sign and a blue U-turn sign on a pole at a rural intersection. The stop sign is in the foreground and the U-turn sign is behind it. The signs are clearly visible and easy to identify. The image conveys the message that drivers must stop at the intersection and that U-turns are allowed. |  This abstract painting features colorful circles arranged in squares. The bright, saturated colors and dynamic composition create a sense of joy and optimism. The painting is reminiscent of the work of Wassily Kandinsky, a pioneer of abstract art. |  This close-up of a campfire at night captures the warmth and beauty of a gathering place. The flames reach up to the sky, casting a warm glow on the surrounding logs and rocks. The embers glow brightly, and the smoke rises in a thin wisp. The background is dark, with only a few stars visible. The image evokes the feeling of sitting around a campfire at night, surrounded by friends and family. It is a warm and inviting place to share stories and enjoy each other’s company.
GPT-4 [3] |  This image shows a traffic sign that indicates a stop and a roundabout ahead. The sign is located on the side of a road with a pavement and some bushes. The sky is blue with some clouds and there are some buildings in the background. The image is realistic and has a lot of details. |  This is an image of a colorful abstract painting that shows the artist’s creativity and expression. The painting consists of 16 squares, each with a different color and pattern. The squares are arranged in a 4x4 grid, creating a contrast and harmony among them. The colors used are orange, red, blue, green, purple, and yellow, which are bright and vivid. The patterns are concentric circles, ovals, and rectangles, which are simple and geometric. The background is a light blue color, which creates a sense of space and calmness. The painting is done in a loose, brushy style, which adds texture and movement to the image. |  The image shows a cozy campfire in the middle of a dark night. The fire is crackling and glowing with warm colors, creating a contrast with the blackness of the surroundings. The fire is built on a pile of rocks and logs, providing fuel and stability. The image also has a blurred square in the bottom right corner, which might be hiding some people who are enjoying the fire. The image captures the atmosphere of a relaxing and adventurous night outdoors.
PixLore (ours) |  A photo of a road with a stop sign on the side. The stop sign is positioned on the side of the road, indicating that it is a designated intersection. The sky above is cloudy, creating a gloomy atmosphere. The road is surrounded by a fence, and there is a curb separating the road from the adjacent area. |  A colorful painting depicting a variety of circles and squares is displayed on a wall. The painting is vibrant and eye-catching, with a variety of colors and patterns. The circles and squares create a geometric pattern, adding a touch of creativity to the scene. The painting is a work of art, showcasing the artist’s skill and creativity. The wall is covered in a soft, smooth surface, creating a comfortable and inviting atmosphere. The painting is a perfect addition to any room, adding a pop of color and creativity to the space. |  A cozy campfire is lit up in the night, casting a warm glow on the surrounding area. The fire pit is filled with logs, creating a cozy and inviting atmosphere. The flames are burning brightly, adding to the ambiance of the scene. It seems like a perfect night for a campfire, with the smell of wood and the crackling of the logs filling the air.
|  |  |
Table 1: Comparison of various state-of-the-art image captioning models versus
PixLore (ours)

##  1 Introduction

Image captioning, defined as the automatic generation of textual descriptions
for images, stands in the middle of computer vision and natural language
processing. Its relevance is not confined to academic exploration; it has
tangible applications such as assisting visually impaired individuals in
interpreting visual content, improving search engine capabilities, and
enhancing content discoverability on digital platforms. While there have been
notable strides in this field, the pursuit of models capable of producing
human-like, contextually appropriate, and detailed captions continues.

Early attempts in image captioning primarily relied on simple template-based
methods [4], where captions were generated by filling in predefined templates
with object, action, and attribute labels detected in the image. However,
these methods lacked flexibility and often failed to capture the richness and
diversity of natural language.

With the advent of deep learning, significant advancements were made. The
introduction of the encoder-decoder framework [5], often realized through
combinations of Convolutional Neural Networks (CNNs) for image encoding and
Recurrent Neural Networks (RNNs) for language decoding, marked a pivotal
shift. This approach enabled the generation of more fluid and varied captions
but often struggled with long-term dependencies and contextual nuances.

The breakthrough came with the integration of attention mechanisms [6],
allowing models to focus on relevant parts of the image while generating each
word in the caption. This led to more accurate and context-aware captions.
Subsequent innovations, such as the adoption of Transformer architectures [7],
further propelled the field forward, offering even greater efficiency and
effectiveness in capturing complex inter-modal relationships.

In the current landscape, the fusion of vision and language models has reached
impressive levels of sophistication. Models like GPT-4 [3], with rumored to
have trillions of parameters, set high benchmarks but raise the question: "Can
similar image understanding be achieved with more accessible, smaller-scale
models?"

The current state-of-the-art, BLIP-2 [8], adopts a distinctive two-stage pre-
training strategy, harnessing the capabilities of both frozen image encoders
and expansive language models. Due to its pioneering architecture and training
approach, BLIP-2 has set new standards in various vision-language tasks in
terms of efficiency and performance. Motivated by these attributes, we
selected BLIP-2 as the foundational model for our research. Utilizing the LoRa
[9] method—a fine-tuning technique that retains the pre-trained model weights
while introducing trainable rank decomposition matrices into each layer of the
Transformer architecture—we fine-tuned [percent] of BLIP-2’s parameters using
our novel dataset. Both the model and the dataset will be released to the
public. Remarkably, this fine-tuning was achieved on a commercially available
GPU, the NVIDIA GeForce RTX 3090Ti.

The main point of this research lies in the introduction of a unique dataset
of 100,000 images, randomly selected from the COCO dataset. These images were
processed using an array of leading computer vision models, encompassing tasks
such as object detection, image captioning, and image tagging. The resulting
outputs of these models were converted into a textual string and then
synthesized using a prompt-tuned ChatGPT, yielding comprehensive textual
descriptions that encapsulate the images’ essence and context with impressive
precision. This paper delves into the details of this dataset’s creation,
methodology, and evaluation, underscoring its potential impact on the image
captioning domain.

Our fine-tuned model is named PixLore: A Dataset-driven Approach to Rich Image
Captioning. The key contributions of this paper are:

  * •

An innovative method of amalgamating information from diverse tasks, akin to
indirect Knowledge Distillation.

  * •

A meticulously curated captioning dataset with 100,000 COCO image examples,
each accompanied by a detailed descriptive paragraph.

  * •

A model that leverages with huge multimodal models, such as GPT-4 and Google
Bard.

In the next section we’ll delve into a study of related work and explain the
creation of our dataset and the fine-tuning of the model.

###  1.1 Related Work

Image captioning, the task of generating natural language descriptions for
visual content, has been a focal point of research in the intersection of
computer vision and natural language processing. Historically, the field has
witnessed several advancements, from rule-based systems to the current state-
of-the-art deep learning techniques. This section delves into the evolution of
methodologies and datasets in image captioning, setting the stage for our
novel approach.

####  1.1.1 Evolution of Image Captioning Datasets

Datasets play a pivotal role in the development and evaluation of image
captioning models. Some datasets that are used by recent image captioning
models are:

  * •

Microsoft COCO [10]: 328.000 images of complex everyday scenes containing
common objects in their natural context.

  * •

Flickr30K [11]: 31.000 images collected from Flickr, together with 5 reference
sentences provided by human annotators.

  * •

TextCaps [12]: 28.000 images with 145.000 captions. Specifically focused on
images with text on it.

  * •

nocaps [13]: 15.100 images with 166.100 captions from the Open Images
validation and test sets. The associated training data consists of COCO image-
caption pairs, plus Open Images image level labels and object bounding boxes.

However, many of these datasets, while extensive, often lack in the richness
and diversity of captions. The emergence of large multimodal language models
trained with huge amounts of data has highlighted the potential of generating
detailed and contextually rich captions. These models possess the capability
to generate intricate descriptions from images, surpassing the richness
offered by traditional datasets.

####  1.1.2 Leveraging Complex Capabilities in Image Captioning

Image captioning has recently been influenced by the same trends observed in
visual question-answering. The focus is on improving the understanding of
vision-language models by integrating them with advanced large language
models. Models such as Google Bard and GPT-4 have displayed impressive
abilities, including generating websites from handwritten text and recognizing
comedic elements in images. These capabilities were not commonly found in
earlier vision-language models. Another model, MINIGPT-4 [14], uses the pre-
trained vision components of BLIP-2. It introduces a single projection layer
to align the encoded visual features with a large language model, while
keeping other vision and language components unchanged.

The Caption Anything [15] model offers a unique approach to image captioning
by using the Segment Anything Model (SAM) [16] and ChatGPT [17]. Users can
provide captions for specific parts of an image, which are segmented
automatically using SAM. Additionally, the model can segment all objects in an
image, assign individual captions, and then use ChatGPT to combine these
captions into one cohesive and rich description. This method uses frozen
models without any further training, which differs from our approach.
Moreover, its dependence on large models and frequent API calls can make it
less practical for some applications.

The GPT4Image [18] model also aims to improve image models. It uses insights
from large pre-trained models to help models like CNN and ViT produce better
visual representations. This is done by creating a high-quality description
set using a multimodal Large Language Model (LLM). The extracted text
embeddings act as additional guidance and are aligned with image
representations from vision models. This process improves the performance of
standard vision models for tasks like image classification.

Our method utilizes the capabilities of large language models to produce
detailed textual descriptions. By processing the outputs from various computer
vision models through ChatGPT, we can generate captions that are detailed and
have stylistic elements similar to models like Google’s Bard and GPT-4. This
approach leads to a unique dataset filled with rich captions that are both
informative and engaging.

Figure 1: Each image goes through a variety of Computer Vision models that aid
ChatGPT in the creation of very rich captions

##  2 Dataset Used

###  2.1 Dataset Creation

The creation of a robust and diverse dataset is pivotal for the success of any
machine learning model, especially in the domain of image captioning. This
section elucidates the meticulous process undertaken to curate our unique
dataset, which subsequently played a cardinal role in enhancing the
performance of our image captioning engine.

####  2.1.1 Image Selection

The initial step in our dataset creation involved the selection of images. For
this purpose, the Common Objects in Context (COCO) dataset was employed. COCO,
a renowned dataset in the computer vision community, offers a rich variety of
images spanning multiple contexts and scenarios, making it an ideal choice for
our endeavor. From this vast repository, a total of 100,000 images were
randomly selected. This random selection ensured a broad representation,
capturing the diversity and richness inherent in the COCO dataset.

####  2.1.2 Model-based Image Processing

The dataset creation process involved a systematic approach wherein each
image, sourced from the selected collection, was subjected to analysis using a
series of advanced computer vision models. These models, each designed with
specific functionalities, provided a range of interpretations and annotations
for the images, ensuring a comprehensive understanding of the visual content.

The models employed in this analytical process are as follows:

  * •

DETR [19]: State-of-the-art object detection model based on the Transformer
architecture. When processing an image, DETR identifies objects within the
visual frame and provides annotations. These annotations include categorical
labels that describe the objects and spatial coordinates that indicate their
positions within the image.

  * •

Recognize Anything [20]: This image tagging model has the capability to
identify and tag a wide range of common categories within an image. The tags
generated by this model serve as descriptors, offering a summarized
representation of the image’s content.

  * •

OwLViT [21]: OwLViT operates as an open-vocabulary object detection model. It
takes a unique approach by using the outputs from the "Recognize Anything"
model as its classes. For each tag produced by "Recognize Anything", OwLViT
detects its presence and provides both its location and class name. This
method complements the information provided by DETR, adding depth to the
scene’s understanding.

  * •

Tag2Text [22]: Building upon the tags from "Recognize Anything", Tag2Text
crafts captions that describe the image. This model’s output provides a
narrative that goes beyond simple object identification, offering a more
holistic view of the image’s content.

  * •

BLIP-2: BLIP-2 is an image captioning model that, despite its reduced number
of trainable parameters compared to some other models, has shown proficiency
in its task. It provides detailed captions that describe the visual content of
images.

To consolidate the diverse outputs from these models, each was transformed
into a text string. This string captures the essence of the information
derived from each model’s output. It’s worth noting that in this framework,
ChatGPT interacts exclusively with these text strings, as it doesn’t possess
the capability to process images directly, thus becoming a kind of "imageless"
image captioning. The final prompt, which integrates the data from all the
models and is then presented to ChatGPT, can be viewed in Figure 2.

⬇

# INPUTS

## IMAGE TAGS

’back’, ’blanket’, ’cat’, ’control’, ’couch’, ’couple’, ’curl’, ’pad’, ’lay’,
’pillow’, ’pink’, ’relax’, ’remote’, ’sleep’, ’stretch’, ’tabby’

## OBJECTS MODEL 1

cat at [344.06, 24.85, 640.34, 373.74]

remote at [328.13, 75.93, 372.81, 187.66]

remote at [39.34, 70.13, 175.56, 118.78]

cat at [15.36, 51.75, 316.89, 471.16]

couch at [-0.19, 0.71, 639.73, 474.17]

##OBJECTS MODEL 2

couch at [0.51, 1.19, 637.86, 479.38]

remote at [39.66, 72.18, 175.3, 117.2]

remote at [334.52, 76.81, 369.98, 182.76]

cat at [335.98, 14.93, 637.93, 373.28]

cat at [8.61, 54.8, 315.07, 479.78]

## SHORT CAPTION

Caption A: two cats laying on a couch with remote controls on the back

Caption B: a couple of cats on a couch with a remote

Figure 2: Joined prompt for ChatGPT from all the models mentioned. Each model
corresponds with a ##SECTION_NAME output converted to plain text. The image on
the right is the one used in the example. For reference, the output of ChatGPT
for this example is: Two adorable cats are peacefully lounging on a
comfortable couch, one on the left and the other on the right. They appear to
be curled up and enjoying a restful moment. On the couch’s backrest, you can
see two remote controls neatly placed. The cats have distinctive tabby fur
patterns, and their soft pink blankets are spread out beneath them. It seems
like they are in complete control of their relaxation, creating a serene and
cozy atmosphere.

###  2.2 Fine-tuning

The process of fine-tuning the BLIP-2 model with our novel dataset was a
crucial step in achieving the results presented in this paper. This section
delves into the specifics of how the model was adapted to our dataset,
leveraging the strengths of both the BLIP-2 architecture and the LoRa method.

The dataset underwent preprocessing consistent with the methods employed in
the original BLIP-2 paper [8]. This ensured that the data was in an optimal
format and structure, ready to be ingested by the BLIP-2 model for fine-
tuning.

For the fine-tuning process, the specific configuration of BLIP-2 used was
BLIP-2 2.7B in 16-bit half-precision sourced from HuggingFace [23]. This
configuration was chosen for its balance between computational efficiency and
performance capabilities.

The model was trained over a span of 5 epochs. This duration was chosen to
prevent potential overfitting to the dataset. Given hardware constraints, a
modest batch size of 3 was employed during training. The AdamW optimizer [24]
was utilized with a learning rate set to 5e-4. This low learning rate was
specifically chosen to prevent catastrophic forgetting, ensuring that the
model retained its pre-trained knowledge while adapting to the new dataset.

The LoRa method was seamlessly integrated into the fine-tuning process using
the peft [25] library of adapters. The model, being sourced from HuggingFace,
was loaded using the transformers [26] library, making the integration
straightforward. Specific hyperparameters associated with the LoRa method were
adjusted to best suit our dataset:

  * •

Rank of the update matrices (r): 16

  * •

Scaling factor (lora_alpha): 16

  * •

Dropout (lora_dropout): 0.1

  * •

Bias training (bias): "all"

  * •

LoRa application targets: queries and keys (target_modules)

The only modification made to the BLIP-2 model configuration was extending the
max_length parameter in the tokenizer to 256. This adjustment was necessary to
accommodate the richer text descriptions generated from our dataset.

##  3 Evaluation

For the evaluation of our model, PixLore, we selected a new set of images that
were not part of the training dataset. These images were randomly chosen from
the COCO dataset, ensuring that they were distinct from the 100k images used
during the training phase. Captions for these evaluation images were generated
using our fine-tuned BLIP-2 model, with default parameters set at a
temperature of 1, top_k of 50, and top_p of 1.

The evaluation was conducted with the help of human evaluators, primarily from
the general public, with a significant portion being Data Scientist fellows.
They evaluated more than 100 images. This mix ensured a balance between layman
perspectives and expert insights. Evaluators were given specific guidelines to
follow during the evaluation process. They were instructed to assess each
caption based on various criteria, including composition, colors and mood,
main subjects, background and foreground, lighting and perspective, size and
context, and specific features. The goal was to score each given caption with
0 to 5 points. More information about this survey can be found on Appendix A.

Each evaluator was presented with an image accompanied by five distinct
captions. These captions were generated by PixLore (two versions), Google’s
Bard, OpenAI’s GPT-4, and BLIP-2-Large. To ensure unbiased evaluations, this
was conducted as a blind test, with evaluators unaware of which model
generated which caption. Each evaluator was tasked with assessing 3 randomly
chosen images from a pool of 50 evaluation images.

Captions from Google’s Bard and OpenAI’s GPT-4 were generated using their
respective official websites. Each model was provided with an image and the
prompt: "Describe this image in a single paragraph." This ensured a consistent
approach to caption generation across all models.

The results from the human evaluation were analyzed by calculating the
percentage of times our model’s caption was preferred over the other models.
This straightforward metric provided a clear indication of PixLore’s
performance in comparison to industry benchmarks.

Our model, PixLore, outperforms GPT-4, Bard, and BLIP in 26.14%, 69.32%, and
61.36% of cases, respectively. This indicates a significant preference for our
model among users when compared to these well-known alternatives. The Figure 3
visually represents the human evaluation scores out of 5.

Figure 3: Diagram of human scores (out of 5) on PixLore

Human evaluation results show that our model Pixlore scores only about 12.80%
lower than GPT-4 despite having approximately 0.16% of GPT-4’s parameters.
Additionally, Pixlore outperforms Bard and BLIP, which score approximately
35.18% and 27.98% lower than Pixlore, respectively, even though Bard has about
50 times more parameters as Pixlore and Blip being the baseline of our model.

##  4 Conclusion and Future Work

Image captioning models have recently seen a trend towards increasing model
sizes, with parameter counts reaching into the billions and even trillions.
This paper presents a model that, despite having a significantly smaller
parameter count, achieves competitive performance. This achievement
underscores the potential of targeted fine-tuning and the importance of a
well-curated dataset.

The model’s performance, when compared to models like Bard’s Palm-2 with 137
billion parameters and the speculated 1.7 trillion parameters of GPT-4,
highlights the efficiency of the approach taken in this work. The reduced
parameter count not only ensures computational efficiency but also
demonstrates that state-of-the-art results can be achieved without the need
for extremely large models.

The integration of outputs from various computer vision models using ChatGPT
offers a novel approach to image captioning. Instead of relying on traditional
deterministic rules or simple aggregation methods, this work leverages
ChatGPT’s capability to understand tasks and the inputs provided. This method
effectively acts as an indirect form of knowledge distillation. Rather than
directly extracting knowledge from teacher models, the model assimilates
information from various state-of-the-art computer vision models, effectively
transforming its capabilities in image captioning.

The detailed descriptions generated by the model hint at the latent potential
of its image encoder. While the specific applications for scene understanding
have not been delineated in this work, the encoder’s ability to generate
detailed and nuanced descriptions suggests its potential utility in
applications requiring detailed scene interpretations, such as real-time scene
analysis or augmented reality systems.

The "no free lunch theorem" in machine learning posits that no single model
can be universally optimal across all tasks. This work provides empirical
evidence supporting this theorem. A model fine-tuned for a specific task,
especially when trained on a rich curated dataset, can potentially outperform
more general models. The dataset-driven approach adopted in this work not only
achieves commendable performance but does so with a significantly reduced
parameter count, leading to benefits like reduced computational costs and
faster inference times.

The promising results demonstrated by our image captioning model invite
several avenues for future research, particularly in the realm of scene
understanding tasks. The fine-grained object recognition and holistic scene
comprehension capabilities of our vision encoder suggest its potential as a
robust feature extractor for complex scene understanding applications. Future
work could explore the integration of our model within systems that require a
deeper semantic understanding of visual inputs, such as autonomous navigation
systems where accurate environmental interpretation is crucial, or in
intelligent surveillance systems where distinguishing subtle nuances could be
paramount for safety and security.

Additionally, the use of our richly annotated caption dataset offers a fertile
ground for the pretraining of image-text models like CLIP [27]. By leveraging
our dataset, which has been proven to enhance captioning quality, we could
augment the richness of the features that such models learn. This could, in
turn, enhance their performance across a wide range of vision-and-language
tasks. Furthermore, the extension of the token context window from the
standard 77 tokens to 256 tokens, as used in our work, suggests an interesting
research direction. This increased context window could allow for the
capturing of more complex relationships and subtleties within scenes,
potentially leading to a more nuanced and comprehensive understanding of
visual content.

Moreover, the fine-tuning of text-to-image models using our approach presents
a tantalizing opportunity to improve the generation of detailed images from
longer, more complex descriptions. By feeding these models with fine-grained,
descriptive captions, they could learn to render images that better reflect
the intricacies of the input text, thus enhancing the synthesis of rich visual
scenes. This could significantly benefit domains such as digital art and
design, where the ability to generate detailed visual content from descriptive
language can streamline creative workflows.

In conclusion, this paper emphasizes the significance of fine-tuning, the
innovative approach of using ChatGPT for indirect knowledge distillation, and
the paramount importance of a rich curated dataset in the domain of image
captioning. As research in this field progresses, this work provides a
foundation, emphasizing the balance between model efficiency, innovation, and
data quality.

## References

  * [1] Salesforce  “blip-image-captioning-large”, https://huggingface.co/Salesforce/blip-image-captioning-large, 2023
  * [2] Google  “PaLM 2 Technical Report”, 2023  arXiv:2305.10403 [cs.LG]
  * [3] OpenAI (2023)  “GPT-4 Technical Report”, 2023  arXiv:2303.08774 [cs.LG]
  * [4] Girish Kulkarni et al.  “Baby talk: Understanding and generating simple image descriptions”  In _CVPR 2011_ , 2011, pp. 1601–1608  DOI: 10.1109/CVPR.2011.5995466
  * [5] Oriol Vinyals, Alexander Toshev, Samy Bengio and Dumitru Erhan  “Show and Tell: A Neural Image Caption Generator”, 2014  arXiv:1411.4555 [cs.LG]
  * [6] Kelvin Xu et al.  “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”, 2015  arXiv:1502.03044 [cs.LG]
  * [7] Ashish Vaswani et al.  “Attention Is All You Need”, 2017  arXiv:1706.03762 [cs.LG]
  * [8] Junnan Li, Dongxu Li, Silvio Savarese and Steven Hoi  “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models”, 2023  arXiv:2301.12597 [cs.LG]
  * [9] Edward Hu et al.  “LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS”, 2021  arXiv:2106.09685 [cs.LG]
  * [10] Tsung-Yi Lin et al.  “Microsoft COCO: Common Objects in Context”, 2014  arXiv:1405.0312 [cs.LG]
  * [11] Peter Young, Alice Lai, Micah Hodosh and Julia Hockenmaier  “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions”, 2014  URL: https://shannon.cs.illinois.edu/DenotationGraph/
  * [12] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach and Amanpreet Singh  “TextCaps: a Dataset for Image Captioning with Reading Comprehension”, 2020  arXiv:2003.12462 [cs.LG]
  * [13] Harsh Agrawal et al.  “nocaps: novel object captioning at scale”, 2020  arXiv:2003.12462 [cs.LG]
  * [14] Deyao Zhu et al.  “MINIGPT-4: ENHANCING VISION-LANGUAGE UNDERSTANDING WITH ADVANCED LARGE LANGUAGE MODELS”, 2023  arXiv:2304.10592 [cs.LG]
  * [15] Teng Wang et al.  “Caption Anything: Interactive Image Description with Diverse Multimodal Controls”, 2023  arXiv:2305.02677 [cs.LG]
  * [16] Alexander Kirillov et al.  “Segment Anything”, 2023  arXiv:2304.02643 [cs.LG]
  * [17] OpenAI  “Introducing ChatGPT”, 2022  URL: https://openai.com/blog/chatgpt
  * [18] Ning Ding et al.  “MINIGPT-4: ENHANCING VISION-LANGUAGE UNDERSTANDING WITH ADVANCED LARGE LANGUAGE MODELS”, 2023  arXiv:2306.00693 [cs.LG]
  * [19] Nicolas Carion et al.  “End-to-End Object Detection with Transformers”, 2020  arXiv:2005.12872 [cs.LG]
  * [20] Youcai Zhang et al.  “Recognize Anything: A Strong Image Tagging Model”, 2023  arXiv:2306.03514 [cs.LG]
  * [21] Youcai Zhang et al.  “Simple Open-Vocabulary Object Detection with Vision Transformers”, 2022  arXiv:2205.06230 [cs.LG]
  * [22] Xinyu Huang et al.  “TAG2TEXT: GUIDING VISION-LANGUAGE MODEL VIA IMAGE TAGGING”, 2023  arXiv:2303.05657 [cs.LG]
  * [23] ybelkada  “blip2-opt-2.7b-fp16-sharded”, https://huggingface.co/ybelkada/blip2-opt-2.7b-fp16-sharded, 2023
  * [24] Ilya Loshchilov and Frank Hutter  “DECOUPLED WEIGHT DECAY REGULARIZATION”, 2017  arXiv:1711.05101 [cs.LG]
  * [25] Sourab Mangrulkar et al.  “PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods”, https://github.com/huggingface/peft, 2022
  * [26] Thomas Wolf et al.  “Transformers: State-of-the-Art Natural Language Processing”  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ Online: Association for Computational Linguistics, 2020, pp. 38–45  URL: https://www.aclweb.org/anthology/2020.emnlp-demos.6
  * [27] Alec Radford et al.  “Learning Transferable Visual Models From Natural Language Supervision”, 2021  arXiv:2103.00020 [cs.CV]

##  Appendix A Human Evaluation Survey

In order to conduct a robust human evaluation of the generated captions, we
initiated our analysis by selecting a subset of 50 random images from the test
subset of the COCO dataset. The PixLore and BLIP models were executed locally
to generate captions for these images, ensuring a direct and consistent
evaluation environment. For GPT-4 and Google Bard, we utilized their
respective web applications, submitting the images individually with a
standardized prompt: "Describe this image in a single paragraph". This prompt
was deliberately crafted to be unambiguous, minimizing any potential bias that
could arise from more complex prompt-engineering. Moreover, the inclusion of
the specification "in a single paragraph" aimed to align the task with the
evaluative scope while simultaneously circumventing the 256-token response
limitation inherent in our model. This limit was preemptively deemed
sufficient for the descriptive task at hand, based on the premise that a well-
articulated caption does not necessitate verbosity.

For the human evaluation, we designed a straightforward and user-friendly
survey using Streamlit. When participants accessed the survey, they were
randomly shown 3 images from the set, alongside the captions created by Google
Bard, GPT-4, BLIP, and PixLore (two versions). The captions were also sorted
randomly for each image. Participants scored from 0 to 5 each captions. This
allowed to independently grade each caption. Aware that judging a caption is
subjective, we provided clear instructions to help participants decide what
aspects might make a caption more effective, such as accuracy, detail, and
clarity. These guidelines, along with a sample question, is displayed in
Figure 4.

Figure 4: Sample of the survey header, with the instructions given to the
evaluators; and a sample of a question, with the five options of the models.
The image is a random image of the test set of the COCO dataset and the
captions are ordered randomly.

## Acknowledgments

This research supported in part by Cognizant and the Universitat de Valencia.


# Group-based Sparse Representation for Image Compressive Sensing
Reconstruction with Non-Convex Regularization

Zhiyuan Zhaa, Xinggan Zhanga, Qiong Wanga, Lan Tanga,b, Xin Liuc aSchool of
Electronic Science and Engineering, Nanjing University, Nanjing 210023, China.
b National Mobile Commun. Research Lab., Southeast University, Nanjing 210023,
China. c The Center for Machine Vision and Signal Analysis, University of
Oulu, 90014, Finland.

###### Abstract

Patch-based sparse representation modeling has shown great potential in image
compressive sensing (CS) reconstruction. However, this model usually suffers
from some limits, such as dictionary learning with great computational
complexity, neglecting the relationship among similar patches. In this paper,
a group-based sparse representation method with non-convex regularization
(GSR-NCR) for image CS reconstruction is proposed. In GSR-NCR, the local
sparsity and nonlocal self-similarity of images is simultaneously considered
in a unified framework. Different from the previous methods based on sparsity-
promoting convex regularization, we extend the non-convex weighted $\ell_{p}$
(0$<p<$1) penalty function on group sparse coefficients of the data matrix,
rather than conventional $\ell_{1}$-based regularization. To reduce the
computational complexity, instead of learning the dictionary with a high
computational complexity from natural images, we learn the principle component
analysis (PCA) based dictionary for each group. Moreover, to make the proposed
scheme tractable and robust, we have developed an efficient iterative
shrinkage/thresholding algorithm to solve the non-convex optimization problem.
Experimental results demonstrate that the proposed method outperforms many
state-of-the-art techniques for image CS reconstruction.

###### keywords:

Image CS reconstruction, group sparse representation, nonlocal self-
similarity, non-convex weighted $\ell_{p}$ minimization, iterative
shrinkage/thresholding algorithm.

††journal: Neurocomputingmytitlenotemytitlenotefootnotetext: Fully documented
templates are available in the elsarticle package on CTAN.

##  1 Introduction

Compressive sensing (CS) [1, 2, 3], which aims to recover signals from fewer
measurements than suggested by the Nyquist sampling theory, is based on the
hypothesis that the signals in question have compressible representations. The
most attractive aspect of CS-based compression is that the sampling and
compression are conducted simultaneously, and almost all computational cost is
derived from the decoder stage, and thus, leading to a low computational cost
of the encoder stage. Due to the superior property of CS, it has been widely
applied to various areas, such as MRI image [4], remoting sensing [5], single-
pixel camera [6] and sensor networks [7].

In the theory of CS, if a signal is sparse in some transform domain, it is
often sampled by the random projection and reconstructed by solving the
$\ell_{0}$ minimization problem with the prior information which usually makes
up the regularization terms. However, due to the fact that $\ell_{0}$
minimization is a difficult combinatorial optimization problem, solving this
problem is NP-hard. For this reason, it has been proposed to replace the
$\ell_{0}$ norm by its convex $\ell_{1}$ counterpart to make the optimization
easy. For instance, Candès $\emph{et al}.$ [1] proposed that solving
$\ell_{1}$ minimization problem can recover a $K$-sparse signal
$\textbf{\emph{X}}\in\Re^{N}$ from $M=O(Klog(N/K))$ random measurements. To
solve the above $\ell_{1}$ minimization problem, many CS reconstruction
algorithms have been proposed, such as linear programming [3], gradient
projection sparse reconstruction [8], match pursuit [9] and iterative
thresholding [10].

As a basic image inverse problem in the filed of image restoration, maybe the
hottest topic is image CS reconstruction, which has attracted a lot of
research interest in the past few years [11, 12, 16, 17, 18, 13, 14, 15, 20,
19, 21, 22, 23, 24, 25, 26, 27, 28]. Image CS reconstruction aims to
reconstruct high quality image from fewer measurements, which may even be far
below the traditional Nyquist sampling rate. Due to the ill-posed nature of
image CS reconstruction, it has been well-known that the prior knowledge of
images plays a critical role in improving the performance of image CS
reconstruction algorithms. Therefore, how to design an effective
regularization term to describe the image priors is vital for image CS
reconstruction tasks.

Early regularization models mainly consider the prior on the level of pixels,
such as Tikhonov regularization [29] and total variation (TV) regularization
[13, 14, 15], utilize the local structure patterns of an image and high
effectiveness to preserve image edges and recover the smooth regions. However,
some undesirable properties are produced, including smearing out the image
details and over-smoothing the images.

Another popular prior is based on image patch, which has shown promising
performance in image CS reconstruction [17, 18, 20, 19]. The well-known work
is sparse representation-based model [30, 31], which assumes that image patch
can be precisely encoded as a sparse linear combination of basic elements.
These elements, called atoms, compose a dictionary [32, 33]. The dictionary is
usually learned from a natural image dataset [51].Compared with the
traditional analytically designed dictionaries, such as DCT [16, 34] and
wavelet [22], dictionaries learned directly from images are superior to be
adapted to image local structures [30, 31], and thus could improve the
sparsity which results in better performance. For example, Zhang $\emph{et
al}.$ [20] proposed a method for image CS reconstruction using adaptively
learned sparsifying basis via $\ell_{0}$ minimization. Zha $\emph{et al}.$
[19] proposed an adaptive sparse nonlocal regularization (ASNR) model for
image CS reconstruction. However, two main problems are still existing for
patch-based sparse representation model. First, it is computationally
expensive to learn an off-the-shelf dictionary. Second, this sparse
representation-based model usually neglects the correlations between sparsely
coded patches.

Image patches that have similar patterns can be spatially far from each other
and thus can be collected in the whole image. The nonlocal self-similarity
(NSS) prior characterizes the repetitiveness of textures and structures
reflected by natural images within nonlocal regions, which can be exploited to
retain edges and sharpness effectively. The seminal work of nonlocal means
(NLM) denoising [35] has motivated a wide range of studies on NSS and a flurry
of NSS-based methods have been proposed for image CS reconstruction [16, 22,
36]. For instance, Zhang $\emph{et al}.$ [36] proposed a nonlocal total
variation (NLTV) regularization model for image CS reconstruction. Zhang
$\emph{et al}.$ [16] proposed a framework via collaborative sparsity, which
enforces local 2D sparsity and nonlocal 3D sparsity simultaneously, in an
adaptive hybrid space-transform domain. Nasser $\emph{et al}.$ [22] proposed a
new technique for high-fidelity image CS reconstruction via joint adaptive
sparsity regularization (JASR) in transform domain.

Recent advances have suggested that, by exploiting the NSS prior and
clustering similar patches, group-based sparse representation has shown great
potential in various image inverse problems [37, 38, 39, 40]. In this paper,
we propose a new method for image CS reconstruction, using group-based sparse
representation framework with non-convex regularization (GSR-NCR). The GSR
offers a powerful mechanism of combining local sparsity and NSS of images
simultaneously. Unlike the previous sparsity-promoting convex regularization
methods, we extend the non-convex weighted $\ell_{p}$ (0$<p<$1) penalty
function on group sparse coefficients of the data matrix, rather than
conventional $\ell_{1}$-based regularization. In order to reduce the
computational complexity, we learn the principle component analysis (PCA)
based dictionary for each group to substitute for the dictionary with a high
computational complexity learned from natural images. In addition, to make the
optimization tractable, an efficient iterative shrinkage/thresholding
algorithm is adopted to solve the non-convex optimization problem.
Experimental results show that the proposed method can outperform many
exisiting state-of-the-art image CS reconstruction methods.

The reminder of this paper is organized as follows. Section 2 briefly
introduces CS theory, patch-based sparse representation modeling and group-
based sparse representation modeling. Section 3 presents the modeling of
group-based sparse representation with non-convex regularization (GSR-NCR) for
image CS reconstruction and develops an iterative shrinkage/thresholding
algorithm to solve the proposed GSR-NCR model. Section 4 presents the
experimental results. Finally, some conclusions are given in Section 5.

##  2 Background and related work

###  2.1 Compressive Sensing

Compressive sensing (CS) has attracted considerable attention from
signal/image processing communities [1, 2, 3]. In the theory of CS,
$\textbf{\emph{X}}\in\Re^{N}$ is a finite length signal. X is said to be
sparse if X can be represented as a superposition of a small number of vectors
taken from a known sparsifying transform domain basis $\boldsymbol{\Psi}$,
such that $\boldsymbol{\theta}={\boldsymbol{\Psi}}^{T}\textbf{\emph{X}}$
contains only a small set of non-zero entries. The number of significant
elements within the coefficient vector $\boldsymbol{\theta}$ is regarded as
the quantitative criteria of the sparsity of X in $\boldsymbol{\Psi}$. To be
concrete, one seeks the perfect reconstruction of a signal X from its $M$
randomized linear measurements, i.e.,
$\textbf{\emph{Z}}={\boldsymbol{\phi}}\textbf{\emph{X}}$, where
$\textbf{\emph{Z}}\in\Re^{M}$, ${\boldsymbol{\phi}}\in\Re^{M\times N}$
represents the random projection matrix and satisfies $M<N$. The goal of CS
recovery is to reconstruct X from Z with subrate being $S=M/N$, which is
usually formulated as the following $\ell_{0}$ minimization problem,

$\arg\min_{{\boldsymbol{\theta}}}||\boldsymbol{\theta}||_{0},\qquad
s.t.\qquad{\textbf{\emph{Z}}}={\boldsymbol{\phi}}{\boldsymbol{\Psi}}\boldsymbol{\theta}$
(1)

where $||*||_{0}$ is $\ell_{0}$-norm, counting the non-zero entries of
${\boldsymbol{\theta}}$.

However, since $||*||_{0}$ norm minimization is discontinuous and an NP-hard
problem, it is usually relaxed to the convex $\ell_{1}$-norm minimization.
Therefore, Eq. (1) can be rewritten as the following unconstrained
optimization problem,

${\boldsymbol{\theta}}=\arg\min_{{\boldsymbol{\theta}}}\left(\frac{1}{2}||{\textbf{\emph{Z}}}-{\boldsymbol{\phi}}{\boldsymbol{\Psi}}\boldsymbol{\theta}||_{2}^{2}+\lambda||\boldsymbol{\theta}||_{1}\right)$
(2)

where $\lambda$ is regularization parameter. According to [1], CS is capable
of recovering a $K$-sparse signal X (with highly probability) from Z of size
$M$, where the number of random measurements satisfies $M=O(Klog(N/K))$.

###  2.2 Patch-based Sparse Representation

Traditional patch-based sparse representation model has been proven to be very
effective in image CS reconstruction [17, 18, 20, 19]. It assumes that each
image patch can be precisely modeled as a sparse linear combination of basic
elements [30, 31]. These elements are called atoms and they compose a
dictionary [32, 33]. Mathematically, for an image
$\textbf{\emph{X}}\in\Re^{N}$, let
$\textbf{\emph{x}}_{i}=\textbf{\emph{R}}_{i}\textbf{\emph{X}}$, $i=1,2,...n$
denotes an image patch of size $\sqrt{m}\times\sqrt{m}$ extracted at location
$i$, where $\textbf{\emph{R}}_{i}$ is the matrix extracting patch
$\textbf{\emph{x}}_{i}$ from X at location $i$. Given a dictionary
$\textbf{\emph{D}}\in\Re^{m\times M},m\leq M$, the sparse representation
processing of each patch $\textbf{\emph{x}}_{i}$ is to discover a sparse
vector ${\boldsymbol{\alpha}}_{i}$ such that
${\boldsymbol{\alpha}}_{i}=\textbf{\emph{D}}^{-1}\textbf{\emph{x}}_{i}$, where
${\boldsymbol{\alpha}}_{i}$ is a sparse vector whose entries are mostly zero
or close to zero. Then the whole image X can be reconstructed by averaging all
the reconstructed patches $\\{\textbf{\emph{x}}_{i}\\}$, which can be
expressed as

$\textbf{\emph{X}}\approx\textbf{\emph{D}}{\boldsymbol{\alpha}}=\left(\sum\limits_{i=1}^{n}{\textbf{\emph{R}}_{i}^{T}}{\textbf{\emph{R}}_{i}}\right)^{-1}\left(\sum\limits_{i=1}^{n}{\textbf{\emph{R}}_{i}^{T}}{\textbf{\emph{D}}}{\boldsymbol{\alpha}_{i}}\right)$
(3)

where $\boldsymbol{\alpha}$ denotes the concatenation of all
$\boldsymbol{\alpha}_{i}$, that is,
$\boldsymbol{\alpha}=[\boldsymbol{\alpha}_{1}^{T},\boldsymbol{\alpha}_{2}^{T},...,\boldsymbol{\alpha}_{n}^{T}]^{T}$,
which is the patch-based redundant sparse representation for X.

Now, we merge Eq. (3) into Eq. (2), the patch-based sparse representation
scheme for image CS reconstruction is formulated as

${\boldsymbol{\alpha}}_{i}=\arg\min_{{\boldsymbol{\alpha}}_{i}}\sum_{i=1}^{n}\left(\frac{1}{2}||{\textbf{\emph{z}}}_{i}-{\boldsymbol{\phi}}{\textbf{\emph{D}}}\boldsymbol{\alpha}_{i}||_{2}^{2}+\lambda||\boldsymbol{\alpha}_{i}||_{1}\right)$
(4)

where D replaces $\boldsymbol{\Psi}$ in Eq. (2), standing for a learning
dictionary, and $\boldsymbol{\alpha}_{i}$ is a patch-based sparse
representation coefficient for each patch ${\textbf{\emph{x}}}_{i}$ over the
dictionary D. ${\textbf{\emph{z}}}_{i}$ is the linear measurements of each
patch ${\textbf{\emph{x}}}_{i}$.

However, there exists two main issues for patch-based sparse representation
model. On one hand, since dictionary learning is a large-scale and highly non-
convex problem, it is computationally expensive to solve the sparsity
optimization problem. On the other hand, the patch-based sparse representation
model usually assumes the independence between sparsely coded patches, which
takes no account of the correlation of similar patches in essence.

Figure 1: Illustrations for the group construction. Extract each exemplar
patch vector $\textbf{\emph{x}}_{i}$ from image $\textbf{\emph{X}}_{G}$. For
each patch $\textbf{\emph{x}}_{i}$, denote ${\textbf{\emph{S}}}_{G_{i}}$ is
the set, which composed of its most $c$ similar patches. Stack all the patches
in ${\textbf{\emph{S}}}_{G_{i}}$ to construct the data matrix to generate the
group, denoted by ${\textbf{\emph{X}}}_{G_{i}}$.

###  2.3 Group-based Sparse Representation

Recent studies have shown that structured or group sparsity can offer more
promising performance for image restoration tasks [37, 38, 39, 40]. Since the
unit of our proposed sparse representation model is group, this section will
give briefs to introduce how to construct the groups. Specifically, as shown
in Fig. 1, image $\textbf{\emph{X}}_{G}$ with size N is divided into n
overlapped patches $\textbf{\emph{x}}_{i}$ of size
$\sqrt{m}\times\sqrt{m},i=1,2,...,n$. Then for each exemplar patch
$\textbf{\emph{x}}_{i}$, denoted by small red square in Fig. 1, within the
$H\times H$ sized searching window (big blue square), its most similar $c$
patches (small green squares) are selected to form a set
${\textbf{\emph{S}}}_{G_{i}}$. Since then, all the patches in
${\textbf{\emph{S}}}_{G_{i}}$ are stacked into a matrix
${\textbf{\emph{X}}}_{G_{i}}\in\Re^{{m}\times{c}}$, which contains every
element of ${\textbf{\emph{S}}}_{G_{i}}$ as its column, i.e.,
${\textbf{\emph{X}}}_{G_{i}}=\\{{\textbf{\emph{x}}}_{G_{i,1}},{\textbf{\emph{x}}}_{G_{i,2}},...,{\textbf{\emph{x}}}_{G_{i,c}}\\}$.
The matrix ${\textbf{\emph{X}}}_{G_{i}}$ consisting of all the patches with
similar structures is called as a group, where ${\textbf{\emph{x}}_{G_{i,c}}}$
denotes the $c$-th similar patch (column form) of the $i$-th group. Finally,
similar to patch-based sparse representation [30, 31], given a dictionary
${\textbf{\emph{D}}}_{G_{i}}$, which is often learned from each group, such as
DCT [16], PCA-based dictionary [41]. Therefore, in image CS reconstruction,
similar to Eq. (4), each group ${\textbf{\emph{X}}}_{G_{i}}$ can be sparsely
represented as
$\boldsymbol{\alpha}_{G_{i}}={{\textbf{\emph{D}}}_{G_{i}}}^{-1}\textbf{\emph{X}}_{G_{i}}$
and solved by the following $\ell_{1}$-norm minimization problem,

${\boldsymbol{\alpha}}_{G_{i}}=\arg\min_{{\boldsymbol{\alpha}}_{G_{i}}}\sum_{i=1}^{n}\left(\frac{1}{2}||{\textbf{\emph{Z}}}_{G_{i}}-{\boldsymbol{\phi}}{\textbf{\emph{D}}}_{G_{i}}\boldsymbol{\alpha}_{G_{i}}||_{F}^{2}+\lambda||\boldsymbol{\alpha}_{G_{i}}||_{1}\right)$
(5)

where ${\textbf{\emph{Z}}}_{G_{i}}$ is the linear measurements of each group
${\textbf{\emph{X}}}_{G_{i}}$.

##  3 Image CS reconstruction using group-based sparse representation model
with non-convex weighted $\ell_{p}$ Minimization

Typical patch-based sparse representation methods for image CS reconstruction
usually suffer from a common drawback that the dictionary learning with great
computational complexity and neglecting the relationships among similar
patches [37, 38, 39, 40]. The sparsity-promoting convex $\ell_{1}$
minimization is usually regarded as a standard scheme for recovering a sparse
signal. However, a fact that cannot be ignored is that, $\ell_{1}$
minimization is hard to achieve the desired sparsity solution in some
practical problems, such as image inverse problems [42]. Based on the fact
above, this paper proposes a new method for image CS reconstruction using
group-based sparse representation with non-convex weighted $\ell_{p}$
minimization. To make the optimization tractable, an iterative
shrinkage/thresholding (IST) algorithm [43] is developed to solve the above
non-convex weighted $\ell_{p}$ minimization problem efficiently.

###  3.1 Modeling of the Proposed Image CS Reconstruction

To obtain sparsity solution more accurately, inspired by the success of
$\ell_{p}$ ($0<p<1$) sparse optimization [44, 45, 46] and our previous work
[42], we apply the non-convex weighted $\ell_{p}$ ($0<p<1$) penalty function
on group sparse coefficients of the data matrix to replace the convex
$\ell_{1}$ norm. To be concrete, different from Eq. (5), the proposed group-
based sparse representation for image CS reconstruction with non-convex
weighted $\ell_{p}$ minimization is formulated as

${\boldsymbol{\alpha}}_{G}=\arg\min_{{\boldsymbol{\alpha}}_{G}}\frac{1}{2}||{\textbf{\emph{Z}}}_{G}-{\boldsymbol{\phi}}{\textbf{\emph{D}}}_{G}\boldsymbol{\alpha}_{G}||_{2}^{2}+||{\textbf{\emph{W}}}_{G}\boldsymbol{\cdot}\alpha_{G}||_{p}$
(6)

where $\cdot$ represents the dot product and ${\textbf{\emph{W}}}_{G}$ is a
weight assigned to $\boldsymbol{\alpha}_{G}$. The weight
${\textbf{\emph{W}}}_{G}$ will enhance the representation capability of group
sparse coefficient $\boldsymbol{\alpha}_{G}$.

###  3.2 Solving the Non-convex Weighted $\ell_{p}$ Minimization by the
Iterative Shrinkage/Thresholding (IST) Algorithm

Solving the objective function of Eq. (6) is very difficult, since it is a
large scale non-convex optimization problem. To make the proposed scheme
tractable and robust, in this paper we adopt the iterative
shrinkage/thresholding (IST) algorithm [43] to solve Eq. (6). We will briefly
introduce IST algorithm. More specifically, consider the following general
optimization problem,

${\rm
min}_{\textbf{\emph{u}}\in\Re^{N}}f(\textbf{\emph{u}})+g(\textbf{\emph{u}})$
(7)

where $f(\textbf{\emph{u}})$ is a smooth convex function with gradient, which
is Lipschitz continuous. $g(\textbf{\emph{u}})$ is a continuous convex
function which is possibly non-smooth. The IST algorithm to solve Eq. (7) with
a constant step $\rho$ is formulated as

$\textbf{\emph{z}}^{(k+1)}=\textbf{\emph{u}}^{(k)}-\rho\nabla
f(\textbf{\emph{u}}^{(k)})$ (8)
$\textbf{\emph{u}}^{(k+1)}=\arg\min\limits_{\textbf{\emph{u}}}\frac{1}{2}||\textbf{\emph{u}}-\textbf{\emph{z}}^{(k+1)}||_{2}^{2}+\lambda
g(\textbf{\emph{u}})$ (9)

where $k$ denotes the iteration number. Then, by invoking IST algorithm, the
proposed non-convex weighted $\ell_{p}$ minimization problem Eq. (6) with the
constraint
$\textbf{\emph{u}}_{G}={\textbf{\emph{D}}}_{G}\boldsymbol{\alpha}_{G}$ can be
rewritten as

${\textbf{\emph{u}}_{G}}^{(k)}={\textbf{\emph{D}}}_{G}^{(k)}\boldsymbol{\alpha}_{G}^{(k)}$
(10)
${\textbf{\emph{Y}}_{G}}^{(k+1)}={\textbf{\emph{u}}_{G}}^{(k)}-\rho{\boldsymbol{\phi}}^{T}(\boldsymbol{\phi}{\textbf{\emph{u}}_{G}}^{(k)}-{\textbf{\emph{Z}}}_{G})$
(11)
$\boldsymbol{\alpha}_{G}^{(k+1)}=\arg\min\limits_{\boldsymbol{\alpha}_{G}}\frac{1}{2}||{\textbf{\emph{D}}}_{G}\boldsymbol{\alpha}_{G}-{\textbf{\emph{Y}}}_{G}^{(k+1)}||_{2}^{2}+||{\textbf{\emph{W}}}_{G}\cdot\boldsymbol{\alpha}_{G}||_{p}$
(12)

Obviously, the crux for solving Eq. (6) is translated into solving Eq. (12).
Next, we will show that there is an efficient solution to Eq. (12). To avoid
confusion, the subscribe $k$ may be omitted for conciseness.

However, due to the complex structure of
$||{\textbf{\emph{W}}}_{G}\boldsymbol{\cdot}\alpha_{G}||_{p}$, it is difficult
to solve Eq. (12), Let
$\textbf{\emph{X}}_{G}={\textbf{\emph{D}}}_{G}\boldsymbol{\alpha}_{G}$, Eq.
(12) can be rewritten as

$\boldsymbol{\alpha}_{G}=\arg\min\limits_{\boldsymbol{\alpha}_{G}}\frac{1}{2}||\textbf{\emph{X}}_{G}-{\textbf{\emph{Y}}}_{G}||_{2}^{2}+||{\textbf{\emph{W}}}_{G}\cdot\boldsymbol{\alpha}_{G}||_{p}$
(13)

To enable a tractable solution of Eq. (13), in this paper, a general
assumption is made, with which even a closed-form solution can be achieved.
Specifically, ${\textbf{\emph{Y}}}_{G}$ can be regarded as some type of noisy
observation of $\textbf{\emph{X}}_{G}$, and then the assumption is made that
each element of
$\textbf{\emph{E}}=\textbf{\emph{X}}_{G}-{\textbf{\emph{Y}}}_{G}$ follows an
independent zero-mean distribution with variance ${\sigma}^{2}$. The following
conclusion can be proved with this assumption.

Theorem 1 Define $\textbf{\emph{X}}_{G},{\textbf{\emph{Y}}}_{G}\in\Re^{N}$,
${\textbf{\emph{X}}}_{G_{i}}$, ${\textbf{\emph{Y}}}_{G_{i}}\in\Re^{m\times
c}$, and ${\textbf{\emph{e}}}{(j)}$ as each element of error vector e, where
$\textbf{\emph{e}}=\textbf{\emph{X}}_{G}-\textbf{\emph{Y}}_{G},j=1,...,N$.
Assume that ${\textbf{\emph{e}}}{(j)}$ follows an independent zero mean
distribution with variance ${\sigma}^{2}$, and thus for any $\varepsilon>0$,
we can represent the relationship between
$\frac{1}{N}||\textbf{\emph{X}}_{G}-\textbf{\emph{Y}}_{G}||_{2}^{2}$ and
${\frac{1}{K}}\sum_{i=1}^{n}||{\textbf{\emph{X}}}_{G_{i}}-{\textbf{\emph{Y}}}_{G_{i}}||_{F}^{2}$
by the following property,

$\lim_{{N\rightarrow\infty}\atop{K\rightarrow\infty}}{\textbf{\emph{P}}}{\\{|\frac{1}{N}||\textbf{\emph{X}}_{G}-\textbf{\emph{Y}}_{G}||_{2}^{2}-{\frac{1}{K}}\sum\nolimits_{i=1}^{n}||{\textbf{\emph{X}}}_{G_{i}}-{\textbf{\emph{Y}}}_{G_{i}}||_{F}^{2}|<\varepsilon\\}}=1$
(14)

where ${\textbf{\emph{P}}}(\bullet)$ represents the probability and
${\emph{K}}=\emph{m}\times\emph{c}\times\emph{n}$. The detailed proof of
Theorem 1 can be seen in our previous work [47].

Therefore, based on Theorem 1, we have the following equation with a very
large probability (restricted 1) at each iteration,

$\frac{1}{N}||\textbf{\emph{X}}_{G}-\textbf{\emph{Y}}_{G}||_{2}^{2}={\frac{1}{K}}\sum\nolimits_{i=1}^{n}||{\textbf{\emph{X}}}_{G_{i}}-{\textbf{\emph{Y}}}_{G_{i}}||_{F}^{2}$
(15)

Based on Eqs. (13) and (15), we have

$\displaystyle\min\limits_{{{\boldsymbol{\alpha}}}_{G}}\frac{1}{2}{||\textbf{\emph{X}}_{G}-\textbf{\emph{Y}}_{G}||_{2}^{2}}+||{\textbf{\emph{W}}}_{G}\cdot\boldsymbol{\alpha}_{G}||_{p}$
(16)
$\displaystyle=\min\limits_{{{{\boldsymbol{\alpha}}}}_{G_{i}}}\sum\nolimits_{i=1}^{n}\left(\frac{1}{2}||{\textbf{\emph{X}}}_{G_{i}}-{\textbf{\emph{Y}}}_{G_{i}}||_{F}^{2}+\frac{K}{N}||{\textbf{\emph{W}}}_{G_{i}}\cdot\boldsymbol{\alpha}_{G_{i}}||_{p}\right)$
$\displaystyle=\min\limits_{{{{\boldsymbol{\alpha}}}}_{G_{i}}}\sum\nolimits_{i=1}^{n}\left(\frac{1}{2}||{\textbf{\emph{Y}}}_{G_{i}}-{{\textbf{\emph{D}}}_{G_{i}}{{{\boldsymbol{\alpha}}}}_{G_{i}}}||_{F}^{2}+\frac{K}{N}||{\textbf{\emph{W}}}_{G_{i}}\cdot\boldsymbol{\alpha}_{G_{i}}||_{p}\right)$
$\displaystyle=\min\limits_{{{{\boldsymbol{\alpha}}}}_{G_{i}}}\sum\nolimits_{i=1}^{n}\left(\frac{1}{2}||{\textbf{\emph{Y}}}_{G_{i}}-{{\textbf{\emph{D}}}_{G_{i}}{{{\boldsymbol{\alpha}}}}_{G_{i}}}||_{F}^{2}+\tau||{\textbf{\emph{W}}}_{G_{i}}\cdot\boldsymbol{\alpha}_{G_{i}}||_{p}\right)$

where $\tau=K/N$. Clearly, Eq. (16) can be regarded as a sparse representation
problem by solving $n$ sub-problems for all the group
${\textbf{\emph{X}}}_{G_{i}}$.

Note that, dictionary learning are often learned from images, but we have only
the linear measurements ${{\textbf{\emph{Z}}}_{G}}$. Thus, we need to generate
a initial image from the linear measurements ${{\textbf{\emph{Z}}}_{G}}$. In
this paper, we first use the Multi-hypothesis block-based compressive sensing
(MH-BCS) method [25] to generate the initial image
${{\textbf{\emph{X}}}_{G}}$. To adapt to the local image structures, instead
of learning an over-complete dictionary for each group as in [37], we learn
the principle component analysis (PCA) based dictionary [41] for each group
${\textbf{\emph{Y}}}_{G_{i}}$. The proposed PCA-based dictionary learning
method is efficient and convenient since it only requires one PCA
decomposition operator for each group ${\textbf{\emph{Y}}}_{G_{i}}$, rather
than learning the dictionary from natural image dataset with a high
computational complexity.

Due to the fact that each dictionary ${{\textbf{\emph{D}}}_{G_{i}}}$ is
orthogonal, Eq. (16) is equal to the following formula:

$\displaystyle{\hat{\boldsymbol{\alpha}}}_{G_{i}}=\min\limits_{{{{\boldsymbol{\alpha}}}}_{G_{i}}}\sum\nolimits_{i=1}^{n}\left(\frac{1}{2}||{{{{\boldsymbol{\gamma}}}}_{G_{i}}}-{{{{\boldsymbol{\alpha}}}}_{G_{i}}}||_{F}^{2}+\tau||{\textbf{\emph{W}}}_{G_{i}}\cdot\boldsymbol{\alpha}_{G_{i}}||_{p}\right)$
(17)
$\displaystyle=\min\limits_{\tilde{{{\boldsymbol{\alpha}}}}_{G_{i}}}\sum\nolimits_{i=1}^{n}\left(\frac{1}{2}||{\tilde{{{\boldsymbol{\gamma}}}}_{G_{i}}}-{\tilde{{{\boldsymbol{\alpha}}}}_{G_{i}}}||_{2}^{2}+\tau||{\tilde{\textbf{\emph{w}}}}_{G_{i}}\cdot\tilde{\boldsymbol{\alpha}}_{G_{i}}||_{p}\right)$

where
${\textbf{\emph{Y}}}_{i}={{\textbf{\emph{D}}}_{G_{i}}{{{\boldsymbol{\gamma}}}}_{G_{i}}}$
and
${\textbf{\emph{X}}}_{i}={{\textbf{\emph{D}}}_{G_{i}}{{{\boldsymbol{\alpha}}}}_{G_{i}}}$.
${\tilde{{{\boldsymbol{\alpha}}}}_{G_{i}}}$,
${\tilde{{{\boldsymbol{\gamma}}}}_{G_{i}}}$ and
${\tilde{\textbf{\emph{w}}}}_{G_{i}}$ denote the vectorization of the matrix
${{{{\boldsymbol{\alpha}}}}_{G_{i}}}$, ${{{{\boldsymbol{\alpha}}}}_{G_{i}}}$
and ${\textbf{\emph{W}}}_{G_{i}}$, respectively.

To obtain the solution of Eq. (17) effectively, in this paper, the generalized
soft-thresholding (GST) algorithm [46] is adopted to solve Eq. (17).
Specifically, given $p$, ${\tilde{{{\boldsymbol{\gamma}}}}_{G_{i}}}$ and
${\tilde{\textbf{\emph{w}}}}_{G_{i}}$, there exists a specific threshold,

$\tau_{p}^{\emph{GST}}({\tilde{{\emph{w}}}_{G_{i,j}}})=(2{\tilde{{\emph{w}}}_{G_{i,j}}}(1-p))^{\frac{1}{2-p}}+{\tilde{{\emph{w}}}_{G_{i,j}}}p(2{\tilde{{\emph{w}}}_{G_{i,j}}}(1-p))^{\frac{p-1}{2-p}}$
(18)

where ${\tilde{\gamma}_{G_{i,j}}}$, ${\tilde{\alpha}_{G_{i,j}}}$ and
${\tilde{{\emph{w}}}_{G_{i,j}}}$ are the $j$-th element of
${\tilde{{{\boldsymbol{\gamma}}}}_{i}}$,
${\tilde{{{\boldsymbol{\alpha}}}}_{i}}$ and ${\tilde{\textbf{\emph{w}}}}_{i}$,
respectively. Then, if
${\tilde{\gamma}_{G_{i,j}}}<\tau_{p}^{\emph{GST}}({\tilde{{\emph{w}}}_{G_{i,j}}})$,
${\tilde{\alpha}_{G_{i,j}}}=0$ is the global minimum. Otherwise, the optimum
will be achieved at non-zero point. According to [46], for any
${\tilde{\gamma}_{G_{i,j}}}\in(\tau_{p}^{\emph{GST}}({\tilde{{\emph{w}}}_{G_{i,j}}}),+\infty)$,
Eq. (17) has one unique minimum
${\textbf{\emph{S}}}_{p}^{\emph{GST}}({\tilde{\gamma}_{G_{i,j}}};{\tilde{{\emph{w}}}_{G_{i,j}}})$,
which can be obtained by solving the following equation,

${\textbf{\emph{S}}}_{p}^{\emph{GST}}({\tilde{\gamma}_{G_{i,j}}};{\tilde{{\emph{w}}}_{G_{i,j}}})-{\tilde{\gamma}_{G_{i,j}}}+{\tilde{{\emph{w}}}_{G_{i,j}}}p\left({\textbf{\emph{S}}}_{p}^{\emph{GST}}({\tilde{\gamma}_{G_{i,j}}};{\tilde{{\emph{w}}}_{G_{i,j}}})\right)^{p-1}=0$
(19)

The complete description of the GST algorithm is shown in Algorithm 1. For
more details about the GST algorithm, please refer to [46].

Algorithm 1: Generalized Soft-Thresholding (GST) [46].
---
Input: ${\tilde{\gamma}_{G_{i,j}}},{\tilde{{\emph{w}}}_{G_{i,j}}},p,J$.
1\.
$\tau_{p}^{\emph{GST}}({\tilde{{\emph{w}}}_{G_{i,j}}})=(2{\tilde{{\emph{w}}}_{G_{i,j}}}(1-p))^{\frac{1}{2-p}}+{\tilde{{\emph{w}}}_{G_{i,j}}}p(2{\tilde{{\emph{w}}}_{G_{i,j}}}(1-p))^{\frac{p-1}{2-p}}$;
2\. If
$|{\tilde{\gamma}_{G_{i,j}}}|\leq\tau_{p}^{\emph{GST}}({\tilde{{\emph{w}}}_{G_{i,j}}})$
3\.
${\textbf{\emph{S}}}_{p}^{\emph{GST}}({\tilde{\gamma}_{G_{i,j}}};{\tilde{{\emph{w}}}_{G_{i,j}}})=0$;
4\. else
5\. $k=0,{\tilde{\alpha}_{G_{i,j}}}^{(k)}=|{\tilde{\gamma}_{G_{i,j}}}|$;
6\. Iterate on $k=0,1,...,J$
7\.
${\tilde{\alpha}_{G_{i,j}}}^{(k+1)}=|{\tilde{\gamma}_{G_{i,j}}}|-{{\tilde{\emph{w}}}_{G_{i,j}}}p\left({\tilde{\alpha}_{G_{i,j}}}^{(k)}\right)^{p-1}$;
8\. $k\leftarrow k+1$;
9\.
${\textbf{\emph{S}}}_{p}^{\emph{GST}}({\tilde{\gamma}_{G_{i,j}}};{\tilde{{\emph{w}}}_{G_{i,j}}})={\rm
sgn}({\tilde{\gamma}_{G_{i,j}}}){\tilde{\alpha}_{G_{i,j}}}^{k}$;
10\. End
Ouput:
${\textbf{\emph{S}}}_{p}^{\emph{GST}}({\tilde{\gamma}_{G_{i,j}}};{\tilde{{\emph{w}}}_{G_{i,j}}})$.

Each weight $\textbf{\emph{W}}_{G_{i}}$ is assigned to group sparse
coefficient $\boldsymbol{\alpha}_{G_{i}}$, large values of each
$\boldsymbol{\alpha}_{G_{i}}$ usually include major edge and texture
information. This implies that to reconstruct ${\textbf{\emph{X}}}_{G_{i}}$
from its degraded one, we should shrink large values less, while shrinking
smaller ones more [48]. Inspired by [49], the weight
$\textbf{\emph{W}}_{G_{i}}$ of each group is set as
${\tilde{\textbf{\emph{w}}}}_{G_{i}}=[{\tilde{{\emph{w}}}}_{G_{i,1}},{\tilde{{\emph{w}}}}_{G_{i,2}},...,{\tilde{{\emph{w}}}}_{G_{i,j}}]$
and we have

${\tilde{{\emph{w}}}}_{G_{i,j}}=\frac{2\sqrt{2}\sigma^{2}}{(\tilde{\boldsymbol{\delta}}_{G_{i}}+\epsilon)}$
(20)

where $\tilde{\boldsymbol{\delta}}_{G_{i}}$ denotes the estimated variance of
each group sparse coefficient $\tilde{{{{\boldsymbol{\gamma}}}}}_{G_{i}}$, and
$\epsilon$ is a small constant. Obviously, it can be seen that each value of
weight $\textbf{\emph{W}}_{G_{i}}$ is inverse proportion to each value of
${{{\boldsymbol{\gamma}}}}_{G_{i}}$ [48]. In light of all derivations, the
complete description of the proposed image CS reconstruction using group-based
sparse representation via non-convex weighted $\ell_{p}$ minimization is given
in Table 1.

Table 1: The proposed GSR-NCR method for Image CS reconstruction. Input: The
observed measurement $\textbf{\emph{Y}}_{G}$, the measurement matrix
$\boldsymbol{\phi}$.
---
Initialization: Estimate an initial image ${\textbf{\emph{X}}_{G}}^{(0)}$
using a MH-BCS method [25] and
set parameters _m_ , _c_ , $\rho$, _p_ , $\sigma$, $\epsilon$, _H_ , _J_ ;
For $k=1,2,...,Max\\_iter$ do
Update $\textbf{\emph{Y}}_{G}^{k+1}$ computing by Eq. (11).
Generating the groups $\textbf{\emph{Y}}_{G_{i}}$ by searching similar patches
from $\textbf{\emph{Y}}_{G}$.
For each group $\textbf{\emph{Y}}_{G_{i}}$ do
Constructing dictionary ${{{\textbf{\emph{D}}}}_{G_{i}}}^{k+1}$ for
${{{\textbf{\emph{Y}}}}_{G_{i}}}$ by PCA operator.
Update ${{{\boldsymbol{\gamma}}}}_{G_{i}}^{k+1}$ by
${{{\boldsymbol{\gamma}}}}_{G_{i}}={{{\textbf{\emph{D}}}}_{G_{i}}}^{-1}{{{\textbf{\emph{Z}}}}_{G_{i}}}$.
Update ${\textbf{\emph{W}}_{G_{i}}}^{k+1}$ by Eq. (20).
Update $\boldsymbol{\alpha}_{G_{i}}^{k+1}$ computing by Algorithm 1.
Get the estimation ${{{\textbf{\emph{X}}}}_{G_{i}}}^{k+1}$
=${{{\textbf{\emph{D}}}}_{G_{i}}}^{k+1}$$\boldsymbol{\alpha}_{G_{i}}^{k+1}$.
End for
Aggregate all group ${{{\textbf{\emph{X}}}}_{G_{i}}}^{k+1}$ to form the
recovered image ${{\hat{\textbf{\emph{X}}}}_{G}}^{k+1}$.
End for
Output: ${{\hat{\textbf{\emph{X}}}}_{G}}^{k+1}$.

Figure 2: All test images. From left to right: Barbara, boats, Fence, F.print,
House, Leaves, plants, staw.

##  4 Experimental Results

In this section, we will report the experimental results of the proposed GSR-
NCR for image CS reconstruction. All the experimental images are shown in Fig.
2. To evaluate the quality of the restored images, the PSNR and the recently
proposed powerful perceptual quality metric FSIM [50] are calculated.

###  4.1 Parameter Setting

We generate the CS measurements at the block level by using a Gaussian random
projection matrix to test images, i.e., the block-based CS reconstruction with
block size of $32\times 32$. The parameters are set as follows. The size of
each patch $\sqrt{m}\times\sqrt{m}$ is set to be $7\times 7$. Similar patch
numbers $c=60$, the search window size $H=20$, $\sigma=\sqrt{2}$,
$\epsilon=10^{-14}$, $J=2$. ($\rho,p$) are set to (0.3, 0.5), (1.5, 0.95) and
(1.5, 0.95) when $0.2N$, $0.3N$ and $0.4N$, respectively.

###  4.2 Performance Comparison with the State-of-the-Art methods

We have compared the proposed GSR-NCR against six other competing approaches
including BCS [11], BM3D-CS [12], ADS-CS [17], SGSR [26], ALSB [20] and MRK
[27]. Note that ADS-CS and ALSB are patch-based sparse representation methods
for image CS reconstruction. The PSNR and FSIM results by the competing CS
reconstruction methods are shown in Table 2 and Table 3, respectively. It can
be seen that the proposed GSR-NCR performs competitively compared to other
methods. In terms of PSNR, the proposed GSR-NCR achieves 7.89dB, 2.85dB,
1.11dB, 1.20dB, 3.22dB and 2.72dB improvement on average over the BCS,
BM3D-CS, ADS-CS, SGSR, ALSB and MRK, respectively. Meanwhile, based on the
FSIM, the proposed GSR-NCR achieves 0.1031, 0.0339, 0.0123, 0.0063, 0.0106 and
0.0316 improvement on average over the BCS, BM3D-CS, ADS-CS, SGSR, ALSB and
MRK, respectively. The visual comparisons of the image CS reconstruction are
shown in Figs. 3 \- 6. It can be seen that the BCS, BM3D-CS, ADS-CS, SGSR,
ALSB and MRK methods still suffer from some undesirable artifacts or over-
smooth phenomena. By contrast, the proposed GSR-NCR not only removes most of
the visual artifacts, but also preserves large-scale sharp edges and small-
scale fine image details more effectively.

Figure 3: CS reconstructed image boats with $0.2N$ measurements. (a) Original
image; (b) BCS [11] (PSNR=27.05dB, FSIM=0.865); (c) BM3D-CS [12]
(PSNR=31.02dB, FSIM=0.931); (d) ADS-CS [17] (PSNR=33.15dB, FSIM=0.951); (e)
SGSR [26] (PSNR=32.43dB, FSIM=0.947); (f) ALSB [20] (PSNR= 32.96dB,
FSIM=0.951); MRK [27] (PSNR=32.38dB, FSIM=0.948); GSR-NCR (PSNR=33.31dB,
FSIM=0.953).

Figure 4: CS reconstructed image Barbara with $0.2N$ measurements. (a) Original image; (b) BCS [11] (PSNR=22.24dB, FSIM=0.844); (c) BM3D-CS [12] (PSNR=28.82dB, FSIM=0.907); (d) ADS-CS [17] (PSNR=32.27dB, FSIM=0.950); (e) SGSR [26] (PSNR=33.44dB, FSIM=0.962); (f) ALSB [20] (PSNR= 30.72dB, FSIM=0.932); MRK [27] (PSNR=27.99dB, FSIM=0.914); GSR-NCR (PSNR=33.93dB, FSIM=0.964). Table 2: PSNR (dB) Comparisons of BCS [11], BM3D-CS [12], ADS-CS [17], SGSR [26], ALSB [20], MRK [27] and the Proposed GSR-NCR. Ratio | Method | _Barbara_ | _boats_ | _Fence_ | _F.print_ | _House_ | _Leaves_ | _plants_ | _straw_ | _Average_
---|---|---|---|---|---|---|---|---|---|---
0.2 | BCS | 22.24 | 27.05 | 21.57 | 18.50 | 30.54 | 21.12 | 30.67 | 20.69 | 24.30
BM3S-CS | 28.82 | 31.02 | 26.87 | 19.37 | 35.01 | 28.13 | 34.98 | 20.04 | 28.03
ADS-CS | 32.27 | 33.15 | 28.37 | 22.70 | 35.76 | 27.88 | 35.45 | 23.75 | 29.92
SGSR | 33.44 | 32.43 | 29.42 | 23.60 | 35.81 | 28.79 | 34.64 | 24.54 | 30.33
ALSB | 30.72 | 32.96 | 28.41 | 23.69 | 36.08 | 27.15 | 32.16 | 24.33 | 24.33
MRK | 27.99 | 32.38 | 22.20 | 20.54 | 36.36 | 27.75 | 35.99 | 23.02 | 28.28
| GSR-NCR | 33.93 | 33.31 | 29.10 | 23.66 | 36.57 | 29.03 | 35.72 | 24.42 | 30.72
0.3 | BCS | 25.59 | 28.91 | 23.24 | 19.96 | 32.85 | 23.16 | 32.81 | 22.19 | 26.09
BM3D-CS | 33.01 | 34.04 | 30.67 | 23.01 | 36.88 | 32.52 | 38.30 | 22.37 | 31.35
ADS-CS | 35.81 | 36.35 | 31.29 | 25.33 | 38.21 | 32.55 | 38.45 | 26.58 | 33.07
SGSR | 35.91 | 35.22 | 31.56 | 25.84 | 37.37 | 33.00 | 37.20 | 27.34 | 32.93
ALSB | 35.00 | 36.42 | 30.83 | 25.84 | 38.34 | 31.08 | 38.05 | 26.61 | 32.77
MRK | 32.64 | 34.97 | 24.44 | 24.21 | 38.35 | 32.37 | 39.06 | 25.52 | 31.45
| GSR-NCR | 37.19 | 37.27 | 32.26 | 26.35 | 39.38 | 34.95 | 40.10 | 27.58 | 34.38
0.4 | BCS | 27.10 | 30.56 | 24.81 | 21.67 | 34.65 | 25.07 | 34.77 | 23.71 | 27.79
BM3D-CS | 35.92 | 36.69 | 33.84 | 25.47 | 38.08 | 35.87 | 41.18 | 24.38 | 33.93
ADS-CS | 38.34 | 38.79 | 34.02 | 27.32 | 40.30 | 35.94 | 40.77 | 28.80 | 35.54
SGSR | 37.70 | 37.41 | 33.35 | 27.85 | 38.99 | 35.83 | 39.23 | 29.63 | 35.00
ALSB | 37.19 | 38.92 | 32.83 | 27.70 | 40.25 | 34.57 | 40.66 | 28.54 | 35.08
MRK | 36.17 | 37.20 | 26.63 | 26.83 | 40.04 | 35.53 | 41.64 | 27.69 | 33.97
| GSR-NCR | 39.23 | 39.65 | 34.39 | 28.53 | 41.12 | 38.55 | 42.48 | 30.06 | 36.75
Table 3: FSIM Comparisons of BCS [11], BM3D-CS [12], ADS-CS [17], SGSR [26], ALSB [20], MRK [27] and the Proposed GSR-NCR. Ratio | Method | _Barbara_ | _boats_ | _Fence_ | _F.print_ | _House_ | _Leaves_ | _plants_ | _straw_ | _Average_
---|---|---|---|---|---|---|---|---|---|---
0.2 | BCS | 0.8443 | 0.8654 | 0.7653 | 0.7355 | 0.9011 | 0.7531 | 0.8973 | 0.7606 | 0.8153
BM3S-CS | 0.9072 | 0.9314 | 0.8325 | 0.8184 | 0.9498 | 0.9231 | 0.9450 | 0.7604 | 0.8835
ADS-CS | 0.9498 | 0.9508 | 0.9181 | 0.8976 | 0.9423 | 0.9015 | 0.9458 | 0.8704 | 0.9220
SGSR | 0.9615 | 0.9468 | 0.9398 | 0.9208 | 0.9502 | 0.9381 | 0.9431 | 0.8856 | 0.9357
ALSB | 0.9324 | 0.9514 | 0.9275 | 0.9226 | 0.9563 | 0.9089 | 0.9145 | 0.8830 | 0.9246
MRK | 0.9135 | 0.9476 | 0.7765 | 0.8397 | 0.9586 | 0.9169 | 0.9555 | 0.8418 | 0.8938
| GSR-NCR | 0.9642 | 0.9526 | 0.9377 | 0.9224 | 0.9508 | 0.9431 | 0.9505 | 0.8852 | 0.9383
0.3 | BCS | 0.8782 | 0.8997 | 0.8345 | 0.8149 | 0.9299 | 0.8018 | 0.9276 | 0.8266 | 0.8641
BM3D-CS | 0.9587 | 0.9630 | 0.9572 | 0.9111 | 0.9690 | 0.9601 | 0.9714 | 0.8322 | 0.9403
ADS-CS | 0.9733 | 0.9728 | 0.9521 | 0.9408 | 0.9667 | 0.9550 | 0.9697 | 0.9220 | 0.9565
SGSR | 0.9762 | 0.9684 | 0.9600 | 0.9482 | 0.9648 | 0.9676 | 0.9654 | 0.9316 | 0.9603
ALSB | 0.9729 | 0.9746 | 0.9557 | 0.9475 | 0.9732 | 0.9511 | 0.9736 | 0.9226 | 0.9589
MRK | 0.9611 | 0.9687 | 0.8415 | 0.9225 | 0.9727 | 0.9598 | 0.9768 | 0.9040 | 0.9384
| GSR-NCR | 0.9816 | 0.9783 | 0.9664 | 0.9534 | 0.9795 | 0.9799 | 0.9819 | 0.9351 | 0.9695
0.4 | BCS | 0.9068 | 0.9248 | 0.8807 | 0.8747 | 0.9490 | 0.8422 | 0.9479 | 0.8748 | 0.9001
BM3D-CS | 0.9777 | 0.9805 | 0.9758 | 0.9452 | 0.9781 | 0.9803 | 0.9855 | 0.8848 | 0.9635
ADS-CS | 0.9837 | 0.9835 | 0.9726 | 0.9608 | 0.9803 | 0.9763 | 0.9816 | 0.9487 | 0.9734
SGSR | 0.9836 | 0.9793 | 0.9728 | 0.9653 | 0.9759 | 0.9799 | 0.9777 | 0.9570 | 0.9739
ALSB | 0.9827 | 0.9840 | 0.9702 | 0.9638 | 0.9824 | 0.9738 | 0.9840 | 0.9479 | 0.9736
MRK | 0.9795 | 0.9802 | 0.8979 | 0.9539 | 0.9819 | 0.9783 | 0.9873 | 0.9377 | 0.9620
| GSR-NCR | 0.9879 | 0.9867 | 0.9784 | 0.9702 | 0.9862 | 0.9894 | 0.9892 | 0.9609 | 0.9811

Figure 5: CS reconstructed image Leaves with $0.2N$ measurements. (a) Original
image; (b) BCS [11] (PSNR=21.12dB, FSIM=0.753); (c) BM3D-CS [12]
(PSNR=28.13dB, FSIM=0.923); (d) ADS-CS [17] (PSNR=27.88dB, FSIM=0.902); (e)
SGSR [26] (PSNR=28.80dB, FSIM=0.938); (f) ALSB [20] (PSNR= 27.15dB,
FSIM=0.909); MRK [27] (PSNR=27.75dB, FSIM=0.917); GSR-NCR (PSNR=29.03dB,
FSIM=0.943).

Figure 6: CS recovered House images with $0.3N$ measurements. (a) Original
image; (b) BCS [11] (PSNR=32.85dB, FSIM=0.930); (c) BM3D-CS [12]
(PSNR=36.88dB, FSIM=0.969); (d) ADS-CS [17] (PSNR=38.21dB, FSIM=0.967); (e)
SGSR [26] (PSNR=37.37dB, FSIM=0.965); (f) ALSB [20] (PSNR= 38.34dB,
FSIM=0.973); MRK [27] (PSNR=38.35dB, FSIM=0.973); GSR-NCR (PSNR=39.38dB,
FSIM=0.980).

###  4.3 Effect of the number of the best matched patches

We have discussed how to select the best matching patch numbers $c$ for the
performance of the proposed GSR-NCR. Specifically, to investigate the
sensitivity of our method against $c$, two experiments were conducted with
respect to different $c$, ranging from 20 to 160, in the case of 0.2$N$ and
$0.3N$ measurements, respectively. The results with different $c$ are shown in
Fig. 7. It can be seen that all the curves are almost flat, showing the
performance of the proposed GSR-NCR scheme is insensitive to $c$. The best
performance of each case was usually achieved with $c$ in the range [40,80].
Therefore, in this paper $c$ was empirically set to be 60.

Figure 7: Performance comparison with different matched patch numbers $c$ for
image CS reconstruction . (a) PSNR results achieved by different $c$ in the
case of 0.2$N$ measurements. (b) PSNR results achieved by different $c$ in the
case of 0.3$N$ measurements.

Figure 8: Convergence analysis of the proposed GSR-NCR. (a) PSNR results
versus iteration numbers for image CS reconstruction with 0.3$N$ measurements;
(b) PSNR results versus iteration numbers for image CS reconstruction with
0.4$N$ measurements.

###  4.4 Convergence analysis

Since the proposed GSR-NCR model (Eq. (6)) is non-convex, it is difficult to
give its theoretical proof for global convergence. Here, we only provide
empirical evidence to illustrate the good convergence of the proposed CS
reconstruction method. Fig. 8 illustrates the convergent performance of the
proposed GSR-NCR. It shows the curves of the PSNR values versus the iteration
numbers for four test images with $0.3N$ and $0.4N$ measurements,
respectively. One can observe that with the increase of the iteration numbers,
the PSNR curves gradually increase and ultimately become flat and stable,
showing good stability of the proposed non-convex GSR-NCR model.

##  5 Conclusion

In this paper, we proposed a efficient method for image CS reconstruction
using group-based sparse representation model, which is able to more
accurately enforce the local sparsity and nonlocal self-similarity of images
simultaneously in a unified framework. Different from the typical sparsity-
promoting convex $\ell_{1}$ minimization methods, we extend the non-convex
weighted $\ell_{p}$ ($0<p<1$) penalty function on group sparse coefficients of
the data matrix to replace the convex $\ell_{1}$-norm. To reduce the
computational complexity, we learn the principle component analysis (PCA)
based dictionary for each group to substitute for the dictionary with a high
computational complexity learned from natural image dataset. Furthermore, to
make the proposed model tractable and robust, an efficient iterative
shrinkage/thresholding algorithm was adopted to solve the non-convex
minimization problem. Experimental results have shown that the proposed method
not only outperforms many state-of-the-art methods both quantitatively and
qualitatively, but also results in fine stability.

## References

## References

  * [1] Cand s E J, Romberg J, Tao T. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information[J]. IEEE Transactions on information theory, 2006, 52(2): 489-509.
  * [2] Donoho D L. Compressed sensing[J]. IEEE Transactions on information theory, 2006, 52(4): 1289-1306.
  * [3] Cand s E J, Wakin M B. An introduction to compressive sampling[J]. IEEE signal processing magazine, 2008, 25(2): 21-30.
  * [4] Lustig M, Donoho D, Pauly J M. Sparse MRI: The application of compressed sensing for rapid MR imaging[J]. Magnetic resonance in medicine, 2007, 58(6): 1182-1195.
  * [5] Ma J, Le Dimet F X. Deblurring from highly incomplete measurements for remote sensing[J]. IEEE Transactions on Geoscience and Remote Sensing, 2009, 47(3): 792-802.
  * [6] Duarte M F, Davenport M A, Takbar D, et al. Single-pixel imaging via compressive sampling[J]. IEEE signal processing magazine, 2008, 25(2): 83-91.
  * [7] Bajwa W U, Haupt J D, Sayeed A M, et al. Joint source Cchannel communication for distributed estimation in sensor networks[J]. IEEE Transactions on Information Theory, 2007, 53(10): 3629-3653.
  * [8] Figueiredo M A T, Nowak R D, Wright S J. Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems[J]. IEEE Journal of selected topics in signal processing, 2007, 1(4): 586-597.
  * [9] Tropp J A, Gilbert A C. Signal recovery from random measurements via orthogonal matching pursuit[J]. IEEE Transactions on information theory, 2007, 53(12): 4655-4666.
  * [10] Daubechies I, Defrise M, De Mol C. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint[J]. Communications on pure and applied mathematics, 2004, 57(11): 1413-1457.
  * [11] Mun S, Fowler J E. Block compressed sensing of images using directional transforms[C]//Image Processing (ICIP), 2009 16th IEEE International Conference on. IEEE, 2009: 3021-3024.
  * [12] Egiazarian K, Foi A, Katkovnik V. Compressed sensing image reconstruction via recursive spatially adaptive filtering[C]//Image Processing, 2007. ICIP 2007. IEEE International Conference on. IEEE, 2007, 1: I-549-I-552.
  * [13] Bioucas-Dias J M, Figueiredo M A T. A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration[J]. IEEE Transactions on Image processing, 2007, 16(12): 2992-3004.
  * [14] Wang Y, Yang J, Yin W, et al. A new alternating minimization algorithm for total variation image reconstruction[J]. SIAM Journal on Imaging Sciences, 2008, 1(3): 248-272.
  * [15] Becker S, Bobin J, Cand s E J. NESTA: A fast and accurate first-order method for sparse recovery[J]. SIAM Journal on Imaging Sciences, 2011, 4(1): 1-39.
  * [16] Zhang J, Zhao D, Zhao C, et al. Image compressive sensing recovery via collaborative sparsity[J]. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2012, 2(3): 380-391.
  * [17] Dong W, Shi G, Li X, et al. Image reconstruction with locally adaptive sparsity and nonlocal robust regularization[J]. Signal Processing: Image Communication, 2012, 27(10): 1109-1122.
  * [18] Shen Y, Li J, Zhu Z, et al. Image reconstruction algorithm from compressed sensing measurements by dictionary learning[J]. Neurocomputing, 2015, 151: 1153-1162.
  * [19] Zha Z, Liu X, Zhang X, et al. Compressed sensing image reconstruction via adaptive sparse nonlocal regularization[J]. The Visual Computer, 2016: 1-21.
  * [20] Zhang J, Zhao C, Zhao D, et al. Image compressive sensing recovery using adaptively learned sparsifying basis via L0 minimization[J]. Signal Processing, 2014, 103: 114-126.
  * [21] Dong W, Shi G, Li X, et al. Compressive sensing via nonlocal low-rank regularization[J]. IEEE Transactions on Image Processing, 2014, 23(8): 3618-3632.
  * [22] Eslahi N, Aghagolzadeh A. Compressive sensing image restoration using adaptive curvelet thresholding and nonlocal sparse regularization[J]. IEEE Transactions on Image Processing, 2016, 25(7): 3126-3140.
  * [23] Eslahi N, Aghagolzadeh A, Andargoli S M H. Image/video compressive sensing recovery using joint adaptive sparsity measure[J]. Neurocomputing, 2016, 200: 88-109.
  * [24] Sun J, Li H, Xu Z. Deep ADMM-net for compressive sensing MRI[C]//Advances in Neural Information Processing Systems. 2016: 10-18.
  * [25] Chen C, Tramel E W, Fowler J E. Compressed-sensing recovery of images and video using multihypothesis predictions[C]//Signals, Systems and Computers (ASILOMAR), 2011 Conference Record of the Forty Fifth Asilomar Conference on. IEEE, 2011: 1193-1198.
  * [26] Zhang J, Zhao D, Jiang F, et al. Structural group sparse representation for image compressive sensing recovery[C]//Data Compression Conference (DCC), 2013. IEEE, 2013: 331-340.
  * [27] Canh T N, Dinh K Q, Jeon B. Multi-scale/multi-resolution Kronecker compressive imaging[C]//Image Processing (ICIP), 2015 IEEE International Conference on. IEEE, 2015: 2700-2704.
  * [28] Wen B, Ravishankar S, Bresler Y. FRIST flipping and rotation invariant sparsifying transform learning and applications[J]. Inverse Problems, 2017, 33(074007): 074007.
  * [29] Tikhonov A N, Goncharsky A V, Stepanov V V, et al. Numerical methods for the solution of ill-posed problems[M]. Springer Science : Business Media, 2013.
  * [30] Aharon M, Elad M, Bruckstein A. $rmk$-SVD: An algorithm for designing overcomplete dictionaries for sparse representation[J]. IEEE Transactions on signal processing, 2006, 54(11): 4311-4322.
  * [31] Elad M, Aharon M. Image denoising via sparse and redundant representations over learned dictionaries[J]. IEEE Transactions on Image processing, 2006, 15(12): 3736-3745.
  * [32] Mairal J, Bach F, Ponce J, et al. Online dictionary learning for sparse coding[C]//Proceedings of the 26th annual international conference on machine learning. ACM, 2009: 689-696.
  * [33] Zhang Q, Li B. Discriminative K-SVD for dictionary learning in face recognition[C]//Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010: 2691-2698.
  * [34] Wen B, Ravishankar S, Bresler Y. Structured overcomplete sparsifying transform learning with convergence guarantees and applications[J]. International Journal of Computer Vision, 2015, 114(2-3): 137-167.
  * [35] Buades A, Coll B, Morel J M. A non-local algorithm for image denoising[C]//Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. IEEE, 2005, 2: 60-65.
  * [36] Zhang X, Burger M, Bresson X, et al. Bregmanized nonlocal regularization for deconvolution and sparse reconstruction[J]. SIAM Journal on Imaging Sciences, 2010, 3(3): 253-276.
  * [37] Mairal J, Bach F, Ponce J, et al. Non-local sparse models for image restoration[C]//Computer Vision, 2009 IEEE 12th International Conference on. IEEE, 2009: 2272-2279.
  * [38] Zhang J, Zhao D, Gao W. Group-based sparse representation for image restoration[J]. IEEE Transactions on Image Processing, 2014, 23(8): 3336-3351.
  * [39] Dong W, Shi G, Ma Y, et al. Image restoration via simultaneous sparse coding: Where structured sparsity meets gaussian scale mixture[J]. International Journal of Computer Vision, 2015, 114(2-3): 217-232.
  * [40] Zha Z, Liu X, Zhou Z, et al. Image denoising via group sparsity residual constraint[C]//Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on. IEEE, 2017: 1787-1791.
  * [41] Dong W, Zhang L, Shi G, et al. Nonlocally centralized sparse representation for image restoration[J]. IEEE Transactions on Image Processing, 2013, 22(4): 1620-1630.
  * [42] Zha Z, Liu X, Huang X, et al. Analyzing the group sparsity based on the rank minimization methods[J]. arXiv preprint arXiv:1611.08983, 2016.
  * [43] Beck A, Teboulle M. Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems[J]. IEEE Transactions on Image Processing, 2009, 18(11): 2419-2434.
  * [44] Lyu Q, Lin Z, She Y, et al. A comparison of typical ?p minimization algorithms[J]. Neurocomputing, 2013, 119: 413-424.
  * [45] Chartrand R, Wohlberg B. A nonconvex ADMM algorithm for group sparsity with sparse groups[C]//Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013: 6009-6013.
  * [46] Zuo W, Meng D, Zhang L, et al. A generalized iterated shrinkage algorithm for non-convex sparse coding[C]//Proceedings of the IEEE international conference on computer vision. 2013: 217-224.
  * [47] Zha Z, Zhang X, Wu Y, et al. A Comparative Study for the Weighted Nuclear Norm Minimization and Nuclear Norm Minimization[J].
  * [48] Candes E J, Wakin M B, Boyd S P. Enhancing sparsity by reweighted ? 1 minimization[J]. Journal of Fourier analysis and applications, 2008, 14(5): 877-905.
  * [49] Chang S G, Yu B, Vetterli M. Adaptive wavelet thresholding for image denoising and compression[J]. IEEE transactions on image processing, 2000, 9(9): 1532-1546.
  * [50] Zhang L, Zhang L, Mou X, et al. FSIM: A feature similarity index for image quality assessment[J]. IEEE transactions on Image Processing, 2011, 20(8): 2378-2386.
  * [51] Yang J, Wright J, Huang T S, et al. Image super-resolution via sparse representation[J]. IEEE transactions on image processing, 2010, 19(11): 2861-2873.