# Learning-Driven Lossy Image Compression; A Comprehensive Survey

Sonain Jamil, Md. Jalil Piran, and MuhibUr Rahman S. Jamil is with the
Department of Electronics Engineering, Sejong University, Seoul, South Korea
e-mail<EMAIL_ADDRESS>J. Piran is with the Department of Computer
Engineering, Sejong University, Seoul, South Korea e-mail:
<EMAIL_ADDRESS>Rahman is with the Department of Electrical Engineering,
Polytechnique Montreal, Montreal, QC H3T 1J4, Canada e-mail:
<EMAIL_ADDRESS>received Month xx, xxxx; revised Month
xx, xxxx.

###### Abstract

In the realm of image processing and computer vision (CV), machine learning
(ML) architectures are widely applied. Convolutional neural networks (CNNs)
solve a wide range of image processing issues and can solve image compression
problem. Compression of images is necessary due to bandwidth and memory
constraints. Helpful, redundant, and irrelevant information are three
different forms of information found in images. This paper aims to survey
recent techniques utilizing mostly lossy image compression using ML
architectures including different auto-encoders (AEs) such as convolutional
auto-encoders (CAEs), variational auto-encoders (VAEs), and AEs with hyper-
prior models, recurrent neural networks (RNNs), CNNs, generative adversarial
networks (GANs), principal component analysis (PCA) and fuzzy means
clustering. We divide all of the algorithms into several groups based on
architecture. We cover still image compression in this survey. Various
discoveries for the researchers are emphasized and possible future directions
for researchers. The open research problems such as out of memory (OOM),
striped region distortion (SRD), aliasing, and compatibility of the frameworks
with central processing unit (CPU) and graphics processing unit (GPU)
simultaneously are explained. The majority of the publications in the
compression domain surveyed are from the previous five years and use a variety
of approaches.

###### Index Terms:

learned image compression, deep learning, JPEG, end-to-end image compression,
machine learning.

##  I Introduction

In this modern era of big data, the size of the data is the biggest concern
for scientists and researchers. Due to the limited bandwidth of the channel
and limited memory space, there is the requirement of data compression for the
successful transmission and storage of data without losing significant
information [1]. Data compression can be performed in several ways: audio
compression, image compression, video compression, and document compression
[2]. There are three types of information in an image: useful, redundant, and
irrelevant. Irrelevant information can be ignored for the compression of
images. Redundant information is crucial for highlighting details in images,
whereas useful information is neither redundant nor irrelevant. Without
accurate information, we cannot reconstruct or decompress images properly [3].
In image compression, there are two major categories. The first is lossless
image compression, where no information is lost, and the second is lossy image
compression. Lossless image compression techniques are very efficient for
small-size data.

Lossless techniques such as Huffman coding, run-length encoding (RLE),
arithmetic coding, Lempel-Ziv-Welch (LZW) Coding, and JPEG-LS are efficient
for the small data [14]-[15]. The major drawback of the lossless compression
techniques is less compression efficiency than lossy compression techniques.
That is why many researchers are working on image compression using ML.

There are many surveys focused on image compression. Many surveys address the
pros and cons of conventional image compression algorithms based on discrete
cosine transform (DCT), and discrete wavelet transforms (DWT). Shukla et al.
in [5], surveyed the prediction and transform-based image compression
algorithms. This survey highlighted the importance and application of the
prediction and transform-based conventional algorithms for image compression.
These prediction-based algorithms were based on edge detection, gradient, and
block-based prediction. However, transform-based algorithms were based on
wavelets. The survey presented the comparative analysis of the entropy coding.
However, this lacked the discussion about end-to-end based image compression
architectures using ML. In [6], the authors studied the DCT and DWT-based
algorithms. The survey did not address the recently learned image compression
techniques. Likewise, in [13] the authors surveyed several lossy and lossless
algorithms for image compression. The article highlighted the advantages and
drawbacks of the predictive, entropy coding, and discrete Fourier transform-
based image compression frameworks. The survey did not address the ML-based
image compression architectures. Several surveys highlight the importance of
the conventional DCT, DWT, and entropy-based approaches for image compression.
There is a requirement to survey ML-based still image compression techniques.

A plethora of techniques have been proposed by researchers in recent years to
address the problem of still image compression using ML techniques [4]. So,
there is a need to survey ML-based image compression techniques and highlight
the open research problems. In this survey, we focus on recent ML-based image
compression techniques. The following open research questions need to be
addressed.

  * •

Which end-to-end still image compression using ML gives better compression
efficiency?

  * •

Which learned image compression frameworks give a better visual representation
of the reconstructed image?

  * •

Which compression technique saves GPU memory?

  * •

Which compression technique has a fast response time?

This article presents a detailed survey of using ML algorithms for still image
compression considering these questions.

Figure 1: Organization of the paper.

As presented in Fig. 1, the rest of this paper is organized as follows.
Section II presents the related work. In Section III, still image compression
with conventional compression techniques is presented. The use of ML
techniques for image compression is explained in Section IV. Section V gives
an insight into the future research directions. Finally, Section VI draws the
conclusions.

TABLE I: List of acronyms. Acronym |  Meaning
---|---
AEs |  Autoencoders
BGAN+ |  Binary Generative Adversarial Network
BPG |  Better Portable Graphics
bpp |  Bits per pixel
CAEs |  Convolutional Autoencoders
CLIC |  Challenge on Learned Image Compression
CNNs |  Convolutional Neural Networks
C |  Computer Vision
CPU |  Central Processing Unit
DCT |  Discrete Cosine Transform
DNN |  Deep Neural Network
DWT |  Discrete Wavelet Transform
GANs |  Generative Adversarial Networks
GDN |  Generalized Divisive Normalization
GMM |  Gaussian Mixture Module
GPU |  Graphics Processing Unit
HDR |  High Resolution
HEVC |  High Efficiency Video Coding
IGDN |  Inverse Generalized Divisive Normalization
JPEG |  Joint Photographic Expert Group
JPEG XR |  JPEG Extended Range
KL |  Kullback-Leiber
LSTM |  Long Short-Term Memory
LZW |  Lempel-Ziv-Welch
ML |  Machine Learning
MNIST |  Modified National Institute of Standards and Technology
MSE |  Mean Squared Error
MS-SSIM |  Multi-scale Structural Similarity Index Measure
NL |  Non-Local
NLAM |  Non-local Attention Module
NLD |  Non-linear Decoder
NLE |  Non-linear Encoder
NN |  Neural Network
OOM |  Out Of Memory
PCA |  Principal Component Analysis
PNG |  Portable Network Graphics
PSNR |  Peak Signal-to-Noise Ratio
RD |  Rate Distortion
RLE |  Run Length Encoding
RNNs |  Recurrent Neural Networks
SCT |  Stop Code Tolerant
SRD |  Striped Region Distortion
SSIM |  Structural Similarity Index Measure
SWTA-AE |  Stochastic Winner-Take-All Autoencoder
VAEs |  Variational Autoencoders
VVC |  Versatile Video Coding

##  II Related Work

Still image compression has been a concern for researchers for many years.
Initially, image compression was performed using conventional frameworks, and
with the introduction of ML, algorithms used ML models for image compression.
Several surveys focused on image compression algorithms. In [7], the authors
presented the comprehensive study of the image compression models based on the
neural networks. This study lacked the discussion of other ML-based end-to-end
image compression methods. The survey included old algorithms. Shum et al. in
[8], surveyed the DCT and DWT-based image compression and representation
algorithms. The authors in [5] and [9] conducted a study on conventional DCT
and DWT-based frameworks.

Similarly, the authors in [10] surveyed the entropy-based image compression
algorithms. [6] and [11] surveyed the DCT and DWT-based image compression
techniques in 2014. Likewise in 2017, Setyaningsih et al. [12], surveyed the
hybrid compression techniques. In [13], the authors surveyed the predictive,
entropy encoding, as well as discrete Fourier transform-based image
compression architectures. In [14] and [15] the authors surveyed the
conventional image compression algorithms. In [14] the authors surveyed
arithmetic entropy coding techniques while in [15], the authors presented the
survey of image compression standards such as joint photographic experts group
(JPEG), JPEG-2000, portable network graphics (PNG), WebP. Table II presents
the comprehensive summary of the related surveys.

TABLE II: Summary of the image compression surveys. Survey |  Year |  Scope of the Architecture Surveyed |  Contributions and limitations
---|---|---|---
|  |  AE |  VAE |  CNN |  RNN |  GAN |
[7] |  1999 |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  Survey of NN based frameworks only
[8] |  2003 |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  Survey of DCT and DWT-based image representation and compression algorithms
[5] |  2010 |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  Survey of predictive and transform based image compression methods
[9] |  2012 |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  Survey of entropy based encoding, JPEG, JPEG2000 methods
[10] |  2013 |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  Survey of entropy based encoding methods
[6] |  2014 |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  Survey of DCT and DWT based architectures only
[11] |  2014 |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  Survey of JPEG and DCT based architectures
[12] |  2017 |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  Survey of hybrid DCT, DWT based compression techniques
[13] |  2018 |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  Survey of predictive, entropy coding and discrete Fourier transform based architectures
[14] |  2019 |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  Survey of arithmetic encoding based architectures only
[15] |  2021 |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  Survey of conventional JPEG, JPEG2000, WebP, PNG based architectures
This survey |  2021 |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  Survey of end-to-end ML based image compression frameworks, New outlook to the open research gaps

After analyzing Table II, it is evident that all the existing surveys deal
with the conventional compression techniques. However, there is a plethora of
learning-driven lossy image compression frameworks and these architectures
have limitations, pros and cons. There is a need for to survey learning-driven
lossy image compression models.In this survey, we provide an in-depth detailed
discussion regarding still lossy image compression. We present analysis,
comparison, pros and cons of the state-of-the-art techniques used for still
lossy image compression. We outline several open research problems and trends
in this research field for substantial future research.

The upcoming section explains the still image compression.

##  III Still Image Compression

The concept of image compression started in 1992 when Wallace proposed the
JPEG standard for still image compression [16]. In the JPEG compression
standard, DCT is used, which can be calculated as follows [17].

$F(u,v)=\frac{1}{\sqrt{2N}}C(u)C(v)\sum_{x=0}^{N-1}\sum_{y=0}^{N-1}f(x,y)\dots\hfill\atop
cos(\frac{(2x+1)u\pi}{N})cos(\frac{(2y+1)v\pi}{N}),$ (1)

where $x$ is the pixel row, for the integer $0\leqslant x\leqslant N$, $y$ is
the pixel column, for the integer $0\leqslant y\leqslant N$, $F_{u,v}$ is the
reconstructed approximate coefficient at coordinates $(u,v)$, $f_{x,y}$ is the
reconstructed pixel value at coordinates $(x,y)$ and $C(u)$ is

$C(u)=\begin{cases}\frac{1}{\sqrt{N}},&\text{for }u=0\\\ 1,&\text{for
}otherwise.\end{cases}$ (2)

The basic framework of JPEG compression and decompression is shown in Fig. 2.

Figure 2: Block diagram of JPEG compression and decompression.

JPEG was widely used for still image compression until an updated, and
improved version of JPEG known as JPEG-2000 was proposed in 2001 [18].
JPEG-2000 uses DWT instead of DCT. The basic framework of JPEG-2000 is shown
in Fig. 3. The entropy encoding scheme used in JPEG-2000 is Huffman encoding
[19], RLE [20] or arithmetic encoding [21].

Figure 3: Block diagram of JPEG-2000 compression and decompression.

JPEG and JPEG-2000 standards have been widely used. Later on, researchers
developed JPEG extended range (JPEG XR) [22], and JPEG XT [23] to overcome the
limitations of the JPEG standard. However, both these algorithms failed due to
hardware incompatibility. Recently, WebP [24], better portable graphics (BPG)
[25] and JPEG XL [26] algorithm were proposed for the compression of high
resolution (HDR) images. Recently, high-efficiency video coding (HEVC)
displayed the most impressive performance. In [27], the authors explained the
use of HEVC for high-depth medical image compression. They also proposed a
novel algorithm to reduce the complexity of HEVC. In [28], the authors
presented a comparative analysis of all these algorithms. These conventional
algorithms still have limitations. ML has addressed several issues like visual
quality improvement for image compression. The next section presents learning-
driven lossy image compression techniques.

##  IV Learning-Driven Lossy Image Compression

ML has a significant influence on every field of research. There has been a
revolution in image processing due to CNN’s compiling property of features
extractions. Inspired by this, many ML architectures have been proposed to
compress images. We categorize architectures concerning ML models.

###  IV-A Compression with AE

The most widely used learning-driven lossy image compression model was
proposed by Ballé in 2016 [29]. The backbone of this model is AE [38]. An AE
can be divided into three parts input, bottleneck, and output. A bottleneck in
CAE is the place where latent space is represented.

AEs gained popularity in image compression as the basic purpose of an AE is to
reduce the dimensions of the input image. Ollivier et al. in [39], used AE to
compress data by minimizing the code length. This method used the generative
property of AE. In [40], the authors used AE with Kalman filter for the
compression of the modified national institute of standards and technology
(MNIST) images. The model failed to perform better for red-green-blue (RGB)
images. Similarly, in [41], the authors used the Kodak dataset to train AE and
achieved better performance than JPEG, JPEG-2000, and WebP. In [42] and [43],
the authors used AE for the compression of images. In [44], the authors
proposed AE for the image compression. They used generalized divisive
normalization (GDN) and inverse generalized divisive normalization (IGDN)
layers instead of rectified linear unit (ReLU) layers to boost up the training
process. Cheng et al. in [45] and Alexandre et al. in [46], deployed AE for
the compression of high resolution image.

###  IV-B Compression with VAE

VAE is a version of NN and belongs to the probabilistic graphical models and
variational Bayesian methods. Compression with VAE incorporates mean and
variance for latent space distributions. The results achieved by VAE are
better than simple AE. A simple illustration of image compression using VAE is
shown in Fig. 4.

Figure 4: Image Compression with VAE.

Several researchers used VAE for still image compression. In [47], the authors
demonstrated the use of VAE having non-linear transform and uniform quantizer
for image compression. Challenge one learned image compression (CLIC) dataset
used for training the model. The model was complex due to massive training
parameters. Recently, Chen et al. in [48], proposed a VAE based architecture
for high resolution image compression. The authors demonstrated a non-local
attention module (NLAM) to boost the training process. However, the model
complexity increased significantly. Similarly, Larsen et al. in [49], proposed
a VAE to compress images. In [50], the authors used VAE to compress images and
achieved bits per pixel (bpp) of 4.10. However, the model was very complex due
to its architecture.

###  IV-C Compression with CNN

CNN has prime importance in image processing due to its feature extraction
characteristics. CNN is also used for image compression. The simple
illustration of the image compression and artifacts reduction is shown in Fig.
5.

Figure 5: Image Compression and artifacts reduction with CNN.

In [51]–[55], the authors used CNNs to compress images. These CNN-based image
compression models outperformed JPEG and JPEG-2000 in structural similarity
index measure (SSIM) and peak signal-to-noise ratio (PSNR). In [56], the
authors proposed a CNN-based framework for the compression of gigapixel
images. They used unsupervised learning for the training of the neural
network. Similarly, in [57], the authors proposed a state-of-the-art optimized
deep CNN-based technique for the compression of the still images. The proposed
method used an attention module to compress certain regions of the images with
different bits adaptively. The approach outperformed JPEG and JPEG-2000.
However, the results of Ballé [29] are better.

In [58], the authors also used CNN for the compression of high-resolution
images. They used CLIC 2018 dataset for training and achieved 7.81% BD-Rate
reduction over BPG and JPEG-2000. In [59], the authors introduced a new
concept of multi-spectral transform for the multi-spectral image compression
using CNN. It achieved better compression efficiency than state-of-the-art
anchors like JPEG-2000. Likewise, in [60], the authors proposed a data
reduction CNN for the compression of hyperspectral images. Similarly, in [61],
the authors proposed a CNN-based end-to-end compression architecture for the
multi-spectral image compression. The authors extracted spatial features by
partitioning and achieved better performance in terms of PSNR than JPEG-2000.
Similarly, [62]–[65] also used CNN based frameworks.

###  IV-D Compression with RNN

RNN can be utilized for image compression too. A simple RNN-based image
compression architecture has some convolutional layers, GDN layer, RNN
modules, binarized convolutional layers, and IGND layer. A simple illustration
image compression using RNN is shown in Fig. 6 as proposed in [66].

Figure 6: Image Compression with RNN.

Covell et al. proposed RNN based compression architecture using stop code
tolerant (SCT) to train the model. They achieved bpp and PSNR of 0.25 and 27dB
for Kodak and imageNet datasets, respectively. The dataset used in the method
was Celeba dataset [67]. Likewise, in [68], an RNN-based method was proposed
to compress still images. The training dataset used in this article was Kodak
and achieved bpp of 0.5 and SSIM of 0.77. The model’s performance was superior
to JPEG, JPEG-2000, and WebP.

Toderici et al. in [69], demonstrated the use of RNN with entropy encoding to
compress images. They achieved bpp of 0.5, PSNR of 33.59dB, SSIM of 0.8933,
and multi-scale structural similarity index (MS-SSIM) of 0.9877. In [70], the
authors proposed a DL-based end-to-end image compression model to compress
images based on semantic analysis. The authors used RNN for the compression
purpose and achieved better results than the conventional compression
techniques like JPEG. The compression rate at 0.75 bpp was obtained 32 using
this method. Furthermore, [71]-[74] also used RNN for image compression and
achieved a better compression ratio than anchors such as JPEG and JPEG-2000.

###  IV-E Compression with GAN

GANs are algorithmic architectures. GANs use two NN, pitting one against the
other to generate new, synthetic data instances that can pass for actual data.
These have the generator part and discriminator portion. GANs are also
utilized for image compression. The simple flow diagram of image compression
using GAN is shown in Fig. 7, where encoder and decoders are GAN modules.

Figure 7: Image Compression with GAN.

Torfason et al. in [75], illustrated the use of GAN for the compression and
classification of semantic data. Similarly, [76] proposed an extreme image
compression architecture using GAN. They used two types of modules for image
compression. The first module was generative compression, where no semantic
label maps were required. The type applied was selective generative
compression, where semantic label maps were used. They provided insight
towards the full resolution image compression and targeted low bit-rate, i.e.,
less than 0.1 bpp. In [77], the authors demonstrated the use of unified binary
GAN (BGAN+) for image compression and image retrieval. The model achieved
better than JPEG and JPEG-2000. The visual quality of the reconstructed image
was much improved than JPEG and JPEG-2000 at low bit-rate such as 0.15 bpp. In
[78]-[83], the authors proposed GAN-based architectures for the image
compression and achieved high compression efficiency. The drawback of the GAN-
based architecture is the cost of deployment.

###  IV-F Compression with PCA

PCA is also used for image compression. In [84], the authors utilized vector
quantization and PCA for the compression of hyperspectral images. The
technique achieved better performance than JPEG-2000. Similarly, in [85], the
authors used PCA and DCT to compress hyperspectral images. The used PSNR
metric for the evaluation of the quality of the image. The method applies to
hyperspectral images only.

In [86], the authors used weighted PCA and JPEG-2000 to compress hyperspectral
images. They demonstrated that the hybrid approach of weighted PCA and
JPEG-2000 achieved better PSNR than the JPEG and JPEG-2000. Similarly, the
authors of [87] proposed PCA for lossy compression and achieved quality
reconstructed image. However, the method depended on the number of components,
and the performance decreased with the increase of PCA components.

###  IV-G Compression with fuzzy means clustering

Martino et al., in [88] proposed a fast algorithm to improve the performance
of the multilevel fuzzy transform image compression method [89]. The authors
demonstrated that the reconstructed image’s visual quality.

Moreover, in [90] the authors proposed a conventional technique of using a
curve-fitting novel hyperbolic function for the compression. The main
advantages of this approach were: strengthening the edges of the image,
removing the blocking effect, improving the SSIM, and increasing the PSNR up
to 20 dB.

TABLE III: Comparison of the learning-driven lossy image compression models. Research |  Technique |  bpp |  Performance Parameters |  Contributions and limitations
---|---|---|---|---
|  |  |  PSNR [dB] |  SSIM |  MS-SSIM |
[39] |  AE |  – |  – |  – |  – |  Generative property of AEs, by minimizing code-length, limited to theoretical perspective.
[40] |  AE |  – |  – |  – |  – |  The combination of the non-recurrent architecture of three-layers with kalman filter, limited to grayscale images.
[41] |  AE |  0.4 |  29 |  0.83 |  0.94 |  Compressive AE with MSE loss function trained by residual network and non-differentiability of the quantization noise reduced using rounding based quantization, visual artifacts problem at low bit rate.
[42] |  AE |  0.2 |  – |  – |  0.92 |  Vector quantization preferred over scalar quantization, joint optimization of learning of latent space representation, worst performance than BPG.
[43] |  AE |  0.1 |  31.5 |  – |  – |  Global rate-distortion (RD) constraint in a framework stochastic winner-take-all autoencoder (SWTA-AE), worst performance than JPEG-2000, changing strides of convolutional layers harms NN.
[44] |  AE |  0.2 |  31 |  – |  – |  GDN as well as IGDN layers, worst performance than HEVC intra.
[45] |  AE |  1.2 |  33 |  – |  – |  Parametric ReLU activation function, to produce feature maps with low dimensionality, uses a CAE instead of a transform and an inverse transform, along with some max-pooling-up-sampling layers, comparable performance with Ball’ e model for grayscale images only.
[46] |  AE |  0.126 |  29.30 |  – |  0.924 |  Feature map coding block with Importance Net, Uses skip connections AEs and trained with loss function of MSE and MS-SSIM, less PSNR than BPG.
[47] |  VAE |  0.15 |  30.76 |  – |  0.955 |  Non-linear encoder (NLE) transform, uniform quantizer, non-linear decoder (NLD) transform, and post-processing module form a VAE framework of article.
[48] |  VAE |  0.2 |  30 |  – |  0.7768 |  Non local (NL) operations to consider local as well as global context, better performance than Ball’ e model.
[49] |  VAE |  – |  – |  – |  – |  Kullback-Leiber (KL) loss function, limited to image generation.
[50] |  VAE |  4.10 |  – |  – |  – |  Recurrent VAE for learning latent space, limited to conceptual compression.
[51] |  CNN |  0.025 |  – |  – |  0.925 |  Initial decoded image and side information require more complexity and storage space, cloud based model, visual artifacts at low bpp.
[52] |  CNN |  0.25 |  30 |  – |  – |  Coarse to fine hyperprior based model, lacks parallel decoding.
[53] |  CNN |  0.0726 |  23.93 |  0.8118 |  – |  Loss function consisting of MSE, adversarial and layer wise loss.
[54] |  CNN |  0.519 |  33.62 |  – |  0.981 |  The Lagrange multiplier for joint rate-distortion, considerable run-time complexity due to the attention modules.
[55] |  CNN |  0.2 |  31 |  – |  0.7878 |  Gaussian mixture module (GMM) alogwith JointIQ-Net to outperform JPEG-2k, BPG, versatile video coding (VVC) Intra scheme.
[56] |  CNN |  – |  – |  – |  – |  Two step CNN based framework and the unsupervised training, limited to histopathology images.
[57] |  CNN |  0.5 |  – |  0.77 |  – |  Better performance than JPEG and JPEG-2k but worst performance than Ballé. [29]
TABLE IV: Comparison of the learning-driven lossy image compression models. Research |  Technique |  bpp |  Performance Parameters |  Contributions and limitations
---|---|---|---|---
|  |  |  PSNR [dB] |  SSIM |  MS-SSIM |
[58] |  CNN |  0.2 |  – |  – |  0.93 |  Trained with perceptual as well as adversarial loss for the generation of sharp details, outperforms JPEG, JPEG-2000, BPG and WebP.
[59] |  CNN |  – |  – |  – |  – |  Multi-spectral transform for the multi-spectral image compression.
[60] |  CNN |  – |  – |  – |  – |  Introduction of data reduction block to save GPU memory.
[61] |  CNN |  – |  – |  – |  – |  Feature extraction by partitioning, better performance than JPEG-2000.
[67] |  RNN |  0.25 |  27 |  – |  – |  RNN trained using SCT, limited to adaptive encoding.
[68] |  RNN |  0.5 |  – |  0.77 |  – |  Focuses on 32x32 size thumbnail images, better than JPEG, JPEG-2000 and WebP.
[69] |  RNN |  0.5 |  33.59 |  0.8933 |  0.9877 |  Long short-term memory (LSTM)-based progressive encoding of thumbnail pictures $(32\times 32)$, decent compression performance at low bit rates, pixel RNN for entropy coding.
[70] |  RNN |  0.75 |  32 |  – |  – |  Semantic analysis, better performance than JPEG.
[75] |  GAN |  0.0983 |  28.54 |  0.85 |  0.973 |  For semantic understanding combined training for compression and classification.
[76] |  GAN |  0.033 |  – |  – |  – |  Full-resolution image compression, applicable only when semantic labels are available.
[77] |  BGAN+ |  0.15 |  – |  – |  0.95 |  Reconstructed image without distorting the small objects present in the image at low bit-rates such as 0.15bpp.
[84] |  PCA |  – |  – |  – |  – |  Better performance than JPEG-2000, limited to hyper-spectral data.
[88]-[89] |  Multilevel Fuzzy Transform |  – |  – |  – |  – |  Reconstructed image with better visual quality, limited to maximum of 1024$\times$1024 image resolutions.
[90] |  Curve-fitting with hyperbolic function |  – |  20 |  – |  – |  Removed blocking effects and improved SSIM, results are dependent on block size and compression ratio.

Table III and IV present a detailed summary of the techniques used for image
compression. Fig. 8 shows the percentage of the different ML architectures
present in literature for still image compression. It is evident that 24% of
architectures in the literature are based on AEs while 12% are based on VAEs.
The majority of the architectures are based on CNN 32%.

Figure 8: Percentage of architectures present for still image compression.

###  IV-H Comparative Analysis of State-of-the-art Techniques

This subsection presents a comparative analysis of state-of-the-art techniques
in terms of compression ratio, computational time, PSNR, MS-SSIM, bpp, and RD-
rate. Fig. 9 shows the minimum bpp of the learning-driven lossy image
compression models. Fig. 10 and Fig. 11 show the corresponding MS-SSIM and
PSNR achieved at minimum bpp by reference respectively. Ayzik et al. [51]
achieved PSNR of 30dB and MS-SSIM of 0.925 at the minimum bpp of 0.025. The
comparison of the three context models as presented in [91] shows that the
encoding time for all three context models, which are masked context [92],
causal context, and causal context plus causal global prediction model. The
reason behind this is that the encoding can be performed in parallel while the
decoding time for all three models is different as the masked context model
takes less time for decoding while the combined causal context and causal
global prediction take approximately 39 seconds, as explained in [91]. The
comparison of the encoding and decoding time of all three models is shown in
Fig. 12. However, the decoding time of masked context-based CAE architectures
used for still image compression is less than the other causal and global
context models. The encoding time for all three types of architectures is the
same. The use of masked convolution [93] has also impacted the visual
representation of the reconstructed image. The reconstructed image has better
visual quality in [91] than other state-of-the-art techniques.

Figure 9: Minimum bpp achieved by different models. Figure 10: MS-SSIM
achieved at minimum bpp by different models. Figure 11: PSNR achieved at
minimum bpp by different models. Figure 12: Comparison of encoding and
decoding time of masked context, causal context and causal context plus causal
global prediction.

Table V presents the comparison of the performance of the recent deep
learning-based learned image compression models.

TABLE V: Summary of the performance of the end-to-end image compression frameworks. Research |  Model |  Minimum bpp |  PSNR (dB) |  MS-SSIM
---|---|---|---|---
[41] |  AE |  0.4 |  29 |  0.94
[42] |  AE |  0.2 |  – |  0.92
[47] |  VAE |  0.15 |  30.76 |  0.955
[48] |  VAE |  0.2 |  30 |  0.7768
[48] |  VAE (loss function MSE) |  0.1276 |  34.63 |  0.9738
[48] |  VAE (loss function MS-SSIM) |  0.1074 |  32.54 |  0.9759
[53] |  CNN |  0.0726 |  23.93 |  0.8118
[54] |  CNN |  0.519 |  33.62 |  0.981
[55] |  CNN |  0.2 |  31 |  0.7878
[69] |  RNN |  0.5 |  33.59 |  0.9877
[75] |  GAN |  0.0983 |  28.54 |  0.973
[94] |  Hyperprior based AE |  0.304 |  34.4 |  –
[95] |  Autoregressive Model |  0.01921 |  18.95 |  –
[96] |  Invertible NN (MSE loss function) |  0.130 |  31.51 |  0.972
[96] |  Invertible NN (MS-SSIM loss function) |  0.124 |  28.01 |  0.978
[97] |  Conditional Autoencoder |  0.1697 |  32.2332 |  0.9602
[98] |  Asymmetric Gained VAE (loss function MSE) |  0.107 |  30.68 |  0.93
[98] |  Asymmetric Gained VAE (loss function MS-SSIM) |  0.109 |  27.76 |  0.94

We also compare the run time complexity of the various benchmark learning-
driven algorithms with standard codecs. Table VI shows the comprehensive
comparison.

TABLE VI: Run-time complexity of various learning-driven algorithms with standard codecs. Codecs |  Encoding Time (ms) |  Decoding Time (ms)
---|---|---
JPEG |  18.600 |  13.000
JPEG-2000 |  367.400 |  80.400
WebP |  67.000 |  83.700
[29] |  242.120 |  338.090
[30] |  64.700 |  12.100
[31] |  8.600 |  9.900
[32] |  3.500 |  4.000
[33] |  75.120 |  73.230
[34] |  79.500 |  17.400
[35] |  42.000 |  32.000
[36] |  74.000 |  984.000
[37] |  24.000 |  32.000
[69] |  1606.900 |  1079.300

###  IV-I Impact of Context-Based Entropy Modeling

In current learned lossy image compression algorithms, the rate loss is
generally the entropy of the codes. To reduce entropy and improve joint rate-
distortion performance, a precise estimate of the probabilistic distribution
of the codecs is critical. Most of the learning-driven architectures have
utilized context-based entropy modeling to improve the RD performance of the
codecs. The architectures consider local as well as non-local (NL) contexts.

The use of global similarity among pixels in image denoising was initially
proposed using NL approaches. Deep neural network (DNN)-based image processing
approaches then incorporate it into DNNs to use the global information and
improve performance in various tasks. The authors in [99] investigated pixel
self-similarity and presented the NL means for image denoising based on a
content weighted NL average of all pixels in the picture. Wang et al. in [100]
defined the non-local operation as a uniform block, which they used in DNNs to
mix local and NL data for object detection. In [101], Liu et al. suggested an
NL recurrent network for image restoration that includes NL operations into a
recurrent network. The authors of [102] used the NL operation to create
attention masks to capture long-range dependence between pixels and pay
greater attention to the challenging sections of picture restoration.

Li et al. [103] proposed learning context to model entropy block in the end-
to-end image compression framework. The analysis, as well as synthesis block,
utilized U-Net [104]-[105] block for encoding and decoding operations. The NL
operation was proposed in entropy block to incorporate the global
similarities.

All the learning-driven lossy image compression models have limitations and
research gaps. The upcoming section highlights the research gaps of learning-
driven lossy image compression frameworks.

##  V Research Gaps and Future Directions

There are several techniques present for still image compression. All
frameworks have their pros and cons. After a detailed survey, we find that
there are still research gaps. All ML-based architectures have problems of
SRD, parallel acceleration, and OOM.

###  V-A SRD in Reconstructed Image

SRD is the striped regions distortion in the reconstructed image using ML-
based architectures. SRD in the reconstructed image is one of the open
research problems in ML-based image compression techniques. There is no
technique available in the literature that addresses this problem. There is a
need to solve this problem in future research.

###  V-B Standardization of Architecture

The second research gap is the standardization of the end-to-end compression
architecture. It is very challenging for the researchers. The deep learning
architectures present in the literature work well only when the training is
performed on GPU and testing is also performed using GPU. Similarly, when the
training is done using CPU and testing is performed on CPU, the results are
better. However, if the training is performed on the GPU and the testing is
done on the CPU and vice versa, the image cannot be reconstructed correctly.
There is a requirement for the standard generalized model that gives an
optimal solution to this problem.

###  V-C Parallel Acceleration

The third research gap is due to serial decoding; parallel acceleration of the
autoregressive entropy model is impossible. So, there is a requirement for a
framework to address the problem of parallel acceleration. In [106], the
authors proposed the learned image block-based framework to address this
problem. The framework has block artifacts problems for meager bit rates. So,
it is still an open research problem.

###  V-D OOM Problem

The fourth research gap is given limited GPU resources; full-resolution
inference frequently produces OOM problems, especially for high-resolution
pictures. Block partition is a good option for dealing with the concerns
mentioned above, but it introduces new challenges in terms of decreasing
duplication between blocks and removing block effects.

###  V-E Aliasing Effect In Reconstructed Image

The fifth research gap is the learning-driven lossy image compression
frameworks’ aliasing effect in the reconstructed image. The reconstructed
images of the CNN and CAE-based architectures have variations in the
directionality of the patterns, which is called aliasing. This problem needs
to be addressed.

These are five major issues in still image compression that can contribute to
future research. The upcoming section concludes the survey.

##  VI Conclusion

Images have three types of information, including helpful information,
redundant information, and useless information, in the era of the modern
world, where the visual quality of images plays a vital role. On the other
hand, the is the requirement for more memory to store as well as more
bandwidth is required to transmit those high-resolution images. To resolve the
problem of memory storage, we perform image compression. Initially, image
compression was based on conventional arithmetic and entropy encoding
techniques such as JPEG and JPEG-2000. However, since the evolution of ML many
several learned image compression models have been proposed in the literature.
This paper surveyed several learned image compression techniques based on AEs,
VAEs, CNNs, RNNs, GANs, PCA, and fuzzy means clustering. The majority of the
architectures are based on CNNs and AEs. These models achieve better
compression efficiency than JPEG-2000, which is an anchor in image
compression. We also highlighted four significant research gaps in ML-based
image compression models. Those gaps are the SRD problems, architecture
standardization, parallel acceleration, aliasing, and OOM. These problems are
yet to be addressed in still image compression.

## References

  * [1] U. Jayasankar, V. Thirumal and D. Ponnurangam, “A survey on data compression techniques: From the perspective of data quality, coding schemes, data type and applications,” Journal of King Saud University - Computer and Information Sciences, vol. 33, no. 2, pp. 119–140, Feb. 2021.
  * [2] Chen S, Zhang S, Zheng X, Ruan X., “Layered adaptive compression design for efficient data collection in industrial wireless sensor networks,” Journal of Network and Computer Applications. 2019 Mar 1;129:37-45.
  * [3] Md. Jalil Piran, SM Riasulislam. QV Pham, DY Suh, and Z. Han, “Multimedia Communication over Cognitive Radio Networks from QoS/QoE Perspective: A Comprehensive Survey,” Journal of Network and Computer Applications, vol. 172, pp. 1-44, Dec. 2020.
  * [4] Gupta O, Raskar R., “Distributed learning of deep neural network over multiple agents,” Journal of Network and Computer Applications. 2018 Aug 15;116:1-8.
  * [5] J. Shukla, M. Alwani and A. K. Tiwari, “A survey on lossless image compression methods,” in Proc. of 2010 2nd International Conference on Computer Engineering and Technology, vol. 6, pp. V6–136, 2010.
  * [6] M. Rehman, M. Sharif and M. Raza, “Image compression: A survey,” Research Journal of Applied Sciences, Engineering and Technology vol. 7, no. 4, pp. 656–672, Jul. 2014.
  * [7] J. Jiang, “Image compression with neural networks-a survey,” Signal processing: image Communication, vol. 14, no. 9, pp. 737–760, Jul. 1999.
  * [8] H.-Y. Shum, S. B. Kang and S.-C. Chan, “Survey of image-based representations and compression techniques,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 11, pp. 1020-1037, Nov. 2003.
  * [9] J. Vrindavanam, S. Chandran and G. K. Mahanti, “A survey of image compression methods,” in Proc. of International Conference and Workshop on Emerging Trends in Technology, pp. 12–17. 2012.
  * [10] G. Vijayvargiya, S. Silakari and R. Pandey, “A survey: various techniques of image compression,” (IJCSIS) International Journal of Computer Science and Information Security, vol. 11, no. 10, Oct. 2013.
  * [11] A. M. Raid, W. M. Khedr, M. A. El-Dosuky and W. Ahmed, “Jpeg image compression using discrete cosine transform-A survey,” International Journal of Computer Science & Engineering Survey (IJCSES), vol.5, no.2, Apr. 2014.
  * [12] E. Setyaningsih and A. Harjoko, “Survey of hybrid image compression techniques,” International Journal of Electrical and Computer Engineering, vol. 7, no. 4, pp. 2206, Aug. 2017.
  * [13] A. J. Hussain, A. Al-Fayadh and N. Radi, “Image compression techniques: A survey in lossless and lossy algorithms,” Neurocomputing, vol. 300, pp. 44–69, Jul. 2018.
  * [14] M. Rahman and M. Hamada, “Lossless image compression techniques: A state-of-the-art survey,” Symmetry, vol. 11, no. 10, pp. 1274, Oct. 2019.
  * [15] M. Rahman, M. Hamada and J. Shin, “The Impact of State-of-the-Art Techniques for Lossless Still Image Compression,” Electronics, vol. 10, no. 3, pp. 360, Feb. 2021.
  * [16] G. K. Wallace, “The JPEG still picture compression standard,” IEEE Transactions on Consumer Electronics, vol. 38, no. 1, pp. xviii-xxxiv, Feb. 1992.
  * [17] E. Y. Lam and J. W. Goodman, “A mathematical analysis of the DCT coefficient distributions for images,” IEEE Transactions on Image Processing, vol. 9, no. 10, pp. 1661–1666, Oct. 2000.
  * [18] A. Skodras, C. Christopoulos and T. Ebrahimi, “The JPEG 2000 still image compression standard,” IEEE Signal Processing Magazine, vol. 18, no. 5, pp. 36–58, Sep. 2001.
  * [19] H. Kasban and S. Hashima, “Adaptive Radiographic Image Compression Technique using Hierarchical Vector Quantization and Huffman Encoding,” J Ambient Intell Human Comput, vol. 10, pp. 2855–2867, Jul. 2019.
  * [20] Y. Qin, Z. Wang, H. Wang and Q. Gong, “Binary image encryption in a joint transform correlator scheme by aid of run-length encoding and QR code,” Optics & Laser Technology, vol. 103, pp. 93–98, Jul. 2018.
  * [21] L. Xiang, Y. Li, W. Hao, P. Yang and X. Shen, “Reversible natural language watermarking using synonym substitution and arithmetic coding,” Computers, Materials & Continua, vol. 55, no.3, pp. 541–559, Jun. 2018.
  * [22] F. Dufaux, G. J. Sullivan and T. Ebrahimi, “The JPEG XR image coding standard [Standards in a Nutshell],” IEEE Signal Processing Magazine, vol. 26, no. 6, pp. 195–204, Nov. 2009.
  * [23] A. Artusi, R. K. Mantiuk, T. Richter et al. “Overview and evaluation of the JPEG XT HDR image compression standard,” Journal of Real-Time Image Processing, vol. 16, pp. 413–428, Apr. 2019.
  * [24] G. Ginesu, M. Pintus and D. D. Giusto, “Objective assessment of the WebP image coding algorithm,” Signal Processing: Image Communication, vol. 27, no. 8, pp. 867–874, Sep. 2012.
  * [25] S. P. Mohanty, E. Kougianos and P. Guturu, “SBPG: Secure Better Portable Graphics for Trustworthy Media Communications in the IoT,” IEEE Access, vol. 6, pp. 5939–5953, Jan. 2018.
  * [26] J. Cho, O. -J. Kwon and S. Choi, “Improvement of JPEG XL Lossy Image Coding Using Region Adaptive DCT Block Partitioning Structure,” IEEE Access, vol. 9, pp. 113213–113225, Aug. 2021.
  * [27] S. S. Parikh, D. Ruiz, H. Kalva, G. Fernandez-Escribano and V. Adzic, “High Bit-Depth Medical Image Compression With HEVC,” IEEE Journal of Biomedical and Health Informatics, vol. 22, no. 2, pp. 552–560, Mar. 2018.
  * [28] T. H. Mandeel, M. Imran Ahmad, N. A. A. Khalid and M. N. Md Isa, “A Comparative Study on Lossless compression mode in WebP, Better Portable Graphics (BPG), and JPEG XL Image Compression Algorithms,” in Proc. of 2021 8th International Conference on Computer and Communication Engineering (ICCCE), pp. 17–22, 2021.
  * [29] J. Ballé, V. Laparra and E. P. Simoncelli, “End-to-end optimized image compression,” arXiv preprint, 2016\. arXiv:1611.01704.
  * [30] J. Ballé, D. Minnen, S. Singh, S. J. Hwang and N. Johnston, “Variational image compression with a scale hyperprior,” in Proc. of International Conf. on Learning Representations, 2018\.
  * [31] O. Rippel and L. Bourdev, “Real-Time Adaptive Image Compression,” in Proc. of the 34th Int. Conf. on Machine Learning, vol. 70, pp. 2922–2930, 2017.
  * [32] D. Mishra, S. K. Singh and R. K. Singh, “Wavelet-based deep auto encoder-decoder (wdaed)-based image compression,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 31, no. 4, pp. 1452–1462, Apr. 2021.
  * [33] C. Cai, L. Chen, X. Zhang and Z. Gao, “End-to-End Optimized ROI Image Compression,” IEEE Trans. on Image Processing, vol. 29, pp. 3442–3457, 2020.
  * [34] C. Cai, L. Chen, X. Zhang and Z. Gao, “Efficient Variable Rate Image Compression with Multi-scale Decomposition Network,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 29, no. 12, pp. 3687–3700, Dec. 2019.
  * [35] A. K. Ashok and N. Palani, “Autoencoders with Variable Sized Latent Vector for Image Compression,” in Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) Workshops, 2018\.
  * [36] M. Li, K. Ma, J. You, D. Zhang and W. Zuo, “Efficient and Effective Context-Based Convolutional Entropy Modeling for Image Compression,” IEEE Transactions on Image Processing, vol. 29, pp. 5900–5911, Apr. 2020.
  * [37] M. Li, W. Zuo, S. Gu, J. You and D. Zhang, “Learning Content-Weighted Deep Image Compression,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no.10, pp. 3446–3461, Oct. 2021.
  * [38] V. Alves de Oliveira, M. Chabert, T. Oberlin, C. Poulliat, M. Bruno, C. Latry, M. Carlavan, S. Henrot, F. Falzon, R. Camarero, “Reduced-Complexity End-to-End Variational Autoencoder for on Board Satellite Image Compression,” Remote Sensing, vol. 13, no.3, pp. 447, Jan. 2021.
  * [39] Y. Ollivier, “Auto-encoders: reconstruction versus compression,” CoRR, 2014\.
  * [40] A. Sento, “Image Compression with Auto-encoder Algorithm using Deep Neural Network (DNN),” in Proc. of 2016 Management and Innovation Technology Int. Conf. (MITicon). IEEE, pp. MIT–99, 2016.
  * [41] L. Theis, W. Shi, A. Cunningham and F. Huszar, “Lossy image compression with compressive autoencoders,” in Proc. of International Conf. on Learning Representations, 2017\.
  * [42] E. Agustsson, F. Mentzer, M. Tschannen, L. Cavigelli, R. Timofte, L. Benini and L. V. Gool, “Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations,” in Proc. of Advances in Neural Information Processing Systems, pp. 1141–1151, 2017.
  * [43] T. Dumas, A. Roumy and C. Guillemot, “Image compression with Stochastic Winner-Take-All Auto-Encoder,” in Proc. of 2017 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp. 1512–1516, 2017.
  * [44] T. Dumas, A. Roumy and C. Guillemot, “Autoencoder Based Image Compression: Can the Learning be Quantization Independent?” in Proc. of 2018 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 1188–1192, 2018.
  * [45] Z. Cheng, H. Sun, M. Takeuchi and J. Katto, “Deep Convolutional AutoEncoder-based Lossy Image Compression,” in Proc. of 2018 Picture Coding Symposium (PCS), IEEE, pp. 253–257, 2018.
  * [46] D. Alexandre, C.-P. Chang, W.-H. Peng and H.-M. Hang, “An autoencoder-based learned image compressor: Description of challenge proposal by nctu,” in Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition Workshops, pp. 2539–2542, 2018.
  * [47] L. Zhou, C. Cai, Y. Gao, S. Su and J. Wu, ”Variational Autoencoder for Low Bit-rate Image Compression,” in Proc. of The IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
  * [48] T. Chen, H. Liu, Z. Ma, Q. Shen, X. Cao and Y. Wang, “End-to-End Learnt Image Compression via Non-Local Attention Optimization and Improved Context Modeling,” IEEE Transactions on Image Processing, vol. 30, pp. 3179–3191, Feb. 2021.
  * [49] A. B. L. Larsen, S. K. Sønderby, H. Larochelle and O. Winther, “Autoencoding beyond pixels using a learned similarity metric,” in Proc. of The 33rd Int. Conf. on Machine Learning (ICML’16), vol. 48, pp. 1558–1566, 2016.
  * [50] K. Gregor, F. Besse, D. J. Rezende, I. Danihelka and D. Wierstra, “Towards Conceptual Compression,” in Proc. of Advances In Neural Information Processing Systems, pp. 3549–3557, 2016.
  * [51] S. Ayzik and S. Avidan, “Deep image compression using decoder side information,” in Proc. of Computer Vision-ECCV 2020: 16th European Conf., Glasgow, UK, pp. 699–714, 2020.
  * [52] Y. Hu, W. Yang, Z. Ma and J. Liu, “Learning end-to-end lossy image compression: A benchmark,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Mar. 2021.
  * [53] S. K. Raman, A. Ramesh, V. Naganoor, S. Dash, G. Kumaravelu and H. Lee, “CompressNet: Generative Compression at Extremely Low Bitrates,” in Proc. of The IEEE Winter Conf. on Applications of Computer Vision, pp. 2325–2333, 2020.
  * [54] Z. Cheng, H. Sun, M. Takeuchi and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, pp. 7939–7948, 2020.
  * [55] J. Lee, S. Cho and M. Kim, “An end-to-end joint learning scheme of image compression and quality enhancement with improved entropy minimization,” arXiv preprint arXiv:1912.12817, 2020.
  * [56] D. Tellez, G. Litjens, J. van der Laak and F. Ciompi, “Neural Image Compression for Gigapixel Histopathology Image Analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 2, pp. 567–578, Feb. 2021.
  * [57] W. Li, W. Sun, Y. Zhao, Z. Yuan and Y. Liu, “Deep Image Compression with Residual Learning,” Applied Sciences, vol. 10, no. 11, pp. 4023, Jun. 2020.
  * [58] H. Liu, T. Chen, Q. Shen, T. Yue and Z. Ma, “Deep Image Compression via End-to-End Learning,” in Proc. of Computer Vision Pattern Recognition, vol. 06, 2018.
  * [59] J. Li and Z. Liu, “Multispectral transforms using convolution neural networks for remote sensing multispectral image compression,” Remote Sensing, vol. 11, no. 7, pp. 759, Mar. 2019.
  * [60] T. M. Zeegers, D. M. Pelt, T. van Leeuwen, R. van Liere and K. J. Batenburg, “Task-Driven Learned Hyperspectral Data Reduction Using End-to-End Supervised Deep Learning,” Journal of Imaging, vol. 6, no. 12, pp. 132, Dec. 2020.
  * [61] F. Kong, K. Hu, Y. Li, D. Li and D. Zhao, “Spectral-Spatial Feature Partitioned Extraction Based on CNN for Multispectral Image Compression,” Remote Sensing, vol. 13, no. 1, pp. 9, Dec. 2020.
  * [62] J. Cai, Z. Cao and L. Zhang, “Learning a single tucker decomposition network for lossy image compression with multiple bits-per-pixel rates,” IEEE Trans. on Image Processing vol. 29, pp. 3612–3625, Jan. 2020.
  * [63] A. Prakash, N. Moran, S. Garber, A. DiLillo and J. Storer, “Semantic Perceptual Image Compression using Deep Convolution Networks,” in Proc. of 2017 Data Compression Conf. (DCC), pp. 250–259, 2017.
  * [64] P. Akyazi and T. Ebrahimi, “Learning-based image compression using convolutional autoencoder and wavelet decomposition,” in Proc. of IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019\.
  * [65] Y. Xue and J. Su, “Attention Based Image Compression Post-Processing Convolutional Neural Network,” in Proc. of CVPR Workshops, 2019\.
  * [66] K. Islam, L. M. Dang, S. Lee and H. Moon, “Image Compression With Recurrent Neural Network and Generalized Divisive Normalization,” in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 1875-1879, 2021.
  * [67] M. Covell, N. Johnston, D. Minnen, S. Jin Hwang, J. Shor, S. Singh, D. Vincent and G. Toderici, “Target-Quality Image Compression with Recurrent, Convolutional Neural Networks.” ArXiv, abs/1705.06687, 2017.
  * [68] G. Toderici, S. M. O’Malley, S. J. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell and R. Sukthankar, “Variable Rate Image Compression with Recurrent Neural Networks,” CoRR, vol. abs/1511.06085, 2016.
  * [69] G. Toderici, D. Vincent, N. Johnston, S. Jin Hwang, D. Minnen, J. Shor and M. Covell, “Full Resolution Image Compression with Recurrent Neural Networks,” in Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, pp. 5306–5314, 2017.
  * [70] C. Wang, Y. Han and W. Wang, “An End-to-End Deep Learning Image Compression Framework Based on Semantic Analysis,” Applied Sciences, vol. 9, no. 17, pp. 3580, Sep. 2019.
  * [71] A. Punnappurath and M. S. Brown, “Learning raw image reconstruction-aware deep image compressors,” IEEE Trans. on pattern analysis and machine intelligence, vol. 42, no. 4, pp. 1013–1019, Apr. 2020.
  * [72] N. Johnston, D. Vincent, D. Minnen, M. Covell, S. Singh, T. Chinen, S. J. Hwang, J. Shor and G. Toderici, “Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networks,” in Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, pp. 4385–4393, 2018.
  * [73] Y. Hu, W. Yang, M. Li and J. Liu, “Progressive Spatial Recurrent Neural Network for Intra Prediction,” IEEE Trans. on Multimedia, vol. 21, no. 12, pp. 3024–3037, Dec. 2019.
  * [74] A. G. Ororbia, A. Mali, J. Wu, S. O’Connell, W. Dreese, D. Miller, C. L. Giles, “Learned Neural Iterative Decoding for Lossy Image Compression Systems,” in Proc. of 2019 Data Compression Conf. (DCC), pp. 3–12, 2019.
  * [75] R. Torfason, F. Mentzer, E. Augustsson, M. Tschannen, R. Timofte and L. V. Gool, “Towards Image Understanding from Deep Compression Without Decoding,” in Proc. of Int. Conf. on Learning Representations, 2018\.
  * [76] E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte and L. V. Gool, “Generative adversarial networks for extreme learned image compression,” in Proc. of the IEEE/CVF International Conference on Computer Vision, pp. 221–231, 2019.
  * [77] J. Song, T. He, L. Gao, X. Xu, A. Hanjalic and T. H. Shen, “Unified Binary Generative Adversarial Network for Image Retrieval and Compression,” International Journal of Computer Vision, vol. 128, no. 8, pp. 2243–2264, Feb. 2020.
  * [78] X. Zhang and X. Wu, “Near-lossless L-infinity constrained Multi-rate Image Decompression via Deep Neural Network,” CoRR, 2018\.
  * [79] B. Kang, S. Tripathi and T. Q. Nguyen, “Toward Joint Image Generation and Compression using Generative Adversarial Networks,” arXiv preprint, 2019, arXiv:1901.07838.
  * [80] L. Wu, K. Huang and H. Shen, “A GAN-based tunable image compression system,” in Proc. of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2334–2342, 2020.
  * [81] D. J. Im, C. D. Kim, H. Jiang and R. Memisevic, “Generating images with recurrent adversarial networks,” CoRR, vol. abs/1602.05110, 2016.
  * [82] E. M. Tolunay and A. Ghalayini, “Generative Neural Network Based Image Compression,” 2018.
  * [83] L. Galteri, L. Seidenari, M. Bertini and A. Del Bimbo, “Deep Generative Adversarial Compression Artifact Removal,” in Proc. of the IEEE Int. Conf. on Computer Vision, pp. 4826–4835, 2017.
  * [84] D. Báscones, C. González and D. Mozos, “Hyperspectral image compression using vector quantization, PCA and JPEG2000,” Remote sensing, vol. 10, no. 6, pp. 907, Jun. 2018.
  * [85] R. J. Yadav and M. S. Nagmode, “Compression of hyperspectral image using PCA–DCT technology,” in Proc. of Innovations in Electronics and Communication Engineering, pp. 269–277, 2018.
  * [86] A. C. Karaca and M. K. G” ull” u, “Target preserving hyperspectral image compression using weighted PCA and JPEG2000,” in Proc. of International Conference on Image and Signal Processing, pp. 508–516, 2018.
  * [87] A. H. Abbas, A. Arab and J. Harbi, “Image compression using principal component analysis,” Mustansiriyah Journal of Science, vol. 29, no. 2, Jan. 2018.
  * [88] F. Di Martino, I. Perfilieva and S. Sessa, “A Fast Multilevel Fuzzy Transform Image Compression Method,” Axioms, vol. 8, no. 4, pp. 135, Dec. 2019.
  * [89] F. Di Martino and S. Sessa, “Multi-level fuzzy transforms image compression,” Journal of Ambient Intelligence and Humanized Computing, vol. 10, no. 7, pp. 2745–2756, Jul. 2019.
  * [90] W. Khalaf, D. Zaghar and N. Hashim, “Enhancement of Curve-Fitting Image Compression Using Hyperbolic Function,” Symmetry, vol. 11, pp. 291, Feb. 2019.
  * [91] Z. Guo, Z. Zhang, R. Feng and Z. Chen, “Causal Contextual Prediction for Learned Image Compression,” IEEE Transactions on Circuits and Systems for Video Technology, Jun. 2021, doi: 10.1109/TCSVT.2021.3089491.
  * [92] D. Minnen, J. Balle, and G. D. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in Proc. of Advances in Neural Information Processing Systems, pp. 10771–10780, 2018.
  * [93] X. He, Q. Hu, X. Zhang, C. Zhang, W. Lin and X. Han, “Enhancing HEVC Compressed Videos with a Partition-Masked Convolutional Neural Network,” in Proc. of 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 216–220, 2018.
  * [94] Y. Hu, W. Yang, and J. Liu, “Coarse-to-fine hyper-prior modeling for learned image compression.” in Proc. of the AAAI Conference on Artificial Intelligence, vol. 34, no. 07, pp. 11013–11020, 2020.
  * [95] D. Minnen and S. Singh, “Channel-Wise Autoregressive Entropy Models for Learned Image Compression,” in Proc. of 2020 IEEE International Conference on Image Processing (ICIP), 2020, pp. 3339–3343.
  * [96] Y. Xie, K. L. Cheng and Q. Chen, “Enhanced invertible encoding for learned image compression,” in Proc. of the 29th ACM International Conference on Multimedia, 2021, pp. 162–170.
  * [97] Y. Choi, M. El-Khamy and J. Lee, “Variable rate deep image compression with a conditional autoencoder,” in Proc. of the IEEE/CVF International Conference on Computer Vision, pp. 3146–3154, 2019.
  * [98] Z. Cui, J. Wang, S. Gao, T. Guo, Y. Feng and B. Bai, “Asymmetric Gained Deep Image Compression With Continuous Rate Adaptation,” in Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10532–10541, 2021.
  * [99] A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” in Proc. of IEEE Conf. Comput. Vis. Pattern Recog., vol. 2. IEEE, pp. 60–65, 2005.
  * [100] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in Proc. of IEEE Conf. Comput. Vis. Pattern Recog., pp. 7794–7803, 2018.
  * [101] D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang, “Non-local recurrent network for image restoration,” Neural Inf. Process. Syst, pp. 1673–1682, Dec. 2018.
  * [102] Y. Zhang, K. Li, K. Li, B. Zhong, and Y. Fu, “Residual non-local attention networks for image restoration,” in Proc. of Int. Conf. Learning Representations, 2019\.
  * [103] M. Li, K. Zhang, J. Li, W. Zuo, R. Timofte and D. Zhang, “Learning Context-Based Nonlocal Entropy Modeling for Image Compression,” IEEE Transactions on Neural Networks and Learning Systems, Aug. 2021.
  * [104] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Proc. of Int. Conf. Medical Image Computing Computer-assisted Intervention. Springer, pp. 234–241, 2015.
  * [105] X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-DenseUNet: hybrid densely connected UNet for liver and tumor segmentation from CT volumes,” IEEE Trans. Medical Imaging, vol. 37, no. 12, pp. 2663–2674, Dec. 2018.
  * [106] Y. Wu, X. Li, Z. Zhang, X. Jin and Z. Chen, “Learned Block-based Hybrid Image Compression,” IEEE Transactions on Circuits and Systems for Video Technology, Oct. 2021.


# Deflection of light by a compact object with electric charge and magnetic
dipole in Einstein-Born-Infeld gravity

Jin Young Kim 111E-mail address<EMAIL_ADDRESS>Department of Physics,
Kunsan National University, Kunsan 54150, Korea

###### Abstract

We consider the bending of light around a compact astrophysical object with
both the electric field and the magnetic field in Einstein-Born-Infeld theory.
From the null geodesic of a light ray passing a massive object with electric
charge and magnetic dipole, the effective metric was obtained from the light
cone condition reflecting the nonlinear electromagnetic effects. We found the
asymptotic form of the effective metric up to the first order in gravitational
constant $G$ and Born-Infeld parameter $1/\beta^{2}$ on the equatorial plane.
Then we compute the bending angle of light from the geodesic equation. The
result includes particular cases where only one type of field is present
taking the appropriate limits.

Deflection of light, nonlinear electrodynamics, charged black hole, magnetar

##  I Introduction

Deflection of light by a massive object is one of the important result of
general relativity. It is the starting point of gravitational lensing theory.
The shadow image of black hole observed by the Event Horizon Telescope has
become an important topic to study the black hole physics [1, 2, 3]. In
general relativity, any physical quantity with energy and momentum can affect
the spacetime structure and can cause the bending of light. When a light ray
is passing around a massive object with electomagnetic field, the
electromagnetic field can also affect the bending angle. The bending angle can
be computed from the geodesic equation using the metric obtained from the
Einstein-Maxwell equation.

Astronomical objects with extremely strong electromagnetic field attract
interest. The magnetic field on the surface of a magnetar is estimated up to
the order of $10^{11}{\rm T}$ [4, 5]. This field strength is the estimated
lower bound of the Born-Infeld parameter [6] where nonlinear electromagnetic
effects can be considered. When the electric or magnetic field made by a
compact astrophysical object is strong enough, one can consider the bending of
light accommodating the nonlinear electromagnetic effects. In Einstein-Maxwell
theory, the presence of electromagnetic field affects only the energy-momentum
tensor and the electromagnetic field equations are linear. So the null
geodesic of the electromagnetic wave is the same as the null geodesic of the
gravitational wave. However, in Einstein-Born-Infeld theory, the null geodesic
of the electromagnetic wave is not the same as that of the gravitational wave
due to the nonlinear electromagnetic effects [7].

In the generalized Born-Infeld-type nonlinear electrodynamics, one can compute
weak bending angles of light by a charged black hole [8] and a magnetar [9]
using the effective null geodesic. These computations correspond to the cases
where only one type of electromagnetic field (purely electric or purely
magnetic) is present. Recently it is observed that black holes can have
magnetic field by the Event Horizon Telescope [10]. It seems of interest to
consider the bending of light by compact astrophysical objects which have both
the electric field and the magnetic field. For example, one can compute the
bending angle of light by a charged black hole with magnetic field or a
magnetar with electric charge in Einstein-Born-Infeld theory.

The organization of the paper is as follows. In Sec. II, we consider the
geometrical aspects of a massive object with both the electric charge and the
magnetic dipole moment. We outline how to obtain the effective metric and the
electromagnetic four-potential in Einstein-Born-Infeld theory. We found the
static axially symmetric solution of the effective metric on the equatorial
plane of a magnetic dipole. In Sec. III, using the geodesic equation on the
equatorial plane, we compute the bending angle of light up to the first order
in gravitational constant $G$ and Born-Infeld parameter $1/\beta^{2}$. We show
that the result includes particular cases where only one type of field is
present. In Sec. IV, we summarize and discuss.

##  II Effective null geodesic of the electromagnetic wave

The Einstein-Born-Infeld action is described by [11, 12]

$S=\int d^{4}x\sqrt{-g}\left(\frac{R}{16\pi}+{\cal L}\right),$ (1)

where ${\cal L}$ is

${\cal
L}=\beta^{2}\left(1-\sqrt{1+\frac{2S}{\beta^{2}}-\frac{P^{2}}{\beta^{4}}}\right).$
(2)

Here $\beta$ is the classical Born-Infeld parameter characterizing the
possible maximum value of the field strength, $S$ and $P$ are Lorentz-
invariants defined by

$S=\frac{1}{4}F_{\mu\nu}F^{\mu\nu}=\frac{1}{2}({\bf B}^{2}-{\bf
E}^{2}),~{}~{}~{}P=\frac{1}{4}F_{\mu\nu}{\tilde{F}}^{\mu\nu}={\bf E}\cdot{\bf
B},$ (3)

$F_{\mu\nu}=\partial_{\mu}A_{\nu}-\partial_{\nu}A_{\mu}$ is the field strength
tensor, and
${\tilde{F}}_{\mu\nu}=\frac{1}{2}\epsilon_{\mu\nu\alpha\beta}F^{\alpha\beta}$
is the dual tensor. We use the unit system with
$1/4\pi\epsilon_{0}=\mu_{0}/4\pi=c=1$. However, we keep the gravitational
constant $G$ and use it as the perturbation parameter.

The equations of motion are obtained by varying the action with respect to
$g_{\mu\nu}$ and $A_{\mu}$ with the Bianchi identity

$R_{\mu\nu}-\frac{1}{2}g_{\mu\nu}R=8\pi GT_{\mu\nu},$ (4)
$\nabla_{\mu}\left[\frac{1}{\sqrt{1+\frac{2S}{\beta^{2}}-\frac{P^{2}}{\beta^{4}}}}\left(F^{\mu\nu}-\frac{P}{\beta^{2}}{\tilde{F}}^{\mu\nu}\right)\right]=0,$
(5)

where $\nabla_{\mu}$ is the covariant derivative and $T_{\mu\nu}$ is the
energy momentum tensor given by

$T_{\mu\nu}=\beta^{2}\left(1-\sqrt{1+\frac{2S}{\beta^{2}}-\frac{P^{2}}{\beta^{4}}}\right)g_{\mu\nu}+\frac{1}{\sqrt{1+\frac{2S}{\beta^{2}}-\frac{P^{2}}{\beta^{4}}}}F_{\mu}^{~{}\rho}\left(F_{\rho\nu}-\frac{P}{\beta^{2}}{\tilde{F}}_{\rho\nu}\right).$
(6)

In the limit $\beta\to\infty$, the above equations reduce to the Einstein-
Maxwell equations

$R_{\mu\nu}-\frac{1}{2}g_{\mu\nu}R=8\pi
G\left(F_{\mu}^{~{}\rho}F_{\rho\nu}-\frac{1}{4}F_{\rho\sigma}F^{\rho\sigma}g_{\mu\nu}\right),$
(7) $\nabla_{\mu}F^{\mu\nu}=0.$ (8)

We want to find the metric tensor and electromagnetic potential by a compact
object having both the electric charge and the magnetic dipole. Because of the
axial symmetry, we use the cylindrical coordinates, $x^{\mu}=(t,r,z,\phi)$,
taking the direction of magnetic dipole as $z$-axis. When a compact object has
only one type of electromagnetic field, the metric tensor has only diagonal
elements because the Poynting vector is zero. When it has both the electric
and the magnetic fields, there are off-diagonal elements in the energy-
momentum tensor so that the metric tensor also has off-diagonal elements. We
take the following form of the metric tensor and the electromgnetic four-
potential

$g_{\mu\nu}=\pmatrix{e^{\rho}&0&0&\omega\cr 0&-e^{\lambda}&0&0\cr
0&0&-e^{\lambda}&0\cr\omega&0&0&-r^{2}e^{-\nu}},$ (9)
$A_{\mu}=(\Phi,0,0,-\psi),$ (10)

where $\rho,\lambda,\nu,\omega,\Phi$, and $\psi$ are functions of $r$ and $z$
only.

Substituting Eqs. (9) and (10) into Eqs. (4) and (5), we have coupled
nonlinear differential equations of six unknown functions,
$\rho,\lambda,\nu,\omega,\Phi$, and $\psi$. The boundary conditions for the
metric and the electromagnetic four-potential should be as follows. At
infinity, the metric functions become asymptotically flat and the
electromagnetic field should vanish. In the limit where the electric charge
and the magnetic dipole moment is zero, the metric should approach the
Schwarzschild metric. In the limit where the mass is zero and
$\beta\rightarrow\infty$, the metric is flat everywhere and the potentials in
Eq. (10) should be the four-potential in flat space given by

$\Phi=\frac{q}{X},~{}~{}~{}\psi=\mu\frac{r^{2}}{X^{3}},$ (11)

where $q$ is the Coulomb charge, $\mu$ is the magnetic dipole moment, and
$X=\sqrt{r^{2}+z^{2}}$ is the spherical distance. The electromagnetic field
tensor can be written as

$F_{\mu\nu}=\partial_{\mu}A_{\nu}-\partial_{\nu}A_{\mu}=\pmatrix{0&-\Phi_{r}&-\Phi_{z}&0\cr\Phi_{r}&0&0&-\psi_{r}\cr\Phi_{z}&0&0&-\psi_{z}\cr
0&\psi_{r}&\psi_{z}&0},$ (12)

where the subscripts $r$ and $z$ denote the partial derivatives.

Finding the complete analytic solution of Eqs. (9) and (10) is not easy.
However, in Einstein-Maxwell theory, the perturbative solution in powers of
$G$ was obtained by Martin and Prechett [13] from Eqs. (7) and (8). Up to the
first order in $G$, the series solutions are

$\displaystyle e^{\rho}$ $\displaystyle=$ $\displaystyle
1-\frac{2GM}{X}+\frac{Gq^{2}}{X^{2}}+\frac{G\mu^{2}z^{2}}{X^{6}},$ (13)
$\displaystyle e^{\lambda}$ $\displaystyle=$ $\displaystyle
1+\frac{2GM}{X}-\frac{Gq^{2}z^{2}}{X^{2}}-\frac{G\mu^{2}(r^{4}-6r^{2}z^{2}+2z^{4})}{2X^{8}},$
(14) $\displaystyle e^{-\nu}$ $\displaystyle=$ $\displaystyle
1+\frac{2GM}{X}-\frac{Gq^{2}}{X^{2}}-\frac{G\mu^{2}z^{2}}{X^{6}},$ (15)
$\displaystyle\omega$ $\displaystyle=$ $\displaystyle-\frac{Gq\mu
r^{2}}{X^{4}},$ (16) $\displaystyle\Phi$ $\displaystyle=$
$\displaystyle\frac{q}{X}\left[1-\frac{GM}{X}+\frac{Gq^{2}}{3X^{2}}-\frac{G\mu^{2}(4r^{2}-z^{2})}{35X^{6}}\right],$
(17) $\displaystyle\psi$ $\displaystyle=$ $\displaystyle\frac{\mu
r^{2}}{X^{3}}\left[1+\frac{GM}{2X}+\frac{Gq^{2}}{5X^{2}}\right].$ (18)

In Einstein-Maxwell theory, the electromagnetic wave and the gravitational
wave follow the same geodesic made by mass, charge, and magnetic dipole.
However, in Einstein-Born-Infeld theory, the null geodesic of the
electromagnetic wave is different from the null geodesic of the gravitational
wave due to the nonlinear coupling with the background electromagnetic field.
We will find the perturbative null geodesic of the electromagnetic wave using
Eqs. (13)-(18).

In Einstein-Born-Infeld gravity, the effective metric for photon can be
obtained from the light cone condition [7, 14, 15, 16, 17] as

$\left(g^{\mu\nu}+\frac{{\cal L}_{SS}}{{\cal
L}_{S}}F^{\mu\alpha}F_{\alpha}^{~{}\nu}\right)k_{\mu}k_{\nu}=0,$ (19)

where $g^{\mu\nu}$ is the metric function of the gravitational wave, $k_{\mu}$
is the four-vector of the electromagnetic wave, ${\cal L}_{S}=\partial{\cal
L}/\partial S$, and ${\cal L}_{SS}=\partial^{2}{\cal L}/\partial S^{2}$. Thus
one can find the effective metric from

${\tilde{g}}^{\mu\nu}=g^{\mu\nu}+\frac{{\cal L}_{SS}}{{\cal
L}_{S}}F^{\mu\alpha}F_{\alpha}^{~{}\nu}.$ (20)

Because the classical Born-Infeld parameter is the possible maximum value of
the electromagnetic field strength, in the weak field region where the
electric field and the magnetic field are not strong enough compared with
$\beta$, one can take

$\frac{{\cal L}_{SS}}{{\cal
L}_{S}}=-\frac{1}{\beta^{2}}\left(1+\frac{2S}{\beta^{2}}-\frac{P^{2}}{\beta^{4}}\right)^{-1}\rightarrow-\frac{1}{\beta^{2}}.$
(21)

With this approximation we can compute the effective metric from

${\tilde{g}}^{\mu\nu}=g^{\mu\nu}-\frac{1}{\beta^{2}}F^{\mu\alpha}F_{\alpha}^{~{}\nu}=g^{\mu\nu}+\delta
g^{\mu\nu},$ (22)

where we defined $\delta
g^{\mu\nu}\equiv-(1/{\beta^{2}})F^{\mu\alpha}F_{\alpha}^{~{}\nu}$.

The details how to find the effective metric are given in Appendix A and the
nonzero components of the covariant metric tensor on the equatorial plane
($z=0$) can be written as

$\displaystyle{\tilde{g}}_{00}$ $\displaystyle=$ $\displaystyle
1-\frac{2GM}{r}+\frac{Gq^{2}}{r^{2}}-\frac{1}{\beta^{2}}\frac{q^{2}}{r^{4}},$
(23) $\displaystyle{\tilde{g}}_{03}$ $\displaystyle=$
$\displaystyle{\tilde{g}}_{30}=-\frac{Gq\mu}{r^{2}}+\frac{1}{\beta^{2}}\frac{q\mu}{r^{4}},$
(24) $\displaystyle{\tilde{g}}_{11}$ $\displaystyle=$
$\displaystyle-\left(1+\frac{2GM}{r}-\frac{G\mu^{2}}{2r^{4}}-\frac{1}{\beta^{2}}\frac{q^{2}}{r^{4}}+\frac{1}{\beta^{2}}\frac{\mu^{2}}{r^{6}}\right),$
(25) $\displaystyle{\tilde{g}}_{22}$ $\displaystyle=$
$\displaystyle-\left(1+\frac{2GM}{r}-\frac{G\mu^{2}}{2r^{4}}\right),$ (26)
$\displaystyle{\tilde{g}}_{33}$ $\displaystyle=$
$\displaystyle-r^{2}\left(1+\frac{2GM}{r}-\frac{Gq^{2}}{r^{4}}+\frac{1}{\beta^{2}}\frac{\mu^{2}}{r^{6}}\right).$
(27)

In the above metric, terms with $G$ originated from the energy-momentum tensor
$T_{\mu\nu}$ and terms with $1/\beta^{2}$ originated from the non-linear
electrodynamic effect. For $q=\mu=0$, the above metric reduces to the
Schwarzschild metric. For $\mu=0$, it reduces to the leading-order Born-Infeld
black hole metric and reduces to the Reissner-Nordstrom metric in the limit
$\beta\rightarrow\infty$. For $q=0$, it reduces to the metric on the equator
of a magnetar [9]. Note that the non-diagonal element ${\tilde{g}}_{03}$
appears when both the electric charge and the magnetic dipole moment are
nonzero. As mentioned before this term appears because the Poynting vector is
nonzero. This metric form is quite similar to the metric of the Kerr-Newman
black hole.

Because the observed universe is charge neutral and no charged black hole has
ever been observed, we cannot compare the relative strength of the charge term
with the mass term. However we can compare the relative strength of the dipole
term with the mass term from the observed data on magnetars. Let us estimate
the order-of-magnitude contribution to the metric function by the mass term
and the magnetic dipole term. For simplicity we consider ${\tilde{g}}_{11}$.
Taking $q=0$ and restoring the units, we have

$-{\tilde{g}}_{11}=1+\frac{2GM}{c^{2}r}-\frac{G}{c^{4}}\frac{\mu_{0}}{4\pi}\frac{\mu^{2}}{r^{4}}+\frac{1}{\beta^{2}}\left(\frac{\mu_{0}}{4\pi}\frac{\mu}{r^{3}}\right)^{2}.$
(28)

One can replace the magnetic dipole moment with the magnetic induction on the
surface ($r=r_{s}$) of the magnetar using
$B_{s}=\frac{\mu_{0}}{4\pi}\frac{\mu}{r_{s}^{3}}$

$-{\tilde{g}}_{11}=1+\frac{2GM}{c^{2}r_{s}}\frac{r_{s}}{r}-\frac{G}{c^{4}}\left(\frac{\mu_{0}}{4\pi}\right)^{-1}B_{s}^{2}r_{s}^{2}\left(\frac{r_{s}}{r}\right)^{4}+\frac{B_{s}^{2}}{\beta^{2}}\left(\frac{r_{s}}{r}\right)^{6}.$
(29)

Because the estimated maximum magnetic field of magnetar is of the order
$10^{11}{\rm T}$, we take $\beta\sim 10^{12}{\rm T}$ and estimate the orders
of magnitude. Taking the mass and radius as those of a typical neutron star,
$M=1.4M_{\rm sun}=2.8\times 10^{30}{\rm kg}$ and $r_{s}=10{\rm km}$, we have

$\frac{2GM}{c^{2}r_{s}}\sim 0.42,$ (30)
$\frac{G}{c^{4}}\left(\frac{\mu_{0}}{4\pi}\right)^{-1}B_{s}^{2}r_{s}^{2}\sim
8.2\times 10^{-8},$ (31) $\frac{B_{s}^{2}}{\beta^{2}}\sim 10^{-2}.$ (32)

The metric contribution by the magnetic dipole originated from the energy
momentum tensor is negligible compared with the mass term. However, the metric
contribution originated from the nonlinear electromagnetic effect is not
negligible when a light ray is passing close to the magnetar.

##  III Geodesic equation and weak bending angle

In this section we compute the bending angle of light by an astrophysical
compact object with electric charge and magnetic dipole. We consider the
trajectory of a light ray passing on the equatorial plane described by the
following form of metric [18, 19, 20],

$ds^{2}=A(r)dt^{2}-B(r)dr^{2}-C(r)d\phi^{2}+2D(r)dtd\phi,$ (33)

where

$\displaystyle A(r)$ $\displaystyle=$ $\displaystyle
1-\frac{2GM}{r}+\frac{Gq^{2}}{r^{2}}-\frac{1}{\beta^{2}}\frac{q^{2}}{r^{4}},$
(34) $\displaystyle B(r)$ $\displaystyle=$ $\displaystyle
1+\frac{2GM}{r}-\frac{G\mu^{2}}{2r^{4}}-\frac{1}{\beta^{2}}\frac{q^{2}}{r^{4}}+\frac{1}{\beta^{2}}\frac{\mu^{2}}{r^{6}},$
(35) $\displaystyle C(r)$ $\displaystyle=$ $\displaystyle
r^{2}\left(1+\frac{2GM}{r}-\frac{Gq^{2}}{r^{4}}+\frac{1}{\beta^{2}}\frac{\mu^{2}}{r^{6}}\right),$
(36) $\displaystyle D(r)$ $\displaystyle=$
$\displaystyle-\frac{Gq\mu}{r^{2}}+\frac{1}{\beta^{2}}\frac{q\mu}{r^{4}}.$
(37)

The geodesic equation can be described by the Lagrangian

${\cal L}_{\rm eff}=\frac{1}{2}g_{\mu\nu}{\dot{x}}^{\mu}{\dot{x}}^{\nu},$ (38)

where the over dot means the derivative with respect to the affine parameter
$p$ which describes the trajectory. Because the metric given by Eq. (33) does
not depend on $t$ and $\phi$, there are two constants of motion from the
associated Killing vectors. The conserved quantities corresponding to the
energy and the angular momentum are

$\frac{\partial{\cal L}_{\rm
eff}}{\partial{\dot{t}}}=A{\dot{t}}+D{\dot{\phi}}=\varepsilon,$ (39)
$\frac{\partial{\cal L}_{\rm
eff}}{\partial{\dot{\phi}}}=D{\dot{t}}-C{\dot{\phi}}=-\ell.$ (40)

One can take $\varepsilon=1$ by absorbing the constant into the redefinition
of the affine parameter. Then, from Eqs. (39) and (40), we have

$\displaystyle{\dot{t}}$ $\displaystyle=$
$\displaystyle\frac{C-D\ell}{AC+D^{2}},$ (41) $\displaystyle{\dot{\phi}}$
$\displaystyle=$ $\displaystyle\frac{D+A\ell}{AC+D^{2}}.$ (42)

Inserting Eqs. (41) and (42) into the null geodesic condition $ds^{2}=0$, we
have

${\dot{r}}=\sqrt{\frac{C-D\ell-A\ell^{2}}{B(AC+D^{2})}}.$ (43)

Figure 1: Bending angle $\Delta\varphi$ of light ray on the equatorial plane
of the magnetic dipole. The distance of closest approach $r_{0}$ is the
minimum of the radial coordinate $r$, $b$ is the impact parameter and $r_{s}$
is the radius of the compact object with magnetic dipole.

On the equatorial plane, one can find the bending angle (Fig. 1) from [18]

$\Delta\varphi=2|\varphi(r_{0})-\varphi_{\infty}|-\pi,$ (44)

where $r_{0}$ is the distance of closest approach and

$\varphi(r_{0})-\varphi_{\infty}=\int_{r_{0}}^{\infty}\frac{d\phi}{dr}dr=\int_{r_{0}}^{\infty}\frac{\sqrt{B}(D+A\ell)}{\sqrt{(AC+D^{2})(C-D\ell-A\ell^{2})}}dr.$
(45)

Because ${\dot{r}}=0$ at the distance of closest approach, the angular
momentum can be represented in terms of $r_{0}$ as

$\ell=\frac{-D_{0}+\sqrt{D_{0}^{2}+4A_{0}C_{0}}}{2A_{0}}.$ (46)

We will compute the integral perturbatively up the first order in $G$ and
$1/\beta^{2}$ in the weak field region. Because $D_{0}^{2}$ in Eq. (46) is
second order, we can write $\ell$ as

$\ell=\sqrt{\frac{C_{0}}{A_{0}}}\left(1-\frac{D_{0}}{2\sqrt{A_{0}C_{0}}}\right).$
(47)

Neglecting $D^{2}$ term in the square root of Eq. (45), we can write

$\frac{d\phi}{dr}=\sqrt{\frac{B}{C}}\frac{1+\frac{D}{\ell}}{\sqrt{\frac{1}{\ell^{2}}\frac{C}{A}-1-\frac{1}{\ell}\frac{D}{A}}}.$
(48)

One can compute the integral by substituting Eq. (47) into Eq. (48). The
details of the integration is given in Appendix B. Inserting the result of the
integration given by Eq. (69) into Eq. (44), we obtain the bending angle up to
the first order as

$\Delta\varphi=\frac{4GM}{r_{0}}-\frac{3\pi}{4}\frac{Gq^{2}}{r_{0}^{2}}+\frac{\pi}{4}\frac{Gq\mu}{r_{0}^{3}}-\frac{3\pi}{32}\frac{G\mu^{2}}{r_{0}^{4}}+\frac{9\pi}{16}\frac{q^{2}}{\beta^{2}r_{0}^{4}}-\frac{9\pi}{16}\frac{q\mu}{\beta^{2}r_{0}^{5}}+\frac{15\pi}{16}\frac{\mu^{2}}{\beta^{2}r_{0}^{6}}.$
(49)

The first four terms having the gravitational constant $G$ come from the
general relativistic effects while the last three terms having the Born-Infeld
parameter $1/\beta^{2}$ reflect the nonlinear electrodynamic effects. Note
that the signs of $Gq^{2}$ and $G\mu^{2}$ terms are negative while the signs
of $q^{2}/\beta^{2}$ and $\mu^{2}/\beta^{2}$ terms are positive. This means
that the charge and the magnetic dipole terms originated from $T_{\mu\nu}$
give repulsive effects while those originated from nonlinear electrodynamic
effects give attractive effects. The result includes particular cases of weak
bending angle in the literature. For $q=\mu=0$, it corresponds to the bending
angle by a Schwarzschild black hole. For $\mu=0$, it corresponds to the
bending angle by an Einstein-Born-Infeld black hole [8]. For $\mu=0$ and
$\beta=\infty$, it corresponds to the bending angle by a Reisner-Nordstrom
black hole. For $q=0$, it corresponds to the bending angle of a light ray
passing on the equator of a magnetar [9].

##  IV Conclusion

We consider the geometrical aspects of light around a compact astrophysical
object with both the electric field and the magnetic field in Einstein-Born-
Infeld theory. We consider the null geodesic of a light ray when it passes a
massive object with electric charge and magnetic dipole. The effective metric
can be obtained from the light cone condition reflecting the nonlinear
electromagnetic effects. We found the asymptotic form of the effective metric
up to the first order in gravitational constant $G$ and Born-Infeld parameter
$1/\beta^{2}$ on the equatorial plane. Then we compute the bending angle of
light from the geodesic equation. We confirm that the result covers particular
cases in the literature.

In this work we compute the effective metric from Eq. (20) which does not show
birefringence. This is possible only in the classical Born-Infeld theory where
the effective action depends only one parameter $\beta$. In general the
effective nonlinear action can be polarization-dependent where the parameters
of the invariants $S$ and $P$ are different [21]. For example, it is well-
known that the effective metric of Euler-Heisenberg Lagrangian is
polarization-dependent [22]. One can find the polarization-dependent effective
metric following the formalism by Novello et al [14].

In this study we compute the bending angle for a special case where a light
ray is passing on the equatorial plane of the magnetic dipole. It will be
interesting to find the bending angle for an arbitrary orientation of the
magnetic dipole. For magnetars considered in this paper, the Schwarzschild
radius and the photon sphere are smaller than the radius of magnetar. However,
for charged black holes with magnetic dipole, the effective metric used in
this paper might be useful to study the strong lensing.

## Acknowledgements

This work was supported by Basic Science Research Program through the National
Research Foundation of Korea (NRF) funded by the Ministry of Education,
Science and Technology (NRF-2019R1F1A1060409).

## Appendix A

In this appendix we compute the effective metric from Eq. (22). The
contravariant tensor of the metric given by Eq. (9) is

$g^{\mu\nu}=\frac{1}{s}\pmatrix{r^{2}e^{-\nu}&0&0&\omega\cr
0&-se^{-\lambda}&0&0\cr 0&0&-se^{-\lambda}&0\cr\omega&0&0&-e^{\rho}},$ (50)

where $s=r^{2}e^{\rho-\nu}+\omega^{2}$. We will compute the effective metric
up to the first order in $G$ and $1/\beta^{2}$. Because the nonlinear
electrodynamic correction in Eq. (22) is of the order $1/\beta^{2}$, we can
use the electromagnetic tensor in flat space given by Eq. (12). The
contravariant form of the field tensor is

$F^{\mu\nu}=\pmatrix{0&\Phi_{r}&\Phi_{z}&0\cr-\Phi_{r}&0&0&-\frac{\psi_{r}}{r^{2}}\cr-\Phi_{z}&0&0&-\frac{\psi_{z}}{r^{2}}\cr
0&\frac{\psi_{r}}{r^{2}}&\frac{\psi_{z}}{r^{2}}&0}.$ (51)

Using Eqs. (50) and (51), the nonzero components of $\delta g^{\mu\nu}$are

$\displaystyle\delta g^{00}$ $\displaystyle=$
$\displaystyle\frac{1}{\beta^{2}}(\Phi_{r}^{2}+\Phi_{z}^{2})e^{\lambda},$ (52)
$\displaystyle\delta g^{03}$ $\displaystyle=$ $\displaystyle\delta
g^{30}=\frac{1}{\beta^{2}}\left(\frac{\Phi_{r}\psi_{r}}{r^{2}}+\frac{\Phi_{z}\psi_{z}}{r^{2}}\right)e^{\lambda},$
(53) $\displaystyle\delta g^{11}$ $\displaystyle=$
$\displaystyle-\frac{1}{\beta^{2}}\left(\Phi_{r}^{2}e^{\rho}+2\frac{\Phi_{r}\psi_{r}}{r^{2}}\omega-\frac{\psi_{r}^{2}}{r^{2}}e^{-\nu}\right),$
(54) $\displaystyle\delta g^{12}$ $\displaystyle=$ $\displaystyle\delta
g^{21}=-\frac{1}{\beta^{2}}\left(\Phi_{r}\Phi_{z}e^{\rho}+\frac{\Phi_{r}\psi_{z}+\psi_{r}\
\Phi_{z}}{r^{2}}\omega-\frac{\psi_{r}\psi_{z}}{r^{2}}e^{-\nu}\right),$ (55)
$\displaystyle\delta g^{22}$ $\displaystyle=$
$\displaystyle-\frac{1}{\beta^{2}}\left(\Phi_{z}^{2}e^{\rho}+\frac{2\Phi_{z}\psi_{z}}{r^{2}}\omega-\frac{\psi_{z}^{2}}{r^{2}}e^{-\nu}\right),$
(56) $\displaystyle\delta g^{33}$ $\displaystyle=$
$\displaystyle\frac{1}{\beta^{2}}\left(\frac{\Phi_{r}^{2}}{r^{2}}+\frac{\psi_{z}^{2}}{r^{2}}\right)e^{\lambda}.$
(57)

The elplicit form of the effective metric can be obtained upon substituting
Eqs. (13)-(18) to $\delta g^{\mu\nu}$. Because we are interested in computing
the bending angle on the equatorial plane ($z=0$), the nonzero comopnent of
the effective metric up to the first order in $G$ and $1/\beta^{2}$ are

$\displaystyle{\tilde{g}}^{00}$ $\displaystyle=$ $\displaystyle
1+\frac{2GM}{r}-\frac{Gq^{2}}{r^{2}}+\frac{1}{\beta^{2}}\frac{q^{2}}{r^{4}},$
(58) $\displaystyle{\tilde{g}}^{03}$ $\displaystyle=$
$\displaystyle{\tilde{g}}^{30}=-\frac{Gq\mu}{r^{4}}+\frac{1}{\beta^{2}}\frac{q\mu}{r^{6}},$
(59) $\displaystyle{\tilde{g}}^{11}$ $\displaystyle=$
$\displaystyle-\left(1-\frac{2GM}{r}+\frac{G\mu^{2}}{2r^{4}}+\frac{1}{\beta^{2}}\frac{q^{2}}{r^{4}}-\frac{1}{\beta^{2}}\frac{\mu^{2}}{r^{6}}\right),$
(60) $\displaystyle{\tilde{g}}^{22}$ $\displaystyle=$
$\displaystyle-\left(1-\frac{2GM}{r}+\frac{G\mu^{2}}{2r^{4}}\right),$ (61)
$\displaystyle{\tilde{g}}^{33}$ $\displaystyle=$
$\displaystyle-\frac{1}{r^{2}}\left(1-\frac{2GM}{r}+\frac{Gq^{2}}{r^{4}}-\frac{1}{\beta^{2}}\frac{\mu^{2}}{r^{6}}\right).$
(62)

## Appendix B

In this appendix we compute the integral in Eq. (45) up to the first order in
$G$ and $1/\beta^{2}$. Because $D_{0}$ is first order in Eq. (47),
substituting $D_{0}/\sqrt{A_{0}C_{0}}=D_{0}/r_{0}$ to the leading order, we
have

$\ell=\sqrt{\frac{C_{0}}{A_{0}}}\left(1-\frac{D_{0}}{2r_{0}}\right).$ (63)

Inserting this, Eq. (48) can be written as

$\frac{d\phi}{dr}=\sqrt{\frac{B}{C}}\frac{1+\frac{D}{r_{0}}}{\sqrt{\frac{C}{A}\frac{A_{0}}{C_{0}}+\frac{r^{2}}{r_{0}^{2}}\frac{D_{0}}{r_{0}}-1-\frac{D}{r_{0}}}}.$
(64)

Using the explicit forms of metric functions in Eqs. (35) and (36), we have

$\sqrt{\frac{B}{C}}=\frac{x}{r_{0}}\left(1+\frac{Gq^{2}}{2r_{0}^{2}}x^{2}-\frac{G\mu^{2}}{4r_{0}^{4}}x^{4}-\frac{q^{2}}{2\beta^{2}r_{0}^{4}}x^{4}\right),$
(65)

where $x$ is defined as $x\equiv r_{0}/r$. Similarly, the numerator and the
denominator can be written as

$1+\frac{D}{r_{0}}=1-\frac{Gq\mu}{r_{0}^{3}}x^{2}+\frac{q\mu}{\beta^{2}r_{0}^{5}}x^{4},$
(66)
$\displaystyle\frac{C}{A}\frac{A_{0}}{C_{0}}+\frac{r^{2}}{r_{0}^{2}}\frac{D_{0}}{r_{0}}-1-\frac{D}{r_{0}}$
$\displaystyle=$
$\displaystyle\frac{1-x^{2}}{x^{2}}\bigg{[}1-\frac{4GM}{r_{0}}\frac{1}{1+x}+\frac{2Gq^{2}}{r_{0}^{2}}-\frac{q^{2}}{\beta^{2}r_{0}^{4}}(1+x^{2})$
(67) $\displaystyle-$
$\displaystyle\frac{\mu^{2}}{\beta^{2}r_{0}^{6}}(1+x^{2}+x^{4})-\frac{Gq\mu}{r_{0}^{3}}(1+x^{2})+\frac{q\mu}{\beta^{2}r_{0}^{5}}(1+x^{2}+x^{4})\bigg{]}.$

Substituting Eqs. (65) - (67) and $dr=-(r_{0}/x^{2})dx$, the integral can be
written as

$\displaystyle\int_{r_{0}}^{\infty}\frac{d\phi}{dr}dr$ $\displaystyle=$
$\displaystyle\int_{0}^{1}\frac{dx}{\sqrt{1-x^{2}}}\bigg{[}1+\frac{2GM}{r_{0}}\frac{1}{1+x}+\frac{Gq^{2}}{2r_{0}^{2}}(x^{2}-2)+\frac{Gq\mu}{2r_{0}^{3}}(1-x^{2})-\frac{G\mu^{2}}{4r_{0}^{4}}x^{4}$
(68) $\displaystyle+$
$\displaystyle\frac{q^{2}}{2\beta^{2}r_{0}^{4}}(1+x^{2}-x^{4})-\frac{q\mu}{2\beta^{2}r_{0}^{5}}(1+x^{2}-x^{4})+\frac{\mu^{2}}{2\beta^{2}r_{0}^{6}}(1+x^{2}+x^{4})\bigg{]}.$

Upon integration we finally have

$\int_{r_{0}}^{\infty}\frac{d\phi}{dr}dr=\frac{\pi}{2}+\frac{2GM}{r_{0}}-\frac{3\pi}{8}\frac{Gq^{2}}{r_{0}^{2}}+\frac{\pi}{8}\frac{Gq\mu}{r_{0}^{3}}-\frac{3\pi}{64}\frac{G\mu^{2}}{r_{0}^{4}}\\\
+\frac{9\pi}{32}\frac{q^{2}}{\beta^{2}r_{0}^{4}}-\frac{9\pi}{32}\frac{q\mu}{\beta^{2}r_{0}^{5}}+\frac{15\pi}{32}\frac{\mu^{2}}{\beta^{2}r_{0}^{6}}.$
(69)

## References

  * [1] K. Akiyama et al. (Event Horizon Telescope Collaboration), Astrophys. J. 875, L1 (2019).
  * [2] K. Akiyama et al. (Event Horizon Telescope Collaboration), Astrophys. J. 875, L5 (2019).
  * [3] K. Akiyama et al. (Event Horizon Telescope Collaboration), Astrophys. J. 875, L6 (2019).
  * [4] R. C. Duncan and C. Thompson, Astrophys. J. 408, 194 (1993).
  * [5] C. Thompson and R. C. Duncan, Mon. Not. R. Astron. Soc. 275, 255 (1995).
  * [6] J. D. Jackson, Classical Electrodynamics (John Wiley and Sons, 1998).
  * [7] J. F. Plebanski, Lectures on Nonlinear Electrodynamics (Nordita, Copenhagen, 1970).
  * [8] J. Y. Kim, Eur. Phys. J. C 81, 508 (2021).
  * [9] J. Y. Kim, Eur. Phys. J. C 82, 485 (2022).
  * [10] K. Akiyama et al. (Event Horizon Telescope Collaboration), Astrophys. J. 910, L13 (2021).
  * [11] M. Born, Proc. R. Soc. London A 143, 410 (1934).
  * [12] M. Born and M. Infeld, Proc. R. Soc. London A 144, 425 (1934).
  * [13] A. W. Martin and P. L. Pritchett, J. Math. Phys. 9, 593 (1968).
  * [14] M. Novello, V. A. De Lorenci, J. M. Salim, and R. Klippert, Phys. Rev. D 61, 045001 (2000).
  * [15] V. A. De Lorenci, N. Figueiredo, H. H. Fliche, and M. Novello, Astron. Astrophys. 369, 690 (2001).
  * [16] N. Breton, Class. Quantum Grav. 19, 601 (2002).
  * [17] E. F. Eiroa, Phys. Rev. D 73, 043002 (2006).
  * [18] S. Weinberg, Gravitation and Cosmology: Principles and Applications of the General Theory of Relativity (Oxford University, New York, 1983).
  * [19] Y.-W. Hsiao, D.-S. Lee, and C.-Y. Lin, Phys. Rev. D 101, 064070 (2020).
  * [20] M. S. Ali and S. Kauhsal, Phys. Rev. D 105, 024062 (2022).
  * [21] S. I. Kruglov, J. Phys. A 43, 375402 (2010).
  * [22] W. Heisenberg and H. Euler, Z. Phys. 98, 714 (1936).