# PF-cpGAN: Profile to Frontal Coupled GAN for Face Recognition in the Wild

Fariborz Taherkhani , Veeru Talreja 11footnotemark: 1, Jeremy Dawson, Matthew
C. Valenti, and Nasser M. Nasrabadi
West Virginia University
{ft0009<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Authors Contributed Equally

###### Abstract

In recent years, due to the emergence of deep learning, face recognition has
achieved exceptional success. However, many of these deep face recognition
models perform relatively poorly in handling profile faces compared to frontal
faces. The major reason for this poor performance is that it is inherently
difficult to learn large pose invariant deep representations that are useful
for profile face recognition. In this paper, we hypothesize that the profile
face domain possesses a gradual connection with the frontal face domain in the
deep feature space. We look to exploit this connection by projecting the
profile faces and frontal faces into a common latent space and perform
verification or retrieval in the latent domain. We leverage a coupled
generative adversarial network (cpGAN) structure to find the hidden
relationship between the profile and frontal images in a latent common
embedding subspace. Specifically, the cpGAN framework consists of two GAN-
based sub-networks, one dedicated to the frontal domain and the other
dedicated to the profile domain. Each sub-network tends to find a projection
that maximizes the pair-wise correlation between two feature domains in a
common embedding feature subspace. The efficacy of our approach compared with
the state-of-the-art is demonstrated using the CFP, CMU Multi-PIE, IJB-A, and
IJB-C datasets.

##  1 Introduction

Due to the emergence of deep learning, face recognition has achieved
exceptional success in recent years [3]. However, many of these deep face
recognition models perform relatively poorly in handling profile faces
compared to frontal faces [36]. In other words, face recognition in the wild,
or unconstrained face recognition, is a challenging problem. Pose, expression,
and lighting variations are considered to be major obstacles in attaining high
unconstrained face recognition performance. Some methods [27, 3] attempt to
address pose-variation issue by learning pose-invariant features, while some
other methods [53, 55, 7, 31, 47] try to normalize images (along with
identity-preservation) to a single frontal pose before recognition. However,
there are three major difficulties related to face frontalization or
normalization in unconstrained environment:

  * •

Complicated face-variations besides pose: In comparison to a controlled
environment, there are more complex face variations, e.g., lighting, head
pose, expression, in real-world scenarios. It is difficult to directly warp
the input face to a normalized view [31].

  * •

Unpaired data: Undoubtedly, obtaining a strictly normalized face is expensive
and time-consuming, but getting an effective pair of target normalized face
(i.e., frontal-facing, neutral expression) and an input face is difficult due
to highly imbalanced datasets[31].

  * •

Presence of artifacts: Synthesized ‘frontal’ faces contain artifacts caused by
occlusions and non-rigid expressions.

Figure 1: Frontal and Profile Images from Celebrity in Frontal Profile (CFP)
Dataset.

In this paper, we hypothesize that the profile face domain possesses a gradual
connection with the frontal face domain in a latent deep feature subspace. We
aim to exploit this connection by projecting the profile faces and frontal
faces into a common latent subspace and perform verification or retrieval in
this latent domain. We propose an embedding model for profile to frontal face
verification based on a deep coupled learning framework which uses a
generative adversarial network (GAN) to find the hidden relationship between
the profile face features and frontal face features in a latent common
embedding subspace.

Our work is conceptually related to the embedding category of super-resolution
[37, 56, 19, 24] in that our approach also performs verification of profile
and frontal face in the latent space but not in the image space. From our
experiments, we observe that transforming profile and frontal face features to
a latent embedding subspace could yield higher performance than image-level
face frontalization, which is susceptible to the negative influence of
artifacts as a result of image synthesis. To our best knowledge, this study is
the first attempt to perform profile-to-frontal face verification in the
latent embedding subspace using generative modeling.

This paper makes the following contributions:

  * •

A novel profile to frontal face recognition model using coupled GAN framework
with multiple loss functions is developed.

  * •

Comprehensive experiments using different datasets and a comparison of the
proposed method with the state-of-the-art methods have been performed,
indicating the efficacy of the proposed GAN framework.

  * •

The proposed framework can potentially be used to improve the performance of
traditional face recognition methods by integrating it as a preprocessing
procedure for face-frontalization schema.

##  2 Related Work

Face recognition using Deep Learning: Before the advent of deep learning,
traditional methods for face recognition (FR) used one or more layer
representations, such as the histogram of the feature codes, filtering
responses, or distribution of the dictionary atoms [50]. FR research was
concentrated more toward separately improving preprocessing, local
descriptors, and feature transformation; however, overall improvement in FR
accuracy was very slow. This all changed with the advent of deep learning, and
now deep learning is the prominent technique used for FR.

Recently various deep learning models such as [43, 8] are used as baseline
model for FR. Simultaneously, various loss functions have been explored and
used in FR. These loss functions can be categorized as the Euclidean-distance-
based loss, angular/cosine-margin-based loss, and softmax loss and its
variations. The contrastive loss and the triplet loss are the commonly used
Euclidean-distance-based loss functions [40, 34, 41, 42]. For avoiding
misclassification of difficult samples [46, 45], the learned face features
need to be well separated. Angular/cosine-margin based loss [10, 25, 2] are
commonly used to make the learned features more separable with a larger
angular/cosine distance. Finally, in the category of softmax loss and its
variants for FR [14, 49, 26], the softmax loss is modified to improve the FR
performance as in [26], where the cosine distance among data features is
optimized along with normalization of features and weights.

Profile-Frontal Face Recognition: Face recognition with pose variation in an
unconstrained environment is a very challenging problem. Existing methods
focus on the pose variation problem by training separate models for learning
pose-invariant features [27, 3], elaborate dense 3D facial landmark detection
and warping [44], and synthesizing a frontal, neutral expression face from a
single image [47, 53, 55, 7, 31]. For instance, Cao _et al_. [3] exploit the
inherent mapping between profile and frontal faces, and transform a deep
profile face representation to a canonical pose by adaptively adding
residuals. FF-GAN [55] solves the problem of large-pose face frontalization in
the wild by incorporating a 3D face model into a GAN. Considering
photorealistic and identity preserving frontal view synthesis, a domain
adaptation strategy for pose invariant face recognition is discussed in [58].
Tran _et al_. [47] propose a GAN framework to rotate the face and disentangle
the identity representation by using the pose code. In [31], a face
normalization model (FNM) uses a generative adversarial network (GAN) network
with 3 distinct losses for generating canonical-view and expression-free
frontal images.

##  3 Generative Adversarial Network

GAN was first introduced by Goodfellow _et al_. [12] and has drawn great
attention from the deep learning research community due to its remarkable
performance on generative tasks. The GAN framework is based on two competing
networks — a generator network G and a discriminator network D. The generator
$G(z;\theta_{g})$ is a differentiable function which maps the noise variable
$z$ from training noise distribution $p_{z}(z)$ to a data space with
distribution $p_{data}$ using the network parameters $\theta_{g}$. On the
other hand, the discriminator $D(.;\theta_{d})$ is also a differentiable
function, which discriminates between the real data $y$ and the generated fake
data $G(z)$ using a binary classification model. Specifically, the min-max
two-player game between the generator and the discriminator provides a simple
and powerful way to estimate target distribution and generate novel image
samples [31]. The loss function $L(D,G)$ for GAN is given as:

$\begin{split}L(D,G)&=E_{y\sim P_{data}(y)}[\log D(y)]\\\ &+E_{z\sim
P_{z}(z)}[\log(1-D(G(z)))].\end{split}$ (1)

The objective (two player minimax game) for GAN is as:

$\begin{split}\min_{G}\max_{D}L(D,G)&=\min_{G}\max_{D}[E_{y\sim
P_{data}(y)}[\log D(y)]\\\ &+E_{z\sim P_{z}(z)}[\log(1-D(G(z)))]].\end{split}$
(2)

Another variant of GAN is the Conditional GAN, which was introduced by Mirza
and Osindero [29]. In conditional GAN, both the generator and discriminator
are conditioned on an additional variable $x$. This additional variable could
be any kind of auxiliary information such as discrete labels [29], and text
[32]. The loss function for the conditional GAN is given as:

$\begin{split}L_{c}(D,G)&=E_{y\sim P_{data}(y)}[\log D(y|x)]\\\ &+E_{z\sim
P_{z}(z)}[\log(1-D(G(z|x)))].\end{split}$ (3)

Hereafter, we will denote the objective for the conditional GAN as
$F_{cGAN}(D,G,y,x)$, which is given by:

$\begin{split}F_{cGAN}(D,G,y,x)&=\min_{G}\max_{D}[E_{y\sim P_{data}(y)}[\log
D(y|x)]\\\ &+E_{z\sim P_{z}(z)}[\log(1-D(G(z|x)))]].\end{split}$ (4)

##  4 Proposed Method

Here, we describe our method for profile to frontal face recognition. In
contrast to the face normalization methods, we do not perform pose
normalization (i.e., frontalization) on each profile image before matching.
Instead, we seek to project the profile and frontal face images to a common
latent low-dimensional embedding subspace using generative modeling. Inspired
by the success of GANs [12], we explore adversarial networks to project
profile and frontal images to a common subspace for recognition.

The framework of proposed profile to frontal coupled generative adversarial
network (PF-cpGAN; shown in Fig. 2) consists of two modules, where each module
contains a GAN architecture made of a generator and a discriminator. The
generators that we have used in both modules are U-net auto-encoders that are
coupled together using a contrastive loss function. In addition to
adversarial, and contrastive loss, we propose to guide the generators using a
perceptual loss [20] based on the VGG 16 architecture, as well as an $L_{2}$
reconstruction error. The perceptual loss helps in sharp and realistic
reconstruction of the images.

Figure 2: Block diagram of PF-cpGAN.

###  4.1 Profile to Frontal Coupled GAN

The main objective of PF-cpGAN is the recognition of profile face images with
respect to a gallery of frontal face images, which have not been seen during
the training. The matching of the profile and the frontal face images is
performed in a common embedding subspace. PF-cpGAN consists of two modules: a
profile GAN module and a frontal GAN module, both consisting of a GAN
(generator + discriminator), and a perceptual network based on VGG-16.

For the generators, we use a U-Net [33] auto-encoder architecture. The primary
reason for using U-Net is that the encoder-decoder structure tends to extract
global features and generates images by leveraging this overall information,
which is very useful for global shape transformation tasks such as profile to
frontal image conversion. Moreover, for many image translation problems, there
is a significant amount of low-level information that needs to be shared
between the input and output, and it is desirable to pass this information
directly across all the layers including the bottleneck. Thus, using skip-
connections, as in U-net, provides a means for the encoder-decoder structure
to circumvent the bottleneck and pass the information over to other layers.

For discriminators, we have used patch-based discriminators [18], which are
trained iteratively along with the respective generators. $L_{1}$ loss
performs very well when trying to preserve the low frequency details but fails
to preserve the high-frequency details. However, using a patch-based
discriminator that penalizes structure at the scale of the patches, ensures
the preservation of high-frequency details, which are usually lost when only
$L_{1}$ loss is used.

The final objective of PF-cpGAN is to find the hidden relationship between the
profile face features and frontal face features in a latent common embedding
subspace. To find this common subspace between the two domains, we couple the
two generators via a contrastive loss function, $L_{cont}$.

This loss function $(L_{cont})$ is a distance-based loss function, which tries
to ensure that semantically similar examples (genuine pairs i.e., a profile
image of a subject with its corresponding frontal image) are embedded closely
in the common embedding subspace, and, simultaneously, semantic dissimilar
examples (impostor pairs i.e., a profile image of a subject and a frontal
image of a different subject) are pushed away from each other in the common
embedding subspace. The contrastive loss function is defined as:

$\begin{split}L_{cont}(z_{1}&(x^{i}_{PR}),z_{2}(x^{j}_{FR}),Y)=\\\
&(1-Y)\frac{1}{2}(D_{z})^{2}+(Y)\frac{1}{2}(\mbox{max}(0,m-D_{z}))^{2},\end{split}$
(5)

where $x^{i}_{PR}$ and $x^{j}_{FR}$ denote the input profile and frontal face
image, respectively. The variable $Y$ is a binary label, which is equal to 0
if $x^{i}_{PR}$ and $x^{j}_{FR}$ belong to the same class (i.e., genuine
pair), and equal to 1 if $x^{i}_{PR}$ and $x^{j}_{FR}$ belong to the different
class (i.e., impostor pair). $z_{1}(.)$ and $z_{2}(.)$ denote only the
encoding functions of the U-Net auto-encoder to transform $x^{i}_{PR}$ and
$x^{j}_{FR}$, respectively into a common latent embedding subspace. The value
$m$ is the contrastive margin and is used to “tighten” the constraint. $D_{z}$
denotes the Euclidean distance between the outputs of the functions
$z_{1}(x^{i}_{PR})$ and $z_{2}(x^{j}_{FR})$.

$D_{z}=\left\lVert z_{1}(x^{i}_{PR})-z_{2}(x^{j}_{FR})\right\rVert_{2}.$ (6)

Therefore, if $Y=0$ (i.e., genuine pair), then the contrastive loss function
$(L_{cont})$ is given as:

$L_{cont}(z_{1}(x^{i}_{PR}),z_{2}(x^{j}_{FR}),Y)=\frac{1}{2}\left\lVert
z_{1}(x^{i}_{PR})-z_{2}(x^{j}_{FR})\right\rVert^{2}_{2},$ (7)

and if $Y=1$ (i.e., impostor pair), then contrastive loss function
$(L_{cont})$ is :

$\begin{split}L_{cont}(z_{1}(x^{i}_{PR}),&z_{2}(x^{j}_{FR}),Y)=\\\
&\frac{1}{2}\mbox{max}\biggl{(}0,m-\left\lVert
z_{1}(x^{i}_{PR})-z_{2}(x^{j}_{FR})\right\rVert^{2}_{2}\biggr{)}.\end{split}$
(8)

Thus, the total loss for coupling the profile generator and the frontal
generator is denoted by $L_{cpl}$ and is given as:

$\begin{split}L_{cpl}=\frac{1}{N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{N}L_{cont}(z_{1}(x^{i}_{PR}),z_{2}(x^{j}_{FR}),Y),\end{split}$
(9)

where N is the number of training samples. The contrastive loss in the above
equation can also be replaced by some other distance-based metric, such as the
Euclidean distance. However, the main aim of using the contrastive loss is to
be able to use the class labels implicitly and find the discriminative
embedding subspace, which may not be the case with some other metric such as
the Euclidean distance. This discriminative embedding subspace would be useful
for matching of the profile images with the frontal images.

###  4.2 Generative Adversarial Loss

Let the generators (profile generator and frontal generator) that reconstruct
the corresponding profile and frontal image from the input profile and frontal
image, be denoted as $G_{PR}$ and $G_{FR}$, respectively. The patch-based
discriminators used for the profile and frontal GANs are denoted as $D_{PR}$
and $D_{FR}$. For the proposed method, we have used the conditional GAN, where
the generator networks $G_{PR}$ and $G_{FR}$ are conditioned on input profile
and frontal face images, respectively. We have used the conditional GAN loss
function [29] to train the generators and the corresponding discriminators in
order to ensure that the discriminators cannot distinguish the images
reconstructed by the generators from the corresponding ground truth images.
Let $L_{PR}$ and $L_{FR}$ denote the conditional GAN loss functions for the
profile and the frontal GANs, respectively, where $L_{PR}$ and $L_{FR}$ are
given as:

$L_{PR}=F_{cGAN}(D_{PR},G_{PR},y^{i}_{PR},x^{i}_{PR}),$ (10)
$L_{FR}=F_{cGAN}(D_{FR},G_{FR},y^{j}_{FR},x^{j}_{FR}),$ (11)

where function $F_{cGAN}$ is the conditional GAN objective defined in (4). The
term $x^{i}_{PR}$ denotes the profile image used as a condition for the
profile GAN, and $y^{i}_{PR}$ denotes the real profile image. Note that the
real profile image $y^{i}_{PR}$ and the network condition given by
$x^{i}_{PR}$ are the same. Similarly, $x^{j}_{FR}$ denotes the frontal image
used as a condition for the frontal GAN and $y^{j}_{FR}$ denotes the real
frontal image. Again, the real frontal image $y^{j}_{FR}$ and the network
condition given by $x^{j}_{FR}$ are the same. The total loss for the coupled
conditional GAN is given by:

$L_{GAN}=L_{PR}+L_{FR}.$ (12)

###  4.3 $L_{2}$ Reconstruction Loss

We also consider the $L_{2}$ reconstruction loss for both the profile GAN and
frontal GAN. The $L_{2}$ reconstruction loss measures the reconstruction error
in terms of the Euclidean distance between the reconstructed image and the
corresponding real image. Let $L_{2_{PR}}$ denote the reconstruction loss for
the profile GAN and is defined as:

$L_{2_{PR}}=\left\lVert G_{PR}(z|x^{i}_{PR})-y^{i}_{PR}\right\rVert^{2}_{2},$
(13)

where $y^{i}_{PR}$ is the ground truth profile image, $G_{PR}(z|x^{i}_{PR})$
is the output of the profile generator.

Similarly, Let $L_{2_{FR}}$ denote the reconstruction loss for the frontal
GAN:

$L_{2_{FR}}=\left\lVert G_{FR}(z|x^{j}_{FR})-y^{j}_{FR}\right\rVert^{2}_{2},$
(14)

where $y^{j}_{FR}$ is the ground truth frontal image, $G_{FR}(z|x^{j}_{FR})$
is the output of the frontal generator.

The total $L_{2}$ reconstruction loss function is given by:

$L_{2}=\frac{1}{N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{N}(L_{2_{PR}}+L_{2_{FR}}).$
(15)

###  4.4 Perceptual Loss

In addition to the GAN loss and the reconstruction loss which are used to
guide the generators, we have also used perceptual loss, which was introduced
in [20] for style transfer and super-resolution. The perceptual loss function
is used to compare high level differences, like content and style
discrepancies, between images. The perceptual loss function involves comparing
two images based on high-level representations from a pretrained CNN, such as
VGG-16 [39]. The perceptual loss function is a good alternative to solely
using $L_{1}$ or $L_{2}$ reconstruction error, as it gives better and sharper
high quality reconstruction images [20].

In our proposed approach, perceptual loss is added to both the profile and the
frontal module using a pre-trained VGG-16 [39] network. We extract the high-
level features (ReLU3-3 layer) of VGG-16 for both the real input image and the
reconstructed output of the U-Net generator. The $L_{1}$ distance between
these features of real and reconstructed images is used to guide the
generators $G_{PR}$ and $G_{FR}$ . The perceptual loss for profile network is
defined as:

$\begin{split}L_{P_{PR}}=&\frac{1}{C_{p}W_{p}H_{p}}\sum_{c=1}^{C_{p}}\sum_{w=1}^{W_{p}}\sum_{h=1}^{H_{p}}\\\
&\left\lVert
V(G_{PR}(z|x^{i}_{PR}))^{c,w,h}-V(y^{i}_{PR})^{c,w,h}\right\rVert,\end{split}$
(16)

where $V(.)$ denotes a particular layer of the VGG-16, where the layer
dimensions are given by $C_{p}$, $W_{p}$, and $H_{p}$.

Likewise the perceptual loss for frontal network is:

$\begin{split}L_{P_{FR}}=&\frac{1}{C_{p}W_{p}H_{p}}\sum_{c=1}^{C_{p}}\sum_{w=1}^{W_{p}}\sum_{h=1}^{H_{p}}\\\
&\left\lVert
V(G_{FR}(z|x^{j}_{FR}))^{c,w,h}-V(y^{j}_{FR})^{c,w,h}\right\rVert.\end{split}$
(17)

The total perceptual loss function is given by:

$L_{P}=\frac{1}{N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{N}(L_{P_{PR}}+L_{P_{FR}}).$
(18)

###  4.5 Overall Objective Function

The overall objective function for learning the network parameters in the
proposed method is given as the sum of all the loss functions defined above:

$\begin{split}L_{tot}=L_{cpl}+\lambda_{1}L_{GAN}+\lambda_{2}L_{P}+\lambda_{3}L_{2},\end{split}$
(19)

where $L_{cpl}$ is the coupling loss, $L_{GAN}$ is the total generative
adversarial loss, $L_{P}$ is the total perceptual loss, and $L_{2}$ is the
total reconstruction error. Variables $\lambda_{1},\lambda_{2},$ and
$\lambda_{3}$ are the hyper-parameters to weigh the different loss terms.

##  5 Experiments

We initially describe our training setup and the datasets that we have used in
our experiments. We show the efficiency of our method for the task of frontal
to profile face verification by comparing its performance with state-of the-
art face verification methods across pose-variation. Moreover, we explore the
effect of face yaw in our algorithm. Finally, we conduct an ablation study to
investigate the effect of each term in our total training loss defined in
(19).

###  5.1 Experimental Details

Datasets: The Celebrities in Frontal-Profile (CFP) dataset [36] is a mixture
of constrained (i.e., carefully collected under different pose, illumination
and expression conditions) and unconstrained (i.e., collected images from the
Internet) settings. CFP includes 500 celebrities, averaging ten frontal and
four profile face images per each celebrity. Following the standard 10-fold
protocol [36], we divide the dataset into 10 folds, each of which consists of
350 same and 350 different pairs generated from 50 subjects (i.e., 7 same and
7 different pairs for each subject).

The CMU Multi-PIE database [13] contains 750,000 images of 337 subjects.
Subjects were imaged from 15 viewing angles and 19 illumination conditions
while exhibiting a range of facial expressions. It is the largest database for
graded evaluation with respect to pose, illumination, and expression
variations. For fair comparison, the database setting was made consistent with
FNM [31], where 250 subjects from Multi-PIE have been used. The training set
and testing split is consistent with FNM.

The IARPA Janus Benchmark A (IJB-A) [22] is a challenging dataset collected
under complete unconstrained conditions covering full pose variation (yaw
angles $-90\degree$ to $+90\degree$). IJB-A contains 500 subjects with 5,712
images and 20,414 frames extracted from videos. Following the standard
protocol in [22], we evaluate our method on both verification and
identification. The IARPA Janus Benchmark B (IJB-B) dataset [51] builds on the
IJB-A by adding more 1345 subjects making it a total of 1845 subjects, and a
total of 21,798 still images and 55,026 frames from 7,011 videos. The IARPA
Janus Benchmark C (IJB-C) dataset [28] builds on IJB-A, and IJB-B datasets and
has a total of 31,334 images for a total number of 3,531 subjects. We have
also evaluated our method on IJB-A and IJB-C datasets.

Implementation Details: We have implemented the U-Net with ResNet-18 [15]
encoder pre-trained on ImageNet. We have added an additional fully-connected
layer after the average pooling layer for ResNet-18 for our U-Net encoder. The
U-Net decoder has the same number of layers as the encoder. The entire
framework has been implemented in Pytorch. For convergence, $\lambda_{1}$ is
set to 1, and $\lambda_{2}$, and $\lambda_{3}$ are both set to 0.25. We used a
batch size of 128 and an Adam optimizer [21] with first-order momentum of 0.5,
and learning rate of 0.0004. We have used the ReLU activation function for the
generator and Leaky ReLU with a slope of 0.3 for the discriminator.

For training, genuine and impostor pairs were required. The genuine/impostor
pairs are created by frontal and profile images of the same/different subject.
During the experiments, we ensure that the training set are balanced by using
the same number of genuine and impostor pairs.

Table 1: Performance comparison on CFP dataset. Mean Accuracy and equal error
rate (EER) with standard deviation over 10 folds.

| Frontal-Profile | Frontal-Frontal
---|---|---
Algorithm | Accuracy | EER | Accuracy | EER
HoG+Sub-SML [36] | 77.31(1.61) | 22.20(1.18) | 88.34(1.31) | 11.45(1.35)
LBP+Sub-SML [36] | 70.02(2.14) | 29.60(2.11) | 83.54(2.40) | 16.00(1.74)
FV+Sub-SML [36] | 80.63(2.12) | 19.28(1.60) | 91.30(0.85) | 8.85(0.74)
FV+DML [36] | 58.47(3.51) | 38.54(1.59) | 91.18(1.34) | 8.62(1.19)
Deep Features [5] | 84.91(1.82) | 14.97(1.98) | 96.40(0.69) | 3.48(0.67)
PR-REM [3] | 93.25(2.23) | 7.92(0.98) | 98.10(2.19) | 1.10(0.22)
PF-cpGAN | 93.78(2.46) | 7.21(0.65) | 98.88(1.56) | 0.93(0.14)

###  5.2 Evaluation on CFP with Frontal-Profile Setting

We first perform evaluation on the Celebrities in Frontal-Profile (CFP)
dataset[36], a challenging dataset created to examine the problem of frontal
to profile face verification in the wild. We follow the standard 10-fold
protocol[36] in our evaluation. The same protocol is applied on both the
Frontal-Profile and Frontal-Frontal settings. For fair comparison and as given
in [36], we consider different types of feature extraction techniques like
HoG[9], LBP[1], and Fisher Vector[38] along with metric learning techniques
like Sub-SML[4], and Diagonal metric learning (DML) as reported in [38]. We
also compare against deep learning techniques, including Deep Features[5], and
PR-REM[3]. The results are summarized in Table 1.

We can observe from Table 1 that our proposed framework, PF-cpGAN, gives much
better performance than the methods that use standard hand-crafted features of
HoG, LBP, or FV, providing minimum of $13\%$ improvement in accuracy with a
$12\%$ decrease in EER for the profile-frontal setting. PF-cpGAN also improves
on the performance of the Deep Features by approximately $9\%$ with a $7.5\%$
decrease in EER for the profile-frontal setting. Finally, PF-cpGAN performs
on-par with the best deep learning method of PR-REM, and, in-fact, does
slightly better than PR-REM by $\approx 0.5\%$ improvement in accuracy with a
$0.7\%$ decrease in EER for the profile-frontal setting. This performance
improvement clearly shows that usage of a GAN framework for projecting the
profile and frontal images in the latent embedding subspace and maintaining
the sematic similarity in the latent space is better than some other deep
learning techniques such as Deep Features or PR-REM.

Table 2: Performance comparison on IJB-A benchmark. Results reported are the
’average$\pm$standard deviation’ over the 10 folds specified in the IJB-A
protocol. Symbol ’-’ indicates that the metric is not available for that
protocol.

Method | Verification | Identification
---|---|---
GAR@ FAR$=0.01$ | GAR@ FAR$=0.001$ | @ Rank-1 | @ Rank-5
OPENBR [23] | $23.6\pm 0.9$ | $10.4\pm 1.4$ | $24.6\pm 1.1$ | $37.5\pm 0.8$
GOTS [23] | $40.6\pm 1.4$ | $19.8\pm 0.8$ | $43.3\pm 2.1$ | $59.5\pm 2.0$
PAM [27] | $73.3\pm 1.8$ | $55.2\pm 3.2$ | $77.1\pm 1.6$ | $88.7\pm 0.9$
DCNN [6] | $78.7\pm 4.3$ | - | $85.2\pm 1.8$ | $93.7\pm 1.0$
DR-GAN [48] | $77.4\pm 2.7$ | $53.9\pm 4.3$ | $85.5\pm 1.5$ | $94.7\pm 1.1$
FF-GAN [54] | $85.2\pm 1.0$ | $66.3\pm 3.3$ | $90.2\pm 0.6$ | $95.4\pm 0.5$
FNM [31] | $93.4\pm 0.9$ | $83.8\pm 2.6$ | $96.0\pm 0.5$ | $98.6\pm 0.3$
PR-REM [3] | $94.4\pm 0.9$ | $86.8\pm 1.5$ | $94.6\pm 1.1$ | $96.8\pm 1.0$
PF-cpGAN | $95.8\pm 0.82$ | $91.2\pm 1.3$ | $97.6\pm 1.0$ | $98.8\pm 0.4$

(a) IJBA

(b) CMU Multi-PIE

(c) Ablation Study

Figure 3: ROC curve comparison against the baselines for different datasets is
shown in (a) and (b). In (c), we show the ROC curves showing the importance of
different loss functions for ablation study.

###  5.3 Evaluation on IJB-A and IJB-C

Figure 4: Reconstruction of frontal images at the output of the frontal U-Net
generator with profile images as input to the profile U-Net generator. Every
odd number column represent the input profile image and every even number
column represents the output frontal image. The input images belong to the
CMU-MultiPIE dataset.

Here, we focus on unconstrained face recognition on IJB-A dataset to quantify
the superiority of our PF-cpGAN for profile to frontal face recognition. Some
of the baselines for comparison on IJB-A are DR-GAN [47], FNM [31], PR-REM
[3],and FF-GAN [55]. We have also compared them with other methods as listed
in [31] and shown in Table 2. As shown in Table 2, we perform better than the
state-of-the-art methods for both verification and identification.
Specifically, for verification, we improve the genuine accept rate (GAR) by at
least $1.4\%$ compared to other methods. For instance, at the false accept
rate (FAR) of 0.01, the best previously-used method is PR-REM, with an average
GAR of $94.4\%$. PF-cpGAN improves upon PR-REM and gives an average GAR of
$95.8\%$ at the same FAR. We also show improvement on identification.
Specifically, the rank-1 recognition rate shows an improvement of around
$1.6\%$ in comparison to the best state-of-the-art method, FNM [31].

Table 3: Performance comparison on IJB-C benchmark. Results reported are the
’average$\pm$standard deviation’ over the 10 folds specified in the IJB-C
protocol. Symbol ’-’ indicates that the metric is not available for that
protocol.

Method | Verification | Identification
---|---|---
GAR@ FAR$=0.01$ | GAR@ FAR$=0.001$ | @ Rank-1 | @ Rank-5
GOTS [28] | $62.1\pm 1.1$ | $36.3\pm 1.2$ | $38.5\pm 1.6$ | $53.8\pm 1.8$
FaceNet [35] | $82.3\pm 1.18$ | $66.3\pm 1.3$ | $70.4\pm 1.2$ | $78.8\pm 2.3$
VGG-CNN [30] | $87.2\pm 1.09$ | $74.3\pm 0.9$ | $79.6\pm 1.04$ | $87.8\pm 1.3$
FNM [31] | $91.2\pm 0.8$ | $80.4\pm 1.8$ | $84.6\pm 0.6$ | $93.7\pm 0.9$
PR-REM [3] | $92.1\pm 0.8$ | $83.4\pm 1.5$ | $83.1\pm 0.4$ | $92.6\pm 1.1$
PF-cpGAN | $93.8\pm 0.67$ | $86.1\pm 0.7$ | $88.3\pm 1.2$ | $94.8\pm 0.6$

We have also plotted receiver operating characteristic (ROC) curve and
compared with the baselines given above. The ROC curves for the IJB-A dataset
are given in Fig. 3(a). As we can clearly see from the curves, the proposed
PF-cpGAN method improves upon other methods and gives much better performance,
even at a FAR of 0.001.

We have also performed the task of verification and identification using the
IJB-C dataset according to the verification and the identification protocol
given in the dataset. The results are provided in Table 3, showing that PF-
cpGAN improve on the existing state-of-the-art methods for both verification
and identification. For instance, at the false accept rate (FAR) of 0.01, the
best previously-used method is PR-REM, with an average GAR of $92.1\%$. PF-
cpGAN improves upon PR-REM and gives an average GAR of $93.8\%$ at the same
FAR. We also observe that, for identification, specifically, rank-1
recognition, shows an improvement over the previous best state-of-the-art
method FNM [31] by about $1.1\%$.

Figure 5: Reconstruction of profile images at the output of the profile U-Net
generator with frontal images as input to the frontal U-Net generator. Every
odd number column represents the input frontal image, and every even number
column represents the output profile image. The input images belong to the
CMU-MultiPIE dataset.

###  5.4 A Further Analysis on Influences of Face Yaw

In addition to complete profile to frontal face recognition, we also perform a
more in-depth analysis on the influence of face yaw angle on the performance
of face recognition to better understand the effectiveness of the PF-cpGAN for
profile to frontal face recognition. We perform this experiment for the CMU
Multi-PIE dataset [13] under setting-1 for fair comparison with other state-
of-the-art methods. As shown in Table 4, we achieve comparable performance
with other state-of-the-art methods for different yaw angles. Under extreme
pose, PF-cpGAN achieves significant improvements (i.e., approx. $77\%$ to
$88\%$ under $\pm 90\degree$).

Table 4: Rank-1 recognition rates ($\%$) across poses and illuminations under
Multi-PIE Setting-1.

Method | $\pm 90\degree$ | $\pm 75\degree$ | $\pm 60\degree$ | $\pm 45\degree$ | $\pm 30\degree$ | $\pm 15\degree$
---|---|---|---|---|---|---
HPN [11] | $29.82$ | $47.57$ | $61.24$ | $72.77$ | $78.26$ | $84.23$
c-CNN [52] | $47.26$ | $60.7$ | $74.4$ | $89.0$ | $94.1$ | $97.0$
TP-GAN [17] | $64.0$ | $84.1$ | $92.9$ | $98.6$ | $99.99$ | $99.8$
PIM [57] | $75.0$ | $91.2$ | $97.7$ | $98.3$ | $99.4$ | $99.8$
CAPG-GAN [16] | $77.1$ | $87.4$ | $93.7$ | $98.3$ | $99.4$ | $99.99$
FNM$+$VGG-Face [31] | $41.1$ | $67.3$ | $83.6$ | $93.6$ | $97.2$ | $99.0$
FNM$+$Light CNN [31] | $55.8$ | $81.3$ | $93.7$ | $98.2$ | $99.5$ | $99.9$
PF-cpGAN | $88.1$ | $94.2$ | $97.6$ | $98.9$ | $99.9$ | $99.9$

For further testing on the Multi-PIE dataset under setting-1, we have also
plotted ROC curves and compared with other state-of-the-art methods. The ROC
curves for Multi-PIE dataset are given in Fig. 3(b). The curves clearly
indicate that the proposed method of PF-cpGAN improves upon other methods and
gives much better performance, even at FAR of 0.001.

###  5.5 Reconstruction of frontal and profile images

As noted in Sec. 1, the PF-cpGAN framework can also be used for reconstruction
of frontal images by using profile images as input and vice versa. The results
of reconstructing frontal images using the profile images as input are given
in Fig. 4, and the results of reconstructing profile images using the frontal
images as input is given in Fig. 5. The reconstruction procedure for frontal
images is given as follows: The profile image is given as input to the profile
U-Net generator and the feature vector generated at the bottleneck of the
profile generator (i.e., at the output of the encoder of the profile U-Net
generator) is passed through the decoder section of the frontal U-Net
generator to reconstruct the frontal image. Similarly the reconstruction
procedure for profile images is given as follows: The frontal image is given
as input to the frontal U-Net generator and the feature vector generated at
the bottleneck of the frontal generator (i.e., at the output of the encoder of
the frontal U-Net generator) is passed through the decoder section of the
profile U-Net generator to reconstruct the profile image. As we can see from
Fig. 4 and Fig. 5, the PF-cpGAN can preserve the identity and generate high-
fidelity faces from an unconstrained dataset such as CMU-MultiPIE. These
results show the robustness and effectiveness of PF-cpGAN for multiple use of
profile to frontal matching in the latent common embedding subspace, as well
as in the reconstruction of facial images.

###  5.6 Ablation Study

The objective function defined in (19) contains multiple loss functions:
coupling loss ($L_{cpl}$), perceptual loss ($L_{P}$), $L_{2}$ reconstruction
loss ($L_{2}$), and GAN loss ($L_{GAN}$). It is important to understand the
relative importance of different loss functions and the benefit of using them
in our proposed method. For this experiment, we use different variations of
PF-cpGAN and perform the evaluation using the IJB-A dataset. The variations
are: 1) PF-cpGAN with only coupling loss and $L_{2}$ reconstruction loss
($L_{cpl}+L_{2}$); 2) PF-cpGAN with coupling loss, $L_{2}$ reconstruction
loss, and GAN loss ($L_{cpl}+L_{2}+L_{GAN}$); 3) PF-cpGAN with all the loss
functions ($L_{cpl}+L_{2}+L_{GAN}+L_{P}$).

We use these three variations of our framework and plot the ROC for profile to
frontal face verification using the features from the common embedding
subspace. We can see from Fig. 3(c) that the generative adversarial loss helps
improve the profile to frontal verification performance, and adding the
perceptual loss (blue curve) results in an additional performance improvement.
The reason for this improvement is that using perceptual loss along with the
contrastive loss leads to a more discriminative embedding subspace leading to
a better face recognition performance.

##  6 Conclusion

We proposed a new framework which uses a coupled GAN for profile to frontal
face recognition. The coupled GAN contains two sub-networks which project the
profile and frontal images into a common embedding subspace, where the goal of
each sub-network is to maximize the pair-wise correlation between profile and
frontal images during the process of projection. We thoroughly evaluated our
model on several standard datasets and the results demonstrate that our model
notably outperforms other state-of-the-art algorithms for profile to frontal
face verification. Moreover, the improvement achieved by different losses in
our proposed algorithm has been studied in an ablation study.

## References

  * [1] T. Ahonen, A. Hadid, and M. Pietikainen.  Face description with local binary patterns: Application to face recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(12):2037–2041, Dec 2006.
  * [2] S. Boumi, A. Vela, and J. Chini.  Quantifying the relationship between student enrollment patterns and student performance.  arXiv preprint arXiv:2003.10874, 2020.
  * [3] K. Cao, Y. Rong, C. Li, X. Tang, and C. C. Loy.  Pose-robust face recognition via deep residual equivariant mapping.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5187–5196, 2018.
  * [4] Q. Cao, Y. Ying, and P. Li.  Similarity metric learning for face recognition.  In 2013 IEEE International Conference on Computer Vision, pages 2408–2415, Dec 2013.
  * [5] J. Chen, V. M. Patel, and R. Chellappa.  Unconstrained face verification using deep cnn features.  In IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–9, March 2016.
  * [6] J.-C. Chen, V. M. Patel, and R. Chellappa.  Unconstrained face verification using deep cnn features.  In 2016 IEEE winter conference on applications of computer vision (WACV), pages 1–9. IEEE, 2016.
  * [7] F. Cole, D. Belanger, D. Krishnan, A. Sarna, I. Mosseri, and W. T. Freeman.  Synthesizing normalized faces from facial identity features.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3386–3395, 2017.
  * [8] A. Dabouei, F. Taherkhani, S. Soleymani, J. Dawson, and N. Nasrabadi.  Boosting deep face recognition via disentangling appearance and geometry.  In The IEEE Winter Conference on Applications of Computer Vision (WACV), 2020.
  * [9] N. Dalal and B. Triggs.  Histograms of oriented gradients for human detection.  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 886–893, June 2005.
  * [10] J. Deng, J. Guo, N. Xue, and S. Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [11] C. Ding and D. Tao.  Pose-invariant face recognition with homography-based normalization.  Pattern Recognition, 66:144–152, 2017.
  * [12] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  In Proc. Neural Information Processing Systems, pages 2672–2680. 2014.
  * [13] R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker.  Multi-pie.  In 2008 8th IEEE International Conference on Automatic Face Gesture Recognition, pages 1–8, Sep. 2008.
  * [14] M. A. Hasnat, J. Bohné, J. Milgram, S. Gentric, and L. Chen.  von mises-fisher mixture model-based deep learning: Application to face verification.  ArXiv, abs/1706.04264, 2017.
  * [15] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2015.
  * [16] Y. Hu, X. Wu, B. Yu, R. He, and Z. Sun.  Pose-guided photorealistic face rotation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8398–8406, 2018.
  * [17] R. Huang, S. Zhang, T. Li, and R. He.  Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2439–2448, 2017.
  * [18] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros.  Image-to-image translation with conditional adversarial networks.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5967–5976, 2017.
  * [19] J. Jiang, R. Hu, Z. Wang, and Z. Cai.  CDMMA: Coupled discriminant multi-manifold analysis for matching low-resolution face images.  Signal Processing, 124:162–172, 2016.
  * [20] J. Johnson, A. Alahi, and L. Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In Proc. European Conference on Computer Vision (ECCV), 2016.
  * [21] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  CoRR, abs/1412.6980, 2015.
  * [22] B. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. E. Allen, P. Grother, A. Mah, M. Burge, and A. K. Jain.  Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus benchmark a.  2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1931–1939, 2015.
  * [23] B. F. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. Allen, P. Grother, A. Mah, and A. K. Jain.  Pushing the frontiers of unconstrained face detection and recognition: IARPA Janus benchmark a.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1931–1939, 2015.
  * [24] P. Li, L. Prieto, D. Mery, and P. J. Flynn.  On low-resolution face recognition in the wild: Comparisons and new techniques.  IEEE Transactions on Information Forensics and Security, page 1–1, 2019.
  * [25] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song.  Sphereface: Deep hypersphere embedding for face recognition.  The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6738–6746, June 2017.
  * [26] Y. Liu, H. Li, and X. Wang.  Rethinking feature discrimination and polymerization for large-scale recognition.  ArXiv, abs/1710.00870, 2017.
  * [27] I. Masi, S. Rawls, G. Medioni, and P. Natarajan.  Pose-aware face recognition in the wild.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4838–4846, June 2016.
  * [28] B. Maze, J. Adams, J. A. Duncan, N. Kalka, T. Miller, C. Otto, A. K. Jain, W. T. Niggel, J. Anderson, J. Cheney, and P. Grother.  IARPA Janus benchmark - c: Face dataset and protocol.  In 2018 International Conference on Biometrics (ICB), pages 158–165, Feb 2018.
  * [29] M. Mirza and S. Osindero.  Conditional generative adversarial nets.  arXiv preprint arXiv:1411.1784, 2014.
  * [30] O. M. Parkhi, A. Vedaldi, and A. Zisserman.  Deep face recognition.  In BMVC, 2015.
  * [31] Y. Qian, W. Deng, and J. Hu.  Unsupervised face normalization with extreme pose and expression in the wild.  In IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [32] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee.  Generative adversarial text to image synthesis.  In International Conference on Machine Learning (ICML), 2016.
  * [33] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241. Springer, 2015.
  * [34] F. Schroff, D. Kalenichenko, and J. Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proc. IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [35] F. Schroff, D. Kalenichenko, and J. Philbin.  Facenet: A unified embedding for face recognition and clustering.  CoRR, abs/1503.03832, 2015.
  * [36] S. Sengupta, J. Chen, C. Castillo, V. M. Patel, R. Chellappa, and D. W. Jacobs.  Frontal to profile face verification in the wild.  In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–9, March 2016.
  * [37] S. Shekhar, V. M. Patel, and R. Chellappa.  Synthesis-based robust low resolution face recognition.  arXiv preprint arXiv:1707.02733, 2017.
  * [38] K. Simonyan, O. M. Parkhi, A. Vedaldi, and A. Zisserman.  Fisher vector faces in the wild.  In BMVC, 2013.
  * [39] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [40] Y. Sun, Y. Chen, X. Wang, and X. Tang.  Deep learning face representation by joint identification-verification.  In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, pages 1988–1996, Cambridge, MA, USA, 2014. MIT Press.
  * [41] F. Taherkhani, H. Kazemi, A. Dabouei, J. Dawson, and N. M. Nasrabadi.  A weakly supervised fine label classifier enhanced by coarse supervision.  In Proceedings of the IEEE International Conference on Computer Vision, pages 6459–6468, 2019.
  * [42] F. Taherkhani, H. Kazemi, and N. M. Nasrabadi.  Matrix completion for graph-based deep semi-supervised learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5058–5065, 2019.
  * [43] F. Taherkhani, N. M. Nasrabadi, and J. Dawson.  A deep face identification network enhanced by facial attributes prediction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 553–560, 2018.
  * [44] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf.  Deepface: Closing the gap to human-level performance in face verification.  In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 1701–1708, June 2014.
  * [45] V. Talreja, T. Ferrett, M. C. Valenti, and A. Ross.  Biometrics-as-a-service: A framework to promote innovative biometric recognition in the cloud.  In 2018 IEEE ICCE, pages 1–6. IEEE, 2018.
  * [46] V. Talreja, M. C. Valenti, and N. M. Nasrabadi.  Multibiometric secure system based on deep learning.  In 2017 IEEE Global conference on signal and information processing (globalSIP), pages 298–302. IEEE, 2017.
  * [47] L. Tran, X. Yin, and X. Liu.  Disentangled representation learning gan for pose-invariant face recognition.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1283–1292, July 2017.
  * [48] L. Tran, X. Yin, and X. Liu.  Disentangled representation learning gan for pose-invariant face recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1415–1424, 2017.
  * [49] F. Wang, X. Xiang, J. Cheng, and A. L. Yuille.  Normface: L2 hypersphere embedding for face verification.  In Proceedings of the 25th ACM International Conference on Multimedia, pages 1041–1049, 2017.
  * [50] M. Wang and W. Deng.  Deep face recognition: A survey.  ArXiv, abs/1804.06655, 2018.
  * [51] C. Whitelam, E. Taborsky, A. Blanton, B. Maze, J. Adams, T. Miller, N. Kalka, A. K. Jain, J. A. Duncan, K. Allen, J. Cheney, and P. Grother.  IARPA Janus benchmark-b face dataset.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 592–600, July 2017.
  * [52] C. Xiong, X. Zhao, D. Tang, K. Jayashree, S. Yan, and T.-K. Kim.  Conditional convolutional neural network for modality-aware face recognition.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3667–3675, 2015.
  * [53] J. Yim, H. Jung, B. Yoo, C. Choi, D.-S. Park, and J. Kim.  Rotating your face using multi-task deep neural network.  2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 676–684, 2015.
  * [54] X. Yin, X. Yu, K. Sohn, X. Liu, and M. Chandraker.  Towards large-pose face frontalization in the wild.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3990–3999, 2017.
  * [55] X. Yin, X. Yu, K. Sohn, X. Liu, and M. K. Chandraker.  Towards large-pose face frontalization in the wild.  2017 IEEE International Conference on Computer Vision (ICCV), pages 4010–4019, 2017.
  * [56] P. Zhang, X. Ben, W. Jiang, R. Yan, and Y. Zhang.  Coupled marginal discriminant mappings for low-resolution face recognition.  Optik, 126(23):4352–4357, 2015.
  * [57] J. Zhao, Y. Cheng, Y. Xu, L. Xiong, J. Li, F. Zhao, K. Jayashree, S. Pranata, S. Shen, J. Xing, et al.  Towards pose invariant face recognition in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2207–2216, 2018.
  * [58] J. Zhao, Y. Cheng, Y. Xu, L. Xiong, J. Li, F. Zhao, K. Jayashree, S. Pranata, S. Shen, J. Xing, S. Yan, and J. Feng.  Towards pose invariant face recognition in the wild.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2207–2216, June 2018.


[1]Tomás Verdugo

[1]Instituto de Astronomía, Observatorio Astronómico Nacional, Universidad
Nacional Autónoma de México, Apartado postal 106, C.P. 22800, Ensenada, México

2] Unidad Académica de Física, Universidad Autónoma de Zacatecas, Calzada
Solidaridad esquina con Paseo a la Bufa S/Nt, C.P. 98060, Zacatecas, México

3] Escuela de Ingeniería, Universidad Central de Chile, Avenida Francisco de
Aguirre 0405, 171-0164, La Serena Coquimbo, Chile

4] Depto. de Física y Matemáticas, Universidad Iberoamericana Ciudad de
México, Prolongación Paseo de la Reforma 880, 01219, México,México

5] Facultad de Ingeniería, Universidad Autónoma de Querétaro, Centro
Universitario Cerro de las Campanas, 76010 Santiago de Querétaro,México

6] Instituto de Física y Astronomía, Universidad de Valparaíso, Avda. Gran
Bretaña, 1111, Valparaíso,Chile

# Synchronize your chrono-brane: Testing a variable brane tension model with
strong gravitational lensing

<EMAIL_ADDRESS>Mario H. Amante<EMAIL_ADDRESS>Juan
Magaña<EMAIL_ADDRESS>Miguel A. García-Aspeitia
<EMAIL_ADDRESS>Alberto Hernández-Almada<EMAIL_ADDRESS>Verónica Motta
<EMAIL_ADDRESS>*  [  [  [  [  [

###### Abstract

Brane world models have shown to be promising to understand the late cosmic
acceleration, in particular because such acceleration can be naturally
derived, mimicking the dark energy behaviour just with a five dimensional
geometry. In this paper we present a strong lensing joint analysis using a
compilation of early-type galaxies acting as a lenses, united with the power
of the well studied strong lensing galaxy cluster Abell 1689. We use the
strong lensing constraints to investigate a brane model with variable brane
tension as a function of the redshift. In our joint analysis we found a value
$n=7.8^{+0.9}_{-0.5}$, for the exponent related to the brane tension, showing
that $n$ deviates from a Cosmological Constant (CC) scenario (n=6). We obtain
a value for the deceleration parameter, $q(z)$ today,
$q(0)=-1.2^{+0.6}_{-0.8}$, and a transition redshift, $z_{t}=0.60\pm 0.06$
(when the Universe change from an decelerated phase to an accelerated one).
These results are in contrast with previous work that favors CC scenario,
nevertheless our lensing analysis is in agreement with a formerly reported
conclusion suggesting that the variable brane tension model is able to source
a late cosmic acceleration without an extra fluid as in the standard one.

###### keywords:

cosmology, gravitational lensing, dark energy, brane cosmology

##  1 Introduction

The accelerated expansion of the universe is one of the most profound
conundrums in modern cosmology, being firstly evidenced by Type Ia Supernovae
[SnIa 1, 2] and confirmed independently by measurements of the Cosmic
Microwave Background Radiation anisotropies [CMB, 3, 4], among other
observations [5]. Under the general relativity theory, the cosmological
constant (CC), arising from quantum vacuum fluctuations, is the simplest
explanation as source of this acceleration because it is consistent with
several cosmological tests [6, 7, 8]. Nevertheless, there is an inconsistency
($\sim 120$ orders in magnitude) between the theoretical estimation and
astrophysical observations, [see 9]. Besides, this CC hypothesis does not
offer an explanation of why the Universe is accelerating today ($z\sim 0.7$)
but not in another cosmic epoch [see 9, 10].

Braneworld models are plausible alternatives to the problem of the accelerated
expansion of the universe [11, 12, 13, 14, 15, 16, 17, 18, 19]. In these
models, our four dimensional observable Universe is a brane embedded in a five
dimensional bulk and the cosmic acceleration has a geometric nature. Some
authors [20] investigated a dark energy (DE) component in a braneworld
geometry with constant brane tension using several cosmological data. They
found a tension on the constraints obtained from low and high redshift data
which suggests that the evidence for the existence of a brane is not
significant. Later on, in a further study [21](hereafter GA2018), it was shown
that it is possible to drive the late-time cosmic acceleration with a five-
dimensional geometry without a DE component by considering a variable brane
tension (hereinafter, VBT) modeled with a polynomial function; the authors
called it the chrono-brane model. By combining several cosmological data,
SNIa, CMB, baryon acoustic oscillations (BAO), and observational Hubble data
(OHD) and their joint analysis, they constrained the exponent of the VBT
function as $n=6.19\pm 0.12$ which provides an extra term in the Friedmman
equation acting as a CC component at cosmic late times. Nevertheless, the VBT
model should be investigated at both galactic and galaxy cluster scales. The
last are the largest structures in the Universe and they provide insights into
the transition of the linear and non linear cosmological perturbations, i.e.
in the late dynamics of the Universe. Thus, it is necessary to consider more
cosmological data to shed light on the properties of the VBT model at these
scales.

In this work we extend that study using observations from strong gravitational
lensing at galaxy and galaxy clusters system scales, which in the last decade
have been considered more frequently in cosmological parameter estimation [see
22, 23, 24, 25, and references therein]. In particular cluster strong lensing
cosmography has been used in the past in a successfully way, but it requires
high-quality data, for example spectroscopy to measure redshifts of multiple
images. However, using a galaxy cluster to constraint cosmological models has
his caveats, many systematic effects can affect an individual strong-lensing
model, for example intrinsic degeneracies in the models, mass components
external to the cluster, or line-of-sight perturbers [24]. As comment by these
authors, a possible solution is to perform a combined cosmographical analysis
using several clusters. In the present work we propose a different approach
which consist in combining two lensing constraints at different scales: the
power of the well known lensing galaxy cluster Abell 1689 [26], and the
capability of early-type galaxies acting as lenses [27].

This paper is organized as follows. In Section 2, we present the theoretical
model. In Section 3, we summarize our data and methodology. In Section 4, we
describe our results, and we discuss them in Section 5. Finally, we present
our conclusions and remarks in Section 6.

Figure 1: 1D marginalized posterior distributions and the 2D confidence
contours ($1\sigma,2\sigma,3\sigma$) for the $\Omega_{m0}$ and $n$ parameters
of the VBT model for early-type galaxies (red) and galaxy cluster Abell 1689
(dark-blue) assuming a Gaussian prior on $\Omega_{m0}$ parameter. The star
indicates the mean value for the joint, and the triangle the mean value
reported by GA2018.

##  2 Theoretical model

The theoretical model that we are going to constrain, namely the VBT model or
chrono-brane model [GA2018], is a function of the scale factor (redshift),
i.e. $\lambda$(a) or $\lambda$(z). In particular, the authors propose a
polynomial function for the brane tension which is dominant in later times in
the Universe evolution, but subdominant in the early Universe. We refer the
reader to such work for a rigorous description, here we discuss briefly the
theoretical framework. We start with the VBT field equation as

$G_{\mu\nu}-8\pi GT_{\mu\nu}=\frac{1}{\lambda}\left[48\pi
G\Pi_{\mu\nu}+\frac{3}{4\pi G}\xi_{\mu\nu}\right],$ (1)

with

$\displaystyle\xi_{\mu\nu}=\mathcal{U}\left(u_{\mu}u_{\nu}+\frac{1}{3}\epsilon_{\mu\nu}\right)+\mathcal{P}_{\mu\nu},$
(2)
$\displaystyle\Pi_{\mu\nu}=-\frac{1}{4}T_{\mu\alpha}T^{\alpha}_{\nu}+\frac{1}{12}T^{\alpha}_{\alpha}T_{\mu\nu}+\frac{1}{24}g_{\mu\nu}[3T_{\alpha\beta}T^{\alpha\beta}-(T^{\alpha}_{\alpha})^{2}].$
(3)

where $G_{\mu\nu}$ is the standard Einstein tensor, $G$ is the Newton
gravitational constant, $\lambda$ is the brane tension, $\xi_{\mu\nu}$ is a
non-local Weyl tensor decomposed in its irreducible form which also contains
$\mathcal{U}$ as the non-local energy density, $\mathcal{P}_{\mu\nu}$ is the
non-local anisotropic stress tensor, $u_{\mu}$ is the four-velocity and
$\epsilon_{\mu\nu}\equiv g_{\mu\nu}+u_{\mu}u_{\nu}$, being $g_{\mu\nu}$ the
metric tensor associated. In addition, $T_{\mu\nu}$ and
$T=T^{\alpha}_{\alpha}$ are the standard energy-momentum tensor and scalar
respectively; finally $\Pi_{\mu\nu}$ contains a quadratic form of the energy-
momentum tensor.

The corrective term that come from brane world is related to the brane tension
that is given by $\lambda$, which in this model is not a constant. Therefore,
the low energy limit is considered when $\lambda\to\infty$, where the
traditional field equation of General Relativity is recovered, and for
$\lambda\to 0$ the extra term play a preponderant role. Finally, notice that
in this case we do not consider extra fields onto the bulk, neglecting those
terms that come from the non local tensor and only taking those fields that
live in the brane.

To study the brane cosmological dynamics, we consider the standard homogeneous
and isotropic line element of Friedmann-Lemaître-Robertson-Walker (FLRW) for a
flat geometry,

$ds^{2}=-dt^{2}+a(t)^{2}(dr^{2}+r^{2}d\Omega^{2}),$ (4)

where $d\Omega^{2}\equiv d\theta^{2}+\sin^{2}\theta d\varphi^{2}$ is the solid
angle in spherical coordinates and $a(t)$ is the scale factor. Moreover, the
energy-momentum tensor will be written as

$T_{\mu\nu}=pg_{\mu\nu}+(\rho+p)u_{\mu}u_{\nu},$ (5)

where $p$ and $\rho$ are the pressure and density of the fluid respectively.
Therefore, if we introduce the previously defined line element in Eq. (1)
together with the perfect fluid energy-momentum tensor (Eq. (5)), in the VBT
model the Friedmann equation can be written in terms of the redshift as

$E(z)^{2}=\Omega_{0m}(z+1)^{3}+\Omega_{0r}(z+1)^{4}+\frac{\mathcal{M}}{(z+1)^{n}}[\Omega_{0m}^{2}(z+1)^{6}+\Omega_{0r}^{2}(z+1)^{8}],$
(6)

where $E(z)\equiv H(z)/H_{0}$, $H_{0}$ is the Hubble constant, $\Omega_{m0}$
is the density parameter for matter (baryons plus dark matter), $\Omega_{0r}$
is the density parameter for radiation, and $\mathcal{M}\equiv
3H_{0}^{2}/16\pi G\lambda_{0}$. The free parameters, related with the brane
tension, $\lambda_{0}$ and $n$ are coupled through the equation
$\lambda(z)=\lambda_{0}(z+1)^{n}$ which is the polynomial function proposed in
GA2018, being $z$ the redshift.

Since $\Omega_{0r}\sim 10^{-5}$, the radiation terms in Eq. (6) can be
neglected when it does not dominate the Universe dynamics, and the Friedmann
equation reads as

$\displaystyle
E(z)^{2}=\Omega_{m0}(z+1)^{3}+\mathcal{M}\Omega_{m0}^{2}(z+1)^{6-n}.$ (7)

An interesting solution is obtained when $n=6$, because the second term in Eq.
(7) will be a constant given by the term $\mathcal{M}\Omega_{m0}^{2}$, acting
as a cosmological constant. Thus, the interpretation of the cosmological
constant will be in the five dimensional context. Additionally to this, we
have the following expression: $\mathcal{M}=(1-\Omega_{m0})\Omega_{m0}^{-2}$,
assuming $z=0$ and the flatness condition.

Moreover, the expression for the deceleration parameter reads

$q(z)=\frac{\Omega_{m0}}{2E^{2}(z)}\Big{\\{}(z+1)^{3}+(4-n)\mathcal{M}\Omega_{m0}(z+1)^{6-n}\Big{\\}},$
(8)

where $E(z)$ is given by Eq. (7).

Table 1: Mean values for the model parameters, $\Omega_{m0}$ and $n$, derived from each data set and the joint analysis. $\rho_{c}$ is the standard critical density of the Universe.  Data set | $\chi^{2}_{min}$ | $\chi^{2}_{red}$ | $\Omega_{m0}$ | $n$ | $\lambda_{0}/\rho_{c}$
---|---|---|---|---|---
SLS | 245.8 | 1.7 | $0.28^{+0.016}_{-0.017}$ | $7.9^{+0.9}_{-0.7}$ | $0.055^{+0.008}_{-0.007}$
Abell 1689 | 30.4 | 2.8 | $0.31\pm 0.017$ | $8.0^{+1.5}_{-0.8}$ | $0.070^{+0.010}_{-0.008}$
Joint | 282 | 1.8 | $0.28^{+0.016}_{-0.018}$ | $7.8^{+0.9}_{-0.5}$ | $0.056^{+0.008}_{-0.007}$

##  3 Data and methodology

We constrain the cosmological parameters of the VBT model, by combining for
the first time the strong lensing effect produced by elliptical galaxies
acting as a lens, and in the galaxy cluster Abell 1689. Here we present a
short description of the methods detailed in the works of [26], [28], and
[27], and how we combine both methodologies.

###  3.1 Strong lensing in elliptical galaxies

We use the compilation of early-type galaxies acting as lenses presented by
[27]. Their fiducial sample consists of $N_{SL}=143$ strong lensing systems
(SLS), with four measured properties: spectroscopically measured stellar
velocity dispersions $\sigma$, the Einstein radius $\theta_{E}$, the lens
redshift $z_{l}$ and the source redshift $z_{s}$.

We can constrain cosmological parameters following the steps provided by [29]
minimizing the chi square function given as

$\chi_{\mbox{Gal}}^{2}=\sum_{i=1}^{N_{SL}}\frac{\left[D^{th}\left(z_{l},z_{s};\bf{\Theta_{Cos}}\right)-D^{obs}(\theta_{E},\sigma^{2})\right]^{2}}{(\delta
D^{\rm{obs}})^{2}},$ (9)

where we define the ratio of two angular diameter distances $D\equiv
D_{ls}/D_{s}$. Thus, the theoretical ratio $D^{th}$ is calculated from Eq. 6,
using the definition of angular diameter
distance111$D(z)=\frac{c}{H_{0}(1+z)}\int_{0}^{z}\frac{dz^{\prime}}{E(z^{\prime})}$.
On the other hand, through the observationally measured properties, we get the
observed counterpart $D^{obs}$ , and $\delta D^{\rm{obs}}$ as the error
propagation of the $D^{obs}$ function. The vector $\mathbf{\Theta_{Cos}}$ is
formed by the parameters $n$ and $\Omega_{m}$.

###  3.2 Strong lensing in Abell 1689 galaxy cluster

While different galaxy clusters has been used to test cosmology [e.g., 24,
30], galaxy cluster Abell 1689 (A 1689) at $z$=0.184, is still competitive to
perform cosmological constraints, and it is a widely studied cluster [see 31].
A 1689 is modeled as a bi-modal mass distribution, with one central, dominant
large-scale potential hosting the brightest cluster galaxy (BCG) at its centre
[see 32]. A second large-scale potential is located at the north-east of the
cluster. Additionally, the model includes the galaxy scale dark matter haloes
associated with 58 cluster members.

Following [26], to reconstruct the A1689 mass model and simultaneously
constrain the cosmological parameters of the VBT cosmology we use the ’family
ratio’ which is defined as the angular diameter distance ratios of two images
from different sources. In particular, we use the same image catalog of [26],
which contains 28 images from $N_{f}$ = 12 families, all with measured
spectroscopic redshifts in the range $1.15<z_{\rm s}<4.86$. For the present
work, we adopt an error $\Delta^{2}$ = 0.5” in the position of the images [see
23], and the models are evaluated and optimized in the image plane.

In this case, the $\chi^{2}$ for a multiple image system $i$ is defined as

$\chi_{i}^{2}=\sum_{j=1}^{n_{i}}\frac{\left|\vec{x}_{\rm
obs}^{j}-\vec{x}^{j}(\mathbf{\Theta_{W}})\right|^{2}}{\Delta^{2}}\;,$ (10)

where $n_{i}$ is the number of multiple images for the source $i$,
$\vec{x}_{\rm obs}^{j}$ is the observed position corresponding to image $j$,
and $\vec{x}^{j}(\mathbf{\Theta_{W}})$ is the position of image $j$ predicted
by the current model, whose total parameters, i.e. the cosmological parameters
and the cluster parameters (see below) are included in the vector
$\mathbf{\Theta_{W}}$. Thus,
$\chi^{2}_{\mbox{Clu}}=\sum_{1}^{N_{f}}\chi_{i}^{2}$, is the total chi square
function in this case.

###  3.3 Combining data sets

The probability distribution function of the model parameters is computed via
LENSTOOL222Publicly available at: https://git-cral.univ-
lyon1.fr/lenstool/lenstool LENSTOOL/ code. This code uses a Bayesian Monte
Carlo Markov chain algorithm to search for the most likely parameters in the
modeling, and it has been used in a large number of clusters studies, and
characterized in [33]. For the present work we have incorporated a routine
into LENSTOOL in order to compute the likelihood for the galactic systems in
combination with the reconstruction of the mass model for A1689.

Considering the Gaussian likelihood $\mathcal{L}\propto
e^{-\chi^{2}_{\mathrm{Tot}}/2}$, the total $\chi^{2}$ reads

$\chi_{\mathbf{Tot}}^{2}=\chi^{2}_{\mbox{Gal}}+\chi^{2}_{\mbox{Clu}},$ (11)

being $\chi^{2}_{\mathrm{Tot}}$ the chi-square function constructed using the
two complementary approaches regarding strong lensing measurements.

In our calculations we have assumed a 3$\sigma$ Gaussian prior in the
$\Omega_{m0}=0.311\pm 0.0056$ parameter according to the observations from
Planck [34], and we fix the $h=0.7403\pm 0.0142$ parameter (due to its
independence with the present method) to the value reported by [35] using a
model independent approach. We also assume a uniform prior for the $n$
parameter within the $0<n<10$ region.

In addition to the cosmological parameters, for A 1689 we have 19 parameters
to be fit: the six parameters (the spatial coordinates $x$, $y$; the
ellipticity, $e$; angle position, $\theta$; $r_{\rm core}$, and the velocity
dispersion, $\sigma$) for each of the main and secondary clumps, five
parameters ($x$, $y$, $r_{\rm core}$, $r_{\rm cut}$, $\sigma_{0}$) for the
dominant BCG in the main clump, and two parameters ($r_{\rm cut}^{*}$,
$\sigma^{*}_{0}$) for the galaxy-scale clumps. The uniform priors in the
parameters were set defining regions around the best values reported by [26]
and [32], however we check that the boundaries were not reached.

##  4 Results

Figure 2: Comparison between the theoretical model and $H(z)$ data (top panel)
and the reconstructed $q(z)$ (bottom panel) using the results of the joint
analysis constraints. The $\Lambda$CDM dynamics is plotted for comparison.
Figure 3: The confidence contours ($1\sigma,2\sigma,3\sigma,4\sigma$, and
$5\sigma$) for the $\lambda_{0}/\rho_{c}$ and $n$ parameters of the VBT model
from the joint analysis. The star indicates the mean value for the joint of
the present work. The square, diamond, pentagon, plus, and triangle markers
represent the mean values obtained from H(z), BAO, CMB, SNIa data, and the
joint reported by GA2018 respectively. The horizontal line represents $n=6$,
i.e., a model with CC.

Table 1 presents the mean values for $\Omega_{m0}$ and $n$ parameters for the
two data sets discussed in this work as well as those for the joint analysis.
In the Figure 1 we show the PDFs and the contours for these parameters. Note
that strong lensing observations at different scales (galaxy cluster and
galaxies) are consistent at 1$\sigma$ between both data sets. Although the
result of our joint model deviates more than 3-$\sigma$ from those reported in
the joint model of GA2018, this trend of larger values for $n$ it is also
obtained with other astrophysical data sets, e.g. with $H(z)$ or baryon
acoustic oscillations (BAO) measurements [GA2018]. The mean value
$n=7.8^{+0.9}_{-0.5}$ points towards a model that departs from $\Lambda$CDM,
with $\lambda(z)$ = $\lambda_{0}(1+z)^{7.8^{+0.9}_{-0.5}}$, we have a term in
the Friedmman equation that takes the role of a DE component at cosmic late
times. Furthermore, this extra dimension VBT model with the joint constraints
can explain the mass distribution in the galaxy cluster A1689 and their strong
lensing features without the existence of dark energy fluid.

One of the advantages of using lensing cosmography, when compared with other
standard probes, is the independence of the method with respect to the Hubble
constant $H_{0}$. Thus, it is interesting to compare how the Hubble parameter
is recovered by employing the results of our lensing fit. Using the result of
the joint analysis we construct the function $H(z)$ and we compare it with the
observational Hubble data measured using only differential age method compiled
by [36, also see [37]]. The comparison is depicted in the top panel of Figure
2. As before, we note that it is not consistent with a CC model. In the bottom
panel of Figure 2 we show the reconstruction of the deceleration parameter
$q(z)$ for the joint analysis. We obtain that the Universe change from a
decelerated phase to an accelerated one at $z_{t}=0.60\pm 0.06$, and deviates
from CC, with $q(0)=-1.2^{+0.6}_{-0.8}$ at $z=0$.

The Figure 3 illustrates the $\lambda_{0}/\rho_{c}-n$ confidence contours
($1\sigma,2\sigma,3\sigma,4\sigma$, and $5\sigma$) for the joint analysis;
where $\rho_{c}$ is the critical density of the Universe. For comparison, we
show the mean values obtained from $H(z)$ measurements, BAO, CMB, SNIa data,
and the joint reported by GA2018. Note the clear deviation from $\Lambda$CDM,
on each data set viewed independently. On one hand, the departure from $n=6$
in our joint analysis it is not surprising, since it is expected when using
lensing as a cosmological test, both, at galaxy scale [38, 27] and using a
galaxy cluster [24, 30]. On the other hand, it is interesting that three mean
values from independent cosmological constraints, falls inside the 2$\sigma$
region obtained from our joint analysis (see Figure 3). By considering
$\rho_{c}=8.070\times 10^{-11}h^{2}\,\text{eV}^{4}$, we estimate
$\lambda_{0}\approx 2.51^{+0.38}_{-0.33}\times 10^{-12}\text{eV}^{4}$ which is
consistent with a tension compatible with the variable tension model where the
brane terms does not affect the early physics, such as Nucleosynthesis among
others, and produce a late time acceleration according to the observations.
Our results are also consistent with the expected value for the CC energy
density which is $\rho<10^{-10}$eV4. In our case, the brane tension mimics the
CC behavior and the value of its energy density coincide with the expected
value of the CC.

##  5 Discussion

In the study by GA2018, they reported a VBT expression with
$\lambda(z)=\lambda_{0}(1+z)^{6.2{\pm 0.12}}$, proposing a brane that mimics
dark energy dynamics and consistent with CC. In our current investigation, the
strong lensing data provide $\lambda(z)=\lambda_{0}(1+z)^{7.8^{+0.9}_{-0.5}}$,
suggesting that the cosmic acceleration may be driven by a phantom dark
energy-like. It is important to note that this effect stems from the dynamics
of extra dimensions rather than a fluid with an equation of state (EoS)
characterized by $w<-1$. Additionally, in the present work, the observed
deviation of the deceleration parameter from its expected value in the CC
cosmology, at z = 0, is noteworthy. The deceleration parameter
$q(z=0)=-1.2^{+0.6}_{-0.8}$, and signals the potential influence of a phantom
dark energy-like component which contributes to an acceleration of the
Universe beyond what is conventionally anticipated in the CC model.

To look deeper into the preceding paragraph, we can write the Friedmann
equation under our constraints,

$E(z)^{2}=\Omega_{m0}(z+1)^{3}+\mathcal{M}\Omega_{m0}(z+1)^{-1.8},$ (12)

thus, it is possible to notice that in the future, $z\to-1$,
$E(z)^{2}\to\infty$ (Big Rip), due to the second term in the right side of Eq.
(12), suggesting the presence of a phantom DE. In this context, it is
important to emphasize that observations do not rule out phantom models in
this context [34, 39, 40].

Moreover, we can write the effective EoS from [GA2018] as

$w_{eff}(z)=\frac{2q-1}{3[1+2\Omega_{m0}\mathcal{M}(z+1)^{3-n}]}+\frac{\Omega_{m0}\mathcal{M}(z+1)^{3-n}[2q-(4-n)]}{3[1+2\Omega_{m0}\mathcal{M}(z+1)^{3-n}]},$
(13)

having for $z=0$ $w_{eff}\simeq 0.013$ a dust fluid, however the acceleration
is maintained because the condition

$w_{acc}<-\frac{1+\Omega_{mo}\mathcal{M}(4-n)}{3[1+2\Omega_{m0}\mathcal{M}]},$
(14)

is satisfied, since the EoS for acceleration demands $w_{acc}<0.47$ at $z=0$.

When we compare our $n$ constraints from the joint analysis of strong lensing
features with the previous one by GA2018, we have a tension of $\Delta
n\approx 2.25\sigma$ that can be appreciated in Figure 3. Note that this can
be written in terms of the parameter $\lambda_{0}$ as
$\Delta\lambda_{0}\approx 0.8\sigma$ and hence a tension of $\sim 3\sigma$ in
the $\lambda_{0}/\rho_{c}$-$n$ parameter space (see Fig. 3). For the
$\Omega_{m0}$ parameter, the tension between the two works is
$\Delta\Omega_{m0}\approx 1.59\sigma$.

To highlight the differences among the joint constraints, we calculated a
combined set of constraints on the parameters $n$ and $\Omega_{m0}$. This was
achieved by multiplying the probability distribution functions of the two
works under consideration: the joint analysis conducted by GA2018 and our own
joint analysis presented in this paper (see Figure 4). Note that there is a
slight tension at the $2\sigma$ confidence level. The disparities in the VBT
($\Omega_{m0},n$) parameters from both joint analyses could be attributed to
various factors, such as errors in the Einstein radius in the SLS, image-
positional errors in the strong lensing features in A1689, among others [see
23, 27]. It is also worth noting that the joint constraints by GA2018 are
influenced by CMB data, and the conflict between early and late data emerges
when we examine the data individually. To shed some light on this, we
calculate the Figure-of-Merit (FoM, see [23] and references therein). This
tool allows us to assess the quality of constraints from different datasets
and is calculated as

$\mathrm{FoM}=\frac{1}{\sqrt{\det\mathrm{cov}(p_{1},p_{2},p_{3},...)}}$ (15)

where $\mathrm{cov}(p_{1},p_{2},p_{3},...)$ is the covariance matrix of the
$p_{i}$ parameters. Larger FoM values imply stronger bounds on the parameters,
as they correspond to smaller error ellipses. By choosing the $\Omega_{m0}$
and $n$ parameters, we obtain the following FoM values:
$24.30,29.92,130.02,672.22,75.91,56.29$ for OHD, SNIa, BAO, CMB, SLS, and
A1689 data, respectively. Note that the constraints of this work are stronger
than those obtained from OHD and SNIa but weaker than those from BAO and CMB
data.

Finally, it is important to bear in mind that the technique presented in this
work is complementary to that of GA2018. This approach could assist in
breaking the degeneracy between VBT parameters, obtaining stronger
constraints, and determining whether this model could account for the late
cosmic acceleration.

Figure 4:  Comparison of confidence contours ($1\sigma$, and $2\sigma$) for
$\Omega_{m0}$ and $n$ parameters. Blue contours from the Joint analysis of
this work. Gray contours from the joint analysis by GA2018. The purple
contours represents the joint analysis of the joints (see text).

##  6 Conclusions and Remarks

In this paper we propose a new approach combining for the first time two
lensing data sets at different scales: the constraints provided by A1689
galaxy cluster and those that come from early-type galaxies acting as lenses.
In order to test the capability of the method we study the VBT (chrono-brane)
model. This model is capable of reproducing a late cosmic acceleration in a
five dimensional scenario, without assuming an extra fluid as in the standard
one, and has been study previously using different data sets. However, in the
present work, we use a different methodology which is independent of $H_{0}$,
avoiding the problem of $H_{0}$ tension presented in other measurements [41,
35].

We found that the joint strong lensing analysis provides, within the errors,
similar values of the parameters related to the brane tension, as those
reported previously by GA2018 using different observational data sets. This
demonstrate that our method can be used to constrain cosmological parameters
as a complement to other standard probes. We found that $n=7.8^{+0.9}_{-0.5}$,
which shows that it deviates from a CC scenario ($n=6$) in the joint analysis
at 3 $\sigma$. This result is in contrast with the one reported by GA2018,
because in the present work, we found an indication of a DE that evolves as
$\sim(z+1)^{-1.8}$ with an origin of extra dimensions. Additionally, the DE
expression has a singularity in $z=-1$ which represents a Big Rip in a finite
value of $z$. This behavior is representative of a Phantom DE normally driven
by a scalar field with a negative kinetic energy [42].

However, our lensing analysis is in agreement with GA2018 in the sense that
the VBT scenario is able to source a late cosmic acceleration without an extra
fluid, the five dimensional geometry naturally produced it cosmic feature
considering a brane with variable tension embedded in an extra dimension
called the bulk. Thus, the VBT (chrono-brane) scenario is a promising
alternative to the problem of the accelerated expansion of the Universe
without considering extra fields like the cosmological constant.

In a future study it is necessary to implement a coupling of the variable
brane tension with the equation of state in order to have a more general
brane-world model. Besides, as the VBT (chrono-brane) has shown to be
consistent with different observations at different scales, it should be
investigated at perturbative level. These further studies could be performed
using the lensing data presented here joined with other data sets, as
observational Hubble data, Type Ia Supernovae, BAO, and CMB data. Moreover,
our method has demonstrate to be competitive with those cosmological probes.
In the coming years, high-quality imaging and spectroscopic data will be
obtained from different astronomical facilities and surveys, driven remarkable
progress in strong lensing modeling of galaxy clusters [e.g. 43], and
producing the discovery of thousands of galaxy-scale lensing system [44, 45].
The inclusion of strong lensing systems along with other astrophysical
observations will allow a more robust analysis to comprehend the dark sectors
of the Universe.

Acknowledgments We thank the anonymous referee for thoughtful remarks and
suggestions. TV, JM, MGA, AHA and VM acknowledge partial support from project
ANID Vinculación Internacional FOVI220144. VM acknowledges partial support
from Centro de Astrofísica de Valparaíso. M.A.G.-A. acknowledges support from
cátedra Marcos Moshinsky (MM), Universidad Iberoamericana for support with the
SNI grant. The numerical analysis was also carried out by Numerical
Integration for Cosmological Theory and Experiments in High-energy
Astrophysics (NICTE-HA) cluster at IBERO University, acquired through cátedra
MM support and Graphic-Power Unit Integrated Numerical Analize (GUINA) at
UCEN-Coquimbo acquired through Fondequip EQM200216. M.H.A acknowledges support
from proyecto CONACYT Estancias Posdoctorales and SNI grant.

## Declarations

  * •

Availability of data and materials

This manuscript has no associated data or the data will not be deposited.
[Authors’ comment: This is theoretical research work, and is based upon the
analysis of public observational data (previously published by different
authors). So no additional data are associated with this work.]

## References

  *   * [1] Riess, A. G. _et al._ Observational evidence from supernovae for an accelerating universe and a cosmological constant.  _The Astronomical Journal_ 116, 1009 (1998).  URL http://stacks.iop.org/1538-3881/116/i=3/a=1009.
  * [2] Perlmutter, S. _et al._ Measurements of $\Omega$ and $\Lambda$ from 42 High-Redshift Supernovae.  _ApJ_ 517, 565–586 (1999).
  * [3] Jaffe, A. H. _et al._ Cosmology from MAXIMA-1, BOOMERANG, and COBE DMR Cosmic Microwave Background Observations.  _Phys. Rev. Lett._ 86, 3475–3479 (2001).
  * [4] Pryke, C. _et al._ Cosmological Parameter Extraction from the First Season of Observations with the Degree Angular Scale Interferometer.  _ApJ_ 568, 46–51 (2002).
  * [5] Weinberg, D. H. _et al._ Observational probes of cosmic acceleration.  _Phys. Rep_ 530, 87–255 (2013).
  * [6] Carroll, S. M.  The Cosmological Constant.  _Living Reviews in Relativity_ 4, 1 (2001).
  * [7] Bull, P. _et al._ Beyond $\Lambda$ CDM: Problems, solutions, and the road ahead.  _Physics of the Dark Universe_ 12, 56–99 (2016).
  * [8] Cyburt, R. H., Fields, B. D., Olive, K. A. & Yeh, T.-H.  Big bang nucleosynthesis: Present status.  _Reviews of Modern Physics_ 88, 015004 (2016).
  * [9] Copeland, E. J., Sami, M. & Tsujikawa, S.  Dynamics of dark energy.  _Int. J. Mod. Phys._ D15, 1753–1936 (2006).
  * [10] Bamba, K., Capozziello, S., Nojiri, S. & Odintsov, S. D.  Dark energy cosmology: the equivalent description via different theoretical models and cosmography tests.  _Ap &SS_ 342, 155–228 (2012).
  * [11] Randall, L. & Sundrum, R.  A large mass hierarchy from a small extra dimension.  _Phys. Rev. Lett._ 83, 3370–3373 (1999).
  * [12] Randall, L. & Sundrum, R.  An Alternative to compactification.  _Phys. Rev. Lett._ 83, 4690–4693 (1999).
  * [13] Dvali, G. R., Gabadadze, G. & Porrati, M.  4-D gravity on a brane in 5-D Minkowski space.  _Phys. Lett._ B485, 208–214 (2000).
  * [14] Gergely, L. Á. & Keresztes, Z.  Irradiated asymmetric Friedmann branes.  _J. Cosmology Astropart. Phys._ 2006, 022 (2006).
  * [15] Gergely, L. A.  Eotvos branes.  _Phys. Rev._ D79, 086007 (2009).
  * [16] Sahni, V. & Shtanov, Y.  Braneworld models of dark energy.  _J. Cosmology Astropart. Phys._ 2003, 014 (2003).
  * [17] Sahni, V.  Cosmological Surprises from Braneworld models of Dark Energy.  _arXiv e-prints_ astro–ph/0502032 (2005).
  * [18] Abdalla, M. C. B., da Silva, J. M. H. & da Rocha, R.  Notes on the two-brane model with variable tension.  _Phys. Rev. D_ 80, 046003 (2009).  URL https://link.aps.org/doi/10.1103/PhysRevD.80.046003.
  * [19] Aros, R. & Estrada, M.  Study of branes with variable tension.  _Commun. Theor. Phys._ 68, 595 (2017).
  * [20] García-Aspeitia, M. A., Magaña, J., Hernández-Almada, A. & Motta, V.  Probing dark energy with braneworld cosmology in the light of recent cosmological data.  _Int. J. Mod. Phys. D_ 27, 1850006 (2017).
  * [21] García-Aspeitia, M. A. _et al._ Brane with variable tension as a possible solution to the problem of the late cosmic acceleration.  _Phys. Rev. D_ 97, 101301 (2018).
  * [22] Cao, S., Biesiada, M., Gavazzi, R., Piórkowska, A. & Zhu, Z.-H.  Cosmology with Strong-lensing Systems.  _ApJ_ 806, 185 (2015).
  * [23] Magaña, J. _et al._ Strong Lensing Modeling in Galaxy Clusters as a Promising Method to Test Cosmography. I. Parametric Dark Energy Models.  _ApJ_ 865, 122 (2018).
  * [24] Caminha, G. B., Suyu, S. H., Grillo, C. & Rosati, P.  Galaxy cluster strong lensing cosmography. Cosmological constraints from a sample of regular galaxy clusters.  _A &A_ 657, A83 (2022).
  * [25] Shajib, A. J. _et al._ Strong Lensing by Galaxies.  _arXiv e-prints_ arXiv:2210.10790 (2022).
  * [26] Jullo, E. _et al._ Cosmological constraints from strong gravitational lensing in clusters of galaxies.  _Science_ 329, 924–927 (2010).
  * [27] Amante, M. H., Magaña, J., Motta, V., García-Aspeitia, M. A. & Verdugo, T.  Testing dark energy models with a new sample of strong-lensing systems.  _MNRAS_ 498, 6013–6033 (2020).
  * [28] Magaña, J., Motta, V., Cárdenas, V. H., Verdugo, T. & Jullo, E.  A Magnified Glance into the Dark Sector: Probing Cosmological Models with Strong Lensing in A1689.  _ApJ_ 813, 69 (2015).
  * [29] Grillo, C., Lombardi, M. & Bertin, G.  Cosmological parameters from strong gravitational lensing and stellar dynamics in elliptical galaxies.  _Astron. Astrophys._ (2007).  [Astron. Astrophys.477,397(2008)].
  * [30] Limousin, M., Beauchesne, B. & Jullo, E.  Dark matter in galaxy clusters: Parametric strong-lensing approach.  _A &A_ 664, A90 (2022).
  * [31] Bina, D. _et al._ MUSE observations of the lensing cluster Abell 1689\.  _A &A_ 590, A14 (2016).
  * [32] Limousin, M. _et al._ Combining Strong and Weak Gravitational Lensing in Abell 1689.  _ApJ_ 668, 643–666 (2007).
  * [33] Jullo, E. _et al._ A Bayesian approach to strong lensing modelling of galaxy clusters.  _New J. Phys._ 9, 447 (2007).
  * [34] Planck Collaboration _et al._ Planck 2018 results. VI. Cosmological parameters.  _A &A_ 641, A6 (2020).
  * [35] Riess, A. G., Casertano, S., Yuan, W., Macri, L. M. & Scolnic, D.  Large Magellanic Cloud Cepheid Standards Provide a 1% Foundation for the Determination of the Hubble Constant and Stronger Evidence for Physics beyond $\Lambda$CDM.  _ApJ_ 876, 85 (2019).
  * [36] Moresco, M. _et al._ A 6% measurement of the Hubble parameter at z~0.45: direct evidence of the epoch of cosmic re-acceleration.  _J. Cosmology Astropart. Phys._ 2016, 014 (2016).
  * [37] Magaña, J., Amante, M. H., Garcia-Aspeitia, M. A. & Motta, V.  The Cardassian expansion revisited: constraints from updated Hubble parameter measurements and type Ia supernova data.  _MNRAS_ 476, 1036–1049 (2018).
  * [38] Chen, Y., Li, R., Shu, Y. & Cao, X.  Assessing the effect of lens mass model in cosmological application with updated galaxy-scale strong gravitational lensing sample.  _MNRAS_ 488, 3745–3758 (2019).
  * [39] Caldwell, R. R.  A phantom menace? Cosmological consequences of a dark energy component with super-negative equation of state.  _Physics Letters B_ 545, 23–29 (2002).
  * [40] Beltrán Jiménez, J., Lazkoz, R., Sáez-Gómez, D. & Salzano, V.  Observational constraints on cosmological future singularities.  _European Physical Journal C_ 76, 631 (2016).
  * [41] Riess, A. G. _et al._ A 2.4% Determination of the Local Value of the Hubble Constant.  _ApJ_ 826, 56 (2016).
  * [42] Caldwell, R. R., Kamionkowski, M. & Weinberg, N. N.  Phantom energy and cosmic doomsday.  _Phys. Rev. Lett._ 91, 071301 (2003).
  * [43] Bergamini, P. _et al._ A state-of-the-art strong-lensing model of MACS J0416.1$-$2403 with the largest sample of spectroscopic multiple images.  _A &A_ 674, A79 (2023).
  * [44] Oguri, M. & Marshall, P. J.  Gravitationally lensed quasars and supernovae in future wide-field optical imaging surveys.  _MNRAS_ 405, 2579–2593 (2010).
  * [45] Collett, T. E.  The Population of Galaxy-Galaxy Strong Lenses in Forthcoming Optical Imaging Surveys.  _ApJ_ 811, 20 (2015).