# FaceGPT: Self-supervised Learning to Chat
about 3D Human Faces

Haoran Wang$1$ Mohit Mendiratta$1$ Christian Theobalt $1$ Adam
Kortylewski$1,2$
$1$ Max Planck Institute for Informatics $2$ University of Freiburg

###### Abstract

We introduce FaceGPT, a self-supervised learning framework for Large Vision-
Language Models (VLMs) to reason about 3D human faces from images and text.
Typical 3D face reconstruction methods are specialized algorithms that lack
semantic reasoning capabilities. FaceGPT overcomes this limitation by
embedding the parameters of a 3D morphable face model (3DMM) into the token
space of a VLM, enabling the generation of 3D faces from both textual and
visual inputs. FaceGPT is trained in a self-supervised manner as a model-based
autoencoder from in-the-wild images. In particular, the hidden states of LLM
is projected into 3DMM parameters and subsequently rendered as 2D face image
to guide the self-supervised learning process via image-based reconstruction.
Without relying on expensive 3D annotations of human faces, FaceGPT obtains a
detailed understanding about 3D human faces, while preserving the capacity to
understand general user instructions. Our experiments demonstrate that FaceGPT
not only achieves high-quality 3D face reconstructions but also retains the
ability for general-purpose visual instruction following. Furthermore, FaceGPT
learns fully self-supervised to generate 3D faces based on complex textual
inputs, which opens a new direction in human face analysis.

##  1 Introduction

In this work, we address the problem of reasoning about 3D human faces from
images and text. Traditional monocular 3D face reconstruction methods
typically estimate the parameters of a 3D morphable model [3, 41, 12, 13, 23]
given 2D face images as input. However, these methods lack the capability to
reason about faces based on textual input. Unlike these systems, humans can
vividly imagine and even draw faces based solely on either face images or
textual descriptions. Motivated by recent advances in large vision-Language
models (VLMs) [26, 48], we aim to explore a path forward towards enabling
vision-language systems to obtain an in-depth reasoning-based understanding of
3D faces.

To investigate this question, we present FaceGPT, a vision-language model with
an intricate ability to reason about 3D human faces from visual and textual
input. We represent faces as 3D morphable model (3DMM) parameters that include
parameters for the 3D face shape, expression, albedo, and scene illumination.
Following related work on image segmentation [22] and human pose estimation
[15], we extend the original token space of the VLM by incorporating a new
<FACE> token that is decoded into 3DMM parameters using an MLP (Fig. 1). Thus,
when combined with a differentiable computer graphics renderer [35], the VLM
model becomes capable to synthesize face images. This enables us to formulate
FaceGPT within a model-based autoencoder framework [41], and hence to train
our model in a fully self-supervised manner from in-the-wild images. To the
best of our knowledge, this is the first work combining vision-language model
with an inverse graphics pipeline. During training, we freeze the visual
encoder of the VLM while training the MLP and the LLM using LORA [20]. The
model is trained with three types of data: (1) Standard multi-modal
instruction tuning data to retain the general capability and quality of the
VLM. (2) In-the-wild face images for training the <face> token and 3DMM
projection layers via self-supervised inverse rendering. (3) Text-to-3DMM data
that enables the generation of 3DMM parameters from text input only. We
construct this dataset from a set of face images in the wild by running an
off-the-shelf self-supervised monocular face reconstruction method [23] and by
generating textual descriptions of the faces via the original VLM.

We evaluate FaceGPT on a variety of tasks, including traditional 3D face
reconstruction, text-to-3DMM face generation and general-purpose visual
instruction following. We demonstrate that FaceGPT becomes a general-purpose
model that achieves competitive results at 3D face reconstruction when
compared to specialized methods, while retaining competitive visual
instruction following capabilities, as well as being capable of reasoning-
based text-to-3D face generation. Beyond face analysis, we believe that the
design principles underlying FaceGPT are general and it also suggests a
pathway towards a self-supervised integration of the "world knowledge" that
VLMs derive from extensive textual data and the structured 3D representations
of the visual world via computer graphics.

In summary, our work makes the following concrete contributions:

  * •

To the best of our knowledge, this is the first work that enables vision-
language models to learn a detailed 3D face understanding in a fully self-
supervised manner.

  * •

We show that VLMs can learn text-based face reconstruction, which predicts 3D
human faces given user instructions, in a fully self-supervised manner.

  * •

Our experiments on traditional 3D face reconstruction, visual instruction
following and text-based face reconstruction, demonstrate the general face
understanding capabilities of FaceGPT.

Figure 1: We introduce FaceGPT, a large vision-language model that learns to
produce 3D human faces (in terms of 3DMM parameters) in a fully-self-
supervised manner. When prompted with face images and task-specific questions,
FaceGPT can output a special <FACE> token of which the corresponding feature
embedding (red) can be decoded into 3DMM parameters, that encode the face
shape $\alpha$, expression $\delta$, the texture $\gamma$, the light $\phi$
and camera $c$ parameters. When decoded with a 3DMM and differentiable
renderer, this enables a fully self-supervised learning via inverse rendering.
FaceGPT can produce 3D human faces from text-only input (first row), as well
as multi-modal input (second row) without requiring any 3D annotations, while
retaining general chatting abilities (last row).

##  2 Related works

###  2.1 Monocular model-based face reconstruction

Realistically reconstructing digital human faces has been a longstanding
challenge in computer vision and graphics due to their vast potential
applications. Traditional methods primarily use parametric 3D Morphable Models
(3DMM) [3, 31, 24] with PCA for dimensionality reduction to simplify high-
dimensional 3D face scans, serving as a 3D prior for representing unique
facial characteristics and providing precise control. Recently, deep learning-
based methods that map 2D images to 3D face models have gained popularity.
Early methods struggled with the need for extensive 3D facial scan data paired
with 2D images, which was labor-intensive and costly. This limitation was
addressed with the introduction of model-based face autoencoders (MoFA) [41]
that enabled self-supervised 3D face reconstruction. MoFA uses a
differentiable rendering layer to minimize differences between input and
rendered images, enabling end-to-end learning without ground-truth 3D faces,
leading to a number of effective extensions in the self-supervised learning
strategies [40, 39, 2, 16, 9]. RingNet [36] and DECA [14] use landmark-based
training, predicting landmarks for input images and treating them as pseudo
ground truth, measuring the distance between 2D face landmarks and their
projections on the 3DMM surface. The FOCUS [23] framework jointly trains a
face autoencoder and an outlier segmentation network, which makes the method
robust to outliers such as occlusion and make-up. These advancements
significantly improved monocular model-based face reconstruction, making it
more efficient and effective. However, these methods are highly specialized
and lack a deep understanding of the semantics of human faces or the ability
to relate faces to language, limiting their overall scope and effectiveness.

###  2.2 Text-to-3D Face Generation and Manipulation

Text-to-3D face generation and manipulation methods aim to integrate textual
information for creating and editing 3D faces. Methods like Dreamface [46] and
Describe3D [43] generate text-conditioned texture maps to render 3D morphable
models (3DMM). TG-3DFace [45] advances this by using tri-plane neural
representations and extending the 3D-aware GAN, EG3D [7], for end-to-end text-
conditioned generation. It incorporates a face parser and attribute classifier
for fine-grained semantics and uses a camera pose conditional discriminator to
guide learning. For text-guided 3D face manipulation, methods like Latent3D
[6] and ClipFace [1] optimize intermediate layers with a CLIP-based loss to
generate UV-texture maps or predict texture and expression latent codes. These
methods, however, rely on 3D scan data and to train new mappers for each text
instruction. Unlike these task-specific approaches, FaceGPT is a general-
purpose model that reasons about 3D human faces from images, text, or both by
leveraging general visual knowledge. Our model can interact with users through
conversations, discussing facial features and providing relevant responses,
while also being capable of following general user instructions.

###  2.3 Multimodal Large Language Models

Large Language Models (LLMs) are rapidly transforming various fields [32, 4,
29, 30]. While proprietary models like OpenAI’s ChatGPT [29] and GPT-4 [30]
dominate the landscape, open-source alternatives such as Vicuna [8], LLaMA
[42], and Alpaca [38] support research efforts. However, LLMs mainly focus on
generating text as output given text-only input. The integration of additional
modalities into LLMs represents an active area of research.

Multi-Modal Large Language Models (MM-LLMs) are emerging, extending LLMs’
capabilities beyond text to encompass a broader spectrum of modalities,
including images, videos, and audio. In the realm of image-text understanding,
recent endeavors like LLaVA [27] and MiniGPT-4 [48] incorporate vision
encoders to interpret images and align their features with language embeddings
using projection layers. Moreover, cutting-edge models such as PandaGPT [37],
ImageBind [18], and NeXT-GPT [44] exhibit impressive versatility in handling
diverse modalities, aligning embeddings from text, images, audio, and video
with language as both input and output. To enhance LLM to natively output more
modalities, approaches like LISA [22] connects LLaVA with a decoder to
generate text and segmentation masks, while ChatPose [15] specializes in human
pose information. However, these methods typically rely on supervised
learning.

In our study, we explore the integration of human faces as a novel modality
for LLMs. We assess (1) LLMs’ ability to generate 3D faces from text or image
inputs and (2) their capability to understand human facial features and
integrate this understanding into their broader functionality without
supervision.

##  3 Method

Figure 2: Architecture of FaceGPT. Our model consists of a multi-modal LLM,
which includes a vision encoder, a vision projection layer, and the LLM
itself, along with a 3DMM projection layer, denoted as $\sigma$, and the
parametric Basel face model [3]. During the training phase, we focus on
training the $\sigma$ projection layer and fine-tuning the LLM, while keeping
other components frozen. The network components are trained using a
differentiable renderer to compute a self-supervised loss.

In this work, our aim is to augment existing Large Vision Language Models
(VLMs) with the ability of reasoning about 3D human faces without requiring
manual human efforts. Inspired by established model-based face reconstruction
methods, we treat the 3D human face as a separate modality within the LLM
framework and represent it with 3D Morphable Model (3DMM) [3]parameters that
represent the face shape, expression, albedo and illumination. In particular,
we introduce a <FACE> token into the token space of the LLM, which is mapped
into the 3DMM parameter space and subsequently rendered into a 2D image, hence
enabling self-supervised 3D facial reconstruction.

###  3.1 Model Architecture

Fig. 2 illustrates the FaceGPT architecture. The VLM has an extended
vocabulary which includes a new token <FACE> that specifically represents the
human face. Given an input image $x_{img}$ and/or text prompt $x_{txt}$, the
VLM $f$ predicts text responses:

${y}_{txt}=f(x_{img},x_{txt}),$ (1)

where ${y}_{txt}=[t_{1},\dots,t_{N}]$ is the output sequence of tokens with
corresponding hidden states $[h_{1},\dots,h_{N}]$. If one of the output tokens
$t_{n}\in y_{txt}$ is our newly defined <FACE> token, we can extract the
hidden state as $h_{\texttt{<FACE>}}=h_{n}\in\mathbb{R}^{4096}$ and project it
using an MLP $\sigma$ into the latent 3DMM parameters
$\theta=\sigma(h_{\texttt{<FACE>}})=[\alpha,\delta,\gamma,\phi,c]\in\mathbb{R}^{257}$,
i.e. the 3D face shape $\alpha\in\mathbb{R}^{80}$, facial expression
parameters $\delta\in\mathbb{R}^{64}$ and texture $\gamma\in\mathbb{R}^{80}$
of a 3DMM, as well as the spherical harmonics illumination
$\phi\in\mathbb{R}^{27}$ and camera parameters $c\in\mathbb{R}^{6}$ of the
scene. The 3D vertices and triangles, as well as the color of the face mesh
are then determined using the standard 3DMM model $M(\theta)$ as described in
[13]. Using an orthographic camera model the reconstructed 3D face mesh can be
rendered into 2D space using a differentiable renderer $\Pi$, hence producing
the final reconstructed face image $\hat{y}_{rec}$. The process can be
summarized as:

$\displaystyle\theta$ $\displaystyle=\sigma(h_{\texttt{<FACE>}})$ (2a)
$\displaystyle\hat{y}_{rec}$ $\displaystyle=\Pi(M(\theta))$ (2b)

Note that the FaceGPT architecture can be seen as a new type of language-based
autoencoder, with a VLM as encoder and a computer graphics decoder that is
based on the 3DMM.

###  3.2 Self-supervised Training

Our objective is to develop a VLM that is able to learn an enhanced
comprehension of 3D human faces in a self-supervised manner. Ultimately, the
model should not only understand user instructions reliably, but also to
accurately reconstruct 3D faces from visual or text input. To accomplish this,
we have devised a self-supervised approach that incorporates 3DMM
understanding into existing VLMs. This method allows us to leverage face data
without the need for costly 3DMM annotations. We also construct a text-to-3DMM
dataset in an unsupervised manner that supports this training paradigm,
enabling our model to learn these capabilities effectively and efficiently.

Training Objective. Inspired by the success of self-supervised learning in
LLMs [32], we augment the existing VLM training objective with a self-
supervised face reconstruction loss $L_{face}$ that avoids the expensive 3D
annotation of human faces in in-the-wild images via image-based reconstruction
to regress 3DMM parameters:

$L=\lambda_{txt}L_{txt}+L_{face}$ (3)

where $\lambda_{txt}$ and $\lambda_{face}$ are the weights for balancing the
losses.

Text prediction loss. We utilize the autoregressive objective $L_{txt}$ to
guide the model to generate correct text output and hence retain its general
capability to follow user instructions:

$L_{txt}=CE(\hat{y}_{txt},y_{txt}),$ (4)

where $y_{txt}$ is the ground truth text response and $CE$ is the cross
entropy loss.

Self-supervised face reconstruction loss. To incorporate the 3DMM as a new
modality into an existing VLM, we add a new token <FACE> in the vocabulary of
the LLM and fine-tune the language modelling head. For fine-tuning the VLM
without relying on manually annotated data, we introduce a 2D self-supervised
loss $L_{face}$, following established protocols of specialized face
reconstruction models [23]:

$L_{face}=\lambda_{pixel}L_{pixel}+\lambda_{per}L_{perc}+\lambda_{LM}L_{LM}+\lambda_{reg}L_{reg},$
(5)

where $L_{pixel}=||A\odot(x_{img}-\hat{y}_{rec})||^{2}_{2}$ refers to the
pixel-wise reconstruction loss between reconstructed images $\hat{y}_{rec}$
and the input images $x_{img}$. To avoid distortions when training from in-
the-wild data, a 2D skin mask $A$ is estimated from the input images using a
simple pre-trained Gaussian mixture model for the skin color [12].

The perceptual loss $L_{perc}=sim(f_{perc}(x_{img}),f_{perc}(\hat{y}_{rec}))$
estimates the cosine similarity between the reconstructed image
$\hat{y}_{rec}$ and the input image $x_{img}$ at the perceptual level using a
pre-trained feature extractor $f_{perc}$.

The landmark loss $L_{LM}=||LM_{img}-LM_{rec}||^{2}_{2}$ measures the L2
distance between the projected 2D landmarks of the estimated 3DMM $LM_{rec}$
and predicted 2D facial landmarks $LM_{img}$ using an off-the-shelf facial
detector [5].

$L_{reg}=||\theta||^{2}_{2}$ regularizes the estimated 3DMM parameters towards
the mean value of the multi-variate Gaussian distribution of the 3DMM.

The weight parameters
$\lambda_{face}=[\lambda_{pixel},\lambda_{per},\lambda_{LM},\lambda_{reg}]$
balance the respective losses, and we set them as established in prior work
[23].

Preserving the ability for natural conversations about faces. We noticed that
VLM’s loose the ability to conduct general conversations about faces when
trained self-supervised face reconstruction loss, and tend to always output
3DMM parameters when queried with a face image. To resolve this problem, we
generate a face conversation dataset with accurate textual face descriptions,
by querying the VLM with face images and conversations that include multiple
questions about facial attributes. During training, we mix task-specific
instructions that explicitly ask for 3DMM parameters with general
conversational data to regularize the training and preserve the ability for
general non-3DMM related conversations about faces. In contrast to prior works
[22, 15] which generally employ a simple static question-answer template that
restrains the model from providing diverse responses, our proposed strategy
can effectively improve the instruction following performance for general face
conversations.

Self-supervised Text-to-3DMM loss. We aim to enable the VLM to perform
semantic face understanding tasks, i.e. to predict faithful 3DMM parameters
when only having textual input $f(\cdot,x_{txt})$. While related works for 3D
human pose estimation [10, 15] rely on human-written fine-grained language
descriptions of 3D human meshes, we aim to achieve text-to-3DMM generation
without any human interaction. To achieve this, we introduce a self-supervised
text-based face reconstruction loss. In particular, we first query the VLM
with the face images in the training data and a question template that
requests detailed face descriptions of a number of facial attributes such as
head shape, skin tone, or facial expressions. This results in a dataset with
pairs of face images and detailed textual descriptions about facial
attributes. During training, we instruct the VLM to estimate 3DMM parameters
only from the available detailed face descriptions $f(\cdot,x_{txt})$. As we
have the corresponding 2D image for each face description, we can re-use the
self-supervised face reconstruction loss Eq. 5 to train the model at the task
of text-to-3DMM generation, without the need for any human-written annotation.

##  4 Experiments

###  4.1 Training Data

Text-to-Face Data. The text-to-face dataset comprises pairs of text
descriptions and face images, facilitating the development of mappings by VLM
between textual descriptions and the 3DMM parameters of faces. During
training, only text is used as input and the corresponding face images are
only used to produce supervision signals. Given the absence of publicly
available datasets linking text descriptions to 3D face meshes and existing
VLMs like LLaVA present powerful Visual Question Answering(VQA) ability, we
generate textual descriptions for face images. We employ templates structured
as follows to guide the learning process in VLMs: "USER: {description}, can
you give the 3DMM parameters of this person. ASSISTANT: Sure, it is <FACE>.",
where {description} is the text description for faces. We utilize high-quality
face images from CelebA-HQ [21] and generate text descriptions using
LLaVA-v1.5-13B [25] to construct the dataset.

Image-to-Face Data. Image-to-face reconstruction data is composed of only
human face images. The face images will be formatted with task-specific
templates like "USER: <IMAGE> Can you give the 3DMM parameters of this person.
ASSISTANT: Sure, it is <FACE>.". To enhance the diversity and relevance of the
conversations centered around human face images, we also generate face-centric
conversations for each face image. This approach enriches the contextual
understanding of the VLM concerning the newly introduced token <FACE>. We
adopt the CelebA-HQ trainset as the image to face reconstruction dataset.

Multimodal Instruction-Following Data. This data is general-purpose VQA data,
and it is used to preserve the VLM’s ability of understanding a user’s
instructions. Following LLaVA v1.5, We use LLaVA-v1.5-mix665k as multimodal
instruction following data.

###  4.2 Experimental settings

Network Architecture. We build our model on Large Vision Language Model
LLaVA-1.5-7B [25] with CLIP-ViT-L-336px as vision encoder and Vicuna v1.5 as
the LLM backbone. LoRA [20] is applied to efficiently fine-tune the VLM. An
MLP head with GeLU activations [19] is appended to the last layer of the VLM
to predict the 3D human face parameters.

3D Face Model. We use the Basel Face Model (BFM) 2017 [17] as the 3D face
model. The face is parameterized as the semantic code vector
$\theta=[\alpha,\delta,\gamma,\phi,c]\in\mathbb{R}^{257}$ in [41], which
includes 3D shape parameters $\alpha\in\mathbb{R}^{80}$, facial expression
parameters $\delta\in\mathbb{R}^{64}$, texture of 3DMM
$\gamma\in\mathbb{R}^{80}$, illumination $\phi\in\mathbb{R}^{27}$ and camera
parameters $c\in\mathbb{R}^{6}$.

LLaVa-Key baseline. As there are no VLM-based methods that can perform 3D face
reconstruction, we introduce the LLaVa-Key. The model is finetuned to predict
facial landmarks in the format of pure text given either images or textual
description of human faces as input. For each input, gradient-based
optimization is applied to these predicted landmarks to fit the 3DMM using Eq.
5 to obtain a 3D face reconstruction. It is important to note that LLaVa-Key
employs test-time fine-tuning, making it an inherently different baseline
compared to our approach and other task-specific methods, which rely solely on
feed-forward prediction. Nevertheless, we still use LLaVA-Key as a baseline to
illustrate the effect of a direct 3DMM embedding in the token space of
FaceGPT, which generally yields superior results, even without any test-time
fine-tuning.

Implementation Details. The training uses 8 NVIDIA 48G A40 GPUs. We utilize
deepspeed [34] engine and ZeRO optimizer [33] for efficient training. We use
AdamW [28] optimizer with the learning rate and weight decay set to $2e-5$ and
0, respectively. We also follow the standard setting of using a WarmupDecayLR
as the learning rate scheduler, where the warm-up iterations are set to 100.
The weights of the text generation loss $\lambda_{txt}$ and the face
reconstruction loss $\lambda_{face}$ are set to 1.0 and 0.1, respectively. And
following [23], those of the pixel loss $\lambda_{pixel}$, the perceptual loss
$\lambda_{per}$, the landmark loss $\lambda_{LM}$, and the regularization loss
$\lambda_{reg}$ are set to 0.5, 0.25, 5e-4 and 0.1, respectively. The
landmarks of human faces are obtained with [5] and we use pre-trained ArcFace
[11] to extract perceptual features. The batch size is set to 8 per GPU and
the gradient accumulation step is set to 4.

Evaluation Metrics. The instruction following ability is measured with GPT-
assisted evaluation, which queries GPT4 to obtain the grading of generated
responses. The 3D human face parameters estimation is measured with the L2
photometric error in RGB space and the L2 landmark error.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
Input Image | DECA | Deep3D | FOCUS | LLaVA-Key | FaceGPT

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

Figure 3: Qualitative results for 3D face reconstruction. Our approach allows
for the regression of pose, shape, expression, skin reflectance, and
illumination from a single monocular image, achieving quality comparable to
recent state-of-the-art methods.

###  4.3 Image-based 3D Face Reconstruction

Table 1: Performance at monocular 3D face reconstruction. Method | photo | keypoint
---|---|---
DECA [13] | 0.216 | 5.2px
Deep3D [12] | 0.073 | 3.2px
FOCUS [23] | 0.077 | 2.2px
LLaVa-Key | 0.110 | 14.0px
FaceGPT | 0.106 | 3.0px

We evaluate FaceGPT at image-based 3D face reconstruction and show a
comparison with SOTA specialized monocular face reconstruction methods and a
VLM-based generalist baseline LLaVa-Key in Table 1.

Moreover, Fig. 3 shows a qualitative comparison of FaceGPT and all baselines
at the classic task of 3D face reconstruction from a single image. As reported
in prior works for VLM-based segmentation [22] and human pose estimation [15],
our FaceGPT model does, as expected, not reach the state-of-the-art
performance of specialized models. However, we note that FaceGPT has a frozen
vision encoder to preserve the generalist behavior, whereas all baseline
models have a fine-tuned task-specific backbone. Moreover, our model does
outperform DECA, which is a highly competitive and widely applied baseline
model [47]. When compared to the VLM baseline model, FaceGPT achieves large
improvements, highlighting the benefit of embedding the 3DMM parameters
directly in the token space of the VLM.

###  4.4 Text-based 3D Face Reconstruction

Table 2 and Fig. 4 show the results of FaceGPT at text-based 3D face
reconstruction. As there are no public unsupervised methods available that
perform this task, we compare to a VLM-based baseline LLaVA-Key, where LLaVA
is fine-tuned to predict the 2D coordinates of face landmarks given text
description as input and an optimization-based method is utilized to fit a
3DMM on predicted landmarks.

Table 2: Performance evaluation of text-based 3D face reconstruction (as L2 3DMM error). Method | Exp. | Shape
---|---|---
LLaVA-Key | 9.39 | 12.10
FaceGPT | 2.89 | 4.04

We can clearly observe the significant advantage that FaceGPT achieves over
the baseline method. Moreover, the qualitative results demonstrate that the
model can achieve faithful text-based 3D face reconstructions, despite being
trained in a self-supervised manner. These results demonstrate that embedding
3D knowledge about our world into a VLM is in principle possible without
detailed human annotations, hence demonstrating the large potential of
combining VLM’s with vision-as-inverse graphics for self-supervised learning.

Figure 4: Qualitative results of text-to -3DMM synthesis comparing the
performance of the baseline VLM LLaVA-key and FaceGPT. Our method can reason
and generate faithful 3D faces using only textual description. We also present
the realistic face image which matches the description as a reference. A 3DMM
reconstruction of reference images estimated by FOCUS method [23] is also
provided for reference. Our approach captures many facial details from the
texts and produces much better visual results compared to the LLaVA-Key
baseline method.

###  4.5 GPT-Assisted Evaluation

Table 5 shows that FaceGPT preserves the ability of instruction following by
following LLaVA’s evaluation [26] protocol, using GPT4-Assisted evaluation on
LLaVA-Bench (COCO). FaceGPT compares favorably to LLaVA-v1.5-7B and reaches
similar or better performance in the benchmark compared to VLMs with much more
parameters. This performance advantage can be attributed to the specialized
face training dataset and the usage of our face-centric conversation. These
elements enhance FaceGPT’s proficiency in interpreting face-related language-
based instructions, thereby improving its overall effectiveness in relevant
tasks.

###  4.6 Ablation study

Influence of LLM on face understanding. Classical methods mainly rely on a
vision encoder for regressing the 3DMM face parameters. In contrast, LLaVA
utilizes a combination of frozen vision encoder and an LLM to perceive
information from images. To study the effect of the LLM head, we train a
baseline model with the vision encoder of LLaVA and an MLP to predict the
human face parameters directly. The results are presented in Table 5 and show
that the additional LLM helps in predicting better face parameters based on
the visual representations from the frozen encoder.

Influence of face-centric conversation generation. To prove the necessity of
our face-centric conversation generation strategy, we train a model only using
a simple single-turn conversation template for face images, which is a common
strategy used in the VLM-based image understanding works like LISA and
ChatPose. The comparison results are demonstrated in Table 5. We observe that
face-centric conversations help a lot in improving model’s ability in
performing detailed description and complex reasoning. The face-centric
conversations only have a small effect on the face reconstruction task, which
is expected, as the generated conversations do not contain information about
the 3DMM parameters of faces.

Comparison between supervised face loss and unsupervised face loss. Recent
VLM-based works on segmentation and pose estimation generally rely on
supervised learning with ground truth data or pseudo ground truth data. To
study the influence of supervision on the model’s capability, we compare the
model trained with supervised 3DMM losses and self-supervised 2D face losses.
For the supervised 3DMM loss, we extract the 3DMM parameters on the same
CelebA-HQ trainset with the state-of-the-art face reconstruction method FOCUS
[23]. L1 loss is used for optimization on 3DMM ground truth. We observe in
Table 6, that the model utilizing a supervised 3DMM loss outperforms its
unsupervised counterpart, indicating that the performance ceiling is not
reached yet and improvements on the self-supervised training could potentially
lead to further performance gains.

Table 3: GPT4-Assisted Evaluation on instruction-following capability. “Conv",
“Details", and “Complex" correspond to three types of questions(conversation,
detailed description, complex reasoning) produced by LLaVA’s data generation
pipeline. GPT4 will be prompted to evaluate the answers from different models
along with the ground truth answer produced by text-only GPT4. It would then
give a score for each answer with an explanation. Previous method like
ChatPose [15] uses gpt-4-0314 for evaluation which is not accessible anymore.
Thus, we provide the evaluation results with a newer gpt-4-0613.

| Evaluation | Conv | Detail | Complex | All
---|---|---|---|---|---
LLaVA-V1.5-13B | gpt-4-0314 | 84.4 | 81.0 | 93.9 | 86.5
ChatPose [15] | 78.8 | 76.2 | 96.7 | 84.0
LLaVA-V1.5-13B [25] | gpt-4-0613 | 80.4 | 81.4 | 90.9 | 84.2
LLaVA-V1.5-7B [26] | 79.9 | 77.6 | 92.4 | 83.4
FaceGPT | 79.6 | 81.5 | 92.6 | 84.6

Table 4: Influence of LLM towards human face understanding

Method | photo | keypoint
---|---|---
CLIP ViT + MLP | 0.133 | 7.6px
FaceGPT | 0.106 | 3.0px

Table 5: Influence of face-centric conversation generation

Method | face convs | photo | keypoint | Conv | Detail | Complex | All
---|---|---|---|---|---|---|---
LLaVA-1.5-7B | ✗ | - | - | 79.9 | 77.6 | 92.4 | 83.4
FaceGPT | ✗ | 0.110 | 3.2px | 78.4 | 80.8 | 89.0 | 82.7
FaceGPT | ✓ | 0.106 | 3.0px | 79.6 | 81.5 | 92.6 | 84.6

Table 6: Comparison when learning with supervised and unsupervised losses. Method | photo. | keyp.
---|---|---
FaceGPT(sup) | 0.092 | 2.1px
FaceGPT(unsup) | 0.106 | 3.0px

##  5 Limitations and Future Work

While FaceGPT demonstrates remarkable results at reasoning about 3D faces from
visual and text input with unsupervised learning, many open challenges remain.
Our work represents merely an initial step into this promising area,
highlighting the vast potential for future exploration and advancement in this
direction. The model does not yet match the state-of-the-art performance of
task-specific 3D face reconstruction methods, and one way to address this
problem is to largely scale up the training to include various face databases.
The extension of FaceGPT to include arbitrary numbers of faces in an image is
another interesting potential research direction. Finally, the model is
specific to faces and relies on the availability of a 3D morphable model for
faces. A generalization to general objects would require the self-supervised
learning to also include the generative object model parameters.

##  6 Conclusion

FaceGPT is the first self-supervised learning framework for Large Vision-
Language Models to reason about 3D human faces. We show that VLMs can learn to
predict detailed 3D human faces from not only images, but also from textual
inputs, in a fully self-supervised manner via inverse rendering. As a
generalist model, FaceGPT achieves strong results across various tasks,
including traditional 3D face reconstruction, visual instruction following and
text-based face reconstruction. We believe our work also has general
implications beyond face analytics, as it points towards a way forward to
enable large multi-modal language-models to reason about our 3D world without
supervision.

## References

  * [1] Shivangi Aneja, Justus Thies, Angela Dai, and Matthias Nießner.  Clipface: Text-guided editing of textured 3d morphable models.  In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023.
  * [2] Anil Bas, Patrik Huber, William AP Smith, Muhammad Awais, and Josef Kittler.  3d morphable models as spatial transformer networks.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 904–912, 2017.
  * [3] V Blanz and T Vetter.  A morphable model for the synthesis of 3d faces.  In 26th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1999), pages 187–194. ACM Press, 1999.
  * [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [5] Adrian Bulat and Georgios Tzimiropoulos.  How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks).  In International Conference on Computer Vision, 2017.
  * [6] Zehranaz Canfes, M. Furkan Atasoy, Alara Dirik, and Pinar Yanardag.  Text and image guided 3d avatar generation and manipulation, 2022.
  * [7] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123–16133, 2022.
  * [8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023.
  * [9] Radek Daněček, Michael J Black, and Timo Bolkart.  Emoca: Emotion driven monocular face capture and animation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20311–20322, 2022.
  * [10] Delmas, Ginger and Weinzaepfel, Philippe and Lucas, Thomas and Moreno-Noguer, Francesc and Rogez, Grégory.  PoseScript: 3D Human Poses from Natural Language.  In ECCV, 2022.
  * [11] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2019.
  * [12] Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong.  Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set.  In IEEE Computer Vision and Pattern Recognition Workshops, 2019.
  * [13] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart.  Learning an animatable detailed 3D face model from in-the-wild images.  ACM Transactions on Graphics (ToG), Proc. SIGGRAPH, 40(4):88:1–88:13, August 2021.
  * [14] Yao Feng, Haiwen Feng, Michael J Black, and Timo Bolkart.  Learning an animatable detailed 3d face model from in-the-wild images.  ACM Transactions on Graphics (ToG), 40(4):1–13, 2021.
  * [15] Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J. Black.  Chatpose: Chatting about 3d human pose.  In CVPR, 2024.
  * [16] Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T Freeman.  Unsupervised training for 3d morphable model regression.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8377–8386, 2018.
  * [17] Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Luthi, Sandro Schönborn, and Thomas Vetter.  Morphable face models-an open framework.  In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 75–82. IEEE, 2018.
  * [18] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra.  Imagebind: One embedding space to bind them all.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023.
  * [19] Dan Hendrycks and Kevin Gimpel.  Bridging nonlinearities and stochastic regularizers with gaussian error linear units.  CoRR, abs/1606.08415, 2016.
  * [20] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2022.
  * [21] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  2018\.
  * [22] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia.  Lisa: Reasoning segmentation via large language model.  In CVPR, 2024.
  * [23] Chunlu Li, Andreas Morel-Forster, Thomas Vetter, Bernhard Egger, and Adam Kortylewski.  Robust model-based face reconstruction through weakly-supervised outlier segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 372–381, 2023.
  * [24] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4d scans.  ACM Trans. Graph., 36(6):194–1, 2017.
  * [25] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning, 2023.
  * [26] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  In NeurIPS, 2023.
  * [27] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  Advances in neural information processing systems, 36, 2024.
  * [28] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2019.
  * [29] OpenAI.  ChatGPT: A conversational language model.  Online, 2024.
  * [30] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.  Gpt-4 technical report, 2024.
  * [31] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter.  A 3d face model for pose and illumination invariant face recognition.  In 2009 sixth IEEE international conference on advanced video and signal based surveillance, pages 296–301. Ieee, 2009.
  * [32] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [33] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  Zero: memory optimizations toward training trillion parameter models.  In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’20. IEEE Press, 2020.
  * [34] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 3505–3506, New York, NY, USA, 2020. Association for Computing Machinery.
  * [35] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv preprint arXiv:2007.08501, 2020.
  * [36] Soubhik Sanyal, Timo Bolkart, Haiwen Feng, and Michael J Black.  Learning to regress 3d face shape and expression from an image without 3d supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7763–7772, 2019.
  * [37] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai.  Pandagpt: One model to instruction-follow them all.  arXiv preprint arXiv:2305.16355, 2023.
  * [38] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Alpaca: A strong, replicable instruction-following model.  Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html, 3(6):7, 2023.
  * [39] Ayush Tewari, Michael Zollhoefer, Florian Bernard, Pablo Garrido, Hyeongwoo Kim, Patrick Perez, and Christian Theobalt.  High-fidelity monocular face reconstruction based on an unsupervised model-based face autoencoder.  IEEE transactions on pattern analysis and machine intelligence, 42(2):357–370, 2018.
  * [40] Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, and Christian Theobalt.  Self-supervised multi-level face model learning for monocular reconstruction at over 250 hz.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2549–2559, 2018.
  * [41] Ayush Tewari, Michael Zollöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Perez, and Christian Theobalt.  MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction.  In The IEEE International Conference on Computer Vision (ICCV), 2017.
  * [42] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [43] Menghua Wu, Hao Zhu, Linjia Huang, Yiyu Zhuang, Yuanxun Lu, and Xun Cao.  High-fidelity 3d face generation from natural language descriptions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4521–4530, 2023.
  * [44] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  arXiv preprint arXiv:2309.05519, 2023.
  * [45] Cuican Yu, Guansong Lu, Yihan Zeng, Jian Sun, Xiaodan Liang, Huibin Li, Zongben Xu, Songcen Xu, Wei Zhang, and Hang Xu.  Towards high-fidelity text-guided 3d face generation and manipulation using only images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15326–15337, 2023.
  * [46] Longwen Zhang, Qiwei Qiu, Hongyang Lin, Qixuan Zhang, Cheng Shi, Wei Yang, Ye Shi, Sibei Yang, Lan Xu, and Jingyi Yu.  Dreamface: Progressive generation of animatable 3d faces under text guidance.  arXiv preprint arXiv:2304.03117, 2023.
  * [47] Ding Zheng, Zhang Cecilia, Xia Zhihao, Jebe Lars, Tu Zhuowen, and Zhang Xiuming.  Diffusionrig: Learning personalized priors for facial appearance editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [48] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.