# Language-guided Semantic Style Transfer of 3D Indoor Scenes

Bu Jin  University of Chinese Academy of SciencesBeijingChina
<EMAIL_ADDRESS>,  Beiwen Tian  Tsinghua UniversityBeijingChina
<EMAIL_ADDRESS>,  Hao Zhao  Peking University, Intel
LabsBeijingChina zhao-hao<EMAIL_ADDRESS>and  Guyue Zhou
Tsinghua UniversityBeijingChina<EMAIL_ADDRESS>
(2022)

###### Abstract.

We address the new problem of language-guided semantic style transfer of 3D
indoor scenes. The input is a 3D indoor scene mesh and several phrases that
describe the target scene. Firstly, 3D vertex coordinates are mapped to RGB
residues by a multi-layer perceptron. Secondly, colored 3D meshes are
differentiablly rendered into 2D images, via a viewpoint sampling strategy
tailored for indoor scenes. Thirdly, rendered 2D images are compared to
phrases, via pre-trained vision-language models. Lastly, errors are back-
propagated to the multi-layer perceptron to update vertex colors corresponding
to certain semantic categories. We did large-scale qualitative analyses and
A/B user tests, with the public ScanNet and SceneNN datasets. We demonstrate:
(1) visually pleasing results that are potentially useful for multimedia
applications. (2) rendering 3D indoor scenes from viewpoints consistent with
human priors is important. (3) incorporating semantics significantly improve
style transfer quality. (4) an HSV regularization term leads to results that
are more consistent with inputs and generally rated better. Codes and user
study toolbox are available at https://github.com/AIR-DISCOVER/LASST.

3D Style Transfer, Differentiable Rendering, Vision and Language

††journalyear: 2022††copyright: acmlicensed††conference: Proceedings of the
1st Workshop on Photorealistic Image and Environment Synthesis for Multimedia
Experiments; October 14, 2022; Lisboa, Portugal.††booktitle: Proceedings of
the 1st Workshop on Photorealistic Image and Environment Synthesis for
Multimedia Experiments (PIES-ME ’22), October 14, 2022, Lisboa,
Portugal††price: 15.00††isbn: 978-1-4503-9500-7/22/10††doi:
10.1145/3552482.3556555††ccs: Computing methodologies Computer vision
representations††ccs: Computing methodologies Artificial intelligence††ccs:
Computing methodologies Computer vision††ccs: Computing methodologies Computer
vision representations††ccs: Computing methodologies Appearance and texture
representations

Figure 1. Our algorithm has two inputs: a colored 3D mesh which is the digital
twin of a user’s room and a set of user-specified text prompts as guidance.
The output is a style transferred 3D mesh following the language guidance.

Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing
to bat.

##  1\. Introduction

3D content creation and editing is a long-existing multimedia demand. With the
surge of metaverse, tech giants and consumers are now looking forward to a
high-quality virtual world that people can live in and interactive with. We
study the problem of 3D indoor scene style transfer, which would promote the
user experience of metaverse residents. Shown in Fig. 1, the system input is a
colored 3D mesh as reconstructed by methods like VoxelHashing (Nießner et al.,
2013) or FlashFusion (Han and Fang, 2018) and the output is a stylized version
corresponding to certain user inputs. We achieve this goal with a new 3D style
transfer algorithm with two notable features: language guidance and semantic
awareness. Firstly, user inputs are specified by natural language, e.g.,
_marble wall_ and _steel counter_ in Fig. 1. This feature allows open-set
arbitrary text inputs that are not restricted to pre-defined style sets.
Secondly, our method stylizes different semantic regions separately, allowing
a delicate control instead of treating the scene as a whole.

Language-driven synthesis of 2D images (Patashnik et al., 2021) and 3D style
transfer via differentiable rendering (Kato et al., 2018) have both been
demonstrated before. Recently, Text2Mesh (Michel et al., 2021) combines these
two methods to build a system that modifies 3D object meshes according to
natural language guidance. However, trivially applying the Text2Mesh method to
3D scenes cannot produce satisfactory results due to three reasons. Firstly,
the Text2Mesh method randomly samples rendering viewpoints for objects, which
is problematic for indoor scenes. The fundamental difference is that objects
(e.g., a shoe) can be observed arbitrarily from any viewpoint but an indoor
scene should be observed according to human viewing preference priors.
Rendering indoor scenes from unnatural viewpoints leads to sub-optimal
gradients from the vision-language model (VLM) because the VLM is pre-trained
on natural images. Secondly, an indoor scene is composed of many semantic
regions and treating the scene as a whole cannot fully leverage the semantic
alignment between vision and language. For example, if the _counter_ region in
Fig. 1 is not known, the _steel_ attribute would be transferred to other
semantic regions. Thirdly, while transferring the style of an object into an
exaggerated one is meaningful in many cases, indoor scene style transfer
generally calls for consistent visual experience. A natural idea is to enforce
a RGB regularization term between stylized meshes and input meshes. But
empirically this loss term cannot effectively guarantee visual consistency.

In order to address aforementioned issues, we propose the LASST algorithm,
short for language-guided semantic style transfer of 3D indoor scenes.
Firstly, we identify the importance of natural rendering viewpoints and design
6DoF pose priors consistent with human preference, which provide meaningful
gradients from the VLM. Secondly, as inspired by 2D semantic style transfer
(Lu et al., 2017)(Lu et al., 2018), we introduce both ground truth and
estimated semantic masks to facilitate aligned feature matching within
semantic regions. In the forward pass, the whole scene is rendered, while in
the backward pass, we apply the _stop gradient_ trick to unrelated semantic
regions. Thirdly, we leverage an HSV regularization loss to prevent the scene
color from drifting too far from inputs, instead of using a regularization in
the RGB space. This alternative leads to stylized meshes that are rated better
in A/B user studies.

In order to illustrate the significance of these three points, we benchmark on
the ScanNet dataset (Dai et al., 2017) and SceneNN dataset (Hua et al., 2016)
with extensive qualitative analyses and A/B user tests. Clear conclusions are
reached, showing that users prefer LASST results generated with natural
viewpoints, semantic alignment and the HSV regularization.

To summarize, our contributions are as follows:

  * •

We propose the LASST algorithm. To the best of our knowledge, LASST is the
first solution for language-guided semantic style transfer of 3D indoor
scenes. It is potentiallly of interest to both multimedia researchers and
practitioners.

  * •

We demonstrate the importance of using natural rendering viewpoints for indoor
scene style transfer.

  * •

We design a simple but effective strategy based upon the stop gradient trick,
to encourage semantic alignment between language guidance and visual data.

  * •

We propose an HSV regularization loss that leads to results more consistent
with inputs.

  * •

We evaluate our design choices with a large-scale A/B user test and provide an
open-source implementation of LASST.

Figure 2. Overview of the LASST algorithm. The input is a colored 3D mesh
$\mathbf{M}$ and a text prompt $\mathbf{P}$. The output is a stylized mesh
$\mathbf{M^{S}}$. Input vertex coordinates are sent into a vertex color
deviation network to generate color residues, which are added onto the input
vertex color values. Semantic masks are acquired through either accessing the
ground truth or running a scene parsing model. The stylized mesh is
differentiablly rendered into multiple views and compared with the text prompt
of a certain category $\mathbf{P_{k}}$ through a pre-trained CLIP. The VLM
semantic loss and the HSV regularization loss are used to back-propagate
gradients to only modify regions corresponding to certain semantics.

Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing
to bat.

##  2\. Related Works

Texturing 3D Meshes. Successfully texturing 3D meshes requires numerical
surface parameterization and state-of-the-art methods pursue conformal
mappings (Gillespie et al., 2021), global consistency (Sharp and Crane, 2018)
and boundary awareness (Li et al., 2018). However, they do not apply to non-
watertight scene meshes as reconstructed using RGB-D streams (e.g. the kitchen
in Fig. 1). Bi _et al._ (Bi et al., 2017) develop a patch search and vote
scheme for the texturing of reconstructed objects. Dessein _et al._ (Dessein
et al., 2014) introduce a Poisson regularization into the texture mapping
problem on object surfaces. Huang _et al._ (Huang et al., 2017) trade off
geometric details for texture quality using planar primitives. Waechter _et
al._ (Waechter et al., 2014) design a comprehensive pipeline for large-scale
3D reconstruction texturing, using photometric fidelity and smoothness
regularizations. While most methods process readily reconstructed meshes, Zhou
_et al._ (Zhou and Koltun, 2014) jointly optimize frame poses and texture.
Different from these methods, we directly optimize mesh texture using neural
network back-propagation, guided by user’s natural language input and being
aware of semantics.

3D Style Transfer. Kato _et al._ (Kato et al., 2018) propose a soft surrogate
for rasterization so that the rendering of meshes is turned into a
differentiable operator. They also demonstrate the application of 3D style
transfer with 2D input guidance. Other differentiable renering formulations
like OpenDR (Loper and Black, 2014), Soft Rasterizer (Liu et al., 2019) and
DIB-Renderer (Chen et al., 2019) also support 3D style transfer in a similar
scheme. 3DStyleNet (Yin et al., 2021) formulates a feed-forward network
trained with content and style losses imposed on differentiable renderings.
Patch-based methods are also used to stylize 3D models (Hauptfleisch et al.,
2020)(Sỳkora et al., 2019). 3D style transfer on point clouds (Cao et al.,
2020) has also been demonstrated. StyleMesh (Höllein et al., 2021) formulates
a 3D mesh style transfer algorithm using depth and normal statistics. To the
best our knowledge, LASST is the first algorithm that achieve visually
pleasing 3D semantic style transfer using language guidance.

Language-guided Synthesis. CLIP (Radford et al., 2021) is a successful vision-
language model that models the affinity of RGB images and concepts expressed
in texts. Except for its surprising zero-shot classification performance on
ImageNet (Russakovsky et al., 2015), CLIP can be used as a _loss function_ for
visual synthesis. Optimizing the image encoder input to minize its distance
with text prompts leads to compelling image generation results (Ramesh et al.,
2021). When combined with diffusion probabilistic models, the visual quality
is strengthened (Nichol et al., 2021). StyleCLIP (Patashnik et al., 2021)
exploits CLIP text embedding to search for optimization directions in pre-
trained StyleGAN models (Karras et al., 2019). Text2Mesh (Michel et al., 2021)
achieves text-driven editing of 3D objects, but has not demonstrate semantic
editing results for large indoor scenes. Interestingly, CLIPort (Shridhar et
al., 2022) recently shows that pre-trained VLM can be used to generate
manipulation control sequences according to human command.

##  3\. Method

###  3.1. Overview

We give an overview of the LASST algorithm, as illustrated in Fig. 2. An
indoor scene $\mathbf{M}$ is represented in the 3D mesh format with vertices
denoted by $\mathbf{V}\in{\mathbb{R}^{n\times 3}}$ and faces denoted by
$\mathbf{F}\in{K^{3\times m}},K\in[1,2,...,n]$. Each vertex $v_{i}$ has an
initial color and an associated semantic label, which could be ground truth or
prediction. The initial colors of a mesh are denoted as $C^{ini}$. The input
text prompt is denoted by $\mathbf{P_{k}}$, where k is the index of sub-
prompts and corresponds to a specific semantic category we aim to stylize.

We perform style transfer for indoor scenes by optimizing a style transfer
network through back-propagation. We refer to our style transfer network as
VCDN (Vertex Color Deviation Network) as it takes vertex coordinates as input
and leverages a multilayer perceptron to predict the color deviation of
vertices. Adding the predicted deviation to the original colors of the input
scene $\mathbf{M}$ results in a stylized scene $\mathbf{M}^{S}$.

Then, a differentiable renderer (Fuji Tsang et al., 2022) is exploited to
obtain the rendering results of the stylized scene from several appropriate
viewpoints. To compare the semantics in different modalities, we utilize a
pre-trained CLIP model (Radford et al., 2021) to extract the features of the
rendered images and the input text prompt. The negation of cosine similarity
between the features are calculated as the loss to optimize (denoted by
$\mathscr{L}_{sem}$). We also calculate the statistical difference between the
stylized and initial scenes in the HSV color space as an additional
regularization loss $\mathscr{L}_{hsv}$ to control the deviation from the
input scenes. Finally we back-propagate the sum of the losses to optimize VCDN
with the CLIP model frozen, while masking out gradients that do not belong to
specific semantic regions.

###  3.2. Vertex Color Deviation Network

As the name suggests, our VCDN predicts the residues $C^{res}$ of vertex
colors that deviate from $C^{ini}$. For each scene, we normalize the input
vertex coordinates into a unit ball first. Then for each vertex $v_{i}\in V$,
its position encoding $PE(v_{i})$ can be computed by:

(1)
$PE(v_{i})=[\cos{(2{\pi}\textbf{B}v_{i})},\sin{(2{\pi}\textbf{B}v_{i})}]^{T}$

where B is a random matrix whose values are sampled from Gaussian
distributions. The position encoding $PE(v_{i})$ is sent to a multi-layer
perceptron ${\phi}$ to get the vertex color residues $C^{res}$:

(2) $C^{res}=\phi(PE(v_{i}))$

Finally the stylized mesh $M^{S}$ is generated by adding the predicted vertex
color residues $C^{res}$ to initial vertex colors $C^{ini}$. The vertex color
of $\mathbf{M^{S}}$ is $C^{res}+C^{ini}$. Then the $\mathbf{M^{S}}$ is passed
to the renderer as input to calculate the semantic loss.

Figure 3. Our online user study interface. Users interact with 3D models and
select a result that betters fits the text prompt. Figure 4. Our sampling
strategy generates better results than Text2Mesh’s random sampling strategy.
The lower two rows are the intermediate rendering results of the scene in the
rightmost column, for the _steel refrigerator_ prompt.

###  3.3. Semantic Loss via Pre-trained VLM

We define a differentiable renderer $\mathscr{R}$ that renders the 3D stylized
mesh $M^{S}$ into 2D images. To focus on the objects of interest, we render
the mesh from views $\theta_{j},j=1,...,n_{v}$ (see Sec. 3.6):

(3) $I_{j}=\mathscr{R}(M^{S},\theta_{j},f_{j}),j=1,...,n_{v}$

where $f_{j}$ is the focal length of the camera, resulting in $n_{v}$ rendered
images $I_{j}$. To map the rendered images and text prompt $P_{k}$ to the same
high-dimensional space, we exploit a large pre-trained vision-language model
CLIP as our embedding tool. Along with text prompt $P_{k}$, the rendered
images $I_{j}$ are passed as input to the CLIP model:

(4) $\displaystyle S^{I}=\frac{1}{n_{v}}\sum_{j}\
E_{img}(I_{j})\in{\mathbb{R}^{512}}$ (5) $\displaystyle
S^{P_{k}}=E_{text}(P_{k})\in{\mathbb{R}^{512}}$

where $S^{I}$ and $S^{P_{k}}$ are in the same multi-modal embedding space the
CLIP creates. Then we define the image-text semantic loss $\mathscr{L}_{sem}$
as the opposite number of the cosine similarity between $S^{I}$ and
$S^{P_{k}}$:

(6) $\mathscr{L}_{sem}=-sim_{cos}(S^{I},S^{P_{k}})$

###  3.4. Local Awareness

With the semantic mask of the input labels $l$, the stylized room is separated
into two parts: mesh with target label $M^{l}$ and mesh with other label
$M^{\overline{l}}$. In the forward process, the whole room mesh is passed to
the renderer $\mathscr{R}$ to get the images of the room (instead of only some
objects in it), see Fig. 4. While in the backward process, the gradient of the
$M^{\overline{l}}$ is masked out. The whole process can be formulated as:

(7)
$I_{j}={\mathscr{R}}(M^{l},\theta_{j},f_{j})+{\mathscr{R}}^{SG}(M^{\overline{l}},\theta_{j},f_{j})\\\
$

where $\mathscr{R}$ and ${\mathscr{R}}^{SG}$ are differentiable renderers. The
superscript $SG$ denotes Stop Gradient trick.

###  3.5. HSV Regularization

The HSV regularization is exploited to prevent the scene color from drifting
too far from inputs. For each vertex $v_{i}$ with RGB color $(R,G,B)$, its
color in HSV color space is given by:

(13) $\displaystyle H$
$\displaystyle=\left\\{\begin{array}[]{rcl}0^{\circ},&&{max=min}\\\
60^{\circ}+\frac{G-B}{max-min}+0^{\circ},&&{max=R,B\leq G}\\\
60^{\circ}+\frac{G-B}{max-min}+360^{\circ},&&{max=R,B>G}\\\
60^{\circ}+\frac{B-R}{max-min}+120^{\circ},&&{max=G}\\\
60^{\circ}+\frac{R-G}{max-min}+240^{\circ},&&{max=B}\end{array}\right.$ (16)
$\displaystyle S$ $\displaystyle=\left\\{\begin{array}[]{rcl}0,&&{max=0}\\\
1-\frac{min}{max},&&{otherwise}\end{array}\right.$ (17) $\displaystyle V$
$\displaystyle=max$

where max is the maximum of R,G,B and min is the minimum. Note that
$H=0^{\circ}$ and $H=360^{\circ}$ represent the same hue. Given an original
mesh and its stylized mesh, we acquire HSV color $(h_{1},s_{1},v_{1})$ and
$(h_{2},s_{2},v_{2})$. Then the HSV loss is calculated by:

(18) $\begin{split}\mathscr{L}_{HSV}=&w_{1}\mu[(\cos h_{1}-\cos h_{2})+(\sin
h_{1}-\sin h_{2})]\\\
&+w_{2}\mu(s_{1}-s_{2})+w_{3}\mu(v_{1}-v_{2})\end{split}$

where $\mu(x)$ is the average value of $x$ on all vertices and $w_{i}$ is
hyperparameter which controls the weight of HSV loss.

###  3.6. Viewpoint Selection

As mentioned before, during each optimization procedure we stylize one
semantic region in an indoor scene. Therefore we, or the camera, should focus
on the target region to prevent the situation that the rendered images fully
or largely ignore our targets.

Given a mesh that has been normalized into a unit ball, we first choose a
viewpoint $\theta$ and a focal length $f$ randomly. Then we apply random
disturbance to the viewpoint and focal length, resulting in $\theta^{\prime}$
and $f^{\prime}$. We get the rendered image from renderer defined previously
using parameters $\theta^{\prime}$ and $f^{\prime}$. Then we compute the ratio
$R$ of the target region in the full image and choose $n_{v}$ images where
$R\in(R_{min},R_{max})$. The $R_{min}$ ensures that an enough portion of
pixels of the rendered images display the target region while the $R_{max}$
keep the focal length in a reasonable scale so that most part of the target
region is visible. Then we apply $\theta_{j}$, $f_{j}$ to the renderer when we
train the Vertex Color Deviation Network.

Additionally, for a camera located in coordinates(x,y,z), the camera pitch
angle $\vec{\gamma}$ is computed as:

(19) $\vec{\gamma}=(-z\times x,-z\times y,x\times x+y\times y)$

which prevents the room to be rendered from an unnatural angle (e.g., the
upside down cases in Fig. 4).

Figure 5. Without semantic region masks, the style prompts are wrongly
transferred. Figure 6. With the HSV regularization term, LASST generates style
transfer results that look more similar with the input.

##  4\. Experiments

Figure 7. The comparision between stylization with ground truth masks and
predicted masks.

Enjoying the baseball game from the third-base seats. Ichiro Suzuki preparing
to bat.

In this section, we introduce the implementation details and experimental
designs. Then we provide ablation studies of the proposed design choices, both
qualitatively and quantitatively.

###  4.1. Dataset

The datasets used in our methods include the ScanNetV2 dataset (Dai et al.,
2017) and SceneNN dataset (Hua et al., 2016). ScanNetV2 is a large-scale RGB-D
video dataset with 3D reconstructions of indoor scenes with over 1500 scans.
SceneNN is an RGB-D scene dataset consisting of more than 100 indoor scenes.
All scenes are reconstructed into triangle meshes and have per-vertex and per-
pixel annotation.

###  4.2. Implementation Details

For each semantic part of the scene, we update the VCD network for 700
iterations using an Adam optimizer.

The differentiable renderer used in our framework is provided by Kaolin python
library (Fuji Tsang et al., 2022). For each forward pass, we render the scenes
with $n_{v}=5$ selected viewpoints without shading. The $R_{min}$ and
$R_{max}$ are 0.25 and 0.70 respectively in our selection.

As augmentations, we adopt random resizing and cropping to the rendered
results and apply random perspective transforms to the images. Augmented
images, along with text prompts, are sent to a pre-trained ViT-B/32 CLIP model
to extract features for semantic loss calculation.

The hsv loss weight $(w_{1},w_{2},w_{3})$in our experiments is
$(0.2,0.3,0.3)$, which is obtained from user study results.

###  4.3. User Study Protocol

We conduct the quantitative experiments by user study with an interactive web-
based program, which we build to collect human preferences of scene pairs in
the ablation studies. Each scene pair for comparison consists of two scenes
stylized from the same initial scene with the same text prompt but with
different experiment settings. Users are asked to selected one that better
fits the text prompts from the two stylized scenes. During experiments, we
divide the scene pairs into groups of 30 to avoid inaccuracy caused by boredom
of users. Each of 40 users checked $10\times 30$ pairs of comparisons.

We show the layout of our user study tool in Fig.3. The original scene and two
different stylized scenes are displayed as wavefront-format 3D models in the
web pages. By dragging and scrolling, users are free to zoom in or out as well
as translate and rotate each one of the models to observe from a better
viewpoint.

###  4.4. Ablation Study

We conduct comprehensive ablation study on multiple datasets to investigate
the capability of our LASST algorithm.

Viewpoint Selection. We investigate the effects of our viewpoint sampling
strategy compared with the strategy in Text2Mesh (Michel et al., 2021) with
random camera position and fixed camera pitch direction, as is shown in Fig.
4. In the lower two rows, we visualize intermediate rendering results. Our
renderer focuses on the target object with input label (e.g., refrigerator),
while the Text2Mesh sampling ignores the position and pose of the target,
leading to some rendered images under unnatural viewpoints, which deviate far
from natural images used for CLIP pre-training. Also in Fig. 4 some results
are displayed and we can see clearly our sampling method improves the final
stylized room. As shown in Table. 1, 78.95% of user ratings prefer LASST
sampling over Text2Mesh sampling.

Semantic Mask. In our method, we exploit semantic masks to apply stylization
in specific regions of the room, instead of the whole room. The semantic mask
helps LASST better align vision and language and stylize the room more
precisely. For example in the first column of Fig. 5, given an initial room
mesh and text prompt _steel table_ , we expect a room with a steel table like
the third row. But in the second row the whole room is stylized, which fails
to meet the user’s demand. Moreover, graffiti-like english characters _STEEL_
are attached to the wall. This is not surprising as vision language models can
understand characters in images, as demonstrated in the _OPENAI written on it_
demo in https://openai.com/blog/dall-e/. Also, we try to style transfer the
room with several prompts like ”wooden floor, silk sofa, wooden table”, which
is shown in the last column of Fig. 5. Then whole room is stylized as a wooden
room even if we have ”silk sofa” in text prompt, which shows the stylization
would miss some information in the text prompts without semantic mask. As
shown in Table. 1, 86.98% of user ratings prefer results generated with
semantic masks than without.

Table 1. Quantitative A/B user study results. The number means percentage of user tests that prefer A over B. A/B Preference Rate | ScanNet | SceneNN
---|---|---
Sampling(LASST v.s. Text2Mesh) | 78.95% | 76.04%
Semantic Mask(w/ v.s. w/o) | 86.98% | 85.45%
Semantic Mask(GT v.s. Prediction) | 71.43% | 59.09%
Regularization(HSV v.s. None) | 64.80% | 65.07%
Regularization(HSV v.s. RGB) | 60.51% | 60.27%
Table 2. CLIP Score and Runtime Comparison. CLIP scores measures the compatibility between an image and a caption by cosine similarity. Regularization | Sampling | CLIP Score | Optimization Time/s
---|---|---|---
None | LASST | 0.2972 | 375.53
None | Text2Mesh | 0.2744 | 355.27
RGB | LASST | 0.2887 | 395.52
HSV | LASST | 0.2669 | 385.84

HSV Regularization. Our method uses HSV regularization to encourage
consistency with initial scenes. We compare settings applying style transfer
with or without HSV regularization. As is shown in Fig. 6, we find that the
stylized room with HSV regularization well maintains the color impression of
input scenes and still achieves good similarity with text prompts. For example
given a room with text prompt _leather sofa_ (see the first column of Fig. 6),
both methods can stylize the input sofa as leather sofa. However, the result
without regularization makes the sofa too bright and stand out in the scene
context. As shown in Table. 1, 64.85% of user ratings prefer results generated
with HSV regularization than without.

Additionally, we also compare the style transfer effects with RGB
regularization. As shown in Table. 1, 60.51% of user ratings prefer results
generated with HSV than RGB regularization.

Using Estimated Semantic Masks. In previous experiments, we conduct our
experiments with the ground truth semantic masks. However, when stylizing an
indoor scene in real multimedia applications, we need to first conduct
semantic segmentation for the room. Thus we investigate the differences
between the results with ground truth labels and estimated labels.

The room mesh is first passed as input to a semantic segmantation model, e.g.
MinkowskiNet(Choy et al., 2019)(Luo et al., 2021). Then we exploit the output
predicted labels and ground truth labels respectively in our framework. The
semantic segmantation results and final stylized results are displayed in Fig.
7. We can see that the results with ground truth semantics and results with
semantic prediction are generally the same, despite some regions with wrongly
predicted labels (e.g., the wall between two refrigerators). As shown in
Table. 1, 71.43% of user ratings prefer results generated with ground truth
labels than estimated labels.

###  4.5. CLIP Score and Runtime Comparison

Although A/B user test is the golden evaluation standard for multimedia
applications, we also report the CLIP score after convergence in several
settings. As shown in Table. 2, using our sampling strategy can improve the
CLIP score from 0.2744 to 0.2972, when compared with Text2Mesh random
sampling. This is understandable, because randomly sampled viewpoints leads to
unnatural views that is _unfamiliar_ to the CLIP model. Meanwhile, using a
regularization (RGB or HSV) term inevitably impacts the main loss, but
according to our user study the user preference is improved.

On average, the stylization of a room mesh in the ScanNet dataset or SceneNN
dataset takes about six minites on a single Nvidia V100 GPU. The optimization
time is listed in Table. 2. When compared the optimization time in different
settings and find that generally our design choices improve the style transfer
quality with limited impact on algorithm speed.

##  5\. Conclusion

In this paper we propose the first language-driven semantic style transfer
algorithm for 3D indoor scenes, named LASST. The inputs are a 3D indoor scene
mesh and several phrases specifying target styles. The mesh is rendered
differentiablly into multiple 2D images and compared with text prompts in the
feature space of a pre-trained vision language model. We identify the
importance of better vision language alignment through semantic masks, a
viewpoint sampling strategy that incorporates human viewing priors, and an HSV
regularization loss that discourages too much drift from input colors. With a
large scale A/B user tests, we demonstrate that our design choices are well
recognized. We also provide comprehensive qualitative results showing the pros
and cons of our method.

## References

  * (1)
  * Bi et al. (2017) Sai Bi, Nima Khademi Kalantari, and Ravi Ramamoorthi. 2017\.  Patch-based optimization for image-based texture mapping.  _ACM Trans. Graph._ 36, 4 (2017), 106–1.
  * Cao et al. (2020) Xu Cao, Weimin Wang, Katashi Nagao, and Ryosuke Nakamura. 2020\.  Psnet: A style transfer network for point cloud stylization on geometry and color. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 3337–3345.
  * Chen et al. (2019) Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. 2019.  Learning to predict 3d objects with an interpolation-based differentiable renderer.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Choy et al. (2019) Christopher Choy, JunYoung Gwak, and Silvio Savarese. 2019\.  4d spatio-temporal convnets: Minkowski convolutional neural networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3075–3084.
  * Dai et al. (2017) Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017.  ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In _Proc. Computer Vision and Pattern Recognition (CVPR), IEEE_.
  * Dessein et al. (2014) Arnaud Dessein, William AP Smith, Richard C Wilson, and Edwin R Hancock. 2014.  Seamless texture stitching on a 3D mesh by poisson blending in patches. In _2014 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2031–2035.
  * Fuji Tsang et al. (2022) Clement Fuji Tsang, Maria Shugrina, Jean Francois Lafleche, Towaki Takikawa, Jiehan Wang, Charles Loop, Wenzheng Chen, Krishna Murthy Jatavallabhula, Edward Smith, Artem Rozantsev, Or Perel, Tianchang Shen, Jun Gao, Sanja Fidler, Gavriel State, Jason Gorski, Tommy Xiang, Jianing Li, Michael Li, and Rev Lebaredian. 2022\.  Kaolin: A Pytorch Library for Accelerating 3D Deep Learning Research.  https://github.com/NVIDIAGameWorks/kaolin.
  * Gillespie et al. (2021) Mark Gillespie, Boris Springborn, and Keenan Crane. 2021\.  Discrete conformal equivalence of polyhedral surfaces.  _ACM Transactions on Graphics (TOG)_ 40, 4 (2021), 1–20.
  * Han and Fang (2018) Lei Han and Lu Fang. 2018\.  FlashFusion: Real-time Globally Consistent Dense 3D Reconstruction using CPU Computing.. In _Robotics: Science and Systems_ , Vol. 1. 7.
  * Hauptfleisch et al. (2020) Filip Hauptfleisch, Ondrej Texler, Aneta Texler, Jaroslav Krivánek, and Daniel Sỳkora. 2020\.  StyleProp: Real-time Example-based Stylization of 3D Models. In _Computer Graphics Forum_ , Vol. 39. Wiley Online Library, 575–586.
  * Höllein et al. (2021) Lukas Höllein, Justin Johnson, and Matthias Nießner. 2021\.  StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions.  _arXiv preprint arXiv:2112.01530_ (2021).
  * Hua et al. (2016) Binh-Son Hua, Quang-Hieu Pham, Duc Thanh Nguyen, Minh-Khoi Tran, Lap-Fai Yu, and Sai-Kit Yeung. 2016\.  Scenenn: A scene meshes dataset with annotations. In _2016 fourth international conference on 3D vision (3DV)_. Ieee, 92–101.
  * Huang et al. (2017) Jingwei Huang, Angela Dai, Leonidas J Guibas, and Matthias Nießner. 2017.  3Dlite: towards commodity 3D scanning for content creation.  _ACM Trans. Graph._ 36, 6 (2017), 203–1.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 4401–4410.
  * Kato et al. (2018) Hiroharu Kato, Y. Ushiku, and Tatsuya Harada. 2018.  Neural 3D Mesh Renderer.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2018), 3907–3916.
  * Li et al. (2018) Minchen Li, Danny M Kaufman, Vladimir G Kim, Justin Solomon, and Alla Sheffer. 2018.  OptCuts: Joint optimization of surface cuts and parameterization.  _ACM Transactions on Graphics (TOG)_ 37, 6 (2018), 1–13.
  * Liu et al. (2019) Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019\.  Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 7708–7717.
  * Loper and Black (2014) Matthew M Loper and Michael J Black. 2014.  OpenDR: An approximate differentiable renderer. In _European Conference on Computer Vision_. Springer, 154–169.
  * Lu et al. (2018) Ming Lu, Feng Xu, Hao Zhao, Anbang Yao, Yurong Chen, and Li Zhang. 2018\.  Exemplar-based portrait style transfer.  _IEEE Access_ 6 (2018), 58532–58542.
  * Lu et al. (2017) Ming Lu, Hao Zhao, Anbang Yao, Feng Xu, Yurong Chen, and Li Zhang. 2017\.  Decoder Network over Lightweight Reconstructed Feature for Fast Semantic Style Transfer.  _2017 IEEE International Conference on Computer Vision (ICCV)_ (2017), 2488–2496.
  * Luo et al. (2021) Liyi Luo, Beiwen Tian, Hao Zhao, and Guyue Zhou. 2021\.  Pointly-supervised 3D Scene Parsing with Viewpoint Bottleneck.  _arXiv preprint arXiv:2109.08553_ (2021).
  * Michel et al. (2021) Oscar Jarillo Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. 2021.  Text2Mesh: Text-Driven Neural Stylization for Meshes.  _ArXiv_ abs/2112.03221 (2021).
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021\.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ (2021).
  * Nießner et al. (2013) Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Marc Stamminger. 2013.  Real-time 3D reconstruction at scale using voxel hashing.  _ACM Transactions on Graphics (ToG)_ 32, 6 (2013), 1–11.
  * Patashnik et al. (2021) Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and D. Lischinski. 2021.  StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery.  _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ (2021), 2065–2074.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International Conference on Machine Learning_. PMLR, 8748–8763.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021\.  Zero-shot text-to-image generation. In _International Conference on Machine Learning_. PMLR, 8821–8831.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015\.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ 115, 3 (2015), 211–252.
  * Sharp and Crane (2018) Nicholas Sharp and Keenan Crane. 2018.  Variational surface cutting.  _ACM Transactions on Graphics (TOG)_ 37, 4 (2018), 1–13.
  * Shridhar et al. (2022) Mohit Shridhar, Lucas Manuelli, and Dieter Fox. 2022\.  Cliport: What and where pathways for robotic manipulation. In _Conference on Robot Learning_. PMLR, 894–906.
  * Sỳkora et al. (2019) Daniel Sỳkora, Ondřej Jamriška, Ondrej Texler, Jakub Fišer, Michal Lukáč, Jingwan Lu, and Eli Shechtman. 2019.  StyleBlit: Fast Example-Based Stylization with Local Guidance. In _Computer Graphics Forum_ , Vol. 38. Wiley Online Library, 83–91.
  * Waechter et al. (2014) Michael Waechter, Nils Moehrle, and Michael Goesele. 2014\.  Let there be color! Large-scale texturing of 3D reconstructions. In _European conference on computer vision_. Springer, 836–850.
  * Yin et al. (2021) Kangxue Yin, Jun Gao, Maria Shugrina, Sameh Khamis, and Sanja Fidler. 2021.  3dstylenet: Creating 3d shapes with geometric and texture style variations. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 12456–12465.
  * Zhou and Koltun (2014) Qian-Yi Zhou and Vladlen Koltun. 2014.  Color map optimization for 3d reconstruction with consumer depth cameras.  _ACM Transactions on Graphics (ToG)_ 33, 4 (2014), 1–10.


# Convolutional restricted Boltzmann machine (CRBM) correlated variational
wave function for the Hubbard model on a square lattice: Mott metal-insulator
transition

Karthik V<EMAIL_ADDRESS>Amal Medhi<EMAIL_ADDRESS>Indian
Institute of Science Education and Research Thiruvananthapuram, Kerala 695551,
India

###### Abstract

We use a convolutional restricted Boltzmann machine (CRBM) neural network to
construct a variational wave function (WF) for the Hubbard model on a square
lattice and study it using the variational Monte Carlo (VMC) method. In the
wave function, the CRBM acts as a correlation factor to a mean-field BCS
state. The number of variational parameters in the WF does not grow
automatically with the lattice size and it is computationally much more
efficient compared to other neural network based WFs. We find that in the
intermediate to strong coupling regime of the model at half-filling, the wave
function outperforms even the highly accurate long range backflow-Jastrow
correlated wave function. Using the WF, we study the ground state of the half-
filled model as a function of onsite Coulomb repulsion $U$. We consider two
cases for the next-nearest-neighbor hopping parameter, e.g., $t^{\prime}=0$ as
well as a frustrated model case with $t^{\prime}\neq 0$. By examining several
quantities, e.g., double occupancy, charge gap, momentum distribution, and
spin-spin correlations, we find that the weekly correlated phase in both cases
is paramagnetic metallic (PM). As $U$ is increased, the system undergoes a
first-order Mott transition to an insulating state at a critical $U_{c}$, the
value of which depends upon $t^{\prime}$. The Mott state in both cases is spin
gapped with long range antiferromagnetic (AF) order. Remarkably, the AF order
emerges spontaneously from the wave function which does not have any
explicitly broken symmetry in it. Apart from some quantitative differences in
the results for the two values of $t^{\prime}$, we find some interesting
qualitative differences in the way the Mott transition takes place in the two
cases.

##  I Introduction

The Hubbard Hamiltonian has been studied extensively over the years using a
variety of analytical and numerical methods as a paradigmatic model for
various correlated electron phenomenaImada  _et al._ (1998); LeBlanc  _et al._
(2015). The model on a two-dimensional lattice is particularly interesting
because of its relevance to the high-temperature superconductorsLee  _et al._
(2006a). Theoretically, the model is handled well both in the weak and strong
coupling limits by different methods, but not so in the most interesting
intermediate coupling regime. Among various numerical methods, the variational
Monte Carlo (VMC) technique has been a very useful over the years in the study
of the ground state properties of the model and its various extensionsCeperley
_et al._ (1977); Tahara and Imada (2008); Sorella (2013); Paramekanti  _et
al._ (2004); SHIBA and YOKOYAMA (1987). Unlike other methods, VMC does not
suffer from any particular difficulty at any coupling strength, but its
results are always biased by the choice of the variational wave function (WF)
which is generally constructed based on the ground state of an underlying
mean-field Hamiltonian. However, with the advent of machine learning
algorithms based on artificial neural networks (ANNs), it was realized that
ANNs can also be used to represent a quantum many-body wave function
Carrasquilla and Melko (2017); Carleo and Troyer (2017); van Nieuwenburg  _et
al._ (2017); Broecker  _et al._ (2017); Ch’ng  _et al._ (2017); Deng  _et al._
(2017). In such wave functions termed as neural-network quantum states (NQS),
the number of variational degrees of freedom is large and it offers the
possibility to overcome the fundamental limitation of the variational method
by enabling the construction of a highly unbiased variational wave function.

A number of seminal works have already demonstrated the power of NQS to learn
quantum many-body physicsCarleo  _et al._ (2019); Ch’ng  _et al._ (2017);
Carrasquilla (2020); Melko  _et al._ (2019); Carrasquilla and Torlai (2021).
To mention some of these, Carleo and TroyerCarleo and Troyer (2017) first
demonstrated the use of a restricted Boltzmann machine (RBM) based variational
wave function to represent the ground state and study the dynamics of a few
prototypical spin Hamiltonians accurately and efficiently. The method involved
training the network by using a reinforcement learning mechanism which
essentially is an iterative tuning of the network parameters so as to minimize
the variational energy. Torlai et al.Torlai  _et al._ (2018) used an NQS wave
function based on RBM network to perform a quantum state tomography and
thereby learn the ground state of a quantum spin Hamiltonian. In Ref. Carleo
_et al._ , 2018, the authors used a wave function based on deep Boltzmann
machine (DBM) and performed imaginary time evolution to obtain accurate ground
state of the transverse-field Ising and the Heisenberg model. Choo et al.Choo
_et al._ (2018), by incorporating translational symmetries explicitly into the
NQS wave function, managed to generate also the excited states for the
Heisenberg and Bose-Hubbard model. RBM were also shown to be very efficient in
representing topological quantum states due to the non-local geometry of its
architectureLu  _et al._ (2019); Glasser  _et al._ (2018). However, most of
these applications so far have been to bosonic systems. One main reason for
this is that the fermionic systems have an additional complexity that comes
from the non-trivial sign structures of its wave functionsZhang and Weng
(2014). The functions that can be represented by neural-networks though highly
non-linear, are essentially smooth and generally fail to represent the sign
structures of fermionic many-body wave functions. For instance, the feed-
forward neural networks, a type of ANN, were shown to be unable to correctly
account for the sign structure of even a free fermionic wave functionCai and
Liu (2018). Nomura et al.Nomura  _et al._ (2017) constructed a wave function
by combining an RBM with a mean-field part where the RBM part introduces
correlation effects, a role usually played by Jastrow type correlation
factorsSorella (2013). It was shown that the combination wave function give
substantially lower energy than the conventional projected variational wave
functions for the fermionic Hubbard model as well as the Heisenberg model.
There also exist other schemes based ANNs for fermionic systems but these are
restricted to small system size due to computational complexityInui  _et al._
(2021); Luo and Clark (2019); Pfau  _et al._ (2020).

In this work, we construct a ground state variational wave function for the
fermionic Hubbard model on a square lattice using a convolutional restricted
Boltzmann machine (CRBM) network where the CRBM is used as a correlation
factor to a mean-field BCS state. We show that the wave function is not only
computationally efficient compared to RBM wave function, it is also highly
accurate in terms of the variational energy. Using VMC, we study the wave
function as a ground state of the model at half-filling as a function of
onsite Coulomb repulsion $U$. We consider two cases of the model when the
next-nearest-neighbor hopping parameter $t^{\prime}=0$ and when the model is
frustrated with $t^{\prime}\neq 0$. The wave function yields energies which
are significantly lower compared to those of the corresponding RBM or Jastrow
projected wave functions, especially in the strong coupling limit. Indeed, in
this limit it outperforms in terms of energy even the highly accurate long
range backflow-Jastrow correlated wave function for the model. The wave
function correctly captures the presence of doublon-holon (DH) binding in the
strong coupling regime in spite of it having no explicit DH binding term. It
also give rise to long range antiferromagnetic (AF) order in the Mott
insulating state spontaneously even though the wave function contains no
explicit magnetic order. We examine the nature of Mott metal-insulator
transition in the model by calculating various quantities, such as double
occupancy, charge gap, momentum distribution and quasiparticle weight, and
spin-spin correlation. The results are thoroughly discussed and compared with
other variational results. The rest of the paper is organized as follows. In
section II, we describe the model and the CRBM wave function. In section III,
we describe and discuss the results, and finally in section IV, we make the
concluding remarks.

##  II Model and method

We consider the fermionic Hubbard model on a two-dimensional (2D) square
lattice at half-filling. The Hamiltonian is given by,

$\displaystyle\mathcal{H}=-\sum_{i,j\sigma}t_{ij}\left(c^{\dagger}_{i\sigma}c_{j\sigma}+hc\right)+U\sum_{i}n_{i\uparrow}n_{i\downarrow}$
(1)

where the operator $c^{\dagger}_{i\sigma}$ creates an electron at site ‘$i$’
with spin $\sigma$. $c_{i\sigma}$ is the corresponding annihilation operator
and $n_{i\sigma}=c^{\dagger}_{i\sigma}c_{i\sigma}$ is the number operator. The
first term represents hoppings of electrons from site to site. The hopping
integrals are $t_{ij}=t$ for $i$, $j$ nearest-neighbor (NN) sites,
$t_{ij}=-t^{\prime}$ for $i$, $j$ next-nearest-neighbor (NNN) sites, and zero
otherwise. Due to the particle-hole symmetry at half-filling, the model is
equivalent to the case with opposite sign of $t^{\prime}$. The Hamiltonian is
the simplest paradigmatic model that capture the essential physics of several
interesting correlated electron phenomenon in condensed matter physics,
including Mott metal-insulator transition, high-temperature cuprate
superconductors, etc.MOTT (1968); Imada  _et al._ (1998); Orenstein and Millis
(2000); Lee  _et al._ (2006b). The Mott physics in the Hubbard model has been
studied extensively using various methods Georges  _et al._ (1996); Zhang  _et
al._ (1993); Rozenberg  _et al._ (1994); Park  _et al._ (2008); Rozenberg  _et
al._ (1999); Bulla  _et al._ (2001); Lu (1994); Ferrero  _et al._ (2005);
Rüegg  _et al._ (2005); Mukherjee and Lal (2020), including variational theory
within the framework of various Gutzwiller-Jastrow type wave functions
(WFs)Yokoyama and Shiba (1990); Yokoyama  _et al._ (2006); Capello  _et al._
(2005, 2006); Tocchio  _et al._ (2008, 2011). These projected variational wave
functions are typically of the form, $|\Psi_{var}\rangle={\cal P}\ket{\Phi}$,
where $\ket{\Phi}$ is a one-body wave function which is generally taken to be
ground state of an underlying mean-field Hamiltonian. ${\cal P}$ is a
projection operator or correlation factor which introduces many-body
correlation into the wave function. The simplest case is the Gutzwiller (GW)
projector ${\cal P}_{G}$ which describes an on-site density-density
correlation, ${\cal
P}_{G}=\prod_{i}\left[1-(1-g)n_{i\uparrow}n_{i\downarrow}\right]$, $0<g\leq
1$. It penalizes electronic configurations with doubly occupied sites thereby
giving a better description of the ground state of the Hubbard model compared
to the uncorrelated state. However, GW projector was found to be inadequate to
describe the Mott insulating state. This is because in the Mott state, local
charge fluctuations creates configurations with doubly occupied sites
(doublon) next to empty sites (holon). The insulating nature of the state
means that the doublons and holons must be bound to each other and the GW
projector does not capture this effectKaplan  _et al._ (1982); Yokoyama  _et
al._ (2006); Capello  _et al._ (2005, 2006). It was shown that the situation
can be greatly improved by considering a correlation factor of the form ${\cal
P}={\cal P}_{DH}{\cal P}_{G}$, where the factor ${\cal P}_{DH}$ incorporates
doublon-holon (DH) binding in the wave function and is given byYokoyama  _et
al._ (2006), ${\cal P}_{DH}=\prod_{i}\left(1-\eta Q_{i}\right)$, with
$Q_{i}=\prod_{\delta}\left[d_{i}(1-h_{i+\delta})+h_{i}(1+d_{i+\delta})\right]$.
Here $d_{i}=n_{i\uparrow}n_{i\downarrow}$ is doublon and
$h_{i}=(1-n_{i\uparrow})(1-n_{i\downarrow})$ is holon operator, $\delta$
denotes the nearest-neighbor sites, and $\eta$ is a variational parameter.
Alternatively, one can also consider Jastrow factor of the formCapello  _et
al._ (2005), ${\cal
P}_{J}=\exp\left(\sum_{ij}\frac{1}{2}v_{ij}(n_{i}-1)(n_{j}-1)+w_{ij}h_{i}d_{j}\right)$
which introduces long range correlations including DH binding. The description
can be further improved by introducing a backflow correlation term in addition
to the Jastrow factor and such a wave function considered for the half-filled
Hubbard model was shown to be much better in terms of the ground state energy
and in the description of the Mott insulating state in the modelTocchio  _et
al._ (2011).

Here, we consider a variational wave function where the correlator is based on
an artificial neural-network (ANN). Such a wave function using the restricted
Boltzmann machine (RBM) network as a correlator was already considered for the
Hubbard as well as the Heisenberg modelNomura  _et al._ (2017). Here we
consider a convolutional restricted Boltzmann machine (CRBM) network as a
correlator. The CRBM is computationally much more efficient than an RBM owing
to much lesser number of network connections and consequently fewer number of
variational parameters in CRBM. In fact, the number of variational parameters
in CRBM can be tuned and does not grow automatically with the lattice size
which is a much desirable feature computationally. In the followings, first we
describe the RBM network followed by a detailed description of the structure
of the CRBM network. The variational wave function using the CRBM as a
correlator is described next.

###  II.1 RBM network

A restricted Boltzmann machine (RBM) networkCarleo  _et al._ (2018) consists
of two layers of artificial neurons, one visible layer connected to a hidden
layer as shown in Fig. 1. There is no intralayer connection between the
neurons.

Figure 1: A restricted Boltzmann machine (RBM) network. The artificial neurons
in the visible layer take the inputs $\\{n_{1},n_{2},\ldots,n_{2L}\\}$ that
represent the sets of electron occupation numbers. The neurons in the hidden
layer define variables $h_{i}$ which can take values $\pm 1$.

The number of neurons in the visible layer is $N_{v}=2L$ where $L$ is the
number of lattice sites and that in the hidden layer is $N_{h}$. The energy
function of an RBM is given by,

$\displaystyle
E_{RBM}=-\sum_{i,j}w_{ij}h_{i}n_{j}-\sum_{j}a_{j}n_{j}-\sum_{i}b_{i}h_{i}$ (2)

The hidden variables $h_{i}$ take values $h_{i}=\pm 1$. The set $W=\\{\bf
a,b,w\\}$ denotes the set of all the network parameters $a_{i}$, $b_{j}$, and
$w_{ij}$. Carrying out the summation over the hidden variables, the
probability distribution over the set of input values $\\{n_{1},n_{2},\ldots
n_{2L}\\}$ is given by,

$\displaystyle\Psi_{RBM}(R,W)=\frac{1}{Z}\sum_{\bf h}e^{-E_{RBM}}$ (3)

where $Z$ is the partition function. The number of parameters in the network
can be drastically reduced by using the symmetries of the Hamiltonian. Here we
take the hidden variable density to be $\alpha=N_{h}/N_{v}=1$. In this case,
it can easily be seen that imposition of lattice translational symmetry leads
to a single bias parameter $a$ for the visible units and a parameter $b$ for
the hidden units. Also the number of unique elements in the weight matrix $w$
reduces to $2L$. Thus the total number of network parameters becomes $(2L+2)$
and the probability function reduces to,

$\displaystyle\Psi_{RBM}(R,{W})=\frac{1}{Z}e^{a\sum_{i}n_{i}}\prod_{i}2\cosh\Bigl(\sum_{j}w_{ij}n_{j}+b\Bigr{missing})$
(4)

###  II.2 CRBM network

The convolutional restricted Boltzmann machine (CRBM) architecture is shown
schematically in Fig. 2.

Figure 2: Schematic diagram of a CRBM network. The visible layer ($V$)
contains $N_{v}=2L$ neurons as in RBM. The hidden layer is enlarged compared
to RBM and consists of $M$ blocks $H_{1},\,H_{2},\,\ldots,H_{M}$, each
containing $N_{v}$ neurons. There is a convolutional layer in between which
also consists of $M$ blocks $C_{1},\,C_{2},\,\ldots,C_{M}$. The $m$-th block
applies a convolutional filter with kernel $K_{m}$ to the visible layer and
generates an output which is in the form of $N_{v}$ output neurons comprising
the $C_{m}$ block. There are one-to-one connections between the neurons in the
$C_{m}$ and $H_{m}$ blocks.

It can be thought of as consisting of three layers with an additional
convolutional layer in between the visible and the hidden layersNorouzi
(2009); Alcalde Puente and Eremin (2020). Since the underlying lattice is two-
dimensional (2D), it is necessary to consider layers also to be 2D arrays of
neurons instead of the linear arrangement of Fig. 1. The visible layer again
contain $N_{v}=2L$ neurons, like in RBM. But the hidden layer ($H$ layer) in
CRBM is enlarged to make it into $M$ blocks $H_{1},\,H_{2}\,\ldots,H_{M}$,
each block containing $N_{h}=N_{v}$ neurons. The middle convolutional layer
($C$ layer) also consists of $M$ blocks $C_{1},\,C_{2},\,\ldots,\,C_{M}$. The
input layer, the blocks $C_{m}$-s, $H_{m}$-s are all taken to be $2P\times P$
arrays of neurons, where $P=\sqrt{L}$. The $m$-th block applies a convolution
filter with kernel $K_{m}$ to the input layer and produces a result which is
in the form of $N_{v}$ number of convolutional output neurons ($2P\times P$
array) comprising the $C_{m}$ block. The convolutional output blocks are
connected directly to the hidden layer blocks. That is, the $i$-th neuron in
$m$-th convolutional block is connected only to the $i$-th neuron of the
$m$-th hidden layer block, and not to any other hidden unit. Thus, there is a
one-to-one connection between the neurons in the blocks $C_{m}$ and $H_{m}$,
with a total of $N_{v}$ connections between the two sub-layers. Therefore for
$M\ll L$, the total number of connections in the CRBM is much less compared to
that in the RBM, making the CRBM representation much more efficient in
comparison.

Figure 3: Schematic diagram showing the convolutional operation by one of the
$C$-blocks in Fig. 2. The convolutional layer slides a filter over the visible
layer with a stride of one. The operation generates the convolutional output
layer which is of the same dimension as the input layer.

Next, consider the convolutional operation by the $m$-th block (Fig. 3). The
convolutional filter in the block is defined by its kernel $K_{m}$ which is a
matrix of $D\times D$ parameters ($D\leq P$). Action of the filter on a
$D\times D$ window of the visible layer is given by the generalized dot
product $K_{m}\odot V_{k}$, where $V_{k}$ is the matrix formed by the input
values given to the neurons in this window. This is shown schematically in
Fig. 3. We slide the filter with a stride of one step along both the
directions to cover the whole of the visible layer. This creates $N_{v}$
number of windows of the visible layer and hence $N_{v}$ number of
convolutional outputs $K_{m}\odot V_{k}$, $k=1,\,2,\,\ldots N_{v}$. Denoting
the $i$-th hidden variable in the $m$-th block by $h_{mi}$, the joint energy
function in the CRBM is given by,

$\displaystyle E_{CRBM}=-\sum_{m,k}h_{mk}(K_{m}\odot
V_{k})-a\sum_{j}n_{j}-\sum_{mi}b_{m}h_{mi}$ (5)

It may be mentioned that by construction CRBM conservs the lattice
translational symmetry imposed in the corresponding RBM. The probability
distribution over the visible variables in CRBM is given by,

$\displaystyle\Psi_{CRBM}=$ $\displaystyle\frac{1}{Z}\sum_{\bf
h}e^{-E_{CRBM}}$ $\displaystyle=$
$\displaystyle\frac{1}{Z}e^{a\sum_{j}n_{j}}\prod_{mk}2\cosh\left(K_{m}\odot
V_{k}+b_{m}\right)$ (6)

where $Z$ is a normalization constant. The total number of parameters in the
CRBM is $M(2D^{2}+1)$. This number depends only upon the two network
structural parameters $M$ and $D$, and not upon the lattice size, $L$.
Therefore the number of variational parameters in the CRBM wave function does
not grow directly with the lattice size which is a big advantage in the
optimization process. In practice, the parameters $M$ and $D$ are determined
by tuning its values so as to obtain the best variational energy. The lattice
size might affect these values somewhat, but the number of variational
parameters is still expected to be much less than that in the corresponding
RBM wave function.

###  II.3 CRBM correlated wave function

The CRBM correlated wave function that we consider here is given by,

$\displaystyle|\Psi_{var}\rangle={\cal P}_{CRBM}\ket{BCS}_{N}$ (7)

where $\ket{BCS}_{N}$ is the ground state (with fixed number of particles) of
the following mean-field Hamiltonian,

$\displaystyle{\cal H}_{MF}=$
$\displaystyle\sum_{\mathbf{k}\sigma}\varepsilon_{\mathbf{k}}c^{\dagger}_{\mathbf{k}\sigma}c_{\mathbf{k}\sigma}-\sum_{\mathbf{k}}\left(\Delta_{\mathbf{k}}c^{{\dagger}}_{\mathbf{k}\uparrow}c^{{\dagger}}_{-\mathbf{k}\downarrow}+hc\right)$
(8)

Here $\varepsilon_{\mathbf{k}}=-2t(\cos k_{x}+\cos k_{y})+4t^{\prime}\cos
k_{x}\cos k_{y}-\mu$, $\mu$ being the chemical potential. We take
superconducting pairing amplitude $\Delta_{\mathbf{k}}$ to be of
$d_{x^{2}-y^{2}}$-wave ($d$-wave) symmetry with
$\Delta_{\mathbf{k}}=\Delta_{SC}(\cos k_{x}-\cos k_{y})$, where $\Delta_{SC}$
is the SC gap parameter. The quantities $\Delta_{SC}$ and $\mu$ are the
variational parameters in the one-body part of the wave function. We consider
the wave function in fixed particle number ($2N$) representation with equal
number of up and down spins. In terms of real-space (Wannier) basis,
$\ket{BCS}_{N}$ can be expressed as,

$\displaystyle\ket{BCS}_{N}=$ $\displaystyle\sum_{n_{1},n_{2},\ldots
n_{2L}}\Psi_{BCS}(n_{1},n_{2},\ldots n_{2L})$
$\displaystyle\times(c^{{\dagger}}_{1\uparrow})^{n_{1}}\ldots(c^{{\dagger}}_{L\uparrow})^{n_{L}}(c^{{\dagger}}_{1\downarrow})^{n_{L+1}}\ldots(c^{{\dagger}}_{L\downarrow})^{n_{2L}}\ket{0}$
$\displaystyle=$ $\displaystyle\sum_{R}\Psi_{BCS}(R)\ket{R}$ (9)

Here $n_{i}$-s are the occupation numbers which can take value $0$ or $1$. $L$
is the number of lattice sites. The summation is over the set of values
$\\{n_{1},\ldots n_{2L}\\}\equiv R$ subject to constraint
$\sum_{i=1}^{L}n_{i}=\sum_{i={L+1}}^{2L}n_{i}=N$. The amplitudes
$\Psi_{BCS}(R)$-s are the determinantal coefficients corresponding to the
electronic configurations. The action of ${\cal P}_{CRBM}$ is given by,

$\displaystyle\ket{\Psi_{var}}={\cal
P}_{CRBM}\ket{\Phi}=\sum_{R}\Psi_{CRBM}(R)\Psi_{BCS}(R)\ket{R}$ (10)

where $\Psi_{CRBM}(R)$ is output (Eq. (6)) of the CRBM network for a given set
of input values $R=\\{n_{1},\ldots n_{2L}\\}$. The variational parameters in
the wave function consists of the network parameters plus the parameters in
the mean-field part of the wave function.

###  II.4 Variational Monte Carlo

Having constructed the variational wave function, we use the variational Monte
Carlo (VMC) methodCeperley  _et al._ (1977); Sorella (2013); Tahara and Imada
(2008) to compute the variational energy,

$\displaystyle E_{var}(\bm{\alpha})=\frac{\bra{\Psi_{var}}{{\cal
H}}\ket{\Psi_{var}}}{\bra{\Psi_{var}}\Psi_{var}\rangle}$ (11)

and minimize it with respect to the variational parameters $\bm{\alpha}$. We
use the stochastic reconfiguration (SR) techniqueSorella (2013); Tahara and
Imada (2008) for optimization which generally works all even for large number
of variational parameters. In this method, the variational parameters
$\bm{\alpha}$ are updated as, $\bm{\gamma}=\bm{\alpha}-\Delta
tS^{-1}\mathbf{g}$ where $\mathbf{g}=\bm{\nabla}_{\alpha}E_{var}$ is the
energy gradient. $S$ is the overlap matrix with the matrix elements given by,
$S_{kl}=\langle\Psi_{\alpha_{k}}\ket{\Psi_{\alpha_{l}}}$,
$\ket{\Psi_{\alpha_{k}}}=\frac{\partial}{\partial\alpha_{k}}\ket{\Psi_{\bm{\alpha}}}$.
In the VMC simulations, typically we take $\sim 10^{5}$ measuring steps after
warming up the system for $\sim 10^{3}$ steps. Each MC step consists of $L$
number of MC moves that include both the hopping and exchange moves as
mentioned before. The number of variational parameters in the CRBM wave
function depends upon the two network structural parameters $M$ and $D$, and
independent of the lattice size. For the values of $M$ and $D$ considered
here, the number of variational parameters becomes of the order of 150. The
optimization step even with the SR method sometime does need large number of
iterations to converge which creates a bottleneck in the computations.

##  III Results

We consider the Hamiltonian on a square lattice of size $10\times 10$ with
$L=100$ sites, and a band filling of one particle per site (half-filling). We
consider two values of the next-nearest hopping parameter $t^{\prime}$, e.g.
$t^{\prime}/t=0$ and $t^{\prime}/t=0.5$. The non-zero value $t^{\prime}/t$
brings in frustration in the antiferromagnetic order expected at half-filling.
We study the model as a function of Hubbard onsite interaction $U/t$. In the
results below, all the energy values quoted are energy per site and in units
of $t$.

Before going ahead with the calculations, we need to determine the optimal
configuration of the CRBM network. It has two crucial structural parameters -
the dimension $D$ of the convolutional kernel and the number of convolution
blocks $M$. In principle, $D$ can vary from 1 to $L$. The convolution can be
interpreted as a ‘feature extraction’ operationNorouzi (2009). A value $D=1$
implies a trivial operation while $D=L$ implies flattening of all features.
Here we take $D=4$ which we find to be an optimal value in terms of
performance and efficiency. Regarding value of $M$, we checked the energy
obtained by repeating the optimizations for different values of $M$. An
example plot is shown in Fig. 4.

Figure 4: Energy obtained by repeating optimizations for different values of
the CRBM parameter $M$. The energy reaches a saturation value at $M\sim 5$.
The value of the parameter $D$ is 4. Model parameters, $U/t=8$ and
$t^{\prime}/t=0$.

It shows that at $U/t=8$, the best energy is obtained for a minimum $M$ value
of $5$. In fact, the optimal value of $M$ depends upon the value of $U$. It is
smaller for smaller $U$. Therefore, a value of $5$ works well for the range of
$U/t$ considered here and hence we set $M=5$ for the rest of the calculations.

We optimize the CRBM wave function defined in Eq. (10) by minimizing the
corresponding variational energy for a range of model parameter values. For
comparison, we also calculate the energies of four other variational wave
functions. These are (i) $\ket{\Psi_{GW}}={\cal P}_{G}\ket{BCS}_{N}$ (GW) (ii)
$\ket{\Psi_{GW+DH}}={\cal P}_{G}{\cal P}_{DH}\ket{BCS}_{N}$ (GW+DH) (iii)
$\ket{\Psi_{Jastrow}}={\cal P}_{J}\ket{BCS}_{N}$ (Jastrow) and (iv)
$\ket{\Psi_{RBM}}={\cal P}_{RBM}\ket{BCS}_{N}$ (RBM), where the projection
operators are as defined above. In $\ket{\Psi_{RBM}}$, we use an RBM network
as the correlator. The comparison of energies of these wave functions are
shown in Fig. 5.

(a) (b)

Figure 5: (a) Energies of the five wave functions described in the text as a
function of $U/t$. (b) The difference $\Delta E=E_{X}-E_{GW}$, where $X$
stands for the four wave function names shown in the figure.

It shows that, in the weak coupling limit, the energies of these WFs are more
or less similar with minor differences among them. However, the energies start
to differ in the strong coupling regime. This is clear from Fig. 5(b) where we
have plotted the difference $\Delta E=E_{X}-E_{GW}$, where $X$ stands for the
four other wave functions shown in the figure. As the figure shows, $\Delta E$
is negative in the intermediate to strong coupling regime, indicating that the
energy of the GW wave function is highest here. The energies of the GW+DH and
RBM wave functions are comparable and lower than that of the Jastrow wave
function. The best wave function among the five is the CRBM wave function
which yields energies significantly lower than the other four. We also compare
the CRBM energies with those for the long range backflow-Jastrow correlated
wave function used in Ref. Tocchio  _et al._ , 2011, which was shown to be the
most accurate among the Jastrow type projected wave functions. We find that
the CRBM wave function even outperforms the backflow-Jastrow wave function.
For example, the backflow WF give energies per site equal to $-0.5961(1)$,
$-0.4803(1)$, $-0.4022(1)$ and $-0.3451(1)$ at $U/t$ equal to $6$, $8$, $10$,
and $12$, respectively. For the same values of $U/t$, the CRBM wave function
give energies per site equal to $-0.6076(2)$, $-0.4882(2)$, $-0.4048(2)$, and
$-0.3456(2)$, respectively. These energies are clearly lower than the backflow
WF energies. It must be mentioned that the lattice sizes used in these two
studies are not the same, and hence the comparison is not strictly rigorous.
Still, it gives an idea about how accurate the CRBM wave function is. In Ref.
Nomura  _et al._ , 2017, the authors used an RBM correlator in conjunction
with a many-variable one-body wave function. The energies of this wave
function is actually lower than the CRBM energies, though such a wave function
is computationally much more expensive.

Next, we examine the ground state phase of the model as a function of $U/t$ as
described by the CRBM wave function. As mentioned before, we have done
calculations for two different values of $t^{\prime}$, e.g. $t^{\prime}/t=0$
and $t^{\prime}/t=0.5$. Although the model has been studied extensively in the
past using a variety of methods, several important aspects of its phase
diagram including the nature of the zero temperature Mott transition, are not
yet established unambiguouslySchäfer  _et al._ (2015); LeBlanc  _et al._
(2015); Mukherjee and Lal (2020). For example, questions exist whether the
ground state of the unfrustrated model ($t^{\prime}/t=0$) on square lattice is
antiferromagnetic (AF) insulating at any $U/t>0$, or is it paramagnetic with a
finite value of critical interaction $U_{c}/t$ for Mott metal-insulator
transition. What is the nature of Mott insulating state, is it magnetically
ordered or a spin liquid? If the unfrustrated model is AF insulating, is there
a critical value of $t^{\prime}$ beyond which the ground state become
paramagnetic metallic? The answers to these questions somewhat vary among
different methods. Within the variational theory, the phase diagram of the
two-dimensional Hubbard model has been studied by looking at the competition
between wave functions with different symmetries. In these wave functions, the
one-body part is taken to be either pure BCS type without any magnetic order
or an AF type with explicitly broken spin rotational symmetry. By using the
GW-DH projector as the correlator, Yokoyama et al.Yokoyama  _et al._ (2006)
have shown that at $t^{\prime}=0$, the lowest energy state at any $U/t>0$ is
the symmetry broken one with long range AF order and insulating. For
$t^{\prime}/t>0$, there exists a non-zero value of critical interaction
$U_{c}$ below which the state is paramagnetic metallic (PM). For
$U/t>U_{c}/t$, the state is AF insulating for small $t^{\prime}/t$, but
paramagnetic insulating at large $t^{\prime}/t$. The transition to the AF
insulating state at $t^{\prime}/t=0$ is continuous. It gradually turns first-
order at larger $t^{\prime}/t$. However, if considered within the non-magnetic
projected BCS wave function only, the state is found to be PM for any
$t^{\prime}/t$ below a critical $U_{c}/t\neq 0$. In this case, the Mott
transition is of first-order nature at any $t^{\prime}/t$, and the insulating
state is non-magnetic. In another work, Tocchio et alTocchio  _et al._ (2008)
used variational WFs with backflow correlations in addition to a long range
Jastrow projector. The backflow correlated WFs is much more accurate with
lower variational energy compared to only DH or Jastrow projected WFs. The
study also showed that for small $U/t$ and non-zero $t^{\prime}/t$, the ground
is paramagnetic metallic. It becomes insulating with a long range AF order as
$U$ is increased above a critical value. Interestingly, the backflow WF give a
region in the phase diagram at large enough values of $U/t$ and
$t^{\prime}/t$, where the state is insulating spin-liquid without any long
range magnetic order.

In contrast to the above studies which considered wave functions with
different symmetries, here we study the ground state within the single
variational wave function. We show that, though no magnetic order is put
explicitly into the CRBM+BCS wave function, it spontaneously gives rise to
long range AF order in the insulating state. This is remarkable as this was
not observed with any of the other variational WFs considered in the previous
studies. In order to characterize the ground state, first we compute the
doublon-holon (DH) correlation function defined as,

$\displaystyle C_{ij}(r)=\frac{\langle d_{i}h_{j}\rangle-\langle
d_{i}\rangle\langle h_{j}\rangle}{\langle d_{i}\rangle\langle h_{j}\rangle}$
(12)

where $d_{i}=n_{i\uparrow}n_{i\downarrow}$ and
$h_{i}=(1-n_{i\uparrow})(1-n_{i\downarrow})$ are the doublon and holon
operator, respectively. The results for the nearest-neighbor (NN) and next-
nearest-neighbor (NNN) correlations as a function of $U$ are shown in Fig. 6.

(a) (b)

Figure 6: Doublon-holon correlations at (a) nearest-neighbor and (b) next-
nearest-neighbor distances as a function of $U/t$. Results are shown for two
values of $t^{\prime}/t$.

It shows that $C_{ij}(r)$ is very close to zero at small $U/t$. As $U/t$ is
increased, it shows a jump at a critical $U_{c}/t$ and steadily rises after.
The NNN values are an order of magnitude smaller than the NN values,
indicating the bindings of doublons and holons within short distances from
each other. Thus the wave function is able to capture the correct physics of
doulon-holon bindings in the strong coupling regime in spite of it not having
any explicit doublon-holon correlation factor. Next, we compute the double
occupancy, $D$ defined as

$\displaystyle D=\frac{1}{N_{\mathrm{s}}}\sum_{i}n_{i\uparrow}n_{i\downarrow}$
(13)

It is a crucial quantity that can indicate the presence of Mott transition.
The value of $D$ is shown in Fig. 7.

Figure 7: Double occupancy $D$ as a function of $U/t$ at the two values of
$t^{\prime}$ shown.

Starting from a value of $0.25$ at $U/t=0$, $D$ decreases as $U/t$ is
increased. It shows a sudden drop at a critical $U_{c}/t$ indicating the onset
of Mott transition. The jump in the value of $D$ is clearly present at both
the values of $t^{\prime}$, though it is sharper in the frustrated case. It
may be mentioned that, even in the long ranged backflow correlated WF used in
previous studiesTocchio  _et al._ (2011), the $D$ at Mott transition at
$t^{\prime}/t=0$ shows only a kink not a jump. We find that the values of
critical interaction are $U_{c}/t\sim 5.4$ for $t^{\prime}/t=0$ and
$U_{c}/t\sim 6.6$ for $t^{\prime}/t=0.5$. These values are in roughly in good
agreement with previous resultsYokoyama  _et al._ (2006); Tocchio  _et al._
(2008). The occurence of Mott transition can be confirmed directly by looking
at the charge gap which can be estimated from the knowledge of the ground
state WF itself. Given the ground state $\ket{\Psi}$, an excited state with
momentum $\mathbf{q}$ is given by $n_{\mathbf{q}}\ket{\Psi}$, and the charge
gap in limit $q\rightarrow 0$ for square lattice can be shown to beTocchio
_et al._ (2011),

$\displaystyle E_{g}=-\frac{1}{4}\left(\lim_{q\rightarrow
0}\frac{|\mathbf{q}|^{2}}{N(\mathbf{q})}\right)\left({\cal K}_{1}+2{\cal
K}_{2}\right)$ (14)

where $N(\mathbf{q})=\langle n_{\mathbf{q}}n_{-\mathbf{q}}\rangle$ is the
charge structure factor. ${\cal K}_{1}$ and ${\cal K}_{2}$ are the NN and the
NNN kinetic energy per site, respectively. The results for $E_{g}$ are shown
in Fig. 8.

Figure 8: Charge gap $E_{g}$ as a function of $U/t$.

It confirms that the state below $U_{c}/t$ is metallic with no charge gap,
while above $U_{c}/t$ it is insulating with a finite charge gap. Next we look
at the momentum distribution function, $n(\mathbf{k})=\langle
c^{\dagger}_{k\sigma}c_{k\sigma}\rangle$. The $n(\mathbf{k})$ values
calculated as a function of $\mathbf{k}$ along the symmetry path
$\Gamma(0,0)$-$X(\pi,0)$-$M(\pi,\pi)$-$\Gamma(0,0)$ for different values of
$U/t$ for the case $t^{\prime}/t=0$ are shown in Fig. 9(a). In the metallic
state, $n(\mathbf{k})$ has a discontinuity at the Fermi surface, $k=k_{F}$ in
the nodal $\Gamma-M$ direction. The magnitude of the jump gives the
quasiparticle weight $Z$ which roughly corresponds to the inverse effective
mass of the quasiparticlesParamekanti  _et al._ (2004); Yokoyama  _et al._
(2006). We plot the values of $Z$ so determined as a function of $U/t$ in Fig.
9(a). For both the cases of $t^{\prime}/t$, $Z$ decreases with increasing
$U/t$ and show a sudden drop at the respective critical interaction $U_{c}/t$.
Interestingly, $Z$ does not vanish completely in the Mott state for the
unfrustrated case at $t^{\prime}/t=0$, though it becomes very small. Thus it
suggests that the Mott transition in this case takes place via vanishing of
spectral weight at the Fermi level rather than via divergence of effective
mass. In contrast, $Z$ vanishes completely in the Mott state for
$t^{\prime}/t=0.5$ case suggesting divergence of the effective mass in this
case. For both the cases, the transition is found to be first-order nature as
evidenced from discontinuities of various quantities at the critical point.

(a) (b)

Figure 9: (a) Momentum distribution at various values of $U/t$ along the
symmetry path in the first Brillouin zone ($t^{\prime}/t=0$). (b)
Quasiparticle weight $Z$ determined from the jump at the Fermi surface, as a
function of $U/t$, for the two cases of $t^{\prime}/t$.

Finally, we look at the magnetic correlations in the CRBM wave function. We
calculate the spin structure factor $S(\mathbf{q})$ defined as,

$\displaystyle
S(\mathbf{q})=\frac{1}{L^{2}}\sum_{i,j}e^{i\mathbf{q}\cdot(\mathbf{r}_{i}-\mathbf{r}_{j})}\langle
S^{z}_{i}S^{z}_{j}\rangle$ (15)

where $S^{z}_{i}=(n_{i\uparrow}-n_{i\downarrow})$ is the $z$-component of the
spin operator at site $i$. The results for $S(\mathbf{q})$ calculated at
various values of $U/t$ are shown in Fig. 10.

(a) (b)

Figure 10: Spin structure factor $S(\mathbf{q})$ as a function of $\mathbf{q}$
at various values of $U/t$ for the cases (a) $t^{\prime}/t=0$ and (b)
$t^{\prime}/t=0.5$. The inset in each case show $S(\mathbf{q})$ versus
$\mathbf{q}$ for small $|\mathbf{q}|$. It shows that in the limit
$|\mathbf{q}|\rightarrow 0$, $S(\mathbf{q})\propto|\mathbf{q}|$ for $U<U_{c}$
(gapless), while $S(\mathbf{q})\propto|\mathbf{q}|^{2}$ for $U>U_{c}$ (spin
gap).

As the figure shows, $S(\mathbf{q})$ is very small at all $\mathbf{q}$ for
$U<U_{c}$ indicating very weak magnetic correlations in the metallic state.
For $U>U_{c}$, $S(\mathbf{q})$ is very sharply peaked at
$\mathbf{q}=(\pi,\pi)$ which indicates onset on long range AF correlations as
soon as the system enters the Mott state. The non-zero values of $t^{\prime}$
considered here does not seem have any impact on the AF correlations. In fact,
we find that the sublattice magnetization $m=|S^{z}_{i}|$ which is close zero
in the metallic phase, jumps to around $0.8$ at transition, a value close to
the saturation limit. If we compare with results in Ref. Yokoyama  _et al._ ,
2006, the AF correlations in the insulating state in the GW+DH wave function
used in this study is much weaker compared to what is found here. The insets
in the figures show $S(\mathbf{q})$ for a small range of $|\mathbf{q}|$.
Clearly in the limit $|\mathbf{q}|\rightarrow 0$, the
$S(\mathbf{q})\propto|\mathbf{q}|$ in the metallic phase suggesting absence of
spin gap in this phase. On the other hand,
$S(\mathbf{q})\propto|\mathbf{q}|^{2}$ for $U>U_{c}$ suggesting that the Mott
state is also spin gapped.

##  IV Conclusion

In summary, we have studied the ground state phase of the half-filled Hubbard
model on a square lattice using a variational wave function which is
constructed by applying a convolutional restricted Boltzmann machine (CRBM)
neural network as correlator to a mean-field BCS wave function. The number of
variational parameters in the wave function does not automatically grow with
the lattice size and can be tuned. The wave function is also highly accurate,
especially in the strong coupling limit where it yields variational energies
lower than those of the best known Jastrow variational WFs for the model. The
picture of Mott metal-insulator transition in the model as described by the
CRBM wave function is roughly similar to that obtained by using other
variational wave functionsYokoyama  _et al._ (2006); Tocchio  _et al._ (2008,
2011), with some interesting differences. Regarding the shortcomings of the
CRBM wave function, the results for the energies shows that the it does not
necessarily perform better in the weak coupling limit. Other more accurate
methods strongly suggest that for the unfrustrated model with
$t^{\prime}/t=0$, there exist short range AF fluctuations even in the weak
coupling limit and the ground state in this case is insulating at all
$U_{c}/t>0$Schäfer  _et al._ (2015); LeBlanc  _et al._ (2015); Mukherjee and
Lal (2020). This physics is not captured correctly in the CRBM wave function.
This of course can be remedied readily by putting an AF order manually into
the mean-field part of the wave function, though it would be much desirable to
have the correlations generated spontaneously in the same manner as in the
Mott insulating state. It is also interesting to study the wave function for a
range of $t^{\prime}/t$ values in order to obtain a full phase diagram as a
function of $U/t$ and $t^{\prime}/t$. We leave it for a future study.

## Acknowledgement

The authors thank the Science and Engineering Research Board, DST, Govt of
India for financial support under the Core grant (No: CRG/2021/005792). Also
acknowledge CHPC, IISER Thiruvananthapuram for computational facilities.

## References

  * Imada  _et al._ (1998) M. Imada, A. Fujimori, and Y. Tokura, Rev. Mod. Phys. 70, 1039 (1998).
  * LeBlanc  _et al._ (2015) J. P. F. LeBlanc, A. E. Antipov, F. Becca, I. W. Bulik, G. K.-L. Chan, C.-M. Chung, Y. Deng, M. Ferrero, T. M. Henderson, C. A. Jiménez-Hoyos, E. Kozik, X.-W. Liu, A. J. Millis, N. V. Prokof’ev, M. Qin, G. E. Scuseria, H. Shi, B. V. Svistunov, L. F. Tocchio, I. S. Tupitsyn, S. R. White, S. Zhang, B.-X. Zheng, Z. Zhu, and E. Gull (Simons Collaboration on the Many-Electron Problem), Phys. Rev. X 5, 041041 (2015).
  * Lee  _et al._ (2006a) P. A. Lee, N. Nagaosa, and X.-G. Wen, Rev. Mod. Phys. 78, 17 (2006a).
  * Ceperley  _et al._ (1977) D. Ceperley, G. V. Chester, and M. H. Kalos, Phys. Rev. B 16, 3081 (1977).
  * Tahara and Imada (2008) D. Tahara and M. Imada, Journal of the Physical Society of Japan 77, 114701 (2008), https://doi.org/10.1143/JPSJ.77.114701 .
  * Sorella (2013) S. Sorella, in  _Strongly Correlated Systems_, Springer Series in Solid-State Sciences, Vol. 176, edited by A. Avella and F. Mancini (Springer, Berlin, Heidelberg, 2013) Chap. Variational Monte Carlo and Markov Chains for Computational Physics.
  * Paramekanti  _et al._ (2004) A. Paramekanti, M. Randeria, and N. Trivedi, Phys. Rev. B 70, 054504 (2004).
  * SHIBA and YOKOYAMA (1987) H. SHIBA and H. YOKOYAMA, in  _Proceedings of the Yamada Conference XVIII on Superconductivity in Highly Correlated Fermion Systems_, edited by M. Tachiki, Y. Muto, and S. Maekawa (Elsevier, 1987) pp. 264–267.
  * Carrasquilla and Melko (2017) J. Carrasquilla and R. G. Melko, Nature Physics 13, 431–434 (2017).
  * Carleo and Troyer (2017) G. Carleo and M. Troyer, Science 355, 602–606 (2017).
  * van Nieuwenburg  _et al._ (2017) E. P. L. van Nieuwenburg, Y.-H. Liu, and S. D. Huber, Nature Physics 13, 435–439 (2017).
  * Broecker  _et al._ (2017) P. Broecker, J. Carrasquilla, R. G. Melko, and S. Trebst, Scientific Reports 7, 8823 (2017).
  * Ch’ng  _et al._ (2017) K. Ch’ng, J. Carrasquilla, R. G. Melko, and E. Khatami, Phys. Rev. X 7, 031038 (2017).
  * Deng  _et al._ (2017) D.-L. Deng, X. Li, and S. Das Sarma, Phys. Rev. X 7, 021021 (2017).
  * Carleo  _et al._ (2019) G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby, L. Vogt-Maranto, and L. Zdeborová, Rev. Mod. Phys. 91, 045002 (2019).
  * Carrasquilla (2020) J. Carrasquilla, Advances in Physics: X 5, 1797528 (2020), https://doi.org/10.1080/23746149.2020.1797528 .
  * Melko  _et al._ (2019) R. G. Melko, G. Carleo, J. Carrasquilla, and J. I. Cirac, Nature Physics 15, 887 (2019).
  * Carrasquilla and Torlai (2021) J. Carrasquilla and G. Torlai, PRX Quantum 2, 040201 (2021).
  * Torlai  _et al._ (2018) G. Torlai, G. Mazzola, J. Carrasquilla, M. Troyer, R. Melko, and G. Carleo, Nature Physics 14, 447–450 (2018).
  * Carleo  _et al._ (2018) G. Carleo, Y. Nomura, and M. Imada, Nature Communications 9, 5322 (2018).
  * Choo  _et al._ (2018) K. Choo, G. Carleo, N. Regnault, and T. Neupert, Phys. Rev. Lett. 121, 167204 (2018).
  * Lu  _et al._ (2019) S. Lu, X. Gao, and L.-M. Duan, Phys. Rev. B 99, 155136 (2019).
  * Glasser  _et al._ (2018) I. Glasser, N. Pancotti, M. August, I. D. Rodriguez, and J. I. Cirac, Phys. Rev. X 8, 011006 (2018).
  * Zhang and Weng (2014) L. Zhang and Z.-Y. Weng, Phys. Rev. B 90, 165120 (2014).
  * Cai and Liu (2018) Z. Cai and J. Liu, Phys. Rev. B 97, 035116 (2018).
  * Nomura  _et al._ (2017) Y. Nomura, A. S. Darmawan, Y. Yamaji, and M. Imada, Phys. Rev. B 96, 205152 (2017).
  * Inui  _et al._ (2021) K. Inui, Y. Kato, and Y. Motome, Phys. Rev. Res. 3, 043126 (2021).
  * Luo and Clark (2019) D. Luo and B. K. Clark, Phys. Rev. Lett. 122, 226401 (2019).
  * Pfau  _et al._ (2020) D. Pfau, J. S. Spencer, A. G. D. G. Matthews, and W. M. C. Foulkes, Phys. Rev. Res. 2, 033429 (2020).
  * MOTT (1968) N. F. MOTT, Rev. Mod. Phys. 40, 677 (1968).
  * Orenstein and Millis (2000) J. Orenstein and A. J. Millis, Science 288, 468 (2000).
  * Lee  _et al._ (2006b) P. A. Lee, N. Nagaosa, and X.-G. Wen, Rev. Mod. Phys. 78, 17 (2006b).
  * Georges  _et al._ (1996) A. Georges, G. Kotliar, W. Krauth, and M. J. Rozenberg, Rev. Mod. Phys. 68, 13 (1996).
  * Zhang  _et al._ (1993) X. Y. Zhang, M. J. Rozenberg, and G. Kotliar, Phys. Rev. Lett. 70, 1666 (1993).
  * Rozenberg  _et al._ (1994) M. J. Rozenberg, G. Kotliar, and X. Y. Zhang, Phys. Rev. B 49, 10181 (1994).
  * Park  _et al._ (2008) H. Park, K. Haule, and G. Kotliar, Phys. Rev. Lett. 101, 186403 (2008).
  * Rozenberg  _et al._ (1999) M. J. Rozenberg, R. Chitra, and G. Kotliar, Phys. Rev. Lett. 83, 3498 (1999).
  * Bulla  _et al._ (2001) R. Bulla, T. A. Costi, and D. Vollhardt, Phys. Rev. B 64, 045103 (2001).
  * Lu (1994) J. P. Lu, Phys. Rev. B 49, 5687 (1994).
  * Ferrero  _et al._ (2005) M. Ferrero, F. Becca, M. Fabrizio, and M. Capone, Phys. Rev. B 72, 205126 (2005).
  * Rüegg  _et al._ (2005) A. Rüegg, M. Indergand, S. Pilgram, and M. Sigrist, Eur. Phys. J. B 48, 55 (2005).
  * Mukherjee and Lal (2020) A. Mukherjee and S. Lal, New Journal of Physics 22, 063007 (2020).
  * Yokoyama and Shiba (1990) H. Yokoyama and H. Shiba, Journal of the Physical Society of Japan 59, 3669 (1990), https://doi.org/10.1143/JPSJ.59.3669 .
  * Yokoyama  _et al._ (2006) H. Yokoyama, M. Ogata, and Y. Tanaka, Journal of the Physical Society of Japan 75, 114706 (2006).
  * Capello  _et al._ (2005) M. Capello, F. Becca, M. Fabrizio, S. Sorella, and E. Tosatti, Phys. Rev. Lett. 94, 026406 (2005).
  * Capello  _et al._ (2006) M. Capello, F. Becca, S. Yunoki, and S. Sorella, Phys. Rev. B 73, 245116 (2006).
  * Tocchio  _et al._ (2008) L. F. Tocchio, F. Becca, A. Parola, and S. Sorella, Phys. Rev. B 78, 041101 (2008).
  * Tocchio  _et al._ (2011) L. F. Tocchio, F. Becca, and C. Gros, Phys. Rev. B 83, 195138 (2011).
  * Kaplan  _et al._ (1982) T. A. Kaplan, P. Horsch, and P. Fulde, Phys. Rev. Lett. 49, 889 (1982).
  * Norouzi (2009) M. Norouzi,  _Convolutional restricted boltzmann machines for feature learning_ , Master’s thesis, School of Computing Science, Simon Fraser University (2009).
  * Alcalde Puente and Eremin (2020) D. Alcalde Puente and I. M. Eremin, Phys. Rev. B 102, 195148 (2020).
  * Schäfer  _et al._ (2015) T. Schäfer, F. Geles, D. Rost, G. Rohringer, E. Arrigoni, K. Held, N. Blümer, M. Aichhorn, and A. Toschi, Phys. Rev. B 91, 125109 (2015).