# Relation Rectification in Diffusion Model

Yinwei Wu Xingyi Yang Xinchao Wang111* Corresponding Author
National University of Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Despite their exceptional generative abilities, large T2I diffusion models,
much like skilled but careless artists, often struggle with accurately
depicting visual relationships between objects. This issue, as we uncover
through careful analysis, arises from a misaligned text encoder that struggles
to interpret specific relationships and differentiate the logical order of
associated objects. To resolve this, we introduce a novel task termed Relation
Rectification, aiming to refine the model to accurately represent a given
relationship it initially fails to generate. To address this, we propose an
innovative solution utilizing a Heterogeneous Graph Convolutional Network
(HGCN). It models the directional relationships between relation terms and
corresponding objects within the input prompts. Specifically, we optimize the
HGCN on a pair of prompts with identical relational words but reversed object
orders, supplemented by a few reference images. The lightweight HGCN adjusts
the text embeddings generated by the text encoder, ensuring accurate
reflection of the textual relation in the embedding space. Crucially, our
method retains the parameters of the text encoder and diffusion model,
preserving the model’s robust performance on unrelated descriptions. We
validated our approach on a newly curated dataset of diverse relational data,
demonstrating both quantitative and qualitative enhancements in generating
images with precise visual relations. Project page: https://wuyinwei-
hah.github.io/rrnet.github.io/ .

Figure 1: Results Visualization of Relation Rectification. (a) Our approach
enables diffusion model to successfully generate images with the correct
directional relation in response to the textual prompt, which they originally
failed. (b) Our method can synthesize relation of diverse and unseen objects
in zero-shot manner.

##  1 Introduction

Diffusion-based text-to-image (T2I) models[14, 32] have set new benchmarks in
the synthesis of images from textual descriptions, achieving remarkable
fidelity and detail. Nevertheless, a nontrivial gap persists in their ability
to interpret and visually articulate the interacting objects within a given
prompt, especially when the description includes directional or relational
terms. For instance, a prompt such as “a book is placed on a bowl” frequently
leads to a visual representation where the intended directionality of the
interaction is misconstrued, resulting in scene akin to “a bowl is placed on a
book”. This indicates a crucial limitation in the model’s relational
understanding, a pivotal aspect of cognitive comprehension that remains to be
fully integrated into the generated images.

The problem of relation mis-interpretion is not unique to diffusion models;
indeed, it is endemic to the broader class of Vision-Language Models (VLMs)
[50]. The crux of this issue stems from the standard contrastive or mask-
filling training of VLMs [41, 12, 17], which prioritize global semantics but
fails to capture correct relationships between objects. Consequently, these
models, especially diffusion-based T2I models, often understand texts as “Bag-
of words” [50], neglecting the compositional semantics necessary for accurate
image generation.

In response, some researchers have incorporated auxiliary input, such as
canvas layouts, to guide image synthesis [52, 47]. However, such interventions
circumvent rather than resolve the primary limitation: the text encoder’s
inadequate response to the directionality of textual relations. Addressing
this fundamental issue remains a pivotal concern for advancing the field.

To address the issue, we introduce a new task named Relation Rectification.
Given a pair of prompts that describe the same relation but with the positions
of the objects reversed (for instance, “The bowl is inside the cloth” vs. “The
cloth is inside the bowl”), termed object-swapped prompts (OSPs). The task
aims to allow the model to give different responses to OSPs according to the
differences in object relationships, rather than simplifying prompts to just
”Bags-of-words”.

Upon investigating the inner workings of the diffusion model, we found that
the embedding of the special token $[EOT]$, signifying the end of text, plays
a pivotal role inguiding the generation of relationships. We further
identified a critical issue: the embeddings of $[EOT]$ generated from OSPs are
nearly identical, rendering the directionality of the relations
indistinguishable.

To address this, we introduce RRNet, a novel framework designed to augment the
relation understanding of diffusion models like Stable Diffusion (SD). The
essence of RRNet is to explicitly encode the directional aspect of
relationships within a sentence. Specifically, we conceptualize OSPs as pairs
of _directed heterogeneous graphs_. To process these graphs, we utilize a
Heterogeneous Graph Convolutional Network (HGCN), which generates adjustment
vectors to distinctly separate the $[EOT]$ embeddings of OSPs. During
training, we update the lightweight HGCN only, while maintaining the
parameters of SD to be fixed.

To evaluate the efficacy of our approach, we compiled Relation Rectification
Benchmark, a new dataset for evaluation and rigorously test RRNet across a
spectrum of relationships. The experimental results indicate that despite a
minor decrease in image fidelity, RRNet enhances the accuracy of SD’s
relationship generation by up to 25Additionally, our method significantly
enhances interpretability, clearly depicting the directional transitions in
relationships. Furthermore, RRNet demonstrates robust generalization
capabilities, effectively handling even unseen objects in the dataset. This
comprehensive testing underscores RRNet’s potential in improving relationship
interpretation in image generation tasks.

Our contributions are summarized as belows:

  * •

We introduce the novel task of Relation Rectification, focused on enhancing
SD’s capability to accurately generate images that reflect the directional
relationships outlined in text prompts.

  * •

We identify that the primary limitation of vanilla SD in relation
rectification arises from the indistinguishable text embeddings of OSPs.

  * •

We proposed RRNet, a HGCN based model that designed to aid SD in accurately
generating images with directional relationships. Our approach requires only
the training of a lightweight HGCN, and can effectively address the task of
relation rectification.

  * •

We contribute the Relation Rectification Benchmark, serving as a valuable
evaluation tool for assessing the effectiveness of relation rectification
methods.

##  2 Related work

Diffusion Models. Diffusion models [32, 14, 5, 6, 45] view image generation as
process of gradual denoising from isotropic noise. Recent advances have seen
diffusion-based models reach the forefront in the field of T2I generation [10,
37, 39, 35, 33, 48]. Notably, the Stable Diffusion (SD) [37] operates by
denoising within the latent space, conditioned on the text embeddings from
pre-trained text encoders [34]. However, text embeddings generated from
sentences that contain directional relations are often inaccurate, leading to
difficulties for SD in accurately generating the relationships. Our method
corrects these inaccurate embeddings. By doing so, it allows SD to more
accurately capture relationship within text, leading to better image quality.

Personalized Diffusion. Personalized diffusion focuses on creating images that
align with specific, personalized visual concepts [18, 23, 8, 38] by tuning
the general-purpose diffusion model. Leveraging off-the-shelf model tuning
techniques [16, 49, 15], it modifies aspects such as text embeddings or
attention patterns within the diffusion process. These modifications guide the
diffusion to achieve a customized image generation. Unlike other methods that
concentrate on generating particular visual content, we focus on improving the
model’s ability to generate precise visual relationships by adjusting the
embeddings.

Vision-Language Models. Vision-Language Models (VLMs) [34, 24, 19] aim to
learn a unified cross-modality representation space between image and text by
pre-training on a large number of image-text pairs. For example, CLIP [34]
learned text-image shared representations by using contrast learning on large-
scale image-text pairs. However, recent work [50] suggests that the
contrastive objective used to train CLIP does not explicitly encourage it to
learn sentence order information, leading to CLIP’s tendency to interpret
sentences as Bags-of-Words. Our work not only identifies the impact of this
characteristic on T2I diffusion models which using CLIP as a text encoder, but
also proposes an effective solution to mitigate the negative effects it
introduces.

Compositional Image Generation. Compositional Image Generation aims to enable
generative models to generate images from prompts that describe the
composition of certain concepts. Composable Diffusion [29] utilizes multiple
diffusion models to control the generation of different concepts, thereby
achieving precise object positioning. GLiGEN [26] and ControlNet [52] guide
compositional generation by providing additional multimodal conditions for
training. Another branch of works [47, 7, 3, 2] modify the cross-attention
layout within the diffusion model to control object generation positions in a
training-free manner. However, previous research primarily examines simple [9]
spatial relationships. Our work delves into more intricate spatial and action-
based relationships.

Graph Convolutional Network. Graphs differ fundamentally from image and text
data. They comprise nodes and edges, with edges representing the relationships
between nodes. Recent advancements in Graph Convolutional Networks (GCNs) [22,
11, 30] have significantly improved graph learning tasks. GCNs have been
successfully applied in diverse domains [36, 25, 20, 46]. Heterogeneous graphs
represent a distinct category of graphs, and they are capable of depicting
various types of nodes and their interrelations. To manage this particular
type of graph, Heterogeneous GCNs (HGCN) [44, 51] are proposed. More
specifically, HGCNs adopt various functions to handle information emanating
from various types of nodes. In our work, we introduce HGCN to tackle the
challenge posed by the text encoder of SD, specifically its insensitivity to
the directional relations within sentences.

##  3 Approach

###  3.1 Preliminaries

Figure 2: Effect of masking out text embeddings corresponding to different
tokens. We found that masking out the embedding of $[EOT]$ dramatically
destroy the semantic of generated images, including relationships, whereas
masking out the embeddings corresponds to words results in only marginal
changes.

Text-to-Image Diffusion Models and its Text Encoder. We apply our method over
a pretrained T2I SD. Given an initial Gaussian noise $z\sim\mathcal{N}(0,I)$
and a textual prompt $y$, the SD can generate an image that matches the prompt
by stepwise denoising. The SD is trained by minimizing LDM loss as follows:

$\mathcal{L}_{LDM}=\mathbb{E}_{z\sim\mathcal{N}(0,I),y,t}[||\epsilon-\epsilon_{\theta}(z_{t},t,c(y))||^{2}_{2}],$
(1)

where the $\epsilon_{\theta}$ is parameterized as a U-Net. At each timestep
$t$, the denoising network $\epsilon_{\theta}$ aims at predicting the noise
$\epsilon$ added to the latent code. The textual prompt $y$ that describes the
final image content guides the denoising process. Before provided to the SD,
the prompt is encoded by a text encoder $c$, to produce text embeddings
$c(y)\in\mathbb{R}^{K\times d}$, where $K$ represents the number of tokens and
$d$ the token embedding dimension. Typically, the text embeddings $c(y)$
generated by $c$ have a length $K$ of 77. In addition to embeddings
representing each word in $y$, embeddings of special tokens are also included.
Notably, the special token $[EOT]$ marks the end of Text, and its embedding
(denoted as $V_{eot}$) is used to represent semantic information for the
entire sentence during the contrast learning of $c$.

Key Finding: $V_{eot}$ controls the relation. In addressing our specific
problem, we identified that the $V_{eot}$ vector plays a pivotal role in
controlling the relationships and semantics in generated images. Our
experiments, as shown in Figure 2, revealed a critical observation: when
$V_{eot}$ is masked out, SD struggles to generate images that accurately
depict valid relationships. This finding underscores the heavy reliance of SD
on $V_{eot}$ for relation generation. We hypothesize that this is due to
$V_{eot}$ accumulating rich semantics from other tokens, including both object
and relational information, thereby serving as a crucial component in
relationship generation. Further, we observed that the $V_{eot}$ vectors of
OSPs have a cosine similarity close to 1, indicating they are nearly
indistinguishable to SD.

Based on these findings, we plan to differentiate the $V_{eot}$ vectors of
OSPs to enhance SD’s relationship generation accuracy.

Graph Convolutional Network. GCN excels in processing graph data, A graph can
be defined as a tuple of $G=(\mathcal{V},\mathcal{E})$, $\mathcal{V}$
represents for nodes set and $\mathcal{E}$ is the edges set. For the directed
graph used in our work, each directed edge $e_{i,j}\in\mathcal{E}$ connects
node $v_{i}$ to node $v_{j}$.

GCN updates a node’s representation by aggregating information from its
neighbors. For a GCN comprising $L$ convolutional layers, the update formula
for the node representation of the $l$ th layer can be expressed as follows:

$h_{i}^{(l+1)}=\sigma(b^{(l)}+\sum_{j:(e_{j,i})\in\mathcal{E}}\alpha_{j,i}h_{j}^{(l)}W^{(l)}),$
(2)

where $h^{(l)}$ is the representation of the node at layer $l$. $\alpha_{j,i}$
denotes the edge weight. $W$ and $b$ are learnable parameters and $\sigma$ is
the activation function.

To differentiate the $V_{eot}$ of OSPs, we use graphs to represent the
directions of the relations in sentences. Our approach utilizes a
heterogeneous graph to model diverse information as different node types.
Objects and relations are distinct node types,
$\mathcal{V}_{O},\mathcal{V}_{R}\in\mathcal{V}$. Their information will be
aggregated into $\mathcal{V}_{\Delta EOT}$, which is responsible for learning
adjustment vectors to separate the $V_{eot}$ of OSPs.

In our HGCN, the weight matrix $W$ varies by node types, allowing specific
aggregation for each information type.

###  3.2 Problem Definition

Figure 3: Our RRNet Architecture. Given OSPs and their exemplar images, the
RRNet learns to produce adjustment vectors $h_{\Delta EOT}^{(L)}$, which will
then be added on original $V_{eot}$ of the prompts. The rectified embeddings
then will used as the condition to guidance the generation process of a frozen
SD. The upper left part is the heterogeneous graph RRNet uses to model the
relation direction. Upon optimization with negative loss and denoising loss,
the SD will be able to generate images with correct relation direction.

In our paper, we define _relation rectification_ as the task of enabling T2I
SD to generate images that more accurately represent the described
relationships of OSPs. For a prompts $y$ contains directional relationship, we
aim to adjust its text embeddings $c(y)$ using an additional model $\phi$ such
that:

$\arg\max_{\phi}\,P(x|\phi(c(y)))-P(\widetilde{x}|\phi(c(y)))$ (3)

where $P$ is the generating distribution of SD, and $x$ and $\widetilde{x}$
are two sets of images describe by $y$ and its object-swapped counterpart
$\widetilde{y}$. Achieving this objective requires two key criteria:

C1. The text embeddings of OSPs must be distinguishable after adjustment.

C2. The adjusted embeddings should remain valid for SD’s input domain,
effectively guiding the generation of the correct relational direction.

As such, we give our solution to each criterion.

Solution to C1. We introduce the Relation Rectification Net (RRNet), which
conceptualizes OSPs as heterogeneous graphs to capture the directional
relations. RRNet uses these graphs to produce adjustment vectors that separate
the embeddings of OSPs. We discuss RRNet in Sec 3.2.1.

Solution to C2. We utilize SD’s capability to understand visual semantics,
guiding RRNet to create effective adjustment vectors. By analyzing example
images corresponding to OSPs, SD discerns various relational directions,
imparting this knowledge to RRNet for modifying text embeddings. More details
are discussed in Sec 3.2.2.

####  3.2.1 Relation Rectification Net

As we have identified, the relationship information for SD is predominantly
encoded within the $V_{eot}$. To effectively rectify and optimize this vector
for accurate relation generation, we’ve developed a model that incorporates a
HGCN. This HGCN is specifically engineered to process heterogeneous graphs
derived from OSPs, subsequently outputting adjustment vectors that refine the
$V_{eot}$.

Heterogeneous Graph Construction. A pair of OSPs contain objects $A,B$ and a
relationship $R$ conceptualized as two distinct triplets: $<A,R,B>$ and
$<B,R,A>$. It is important to note that $<A,R,B>\neq<B,R,A>$ due to the
directional nature of $R$. These triplets are directly modeled as directed
graphs, for example $<A,R,B>$ as a graph with directed edges $A\rightarrow R$
and $R\rightarrow B$.

Considering the different semantics of object and relation nodes, we employ
heterogeneous graphs. Here, objects and relations are represented as two
distinct node types, $v_{r}\in\mathcal{V}_{R}$ for relations and and
$v_{o}\in\mathcal{V}_{O}$ for objects.

In addition to $\mathcal{V}_{R}$ and $\mathcal{V}_{O}$, a third kind of nodes,
$\mathcal{V}_{\Delta EOT}$, is utilized to learn the adjustment vector
$h_{\Delta EOT}$. The learned $h_{\Delta EOT}$ will be used to adjust the
relation direction information in original $V_{eot}$ of OSPs.

Relation Adjustment. With the graph topology established, we initiate the node
representations. For object nodes $\mathcal{V}_{O}$ and relation nodes
$\mathcal{V}_{R}$, we employ CLIP’s word embeddings, rich in semantic content,
as their initial representations. Node $\mathcal{V}_{\Delta EOT}$ is
initialized randomly.

In the HGCN layers, the information from $\mathcal{V}_{O}$ and
$\mathcal{V}_{R}$ aggregates into $\mathcal{V}_{\Delta EOT}$ along edges. The
update formula for $\mathcal{V}_{\Delta EOT}$ is:

$h_{\Delta
EOT}^{(l+1)}=\sum_{\mathcal{E}\in\hat{\mathcal{\mathcal{E}}}}\sigma(b^{(l)}+\sum_{j:(e_{j,i})\in\mathcal{E}}\alpha_{j,i}h_{j}^{(l)}W_{\mathcal{E}}^{(l)}),$
(4)

where $\hat{\mathcal{E}}$ is the set of all edge types, and
$W_{\mathcal{E}}^{(l)}$ denotes the weight of the edge of type $\mathcal{E}$.
After several convolution layers, relation direction and node representations
of $A,R$ and $B$ merge into $\mathcal{V}_{\Delta EOT}$. As illustrated in the
bottom left part of Figure 3, the final adjusted $\overset{*}{V_{eot}}$ is
obtain via:

$\overset{*}{V_{eot}}=V_{eot}+\lambda\cdot h_{\Delta EOT}^{(L)},$ (5)

here the $\lambda\in[0,1]$ moderates the adjustment strength. By adjusting the
value of $\lambda$, a trade-off between image quality and relation generation
accuracy can be made. This additive operation is inspired by previous works
[4, 42] on concept algebraic in VLMs’ embedding space.

Object Node Disentanglement. Our objective extends beyond just memorizing
relationships; we aim to separate the features of individual objects,
represented by nodes $A$ and $B$, from the relationship $R$. This separation
is essential to prevent the blending of object features with relations.

To achieve this, as depicted in Figure 3, RRNet includes a dedicated node,
$v_{\Delta EOT}^{A}\in\mathcal{V}_{\Delta EOT}$ for object $A$. $V_{eot}^{A}$
is extracted from template sentence “This is a photo of {A}”. We aim to
preserve $A$ ’s accurate semantic in $V_{eot}^{A}$ after applying the
adjustments through equation 5 with $h_{\Delta EOT}^{A}$. The $h_{\Delta
EOT}^{A}$ is obtained from the final layer representation of $v_{\Delta
EOT}^{A}$, as depicted in equation 4. This method ensures the disentanglement
of node $A$ from node $B$ and relation $R$.

However, since $V_{eot}^{A}$ already aligns closely with object $A$’s semantic
and might not require adjustment, $h_{\Delta EOT}^{A}$ could potentially learn
a trivial solution, such as zero. To prevent this, we add Gaussian noise to
$V_{eot}^{A}$, so that the $h_{\Delta EOT}^{A}$ is forced to learning a valid
semantic vector.

For the disentanglement of object B, owing to the concurrent training of RRNet
from $<A,R,B>$ and $<B,R,A>$ directions, when training in the $<B,R,A>$
direction, B is also disentangled

####  3.2.2 Relation Compelling Losses

In our study, we have developed a relation compelling loss, consisting of both
_positive_ and _negative_ components, tailored to enhance SD’s interpretation
of relationships.

Positive Loss. Our goal is to guide SD in identifying text embeddings that
generate images with specific relationship semantics, as outlined in equation
3. To achieve this, we directly utilize the denoising loss as below:

$\begin{split}\mathcal{L}_{denoise}=\mathbb{E}_{z\sim\mathcal{N}(0,I),y,t}[||\epsilon-\epsilon_{\theta}(x_{t},t,\phi(c(y)))||^{2}_{2}],\end{split}$
(6)

where $\phi$ is the RRNet, and $\phi(c(y)))$ adjusts the text embeddings of
prompt $y$. The $x_{t}$ are exemplar images corresponding to $y$. Through
$\mathcal{L}_{denoise}$, RRNet learns to generate adjustment vectors that
align text embeddings of $y$ with the relation semantics in the exemplar
images.

Negative Loss. Solely relying on $\mathcal{L}_{denoise}$ risks the model
focusing on superficial features such as image background, rather than the
desired relations. To address this and ensure $V_{eot}$ separation in OSPs, we
introduce a negative loss:

$\begin{split}\mathcal{L}_{neg}=\mathbb{E}_{z\sim\mathcal{N}(0,I),\widetilde{y},t}[-||\epsilon-\epsilon_{\theta}(x_{t},t,\phi(c(\widetilde{y})))||^{2}_{2}],\end{split}$
(7)

where $\widetilde{y}$ is the OSP counterpart of $y$.

The final loss becomes:

$\begin{split}\mathcal{L}=\eta\cdot\mathcal{L}_{denoise}+\xi\cdot\mathcal{L}_{neg}\end{split}$
(8)

here, the $\eta$ and $\xi$ ere hyperparameters balancing the two losses.
Intuitively, $\mathcal{L}_{denoise}$ helps RRNet in discerning relation
semantics, while $L_{neg}$ mitigates unintended semantics, aiding in
separating $V_{eot}$ of OSPs.

During training, for each OSP pair represented as $<A,R,B>$ and $<B,R,A>$, our
dataset includes four types of image-text pairs: two types for the original
OSPs and two for disentangling the object node features. The computation of
loss for each image-text pair involves the incorporation of the other three
pairs using Equation 8.

Once trained, RRNet is capable of generating adjustment vectors that
effectively separate the $V_{eot}$ of OSPs, thus enhancing SD’s precision in
relationship generation.

##  4 Experiment

###  4.1 Experimental Setup

Dataset. For a comprehensive benchmarking, we compiled a dataset with 21
relationships, split into 8 positional (like below, on the left) and 13 action
(such as touch, follow) types. Each includes a pair of object-swapped prompts
(OSPs) and corresponding images. We generated 100 images per prompt, totaling
4200 images, to rigorously assess the accuracy of relation generation. For
detailed dataset statistics, please refer to Appendix D.

Figure 4: Qualitative Results. By increasing the weight $\lambda$ of the
adjustment vector, we show the process of correcting relation direction in the
generated images that were originally incorrect.

Implementation Details. We train RRNet on Stable Diffusion 2-1 for 100 epochs.
During training, $\lambda$ in Equation 5 is set to 1. For the loss in Equation
8, we found that setting the weight $\eta$ of denoising loss to 10 and weight
$\xi$ for negative loss to 2 works well for most cases. Training for each
relationship only takes about 20 minutes on 1 NVIDIA V100 GPU.

During generation, the weight $\lambda$ can be set within $[0,1]$. In our
experiments, we assess the generation outcomes using various values of
$\lambda$. For the denoising process, we use the PNDM scheduler [21] and adopt
30 steps. The classifier-free guidance is applied for better image quality.

Baselines. In the absence of existing approaches designed to generate correct
relations, we established our own baselines. The first baseline is SD itself.
Additionally, we compare our method with personalized diffusion models [23, 8,
38], where we optimize the CLIP text encoder within the SD model. To ensure a
fair comparison, we optimize the text encoder using the same loss function
that is applied in these personalized diffusion methods. More comparative
experiments are provided in the Appendix A.

Evaluation Metrics. We report two evaluation metrics

  * •

Relationship Generation Accuracy. To evaluate the accuracy of relationship
generation, we employ vision-language chatbots proficient in image semantics,
specifically Qwen-VL-Chat [1] and LLaVA [28]. The evaluation approach
involves: (1) Using a sentence representing triplet $<A,R,B>$ and a
corresponding image as a prompt. (2) Verifying the presence of entities A and
B in the image. (3) Guiding the chatbots to determine the relationship between
A and B. (4) Having the chatbots choose the most plausible relationship from
$<A,R,B>$, $<B,R,A>$, or $Neither$.

In steps (1) and (2) described above, chatbots perform the task of detecting
object generation, from which we derive the Object Generation Accuracy (OGA).
This metric is used to assess the accuracy of entity generation.

  * •

Fréchet Inception Distance (FID). FID [13] is used to evaluate the quality of
generated images. It can measure the feature similarity between generated and
real images via a pre-trained inception model [40]. We caculate the FID score
between 4,200 generated images and all images from validation set of COCO
[27].

Figure 5: Qualitative Comparisons. Our method outperforms the baselines in
terms of the relationship generation.

###  4.2 Comparisons

Method | Position(Qwen) $\uparrow$ | Position(LLaVA)$\uparrow$ | Action(Qwen) $\uparrow$ | Action(LLaVA)$\uparrow$ | OGA$\uparrow$ | FID$\downarrow$
---|---|---|---|---|---|---
RR Dataset | 0.849 | 0.763 | 0.616 | 0.652 | 1.000 | N/A
Stable Diffusion | 0.467 | 0.542 | 0.399 | 0.543 | 0.898 | 83.73
Personalized Diffusion | 0.509 | 0.558 | 0.273 | 0.518 | 0.844 | 91.93
RRNet ($\lambda$=0.2) | 0.564 | 0.597 | 0.469 | 0.565 | 0.937 | 89.13
RRNet ($\lambda$=0.4) | 0.646 | 0.648 | 0.492 | 0.603 | 0.964 | 94.73
RRNet ($\lambda$=0.6) | 0.697 | 0.684 | 0.500 | 0.632 | 0.970 | 100.78
RRNet ($\lambda$=1) | 0.729 | 0.724 | 0.490 | 0.651 | 0.971 | 110.01
Table 1: Quantitative Results of Positional Relation Generation. We evaluate
the generation accuracy through Qwen-VL-Chat and LLaVA chatbots, and the
generated image quality is evaluated via FID. The Position and Action
represent for evaluation on positional relationships part and action
relationships part of the dataset, respectively. Object generation accuracy
(OGA) is the average calculated from all position and action relations.

Quantitative analysis. Table 1 presents the quantitative results on positional
and action relationships generation. Our method obtains a better relationship
generation accuracy and object generation accuracy compared to baselines. By
increasing the adjustment weight $\lambda$ in equation 5, the RRNet can
achieve a higher generation accuracy.

However, it can be observed that a larger $\lambda$ could also leads to a
larger FID score, indicating lower image fidelity. To balance relationship
generation accuracy and image diversity, we discovered that setting $\lambda$
between 0.4 and 0.6 significantly improves relationship accuracy while
preserving image quality.

Qualitative analysis. Figure 4 and Figure 1(a) illustrate the interpretability
of our adjustment vector approach. As $\lambda$ increases, the generated
images progressively shift to the correct relationship direction. For example,
the second row of Figure 4 shows an astronaut transitioning from riding a
horse ($\lambda=0$) to eventually leading the horse ($\lambda=0.6$). This
progression visually illustrates the transition of relationships towards the
correct direction, demonstrating RRNet’s capability in rectifying relations.

In Figure 5, we compare our method against baselines. Original SD generations
often mix relationship directions, failing to distinguish them accurately. The
personalized diffusion approach, Personalized diffusion, while learning the
relationship, often alters object semantics. As observed in the bottom row, it
successfully learns the meaning of “In water”, but sacrifices the meaning of
“A bowl”, which in turn incorrectly generates cup and bucket similar to bowl.
Our approach, in contrast, correctly portrays relationship directions and
maintains the objects’ original meanings in the generated images.

###  4.3 User Study

Setup. We conducted a user study with 63 evaluators to evaluate relationship
generation accuracy. Ten relationships were chosen from our dataset, with
images generated by RRNet and baselines. For each, evaluators selected the
image that best depicted the described relationship from two randomly chosen
images per method.

Results. The outcomes of this user study are detailed in Table 3. We observe
that the generation of RRNet are more favored by evaluators, with an absolute
improvement of 68.1% on personalized diffusion and 58.58% on SD.

Method | Position(Qwen) $\uparrow$ | Position(LLaVA)$\uparrow$ | Action(Qwen) $\uparrow$ | Action(LLaVA)$\uparrow$ | OGA$\uparrow$
---|---|---|---|---|---
RRNet | 0.697 | 0.684 | 0.500 | 0.632 | 0.970
w/o HGCN | 0.509 | 0.558 | 0.273 | 0.518 | 0.844
w/o negative loss | 0.534 | 0.633 | 0.451 | 0.555 | 0.949
w/o node disentanglement | 0.543 | 0.635 | 0.426 | 0.535 | 0.938
Table 2: Quantitative Results of Variants of RRNet. Method | User Preference Rate $\uparrow$
---|---
Stable Diffusion | 16.98%
Personalized Diffusion | 7.46%
RRNet ($\lambda$=0.6) | 75.56%
Table 3: Results of User Study. The result shows the percentage of evaluators
prefer the image produced by RRNet verse baselines.

###  4.4 Generalization to Unseen Objects

Figure 6: Example of Generalization. By switching the objects in the prompts,
RRNet can still generate correct relationships.

In our framework, for each set of OSPs, the RRNet is trained with images
contains only one pair of objects. It is normal to wonder if the trained RRNet
can generalize well to the unseen objects.

As shown in Figure 6 and Figure 1(b), by constructing new graphs, RRNet
demonstrates its ability to handle many objects unseen in the training
dataset.

###  4.5 Ablation Study

Figure 7: Ablation Results.

Here, we demonstrate the importance of our three designs, 1) HGCN 2) negative
loss, and 3) object node disentanglement. For a fair comparison, we kept the
training hyperparameters the same and set $\lambda$=0.6 during image
generation.

HGCN. Excluding the HGCN, which accounts for graph structure, RRNet reverts to
a basic personalized diffusion model. This scenario has been analyzed and
compared in Table 5. The results indicate that the construction of relational
graphs is crucial for effective relation modeling in RRNet.

Negative loss. As depicted in Figure 7, the absence of negative loss leads to
generated images that somewhat conform to both relational directions. For
instance, in the upper row, images of “A bottle contains a car, and a car
contains a bottle” are generated. Meanwhile, as can be seen from Table 2,
without negative loss, the performance of the model decreases a lot.

Object node disentanglement. Omitting this component leads to the blending of
semantics of nodes representing different objects. As shown in Figure 7, this
results in generated images with merged features, such as a bottle with car-
like traits or a bowl resembling cloth. Quantitative results in Table 2
further show reduced accuracy in generating both relationships and objects.

##  5 Conclusion

We presented a novel framework, named RRNet, to enhance the ability of
diffusion models in generating images with more accurate relation directions.
The key of our approach is to employ a HGCN to explicitly model the relational
direction present in the prompts. RRNet will produce an adjustment vector to
rectify the direction of the relations in the text embeddings that are
provided to SD. We experimentally demonstrate the interpretability as well as
the robust performance of our approach.

## Acknowledgements

This research is supported by the Ministry of Education, Singapore, under its
Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-0006).

## References

  * [1] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.  Sep 2023.
  * [2] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  Jan 2023.
  * [3] Minghao Chen, Iro Laina, and Andrea Vedaldi.  Training-free layout control with cross-attention guidance.  Apr 2023.
  * [4] Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka.  Debiasing vision-language models via biased prompts.  Jan 2023.
  * [5] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis.  Neural Information Processing Systems,Neural Information Processing Systems, Dec 2021.
  * [6] Chengbin Du, Yanxi Li, Zhongwei Qiu, and Chang Xu.  Stable diffusion is unstable.  Advances in Neural Information Processing Systems, 36, 2023.
  * [7] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, XinEric Wang, WilliamYang Wang, UcSanta Barbara, Santa Uc, Cruz Cruz, and Google Google.  Training-free structured diffusion guidance for compositional text-to-image synthesis.
  * [8] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, AmitH Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.
  * [9] Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang.  Benchmarking spatial relationships in text-to-image generation.
  * [10] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2022.
  * [11] WilliamL. Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  Neural Information Processing Systems,Neural Information Processing Systems, Jun 2017.
  * [12] Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, and Amir Globerson.  Incorporating structured representations into pretrained vision & language models using scene graphs.  May 2023.
  * [13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Neural Information Processing Systems,Neural Information Processing Systems, Jan 2017.
  * [14] Jonathan Ho, Ajay Jain, Pieter Abbeel, and UC Berkeley.  Denoising diffusion probabilistic models.
  * [15] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International conference on machine learning, pages 2790–2799. PMLR, 2019.
  * [16] EdwardJ. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv: Computation and Language,arXiv: Computation and Language, Jun 2021.
  * [17] Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Tangjie Lv, Zhipeng Hu, and Wen Zhang.  Structure-clip: Enhance multi-modal language representations with structure knowledge.  arXiv preprint arXiv:2305.06152, 2023.
  * [18] Ziqi Huang, Tianxing Wu, Yuming Jiang, KelvinC.K. Chan, and Ziwei Liu.  Reversion: Diffusion-based relation inversion from images.  Mar 2023.
  * [19] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, QuocV. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  Cornell University - arXiv,Cornell University - arXiv, Feb 2021.
  * [20] Yongcheng Jing, Yining Mao, Yiding Yang, Yibing Zhan, Mingli Song, Xinchao Wang, and Dacheng Tao.  Learning graph neural networks for image style transfer.  Jul 2022.
  * [21] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  Advances in Neural Information Processing Systems, 35:26565–26577, 2022.
  * [22] Thomas Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv: Learning,arXiv: Learning, Sep 2016.
  * [23] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  Dec 2022.
  * [24] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
  * [25] Rui Li, Xin Yuan, Mohsen Radfar, Peter Marendy, Wei Ni, Terrence J. O’Brien, and Pablo Casillas-Espinosa.  Graph signal processing, graph neural network and graph learning on biological data: A systematic review.  IEEE Reviews in Biomedical Engineering, page 109–135, Jan 2023.
  * [26] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and YongJae Lee.  Gligen: Open-set grounded text-to-image generation.  Jan 2023.
  * [27] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: Common Objects in Context, page 740–755.  Jan 2014.
  * [28] Haotian Liu, Chunyuan Li, Qingyang Wu, YongJae Lee, Madison Madison, and Microsoft Research.  Visual instruction tuning.
  * [29] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and JoshuaB. Tenenbaum.  Compositional visual generation with composable diffusion models.  Jun 2022.
  * [30] Zhiyuan Liu and Jie Zhou.  GRAPH ATTENTION NETWORKS, page 39–41.  Jan 2020.
  * [31] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization, 2019.
  * [32] Alex Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  Cornell University - arXiv,Cornell University - arXiv, Feb 2021.
  * [33] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.
  * [34] Alec Radford, JongWook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  Cornell University - arXiv,Cornell University - arXiv, Feb 2021.
  * [35] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.
  * [36] Alireza Rezvanian and Mohammad Reza Meybodi.  Stochastic graph as a model for social networks.  Computers in Human Behavior, 64:621–640, Nov 2016.
  * [37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer.  High-resolution image synthesis with latent diffusion models.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2022.
  * [38] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  Aug 2022.
  * [39] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar, Seyed Ghasemipour, Burcu Karagol, SSara Mahdavi, RaphaGontijo Lopes, Tim Salimans, Jonathan Ho, DavidJ Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.
  * [40] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016.
  * [41] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross.  Winoground: Probing vision and language models for visio-linguistic compositionality.
  * [42] Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parminder Bhatia, and Stefano Soatto.  Linear spaces of meanings: Compositional structures in vision-language models.  Feb 2023.
  * [43] Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang.  Deep graph library: A graph-centric, highly-performant package for graph neural networks.  arXiv preprint arXiv:1909.01315, 2019.
  * [44] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu.  Heterogeneous graph attention network.  In The World Wide Web Conference, May 2019.
  * [45] Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, Mingyuan Zhou, et al.  Patch diffusion: Faster and more data-efficient training of diffusion models.  Advances in Neural Information Processing Systems, 36, 2024.
  * [46] Lingfei Wu, Yu Chen, Kai Shen, Xiaojie Guo, Hanning Gao, Shucheng Li, Jian Pei, and Bo Long.  Graph neural networks for natural language processing: A survey., Jan 2023.
  * [47] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and MikeZheng Shou.  Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion.  Jul 2023.
  * [48] Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang.  Diffusion probabilistic model made slim.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22552–22562, 2023.
  * [49] Xingyi Yang, Daquan Zhou, Songhua Liu, Jingwen Ye, and Xinchao Wang.  Deep model reassembly.  Advances in neural information processing systems, 35:25739–25753, 2022.
  * [50] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.  When and why vision-language models behave like bag-of-words models, and what to do about it?  Oct 2022.
  * [51] Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V. Chawla.  Heterogeneous graph neural network.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery; Data Mining, Jul 2019.
  * [52] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.

This supplementary document offers additional comparisons and detailed
information regarding our proposed RRNet model. In Sec A, we offer a further
comparison between proposed RRNet and more baseline methods. In Sec B, we
present more visualized results. We then provide additional analysis of
$V_{eot}$ in Sec C. The implementation and additional experimental details are
included in Sec D. Finally, we discuss the limitations of RRNet in Sec E.

##  Appendix A Additional Comparisons

Method | GPU Memory Usage (MiB)
---|---
TI | 25458
Unet Update | 30060
Unet Update + TI | OOM
RRNet | 25040
Table 4: Memory usage of training. Method | Position(Qwen) $\uparrow$ | Position(LLaVA)$\uparrow$ | Action(Qwen) $\uparrow$ | Action(LLaVA)$\uparrow$ | OGA$\uparrow$ | FID$\downarrow$
---|---|---|---|---|---|---
TI | 0.560 | 0.592 | 0.443 | 0.574 | 0.929 | 91.634
Unet Update | 0.473 | 0.563 | 0.404 | 0.529 | 0.912 | 87.294
RRNet ($\lambda$=0.2) | 0.564 | 0.597 | 0.469 | 0.565 | 0.937 | 89.13
RRNet ($\lambda$=0.6) | 0.697 | 0.684 | 0.500 | 0.632 | 0.970 | 100.78
Table 5: Further comparisons.

Additional baselines. To provide a more comprehensive comparison, we compare
with three other baseline methods.

  * •

Textual Inversion. The first method is a variation of textual inversion [8].
In this approach, we employ the its loss function to fit the diffusion model
to the given set of images by optimizing the input embeddings.

  * •

Unet Update. The second method involves maintaining the text encoder of SD
fixed while fine-tuning the Unet component. Specifically, we implement Low-
Rank Adaptation (LoRA) [16] to prevent catastrophic forgetting in SD. The
training parameters are set with a learning rate of 1e-4 and a rank of 4, over
a duration of 100 epochs.

  * •

Reversion. Furthermore, we tested the Reversion method on our dataset. The
empirical results indicated that the images generated through Reversion were
of poor quality. This suboptimal performance could be attributed to the
inherent complexities associated with the Relation Rectification Task. Due to
these findings, we decided against including the quantitative results of this
method in our report. For reference, examples of these generated images are
depicted in Figure 8.

A potential fourth method, optimizing the text encoder and Unet parameters
simultaneously, was considered but deemed impractical due to its excessive
computational demands. The memory requirements for training using different
methods are compared in Table 4. Training with a batch size of 1 is not
unfeasible on a single V100-32GB.

Quantitative analysis. The results are presented in the Table 5. It can be
observed that among the baselines, TI demonstrates higher performance in
relationship generation accuracy metrics compared to Unet Update. Our RRNet,
employing a mere $\lambda=0.2$, closely approaches the performance of TI
across various relationship generation accuracy metrics, while also
maintaining a lower FID. Increasing $\lambda$ to 0.6 significantly elevates
RRNet’s performance above the baselines in all relationship generation
accuracy metrics. Specifically, in the Position (Qwen) and Action (Qwen)
metrics, RRNet ($\lambda$=0.6) outperforms TI by 13.7% and 5.7%, respectively.
Furthermore, when using LLaVA as the detector, RRNet ($\lambda$=0.6) also
exceeds TI by 9.2% and 5.8% respectively in position and action relationship
generation metrics. In terms of OGA, RRNet ($\lambda$=0.6) surpasses TI by a
margin of 4.1%.

For those two baselines, since the issue with the text encoder treating
sentences as a Bag of Words (BOW) remains unresolved, they still struggle to
accurately distinguish the direction of relationships. They tend more to
directly fit the dataset rather than truly learning to represent the
relationships between objects.

Figure 8: In our dataset, the Reversion fails to achieve accurate relationship
generation.

##  Appendix B Results Visualization

We provide additional qualitative results to demonstrate the effect of RRNet
on relation rectification for SD. The generation results of action OSPs are
shown in Figure 9 and the results of positional OSPs are showcased in Figure
10.

It can be observed that with RRNet’s assistance, SD can accurately generate
relationships in both directions.

Figure 9: Additional results of action relation rectification with RRNet.
Figure 10: Additional results of positional relation rectification with RRNet.
Figure 11: After replacing the $V_{eot}$, a phenomenon of entity replacement
occurred.

##  Appendix C Additional Analysis of $V_{eot}$

In the Sec 3, we identify the $V_{eot}$ as pivotal in generating
relationships. Specifically, as detailed in previous work [3], the $V_{eot}$
plays a crucial role in foreground generation, encompassing both the objects
and their interrelationships.

Here, we conducted an experiment by replacing the $V_{eot}$ of two prompts to
validate its significance.

Observation. Our experiment, illustrated in the first row of Figure 11,
involved replacing the $V_{eot}$ from “A corgi” with that from “A cat inside
the box”. This resulted in an image of “A corgi inside the box”. This
experiment suggests that the primary role of $V_{eot}$ is to control the
foreground layout, dictating where each object appears. Since the semantic of
“A corgi” closely match that of “A cat”, it would match the anchor for “A cat”
and instead generate a corgi.

Discussion. We uncovered a fascinating aspect of $V_{eot}$ related to the
diffusion generation mechanism. Analogous to painting, the $V_{eot}$ shapes
the basic layout of the foreground, setting anchor points for each object’s
generation. It establishes an anchor point for each foreground object to be
generated, without detailing each object specifically. The word embeddings of
these objects then align with the closest anchor, leading to their
manifestation at specific locations. The occurrences in Figure [Mask
Phenomenon] can be seen as a failure in object generation due to the lack of
precise positional information in $V_{eot}$.

Therefore, in our work, we achieve the effect of correctly controlling the
generation of the foreground by adjusting $V_{eot}$.

##  Appendix D Experiment Settings

###  D.1 Detailed dataset statistics.

The RR dataset comprises 21 relationships, consisting of 8 positional and 13
action types. Each relationship includes 4 types of prompts, two for OSPs
represented as $<A,R,B>$ and $<B,R,A>$, and two generated from template
sentence “This is a photo of {obj}” for object disentanglement purposes. We
collect 3-5 images for each prompt to serve as exemplars for training.

###  D.2 Relationship Detection with Chatbots.

Prompts for relationship detection. We employ Vision-language chatbots to
facilitate the detection of relationships in images. To ensure these chatbots
focus more on the relationships between objects, we have developed a series of
prompt templates. For a sentence can be abstracted as triplet $<A,R,B>$,
prompt templates are listed follows:

  1. 1.

“Is there any object A in the image?”

  2. 2.

“Is there any object B in the image?”

  3. 3.

“Are both object A and object B present in the image?”

  4. 4.

“Can you infer the relationship that exists between object A and object B in
the image?”

  5. 5.

“Is there ARB or BRA or neither?”

“object A” and “object B” are placeholders substituted by entities in the
OSPs. “ARB” and “BRA” are OSPs. During the evaluation, each generated image
undergoes assessment through above five questions.

Object detection. Question 1 and Question 2 are designed for object detection.
During our experiments, we observed a high false-positive rate in object
detection using the first two questions, primarily because the model
occasionally generates a composite object embodying features of both objects A
and B. Consequently, even if there is a single object displaying both sets of
features, the chatbots are likely to affirmatively respond to the first two
questions. To mitigate this issue, we introduced Question 3, aimed at
filtering out objects that exhibit mixed features. For object detection, a
generation is deemed correct only if it successfully passes Questions 1, 2,
and 3.

Relationship detection. Question 4 is designed to steer the chatbot towards
creating contexts that emphasize the relationships between objects, thereby
setting the stage for Question 5. Regarding relationship generation, a
relationship generation is classified as correct only if the chatbot’s
response to the Question 5 aligns with the prompt for generating the given
image.

###  D.3 Additional Implementation details.

Implementation details of HGCN. In our implementation, the HGCN is built with
DGL [43]. We use the HGCN based on Graph Attention Networks (GAT) [30]. The
dimension of HGCN’s hidden layers is 512. The learning rate of HGCN is set to
3e-4. The optimizer used is AdamW [31]. The training batch size is set to 1.

##  Appendix E Limitations

###  E.1 Unseen concepts

Figure 12: Failure case. The model has no sense of how horse rides on other
objects. Figure 13: Generation of Complex Relationships.

Our method is capable of steer the generation of SD by adjusting the direction
of the relations in the text embeddings. Since we do not modify any parameters
in denoising network, there are limitations to our approach for concepts that
do not exist in SD. We show a failure case in Figure 12. Although our method
separates the “A horse rides an astronaut” from “An astronaut rides a horse”,
SD lacks the concept of a horse riding anything, leaving the RRNet
directionless in adjusting the horse-astronaut relationship. Consequently,
most generated images are mere variations of existing dataset images. we
observe that for such abstract relations, we need a larger $\lambda$ to ensure
the generated images meaningful, which in term undermines the images’ quality.

###  E.2 Multi-relationships generation

we employed multiple RRNets, initially _trained on simple paired relations_ ,
to handle more complex scenarios in image generation. The results are
illustrated in the Figure 13. Although generated results looks reasonable,
there is a noticeable drop in performance as the complexity of relationships
and objects increases. The occurrence of this phenomenon is likely due to
multiple RRNETs simultaneously adjusting the $V_{eot}$, resulting in semantic
confusion within $V_{eot}$. Therefore, we believe that exploring how to
construct a more complex graph to generate one adjustment vector capable of
jointly rectifying multiple relational semantics is a highly promising avenue
for future research.


# On proof of the Wei-Yue Ding’s conjecture for Schrödinger map flow

Sheng Wang  Shanghai Center for Mathematical Sciences, Fudan University,
Shanghai 200433, P.R. China<EMAIL_ADDRESS>and  Yi Zhou  School of
Mathematical Sciences, Fudan University, Shanghai 200433, P.R. China
<EMAIL_ADDRESS>
###### Abstract.

Wei-Yue Ding [8] proposeed a proposition about Schrödinger map flow in 2002
International Congress of Mathematicians in Beijing, which is called Wei-Yue
Ding conjecture by Rodnianski-Rubinstein-Staffilani [14]. They proved [14]
that Schrödinger map flow for maps from the real line into Kähler manifolds
and for maps from the circle into Riemann surfaces is globally well-posed
which is the first significant advance in this conjecture by translating the
Schrödinger map flow into nonlinear Schrödinger-type equations or (systems)
and partially solved this conjecture. In this article, we will derive a new
div-curl type lemma and combined it with energy and “momentum” balance law to
get some space-time estimates. Based on this, we prove the Schrödinger map
flow for maps from the circle into Kähler manifolds is globally regular, this
settles the Wei-Yue Ding’s conjecture.

###### Key words and phrases:

Wei-Yue Ding’s Conjecture; Kähler manifolds; Div-Curl Lemma; Gobal Well-
posedness.

###### 2010 Mathematics Subject Classification:

Primary: 35Q55; Secondary: 35B44

##  1\. Introduction

Let $(M,g)$ be an $m$-dimensional complete Riemannian manifold, and let
$(N,\omega,J,h)$ be a $2n$-dimensional complete symplectic manifold with a
compatible almost complex structure $J$, which means
$(J\cdot,J\cdot)=(\cdot,\cdot)$ and $h(\cdot,\cdot)=\omega(\cdot,J\cdot)$.
This defines a complete Riemannian metric on $N$.

We introduce the Fréchet manifold $X:=C^{\infty}(M,N)$ which is the space of
all smooth maps from $M$ to $N$, endowed with a symplectic structure,

$\Omega(V,W)|_{u}=\int_{M}u^{\star}\omega(V,W)dV_{M,g},\quad\forall\;V,W\in
T_{u}X=\Gamma(M,u^{\star}TN),$ (1.1)

where the tangent space to $X$ at a map $u:M\rightarrow N$ is the space of
smooth sections of $u^{\star}TN\rightarrow M$ and where $dV_{M,g}$ denotes the
volume form on $M$ induced by Riemannian metric $g$. The form $\Omega$ is non-
degenerate, i.e., endows $X$ with an injective map $TX\rightarrow T^{\star}X$.

The energy function on $X$ can be represented by

$\mathcal{E}(u)=\frac{1}{2}\int_{M}|du|_{g^{\sharp}\otimes
u^{\star}h}^{2}dV_{M,g},$ (1.2)

where we denote by $g^{\sharp}$ the metric induced by $g$ on $T^{\star}M$ and
where we view $du$ as a section of $T^{\star}M\otimes u^{\star}TN\rightarrow
M$ and equip this bundle with the metric $g^{\sharp}\otimes u^{\star}h$.

As the similar arguments in [14], by Hamilton’s principle, we can obtain the
corresponding Hamiltonian flow:

$\left\\{\begin{aligned} &\partial_{t}u=-J\tau(u),\\\ &u|_{\\{0\\}\times
M}=u_{0},\end{aligned}\right.$ (1.3)

on $\left(X,\Omega\right)$ from (1.2), where $\tau(u):=tr_{g^{\sharp}}\nabla
du$ is called the tension field of $u$ and $\nabla$ is the connection on
$T^{\star}M\otimes u^{\star}TN\rightarrow M$ induced from the Levi-Civita
connection on $(M,g)$ and the pulled-back one from $(N,h)$.

In fact, the Hamiltonian flow (1.3) is also called Schrödinger map flow,
introduced by Ding-Wang in [6].

An interesting problem about this flow (1.3) is the existence and uniqueness.
But if we consider the general case, we may not have the global existence or
well-posedness because this flow is defined on infinite-dimensional spaces. So
we usually restrict to the Kähler case. Ding-Wang [7] proved the local well-
posedness, and then McGahagan [13] gave the similar results with another way
by parabolic approximation. This result can be described as follows:

######  Theorem 1.1 (Local well-posedness).

Suppose $\left(M,g\right)$ is an $m$-dimensional complete Riemannian manifold
and $\left(N,J,h\right)$ is a complete Kähler manifold with bounded geometry.
If the initial data $u_{0}\in H^{k}\left(M,N\right)$ with $k>\frac{m}{2}+1$,
then the system (1.3) admits a unique flow $u\in
C^{0}\left(\left[0,T\right],H^{k}\left(M,N\right)\right)$ where $T<T_{0}$ and
$T_{0}$ only depends on the geometry of the manifold $N$ and $\left\|\nabla
u_{0}\right\|_{H^{\left[m/2\right]+1}}$. In addition, we have the following
estimates:

$\left\|\nabla
u\right\|_{H^{\left[m/2\right]+1}}\leqslant\frac{C_{1}}{\left(T_{0}-t\right)^{C_{2}}}\quad\forall
t\in\left[0,T_{0}\right),$

where the constants $C_{1}$ and $C_{2}$ only depend on the geometry of the
manifold $N$ and $\left\|\nabla u_{0}\right\|_{H^{\left[m/2\right]+1}}$.
Particularly, when the initial data $u_{0}\in H^{k}\left(M,N\right)$ with any
$k\geqslant 2$, then the flow $u\in C^{\infty}\left(\left[0,T\right]\times
M,N\right)$.

We recall the Wei-Yue Ding’s conjecture:

###  1.1. Conjecture [8]

The Schrödinger map flow is globally well-posed for maps from one-dimensional
domains into compact Kähler manifolds.

Rodnianski-Rubinstein-Staffilani [14] first studied this conjecture, and have
taken an important step forward in the research of the conjecture. They proved
that the Schrödinger map flow for maps from the real line into Kähler
manifolds and for maps from the circle into Riemann surfaces is globally well-
posed. These results can be described as follows:

######  Theorem 1.2.

Suppose $\left(M,g\right)=\left(\mathbb{R},dx\otimes dx\right)$ and
$\left(N,J,h\right)$ is a complete Kähler manifold with bounded geometry. If
the initial data $u_{0}\in H^{k}\left(\mathbb{R},N\right)$ with $k\geqslant
2$, then the system (1.3) admits a unique flow $u\in
C^{0}\left(\left[0,\infty\right),H^{k}\left(\mathbb{R},N\right)\right)$.
Particularly, when the initial data $u_{0}\in H^{k}\left(\mathbb{R},N\right)$
with any $k\geqslant 2$, then the flow $u\in
C^{\infty}\left(\left[0,\infty\right)\times\mathbb{R},N\right)$.

######  Theorem 1.3.

Suppose $\left(M,g\right)=\left(\mathbb{S}^{1},dx\otimes dx\right)$ and
$\left(N,J,h\right)$ is a complete Riemann surface with bounded geometry. If
the initial data $u_{0}\in H^{k}\left(\mathbb{S}^{1},N\right)$ with
$k\geqslant 2$, then the system (1.3) admits a unique flow $u\in
C^{0}\left(\left[0,\infty\right),H^{k}\left(\mathbb{S}^{1},N\right)\right)$.
Particularly, when the initial data $u_{0}\in
H^{k}\left(\mathbb{S}^{1},N\right)$ with any $k\geqslant 2$, then the flow
$u\in C^{\infty}\left(\left[0,\infty\right)\times\mathbb{S}^{1},N\right)$.

In the Rodnianski-Rubinstein-Staffilani work [14], they need to translate the
Schrödinger map flow into nonlinear Schrödinger-type equations or (systems).
For the maps from real line into Kähler manifolds, they can find a set of
global parallel frams to translate the Schrödinger map flow into nonlinear
Schrödinger-type equations or (systems) and use the Strichartz estimates to
obatin the $L^{4}$-norm priori estimates to establish the global well-
posedness. For the maps from circle into Riemann surfaces, they use the
holonomy method and some transformation to translate the Schrödinger map flow
into cubic nonlinear Schrödinger and adapt the Bourgain’s results [4] to get a
priori estimates for $L^{4}$-norm to establish the global well-posedness.

For the maps from circle into Kähler manifolds, it is unable to translate the
Schrödinger map flow into nonlinear Schrödinger-type equations or (systems) so
that the classical methods for studying the dispersive equation (such as
Strichartz estimates) are no longer suitable for the research of Schrödinger
map flow. So in this work, we study the Schrödinger map flow from a new
perspective. we don’t need to translate the Schrödinger map flow into
nonlinear Schrödinger-type equations or (systems), and don’t need to use the
Strichartz estimates. Just like our previous work [15] for time-like extremal
hypersurface equations, we also derive a new div-curl type lemma 4.1 that
connects energy and “momentum” in the periodic case. In our method, we can use
the new div-curl type lemma 4.1 to study some balance law (i,e.
$\partial_{t}f+\partial_{x}f=f^{re}$). Particularly, we can use the new div-
curl lemma 4.1 to fully explore the energy balance law and “momentum” balance
law. Interestingly, in our approch, the “momentum” balance law is as important
as the energy balance law, this gives a new perspective in the study of the
estimates of the nonlinearity. Based on this and making full use of the almost
complex structure $J$ in Kähler manidolds, we can prove that the Schrödinger
map flow for maps from the $\mathbb{S}^{1}$ into Kähler manifolds is globally
well-posed. So far, the Wei-Yue Ding’s conjecture has been completely proved.

Now, we describe our main results as follow:

######  Theorem 1.4.

Suppose $\left(M,g\right)=\left(\mathbb{S}^{1},dx\otimes dx\right)$ and
$\left(N,J,h\right)$ is a complete Kähler manifold with bounded geometry. If
the initial data $u_{0}\in H^{k}\left(\mathbb{S}^{1},N\right)$ with
$k\geqslant 2$, then the system (1.3) admits a unique flow $u\in
C^{0}\left(\left[0,\infty\right),H^{k}\left(\mathbb{S}^{1},N\right)\right)$.
Particularly, when the initial data $u_{0}\in
H^{k}\left(\mathbb{S}^{1},N\right)$ with any $k\geqslant 2$, then the flow
$u\in C^{\infty}\left(\left[0,\infty\right)\times\mathbb{S}^{1},N\right)$.

######  Remark 1.5.

We also derive a div-curl type lemma 4.2 in the real line. Using this lemma to
repeat our argument, the Wei-Yue Ding’s conjecture for the real line case can
be proved in an elementary way.

For the higher dimensional case, there is also a lot of outstanding work.
Chang-Shatah-Uhlenbeck [5] studied the raially symmetric Schrödinger map flow
maps from the $\mathbb{R}^{2}$ into Riemann surface with small energy. Without
raially symmetric assumption, Bejenaru-Ionescu-Kenig-Tataru [2] obtained the
global solutions for Schrödinger map flow maps from the $\mathbb{R}^{d}$ for
$d\geqslant 2$ into $\mathbb{S}^{2}$ with small energy in critical sobolev
space. Besides, Bejenaru-Ionescu-Kenig-Tataru [3] researched equivariant
solutions for the Schrödinger map problem from $\mathbb{R}^{1+2}$ to
$\mathbb{S}^{2}$ with energy less than $4\pi$ and showed that they are global
in time and scatter. Huang-Wang-Zhao [9] considered the Schrödinger map flow
maps from two dimensional hyperbolic space $\mathbb{H}^{2}$ into sphere
$\mathbb{S}^{2}$ with small initial data. For the Besov regularity, the global
existence for small data in critical Besov spaces was proved by Ionescu-Kenig
[10] and Bejenaru [1] independently. Recently, Li extended the result of
Bejenaru-Ionescu-Kenig-Tataru [2] to general Kähler manifolds, see [11] and
[12].

Next, we give the framework of this article.

###  1.2. Outline of the article.

This article is organized as follows: In section 2, we introduce some
notations and preliminaries, which makes the proof more easier to understand
in the later section. In section 3, we establish some balance law (i,e.
$\partial_{t}f+\partial_{x}f=f^{re}$) associated with energy and “momentum”.
In section 4, we will derive a new div-curl lemma 4.1 in the periodic case,
which palys a key role to get the regularity estimates of solution. In section
5, we note that “div-curl” structure is an “inner product” in some sense, and
calculate these “inner product” induced by balance law. In section 6, we use
the div-curl lemma 4.1 and continuous induction method to obatin the priori
estimates for the solution. Based on this and combining the local well-
posedness results of Ding-Wang [7], we can use the standard interval
decomposition method and extension method to obtain a global solution.

##  2\. Notations and Preliminaries

In this section, we will make some notations and give some preliminaries.

We mark $A\lesssim B$ to mean there exists a constant $C>0$ such that
$A\leqslant CB$. We indicate dependence on parameters via subscripts, e.g.
$A\lesssim_{x}B$ indicates $A\leqslant CB$ for some $C=C(x)>0$.

To avoid ambiguity, let’s make some symbolic conventions.

Suppose $\left(M,g\right)$ is a Riemann manifold with Riemann metric $g$. For
any tangent vector fields $X$, $Y$ $\in TM$, we write $<X,Y>_{g}$ to represent
the Riemann inner product for tangent vector fields $X$ and $Y$ and define
$|X\rvert^{2}_{g}$ := $<X,X>_{g}$. For tangent vector fields $X$, $Y$, $Z$,
$W$ $\in TM$, we define the curvature operators and curvature tensors:

$\displaystyle
R(X,Y)Z:=\nabla_{X}\nabla_{Y}Z-\nabla_{Y}\nabla_{X}Z-\nabla_{\left[X,Y\right]}Z,$
$\displaystyle R\left(X,Y,Z,W\right):=<R\left(Z,W\right)Y,X>_{g}.$

Under this notation convention, the Schrödinger map flow maps from the real
line into Kähler manifolds and for maps from the circle into Kähler manifolds
can be rewritten:

$J\partial_{t}u=\nabla_{x}\partial_{x}u.$ (2.1)

.

Since we only consider that the target manifold with bounded geometry which
means uniform bounds on the curvature tensor and its derivatives. So for
simplicity of writing, we omit the dependence on curvature. That means we can
omit the subscripts ”$R$” in $A\lesssim_{R}B$.

For convenience, let us define the following quantity:

$m\left(t\right):=\int_{0}^{1}<\partial_{x}u,\partial_{x}u>_{g}$

$E\left(t\right):=\int_{0}^{1}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$

$\xi_{1}\left(t\right):=\int_{0}^{t}\int_{0}^{1}detA_{m}$

$\xi_{2}\left(t\right):=\int_{0}^{t}\int_{0}^{1}detA$

where the $detA_{m}$, $detA$ are defined in section 5.

##  3\. law of equilibrium

Let us take the inner product of both sides of (2.1) with the Riemann metric
with the $\partial_{t}u$ and use the antisymmetry of the almost structure $J$,
we get:

$<\nabla_{x}\partial_{x}u,\partial_{t}u>_{g}=-<\nabla_{x}\partial_{x}u,J\nabla_{x}\partial_{x}u>_{g}=0.$

According to some simple calculations and the antisymmetry of the almost
complex structure $J$, we rewrite above as follows:

$\partial_{x}<\partial_{x}u,\partial_{t}u>_{g}-<\partial_{x}u,\nabla_{x}\partial_{t}u>_{g}=0,$

and

$\partial_{x}<\partial_{x}u,-J\nabla_{x}\partial_{x}u>_{g}-<\partial_{x}u,\nabla_{x}\partial_{t}u>_{g}=0.$

So we obtain the conservation law:

$\frac{1}{2}\partial_{t}<\partial_{x}u,\partial_{x}u>_{g}-\partial_{x}<J\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}=0.$
(3.1)

Taking the covariant derivative of both sides of (2.1) with respect to the
variable $x$, we have:

$J\nabla_{x}\partial_{t}u=\nabla_{x}\nabla_{x}\partial_{x}u.$

According to the commutativity of covariant derivatives and differential
operators, we can rewrite above as follow:

$J\nabla_{t}\partial_{x}u=\nabla_{x}\nabla_{x}\partial_{x}u.$ (3.2)

Taking the inner product of $J\partial_{x}u$ and $\nabla_{x}\partial_{x}u$ in
the Riemann metric sense, and taking the derivative directly with respect to
the variable $t$, we know:

$\displaystyle\partial_{t}<J\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle=<\nabla_{x}\partial_{x}u,J\nabla_{t}\partial_{x}u>_{g}+<\nabla_{t}\nabla_{x}\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle=<\nabla_{x}\partial_{x}u,J\nabla_{t}\partial_{x}u>_{g}+<\nabla_{t}\nabla_{x}\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle-<\nabla_{x}\nabla_{t}\partial_{x}u,J\partial_{x}u>_{g}+<\nabla_{x}\nabla_{t}\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle=<\nabla_{x}\partial_{x}u,J\nabla_{t}\partial_{x}u>_{g}+<\nabla_{x}\nabla_{t}\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle+<R\left(\partial_{t}u,\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle=<\nabla_{x}\partial_{x}u,J\nabla_{t}\partial_{x}u>_{g}+\partial_{x}<\nabla_{t}\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle-<\nabla_{t}\partial_{x}u,J\nabla_{x}\partial_{x}u>_{g}+<R\left(\partial_{t}u,\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle=2<\nabla_{x}\partial_{x}u,J\nabla_{t}\partial_{x}u>_{g}+\partial_{x}<\nabla_{t}\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle+<R\left(\partial_{t}u,\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle=2<\nabla_{x}\partial_{x}u,\nabla_{x}\nabla_{x}\partial_{x}u>_{g}-\partial_{x}<J\nabla_{x}\nabla_{x}\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle-<R\left(J\nabla_{x}\partial_{x}u,\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle=2<\nabla_{x}\partial_{x}u,\nabla_{x}\nabla_{x}\partial_{x}u>_{g}-\partial_{x}<\nabla_{x}\nabla_{x}\partial_{x}u,\partial_{x}u>_{g}$
$\displaystyle+<R\left(\partial_{x}u,J\nabla_{x}\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle=\partial_{x}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}-\partial_{x}<\nabla_{x}\nabla_{x}\partial_{x}u,\partial_{x}u>_{g}$
$\displaystyle+\frac{1}{4}\partial_{x}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle-\frac{1}{4}<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}.$
(3.3)

So we have the balance law:

$\displaystyle\partial_{t}<J\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}-\partial_{x}\left(<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}-<\nabla_{x}\nabla_{x}\partial_{x}u,\partial_{x}u>_{g}\right)$
$\displaystyle-\frac{1}{4}\partial_{x}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}=-\frac{1}{4}<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}.$
(3.4)

##  4\. New div-curl type lemma

In this section, we will prove a new div-curl lemma, which is useful for us to
get the regularity estimates of solution.

######  Lemma 4.1.

Suppose that

$\left\\{\begin{aligned} &\partial_{t}f^{11}+\partial_{x}f^{12}=G^{1}\\\
&\partial_{t}f^{21}-\partial_{x}f^{22}=G^{2},\end{aligned}\right.$ (4.1)

where $f^{i,j}$, $G^{i}$ are periodic in x with periodic 1.

Then there hold

$\displaystyle\int_{0}^{T}\int_{0}^{1}f^{11}f^{22}+f^{12}f^{21}$
$\displaystyle\lesssim\left(\|f^{11}\left(0\right)\|_{L^{1}}+\sup\limits_{0\leqslant
t\leqslant T}\|f^{11}\left(t\right)\|_{L^{1}}+\int_{0}^{T}\int_{0}^{1}\lvert
G^{1}\rvert\right)$
$\displaystyle\cdot\left(\|f^{21}\left(0\right)\|_{L^{1}}+\sup\limits_{0\leqslant
t\leqslant T}\|f^{21}\left(t\right)\|_{L^{1}}+\int_{0}^{T}\int_{0}^{1}\lvert
G^{2}\rvert\right)$
$\displaystyle+\int_{0}^{T}\left(\int_{0}^{1}f^{11}\int_{0}^{1}f^{22}+\int_{0}^{1}f^{12}\int_{0}^{1}f^{21}\right)$
(4.2)

provided that the right side is bounded.

###### Proof.

We first assume that

$\int_{0}^{1}f^{11}=0.$ (4.3)

Noting that:

$\partial_{t}\left(\int_{y}^{x}f^{11}\right)+f^{12}\left(x\right)-f^{12}\left(y\right)=\int_{y}^{x}G^{1}.$

So we have:

$\partial_{t}\left(\int_{0}^{1}dy\int_{y}^{x}f^{11}\right)+f^{12}\left(x\right)=\int_{0}^{1}f^{12}dy+\int_{0}^{1}dy\int_{y}^{x}G^{1}.$

Then

$f^{21}\partial_{t}\left(\int_{0}^{1}dy\int_{y}^{x}f^{11}\right)+f^{12}\left(x\right)f^{21}\left(x\right)=f^{21}\left(x\right)\int_{0}^{1}f^{12}dy+f^{21}\left(x\right)\int_{0}^{1}dy\int_{y}^{x}G^{1}$
(4.4)

$\partial_{t}f^{21}\left(\int_{0}^{1}dy\int_{y}^{x}f^{11}\right)-\partial_{x}f^{22}\int_{0}^{1}dy\int_{y}^{x}f^{11}=G^{2}\int_{0}^{1}dy\int_{y}^{x}f^{11}$
(4.5)

(4.4)+(4.5):

$\displaystyle\partial_{t}\left(\int_{0}^{1}f^{21}\int_{0}^{1}dy\int_{y}^{x}f^{11}\right)+\int_{0}^{1}\left(f^{12}f^{21}-\partial_{x}f^{22}\int_{0}^{1}dy\int_{y}^{x}f^{11}\right)$
$\displaystyle=\int_{0}^{1}f^{21}\int_{0}^{1}dy\int_{y}^{x}G^{1}+\int_{0}^{1}G^{2}\int_{0}^{1}dy\int_{y}^{x}f^{11}+\int_{0}^{1}f^{21}\int_{0}^{1}f^{12}dy$
(4.6)

Using (4.3), and by periodic in $x$ with periodic 1, we know:

$f^{22}\left(t,1\right)\int_{0}^{1}dy\int_{y}^{1}f^{11}-f^{22}\left(t,0\right)\int_{0}^{1}dy\int_{y}^{0}f^{11}=f^{22}\left(t,0\right)\int_{0}^{1}dy\int_{0}^{1}f^{11}=0$

So we have:

$\displaystyle\int_{0}^{T}\int_{0}^{\infty}f^{11}f^{22}+f^{12}f^{21}=$
$\displaystyle\left(\int_{0}^{1}f^{21}\int_{0}^{1}dy\int_{y}^{x}f^{21}\right)\bigg{|}^{T}_{0}$
$\displaystyle+\int_{0}^{T}\int_{0}^{1}f^{21}\int_{0}^{1}dy\int_{y}^{x}G^{1}$
$\displaystyle+\int_{0}^{T}\int_{0}^{1}G^{2}\int_{0}^{1}dy\int_{y}^{x}f^{11}$
$\displaystyle+\int_{0}^{T}\int_{0}^{1}f^{21}\int_{0}^{1}f^{12}$
$\displaystyle:=\mathcal{A}_{1}+\mathcal{A}_{2}+\mathcal{A}_{3}+\mathcal{A}_{4},$
(4.7)

where

$\displaystyle\lvert\mathcal{A}_{1}\rvert$
$\displaystyle\lesssim\|f^{11}\left(0\right)\|_{L^{1}}\|f^{21}\left(0\right)\|_{L^{1}}+\|f^{11}\left(T\right)\|_{L^{1}}\|f^{21}\left(T\right)\|_{L^{1}},$
$\displaystyle\lvert\mathcal{A}_{2}\rvert$
$\displaystyle\lesssim\int_{0}^{T}\|f^{21}\left(t\right)\|_{L^{1}}\|G^{1}\left(t\right)\|_{L^{1}}$
$\displaystyle\lesssim$ $\displaystyle\sup\limits_{0\leqslant t\leqslant
T}\|f^{21}\left(t\right)\|_{L^{1}}\left(\int_{0}^{T}\int_{0}^{1}\lvert
G^{1}\rvert\right),$ $\displaystyle\lvert\mathcal{A}_{3}\rvert$
$\displaystyle\lesssim\int_{0}^{T}\|f^{11}\left(t\right)\|_{L^{1}}\|G^{2}\left(t\right)\|_{L^{1}}$
$\displaystyle\lesssim$ $\displaystyle\sup\limits_{0\leqslant t\leqslant
T}\|f^{11}\left(t\right)\|_{L^{1}}\left(\int_{0}^{T}\int_{0}^{1}\lvert
G^{2}\rvert\right).$

If (4.3) fails, we define:

$\displaystyle\widetilde{f}^{11}:=f^{11}-\int_{0}^{1}f^{11},$
$\displaystyle\widetilde{G}^{1}:=G^{1}-\int_{0}^{1}G^{1}.$ (4.8)

We know:

$\partial_{t}\widetilde{f}^{11}+\partial_{x}f^{12}=\widetilde{G}^{1}$ (4.9)

Repeating the process above, we complete the proof. ∎

######  Lemma 4.2.

Suppose that

$\left\\{\begin{aligned} &\partial_{t}f^{11}+\partial_{x}f^{12}=G^{1},\\\
&\partial_{t}f^{21}-\partial_{x}f^{22}=G^{2},\end{aligned}\right.$ (4.10)

$\displaystyle f^{11},f^{12},f^{21},f^{22}\rightarrow 0,x\rightarrow\infty.$
(4.11)

Then there hold

$\displaystyle\int_{0}^{T}\int_{-\infty}^{+\infty}f^{11}f^{22}+f^{12}f^{21}$
$\displaystyle\lesssim\left(\|f^{11}\left(0\right)\|_{L^{1}}+\sup\limits_{0\leqslant
t\leqslant
T}\|f^{11}\left(t\right)\|_{L^{1}}+\int_{0}^{T}\int_{-\infty}^{+\infty}\lvert
G^{1}\rvert\right)$
$\displaystyle\cdot\left(\|f^{21}\left(0\right)\|_{L^{1}}+\sup\limits_{0\leqslant
t\leqslant
T}\|f^{21}\left(t\right)\|_{L^{1}}+\int_{0}^{T}\int_{-\infty}^{+\infty}\lvert
G^{2}\rvert\right)$

provided that the right side is bounded.

###### Proof.

$\partial_{t}\left(\int_{-\infty}^{x}f^{11}\right)+f^{12}=\int_{-\infty}^{x}G^{1}.$

Then

$f^{21}\int_{-\infty}^{x}\partial_{t}f^{11}+f^{12}f^{21}=\int_{-\infty}^{x}G^{1}f^{21},$
(4.12)

$\partial_{t}f^{21}\int_{-\infty}^{x}f^{11}-\partial_{x}f^{22}\int_{-\infty}^{x}f^{11}=G^{2}\int_{-\infty}^{x}f^{11}.$
(4.13)

(4.12)+(4.13):

$\displaystyle\int_{-\infty}^{+\infty}\partial_{t}\left(\int_{-\infty}^{x}f^{11}f^{21}\right)+$
$\displaystyle\int_{-\infty}^{+\infty}\left(f^{12}f^{21}-\partial_{x}f^{22}\int_{-\infty}^{x}f^{11}\right)$
$\displaystyle=\int_{-\infty}^{+\infty}\left(f^{21}\int_{-\infty}^{x}G^{1}+G^{2}\int_{-\infty}^{x}f^{11}\right).$

We have:

$\displaystyle\int_{0}^{T}\int_{-\infty}^{+\infty}f^{11}f^{22}+f^{12}f^{21}=$
$\displaystyle\int_{-\infty}^{+\infty}\left(\int_{-\infty}^{x}f^{11}f^{21}\right)\left(0\right)-\left(\int_{-\infty}^{x}f^{11}f^{21}\right)\left(T\right)$
$\displaystyle+\int_{0}^{T}\int_{-\infty}^{+\infty}\left(f^{21}\int_{-\infty}^{x}G^{1}+G^{2}\int_{-\infty}^{x}f^{11}\right)$
$\displaystyle:=\mathcal{A}_{1}+\mathcal{A}_{2}+\mathcal{A}_{3},$ (4.14)

where

$\displaystyle\lvert\mathcal{A}_{1}\rvert$
$\displaystyle\lesssim\|f^{11}\left(0\right)\|_{L^{1}}\|f^{21}\left(0\right)\|_{L^{1}}+\|f^{11}\left(T\right)\|_{L^{1}}\|f^{21}\left(T\right)\|_{L^{1}},$

$\displaystyle\lvert\mathcal{A}_{2}\rvert$
$\displaystyle\lesssim\int_{0}^{T}\|f^{21}\left(t\right)\|_{L^{1}}\|G^{1}\left(t\right)\|_{L^{1}}$
$\displaystyle\lesssim$ $\displaystyle\sup\limits_{0\leqslant t\leqslant
T}\|f^{21}\left(t\right)\|_{L^{1}}\left(\int_{0}^{T}\int_{-\infty}^{+\infty}\lvert
G^{1}\rvert\right),$

$\displaystyle\lvert\mathcal{A}_{3}\rvert$
$\displaystyle\lesssim\int_{0}^{T}\|f^{11}\left(t\right)\|_{L^{1}}\|G^{2}\left(t\right)\|_{L^{1}}$
$\displaystyle\lesssim$ $\displaystyle\sup\limits_{0\leqslant t\leqslant
T}\|f^{11}\left(t\right)\|_{L^{1}}\left(\int_{0}^{T}\int_{-\infty}^{+\infty}\lvert
G^{2}\rvert\right).$

Based analysis above, we complete the proof. ∎

######  Remark 4.3.

In fact, the quantity $f^{11}f^{22}+f^{12}f^{21}$ is an “inner product” in
some sense. But by coincidence, this quantity equals to the determinant of a
matrix. We can represent the matrix in the following form

$\mathbb{A}=\begin{pmatrix}f^{11}&-f^{12}\\\ f^{21}&f^{22}\end{pmatrix}.$

##  5\. Some calculus from law of equilibrium

In this section, we will calculate some “inner product” induced by balance
law, which paly a crucial role in the estimates of regularity in the later
sections.

From the above remark 4.3, we can calculate some determinant of matrix to
obtain some “inner product” .

$A:=\begin{pmatrix}a&b\\\ c&d\end{pmatrix},$ (5.1)

$a:=\frac{1}{2}<\partial_{x}u,\partial_{x}u>_{g},$

$b:=<J\partial_{x}u,\nabla_{x}\partial_{x}u>_{g},$

$c:=<J\partial_{x}u,\nabla_{x}\partial_{x}u>_{g},$

$d:=<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}-<\nabla_{x}\nabla_{x}\partial_{x}u,\partial_{x}u>_{g}+\frac{1}{4}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}.$

$\displaystyle detA$ $\displaystyle=ad-bc$
$\displaystyle=\frac{1}{2}<\partial_{x}u,\partial_{x}u>_{g}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}-\frac{1}{2}<\partial_{x}u,\partial_{x}u><\nabla_{x}\nabla_{x}\partial_{x}u,\partial_{x}u>_{g}$
$\displaystyle+\frac{1}{8}<\partial_{x}u,\partial_{x}u>_{g}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}-\lvert<J\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}\rvert^{2}$
$\displaystyle=detA_{m}+<\partial_{x}u,\partial_{x}u>_{g}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}-\lvert<J\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}\rvert^{2}$
$\displaystyle+\frac{1}{8}<\partial_{x}u,\partial_{x}u>_{g}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle-\frac{1}{2}\partial_{x}\left(<\partial_{x}u,\partial_{x}u>_{g}<\nabla_{x}\partial_{x}u,\partial_{x}u>_{g}\right),$
(5.2)

where

$detA_{m}:=\lvert<\nabla_{x}\partial_{x}u,\partial_{x}u>_{g}\rvert^{2}.$

##  6\. Global well-posedness for the Schrödinger map flow

From (3.1), it is easy to see $m\left(t\right)=m\left(0\right)$.

First, taking the inner product of both sides of (3.2) with the Riemann metric
with the $\nabla_{t}\partial_{x}u$, we obatin:

$\displaystyle
0=<\nabla_{t}\partial_{x}u,J\nabla_{t}\partial_{x}u>_{g}=<\nabla_{t}\partial_{x}u,\nabla_{x}\nabla_{x}\partial_{x}u>_{g}.$
(6.1)

Integrating both sides of the equation (6.1), and using integration by parts,
according to some simple calculations, we know:

$\displaystyle 0$
$\displaystyle=\int_{0}^{1}<\nabla_{t}\partial_{x}u,\nabla_{x}\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle=-\int_{0}^{1}<\nabla_{x}\nabla_{t}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}+\int_{0}^{1}<\nabla_{t}\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle-\int_{0}^{1}<\nabla_{t}\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle=-\frac{1}{2}\partial_{t}\int_{0}^{1}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}+\int_{0}^{1}<R\left(\partial_{t}u,\partial_{x}u\right)\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}.$

Combining equation (2.1) and the antisymmetry of the curvature, we integrate
by parts to get:

$\displaystyle
0=-\frac{1}{2}\partial_{t}\int_{0}^{1}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}+\int_{0}^{1}<R\left(\partial_{x}u,J\nabla_{x}\partial_{x}u\right)\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle=-\frac{1}{2}\partial_{t}\int_{0}^{1}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}+\frac{1}{2}\int_{0}^{1}<\nabla_{x}\left(R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u\right),\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle-\frac{1}{2}\int_{0}^{1}<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle-\frac{1}{2}\int_{0}^{1}<R\left(\partial_{x}u,J\partial_{x}u\right)\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle=-\frac{1}{2}\partial_{t}\int_{0}^{1}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}+\frac{1}{2}\int_{0}^{1}<\nabla_{x}\left(R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u\right),\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle-\frac{1}{2}\int_{0}^{1}<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
(6.2)

According to some simple calculations, we have:

$\displaystyle\frac{1}{2}\partial_{t}\int_{0}^{1}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle=-\frac{1}{2}\int_{0}^{1}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,\nabla_{x}\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle-\frac{1}{2}\int_{0}^{1}<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle=-\frac{1}{2}\int_{0}^{1}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\nabla_{t}\partial_{x}u>_{g}$
$\displaystyle-\frac{1}{2}\int_{0}^{1}<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle=-\frac{1}{8}\partial_{t}\int_{0}^{1}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle+\frac{1}{8}\int_{0}^{1}<\left(\partial_{t}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle-\frac{1}{2}\int_{0}^{1}<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
(6.3)

We integrate the above equation from $0$ to $t$ to obatin:

$\displaystyle\frac{1}{2}E\left(t\right)+\frac{1}{8}\int_{0}^{1}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle=\frac{1}{2}E\left(0\right)+\frac{1}{8}\int_{0}^{1}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}\big{|}_{t=0}$
$\displaystyle+\frac{1}{8}\int_{0}^{t}\int_{0}^{1}<\left(\partial_{t}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle-\frac{1}{2}\int_{0}^{t}\int_{0}^{1}<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}.$
(6.4)

By sobolev embedding and interpolation, we know:

$\displaystyle\int_{0}^{1}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle\lesssim\int_{0}^{1}\lvert\partial_{x}u\rvert^{4}_{g}$
$\displaystyle\lesssim\|\nabla_{x}\partial_{x}u\|_{L_{g}^{2}}\|\partial_{x}u\|^{3}_{L_{g}^{2}}+\|\partial_{x}u\|^{4}_{L_{g}^{2}}$
$\displaystyle\lesssim
E^{1/2}\left(t\right)m^{3/2}\left(0\right)+m^{2}\left(0\right),$ (6.5)

$\displaystyle\int_{0}^{t}\int_{0}^{1}<\left(\partial_{t}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle\lesssim\int_{0}^{t}\int_{0}^{1}\lvert\nabla_{x}\partial_{x}u\rvert_{g}\lvert\partial_{x}u\rvert^{4}_{g}$
$\displaystyle\lesssim\int_{0}^{t}\|\lvert\nabla_{x}\partial_{x}u\rvert_{g}\|_{L^{2}}\|\lvert\partial_{x}u\rvert^{4}_{g}\|_{L^{2}}$
$\displaystyle\lesssim\int_{0}^{t}\|\lvert\nabla_{x}\partial_{x}u\rvert_{g}\|_{L^{2}}\|\lvert\partial_{x}u\rvert^{2}_{g}\|^{2}_{L^{4}}$
$\displaystyle\lesssim\int_{0}^{t}\|\lvert\nabla_{x}\partial_{x}u\rvert_{g}\|_{L^{2}}\|\partial_{x}\lvert\partial_{x}u\rvert^{2}_{g}\|_{L^{2}}\|\lvert\partial_{x}u\rvert^{2}_{g}\|_{L^{1}}+\|\lvert\nabla_{x}\partial_{x}u\rvert_{g}\|_{L^{2}}\|\lvert\partial_{x}u\rvert^{2}_{g}\|^{2}_{L^{1}}$
$\displaystyle\lesssim
2\sup\limits_{t}E^{1/2}\left(t\right)m\left(0\right)\xi^{1/2}_{1}\left(t\right)t^{1/2}+\sup\limits_{t}E^{1/2}\left(t\right)m^{2}\left(0\right)t,$
(6.6)

$\displaystyle\int_{0}^{t}\int_{0}^{1}<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle\lesssim\int_{0}^{t}\int_{0}^{1}\lvert\nabla_{x}\partial_{x}u\rvert_{g}\lvert\partial_{x}u\rvert^{4}_{g}$
$\displaystyle\lesssim
2\sup\limits_{t}E^{1/2}\left(t\right)m\left(0\right)\xi^{1/2}_{1}\left(t\right)t^{1/2}+\sup\limits_{t}E^{1/2}\left(t\right)m^{2}\left(0\right)t.$
(6.7)

Combining (6), (6), (6), and (6), we get:

$\displaystyle\frac{1}{4}E\left(t\right)$
$\displaystyle\lesssim\frac{1}{2}E\left(0\right)+m^{3}\left(0\right)+m^{2}\left(0\right)$
$\displaystyle+\frac{3}{4}\left(2\sup\limits_{t}E^{1/2}\left(t\right)m\left(0\right)\xi^{1/2}_{1}\left(t\right)t^{1/2}+\sup\limits_{t}E^{1/2}\left(t\right)m^{2}\left(0\right)t\right)$
(6.8)

According to the div-curl lemma 4.1, we have:

$\displaystyle\xi_{2}\left(t\right)$
$\displaystyle\lesssim\int_{0}^{t}\int_{0}^{1}<\partial_{x}u,\partial_{x}u>_{g}\int_{0}^{1}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle-\int_{0}^{t}\int_{0}^{1}<\partial_{x}u,\partial_{x}u>_{g}\int_{0}^{1}<\nabla_{x}\nabla_{x}\partial_{x}u,\partial_{x}u>_{g}$
$\displaystyle+\frac{1}{4}\int_{0}^{t}\int_{0}^{1}<\partial_{x}u,\partial_{x}u>_{g}\int_{0}^{1}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle+\sup\limits_{t}\|<\partial_{x}u,\partial_{x}u>_{g}\|_{L^{1}}\sup\limits_{t}\|<\nabla_{x}\partial_{x}u,J\partial_{x}u>_{g}\|_{L^{1}}$
$\displaystyle+\frac{1}{4}\sup\limits_{t}\|<\partial_{x}u,\partial_{x}u>_{g}\|_{L^{1}}\int_{0}^{t}\int_{0}^{1}\lvert<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}\rvert$
$\displaystyle\lesssim
2\int_{0}^{t}\int_{0}^{1}<\partial_{x}u,\partial_{x}u>_{g}\int_{0}^{1}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle+\frac{1}{4}\int_{0}^{t}\int_{0}^{1}<\partial_{x}u,\partial_{x}u>_{g}\int_{0}^{1}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle+m\left(0\right)\left(\sup\limits_{t}E^{1/2}\left(t\right)m^{1/2}\left(0\right)+\frac{1}{4}\int_{0}^{t}\int_{0}^{1}\lvert<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}\rvert\right)$
(6.9)

By sobolev embedding and interpolation, we know:

$\displaystyle
2\int_{0}^{t}\int_{0}^{1}<\partial_{x}u,\partial_{x}u>_{g}\int_{0}^{1}<\nabla_{x}\partial_{x}u,\nabla_{x}\partial_{x}u>_{g}$
$\displaystyle\leqslant 2m\left(0\right)\sup\limits_{t}E\left(t\right)t$
(6.10)

$\displaystyle\frac{1}{4}\int_{0}^{t}\int_{0}^{1}<\partial_{x}u,\partial_{x}u>_{g}\int_{0}^{1}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}$
$\displaystyle\lesssim\frac{1}{4}m\left(0\right)\int_{0}^{t}\int_{0}^{1}\lvert\partial_{x}u\rvert^{4}_{g}$
$\displaystyle\lesssim\frac{1}{4}m\left(0\right)\int_{0}^{t}\|\partial_{x}\lvert\partial_{x}u\rvert^{2}_{g}\|^{2/3}_{L^{2}}\|\lvert\partial_{x}u\rvert^{2}_{g}\|^{4/3}_{L^{1}}+\|\lvert\partial_{x}u\rvert^{2}_{g}\|^{2}_{L^{1}}$
$\displaystyle\lesssim\frac{1}{2}m^{7/3}\left(0\right)\xi^{1/3}_{1}\left(t\right)t^{2/3}+\frac{1}{4}m^{3}\left(0\right)t$
(6.11)

$\displaystyle\frac{1}{4}\int_{0}^{t}\int_{0}^{1}\lvert<\left(\partial_{x}R\right)\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}\rvert$
$\displaystyle\lesssim\frac{1}{4}\int_{0}^{t}\int_{0}^{1}\lvert\partial_{x}u\rvert^{5}_{g}$
$\displaystyle\lesssim\frac{1}{4}\int_{0}^{t}\|\partial_{x}\lvert\partial_{x}u\rvert^{2}_{g}\|_{L^{2}}\|\lvert\partial_{x}u\rvert^{2}_{g}\|^{3/2}_{L^{1}}+\|\lvert\partial_{x}u\rvert^{2}_{g}\|^{5/2}_{L^{1}}$
$\displaystyle\lesssim\frac{1}{4}m^{5/2}\left(0\right)t+\frac{1}{2}m^{3/2}\left(0\right)\xi^{1/2}_{1}\left(t\right)t^{1/2}$
(6.12)

Besides, from (5), we get:

$\xi_{1}\left(t\right)\leqslant\xi_{2}\left(t\right)+\frac{1}{8}\int_{0}^{t}\int_{0}^{1}\lvert<\partial_{x}u,\partial_{x}u>_{g}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}\rvert$
(6.13)

Similarly, we obatin:

$\displaystyle\frac{1}{8}\int_{0}^{t}\int_{0}^{1}\lvert<\partial_{x}u,\partial_{x}u>_{g}<R\left(\partial_{x}u,J\partial_{x}u\right)\partial_{x}u,J\partial_{x}u>_{g}\rvert$
$\displaystyle\lesssim\frac{1}{8}\int_{0}^{t}\int_{0}^{1}\lvert\partial_{x}u\rvert^{6}_{g}$
$\displaystyle\lesssim\frac{1}{8}\int_{0}^{t}\|\partial_{x}\lvert\partial_{x}u\rvert^{2}_{g}\|^{4/3}_{L^{2}}\|\lvert\partial_{x}u\rvert^{2}_{g}\|^{5/3}_{L^{1}}+\|\lvert\partial_{x}u\rvert^{2}_{g}\|^{3}_{L^{1}}$
$\displaystyle\lesssim\frac{1}{8}m^{3}\left(0\right)t+\frac{1}{4}m^{5/3}\left(0\right)\xi^{2/3}_{1}\left(t\right)t^{1/3}$
(6.14)

Combining (6), (6), (6), (6), (6.13), and (6), we have:

$\displaystyle\xi_{1}\left(t\right)$
$\displaystyle\lesssim\frac{1}{8}m^{3}\left(0\right)t+\frac{1}{4}m^{5/3}\left(0\right)\xi^{2/3}_{1}\left(t\right)t^{1/3}+2m\left(0\right)\sup\limits_{t}E\left(t\right)t$
$\displaystyle+\frac{1}{2}m^{7/3}\left(0\right)\xi^{1/3}_{1}\left(t\right)t^{2/3}+\frac{1}{4}m^{3}\left(0\right)t$
$\displaystyle+m\left(0\right)\left(\sup\limits_{t}E^{1/2}\left(t\right)m^{1/2}\left(0\right)+\frac{1}{4}m^{5/2}\left(0\right)t+\frac{1}{2}m^{3/2}\left(0\right)\xi^{1/2}_{1}\left(t\right)t^{1/2}\right)$
(6.15)

Combining (6), (6), and using the continuous induction method and standard
bootstrap arguments, we know there exists a
$T^{1}=T^{1}\left(m\left(0\right)\right)$, and when
$t\in\left[0,T^{1}\right)$, we have:

$\displaystyle E\left(t\right)\leqslant
C\left(m\left(0\right)\right)\left(E\left(0\right)+1\right).$ (6.16)

From the local well-posedness results of Ding-Wang [7], we can use the
standard interval decomposition method and extension method to obtain a global
solution $u\in
C^{0}\left(\left[0,\infty\right),H^{2}\left(\mathbb{S}^{1},N\right)\right)$.

Acknowledgement:

This work was supported by the National Natural Science Foundation of China
(No. 12171097), the Key Laboratory of Mathematics for Nonlinear Sciences
(Fudan University), Ministry of Education of China, P.R.China. Shanghai Key
Laboratory for Contemporary Applied Mathematics, School of Mathematical
Sciences, Fudan University, P.R. China, and by Shanghai Science and Technology
Program [Project No. 21JC1400600].

## References

  * [1] I. Bejenaru, On Schrödinger maps, Amer. J. Math., 130 (2008), 1033-1065.
  * [2] I. Bejenaru, A. D. Ionescu, C. E. Kenig, and D. Tataru, Global Schrödinger maps in dimensions $d\geqslant 2$: small data in the critical Sobolev spaces, Ann. of Math., 2 (2011), 1443–1506.
  * [3] I. Bejenaru, A. D. Ionescu, C. E. Kenig, and D. Tataru, Equivariant Schrödinger maps in two spatial dimensions, Duke Math. J., 162 (2013), 1967–2025.
  * [4] J. Bourgain, Fourier transform restriction phenomena for certain lattice subsets and applications to nonlinear evolution. I. Schrödinger equations, Geom. Funct. Anal., 3 (1993), 107-156.
  * [5] N. H. Chang, J. Shatah, and K. Uhlenbeck, Schrödinger maps, Comm. Pure Appl. Math., 53 (2000), 590–602.
  * [6] W. Y. Ding, Y. D. Wang, Schrödinger flow of maps into symplectic manifolds, Sci. China Ser. A., 41 no. 7 (1998), 746-755.
  * [7] W. Y. Ding, Y. D. Wang, Local Schrödinger flow into Kähler manifolds, Sci. China Ser. A., 44 no. 11 (2001), 1446–1464.
  * [8] W. Y. Ding, On the Schrödinger flow, pp. 283-291 in Proceedings of the International Congress of Mathematicians (Beijing, 2002), vol. 2, Higher Education Press, 2002.
  * [9] J. X. Huang, Y. D. Wang, and L. F. Zhao, Equivariant Schrödinger map flow on two dimensional hyperbolic space, Discrete and Continuous Dynamical Systems., 40 no. 7 (2020), 4379-4425.
  * [10] A. D. Ionescu, C. E. Kenig, Low-regularity Schrödinger maps. II . Global well-posedness in dimensions $d\geqslant 3$, Comm. Math. Phys., 271 (2007), 523-559.
  * [11] Z. Li, Global Schrödinger map flows to Kähler manifolds with small data in critical Sobolev spaces: Energy critical case, arXiv: 1811. 10924v4.
  * [12] Z. Li, Global Schrödinger map flows to Kähler manifolds with small data in critical Sobolev spaces: High dimensions, arXiv: 1903. 05551v2.
  * [13] H. McGahagan, An approximation scheme for Schrödinger maps, Comm. Partial Differential Equations., 44 (2007), 375-400.
  * [14] I. Rodnianski, Y. A. Rubinstein, and G. Staffilani, On the global well-posedness of the one-dimensional Schrödinger map flow, Anal. PDE., no. 2 (2009), 187-209.
  * [15] S. Wang, Y. Zhou, Global well-posedness for radial extremal hypersurface equation in (1+3)-dimensional Minkowski space-time in critical Sobolev space, arXiv:2212.08828.