(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Tencent AI Lab
11email: {ffeishen, huye, siboliu, junejzhang, xvencewang, haroldhan,
<EMAIL_ADDRESS>
# Boosting Consistency in Story Visualization with Rich-Contextual Conditional
Diffusion Models

Fei Shen  Hu Ye  Sibo Liu  Jun Zhang Corresponding author Cong Wang  Xiao Han
Wei Yang

###### Abstract

Recent research showcases the considerable potential of conditional diffusion
models for generating consistent stories. However, current methods, which
predominantly generate stories in an autoregressive and excessively caption-
dependent manner, often underrate the contextual consistency and relevance of
frames during sequential generation. To address this, we propose a novel Rich-
contextual Conditional Diffusion Models (RCDMs), a two-stage approach designed
to enhance story generation’s semantic consistency and temporal consistency.
Specifically, in the first stage, the frame-prior transformer diffusion model
is presented to predict the frame semantic embedding of the unknown clip by
aligning the semantic correlations between the captions and frames of the
known clip. The second stage establishes a robust model with rich contextual
conditions, including reference images of the known clip, the predicted frame
semantic embedding of the unknown clip, and text embeddings of all captions.
By jointly injecting these rich contextual conditions at the image and feature
levels, RCDMs can generate semantic and temporal consistency stories.
Moreover, RCDMs can generate consistent stories with a single forward
inference compared to autoregressive models. Our qualitative and quantitative
results demonstrate that our proposed RCDMs outperform in challenging
scenarios. The code and model will be available at
https://github.com/muzishen/RCDMs.

###### Keywords:

Story visualization Diffusion model Rich-Contextual

##  1 Introduction

Story visualization [16, 24, 21] aims to depict a continuous narrative through
multiple captions and reference clips. It has profound applications in game
development and comic drawing. Due to the technological leaps in generative
models such as generative adversarial network (GAN) [4, 22, 42, 39, 5, 11] and
diffusion model [25, 45, 29, 31, 41], text-to-image synthesis methods [43, 26,
46, 40, 15] can now generate visually faithful images through text
descriptions. However, given multiple captions to generate a continuous story
with style and temporal consistency still poses significant challenges.

Existing methods typically employ autoregressive generation and can be broadly
classified into GAN-based [16, 19, 18, 14] and diffusion model-based [21, 20,
24, 32]. GAN-based methods typically comprise a text encoder, image generator,
image separation, and story discriminator. These components work together to
maintain the consistency of the entire sequence of images. However, the images
generated by these methods often display distorted objects, mismatched
semantics, and localized blurring, especially when creating images from
complex scene descriptions. Subsequently, GAN-based methods [19, 18]
progressively focus on generating more consistent images by improving the
performance of the text encoder, such as caption enhancement [19], structured
text parsing [18], and ID attention mechanism [3]. While these methods [19,
17, 18] can generate images that satisfy the requirements of character
consistency, they often struggle to maintain a consistent style and capture
realistic scene details. Furthermore, since the adversarial nature of the min-
max objective, GAN-based methods can be prone to unstable training dynamics,
which limits the diversity of the stories.

Figure 1: (a) Existing methods, which employ autoregressive models and rely on
the current caption for guidance, suffer from weak conditioning, leading to a
decrease in the consistency of the generated story. (b) RCDMs initially
predict the frame-contextual information at the feature level, then
simultaneously infuse image-level and feature-level contextual information to
generate coherent stories in a single forward inference.

Advanced diffusion models like Imagen [30] and Stable Diffusion [27] have
recently demonstrated unprecedented text-to-image synthesis capabilities.
Story visualization based on diffusion models [21, 20, 24, 32] generates
better consistent images through a multi-step denoising process, using the
features of the current caption and historical frames as conditions. However,
these methods [21, 20, 24, 32] only consider clip information at the feature
level due to the injection of the current caption and the known clip into the
model via the inherent cross-attention module, overlooking the frame
information of known at the image level and text information of the unknown
clip. Besides, from Fig. 1 (a), GAN-based and diffusion model-based
autoregressive generation methods [21, 20, 38, 13] depend on the current
caption for frame-by-frame forward generation. This reliance can lead these
methods to easily overlook the rich contextual information within the captions
and the consistency of the story.

This paper presents Rich-contextual Conditional Diffusion Models (RCDMs) to
tackle the issues above through two stages, as shown in Fig. 1 (b). Firstly,
we propose a frame-prior transformer diffusion model to predict the semantic
embeddings of frames within an unknown clip. This prediction task is
significantly less complex than directly generating consistent a story. It
enables the model to focus solely on the task at the semantic feature level,
thereby circumventing the rigorous consistency requirements associated with
story generation at the image level. The frame-prior transformer diffusion
model takes the frames of the known clip and all captions as conditions,
leveraging a combination of multiple cascaded transformer blocks and frame
attention blocks to predict the frame semantic embedding of the unknown clip.
In the second stage, instead of over-reliance on caption conditions, we
integrate the text embeddings of all captions, the frame semantic features of
the unknown clip, and the images of the known clip to serve as a rich
contextual guide for generating frames in the unknown clip. Based on the above
rich-contextual conditional, we devise a frame-contextual 3D diffusion model
to generate consistent stories by jointly infusing conditions at both the
image and feature levels. Besides, RCDMs can generate consistent stories with
a single forward inference compared to autoregressive models. The main
contributions are briefly summarized as follows:

  * $\bullet$

We propose a frame-prior transformer diffusion model that, by using the frames
from the known clip and all captions as conditions, predicts the semantic
embeddings of frames in an unknown clip, thereby providing a rich semantic
feature for the next stage.

  * $\bullet$

We devise a frame-contextual 3D diffusion model that jointly infuses image-
level and feature-level rich-contextual conditional to generate stories with
stylistic and temporal consistency.

  * $\bullet$

We conduct comprehensive experiments on two datasets to demonstrate the
competitive performance of proposed RCDMs. Additionally, we perform a user
study to evaluate the superiority of RCDMs qualitatively.

##  2 Related Work

###  2.1 Text-to-Image Synthesis

Recent advances in text-to-image synthesis mainly focus on generative
adversarial networks (GANs) [22, 39, 44, 34, 43] and diffusion models. For
example, MirrorGAN [22] introduces the concept of redescription, progressively
enhancing images’ diversity and semantic consistency by focusing on local
words and global captions as conditions. Similarly, AttnGAN [39] synthesizes
fine details by calculating fine-grained image-caption matching losses in
different image sub-regions, focusing on relevant words in natural language
descriptions. Moreover, StackGAN [44] uses a two-stage approach, first
sketching the shape and color of objects based on the caption and then
generating images based on the results of the first stage and the caption. On
the other hand, XMC-GAN [43] employs an attention self-modulation generator
and a contrastive discriminator to capture and reinforce the correspondence
between captions and images, maximizing their mutual information to generate
images. While these methods [22, 39, 44, 43] can generate images that meet
semantic requirements, GAN training heavily depends on selecting
hyperparameters and can easily lose scene details.

Unlike GANs, diffusion models [33, 36, 9, 28, 35] do not suffer from mode
collapse and potentially unstable training and can generate more diverse
images. For example, DALL-E2 [25] introduces a two-stage model. Initially, it
generates learned CLIP image embeddings in an autoregressive fashion, using
text descriptions as a guide via a prior model. Subsequently, a diffusion
model-based decoder is employed to generate semantically consistent images
based on these embeddings. Following this, Imagen [29] aims to augment text
comprehension capabilities by utilizing a large transformer language model,
thereby enhancing image-text alignment and the fidelity of the samples. Stable
Diffusion [28] presents a novel approach of applying diffusion models in the
latent space, striking an almost perfect balance between simplification and
detail preservation. However, these text-to-image methodologies [29, 6, 28,
37] primarily concentrate on aligning individually generated images with their
corresponding text descriptions. They overlook the crucial elements of style
and temporal consistency across multiple frames, which are essential for
effective story visualization.

###  2.2 Story Visualization

StoryGAN [16] is a pioneer in story visualization, proposing a sequential
conditional GAN framework with a context encoder that can dynamically track
the story flow and a story-level discriminator. DuCo-StoryGAN [19] presents a
dual learning framework to enhance the semantic consistency between the story
and the generated images by improving the captions. VLC-StoryGAN [18]
introduces a Transformer-based recursive architecture to encode structured
text input through constituency parsing trees while using common sense
information to enhance the structured input of the text, making it more in
line with the sequence structure of the story. Similarly, Word-Level SV [14]
also focuses on text input and proposes a discriminator with fused features
and extended spatial attention to improve image quality and story consistency.
Besides, VP-CSV [3] devises a two-stage framework that predicts character
tokens and the remaining tokens to better ensure character consistency.

Story visualization [21, 1, 20, 32] has achieved great development during
these years, especially with the unprecedented success of diffusion model. For
example, StoryDALL-E [20] explores story visualization through an
autoregressive Transformer, focusing on full model fine-tuning based on pre-
trained models and parameter-efficient adaptive adjustments based on prompts.
AR-LDM [21] proposes an autoregressive latent diffusion model based on
historical captions and generated images to align and enhance story
consistency. Similarly, Story-LDM [24] proposes an autoregressive diffusion
framework with a visual memory module to capture character and background
context implicitly. However, the above autoregressive methods [16, 1, 20, 32],
including GAN-based and diffusion model-based, rely on the current caption for
guidance and suffer from weak conditioning, leading to a decrease in the
consistency of the generated story.

##  3 Method

The proposed Rich-contextual Conditional Diffusion Models (RCDMs) aims to
generate consistent stories by jointly infusing rich contextual conditions at
both the image and feature levels. In this section, we introduce the following
three aspects: preliminaries (Section 3.1), frame-prior transformer diffusion
model (Section 3.2), and frame-contextual 3D diffusion model (Section 3.3).

###  3.1 Preliminaries

Diffusion models represent a category of generative models trained to reverse
a diffusion process. This process systematically adds Gaussian noise to the
data through a fixed Markov chain of timestep ${t}$, while concurrently
training a denoising model to generate samples starting from the Gaussian
noise. To learn such a diffusion model ${\epsilon}_{\theta}$ parameterized by
${\theta}$, for each timestep ${t}$, the training objective usually adopts a
mean square error loss ${L_{DM}}$, as follows,

$L_{DM}=\mathbb{E}_{{x}_{0},{\epsilon},{c},t}\|{\epsilon}-{\epsilon}_{\theta}\big{(}{x}_{t},{c},t\big{)}\|^{2},\vspace{-5pt}$
(1)

where ${x}_{0}$ represents the real data with an additional condition ${c}$.
The timestep ${t}$ of diffusion process is denoted by ${t\in[0,T]}$. The noisy
data at ${t}$ step, ${x}_{t}$, is defined as
$\alpha_{t}{x}_{0}+\sigma_{t}{\epsilon}$, where $\alpha_{t}$ and $\sigma_{t}$
are predefined functions of $t$ that determine the diffusion process. Once the
model ${\epsilon}_{\theta}$ is trained, images can be synthesized from random
noise through an iterative process.

During the sampling stage, the predicted noise is calculated based on the
predictions of both the conditional model ${\epsilon}_{\theta}({x}_{t},{c},t)$
and the unconditional model ${\epsilon}_{\theta}({x}_{t},t)$ via classifier-
free guidance [10].

$\hat{{\epsilon}}_{\theta}({x}_{t},{c},t)={w}{\epsilon}_{\theta}({x}_{t},{c},{t})+(1-{w}){\epsilon}_{\theta}({x}_{t},{t}).\vspace{-5pt}$
(2)

Here, ${w}$ is the guidance scale used to adjust condition ${c}$.

Figure 2: Illustration of the frame-prior transformer diffusion model. The
frame-prior transformer diffusion model predicts the frame semantic embeddings
of unknown clips by aligning the semantic correlations between the captions
and frames of known clips.

###  3.2 Frame-Prior Transformer Diffusion Model

In the first stage, we propose a frame-prior transformer diffusion model to
predict the frame semantic embeddings of unknown clips by aligning the
semantic correlations between the captions and frames of known clips. As shown
in Fig. 2, it is composed of a frozen image encoder, a frozen text encoder, an
unshared linear layer, and a stack of multiple transformer blocks and frame
attention blocks. Here, we utilize the pooled representation extracted from
the CLIP image encoder as the frame semantic embeddings for unknown clips. Our
choice is inspired by the capability of CLIP [23], which is trained on a
large-scale dataset of image-text pairs through contrastive learning. This
enables it to encapsulate a rich variety of image content and stylistic
information, pivotal in steering the subsequent process of story synthesis.

Specifically, we first split the ground truth into known and unknown clips and
introduce noise into the unknown clips. Subsequently, all clips (known and
unknown) and all captions are fed into the frozen image encoder and the frozen
text encoder, respectively. We then use an unshared linear layer to obtain the
pooled visual representation $v_{p}$ and hidden visual representation $v_{h}$
of the image and pooled textual representation $t_{p}$ and hidden textual
representation $t_{h}$ of the caption. For consecutive frames, we input them
as a 4D tensor $x\in\mathbb{R}^{b\times f\times n\times d}$, where $b$, $f$,
$n$, and $d$ represent the batch size, temporal length, token length and each
token dimension, respectively. When the internal feature map passes through
the transformer block, the temporal length $f$ is reshaped to the batch size
$b$ and ignored, allowing the model to process each frame independently. We
reshape the feature map back into a 4D tensor after the transformer block,
frame attention block reshape $n$ into $b$ to learn and maintain the temporal
consistency of the story, and then reshape it back after the module while
ignoring the temporal length. Inspired by [2, 7], the frame attention block
consists of several self-attention modules. This allows the model to guide
self-attention along the temporal length $f$, effectively capturing the
dynamic content within the narrative. Besides, we add an extra embedding to
represent the unnoised frame semantic embedding of the known clip to be
predicted.

Following unCLIP [25], the frame-prior transformer diffusion model is trained
to predict the unnoised frame semantic embedding directly rather than the
noise added to the frame embedding. The training loss $L_{prior}$ of frame-
prior transformer diffusion model ${x}_{\theta}$ is defined as follows,

$L_{\text{prior}}=\mathbb{E}_{{x}_{0},{\epsilon},{v_{p}},{v_{h}},{t_{p}},{t_{h}},{t}}\left\|{x_{0}}-{x_{\theta}}\left({x_{t}},{v_{p}},{v_{h}},{t_{p}},{t_{h}},{t}\right)\right\|^{2}.$
(3)

Once the model learns the conditional distribution, the inference is performed
according to Eq. 4, as follows, $w$ is the guidance scale.

$\hat{{x}}_{\theta}\left({x_{t}},{v_{p}},{v_{h}},{t_{p}},{t_{h}},{t}\right)=w{x}_{\theta}\left({x_{t}},{v_{p}},{v_{h}},{t_{p}},{t_{h}},{t}\right)+(1-w){x}_{\theta}\left({x}_{t},t\right).\vspace{-5pt}$
(4)

Figure 3: Overview of the frame-contextual 3D diffusion model. The frame-
contextual 3D diffusion model infuses both image-level and feature-level
context information to generate stories with stylistic and temporal
consistency.

###  3.3 Frame-Contextual 3D Diffusion Model

In the second stage, we propose a frame-contextual 3D diffusion model that
utilizes a variety of rich contextual conditions, including reference images
from the known clip, the anticipated frame semantic embedding from the unknown
clip, and text embeddings from all captions, to generate consistent stories.
Therefore, our method can generate style and temporal consistency stories by
jointly injecting these rich contextual conditions at the image and feature
levels. From Fig. 3, the frame-contextual 3D diffusion model comprises a
frozen VAE, a model stacked with multiple U-Net blocks and frame attention
(FA) blocks, a multimodal interaction module, and a stacked semantic module.
Here, the design of the FA block is the same as in the previous stage, and
focus on the correlation between frames to maintain consistency across frames.

Image-Level Condition. Since the VAE [12] enables almost lossless
reconstruction, introducing known clip conditions at the image level can steer
the story continuity, which has been neglected in previous studies.
Specifically, similar to the previous stage, we first divide the ground truth
into known and unknown clips and directly mask the unknown clip. The ground
truth, known clip, and masked unknown clip are all fed into the frozen VAE
encoder to extract latent space features. We then concatenate the latent space
features of the known clip and masked unknown clip along the width dimension,
and simultaneously concatenate them with the latent space features of the
ground truth along the channel dimension. Moreover, if the images in the known
clip have 0-pixel values similar to the mask, it can easily mislead the model
into thinking these areas need to be generated. Therefore, we introduce a
single-channel marker symbol that matches the width and height of the
concatenated features (omitted in the figure). We use 0 and 1 to represent
masked and unmasked pixels, respectively. This approach helps to reduce model
confusion and ensures accurate identification of the generation area.

Feature-Level Condition. Existing methods depend solely on the current
caption’s text embeddings, lacking in maintaining the contextual consistency
of the overall narrative. Contrarily, we incorporate the frame semantic
embedding of the unknown clip obtained from the previous stage and the text
embeddings of all captions as extra feature-level conditions. These rich
contextual conditions facilitate the generation of consistent narratives.
Furthermore, to enhance the features of the text and frames in both known and
unknown segments, we separately introduce a multimodal interaction module and
a semantic stacking module. Specifically, we first employ frozen text and
image encoders (omitted in the figure) to extract the text embeddings from all
captions, and the image embeddings from known clips, respectively. We further
divide text embeddings $F_{t}\in\mathbb{R}^{f\times n\times d}$ along the
temporal dimension into $F_{t}^{k}\in\mathbb{R}^{f^{k}\times n\times d}$ and
$F_{t}^{u}\in\mathbb{R}^{f^{u}\times n\times d}$ based on known/unknown clips
to align the text and image modalities, where $f=f^{k}+f^{u}$. For known clip,
we first feed the text embeddings $F_{t}^{k}$ and image embeddings
$F_{v}^{k}\in\mathbb{R}^{f^{k}\times n\times d}$ of the known clip into a
multimodal interaction module. The multimodal interaction module comprises two
projection layers and a multi-head cross-attention module. The two projection
layers are respectively used to project the embeddings of images and text onto
the same dimension. Then, the projected text and image embeddings are then fed
into the multi-head cross-attention module to obtain the interaction features
$F_{i}^{k}\in\mathbb{R}^{f^{k}\times n\times d}$ of the known clip.

Considering the discrepancy in the number of tokens and dimensions between the
frame semantic embedding and the text embedding, we introduce a semantic
stacking module specifically for unknown clips. This module aims to enhance
the feature-level semantic information of the unknown clip. The semantic
stacking module is composed of a projection layer and a multi-head cross-
attention module. The projection layer’s role is to convert the text embedding
of the unknown clip into the same feature dimension as the image embedding of
the unknown clip, which was obtained from the previous stage. Assume that
$F^{u}_{v}\in\mathbb{R}^{f^{u}\times 1\times d}$ and
$F^{u}_{t}\in\mathbb{R}^{f^{u}\times n\times d}$ respectively are image
embedding and projected text embedding of the unknown clip. To align text and
image modal, we first obtain the extracted pooled representation
$e^{g}\in\mathbb{R}^{f^{u}\times 1\times d}$ of unknown clip captions. Then,
the pooled representation $F^{u}_{v}$ of the unknown clip is fed into the
multi-head cross-attention module to obtain the interaction features of the
unknown clip. These are then stacked with the hidden representation of the
unknown clip along the length dimension to obtain the stacked features
$F_{s}^{u}\in\mathbb{R}^{f^{u}\times n\times d}$ of the unknown clip. Finally,
the interaction features $F_{i}^{k}$ and the stacked features $F_{s}^{u}$ are
concatenated along the temporal dimension and fed into the U-Net block via
inherent cross-attention mechanism in diffusion models.

The loss function $L_{\mathrm{contextual}}$ of frame-contextual 3D diffusion
model according to Eq. 5, as follows. Here, ${F}_{I}$ denotes the feature of
the image-level conditional.

$L_{\mathrm{contextual}}=\mathbb{E}_{{x}_{0},{\epsilon},{F}_{I},{F}_{i}^{k},{F}_{s}^{u},t}\|{\epsilon}-{\epsilon}_{\theta}\big{(}{x}_{t},{F}_{I},{F}_{i}^{k},{F}_{s}^{u},t\big{)}\|^{2}.\vspace{-5pt}$
(5)

In the inference stage, we also use classifier-free guidance according to Eq.
6.

$\begin{split}\hat{{\epsilon}}_{\theta}({x}_{t},{F}_{I},{F}_{i}^{k},{F}_{s}^{u},t)=w{\epsilon}_{\theta}({x}_{t},{F}_{I},{F}_{i}^{k},{F}_{s}^{u},t)+(1-w){\epsilon}_{\theta}({x}_{t},t).\end{split}\vspace{-5pt}$
(6)

##  4 Experiments

Datasets. We conduct experiments on the FlintstonesSV [18] and PororoSV [16]
datasets. The former contains 20,132 training sequences and 2,309 testing
sequences, encompassing 7 main characters. PororoSV includes 10,191 training
sequences and 2,208 testing sequences, covering 9 main characters. Following
[21, 24, 32], for story visualization, we designated the first frame as the
source frame and generated the remaining four based on this source frame.

Metrics. We conduct a comprehensive evaluation of the model, considering both
objective and subjective metrics. Objective indicators include classification
accuracy of characters (Char-Acc) and F1-score of characters (Char-F1), both
extracted using InceptionV3. Additionally, we also consider the fréchet
inception distance (FID) [8] score. This metric provides a quality assessment
by comparing the distribution of feature vectors derived from both real and
generated images. In contrast, subjective assessments prioritize user-oriented
metrics [32], including the percentage of visual quality, text-image
relevance, and temporal consistency.

Table 1: Quantitative comparison of the proposed RCDMs with several SOTA models. Datasets | Methods | FID ($\downarrow$) | Char-Acc ($\uparrow$) | Char-F1 ($\uparrow$)
---|---|---|---|---
FlintstonesSV [18] |  LDM [27] | 82.53 | 9.17 | 22.68
StoryGAN [16] | 74.63 | 16.57 | 39.68
Story-DALL-E [20] | 26.49 | 55.19 | 73.43
Story-LDM [24] | 24.24 | 57.19 | 76.59
AR-LDM [21] | 19.28 | 62.58 | 79.25
RCDMs (Ours) | 14.96 | 78.44 | 85.51
PororoSV [16] |  LDM [27] | 64.52 | 4.31 | 12.74
StoryGAN [16] | 49.27 | 9.34 | 18.59
CP-CSV [3] | 40.56 | 10.03 | 21.78
DuCo-StoryGAN [19] | 37.15 | 13.97 | 38.01
Story-DALL-E [20] | 35.90 | 27.14 | 42.45
Story-LDM [24] | 26.64 | 29.19 | 47.56
AR-LDM [21] | 17.40 | 35.18 | 55.29
RCDMs (Ours) | 16.25 | 41.48 | 59.03

Implementations. We perform our experiments on 8 NVIDIA V100 GPUs. Our
configurations can be summarized as follows, (1) In the frame-prior
transformer diffusion model, there are 10 layers of cascaded transformer and
frame attention blocks, and the width of each transformer block is 2048. For
the frame-contextual 3D diffusion model, we use the pretrained Stable
Diffusion V1.5 111https://huggingface.co/runwayml/stable-diffusion-v1-5 and
modify the first convolution layer to adapt additional conditions. (2) We
employ the AdamW optimizer with a fixed learning rate of $1e^{-5}$ in all
stages. (3) Following [21, 24], we train our models using images of sizes 512
$\times$ 512 for FlintstonesSV and PororoSV dataset. (4) We employ a data
augmentation strategy of dropping images in all two stages, with the drop
count ranging from 0 to 5. We substitute the dropped images with black images.
(5) In the inference stage, we use the DDIM [9] sampler with 20 steps and set
the guidance scale $w$ to 2.0 for RCDMs on all stages.  Please refer to
Appendix B for more detail.

Figure 4: Qualitative comparisons with several state-of-the-art models on the
FlintstonesSV and PororoSV datasets.  Please see Appendix C for more examples.

###  4.1 Quantitative and Qualitative Results

We quantitatively compare our proposed RCDMs with several state-of-the-art
methods, including LDM [27], StoryGAN [16], Story-LDM [24], Story-DALL-E [20],
AR-LDM [21], CP-CSV [3], and DuCo-StoryGAN [19].

Quantitative Results. As shown in Table 1, firstly, since LDM [27] generates
each image based solely on individual captions, it performs significantly
worse than all other methods on three metrics. Secondly, compared to GAN and
diffusion model methods, our approach outperforms other models on all three
metrics in FlintstonesSV. For example, compared to StoryGAN [16], which
employs story dynamic tracking, RCDMs score 61.87% and 45.83% higher on Char-
Acc and Char-F1 metrics, respectively. This demonstrates the superiority of
proposed RCDMs in understanding story details through all captions and then
generating all story images at once, as opposed to StoryGAN, which uses an
autoregressive approach to understand captions frame by frame. Lastly,
compared to AR-LDM [21], which also relies on a diffusion model, RCDMs perform
better on the FID metric. Even though AR-LDM already scores well, RCDMs show
better performance, indicating that injecting more semantic information
through the first-stage model can enrich the generation of image details.
Moreover, on Char-Acc and Char-F1 metrics, RCDMs significantly outperform AR-
LDM. This is because we not only introduce more semantic information at the
feature level but also inject more contextual information at the image level.

The comparison results for PororoSV are summarized in Table 1. Notably,
consistent with the trend presented in the FlintstonesSV dataset, proposed
RCDMs outperform all SOTA methods, achieving the best FID, Char-Acc, and
Char-F1. Specifically, compared to the best-performing GAN method, i.e., DuCo-
StoryGAN [19], which enhances the current caption to improve text semantic
understanding, RCDMs inject context by understanding the complete story
semantics. Similarly, compared to AR-LADM [21], we surpass it by 6.30% and
3.74% on Char-Acc and Char-F1, respectively. These results indicate that the
simultaneous use of a frame-prior transformer diffusion model to obtain frame
semantic information of the unknown clip and the injection of image-level
conditions are crucial for understanding and generating stories.

Figure 5: Results of user study. Higher values indicate better performance.

Qualitative Results. As some methods have yet to be open-sourced, we
qualitatively compared RCDMs with StoryGAN [16], Story-DALL-E [20], AR-LDM
[21], and Story-LDM [24] on the FlintstonesSV and PororoSV datasets. As shown
in Fig. 4, several conclusions can be drawn from the results: (1) RCDMs
significantly outperform other SOTA methods regarding image quality. For
example, on the FlintstonesSV dataset, Story-DALL-E, AR-LDM, and the second
frame of Story-LDM, and the third frame of StoryGAN all exhibit scenes and
character limbs that do not match. (2) Regardless of whether it is the
FlintstonesSV dataset or the PororoSV dataset, our proposed RCDMs perform best
regarding character consistency. For example, StoryGAN performs poorly in
terms of character consistency on text-image pairs, such as the second and
third frames on FlintstonesSV dataset, and the first and fourth frames on
PororoSV dataset. A similar situation also occurs on Story-DALL-E. (3) Only
our method can generate story images that reasonably align with the text on
complex micro-actions and expressions. For example, the text prompt for the
fourth frame on the FlintstonesSV dataset is ’laughs and tears up a little,’
and the third frame on the PororoSV dataset is ’look sad’. This can be
attributed to our method’s ability to enhance the semantic consistency of
image-text pairs. (4) Regarding visual consistency, although Story-LDM
integrates an attention-memory module to handle context, it cannot produce a
consistent style scene in a complete story. In contrast, the results of RCDMs
have pleasing visual effects and the ability to maintain temporal consistency,
primarily due to RCDMs injecting rich contextual conditions, including
reference images of the known clip, the predicted frame semantic embedding of
the unknown clip, and text embeddings of all captions at both the feature and
image levels. In summary, our method can always produce more realistic and
consistent story images, proving that RCDMs bring significant advantages by
introducing rich context conditions.

User Study. The above quantitative and qualitative comparison results
demonstrate the substantial advantages of our proposed RCDMs in generating
results. However, the task of synthesizing story visualization is often human
perception-oriented. Therefore, we also conduct a user study involving 20
volunteers with computer vision backgrounds. The volunteers are asked to
choose which method is better regarding visual quality, text-image relevance,
and style/temporal consistency in the generated story. Please refer to
Appendix C for more detail.

As shown in Fig. 5, the higher the score in the three indicators of this
study, the better the performance. RCDMs offer commendable performance on all
three fundamental indicators. For example, the text-image relevance and
consistency scores of RCDMs are 51.570% and 76.855%, respectively, which are
nearly 35.515% and 65.650% higher than the second-best model. This
demonstrates the significant advantages of RCDMs in multimodal semantic
understanding and consistency due to our conditional injection at both the
image and feature levels. In addition, our visual quality score is 67.635%,
indicating that participants prefer our method, demonstrating better story
visualization quality. More detail refer to appendix 0.B.

Table 2: Ablation study on FlintstonesSV dataset. Here, IC stands for image-level conditional. MIM and SSM, respectively, denote the multimodal interaction module and semantic stacking module.  Settings | Components | FlintstonesSV
---|---|---
Stage1 | Stage2 | FID ($\downarrow$) | Char-Acc ($\uparrow$) | Char-F1 ($\uparrow$)
IC | MIM | SSM
LDM [27] | - | - | - | - | 82.53 | 9.17 | 22.68
B0 | ✓ | ✗ | ✗ | ✗ | 21.81 | 56.44 | 70.32
B1 | ✓ | ✓ | ✗ | ✗ | 19.46 | 61.88 | 78.03
B2 | ✓ | ✓ | ✓ | ✗ | 18.36 | 72.53 | 82.06
B3 | ✓ | ✓ | ✗ | ✓ | 17.94 | 73.58 | 82.87
B4 | ✗ | ✓ | ✓ | ✓ | 16.51 | 75.73 | 83.96
Ours | ✓ | ✓ | ✓ | ✓ | 14.96 | 78.44 | 85.51

###  4.2 Ablation Study

We further devise several variants to demonstrate the efficacy of each module
proposed in this study. All these variants belong to the RCDMs framework but
encompass different configurations. B0 denotes the sole use of the frame-prior
transformer diffusion model, devoid of image-level conditions, with feature-
level conditions directly infused into the inherent cross-attention of SD
using caption features. B1 built on the foundation of B0, incorporates image-
level conditions. B2, an extension of B1, additionally employs a multimodal
interaction module. B3 also building upon B1, further utilizes a semantic
stacking module. B4 indicates that there is no stage1 prior model and in the
SSM without stage1, the QKV of attention all originate from the features of
the known clips.

As shown in Table 2, firstly, compared to LDM [27], B0, which adopts the frame
attention module, can significantly improve the consistency of characters and
backgrounds. Subsequently, when the image-level condition setting is added, B1
is 5.44% and 7.71% higher than B0 on Char-Acc and Char-F1, respectively. This
demonstrates that the injection of known clip images can enhance the model’s
contextual information. Secondly, when the MIM and SSM are introduced based on
B1, B2 and B3 are 10.65% and 11.70% higher than B1 on Char-Acc, respectively.
At this point, B2 and B3 have achieved highly competitive performance on the
FlintstonesSV dataset. These results indicate that they also contribute
constructively to the success of our RCDMs. Our proposed RCDMs outperform the
B4 setting, highlighting the importance of acquiring prior features in the
first stage of our method. Finally, when the frame semantic embeddings of the
unknown clip predicted by the frame-prior transformer diffusion model are
added, FID, Char-Acc, and Char-F1 all improve better, especially FID. This
shows that the frame-prior model can better help RCDMs generate stories with
semantic and temporal consistency.

Figure 6: Branching storyline. Generating consistent stories by different
captions.

Figure 7: Inference speed comparison with different SOTA methods.

###  4.3 Additional Results

Branching Storyline. We add an experiment to validate the diversity in
generating branching narratives. Specifically, we used one reference image and
two sets of distinct captions as the story descriptions. We also incorporated
pronouns like “they" to check if the story could maintain coherence. From Fig.
7, we generated two sets of different story images, and the results
demonstrate consistency in style and sequence. Moreover, we observed that for
the reference ’they’, RCDMs can generate these two characters based on the
previously parsed storylines.

Inference Speed. We also conduct a comparison experiment on inference speed,
including AR-LDM [21] and Story-LDM [24], which are also based on the
diffusion model. All experiments are conducted on the same V100 GPU to ensure
a fair comparison. From Fig. 7, the mean inference time for a story by AR-LDM
and Story-LDM is 14.6 seconds and 16.9 seconds, respectively, while proposed
RCDMs only take 12.7 seconds, even though it’s a two-stage model. Since AR-LDM
uses a more heavy multimodal model, and Story-LDM employs an attention module
with memory storage, both of which slow down the inference speed. Notably,
while AR-LDM and Story-LDM generate frames one by one using an autoregressive
architecture, RCDMs infers all story images in a single forward pass. These
results demonstrate the architectural superiority of RCDMs.

Caption-Only Generation. RCDMs also support caption-only generation, as it
adopts a strategy of randomly dropping images during the training process. In
contrast, other SOTA methods typically require an additional model to be
trained to accommodate this scenario. Fig. 8 displays the results generated by
RCDMs using only captions as guidance. Due to the lack of open-source weights
from SOTA methods for comparison, we can only assess RCDMs’ performance
independently. The results suggest that RCDMs can generate story images that
maintain consistency in style and sequence under different
scenarios/characters.

Figure 8: Qualitative results of RCDMs for caption-only story generation.

##  5 Conclusion

This paper presents Rich-contextual Conditional Diffusion Models (RCDMs) to
enhance style and temporal consistency at both the image and feature levels
for story visualization. In the first stage, the frame-prior transformer
diffusion model is presented to predict the frame semantic embedding of the
unknown clip by aligning the semantic correlations between the captions and
frames of the known clip. The second stage establishes a robust model with
rich contextual conditions, including reference images of the known clip, the
predicted frame semantic embedding of the unknown clip, and text embeddings of
all captions. By jointly injecting these rich contextual conditions, RCDMs can
generate style and temporal consistency stories. Both qualitative and
quantitative results demonstrate that RCDMs perform well in challenging
scenarios.

Limitations. Current methods, including RCDMs, typically achieve consistent
story generation on a closed-set dataset, which limits the variety of
characters and scenes. For future work, we wil explore methods with open-set
generation capabilities to allow for a broader range of characters and scenes.

##  6 Ethics and Broader Impacts

This paper presents a synthesis method that creates novel comic narratives
from captions and reference images. While there’s potential for misuse in
fabricating misleading content, this risk is common to all story generation
techniques. However, the cartoon nature of our outputs significantly reduces
ethical concerns related to image and video generation, as they can’t be
mistaken for reality. Moreover, strides have been made in research to detect
and prevent malicious tampering. Our work aids this domain, balancing the
technology’s value and the risks of unrestricted access, ensuring the safe and
beneficial use of RCDMs.

## References

  * [1] Ahn, D., Kim, D., Song, G., Kim, S.H., Lee, H., Kang, D., Choi, J.: Story visualization by online text augmentation with context memory. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3125–3135 (2023)
  * [2] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your latents: High-resolution video synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22563–22575 (2023)
  * [3] Chen, H., Han, R., Wu, T.L., Nakayama, H., Peng, N.: Character-centric story visualization via visual planning and token alignment. arXiv preprint arXiv:2210.08465 (2022)
  * [4] Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., Bharath, A.A.: Generative adversarial networks: An overview. IEEE signal processing magazine 35(1), 53–65 (2018)
  * [5] Ding, X., Wang, Y., Xu, Z., Welch, W.J., Wang, Z.J.: Ccgan: Continuous conditional generative adversarial networks for image generation. In: International conference on learning representations (2020)
  * [6] Ge, S., Park, T., Zhu, J.Y., Huang, J.B.: Expressive text-to-image generation with rich text. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7545–7556 (2023)
  * [7] Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D., Dai, B.: Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 (2023)
  * [8] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017)
  * [9] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
  * [10] Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)
  * [11] Hong, F.T., Zhang, L., Shen, L., Xu, D.: Depth-aware generative adversarial network for talking head video generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3397–3406 (2022)
  * [12] Kingma, D.P., Welling, M., et al.: An introduction to variational autoencoders. Foundations and Trends® in Machine Learning 12(4), 307–392 (2019)
  * [13] Lee, D., Kim, C., Kim, S., Cho, M., Han, W.S.: Autoregressive image generation using residual quantization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11523–11532 (2022)
  * [14] Li, B.: Word-level fine-grained story visualization. In: European Conference on Computer Vision. pp. 347–362. Springer (2022)
  * [15] Li, Y., Wang, H., Jin, Q., Hu, J., Chemerys, P., Fu, Y., Wang, Y., Tulyakov, S., Ren, J.: Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. Advances in Neural Information Processing Systems 36 (2024)
  * [16] Li, Y., Gan, Z., Shen, Y., Liu, J., Cheng, Y., Wu, Y., Carin, L., Carlson, D., Gao, J.: Storygan: A sequential conditional gan for story visualization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6329–6338 (2019)
  * [17] Liu, M.Y., Huang, X., Yu, J., Wang, T.C., Mallya, A.: Generative adversarial networks for image and video synthesis: Algorithms and applications. Proceedings of the IEEE 109(5), 839–862 (2021)
  * [18] Maharana, A., Bansal, M.: Integrating visuospatial, linguistic and commonsense structure into story visualization. arXiv preprint arXiv:2110.10834 (2021)
  * [19] Maharana, A., Hannan, D., Bansal, M.: Improving generation and evaluation of visual stories via semantic consistency. arXiv preprint arXiv:2105.10026 (2021)
  * [20] Maharana, A., Hannan, D., Bansal, M.: Storydall-e: Adapting pretrained text-to-image transformers for story continuation. In: European Conference on Computer Vision. pp. 70–87. Springer (2022)
  * [21] Pan, X., Qin, P., Li, Y., Xue, H., Chen, W.: Synthesizing coherent story with auto-regressive latent diffusion models. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2920–2930 (2024)
  * [22] Qiao, T., Zhang, J., Xu, D., Tao, D.: Mirrorgan: Learning text-to-image generation by redescription. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1505–1514 (2019)
  * [23] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [24] Rahman, T., Lee, H.Y., Ren, J., Tulyakov, S., Mahajan, S., Sigal, L.: Make-a-story: Visual memory conditioned consistent story generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2493–2502 (2023)
  * [25] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022)
  * [26] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: International Conference on Machine Learning. pp. 8821–8831. PMLR (2021)
  * [27] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [28] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [29] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35, 36479–36494 (2022)
  * [30] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35, 36479–36494 (2022)
  * [31] Shen, F., Ye, H., Zhang, J., Wang, C., Han, X., Wei, Y.: Advancing pose-guided image synthesis with progressive conditional diffusion models. In: The Twelfth International Conference on Learning Representations (2023)
  * [32] Shen, X., Elhoseiny, M.: Large language models as consistent story visualizers. arXiv preprint arXiv:2312.02252 (2023)
  * [33] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 (2020)
  * [34] Tao, M., Tang, H., Wu, F., Jing, X.Y., Bao, B.K., Xu, C.: Df-gan: A simple and effective baseline for text-to-image synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16515–16525 (2022)
  * [35] Wang, C., Tian, K., Guan, Y., Zhang, J., Jiang, Z., Shen, F., Han, X., Gu, Q., Yang, W.: Ensembling diffusion models via adaptive feature aggregation. arXiv preprint arXiv:2405.17082 (2024)
  * [36] Wang, C., Tian, K., Zhang, J., Guan, Y., Luo, F., Shen, F., Jiang, Z., Gu, Q., Han, X., Yang, W.: V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint arXiv:2406.02511 (2024)
  * [37] Wu, Q., Liu, Y., Zhao, H., Bui, T., Lin, Z., Zhang, Y., Chang, S.: Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7766–7776 (2023)
  * [38] Xiao, Y., Wu, L., Guo, J., Li, J., Zhang, M., Qin, T., Liu, T.y.: A survey on non-autoregressive generation for neural machine translation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
  * [39] Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., He, X.: Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1316–1324 (2018)
  * [40] Yang, Z., Wang, J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C., Zeng, M., et al.: Reco: Region-controlled text-to-image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14246–14255 (2023)
  * [41] Ye, H., Zhang, J., Liu, S., Han, X., Yang, W.: Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 (2023)
  * [42] Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-attention generative adversarial networks. In: International conference on machine learning. pp. 7354–7363. PMLR (2019)
  * [43] Zhang, H., Koh, J.Y., Baldridge, J., Lee, H., Yang, Y.: Cross-modal contrastive learning for text-to-image generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 833–842 (2021)
  * [44] Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas, D.N.: Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 5907–5915 (2017)
  * [45] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3836–3847 (2023)
  * [46] Zhang, Z., Han, L., Ghosh, A., Metaxas, D.N., Ren, J.: Sine: Single image editing with text-to-image diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6027–6037 (2023)

## Supplementary Material

This supplementary material offers a more detailed exploration of the
experiments and methodologies proposed in the main paper. Section 0.A provides
a series of symbols and definitions for enhanced comprehension. Section 0.B
delves deeper into the implementation specifics of our experiments. Section
0.C presents additional experimental outcomes, including a broader range of
supplemented results from the RCDMs method, qualitative comparison examples
with state-of-the-art methods, and a detailed explanation of our user studies.

##  Appendix 0.A Some Notations and Definitions

Table 3: Some notations and definitions. Notation | Definition
---|---
${x_{0}}$ | Real image
${c}$ | Additional condition
${t}$ | Timestep
${\theta}$ | Diffusion model
${\epsilon}$ | Gaussian noise
${w}$ | Guidance scale
${v_{p}}$ | Pooled visual representation of the frame
${v_{h}}$ | Hidden visual representation of the frame
${t_{p}}$ | Pooled textual representation of the caption
${t_{h}}$ | Hidden textual representation of the caption
${F_{I}}$ | Feature of the image-level conditional
${F_{i}^{k}}$ | Interaction feature of the known clip
${F_{s}^{u}}$ | Stacking feature of the unknown clip

##  Appendix 0.B Implement Details

Our experiments are conducted on 8 NVIDIA V100 GPUs. We follow the standard
training strategies and hyperparameters of diffusion models. We utilize the
AdamW optimizer with a consistent learning rate of $1e^{-5}$ across all
stages. The probability of random dropout for condition $c$ is set at 10%. We
employ OpenCLIP ViT-bigG/14 222https://huggingface.co/laion/CLIP-ViT-
bigG-14-laion2B-39B-b160k as the image and text encoder in all stages. For the
frame-prior transformer diffusion model, it consists of 10 transformer blocks,
each with a width of 2,048. The model is trained for 100k iterations with a
batch size of 8, using a cosine noising schedule with 1000 timesteps. We
modify the first convolution layer for the frame-contextual 3D diffusion model
to accommodate additional conditions. The model is trained for 500k
iterations, each with a batch size of 8, and a linear noise schedule with 1000
timesteps is applied. In the inference stage, we use the DDIM sampler with 20
steps and set the guidance scale $w$ to 2.0 for PCDMs in all stages.

Dataset. There are 7 main characters in PororoSV: Fred, Barney, Wilma, Betty,
Pebbles, Dino, and Slate. Profile pictures of them are given in Fig. 9. There
are 9 main characters in PororoSV: Pororo, Loopy, Eddy, Harry, Poby, Tongtong,
Crong, Rody, and Petty. Profile pictures of them are given in Fig. 10.

User Study. Details of the user study and need more volunteers. We carried out
a user study using 200 randomly selected sets of stories derived from the
FlintstonesSV and PororoSV datasets. These stories were assessed across
various dimensions by 100 volunteers.

Figure 9: Main character names and corresponding images in FlintstonesSV. The
images are from https://flintstones.fandom.com/. Figure 10: Main character
names and corresponding images in PororoSV. The images are from
https://pororo.fandom.com/.

##  Appendix 0.C Additional Results

We provide additional examples for RCDMs in Fig. 11, Fig. 12, and Fig. 13. We
show additional examples for comparison with the state-of-the-art (SOTA)
methods in Fig. 14 and Fig. 15. We provide the user study interface as shown
in Fig. 16.

Figure 11: More qualitative results of RCDMs on the FlintstonesSV dataset.
Figure 12: More qualitative results of RCDMs on the FlintstonesSV dataset.
Figure 13: More qualitative results of RCDMs on the PororoSV dataset.  Figure
14: More qualitative comparisons between RCDMs and SOTA methods.  Figure 15:
More qualitative comparisons between RCDMs and SOTA methods.  Figure 16: An
example question used in our user study for story visualization.


# Untrusted NOMA with Imperfect SIC: Outage Performance Analysis and
Optimization

Sapna Thapar1, Deepak Mishra2, and Ravikant Saini1 1Department of Electrical
Engineering, Indian Institute of Technology Jammu, India
2School of Electrical Engineering and Telecommunications, University of New
South Wales, Australia
Emails<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Non-orthogonal multiple access (NOMA) has come to the fore as a spectral-
efficient technique for fifth-generation and beyond communication networks. We
consider the downlink of a NOMA system with untrusted users. In order to
consider a more realistic scenario, imperfect successive interference
cancellation is assumed at the receivers during the decoding process. Since
pair outage probability (POP) ensures a minimum rate guarantee to each user,
it behaves as a measure of the quality of service for the pair of users. With
the objective of designing a reliable communication protocol, we derive the
closed-form expression of POP. Further, we find the optimal power allocation
that minimizes the POP. Lastly, numerical results have been presented which
validate the exactness of the analysis, and reveal the effect of various key
parameters on achieved pair outage performance. In addition, we benchmark
optimal power allocation against equal and fixed power allocations with
respect to POP. The results indicate that optimal power allocation results in
improved communication reliability.

##  I Introduction

Non-orthogonal multiple access (NOMA) has been recognized as a disruptive
technology, supporting massive connections through high spectral efficiency,
for fifth-generation and beyond wireless networks. With NOMA, a number of
users are accommodated simultaneously within the same resource block by using
superposition coding at the transmitter and successive interference
cancellation (SIC) at the receiver [1]. However, each user in the system has a
threshold data rate demand, i.e., each user desires the base station to
transmit messages with a minimum data rate guarantee under the quality of
service (QoS) agreement. Therefore, ensuring reliability with a minimum rate
guarantee to each user in an untrusted NOMA system is an interesting research
direction.

###  I-A Related Works

Reliable communication signifies that the messages broadcasted from the
transmitter are highly likely to reach their destination. Extensive research
activities have analyzed the reliability in terms of outage probability for
the considered different system models. In [2], the performance analysis of a
downlink NOMA system with randomly deployed mobile users for a Rayleigh fading
channel link was investigated in terms of outage probability. In [3], outage
performance analysis for a two-user NOMA system with Nakagami-m fading channel
link was studied. In [4], closed-form outage probability for each user in a
NOMA system with statistical channel state information at the transmitter and
Nakagami-m fading links was investigated. The exact expressions for the outage
probability for a two-user NOMA system considering a dynamic power allocation
scheme was investigated in [5]. In the direction of an untrusted NOMA system,
reliability was discussed in [6], where the authors’ derived pair outage
probability (POP) to analyze the reliability of a two-user NOMA system. Then,
in [7], POP was analyzed for the optimal decoding order that can ensure
secrecy to each user of the system.

###  I-B Motivation and Key Contributions

The outage analyses studied in the above-mentioned works [2]-[6] are
restricted to the conventional decoding order of NOMA. Unlike the conventional
decoding order, each user can perform SIC and decode data of itself and other
users in any sequence if the users are untrusted [7], [8]. As a result, a
number of decoding orders are possible in an untrusted NOMA system [8] and
each of them is important for different applications. Thus, it is important to
study the outage performance in each decoding order. Besides, a common
assumption considered in [2]-[6] is that perfect SIC is performed by the
receivers. With perfect SIC, the interference from the decoded users is
cancelled entirely while decoding later users which results in better spectral
efficiency. Nevertheless, implementing perfect SIC might not be practical, due
to decoding errors and complexity scaling issues [9], [10]. In consequence,
imperfect SIC, where the residual interference (RI) from imperfectly decoded
users remains while decoding later users should be considered [10], [11]. In
the NOMA literature, some researchers have considered a fixed value of RI
[12], [13] while many have taken RI as a linear function of the interfering
power [9], [10], [14]. However, despite its significance, analyzing the outage
performance of an untrusted NOMA system with imperfect SIC has not received
much attention yet.

Assuming imperfect SIC at receivers, POP of the system was analyzed with RI in
[7]. Here, considering the application of secure communication in NOMA, the
study was carried out for that optimal decoding order which ensures positive
secrecy to each user. Note that in this work, a fixed RI value was considered,
which is a strong and unrealistic assumption. In contrast to this, a more
generalised linear model of imperfect SIC can effectively represent the
relationship between RI and the received signal power. _Therefore, to
appreciate the significant impact of imperfect SIC with linear RI model on
reliable communication for the users, we investigate the pair outage for a
two-user NOMA system, which to the best of our knowledge, has not been
investigated yet in the literature._

The key contributions of this work are summarized below:

  * •

To observe the realistic impact of imperfect SIC with linear model, the
analytical expressions of POP for a two-user untrusted NOMA system is derived.

  * •

To analyze optimal performance, optimal power allocation minimizing POP is
obtained by using the concept of generalized-convexity of POP.

  * •

Numerical results are provided to validate the analytical results, and
highlight deeper insights on the impact of different key parameters on the
system performance.

##  II Untrusted NOMA with Imperfect SIC

In this section, we first present the system model, then provide the possible
decoding orders for an untrusted NOMA system. Finally, the signal-to-
interference-plus-noise-ratio (SINR) expressions are derived for users under
imperfect SIC scenario.

###  II-A System Model

We consider downlink transmission of a NOMA system consisting of one base
station (BS) and two untrusted users. It is not preferable in practice to ask
all users in the system to participate in NOMA jointly due to an increase in
co-channel interference and implementation complexity with more users.
Therefore, dividing users into groups and implementing NOMA in each group is a
better choice. For analytical tractability, we focus on a two-user NOMA system
as taken in many works [15], [16]. The $n$-th user is denoted by $U_{n}$,
where $n\in\mathcal{N}=\\{1,2\\}$. All the nodes in the network are assumed to
be equipped with one antenna. The channel gain from BS to $U_{n}$ is assumed
to be Rayleigh distributed. We denote the channel gain coefficient between BS
and $U_{n}$ by $h_{n}$. The channel power gain $|h_{n}|^{2}$ follows an
exponential distribution with mean parameter $\lambda_{n}=L_{p}d_{n}^{-e}$,
where $d_{n}$ indicates the distance between BS and $U_{n}$, $L_{p}$ denotes
the path loss constant, and $e$ is the path loss exponent. Without loss of
generality, channel power gains are assumed tio be ordered as
$|h_{1}|^{2}>|h_{2}|^{2}$. Therefore, near and far user, i.e., $U_{1}$ and
$U_{2}$ could be regarded as strong and weak user, respectively.

In accordance to the power-domain NOMA principle, the BS transmits the
superimposed signal $\sqrt{\alpha P_{t}}x_{1}$\+ $\sqrt{(1-\alpha)P_{t}}x_{2}$
to both users. Here, $x_{1}$ and $x_{2}$ are the information signals
associated with $U_{1}$ and $U_{2}$, respectively. $P_{t}$ is the total BS
transmission power. $\alpha$ denotes the fraction of $P_{t}$ allocated to
$U_{1}$, and $(1-\alpha)$ is the remaining fraction of $P_{t}$ allocated to
$U_{2}$. At the receiver side, each user performs SIC where inter-user
interference is cancelled out to extract the desired information signal [17].
Without loss of generality, received additive white Gaussian noise is assumed
to have mean equal to zero and variance equal to $\sigma^{2}$ at both the
users. In order to focus on a realistic system model, we consider imperfect
SIC at receivers where RI from imperfectly decoded signals remains while
decoding later users. Considering linear RI model, the corresponding RI factor
is denoted by $\beta$, where $0\leq\beta\leq 1$. Note that $\beta=0$
corresponds to perfect SIC and $\beta=0$ indicates fully imperfect SIC, i.e.,
maximum interference [9], [10].

###  II-B Possible Decoding Orders for Untrusted NOMA

Each receiver performs SIC in a certain sequence. A collection of such
sequences for each of the users is termed as the “decoding order” of the
system. In accordance with an untrusted NOMA system, each user can decode any
user’s signal at any stage [7], [8], [17]. Thus, in a two user system, with
each user having two choices that of decoding self or the other user first,
four decoding orders are possible [7]. We can denote the decoding order as a
$2\times 2$ matrix $\mathbf{D}_{o}$, where $o\in\\{1,2,3,4\\}$ indicates the
index of $o$-th decoding order. The $m$-th column of matrix $\mathbf{D}_{o}$
is denoted by a column vector $\mathbf{d}_{m}$ of size $2\times 1$, which
imply the SIC sequence followed by $U_{m}$, where $m\in\mathcal{N}$. Here,
$[\mathbf{d}_{m}]_{k}=n$ signifies that $U_{m}$ decodes data of $U_{n}$ at
$k$-th stage, where $n,k\in\mathcal{N}$ and
$[\mathbf{d}_{m}]_{1}\neq[\mathbf{d}_{m}]_{2}$. Thus, we can write the four
possible decoding orders as $\mathbf{D}_{1}=[2,1;2,1]$,
$\mathbf{D}_{2}=[2,1;1,2]$, $\mathbf{D}_{3}=[1,2;2,1]$, and
$\mathbf{D}_{4}=[1,2;1,2]$.

###### Remark 1

From the perspective of ensuring secure communication to both the users, it
has been proved in [7, Theorem 2] that the optimal decoding order with respect
to providing maximum secrecy rate for both users is $\mathbf{D}_{2}$.
Therefore, for mathematical tractability, we further derive the analytical
expression of POP with the linear imperfect SIC model for decoding order
$\mathbf{D}_{2}$. However, in a similar manner, we will be analyzing the POP
expressions also for other decoding orders in the extended version of this
work.

###  II-C Achievable SINRs with Imperfect SIC

According to the decoding order $\mathbf{D}_{2}$, $U_{1}$ and $U_{2}$ decode
signal of other user at the first stage, and then decode their own signal at
the second stage after performing SIC [7]. Thus, considering imperfect SIC at
receivers with linear model [8], [10], the achievable SINR $\Gamma_{nm}$ at
$U_{m}$, when $U_{n}$ is decoded by $U_{m}$, where $m,n\in\mathcal{N}$, is
given as

$\displaystyle\Gamma_{21}$
$\displaystyle=\frac{(1-\alpha)|h_{1}|^{2}}{\alpha|h_{1}|^{2}+\frac{1}{\rho_{t}}},$
(1) $\displaystyle\Gamma_{12}$
$\displaystyle=\frac{\alpha|h_{2}|^{2}}{(1-\alpha)|h_{2}|^{2}+\frac{1}{\rho_{t}}},$
(2) $\displaystyle\Gamma_{11}$
$\displaystyle=\frac{\alpha|h_{1}|^{2}}{(1-\alpha)\beta|h_{1}|^{2}+\frac{1}{\rho_{t}}},$
(3) $\displaystyle\Gamma_{22}$
$\displaystyle=\frac{(1-\alpha)|h_{2}|^{2}}{\alpha\beta|h_{2}|^{2}+\frac{1}{\rho_{t}}},$
(4)

where $\rho_{t}\stackrel{{\scriptstyle\Delta}}{{=}}\frac{P_{t}}{\sigma^{2}}$
is the BS transmit signal-to-noise ratio (SNR). The corresponding achievable
data rate at $U_{n}$ is given by the Shannon’s capacity formula as

$\displaystyle R_{nm}=\log_{2}(1+\Gamma_{nm}).$ (5)

##  III Pair Outage Performance Analaysis

In this section, we derive the analytical expressions of POP that ensures
users’ QoS demands for reliable communication over both the links. POP is
defined as the probability at which the achievable data rate at each user
falls below than the required threshold data rate. Thus, POP ensures minimum
data rate guarantee to each user in the system. Let us denote POP for the
system by $P_{o}$. Assuming threshold data rate for $U_{n}$ as
$R_{n}^{\mathrm{th}}$, POP can be mathematically given as

$\displaystyle P_{o}$
$\displaystyle=1-\mathbb{P}\\{\Gamma_{11}>\pi_{1},\Gamma_{21}>\pi_{2},\Gamma_{12}>\pi_{1},\Gamma_{22}>\pi_{2}\\},$
$\displaystyle\stackrel{{\scriptstyle(\mathrm{g})}}{{=}}1-\mathbb{P}\\{\Gamma_{11}>\pi_{1},\Gamma_{21}>\pi_{2}\\}\mathbb{P}\\{\Gamma_{12}>\pi_{1},\Gamma_{22}>\pi_{2}\\},$
$\displaystyle=1-\mathbb{P}\\{|h_{1}|^{2}>\max(\zeta_{1},\zeta_{2})\\}\mathbb{P}\\{|h_{2}|^{2}>\max(\zeta_{3},\zeta_{4})\\},$
$\displaystyle=1-(1-F_{|h_{1}|^{2}}(\max(\zeta_{1},\zeta_{2})))(1-F_{|h_{2}|^{2}}(\max(\zeta_{3},\zeta_{4}))),$
$\displaystyle=1-\bar{F}_{|h_{1}|^{2}}(\max(\zeta_{1},\zeta_{2}))\bar{F}_{|h_{2}|^{2}}(\max(\zeta_{3},\zeta_{4})),$
(6)

where $\mathbb{P}\\{.\\}$ denotes the probability measure,
$\pi_{n}\stackrel{{\scriptstyle\Delta}}{{=}}2^{R_{n}^{\mathrm{th}}}-1$,
$(\mathrm{g})$ follows from the property of independent events [18],
$\zeta_{1}\stackrel{{\scriptstyle\Delta}}{{=}}\frac{\pi_{1}}{(\alpha-\beta(1-\alpha)\pi_{1})\rho_{t}},\zeta_{2}\stackrel{{\scriptstyle\Delta}}{{=}}\frac{\pi_{2}}{(1-\alpha-\alpha\pi_{2})\rho_{t}},\zeta_{3}\stackrel{{\scriptstyle\Delta}}{{=}}\frac{\pi_{1}}{(\alpha-(1-\alpha)\pi_{1})\rho_{t}}$,
and
$\zeta_{4}\stackrel{{\scriptstyle\Delta}}{{=}}\frac{\pi_{2}}{(1-\alpha-\beta\alpha\pi_{2})\rho_{t}}$.
$F_{|h_{1}|^{2}}(x)$ and $\bar{F}_{|h_{1}|^{2}}(x)$, respectively, denote the
cumulative distribution function (CDF) and complementary cumulative
distribution function (CCDF), of channel power gain $|h_{1}|^{2}$. In a
similar manner, $F_{|h_{2}|^{2}}(x)$ and $\bar{F}_{|h_{2}|^{2}}(x)$ are CDF
and CCDF, respectively, of channel power gain $|h_{2}|^{2}$.

We observe that in (9), $\bar{F}_{|h_{1}|^{2}}(\max(\zeta_{1},\zeta_{2}))$ can
be rewritten for two cases $\zeta_{1}>\zeta_{2}$ and $\zeta_{1}<\zeta_{2}$. On
solving the first case $\zeta_{1}>\zeta_{2}$, we find a constraint on $\alpha$
as
$\alpha<\frac{\pi_{1}(1+\pi_{2}\beta)}{\pi_{2}(1+\pi_{1}\beta)+\pi_{1}(1+\pi_{2})}$.
Similarly, in the second case $\zeta_{1}<\zeta_{2}$, the constraint on
$\alpha$ will be
$\alpha>\frac{\pi_{1}(1+\pi_{2}\beta)}{\pi_{2}(1+\pi_{1}\beta)+\pi_{1}(1+\pi_{2})}$.
Besides, from the definition of CDF of exponential distribution, $\zeta_{1}>0$
and $\zeta_{2}>0$, respectively, gives
$\alpha>\frac{\beta\pi_{1}}{1+\beta\pi_{1}}$ and $\alpha<\frac{1}{1+\pi_{2}}$.
Thus, denoting
$\alpha_{1}\stackrel{{\scriptstyle\Delta}}{{=}}\frac{\beta\pi_{1}}{1+\beta\pi_{1}}$,
$\alpha_{2}\stackrel{{\scriptstyle\Delta}}{{=}}\frac{\pi_{1}(1+\pi_{2}\beta)}{\pi_{2}(1+\pi_{1}\beta)+\pi_{1}(1+\pi_{2})}$,
and $\alpha_{3}\stackrel{{\scriptstyle\Delta}}{{=}}\frac{1}{1+\pi_{2}}$,
$\bar{F}_{|h_{1}|^{2}}(\max(\zeta_{1},\zeta_{2}))$ can be expressed as

$\displaystyle\bar{F}_{|h_{1}|^{2}}(\max(\zeta_{1},\zeta_{2}))=\begin{cases}\exp\\{-\frac{\zeta_{1}}{\lambda_{1}}\\},&\alpha_{1}<\alpha<\alpha_{2}\\\
\exp\\{-\frac{\zeta_{2}}{\lambda_{1}}\\},&\alpha_{2}<\alpha<\alpha_{3}\\\
0,&\text{otherwise}.\end{cases}$ (7)

Similarly, $\bar{F}_{|h_{2}|^{2}}(\max(\zeta_{3},\zeta_{4}))$ can be expressed
as follows

$\displaystyle\bar{F}_{|h_{2}|^{2}}(\max(\zeta_{3},\zeta_{4}))$
$\displaystyle=\begin{cases}\exp\\{-\frac{\zeta_{3}}{\lambda_{2}}\\},&\alpha_{4}<\alpha\\!<\alpha_{5}\\\
\exp\\{-\frac{\zeta_{4}}{\lambda_{2}}\\},&\alpha_{5}<\alpha\\!<\alpha_{6}\\\
0,&\text{otherwise}\end{cases}$ (8)

where $\alpha_{4}=\frac{\pi_{1}}{1+\pi_{1}}$,
$\alpha_{5}=\frac{\pi_{1}(1+\pi_{2})}{\pi_{2}(1+\pi_{1})+\pi_{1}(1+\beta\pi_{2})}$,
and $\alpha_{6}=\frac{1}{1+\beta\pi_{2}}$. Here we observe that
$\alpha_{4}>\alpha_{1}$ and $\alpha_{6}>\alpha_{3}$ since $\beta<1$. Thus,
considering (III), (7), (8), and $\alpha_{4}>\alpha_{1}$,
$\alpha_{6}>\alpha_{3}$, the piecewise definition of $P_{o}$ as a function of
$\alpha$ can be obtained as given at the top of next page in (9).

$P_{o}=\begin{cases}1-\bar{F}_{|h_{1}|^{2}}(\zeta_{1})\bar{F}_{|h_{2}|^{2}}(\zeta_{3}),\quad\quad\begin{cases}[\alpha_{4}<\alpha<\alpha_{5}]\wedge[\alpha_{5}<\alpha_{2}]\\\
[\alpha_{4}<\alpha<\alpha_{2}]\wedge[\alpha_{5}>\alpha_{2}]\\\
\end{cases}&\quad\quad\text{Case 1}\\\
1-\bar{F}_{|h_{1}|^{2}}(\zeta_{1})\bar{F}_{|h_{2}|^{2}}(\zeta_{4}),\quad\quad\begin{cases}[\alpha_{5}<\alpha<\alpha_{6}]\wedge[\alpha_{5}>\alpha_{1}]\wedge[\alpha_{6}<\alpha_{2}]\\\
[\alpha_{1}<\alpha<\alpha_{2}]\wedge[\alpha_{5}<\alpha_{1}]\wedge[\alpha_{6}>\alpha_{2}]\\\
[\alpha_{1}<\alpha<\alpha_{6}]\wedge[\alpha_{5}<\alpha_{1}]\wedge[\alpha_{6}<\alpha_{2}]\\\
[\alpha_{5}<\alpha<\alpha_{2}]\wedge[\alpha_{5}>\alpha_{1}]\wedge[\alpha_{6}>\alpha_{2}]\\\
\end{cases}&\quad\quad\text{Case 2}\\\
1-\bar{F}_{|h_{1}|^{2}}(\zeta_{2})\bar{F}_{|h_{2}|^{2}}(\zeta_{3}),\quad\quad\begin{cases}[\alpha_{4}<\alpha<\alpha_{5}]\wedge[\alpha_{4}>\alpha_{2}]\wedge[\alpha_{5}<\alpha_{3}]\\\
[\alpha_{2}<\alpha<\alpha_{3}]\wedge[\alpha_{4}<\alpha_{2}]\wedge[\alpha_{5}>\alpha_{3}]\\\
[\alpha_{2}<\alpha<\alpha_{5}]\wedge[\alpha_{4}<\alpha_{2}]\wedge[\alpha_{5}<\alpha_{3}]\\\
[\alpha_{4}<\alpha<\alpha_{3}]\wedge[\alpha_{4}>\alpha_{2}]\wedge[\alpha_{5}>\alpha_{3}]\\\
\end{cases}&\quad\quad\text{Case 3}\\\
1-\bar{F}_{|h_{1}|^{2}}(\zeta_{2})\bar{F}_{|h_{2}|^{2}}(\zeta_{4}),\quad\quad\begin{cases}[\alpha_{2}<\alpha<\alpha_{3}]\wedge[\alpha_{5}<\alpha_{2}]\\\
[\alpha_{5}<\alpha<\alpha_{3}]\wedge[\alpha_{5}>\alpha_{2}]\\\
\end{cases}&\quad\quad\text{Case 4}\\\
1,\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\text{otherwise.}&\quad\quad\text{Case
5}\end{cases}$ (9)

Here,

$\displaystyle\bar{F}_{|h_{1}|^{2}}(\zeta_{1})$
$\displaystyle=\exp\bigg{\\{}-\frac{\pi_{1}}{(\alpha-\beta(1-\alpha)\pi_{1})\rho_{t}\lambda_{1}}\bigg{\\}},$
(10) $\displaystyle\bar{F}_{|h_{1}|^{2}}(\zeta_{2})$
$\displaystyle=\exp\bigg{\\{}-\frac{\pi_{2}}{(1-\alpha-\alpha\pi_{2})\rho_{t}\lambda_{1}}\bigg{\\}},$
(11) $\displaystyle\bar{F}_{|h_{2}|^{2}}(\zeta_{3})$
$\displaystyle=\exp\bigg{\\{}-\frac{\pi_{1}}{(\alpha-(1-\alpha)\pi_{1})\rho_{t}\lambda_{2}}\bigg{\\}},$
(12) $\displaystyle\bar{F}_{|h_{2}|^{2}}(\zeta_{4})$
$\displaystyle=\exp\bigg{\\{}-\frac{\pi_{2}}{(1-\alpha-\beta\alpha\pi_{2})\rho_{t}\lambda_{2}}\bigg{\\}}.$
(13)

##  IV Pair Outage Probability Minimization

In this section, we obtain optimal power allocation that minimizes pair outage
probability.

###  IV-A Problem Formulation

Observing POP expression as a function of $\alpha$, the POP optimization
problem, using (9), can be formulated as

$\displaystyle\mathcal{O}:\underset{\alpha}{\text{minimize}}$ $\displaystyle
P_{o},$ s.t. $\displaystyle(C1):0<\alpha<1,$

where $(C1)$ denotes the constraint on PA coefficient.

###  IV-B Solution Methodology

POP $P_{o}$, given in (9), is a closed-form piece-wise expression. We study
each case of $P_{o}$ one by one, and find candidate optimal points to obtain
the global-optimal solution. The key result regarding the global-optimal
solution of $\mathcal{O}$ is provided through the Lemma $1$.

###### Lemma 1

The global-optimal power allocation $\alpha^{*}$ is the feasible optimal point
from the set of obtained candidate optimal points that minimizes $P_{o}$,
which can be expressed as

$\displaystyle\alpha^{*}\\!\stackrel{{\scriptstyle\Delta}}{{=}}\\!\underset{\alpha\in\\{\alpha_{c1},\alpha_{r1},\alpha_{r2},\alpha_{r3},\alpha_{r4},\alpha_{c2}\\}}{\mathrm{argmin}}P_{o},$
(14)

where $\alpha_{c1}$, $\alpha_{r1}$, $\alpha_{r2}$, $\alpha_{r3}$,
$\alpha_{r4}$ and $\alpha_{c2}$ are the candidate optimal points.

###### Proof:

Let us consider each case of the closed-form piecewise expression of $P_{o}$
given in (9).

Case 1 : In the first case, when
$P_{o}=1-\bar{F}_{|h_{1}|^{2}}(\zeta_{1})\bar{F}_{|h_{2}|^{2}}(\zeta_{3})$,
the first-order derivative obtained by differentiating $P_{o}$ with respect to
$\alpha$ is given as

$\displaystyle\frac{\text{d}P_{o}}{\text{d}\alpha}$
$\displaystyle=-\left(\dfrac{\pi_{1}s_{1}}{\lambda_{1}\rho_{t}\left(\alpha
s_{1}-\beta\pi_{1}\right)^{2}}+\dfrac{\pi_{1}t_{1}}{\lambda_{2}\rho_{t}\left(\alpha
t_{1}-\pi_{1}\right)^{2}}\right)\times$
$\displaystyle\mathrm{exp}\bigg{\\{}\\!\frac{-\pi_{1}}{\lambda_{1}\rho_{t}\left(\alpha
s_{1}-\beta\pi_{1}\right)}\\!-\\!\frac{\pi_{1}}{\lambda_{2}\rho_{t}\left(\alpha
t_{1}-\pi_{1}\right)}\\!\bigg{\\}}.$ (15)

wher $s_{1}=\beta\pi_{1}+1$ and $t_{1}=\pi_{1}+1$. From (IV-B), we observe
that $\frac{\mathrm{d}P_{o}}{\mathrm{d}\alpha}<0$. Therefore, $P_{o}$ is a
monotonically decreasing function of $\alpha$ in the given range. Thus, the
optimal power allocation is the corner point of the range of $\alpha$ in
case-1. Hence, the optimal solution $\alpha_{c1}=\alpha_{5}$ if
$\alpha_{5}<\alpha_{2}$, or $\alpha_{c1}=\alpha_{2}$ if
$\alpha_{5}>\alpha_{2}$.

Case 2 : In the second case, where
$P_{o}=1-\bar{F}_{|h_{1}|^{2}}(\zeta_{1})\bar{F}_{|h_{2}|^{2}}(\zeta_{4})$,
the derivative $\frac{\text{d}P_{o}}{\text{d}\alpha}$ is obtained as

$\displaystyle\frac{\text{d}P_{o}}{\text{d}\alpha}$
$\displaystyle=\left(\\!\dfrac{\pi_{2}s_{2}}{\lambda_{2}\rho_{t}\left(1-\alpha
s_{2}\right)^{2}}\\!-\\!\dfrac{\pi_{1}\ s_{1}}{\lambda_{1}\rho_{t}\left(\alpha
s_{1}-\beta\pi_{1}\right)^{2}}\\!\right)\times$
$\displaystyle\mathrm{exp}\bigg{\\{}\\!\frac{-\pi_{2}}{\lambda_{2}\rho_{t}\left(1-\alpha
s_{2}\right)}\\!-\\!\frac{\pi_{1}}{\lambda_{1}\rho_{t}\left(\alpha
s_{1}-\beta\pi_{1}\right)}\\!\bigg{\\}}.$ (16)

where $s_{2}=\beta\pi_{2}+1$. We observe that
$\frac{\text{d}P_{o}}{\text{d}\alpha}$ in (IV-B) does not indicate any
monotonicity. But, $P_{o}$ is monotonically increasing function of $\alpha$ if
$\dfrac{\pi_{2}s_{2}}{\lambda_{2}\rho_{t}\left(1-\alpha
s_{2}\right)^{2}}>\dfrac{\pi_{1}\ s_{1}}{\lambda_{1}\rho_{t}\left(\alpha
s_{1}-\beta\pi_{1}\right)^{2}}$, and monotonically decreasing function
otherwise. Thus, we can obtain the point of inflection by solving
$\dfrac{\pi_{2}s_{2}}{\lambda_{2}\rho_{t}\left(1-\alpha
s_{2}\right)^{2}}=\dfrac{\pi_{1}\ s_{1}}{\lambda_{1}\rho_{t}\left(\alpha
s_{1}-\beta\pi_{1}\right)^{2}}$ which can be simplified as a quadratic
equation $m_{1}\alpha^{2}+m_{2}\alpha+m_{3}=0$ where
$m_{1}\stackrel{{\scriptstyle\Delta}}{{=}}r_{1}s_{1}-r_{2}s_{2},m_{2}\stackrel{{\scriptstyle\Delta}}{{=}}2(r_{2}-r_{1}\beta\pi_{1}),m_{3}\stackrel{{\scriptstyle\Delta}}{{=}}\pi_{1}^{2}\pi_{2}s_{2}\beta^{2}\lambda_{1}-\pi_{1}s_{1}\lambda_{2}$
with $r_{1}\stackrel{{\scriptstyle\Delta}}{{=}}\pi_{2}\lambda_{1}s_{1}s_{2},$
and $r_{2}\stackrel{{\scriptstyle\Delta}}{{=}}\pi_{1}\lambda_{2}s_{1}s_{2}$.
The optimal solutions, which are the roots of the quadratic equation, are
given as

$\alpha_{r1},\alpha_{r2}=\frac{-m_{2}\pm\sqrt{m_{2}^{2}-4m_{1}m_{3}}}{2m_{1}}.$
(17)

Case 3 : In the third case, where
$P_{o}=1-\bar{F}_{|h_{1}|^{2}}(\zeta_{2})\bar{F}_{|h_{2}|^{2}}(\zeta_{3})$,
the derivative of $P_{o}$ with respect to $\alpha$ is given as

$\displaystyle\frac{\text{d}P_{o}}{\text{d}\alpha}$
$\displaystyle=\left(\dfrac{\pi_{2}t_{2}}{\lambda_{1}\rho_{t}\left(1-\alpha
t_{2}\right)^{2}}-\dfrac{\pi_{1}t_{1}}{\lambda_{2}\rho_{t}\left(\alpha
t_{1}-\pi_{1}\right)^{2}}\right)\times$
$\displaystyle\mathrm{exp}\bigg{\\{}\frac{-\pi_{2}}{\lambda_{1}\rho_{t}\left(1-\alpha
t_{2}\right)}-\frac{\pi_{1}}{\lambda_{2}\rho_{t}\left(\alpha
t_{1}-\pi_{1}\right)}\bigg{\\}}.$ (18)

where $t_{2}=\pi_{2}+1$. Here also, we observe that
$\frac{\text{d}P_{o}}{\text{d}\alpha}$ given in (IV-B) does not show any
monotonicity. Therefore, similar to the second case, the point of inflection
can be obtained by solving
$\dfrac{\pi_{2}t_{2}}{\lambda_{1}\rho_{t}\left(1-\alpha
t_{2}\right)^{2}}=\dfrac{\pi_{1}t_{1}}{\lambda_{2}\rho_{t}\left(\alpha
t_{1}-\pi_{1}\right)^{2}}$ which is simplified as a quadratic equation
$l_{1}\alpha^{2}+l_{2}\alpha+l_{3}=0$ where
$l_{1}\stackrel{{\scriptstyle\Delta}}{{=}}q_{2}t_{1}-q_{1}t_{2},l_{2}\stackrel{{\scriptstyle\Delta}}{{=}}2(q_{1}-q_{2}\pi_{1}),l_{3}\stackrel{{\scriptstyle\Delta}}{{=}}\pi_{1}^{2}\pi_{2}t_{2}\lambda_{2}-\pi_{1}t_{1}\lambda_{1}$
with $q_{1}\stackrel{{\scriptstyle\Delta}}{{=}}\pi_{1}\lambda_{1}t_{1}t_{2},$
and $q_{2}\stackrel{{\scriptstyle\Delta}}{{=}}\pi_{2}\lambda_{2}t_{1}t_{2}$.
In this case, the optimal solutions given as the roots of the quadratic
equation, are obtained as

$\alpha_{r3},\alpha_{r4}=\frac{-l_{2}\pm\sqrt{l_{2}^{2}-4l_{1}l_{3}}}{2l_{1}}.$
(19)

Case 4 : In the fourth case, where
$P_{o}=1-\bar{F}_{|h_{1}|^{2}}(\zeta_{2})\bar{F}_{|h_{2}|^{2}}(\zeta_{4})$,
the derivative $\frac{\text{d}P_{o}}{\text{d}\alpha}$ is obtained as

$\displaystyle\frac{\text{d}P_{o}}{\text{d}\alpha}$
$\displaystyle=\left(\\!\dfrac{\pi_{2}s_{2}}{\lambda_{2}\rho_{t}\left(1-\alpha
s_{2}\right)^{2}}+\dfrac{\pi_{2}t_{2}}{\lambda_{1}\rho_{t}\left(1-\alpha
t_{2}\right)^{2}}\\!\right)\times$
$\displaystyle\mathrm{exp}\bigg{\\{}\\!\frac{-\pi_{2}}{\lambda_{2}\rho_{t}\left(1-\alpha
s_{2}\right)}\\!-\\!\frac{\pi_{2}}{\lambda_{1}\rho_{t}\left(1-\alpha
t_{2}\right)}\\!\bigg{\\}}.$ (20)

which is greater than zero. Thus, $P_{o}$ is a monotonically increasing
function of $\alpha$. Therefore, the optimal point, in this case, is given by
the lower corner point of considered range. Thus, the optimal solution
$\alpha_{c2}=\alpha_{2}$ when $\alpha_{5}<\alpha_{2}$, or
$\alpha_{c2}=\alpha_{5}$ when $\alpha_{5}>\alpha_{2}$.

Case 5 : In the fifth case, $P_{o}$ is a constant equal to the maximum
feasible value, i.e., $P_{o}=1$.

From the above analysis, we observe that $\alpha_{c1}$ and $\alpha_{c2}$ are
two corner points due to monotonically decreasing and increasing nature of
$P_{o}$, respectively. $\alpha_{r1}$ and $\alpha_{r2}$ are roots of quadratic
equation as explained in Case 2. Similarly, $\alpha_{r3}$ and $\alpha_{r4}$
are the roots of quadratic equation obtained in Case 3. As a result, POP
minimization problem has global-optimal solution $\alpha^{*}$, which is the
feasible optimal point from set {$\alpha_{c1}$, $\alpha_{r1}$, $\alpha_{r2}$,
$\alpha_{r3}$, $\alpha_{r4}$, $\alpha_{c2}$} at which $P_{o}$ is minimum. ∎

##  V Numerical Results

In this section, we present numerical results where we study the outage
performance of the NOMA communication system under various system parameters.
Downlink NOMA system is considered with one BS and two users. Near and far
user, respectively, are assumed to be located at a distance of $d_{1}=50$
meter and $d_{2}=100$ meter from BS. Noise power is taken as $-90$ dBm with
noise signal following Gaussian distribution at both users. Small scale fading
is assumed to follow an exponential distribution with a mean value equal to 1
for both the links [6]. We have averaged simulation results over $10^{6}$
randomly generated channel realizations using Rayleigh distribution at both
strong and weak users. The value of $L_{p}$ and $e$, respectively, are set to
$1$ and $3$. Besides, we have taken $\beta=0.2$, $\alpha=0.5$,
$R_{1}^{\mathrm{th}}=0.1$, $R_{2}^{\mathrm{th}}=0.1$ and $\rho_{t}=60$ dB.

###  V-A Validation of Analysis

Fig. 1 presents the validation of the accuracy of the analytical results. The
simulation and analytical results are, respectively, marked as ‘Sim’ and ‘Ana’
in the following figures. The validation of POP, i.e., $P_{o}$, is shown with
$R_{1}^{\mathrm{th}}$ and $R_{2}^{\mathrm{th}}$ for different values of
$\rho_{t}$. The perfect match between simulated and analytical curves confirms
the accurateness of POP analysis. Besides, it is observed from the results
that with an increase in threshold rates $R_{1}^{\mathrm{th}}$ and
$R_{2}^{\mathrm{th}}$, pair outage $P_{o}$ increases. The reason is that an
outage happens in the system when the maximum achievable data rate falls below
a threshold rate, which clearly means POP increases on increasing threshold
rates at the users. Further, an increase in $\rho_{t}$ results in decrease in
POP. This is because, the data rates achieved by users’ increase with an
increase in the SNR, resulting in a decrease in POP, for given threshold data
rates.

Figure 1: Validation of POP with a variation in threshold data rate
requirements at near and far users for various BS transmit SNR values,
$\alpha=0.5$.

###  V-B Impact of Relative Threshold Data Rate Demands

Figure 2: Variation in POP with different threshold data rate requirements at
near and far users, $\alpha=0.5$ and $\rho_{t}=60$ dB.

The impact of relative threshold data rate demands of the near and the far
user on the pair outage performance of the system is highlighted through Fig.
2. $P_{o}$ has been plotted versus $R_{1}^{\mathrm{th}}$ for different values
of $R_{2}^{\mathrm{th}}$. From the results, it can be observed that the POP
performance is better when $R_{1}^{\mathrm{th}}>R_{2}^{\mathrm{th}}$. Note
that due to the poorer channel condition of the weak user, the data rate
achieved at $U_{2}$ is always lesser compared to data rate achieved by
$U_{1}$. In such situation, increasing $R_{2}^{\mathrm{th}}$ will in turn
increase outage probability for $U_{2}$ which will further increase POP.
Therefore, to achieve better pair outage performance, having the threshold
rate pair ($R_{1}^{\mathrm{th}},R_{2}^{\mathrm{th}}$) with
$R_{2}^{\mathrm{th}}<R_{1}^{\mathrm{th}}$ is a good choice.

###  V-C Insights on Optimal Power Allocation

Figure 3: Optimality of POP with power allocation coefficient $\alpha$
associated to near user for different values of BS transmit SNR,
$R_{1}^{\mathrm{th}}=R_{2}^{\mathrm{th}}=0.1$. Figure 4: POP versus power
allocation coefficient $\alpha$ with different threshold rate requirements at
near and far users, $\rho_{t}=60$ dB.

Both Fig. 3 and Fig. 4 present the numerical proof of the unique solution of
POP with respect to power allocation $\alpha$. In Fig. 3 and Fig. 4,
respectively, we depict $P_{o}$ with $\alpha$ for different values of BS
transmit SNR $\rho_{t}$, and threshold rates $R_{1}^{\mathrm{th}}$ and
$R_{2}^{\mathrm{th}}$. In both Fig. 3 and Fig. 4, the unique optimal solution
of power allocation that gives minimum POP can be easily observed. The
analytical optimal solution has also been marked to validate the analysis. It
is observed from the results that optimal power allocation is independent of
the SNR values, which corroborates our analysis presented through Lemma $1$.
However, for the different pair of threshold data rate demands, optimal
$\alpha$ varies. Thus, it can be concluded the threshold rate pair plays an
important role in the pair outage performance of the system.

###  V-D Performance Comparison

Figure 5: Performance comparison of optimal power allocation with equal power
allocation and fixed power allocation, $d_{1}=50$ meter.

Finally, Fig. 5 demonstrates the performance gain achieved by the optimal
solution over two different benchmark schemes. In particular, we depict the
performance comparison of optimal power allocation (OPA) scheme with equal
power allocation (EPA) and fixed power allocation (FPA) schemes. In EPA
scheme, we have taken $\alpha=0.5$ while in FPA, $\alpha$ is set to 0.4. From
the obtained results, it is observed that the optimal power allocation to
users provides considerable improvement in the pair outage performance of the
system. The average percentage improvement obtained by OPA over EPA and FPA,
are around $1.39\%$ and $14.60\%$, respectively. Besides, in Fig. 5, fixing
$d_{1}$ as $50$ meter, the effect of variation of $d_{2}$ on POP is also
presented. It is observed from the results that with a rise in $d_{2}$, POP
increases. This is because on increasing the $d_{2}$, achievable data rate at
$U_{2}$ decreases due to which POP of the system increases.

##  VI Concluding Remarks

In this work, we focused on analyzing the pair outage performance of a two-
user untrusted NOMA system. In particular, considering imperfect SIC at
receivers, we evaluated the closed-form expression of pair outage probability
POP for a decoding order that has been proven to be optimal from the
perspective of secure communication. We also presented power allocation
optimization to minimize the POP. We provided numerical results to validate
the accuracy of the derived analytical expressions, and presented the impact
of various system parameters on the outage performance. It has been observed
that significant performance gains of about $1.39\%$ and $14.60\%$,
respectively, are achieved by using optimal power allocation compared to equal
and fixed power allocations. In future, we wish to focus on generalizing the
POP study for all possible decoding orders in a multi-user untrusted NOMA
scenario.

## References

  * [1] Z. Ding, X. Lei, G. K. Karagiannidis, R. Schober, J. Yuan, and V. K. Bhargava, “A survey on non-orthogonal multiple access for 5G networks: Research challenges and future trends,” _IEEE J. Sel. Areas Commun._ , vol. 35, no. 10, pp. 2181–2195, Oct. 2017.
  * [2] Z. Ding, Z. Yang, P. Fan, and H. V. Poor, “On the performance of non-orthogonal multiple access in 5G systems with randomly deployed users,” _IEEE Signal Process. Lett._ , vol. 21, no. 12, pp. 1501–1505, 2014\.
  * [3] T. Hou, X. Sun, and Z. Song, “Outage performance for non-orthogonal multiple access with fixed power allocation over nakagami- ${m}$ fading channels,” _IEEE Commun. Lett._ , vol. 22, no. 4, pp. 744–747, 2018.
  * [4] X. Wang, J. Wang, L. He, and J. Song, “Outage analysis for downlink NOMA with statistical channel state information,” _IEEE Wireless Commun. Lett._ , vol. 7, no. 2, pp. 142–145, 2018.
  * [5] Z. Yang, Z. Ding, P. Fan, and N. Al-Dhahir, “A general power allocation scheme to guarantee quality of service in downlink and uplink NOMA systems,” _IEEE Trans. on Wireless Commun._ , vol. 15, no. 11, pp. 7244–7257, 2016\.
  * [6] B. M. ElHalawany and K. Wu, “Physical-layer security of NOMA systems under untrusted users,” in _Proc. IEEE GLOBECOM_ , United Arab Emirates, Dec. 2018, pp. 1–6.
  * [7] S. Thapar, D. Mishra, and R. Saini, “Novel outage-aware NOMA protocol for secrecy fairness maximization among untrusted users,” _IEEE Trans. Veh. Technol._ , vol. 69, no. 11, pp. 13 259–13 272, Sep. 2020.
  * [8] ——, “Decoding orders for securing untrusted NOMA,” _IEEE Networking Lett._ , vol. 3, no. 1, pp. 27–30, Jan. 2021.
  * [9] H. Sun, B. Xie, R. Q. Hu, and G. Wu, “Non-orthogonal multiple access with SIC error propagation in downlink wireless MIMO networks,” in _Proc. IEEE VTC-Fall_ , Montreal, Canada, Sep. 2016, pp. 1–5.
  * [10] X. Wang, R. Chen, Y. Xu, and Q. Meng, “Low-complexity power allocation in NOMA systems with imperfect SIC for maximizing weighted sum-rate,” _IEEE Access_ , vol. 7, pp. 94 238–94 253, July 2019.
  * [11] S. Thapar, I. Amin, D. Mishra, and R. Saini, “Secrecy outage probability analysis for downlink NOMA with imperfect SIC at untrusted users,” in _Proc. IEEE EUSIPCO_ , Virtual Conference, Aug. 2021, pp. 1–6.
  * [12] X. Yue, Z. Qin, Y. Liu, S. Kang, and Y. Chen, “A unified framework for non-orthogonal multiple access,” _IEEE Transactions on Communications_ , vol. 66, no. 11, pp. 5346–5359, May 2018.
  * [13] B. T. F.T. Miandoab, “NOMA performance enhancement-based imperfect SIC minimization using a novel user pairing scenario involving three users in each pair,” _Wireless Networks_ , vol. 26, no. 5, pp. 3735–3748, Mar. 2020\.
  * [14] P. K. Hota, S. Thapar, D. Mishra, R. Saini, and A. Dubey, “Ergodic performance of downlink untrusted NOMA system with imperfect SIC,” _IEEE Commun. Lett._ , vol. 26, no. 1, pp. 23–26, Nov. 2022.
  * [15] Z. Ding, P. Fan, and H. V. Poor, “Impact of user pairing on 5G nonorthogonal multiple-access downlink transmissions,” _IEEE Trans. Veh. Technol._ , vol. 65, no. 8, pp. 6010–6023, Aug. 2016.
  * [16] B. Xia, J. Wang, K. Xiao, Y. Gao, Y. Yao, and S. Ma, “Outage performance analysis for the advanced SIC receiver in wireless NOMA systems,” _IEEE Trans. Veh. Technol._ , vol. 67, no. 7, pp. 6711–6715, Mar. 2018.
  * [17] X. Chen, A. Beiijebbour, A. Li, H. Jiang, and H. Kayama, “Consideration on successive interference canceller (SIC) receiver at cell-edge users for non-orthogonal multiple access (NOMA) with SU-MIMO,” in _Proc. IEEE PIMRC_ , Hong Kong, China, Aug. 2015, pp. 522–526.
  * [18] A. Papoulis and S. U. Pillai, _Probability, random variables, and stochastic processes_. Tata McGraw-Hill Education, 2002.