#  InFusion: Inject and Attention Fusion for Multi Concept Zero-Shot Text-
based Video Editing

Anant Khandelwal
Glance AI
<EMAIL_ADDRESS>
###### Abstract

Large text-to-image diffusion models have achieved remarkable success in
generating diverse, high-quality images. Additionally, these models have been
successfully leveraged to edit input images by just changing the text prompt.
But when these models are applied to videos, the main challenge is to ensure
temporal consistency and coherence across frames. In this paper, we propose
InFusion, a framework for zero-shot text-based video editing leveraging large
pre-trained image diffusion models. Our framework specifically supports
editing of multiple concepts with pixel-level control over diverse concepts
mentioned in the editing prompt. Specifically, we inject the difference in
features obtained with source and edit prompts from U-Net residual blocks of
decoder layers. When these are combined with injected attention features, it
becomes feasible to query the source contents and scale edited concepts along
with the injection of unedited parts. The editing is further controlled in a
fine-grained manner with mask extraction and attention fusion, which cut the
edited part from the source and paste it into the denoising pipeline for the
editing prompt. Our framework is a low-cost alternative to one-shot tuned
models for editing since it does not require training. We demonstrated complex
concept editing with a generalised image model (Stable Diffusion v1.5) using
LoRA. Adaptation is compatible with all the existing image diffusion
techniques. Extensive experimental results demonstrate the effectiveness of
existing methods in rendering high-quality and temporally consistent videos.

Source Prompt: A silver jeep driving down a curvy road in the countryside.

Zero Shot Multi Concept Editing with Stable Diffusion v1.5 Silver Jeep
$\rightarrow$ Porsche car, countryside $\rightarrow$ Landmark of autumn

Zero Shot Multi Concept Editing with Stable Diffusion v1.5 Silver Jeep
$\rightarrow$ Porsche car, countryside $\rightarrow$ snowy winter

##  1 Introduction

With the rise in the creation and consumption of video content on social media
platforms, there is a need for generalised video creation and editing tools.
Despite the recent success of text-to-image diffusion models, their
applicability to video is limited since per-frame editing does not produce
consistent editing across all the frames. To overcome this limitation, recent
research introduced three types of text-to-video diffusion: a) first solution
is to train the model on large-scale video data [8] which require lot of
computing resources b) second solution is to fine-tune the image models on
single video [31] c) third solution is the zero-shot method [11, 19], which
requires no training, is compatible with pre-trained image diffusion models,
and requires fewer computing resources. In this paper, we employ the zero-shot
strategy for text-based video editing. However, the challenges associated with
zero-shot methods are: 1) Temporal Consistency: Cross-Frame Continuity 2)
Zero-Shot: no training or fine-tuning required 3) Flexible: compatible with
off-the-shelf, pre-trained image models. In this paper, we demonstrated the
use of a large-scale pre-trained text-to-image model (i.e., Stable diffusion
v1.5 [22]), which contains almost all the concepts, hence can be used for any
customised generation as opposed to the zero-shot method Fatezero [19], which
requires a one-shot tuned model for customised generation.
In this paper, we introduce a novel zero-shot framework for text guided video
editing with fine grained control over multiple concepts. Our framework,
InFusion, consists of two parts Inject and Attention Fusion. In the first
part, we inject features from residual block in decoder layers and attention
features (obtained from source prompt $P_{s}$) into the denoising pipeline for
editing prompt $P_{e}$. This injection step highlights the target concepts
since we injected the difference between residual block features for ($P_{s}$,
$P_{e}$) and combined them with attention injection (keys and values) to query
the source contents, keeping the unedited concepts as they are and scaling up
the edit concepts in the edit pipeline while scaling down the removed concepts
from the source pipeline. In the second part, we fuse the attention for edited
and unedited concepts using the mask extraction obtained from cross-attention
maps for $P_{e}$ and $P_{s}$, respectively. The fused attention preserves the
source content with editing concepts. Additionally, we mix the cross-attention
from the source and edited prompts to contain the unedited and edited
concepts, respectively. To summarize, our main contributions are as follows:

  * •

A novel zero-shot framework capable of editing multiple concepts with finer
details with a single editing pipeline. It achieves the best temporal
consistency and generates coherent edited videos, with no training involved
either for the generalised image diffusion model or for edited video
generation.

  * •

Inject for fine-grained control over editing concepts and Attention Fusion to
cut the edited part and paste the unedited part from source attention.

  * •

Experimental results demonstrate the flexible structure, shape, colour, and
style of editing with a temporally coherent generation of edited videos.

##  2 Related Work

Large-scale zero-shot methods for text-based image editing triggered interest
in videos as well. Recent developments introduced video editing methods,
namely, Tune-A-Video [31] is a one-shot method that inflates an image
diffusion model into a video model with cross-attention and generates edited
video by fine-tuning on a single video. Other methods based on the same idea
are Edit-A-Video [25], VideoP2P [14] and vid2vid-zero [30] which uses Null-
text inversion [15]for preserving unedited regions. However, all these methods
require fine-tuning of the pre-trained model over the input video. Following
these zero-shot methods are introduced, namely, FateZero [19] proposed
attention blending using features before and after editing, Text2Video-Zero
[11] denoise the latent directly to motions, Pix2Video [4] matches the current
frame with the previous frame in latent space. All the mentioned zero-shot
methods largely rely on manipulation with cross-attention maps for early-step
latent fusion to improve temporal consistency. However, as we demonstrate,
these methods are effective in editing high-level styles and shapes but less
effective in manipulating concepts at fine-grained levels. Our method does the
editing at finer levels using feature injection, which acts at pixel level.
Over and above, we apply attention feature injection and fusion to control
over the concepts mentioned for editing.

##  3 Preliminary

Latent Diffusion Models: Diffusion models [22, 9, 17, 26] are probabilistic
generative models that can generate the desired image from an initialised
Gaussian noise image $x_{T}\sim\mathcal{N}(0,\mathbf{I})$ by progressively
removing the noise at step ranging from $T$ to $0$. In general, the foundation
of diffusion models is based on two complementary random processes i.e.
forward and backward. During forward process or inversion the noise is added
at each step from $0$ to $T$ to clean image $x_{0}$ defined as:

$x_{t}=\sqrt{\alpha_{t}}\cdot x_{0}+\sqrt{1-\alpha_{t}}\cdot z$ (1)

where $z\sim\mathcal{N}(0,\mathbf{I})$ and $\alpha_{t}$ are the noise
schedule.
The backward process or reconstruction is aimed at progressively denoising the
image $x_{T}$, where at each step $t$ the cleaner version of image is obtained
than the previous step $t+1$, and finally to cleaned image at $0$. This is
achieved by a neural network $\epsilon_{\theta}(x_{t},t)$, which predicts the
added noise $z$ at each step. Once trained, this is applied at each backward
step which consists of applying $\epsilon_{\theta}$ to the current $x_{t}$,
and adding a Gaussian noise perturbation to obtain a cleaner $x_{t-1}$,
defined as:

$\displaystyle
p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t),\sigma_{t}),$
(2)
$\displaystyle\mu_{\theta}(x_{t},t)=\frac{1}{\sqrt{\alpha_{t}}}(x_{t}-\epsilon\frac{\sqrt{1-\alpha_{t}}}{\sqrt{1-\bar{\alpha}_{t}}}),$
(3)

where $\bar{\alpha}_{t}=\prod_{i}^{t}\alpha_{t}$, and $\epsilon$ is the
predicted noise. Neural network $\epsilon_{\theta}$ is trained using the mean
squared error given as:

$\mathcal{L}=\mathbb{E}_{x_{0},\epsilon,t}(||\epsilon-\epsilon_{\theta}(x_{t},t)||)$
(4)

Diffusion models are evolving very fast and have been integrated and trained
to generate images conditioned on multiple types of guiding signals, denoted
as $y$ in $\epsilon_{\theta}(x_{t},y,t)$ i.e. another image [24], text [12,
16, 21, 22] or class label [10]. In this work, we leveraged the pre-trained
text-conditioned Latent Diffusion Model (LDM), a.k.a. Stable Diffusion [22],
which performs the diffusion-denoising process in the latent space of the pre-
trained image auto-encoder network. The structure of the denoising backbone
$\epsilon_{\theta}$ is realized as a time-conditional U-Net [23] conditioned
on the guiding text prompt $P$.
Self-Attention and Cross-Attention: Layers of denoising U-Net consists of a
residual block [6], a self-attention block and a cross-attention block [29].
At the denoising step $t$, the residual block convolves features from previous
layer $\phi_{t}^{l-1}$ to produce the intermediate features $f_{t}^{l}$ at the
layer $l$. In self-attention block these intermediate features are projected
to produce the queries $q_{t}^{l}$, keys $k_{t}^{l}$ and values $v_{t}^{l}$.
The output feature of self-attention is then given as:

$\hat{f}_{t}^{l}=A_{t}^{l}v_{t}^{l},\textrm{ where
}A=Softmax(q_{t}^{l}{k_{t}^{l}}^{\mathbf{T}})$ (5)

Finally, the textual prompt $P$ features are projected into keys and values,
which are queried by self-attended spatial features, which when plugged into
the attention equation 5 will compute the features at the output of the cross-
attention block. These attention maps in the stable diffusion collectively
contain the rich information of structure, shape, and layout present in the
spatial features obtained from residual blocks. Cross-attention maps the
spatial pixels to the input text prompt and allows the editing [7] of multiple
granular objects that are present in the source video. Meanwhile, the features
in self-attention layers are employed in a plug-and-play manner [28] to retain
the structure/ layout/ shape of un-edited objects and facilitate style editing
over them with the edited prompt. Collectively, in this work, we leveraged the
combination of cross-attention and self-attention maps to perform consistent
video synthesis with delicate handling of multi-concept editing in a zero-shot
manner that can preserve and retain the layout and structure of edited and un-
edited parts in a prompt, respectively.

Figure 1: InFusion: Leveraging a pre-trained text-to-image model for video
editing ensures temporal consistency and editing accuracy with Inject and
Attention Fusion. The denoising pipeline for source prompt $P_{S}$ generates
the decoder latent from U-Net and attention features from source video, which
are injected into the denoising pipeline (initialised with inverted source
latent $z_{T}$) for edit prompt $P_{e}$.

##  4 InFusion

In this section we present InFusion, a framework designed to do zero shot text
based video editing of multiple concepts and ensuring temporal consistency
between frames of the edited video. Formally, given the source input video
$X_{0}=\\{x_{0}\\}_{i=1}^{N}$ with $N$ frames, source prompt $P_{s}$ and the
target prompt $P_{e}$, the goal of text driven video editing is to generate a
video $Y_{0}=\\{y_{0}\\}_{i=1}^{N}$ which aligns with prompt $P_{e}$,
faithfully preserves the unedited content of source video $X_{0}$ and
maintains the temporal consistency between frames. Our framework is built upon
Stable Diffusion v1.5 [22] a pre-trained and fixed text-to-image LDM model
denoted by $\epsilon_{\theta}(x_{t},P,t)$ where $P$ is the given prompt. This
model is based on the U-Net architecture with $T$ time-steps denoising as
shown in Figure 1 and discussed in Section 3. However, this model can generate
the frames as per the given prompt but to ensure temporal consistency between
frames and retaining the unedited contents of source video we made several
modifications to the pipeline.
Our key finding is that the fine grained control over the generated structure
is achieved by highlighting each concept using the a) edit directions obtained
from difference of spatial features from source and edit prompts b) accurate
mask extraction from source and edited cross-attention maps for fine grained
control over the structure of edited shape c) retain the unedited structure by
combining cross-attention maps from source and edit prompts for source and
edited parts respectively.

Input
layer 4
layer 7
layer 11
$f_{t}^{4}-{{f_{t}}^{*}}^{4}$

Figure 2: Visualising Top-3 principal components of diffusion features
(spatial features) obtained from the decoder of U-Net at different layers.

###  4.1 Inject

Spatial Features: Spatial features in text-to-image generation methods govern
the basic part of specifying the structure/shape/pose/scene layout. Even if
the prompt is descriptive, like "a Porsche car driving down a curvy road in
the countryside" or "a cat jumping over the bed" the model can generate
different images under different initial noise $x_{T}$. We hypothesise that in
text-based editing, the structure/pose can be controlled in a fine-grained
manner using the spatial features, and this hypothesis is motivated by the
analysis in [2, 28], which demonstrated the semantic segments obtained from
spatial features. To further investigate this fact, we did a PCA analysis, as
shown in Figure 2. Specifically, for each input image, we extract the features
$f_{t}^{l}$ from each layer in decoder of $\epsilon_{\theta}$ at each time-
step and compute the first three principal components as displayed in Figure 2
for layers 4,7 and 11. As seen, in the coarsest layer (layer 4), a crude blob
of Jeep structure is visible, but in layers 7 and 11, the Jeep structure is
clearly visible. Interestingly, the colour of the similar object (irrespective
of its pose) is same across all the frames at each layer. In text-based video
editing, we have to retain the source layout, and hence we choose to inject
the source features while editing with the prompt $P_{e}$, but while
injecting, we have to edit the structure of some objects, so we choose to
inject the source features in coarse layers only, since features at higher
layers gradually capture more fine-grained information based on the features
at coarse layers, and since these features eventually contribute to the error
predicted by the U-Net, the tendency will be more towards decreasing the error
at finer levels.
Feature Injection and Edit Direction: We now discuss the translation of the
given source ($x_{0}$, $P_{s}$) to edited video $y_{0}$ from the edited prompt
$P_{e}$. First, the source video is inverted using DDIM[26] to noise denoted
as $z_{T}$. Given the target prompt $P_{e}$, the generation of edited video
$y_{0}$ is performed using the same initial noise $z_{T}$ as shown in Figure
1. At each step $t$ of the backward process from initial noise $z_{T}$ for
source prompt, the guidance features $\\{f_{t}^{l}\\}$ are collected at each
layer from the denoising step $z_{t-1}=\epsilon_{\theta}(x_{t},P_{s},t)$. We
then inject these source guidance features $\\{f_{t}^{l}\\}$ during the
denoising steps of $y_{t}$ from the target prompt $P_{e}$. Specifically, we
replace the resulting features $\\{{f_{t}^{*}}^{l}\\}$ given as follows:

$z_{t-1}^{*}=\epsilon_{\theta}(y_{t},P_{e},t;\\{f_{t}^{l}-{f_{t}^{*}}^{l}\\})\textrm{
where }y_{T}=z_{T}$ (6)

The edited prompt "a Porsche car driving down a curvy road in a landmark of
autumn" contains the following edited concepts: a) "silver jeep $\rightarrow$
Porsche car" b) "countryside $\rightarrow$ landmark of autumn". As shown in
Figure 2 the spatial features $\\{f_{t}^{l}-{f_{t}^{*}}^{l}\\}$ at layer 4,
the jeep structure which was visible in $f_{t}^{l}$ for source prompt $P_{s}$
is now not clearly visible (looks moving towards car structure in some
frames), and the colour of the "countryside" is also changed, depicting that
it is moving from source concepts to edited concepts. Hence, we instead
injected the $\\{f_{t}^{l}-{f_{t}^{*}}^{l}\\}$ features since we want to move
from source concepts to edited concepts as mentioned in a) and b) rather than
retaining them. However, to reflect the complete structure change for edited
concepts, feature injection is not enough since it can only give the edit
directions (after a few steps $S_{1}$ as shown in Figure 1), these are
leveraged in controlling the self-attention to cut the edited concepts from
the source structure and paste the remaining part without any modification.

(a) Self-Attention Control

(b) Mask Extraction from cross-attention maps

Figure 3: a) Self-attention control to query contents from source image in
decoder part of U-Net b) Mask extraction strategy

Self-Attention Control: Figure 3(a) depicts the mechanism of self-attention
control, where the keys ${K_{t}^{s}}^{l}$ and values ${{V_{t}}^{s}}^{l}$ from
source (collected during denoising step of $x_{t}$) are injected during
denoising step of $y_{t}$. Specifically, as shown in Figure 3(a) the queries
$Q$ obtained from the injected spatial features
$\\{f_{t}^{l}-{f_{t}^{*}}^{l}\\}$ downscale the affinities for the edited
concepts (a and b as discussed in feature injection) due to the structure
changed for those concepts in feature injection and hence less similarity for
those concepts with keys ${K_{t}^{s}}^{l}$, less probability for the edited
concepts, which scale down the edited concepts in ${{V_{t}}^{s}}^{l}$. This
ensures the self-attention needed to control the propagation of edited parts.
However, the cross-attention maps, which correlate to the target prompt using
keys and values, make the edit concepts slowly integrate into the source
layout. Once the source layout with edited concepts is brewed in the diffusion
process (after $S_{2}$ steps as shown in Figure 1) we perform the mask-guided
mixing of self-attention and cross-attention from source and target prompt.
The Inject operation is defined as follows:

$z_{t-1}^{*}:=\left\\{\begin{matrix}\epsilon_{\theta}(y_{t},P_{e},t;\\{f_{t}^{l}-{f_{t}^{*}}^{l}\\}),t\in[0,S_{1}),l<L\\\
\epsilon_{\theta}(y_{t},P_{e},t;\\{{K_{t}^{s}}^{l},{{V_{t}}^{s}}^{l}\\}),t\in[S_{1},S_{2}],\forall
l\end{matrix}\right.$ (7)

###  4.2 Attention Fusion

Cut and Paste Self-Attention: We observed that synthesised videos using the
Inject operation faithfully generate the source layout with noise in the
edited concepts like overlapping parts of "jeep" and "Porsche car". Hence, we
propose the use of mask-guided editing of self-attention maps for faithful
reconstruction of edited concepts without any overlapping parts from the
source for those edited concepts. Inspired by the previous works [7, 27, 3],
it is revealed that cross-attention maps correlate to the target prompt and
hence can be used to inject the edited concepts from the attention maps
obtained from the edit prompt while keeping the source layout. Specifically,
at step $t$, we store all the self-attention and cross-attention maps from
source prompt $P_{s}$ and edit prompt $P_{e}$ during denoising steps using
fixed backbone U-Net with Inject in place for the edit prompt $P_{e}$. Then we
average the source and edit cross-attention maps for edited words in $P_{e}$
across all the heads and layers with spatial resolution $16\times 16$, the
resulting maps are denoted as ${A_{c}^{t}}^{src}\in R^{16\times 16\times N}$
(N is the number of tokens edited) and similarly in target attention for
edited words denoted as ${A_{c}^{t}}^{edit}\in R^{16\times 16\times N}$. We
then calculate the masks (shown in Fig. 3(b)) by thresholding the max pooled
maps of both ${A_{c}^{t}}^{src}$ and ${A_{c}^{t}}^{edit}$ denoted as $M_{s}$
and $M_{e}$ respectively. This captures only the foreground edited objects in
the binary mask. The resulting mask-guided self-attention is given as:

$\displaystyle s_{t}^{fused}$
$\displaystyle=M_{e}*s_{t}^{edit}+(1-M_{s})*s_{t}^{src}$ (8) $\displaystyle
s_{t}^{fused}$ $\displaystyle=fill(equal(s_{t}^{fused},0),s_{t}^{edit})$ (9)

which denotes that we take the foreground edited objects from target attention
maps only, which is "cut" and "paste" the remaining background from source
using $1-M_{s}$.
Cross-Attention Fusion: Similarly, the final fused cross-attention is obtained
by taking the target cross-attention maps $c_{t}^{edit}$ for all heads and all
layers for the edited words and taking the source attention $c_{t}^{src}$ for
unedited words. Mathematically, the fused cross-attention is given as:

$c_{t}^{fused}=\alpha_{w}*c_{t}^{edit}+(1-\alpha_{w})*c_{t}^{src}$ (10)

where $\alpha_{w}$ is the $1/0$ array indicating $1$’s where the word index is
edited and $0$ for source words in the edit prompt $P_{e}$ as compared to
source prompt $P_{s}$.

###  4.3 Spatio-Temporal Attention

Inspired by the previous works [31, 19, 33] we also leveraged the spatio-
temporal attention for consistent video synthesis for the edit prompt. Since
it has been observed that spatial features are the basic foundation of
structure in the synthesised video, we initialised the weights of temporal
attention with weights of spatial self-attention. Specifically, we integrated
the key frame attention into the spatial self-attention to align all the
frames with the key frame. Formally, let $z^{i}$ and $z^{k}$ denote the
embedding of the $i$-th frame and key frame, respectively. The modified
spatial attention is given as:

$Q=W^{Q}z^{i},K=W^{K}[z^{i};z^{k}],V=W^{V}[z^{i};z^{k}]$ (11)

where [;] denotes concatenation, $W^{Q},W^{K},W^{V}$ are the projection
matrices of the pre-trained model. Among the possible key-frame choices: a)
$k=\textrm{round}(\frac{N_{F}}{2}$ where $N_{F}$ is the total number of
frames) b) $k=i-1$ c) $k=i+1$. We find that there are no significant
differences in using any of the above choices, and hence, for ease, we choose
to take k=i-1 as the key frame. The resulting spatio-temporal attention map is
represented as $s_{t}\in R^{hw\times 2hw}$ where $2$ denotes the spatio-
temporal correspondence considered in calculating attention at a given time
step. Overall, this concludes the end-to-end zero shot editing with the
general-purpose diffusion model, which requires no fine tuning for editing the
concepts that are already present in the trained model.

##  5 Experimental Results

###  5.1 Implementation Details

All the experiments are conducted on one NVIDIA Tesla A100 40GB GPU. For zero-
shot text-based video editing, we use the trained Stable Diffusion 1.5 [22] as
the base text-to-image model, which has been converted to a video editing
model by incorporating spatio-temporal attention along with Inject and
Attention Fusion as shown in Figure 1. Throughout all experiments, the DDIM
sampler was used with $T=50$ steps and 7.5 as the classifier guidance for the
video editing pipeline. Out of 50 steps, the total number of steps for Inject
operation is 12, of which 6 are for feature injection and the rest 6 are for
self-attention injection, denoted as $S_{1}$ and $S_{2}$ in Figure 1
respectively. We explained above the choice of feature injection for coarse
layers only, hence L = 6 in our case. The mask threshold is set to 0.3 for the
case of editing "Porsche car", but it is subject to change from case to case.
Rest of the steps are consumed by Attention Fusion operation. Following
existing works [19, 33, 1, 5], we evaluated our method on videos from
DAVIS[18] dataset. The source prompt $P_{s}$ for these videos is obtained
using the caption model [13]. We develop the edit prompt $P_{e}$ by adding or
replacing some words.

(a) Input

(b) Ours

(c) FateZero

(d) T2V-Zero

Figure 4: Qualitative comparison of our method with FateZero and Text2Video-
Zero (T2V-Zero). Best viewed with zoom-in

(a) Input

(b) Ours

(c) vid2vid-zero

Figure 5: Qualitative comparison of our method with vid2vid-zero. Best viewed
with zoom-in

(a) Input

(b) CG Style

(c) Ghibli Cartoon

(d) Ink Painting

Figure 6: More Results: Demonstration of editing for cartoon and painting
styles

(a) Source Prompt: Blooming field of red poppy flowers. Target Prompt:
Blooming field of white poppy flowers

(b) Source Prompt: White Snowball Flowers. Target Prompt: Cherry Blossom
Flowers

(c) Source Prompt: Morning view over a farm. Target Prompt: Sunset view over a
farm

Figure 7: More Results: Demonstration of editing over fine-grained structure,
shape and color

###  5.2 Comparison with State-of-the-Art Methods

We compare our approach with four recent zero-shot methods, namely: FateZero
[19], vid2vid-zero [30], Pix2Video [4], and Text2Video-Zero [11], where the
code for all the baselines is publicly available, but they generate plausible
results only for editing templates mentioned in their code base. In
particular, vid2vid-zero generated frames totally different from the input.
For the sake of comparison, we have taken most of the videos that are common
among these baselines. As shown in Figure 4, the case of editing "swan" to
"yellow terosaur", this is mentioned by Fatezero as their limitation in their
paper. As shown, we are able to successfully replace the "swan" to "yellow
terosaur". Additionally, we compared this with Text2Video-Zero as well since
this uses the ControlNet, hence, it has added information to edit the videos,
but it generates frames that are not even close to the target prompt, instead,
it changes the colour of the wall to yellow. Similarly, in Figure 5, we have
compared our method against vid2vid-zero for the input video and target prompt
mentioned in their zero shot results. The generated frames, as per their code
base, show the backside of the "Porsche car" while moving forward, whereas in
the input video, the Jeep is moving forward and is front-facing. We showed
that our results are much better in terms of resemblance to input and target
prompts. Following [19, 33, 5, 31] we also conducted the quantitative
evaluation using the trained CLIP [20] model. Specifically, we show the
"Temporal" [5] to measure the temporal consistency between consecutive frames
by measuring cosine similarity between all pairs of consecutive frames.
Another measure, "Edit Acc" [20, 31] to measure the editing accuracy in the
generated frames, is calculated as the percentage of generated frames having a
higher similarity for the target prompt than the source prompt. Additionally,
we evaluated our method on two user studies metrics (’Edit’ and ’Temporal’)
are measured to measure the editing quality of our system from the perspective
of the applicability of our method in terms of usage in a real environment.
Specifically, we measured the rank of our proposed method for temporal
consistency (’Temporal’) across the frames in edited video and overall frame-
wise editing (’Edit’) for a given target prompt. We asked 20 subjects to rank
the editing method, with nine sets of comparisons in each study. As shown in
Table 1, our proposed method Infusion outperforms for all the CLIP metrics,
hence achieving the best temporal consistence and better per-frame editing
accuracy. Moreover, our method is truly zero shot since we have not used any
other pre-trained diffusion other than Stable Diffusion v1.5[22] as opposed to
Fatezero, which uses the one-shot trained model for the target prompt "A
Porsche car driving down a curvy road in the countryside". Apart from CLIP
metrics, our method is more reliable to put into real world editing since it
earns user preferences the best among all methods in both (Edit and Temporal)
aspects.

Method |  CLIP Metrics $\uparrow$ |  User Study $\downarrow$
---|---|---
Temporal | Edit Acc | Temporal | Edit
Tune-A-Video | 0.934 | 0.738 | 2.79 | 2.73
vid2vid-zero | 0.951 | 0.696 | 2.71 | 2.69
FateZero | 0.954 | 0.894 | 1.89 | 2.52
Pix2Video | 0.912 | 0.701 | 2.60 | 1.98
Text2Video-Zero | 0.959 | 0.902 | 1.81 | 1.77
Ours | 0.971 | 0.915 | 1.78 | 1.31

Table 1: Quantitative evaluation: For both user study and CLIP metrics
Infusion outperforms all the baselines in terms of temporal consistency and
per frame editing accuracy

Input

Ours

w/o $f_{t}^{l}$-inject

w/o att-inject

w/o Fusion

Figure 8: Ablation Study of Feature Injection, Attention Injection and
Attention Fusion. Prompts used are $P_{s}$: "A silver jeep driving down a
curvy road in the countryside", $P_{e}$: "A Porsche car driving down a curvy
road in the countryside". Without any of these components the foreground or
background or both details are missing from the edited video.

###  5.3 Ablation Study

Despite proving the effectiveness of Infusion, in this section we will present
the ablation study (shown in Figure 8) of various components in our editing
method to discuss the importance of each and their contribution towards the
edited video.
Injection is studied in Figure 8 as shown in the third and fourth columns. The
column named "w/o $f_{t}^{l}$-inject" is the ablation study of feature
injection, it depicts the complete change in frames including both foreground
(Porsche car) and background (curvy road and countryside as in source frames).
Though, we wanted to change the structure of "jeep $\rightarrow$ Porsche car",
but without feature injection it changes the complete source layout/background
as well, which is not a desirable change. The column named "w/o att-inject" is
the ablation for attention injection. As shown, without attention injection,
the rest of the background is faded, however, due to feature injection, the
source layout/background are retained ("curvy road and countryside").
Additionally, due to feature injection, the target concept is highlighted
("Porsche car") but the remaining source layout is faded, and hence it is very
much required to fill the source concepts on the highlighted target concepts,
which is proposed to be done with self-attention injection since it can inject
the source concepts while keeping the importance of highlighted target
concepts injected using differential features.
Attention Fusion is also studied in Figure 8, where we have removed self
attention fusion (shown in the fifth column) as discussed in equation 9 but
cross attention mixing is present. The resulting frames show little or no
change, with some shape distortion in the front part of the Jeep, as if the
diffusion is trying to align the Jeep structure with that of the car. As
expected, the change injected using differential feature and attention until
$S_{2}$ steps is wiped, and the concepts that are highlighted also get wiped
due to continued diffusion steps without masking, and hence it will generate a
structure similar to the source.

###  5.4 More Results: Flexible Structure, Color and Style

Our proposed method has shown decent results in editing the structure, colour,
and shape at finer details (Figures 7 and 6) with source content preserved.
Specifically, as shown in Figure 7(a) the source content contains lot of
cluttered red flowers, which are edited to white flowers (fine-grained colour
editing) with decent temporal consistency and accuracy over the frames.
Similarly, Figure 7(b) contains a lot of snowball flowers, which are edited to
"cherry blossom" flowers (fine-grained structure and shape editing) at all
viewing angles demonstrated over frames. Figure 7(c) demonstrated fine-grained
structure editing from "morning" view to "sunset" view with just deleting the
sun and associated rays over the farm and remaining everything intact. Other
results include style editing ranging from cartoon to painting styles, as
shown in Figure 6. Here, as we see, the details like nails, pose, etc. of the
woman are preserved. More results will be added to the github page:
https://infusion-zero-edit.github.io/

## Conclusion

In this work, we have presented the generalised zero-shot text-based video
editing framework with no additional models like ControlNet[32] and with no
training or fine-tuning of the pre-trained image diffusion model. Only pre-
trained Stable Diffusion v1.5[22] is used for all the edited videos without
the need for any customised image diffusion.

## References

  * [1] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In European conference on computer vision, pages 707–723. Springer, 2022.
  * [2] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko.  Label-efficient semantic segmentation with diffusion models.  arXiv preprint arXiv:2112.03126, 2021.
  * [3] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng.  Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing.  arXiv preprint arXiv:2304.08465, 2023.
  * [4] Duygu Ceylan, Chun-Hao Paul Huang, and Niloy J Mitra.  Pix2video: Video editing using image diffusion.  arXiv preprint arXiv:2303.12688, 2023.
  * [5] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  arXiv preprint arXiv:2302.03011, 2023.
  * [6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [7] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [8] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [9] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [10] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  The Journal of Machine Learning Research, 23(1):2249–2281, 2022\.
  * [11] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  arXiv preprint arXiv:2303.13439, 2023.
  * [12] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426–2435, 2022.
  * [13] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  * [14] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia.  Video-p2p: Video editing with cross-attention control.  arXiv preprint arXiv:2303.04761, 2023.
  * [15] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023.
  * [16] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [17] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  * [18] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  arXiv preprint arXiv:1704.00675, 2017.
  * [19] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  arXiv preprint arXiv:2303.09535, 2023.
  * [20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [21] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [22] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [23] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015.
  * [24] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–10, 2022.
  * [25] Chaehun Shin, Heeseung Kim, Che Hyun Lee, Sang-gil Lee, and Sungroh Yoon.  Edit-a-video: Single video editing with object-aware consistency.  arXiv preprint arXiv:2303.07945, 2023.
  * [26] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [27] Raphael Tang, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Jimmy Lin, and Ferhan Ture.  What the daam: Interpreting stable diffusion using cross attention.  arXiv preprint arXiv:2210.04885, 2022.
  * [28] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921–1930, 2023.
  * [29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [30] Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen.  Zero-shot video editing using off-the-shelf image diffusion models.  arXiv preprint arXiv:2303.17599, 2023.
  * [31] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  arXiv preprint arXiv:2212.11565, 2022.
  * [32] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  arXiv preprint arXiv:2302.05543, 2023.
  * [33] Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, and Jun Zhu.  Controlvideo: Adding conditional control for one shot text-to-video editing.  arXiv preprint arXiv:2305.17098, 2023.


# Learning to Mask: Towards Generalized Face Forgery Detection

###### Abstract

Generalizability to unseen forgery types is crucial for face forgery
detectors. Recent works have made significant progress in terms of
generalization by synthetic forgery data augmentation. In this work, we
explore another path for improving the generalization. Our goal is to reduce
the features that are easy to learn in the training phase, so as to reduce the
risk of overfitting on specific forgery types. Specifically, in our method, a
teacher network takes as input the face images and generates an attention map
of the deep features by a diverse multihead attention ViT. The attention map
is used to guide a student network to focus on the low-attended features by
reducing the highly-attended deep features. A deep feature mixup strategy is
also proposed to synthesize forgeries in the feature domain. Experiments
demonstrate that, without data augmentation, our method is able to achieve
promising performances on unseen forgeries and highly compressed data.

Index Terms—  Face forgery detection, Deepfake, Generalization, Multihead
attention, Deep mixup

##  1 Introduction

Recent works have made impressive progress in terms of generalizability, with
one mainstream achieving this by incorporating data augmentation that
reproduces the general pipeline of face forgery [1, 2, 3, 4] to synthesize
more training data. By training on these representative synthetic forgery
faces, detectors usually perform much better on unseen forgery types.

Fig. 1: A simple visualization of the generalization bound trained with
different approaches. The left figure shows the generalization bound derived
from the fitting of the training domain. The right figure shows how the
generalization bound is forced to be extended when the most easily fitted part
in the training domain is removed.

In this paper, we investigate an alternative route to better generalization.
We attribute the poor generalization of face forgery detectors to the
overfitting of the training data. Thus, we do not improve the generalization
by training on external data, in contrast, we avoid overfitting by reducing
the most easily fitted features. Fig. 1 is a toy example in data space to show
how our method improves the generalization. training domain refers to the
training forgery, while unseen domains indicate the forgeries not included in
the training. Fig. 1(a) shows the generalization bound of the model trained on
the training domain, it may include some other unseen domains. Fig. 1(b) shows
the basic motivation of our approach. When we reduce the training domain by
masking the easiest part to fit, the model is encouraged to have a wider
generalization bound such that the performance on unseen domains is better.

To this end, we introduce a teacher-student framework for face forgery
detection. The teacher network is a CNN-VIT hybrid model where the ViT is used
to generate an attention map by diverse multihead attention (MHA). The
attention map indicates the degree of influence of local features on the
binary classification and is used to mask the highly-attended local features
of the student model. The student network only learns with these low-attended
features, in other words, normal samples become hard samples after the
masking, resulting in a wider generalization bound. We also introduce deep
mixup that mixes up samples in the deep feature level, which can further
extend the training distribution by the linear interpolations of features. Our
contributions can be summarized into 3-folds: I. We designed a novel teacher-
student based face forgery detection framework, in which the student network
learns to detect face forgery with only the low-attended features identified
by the teacher network. II. We also introduce the deep mixup, which
interpolates the deep features, to extend the training distribution; IIi.
Experiments demonstrate the improved generalization of our method, as well as
the robustness to compression.

Fig. 2: Overview of the proposed method.

##  2 Related Work

In this part, we briefly review works on face forgery detection, with a
particular focus on recent works that achieve great success in
generalizability. And we also introduce basic concepts of vision transformer
and masked image modeling.

###  2.1 Generalized Face Forgery Detection

Data augmentation based method is the mainstream of current face forgery
detection methods that have good generalizability. These methods synthesize
their own data using the general process following the general face forgery
pipeline: I. Choose a proper target and source face from the database; II.
Crop the face with some strategies and conduct transformations; III. Swap the
faces and employ post-processing. [5] is the first to use such data
augmentation to detect face forgeries based on their observation of the visual
artifacts on forged faces. Subsequently, many methods are proposed to achieve
better generalization through data augmentation, and these methods generally
incorporate additional learning objectives, including forgery boundary
localization [1], self-consistency learning[4], blending type prediction[2].
The incorporation of synthetic data encourages models to learn features of
more commonality, resulting in better generalization.

###  2.2 Vision Transformer and Masked Image Modeling

Vision Transformer (ViT) [6] adopts the Transformer by considering an image as
a sequence of patches, achieving incredible performances on various vision
tasks. The basic component of ViT is the transformer encoder block, in which
multihead attention modules transform patch embeddings (feature vectors) into
queries, keys, and values, to model the relationship among each element of the
embedding sequence. Some recent works using ViT have achieved promising
results in face forgery detection [7, 8]. Masked Image Modeling (MIM) is a
self-supervised learning approach for pretraining ViT backbones by masking
some patches of the input and predicting the full input from visible patches.
MIM has attracted considerable attention due to its superiority in visual
model training. In this paper, we only use a tiny ViT to generate spatial
attention maps from CNN features and use MIM to mask features instead of raw
input.

##  3 Proposed Method

###  3.1 Overview

Fig. 2 shows the overview of our proposed method. Given an input RGB face
image of size $H\times W\times 3$ where $H,W$ stand for its height and width,
it is fed to two homogeneous CNNs. For the teacher network, the CNN backbone
produces a feature map $X_{t}\in\mathbb{R}^{h\times w\times c}$. In the
vanilla transformer, an image is split into patch sequences, flattened, and
linearly embedded with fully connected layers, to generate a vector sequence.
In this work, we directly replace the embedding operation with the CNN
backbone and take $X_{t}$ as the vector sequence, which serves as the input
for the ViT encoder. To this end, $X_{t}$ is reshaped into a feature vector
sequence $[x_{1},x_{2},...,x_{h\times w}]$ where $x_{i}\in\mathbb{R}^{c}$. We
prepend the sequence a learnable class embedding $x_{0}\in\mathbb{R}^{c}$
whose state is the class representation at output, and add a position
learnable embedding sequence to generate the input of ViT
$X_{vit}\in\mathbb{R}^{(h\times w)\times c}$. Meantime, the ViT branch
generates an attention map that indicates the regions contributing the most to
the final decision on fake or real. More details are presented in Section 3.2.
Then, features of the student network are masked using the attention map, and
the student network takes the masked features as input, to make its decision.
Moreover, we use the interpolations of deep features to synthesize more data.

###  3.2 Attention Based Mask Generation with diverse MHA

For each block in the transformer encoder, a diverse MHA with $h$ heads maps
$X_{vit}$ into $Q,K,V$ using 3 different MLP layers, and for each head, we
have an attention map

$\displaystyle A_{i}=\textit{softmax}(Q_{i}K_{i}^{\top}/\sqrt{c/h}).$ (1)

Then final attention map is the average of $h$ head attention matrices:

$\displaystyle A=\frac{1}{h}\sum_{i=1}^{h}A_{i}.$ (2)

Meantime, we further adopt a diversity loss following [9], to encourage more
diversity of different heads in the last encoder block:

$\displaystyle L_{diversity}=\frac{1}{h^{2}}(\sum_{0<i,j<h\atop
i=j}-d_{i,j}+\sum_{0<i,j<h\atop i\neq j}(1-d_{i,j})),$ (3)

where $d_{i,j}$ denotes
$\frac{A_{i}A_{j}^{\top}}{\left\|A_{i}\right\|_{2}\left\|A_{j}\right\|_{2}}$.

Differing from the thought of MIM, we mask the features instead of the raw
input. Put simply, denote the top $k$ highest elements in the attention map
$A$ as a collection $k(A)$, where $i,j$ is the spatial position index. We
first calculate a binary mask $\mathcal{M}$ with

$\displaystyle\mathcal{M}_{i,j}=\begin{cases}0&\text{if}\phantom{se}A_{i,j}\in
k(A),\\\ 1&\text{otherwise}.\end{cases}$ (4)

For the student network, the deep feature $X_{s}\in\mathbb{R}^{h\times w\times
c}$ is masked by $\mathcal{M}$, average pooled, to generate final feature
$F=\textit{Pooling}(X_{s}\odot\mathcal{M})\in\mathbb{R}^{c}$.

###  3.3 Deep Mixup

Following the thought of mixup [10], we use the interpolations to synthesize
more data. Differing from [10], we do not employ mixup on raw RGB input, in
contrast, we apply the operation to deep features. Specifically, within a
training batch, we generate new feature $F^{\prime}$ and corresponding label
$y^{\prime}$ with 2 samples $F_{i}$, $F_{j}$ and their labels by a mixup
factor $\lambda\in[0,1]$:

$\displaystyle F^{\prime}=\lambda F_{i}+(1-\lambda)F_{j},$ (5) $\displaystyle
y^{\prime}=\lambda y_{i}+(1-\lambda)y_{j}.$ (6)

Then a MLP classifier takes as input $F^{\prime}$ and makes the prediction.
Overall, there are 4 losses involved in the training of our framework:

$\displaystyle L_{total}=L^{t}_{cls}+L^{t}_{vit}+\alpha L_{diversity}+L^{s},$
(7)

where $L^{t}_{cls}$, $L^{t}_{vit}$, and $L^{s}$ are the binary classification
losses for classifier branch, ViT branch of the teacher network, and student
network respectively, and $\alpha$ is the weight.

##  4 Evaluations

###  4.1 Settings

Datasets. We use FaceForensics++ (FF++) [11] that consists of DeepFakes (DF),
Face2Face (F2F), FaceSwap(FS), and NeuralTextures (NT), CelebDF [12],
WildDeepfake (WDF) [13] to evaluate the proposed method. All faces are cropped
and resized to 224 $\times$ 224 and normalized to $[0,1]$ by dividing 255.

Implementation Details. The CNN backbone is Xception [14] initialized with
ImageNet pre-trained weights for default. The ViT of the teacher network
consists of 6 transformer blocks with each having a 6-head MHA module with
latent dimension 2048. Both networks are trained using SGD optimizer with a
initial learning rate 1e-2, momentum 0.9, and exponential decay factor 0.9.
The batch size is 32, and each epoch has 300 iterations. We use frame-level
AUC (area under the curve) as the evaluation metric. The mask ratio is 25% and
$\alpha$ is 0.1. Experimental results of other compared methods are cited from
the original papers.

###  4.2 In-dataset Evaluations

First, we evaluate our method against heavy video compression. Table 1 shows
the in-dataset results of our method and compared methods. We train and test
our method on the same heavily compressed datasets which is a challenging
scenario in real world. It is clear that our method has an obvious improvement
compared to the baseline model. Moreover, our overall performances outperform
the state-of-the-art method by 0.46%, 1.80%, 0.69%, and 1.70% respectively,
showing very promising robustness to compression.

Methods | DF | F2F | FS | NT
---|---|---|---|---
Xception [11] | 99.08 | 93.77 | 97.42 | 84.23
SPSL [15] | 98.50 | 94.62 | 98.10 | 80.49
Ours-Xception | 99.54 | 96.42 | 98.79 | 85.93
Table 1: Overall performances on FF++ (c40) in-dataset evaluation.

More results of in-dataset performances are shown in Table 2. We evaluate our
method on lightly compressed FF++, CelebDF, and WDF. It can be seen that on
various datasets, our method is able to achieve competitive results compared
to the state-of-the-art methods. Performances of our method are stable across
different datasets, even though we are not the best on a single dataset, the
overall accuracy is still better.

Methods | FF++ (c23) | CelebDF | WDF | Avg
---|---|---|---|---
SPSL [15] | 95.32 | - | - | -
RFM [16] | 98.79 | 99.94 | 83.92 | 94.21
Add-Net [13] | 97.74 | 99.55 | 86.17 | 94.48
F3-Net [17] | 98.10 | 98.93 | 87.53 | 94.85
Ours-Xception | 98.59 | 99.45 | 86.70 | 94.91
Table 2: Performances of in-dataset evaluation on FF++ (c23), CelebDF, and
WDF.

###  4.3 Generalizability Evaluations

The results of cross-domain evaluations are presented in Table 3. In this
setting, models are trained and evaluated on different datasets. Existing
methods tend to train on the entire 4 subsets of FF++ and test on the unseen
datasets. Our method, in contrast, can achieve much better performances when
trained on only one single dataset. Here we present the results of our methods
trained on different subsets of FF++ (c23), namely FF-DF, FF-F2F and FF-NT.
Compared to the Xception baseline [11], cross-dataset accuracy on CelebDF is
improved by 16.11% with only FF-DF. This considerable improvement does not
come from the attention masking, but also comes from deep mixup that
successfully extends the training distribution with the data from a single
domain.

Methods | Train | In-dataset | CelebDF
---|---|---|---
Xception [11] | FF++ | 99.70 | 65.30
SPSL [11] | FF++ | 99.80 | 67.40
Ours | FF-DF | 99.90 | 81.41
Ours -w/o mixup | FF-DF | 99.91 | 80.04
Ours | FF-F2F | 99.62 | 74.92
Ours -w/o mixup | FF-F2F | 99.65 | 72.48
Ours | FF-NT | 98.54 | 76.52
Ours -w/o mixup | FF-NT | 98.75 | 76.39
Table 3: Cross evaluations on CelebDF, our method is trained on the subset of
FF++.

###  4.4 Ablation Studies

First, we evaluate the effectiveness of different masking strategies and the
results are shown in Table 4. Here high stands for the default setting that
filters out the highly-attended features while low filters out the low-
attended features, and random denotes random masking. All strategies have the
same mask ratio of 25%. Here, the results of FF++ (c23) and WDF are under an
in-dataset setting, and results of CelebDF are from models trained on FF-DF
(c23). We can see that masking highly-attended features may compromise the in-
dataset performances but can improve the generalization with a considerable
gap, while masking low-attended features is the exact opposite. As for the
random masking, the performances are between high and low.

Strategy | FF++ (c23) | WDF | CelebDF |
---|---|---|---|---
high | 98.59 | 86.70 | 81.41 |
low | 99.10 | 87.32 | 77.48 |
random | 99.04 | 87.44 | 80.95 |
Table 4: Effectiveness of the proposed method with different masking
strategies.

In previous experiments, we use homogeneous student and teacher networks as
their features that contribute to prediction should be aligned. We further
evaluate different architectures in Table 5. The results are in-dataset
performance on FF++ (c23). We can see that with different architectures, the
overall accuracy floating is relatively small. This may be because when the
features of different architectures are not aligned, cross-architecture
framework behaves more like random masking.

Student Teacher | ResNet18 | ResNet34 | Xception
---|---|---|---
ResNet18 | 95.59 | 96.10 | 96.02
ResNet34 | 96.31 | 96.24 | 96.52
Xception | 98.64 | 98.69 | 98.59
Table 5: Effectiveness of different architectures for student and teacher
networks.

##  5 Conclusion

This paper presents another path to achieve generalizability in face forgery
detection. In particular, we introduce a teacher-student framework, where the
student network learns on low-attended features guided by the teacher network,
to obtain a wider generalization bound. A deep mixup is also incorporated to
enhance the generalization using one single domain. Experiments on several
challenging datasets have demonstrated that our method shows good
generalization and is robust to compression compared to state-of-the-art
works.

Acknowledgement This work is supported in part by the National Key Research
and Development Plan of China under Grant 2020YFB1005600, in part by the
National Natural Science Foundation of China under grant numbers 62122032, and
U1936118, in part by Qinglan Project of Jiangsu Province, and “333” project of
Jiangsu Province. This work has been partially supported by the China
Scholarship Council (CSC), file No.202109040029.

## References

  * [1] Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo,  “Face x-ray for more general face forgery detection,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 5001–5010.
  * [2] Kaede Shiohara and Toshihiko Yamasaki,  “Detecting deepfakes with self-blended images,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18720–18729.
  * [3] Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, and Jue Wang,  “Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18710–18719.
  * [4] Tianchen Zhao, Xiang Xu, Mingze Xu, Hui Ding, Yuanjun Xiong, and Wei Xia,  “Learning self-consistency for deepfake detection,”  in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 15023–15033.
  * [5] Yuezun Li and Siwei Lyu,  “Exposing deepfake videos by detecting face warping artifacts,”  arXiv preprint arXiv:1811.00656, 2018.
  * [6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  in International Conference on Learning Representations, 2020.
  * [7] Sohail Ahmed Khan and Hang Dai,  “Video transformer for deepfake detection with incremental learning,”  in Proceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 1821–1828.
  * [8] Ping Wang, Kunlin Liu, Wenbo Zhou, Hang Zhou, Honggu Liu, Weiming Zhang, and Nenghai Yu,  “Adt: Anti-deepfake transformer,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 2899–1903.
  * [9] Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo,  “Learning deep bilinear transformation for fine-grained image representation,”  Advances in Neural Information Processing Systems, vol. 32, 2019\.
  * [10] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz,  “mixup: Beyond empirical risk minimization,”  in International Conference on Learning Representations, 2018.
  * [11] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner,  “Faceforensics++: Learning to detect manipulated facial images,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1–11.
  * [12] Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu,  “Celeb-df: A large-scale challenging dataset for deepfake forensics,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3207–3216.
  * [13] Bojia Zi, Minghao Chang, Jingjing Chen, Xingjun Ma, and Yu-Gang Jiang,  “Wilddeepfake: A challenging real-world dataset for deepfake detection,”  in Proceedings of the 28th ACM international conference on multimedia, 2020, pp. 2382–2390.
  * [14] François Chollet,  “Xception: Deep learning with depthwise separable convolutions,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1251–1258.
  * [15] Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, and Nenghai Yu,  “Spatial-phase shallow learning: rethinking face forgery detection in frequency domain,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 772–781.
  * [16] Chengrui Wang and Weihong Deng,  “Representative forgery mining for fake face detection,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14923–14932.
  * [17] Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, and Jing Shao,  “Thinking in frequency: Face forgery detection by mining frequency-aware clues,”  in European conference on computer vision. Springer, 2020, pp. 86–103.


Supplementary Figures

[width=]images/ITI_ABA.jpg

Figure : Model training for image enhancement. Images of domain A are
translated to domain B by generator AB (blue). The enhanced images are
translated back to domain A by generator BA (yellow). The perceptual image
quality is optimized with the use of discriminator B, which distinguishes
between real images of domain B (bottom) and enhanced images by generator AB
(top). The noise term for the reconstruction of the original images by
generator BA is obtained by the noise-estimator (NE).

[width=]images/ITI_ID.jpg

Figure : Model training for the identity transformations. Images are optimized
to remain unchanged for a mapping into the same domain. Top panel: mapping
from domain A to domain A by generator BA (yellow). Bottom panel: mapping from
domain B to domain B by generator AB (blue). The noise term for generator BA
is obtained from the original image by the noise-estimator (NE). In addition
the network is optimized to estimate the same noise before and after
transformation (top panel). In case of different resolutions the low-quality
images are first upsampled for A-A and downsampled for B-B.

[width=0.8]images/Multi-channel.jpg

Figure : Example of the multi-channel use of the discriminator. Each image
channel of the source low-quality domain (top) is simultaneously translated to
the target high-quality domain (bottom) by the generator AB (SOHO/EIT+MDI-to-
SOHO/AIA+HMI). For each translated image channel a separate discriminator is
used. The inter-channel consistency is assessed by a single discriminator that
uses the combined set of channels as input.

##  1 Supplementary Tables

Table : Data sets for training of Instrument-To-Instrument translation. The effective number of samples refers to the number of independent patches that can be extracted from the training set. Observations of Hinode/SOT have a varying field-of-views, the provided value gives the upper limit.  Instrument | Observables | Train | Effective Number of Samples | Test
---|---|---|---|---
SDO | 171, 193, 211, 304, LOS magnetogram | 9562 | 3.8e4/15.3e4 | 2404
SOHO | 171, 195, 284, 304, LOS magnetogram | 4855 | 31.1e4 | 1410
STEREO | 171, 195, 284, 304 | 3146 | 20.1e4 | 822
SDO/HMI | continuum | 2751 | 2.8e4 | 595
Hinode/SOT | continuum | 2643 |  $\sim$5286 | 668
KSO synoptic | H$\alpha$ | 2106 | 3.4e4 | 422
KSO LQ | H$\alpha$ | 1461 | 23.4e4 | 384
KSO Film | H$\alpha$ | 4067 | 1.6e4 | 772
Table : Overview of the Training parameters. For each instrument translation we train our model for the given number of iterations (Iter.). The upsampling parameter gives the number of upsampling operations that are applied to the low-quality images (Up). The diversity parameter is only considered for ground-based observations ($\lambda_{d}$). The patch size is given for the low-quality samples, such that the high-quality patch size corresponds to (Patch Size)$\cdot 2^{\text{Up}}$. Instruments | Iter. | Up | $\lambda_{d}$ | Patch Size [pixels]
---|---|---|---|---
SDO/HMI $\xrightarrow{}$ Hinode continuum | 260 000 | 2 | 0 | 160$\times$160
SOHO/EIT+MDI $\xrightarrow{}$ SDO/AIA+HMI | 460 000 | 1 | 0 | 128$\times$128
STEREO/EUVI $\xrightarrow{}$ SDO/AIA | 520 000 | 2 | 0 | 128$\times$128
KSO LQ $\xrightarrow{}$ KSO HQ H$\alpha$ | 260 000 | 0 | 1 | 256$\times$256
KSO Film $\xrightarrow{}$ KSO CCD H$\alpha$ | 240 000 | 0 | 1 | 256$\times$256
STEREO/EUVI $\xrightarrow{}$ SDO/AIA+HMI | 340 000 | 0 | 0 | 256$\times$256
Table : Evaluation of the intercalibration of SOHO, STEREO and SDO observations. We compare our ITI calibration with the baseline calibration, where we consider smoothed light-curves for STEREO to mitigate the different vantage points to SDO. For SOHO we compare temporally aligned observations. We compute the mean absolute difference (MAE) between the calibrated and SDO light-curves and the cross-correlation (CC) between the calibrated STEREO and SDO light-curves. Instrument | Method | MAE $\downarrow$ | CC $\uparrow$
---|---|---|---
STEREO/EUVI - 171 Å | baseline | 18.6 | 0.68
| ITI | 12.7 | 0.76
STEREO/EUVI - 195 Å | baseline | 29.0 | 0.97
| ITI | 17.5 | 0.97
STEREO/EUVI - 284 Å | baseline | 20.1 | 0.96
| ITI | 10.2 | 0.97
STEREO/EUVI - 304 Å | baseline | 2.1 | 0.91
| ITI | 2.2 | 0.92
STEREO/EUVI - avg | baseline | 17.4 | 0.88
| ITI | 10.7 | 0.90
SOHO/EIT - 171 Å | baseline | 7.7 | -
| ITI | 12.4 | -
SOHO/EIT - 195 Å | baseline | 9.2 | -
| ITI | 8.5 | -
SOHO/EIT - 284 Å | baseline | 3.3 | -
| ITI | 1.8 | -
SOHO/EIT - 304 Å | baseline | 1.3 | -
| ITI | 2.4 | -
SOHO/EIT - avg | baseline | 5.4 | -
| ITI | 6.3 | -

##  2 Instruments and Data

This section describes the characteristics of the considered instruments for
the evaluation of our method. The pair-wise correspondence of the data sets is
discussed in Sect. LABEL:section:iti_data_set_pairs. The Solar Dynamics
Observatory (SDO) is a space-based mission located in a circular
geosynchronous orbit, that provides science data since 1 May 2010
[pesnell2012sdo]. For our study we use EUV filtergrams from the Atmospheric
Imaging Assembly (AIA; [lemen2012aia]), and line-of-sight (LOS) magnetograms
and continuum observations from the Helioseismic and Magnetic Imager (HMI;
[schou2012hmi]). Both instruments provide solar full-disc images with a
spatial sampling of 0.6 arcsec pixels and a spatial resolution of 1.5 arcsec.
The data are recorded by 4096$\times$4096 pixel CCDs. The AIA instrument
operates at a cadence of 12 seconds and provides EUV filtgrams in seven band
passes. In this study, we consider Fe IX (171 $\AA$), Fe XII, XXIV (193
$\AA$), Fe XIV (211 $\AA$) and He II (304 $\AA$) filtergrams. The emission
lines are primarily formed in the solar corona and chromosphere and are
associated with peak temperatures of ion formation at $6.3\cdot 10^{5}$ K (Fe
IX; quiet corona, upper transition region), $1.6\cdot 10^{6}$ K, $2.0\cdot
10^{7}$ K (Fe XII, XXIV; corona, hot flare plasma), $2.0\cdot 10^{6}$ K (Fe
XIV; active-region corona) and $5.0\cdot 10^{4}$ K (He II; chromosphere,
transition region) [lemen2012aia]. The HMI instrument provides maps of the
photospheric magnetic field strength and orientation by observing
polarizations of the Fe I absorption line (6173 $\AA$). The continuum
observations are calculated from six points in the Fe I line [schou2012hmi].
For both the HMI magnetograms and continuum observations, we use the 720
seconds series. The Solar and Heliospheric Observatory (SOHO) is a space-based
mission, located at Lagrange point L1, that was launched in December 1995
[domingo1995soho]. In this study we use data from the Extreme-ultraviolet
Imaging Telescope (EIT; [delaboudiniere1995eit]) and LOS magnetograms from the
Michelson Doppler Imager (MDI; [scherrer1995mdi]). Both instruments provide
full-disc 1024$\times$1024 pixels images of the Sun at a spatial sampling of
2.6 arcsec pixels and about 5 arcsec spatial resolution. From EIT we use Fe IX
(171 $\AA$), Fe XII (195 $\AA$), Fe XV (284 $\AA$) and He II (304 $\AA$)
filtergrams. The peak temperatures for ion formation are $1.3\cdot 10^{6}$ K,
$1.6\cdot 10^{6}$ K, $2.0\cdot 10^{6}$ K and $8.0\cdot 10^{4}$ K, respectively
[delaboudiniere1995eit]. The MDI instrument is the predecessor of SDO/HMI and
derives the LOS magnetograms in the Ni I 6768 A absorption line
[scherrer1995mdi]. The Solar Terrestrial Relations Observatory (STEREO) is a
twin-satellite mission that operates two identical satellites on two orbits
close to 1 AU. The orbits are selected to lead to a yearly separation of the
spacecrafts by about 45 degree, as viewed from the Sun [kaiser2008stereo]. The
mission was launched in 2006 and provides stereoscopic observations of the Sun
since then. In this study, we use the Exteme Ulraviolet Imager (EUVI;
[wulser2004euvi]) of the Sun–Earth Connection Coronal and Heliospheric
Investigation (SECCHI; [howard2008secchi]) instrument. The imager provides
full-disc filtergrams with 2048$\times$2048 pixels with a spatial sampling of
about 1.6 arcsec pixels and 3.2 arcsec spatial resolution [howard2008secchi].
Similarly to SOHO/EIT, filtergrams of Fe IX (171 $\AA$), Fe XII (195 $\AA$),
Fe XV (284 $\AA$) and He II (304 $\AA$) are recorded. The associated peak
temperatures are in the same range as for the SOHO/EIT filters (see above).
The Solar Optical Telescope (SOT) onboard the Hinode satellite (launched in
2006) is a 50 cm aperture telescope that provides high-resolution images of
partial fields of the Sun [tsuneta2008sot]. The instrument comprises multiple
broad- and narrow-band filters and provides spatial resolutions of up to 0.2
arcsec and a pixel scale of 0.054 arcsec pixels. In this study, we use
continuum observations centered at a wavelength of 6684 $\AA$ of the Broadband
Filter Imager (BFI). The BFI instrument provides a field-of-view of
218”$\times$109”, producing images with 4096$\times$2048 pixels, and operates
at a cadence $<10$s [tsuneta2008sot]. Observations are recorded on demand.
Kanzelhöhe Observatory for Solar and Environmental Research (KSO;
https://kso.ac.at/) provides ground-based solar full-disk H$\alpha$
filtergrams at a spatial resolution of 2 arcsec. The data are recorded by a
2048$\times$2048 pixel CCD corresponding to a sampling of about 1 arcsec
pixels. KSO regularly takes H$\alpha$ images at a cadence of 6 seconds and
provides a fully automated data reduction and data provision pipe line, which
allows for data access in near real time [otruba2003halphaKSO, potzi2015real,
potzi2021kso]. The current instrument setup is in operation since 2008. The
H$\alpha$ line is formed by absorption at 6563 $\AA$ in the solar chromosphere
and by cooler plasma in the solar corona (filaments). At KSO, regular
monitoring of the Sun in the H$\alpha$ spectral line is provided since 1973.
The observations during 1973 and 2000 were recorded on photographic film at a
cadence of about 4 minutes and were later digitized in 1024$\times$1024 pixels
format [potzi2007scanning, potzi2008scanningResults].

##  3 Data pre-processing

###  Space-based observations

For each observation we center the Sun, normalize the field-of-view to 1.1
solar radii and crop the frame, such that the extent of the solar-disk is
independent of yearly variations caused by the elliptic orbit (cf.
[galvez2019machine]). We correct for instrumental degradation and normalize
the exposure time [boerner2014photometric]. For STEREO/EUVI, SOHO/EIT,
Hinode/SOT we use the IDL eit_prep, secchi_prep and fg_prep routines,
respectively. For SDO/AIA we use the SDO autocalibration by
[santos2021aiacalibration]. The STEREO/EUVI instrument shows significant
degradations of the 304 $\AA$ channel, that are not considered by the STEREO
prep routine ($\sim 50\%$). We correct for this degradation by extracting the
quiet-Sun regions over the full data set and fitting the degradation as a
first-order polynomial. The maximum intensity value of each instrument channel
is estimated from the maximum intensity values over the full data set by
$\hat{I}_{max}=\text{mean}(I_{max})+0.5\cdot\text{std}(I_{max})$, where
$I_{max}$ refers to the maximum intensity values of the images and
$\text{std}$ to the standard deviation. We clip negative values and values
larger than the estimated maximum intensity and normalize to [0, 1]. We apply
a asinh stretch for EUV filtergrams and scale the data afterwards to the
interval [-1, 1], such that it corresponds to a $\tanh$ activation function

$\hat{x}=\frac{\text{asinh}(x/a)}{\text{asinh}(1/a)}2-1,$ (1)

with $a=0.005$. This stretching function provides for a logarithmic behavior
for large values and a linear behavior for small values. For LOS magnetograms
we use a linear normalization of values between $[-1000,1000]$ Gauss to
$[-1,1]$ and set all off-limb values to zero. For the continuum observations
of SDO/HMI and Hinode we analogously estimate the maximum value and scale
linearly between 0 and the maximum value.

###  Ground-based observations

For all KSO observations we apply the same preprocessing. We shift the Sun to
the image center, crop the field-of-view to 1 solar radius and resize it to
512$\times$512 pixels and 1024$\times$1024 pixels for the Film-to-CCD and Low-
to-High translation, respectively. The center-to-limb variation is corrected
by plotting the theoretical correction $\mu=\cos{\theta}$ against the
intensity values of the image, where $\theta$ refers to the heliocentric
angle, and fitting a fourth order polynomial that gives the intensity
correction $I_{corr}(\mu)$ (cf. [diercke2018counter]). For KSO observations of
the recent instrument (CCD), we scale values between 0.65 and 1.5 to $[-1,1]$
with a asinh stretch $(a=0.5)$. For the KSO film observations we apply the
same scaling but adjust the scaling to values between 0.39 and 1.94 to account
for the variations of the degraded observations. We apply the preprocessing
per frame and set all off-limb pixels to $-1$.

##  4 Model architecture

[width=]images/ITIGenerator.jpg

Figure : Overview of the generator architecture. Our generators are composed
of an input block, three downsampling blocks, an intermediate block, three
upsampling blocks and an output block. The input block transforms the image to
64 filter channels. Each downsampling block reduce the spatial resolution by 4
while increasing the number of filter channels by 2. The number of
convolutional blocks is linearly increased. The upsampling blocks operate in
reverse order. At the intermediate layer the noise term for generator BA is
added by concatenation (dashed line). The noise term is matched to the
dimensions of the intermediate layers by an initial convolution block and an
upsampling block. Features from the downsampling blocks are concatenated to
the upsampling block features at the same resolution. The output image is
obtained by a convolution layer followed by a $\tanh$ activation.

For image translation tasks the concept of multi-scale architectures has shown
great success. Skip connections have demonstrated the ability to enhance
training performance and to preserve spatial details [he2016deep,
ronneberger2015unet]. For our generators we employ convolution blocks similar
to Karras et al. [karras2017progressive] and introduce skip connections at
each resolution level, similar to Ronneberger et al. [ronneberger2015unet].
The full overview of our generator models is given in Supplementary Fig. 4. We
use convolutional blocks composed of a convolutional layer with a kernel size
of 3$\times$3, followed by an instance normalization [ulyanov2016instance] and
a ReLU activation. In order to retain the image dimensions and to avoid
boundary artifacts, we use reflection padding before the convolution layers
[wang2018pix2pixhd]. Our network is composed of three downsampling blocks, an
intermediate block, followed by three upsampling blocks. A downsampling block
consists of $n$ convolutional blocks followed by a convolutional block with
stride 2 (downsampling by 2). While downsampling we double the number of
filter channels. For an upsampling block we first apply bilinear upsampling,
reduce the number of filter channels by a factor of 2 with a convolutional
block and apply $n$ additional convolutional blocks. At each resolution level
we use skip connections between the downsampling and upsampling blocks. Hereby
the features prior to downsampling are concatenated with the features after
the first convolutional block in the upsampling block. The input image is
transformed by an initial convolutional block with a kernel size of 7$\times$7
and 64 filter channels. The three downsampling blocks are consecutively
applied with $n=1,2,3$. The intermediate layer consists of three convolution
blocks with the same number of filters as the last downsampling block (512).
The upsampling blocks are applied in the inverse order ($n=3,2,1$). The output
image is obtained by a convolution layer with kernel size 7$\times$7, followed
by a $\tanh$ activation function. For the generator BA we include the noise
term by concatenation with the features of the last downsampling block (prior
to the intermediate block). The noise term is transformed by a convolutional
block with 512 filter channels and matched to the dimensions of the last
downsampling block by an upsampling block with $n=3$ and 512 filter channels
(dashed line in Supplementary Fig. 4). Our networks are fully convolutional,
therefore the training can be performed with image patches, while the
evaluation is performed with full resolution images. For the noise term we use
16 channels and the spatial dimensions are adjusted to the considered image
size ($1/16$ of the input image resolution). The instance normalization
accounts for the global style transfer (see Sect.
LABEL:section:iti_diversity), while the noise term accounts for localized
degradations in the image. For the discriminators we use the architecture
introduced in Wang et al. [wang2018pix2pixhd], where each discriminator is
composed of three individual networks that operate on different scales. The
discriminators are composed of four consecutive stride 2 convolutions with
instance normalization and ReLU activation. We start with 64 filter channels
and consecutively increase them by a factor 2 for each layer. The
discriminators output is obtained by a convolution layer with one filter
channel. Therefore, each discriminator provides a grid of predictions instead
of a single output [isola2017image]. For the noise estimator we use the same
architecture as for the discriminators, adjust the number of filters of the
last layer to the noise dimensions (16) and apply a final sigmoid activation
function. For an image enhancement that involves a resolution increase, we
extend the generator AB by additional upsampling blocks and the generator BA
by additional downsampling blocks. In the case of images with multiple
channels (e.g., multiple filtergrams, magnetograms), we translate all channels
simultaneously. For each image channel we use an additional discriminator that
ensures the correct representation of the generated channel, independent of
the other image channels. The primary discriminator is adjusted to take all
filter channels into account simultaneously to ensure the consistency between
the channels. For an unequal number of channels between domain A and B, we
adjust the identity cycles by truncating additional channels and extending
missing channels with zeros.

###  Ablation study

We perform an ablation study to identify the most important components of our
network architecture. The primary difference to existing domain translation
models are the introduced skip-connections, the running statistics of the
Instance-Normalization, the one-sided noise factor and the channel-wise
discriminators. We found that the InstanceNorm with running statistical values
of the normalization parameters is required to obtain an equal translation
across the full images. The application of a sample-wise normalization would
lead to a translation dependent on the selected patch, while we aim to train
with small patches and then extend our model to full-resolution images for
inference. The noise term is considered necessary to provide a consistent
mapping between the image domains (one-to-many mapping). In Table 4 we vary
our model configuration in terms of number of filters (i.e., model capacity),
depth of the generator (i.e., field-of-view), skip-connections, and
discriminator mode. We refer to ’multi’ when we use an additional
discriminator for each image channel, and to ’single’ if we only employ a
discriminator for the combined set of channels. Each model is trained for
180,000 iterations for the SOHO-To-SDO translation task (Sect.
LABEL:section:stereo_soho_sdo). We evaluate each model in terms of the FID
averaged over all channels. We associate a lower FID with a stronger
similarity to the reference high-quality image distribution, and consequently
a better performance. The results show that the skip-connections provide
throughout a better performance, which we associate with a better spatial
relation to the input image and a more efficient gradient update. Moreover,
the models without skip-connections are more unstable for deeper
architectures, as can be seen from the diverged training of the depth 4
setting. An insufficient capacity leads to poor results (e.g., 32 filters),
while our configurations with 64 filters show the best results. From the
unpaired data set, our training is typically not limited in terms of data,
with the implication that more parameters lead to a better performance. The
comparison of the discriminator modes shows that the additional channel-wise
discriminators as additional training target significantly improve the overall
performance. The improvement for the depth 4 configuration is only marginal,
while the model is three times larger. For this reason, the version with 64
filters, depth 3 and skip-connections was considered for the presented domain
translations in Sect. LABEL:section:results.

Table : Ablation study of our model configuration. We vary our architecture in dependence of number of filters, generator depth, skip-connections and discriminator mode. The number of parameters refer to the generator AB. We estimate the performance in terms of mean FID (lower better). Filters | Depth | Skip-Connections | Discriminator | Parameters [M] | FID
---|---|---|---|---|---
32 | 3 |  | multi | 3.62 | 15.6
32 | 4 |  | multi | 14.65 | 22.8
64 | 3 |  | multi | 14.44 | 13.6
64 | 4 |  | multi | 58.55 | -
32 | 3 |  | multi | 3.81 | 9.9
32 | 4 |  | multi | 15.44 | 8.5
64 | 3 |  | multi | 15.22 | 7.5
64 | 4 |  | multi | 61.68 | 6.8
64 | 3 |  | single | 15.22 | 14.6
64 | 4 |  | single | 61.68 | 11.2

##  5 Supplementary Evaluation

###  Extended comparison to image deconvolution

For the evaluation of enhanced SDO/HMI continuum observations, we match active
region patches and compute the peak signal-to-noise ratio (PSNR), and the
structural similarity index measure (SSIM). In addition we evaluate the PSNR
prior to normalizing the images and the FID of this subset (Supplementary
Table 5). As can be seen from the calibrated HMI images without the
deconvolution, the offset in calibration dominates the quality metric. For
this reason, the images need to be normalized to assess the content
similarity. ITI shows a worse calibration than the baseline, which owes to the
fact that the baseline is fitted solely with data similar to the test set,
while ITI is trained to enhance full-disk observations. In Supplementary Fig.
5 we give additional samples of the aligned SDO/HMI, ITI and Hinode/SOT
continuum observations, as well as the corresponding difference maps.

Table : Evaluation of HMI-to-Hinode enhanced observations. We compare our ITI method, calibrated HMI images, and additionally deconvolved images with aligned Hinode observations. We evaluate the PSNR, SSIM, and FID for the normalized images. The unnormalized PSNR uses image patches without a prior-normalization. For the full data set we compute the distance to the Hinode image distribution (FID). Method | PSNR $\uparrow$ | unnormalized PSNR $\uparrow$ | SSIM $\uparrow$ | FID $\downarrow$
---|---|---|---|---
HMI - calibrated | 18.4 | 24.7 | 0.55 | 74.1
HMI - deconvolved | 23.3 | 24.4 | 0.59 | 78.7
ITI | 24.7 | 20.3 | 0.64 | 44.6

[width=]images/HMI_to_Hinode_overview.jpg

Figure : Six examples of the HMI-to-Hinode translation. We show the original
SDO/HMI observation, the deconvolved image, the ITI enhanced image, the
aligned Hinode/SOT observation and the corresponding difference maps.

###  Variation of atmospheric degradations

By modeling a large variety of realistic degradation effects we expect that
the reconstruction model can account for multiple atmospheric degradation
effects simultaneously. From the included noise term in our generator BA,
multiple low-quality observations can be synthesized from a single high-
quality observation. We illustrate the diversity of the generated samples by
randomly selecting a high-quality image and generating six corresponding low-
quality images, where we use a different noise term for each example. Four
examples are shown in Supplementary Fig. 5, where panel a contains the high-
quality input images and panel b the synthetic low-quality images. The samples
show a large spread of different cloud distributions, both in density and
position. From a comparison with Fig. LABEL:figure:kso_combined it can be seen
that the synthetic clouds appear similar to real low-quality observations. The
solar features are consistent with the high-quality reference (e.g., active
regions, filaments).

[width=]images/KSO_variation.jpg

Figure : Diversity examples of the KSO low-to-high quality translation. a)
real high-quality observations. b) artificially degraded observations. The
images in one row are generated from the same high-quality input image. The
artificial low-quality images show a variety of realistic atmospheric effects,
while the solar features are mostly unchanged.

###  Detailed comparison of enhanced photographic film observations

A comparison between cutouts of the film scans and the ITI enhanced
observations is shown in Supplementary Fig. 5. All samples show a clear
improvement in image sharpness, that especially affects the plage regions and
leads to more distinct representation of solar filaments. As can be seen from
Supplementary Fig. 5b, the ITI translation also mitigates clouds, similar to
the application in Sect. LABEL:section:kso.

[width=0.8]images/Film_comparison.jpg

Figure : Example of KSO film-to-CCD translation. The ITI translation results
in deblurring of plage regions, sharpening of filaments and enhances the
structure of quiet Sun regions. Atmospheric effects are strongly mitigated in
the enhanced version (middle).

In Supplementary Fig. 5 we provide a comparison of a filtergram at three
different scales. In Supplementary Fig. 5a the calibration of the image
intensity and correction of large scale inhomogeneities is visible. The
cutouts show a comparison between the film observation and the restored image,
where we note a perceptual quality improvement.

[width=0.5]images/Film_sample.jpg

Figure : Detailed comparison of a KSO film-to-CCD translation sample from
1992-03-12 08:51. The ITI observation shows a more homogeneous appearance at
the global scale (top). The comparison of two regions shows that active
regions are well reconstructed (red). At smaller scales (blue), solar features
can be better identified in the ITI observations, but the limits in resolution
are visible.

###  Temporal stability of generated ITI magnetograms

We further compare the temporal stability of our method by estimating the
full-disk magnetograms from 2007-01-14 to 2007-01-18 and compare the sequence
to the SOHO/MDI magnetograms (Movie 3). The ITI magnetograms are overall
consistent and show no large artifacts (e.g., random active regions). The
active region in the ITI magnetograms appears similar to the real observation
and is consistent in its position on the solar disk. The largest
inconsistencies originate from variations in the estimated sunspots and we
note a quality decreases close to the solar limb.

###  Comparison to paired image-to-image translation

Recent studies try to estimate the solar far-side magnetic field with the use
of paired image translations, where first a model is trained to translate from
SDO/AIA EUV filtergrams to SDO/HMI magnetograms, and apply in a second step
the trained model to EUV data from STEREO [kim2019solar,
jeong2020ai_mag_pfss]. Here, we compare our method with the paired image
translation methods. For this we consider the first step (SDO EUV
$\xrightarrow{}$ magnetogram), which we can fully quantify. For the paired
image-to-image translation, we use the pix2pixHD model [wang2018pix2pixhd], in
line with [jeong2020ai_mag_pfss, shin2020caII_to_mag, son2021euv_to_he]. We
note that our method contains the pix2pix training cycle, but it is only
employed with secondary importance. We train both methods with the same data
set, using only observations from SDO, where we reduce the resolution to
512$\times$512 pixels. For pix2pix we use the standard training described in
[wang2018pix2pixhd], without the VGG loss and start with 32 filters in the
first convolutional layer (larger models led to divergence). The model is
trained for 200 epochs. For ITI we train our model to translate from the four
EUV channels (A) to the same set including the magnetogram (B). We train our
model for 160.000 iterations, with the same parameter settings as in Sect.
LABEL:section:stereo_mag (STEREO-To-SDO). For comparison we also train ITI to
estimate the unsigned magnetic flux (no polarities). In Supplementary Table 5
we summarize the results in terms of mean-absolute-error (MAE), MAE of the
unsigned magnetic flux, and FID. The results show that the paired translation
performs best for estimating the polarity of the magnetic field (MAE), while
our ITI method achieves a similar performance with unpaired training. In terms
of perceptual quality (FID), ITI provides the best results. Enforcing the
estimation of magnetic polarities reduces the model performance for estimating
the total flux distribution, as can be seen for the unsigned MAE, where the
best results are obtained by the ITI model that is explicitly trained for this
task. In Sect. LABEL:section:stereo_soho_sdo we used the unsigned magnetic
flux for our training. The reason for this is two fold. 1) There exists no
evidence that the polarity can be directly derived from the EUV observations.
2) The polarity could be inferred from the global magnetic field configuration
(i.e., Hale’s Law). However, this can only be performed by using a data set of
only one solar cycle. Moreover, this estimation breaks down for complex
regions of strong confined flux. This is supported by Supplementary Fig. 5,
where we show a comparison between Pix2Pix and the ITI results. All methods
lack in estimating the correct position and strength of confined flux regions
(i.e., sunspots and pores). For this comparison, none of the methods produced
larger artifacts (e.g., strong magnetic flux outside of active regions). The
pix2pix approach assumes that the EUV observations are similar between SDO/AIA
and STEREO/EUVI, however we found that the EUV filtergrams are vastly
different (Sect. LABEL:section:stereo_soho_sdo). Since the pix2pix methods are
trained for a specific image domain, it can not be assumed that comparable
results are achieved when applied to data that is not within the training
domain. Our method is based on domain translation, thus it explicitly learns
to translate from the STEREO/EUVI to the SDO/AIA+HMI domain, from which we
expect a comparable performance to this reference task. In conclusion, our
unpaired image-to-image translation approach can provide comparable results to
paired translations, while being also applicable for data sets that have no
spatial or temporal alignment.

Table : Evaluation of the SDO EUV to magnetogram training task for the Pix2Pix method and ITI. For all metrics lower values correspond to a better performance. Method | MAE [G] $\downarrow$ | MAE - unsigned [G] $\downarrow$ | FID $\downarrow$
---|---|---|---
Pix2Pix | 8.5 | 6.6 | 4.1
ITI | 9.6 | 7.3 | 1.6
ITI-abs | - | 6.1 | -

[width=]images/pix2pix_vs_iti.jpg

Figure : We compare the original SDO observation with the Pix2Pix paired
translations, the ITI unpaired translations, and the ITI method trained with
absolute magnetic flux (2014-12-17 12:00). Column a shows the reference
SDO/HMI magnetogram, column b the synthetic mangetograms, and column c the
difference maps for the magnetograms of unsigned flux. All methods show larger
deviations for regions of confined magnetic flux (red circle). The polarity
can be confused even on global scales (blue for ITI; yellow for Pix2Pix) and
also on small scales (red for both). The difference maps show that the largest
error occurs due to a misalignment of estimated sunspots, as well as
differences in magnetic field strength.