Brodsky et al’s defence does not work

P. M. Stevenson

T.W. Bonner Laboratory, Department of Physics and Astronomy,
Rice University, Houston, TX 77251, USA

Abstract:

> In their defence of “maximum conformality” methods, Brodsky et al make the
> astonishing claim that any RG transformation $a^{\prime}=a(1+V_{1}a+\ldots)$
> in QCD must have $V_{1}$ proportional to $b=(33-2n_{f})/6$. It is well known
> that this is not true. I emphasize again the correctness and central
> importance of the Celmaster-Gonsalves relation for the prescription
> dependence of the $\Lambda$ parameter.

I recently made a direct attack [1] upon the BLM [2] or PMC (“principle of
maximum conformality”) methods (see [3] and references therein). The key point
arises already at next-to-leading order (NLO) for a physical quantity

${\cal R}=a(1+r_{1}a+\ldots).$ (1)

The result given by BLM/PMC for the $r_{1}$ coefficient is just different when
one starts, not from $\overline{\rm MS}$, but from another scheme related to
it by

$a^{\prime}=a(1+V_{1}a+\ldots).$ (2)

Thus BLM/PMC does not resolve the scheme ambiguity [1, 4]. None of the
variants or generalizations of BLM cures this fundamental defect.

Brodsky et al’s defence [5] goes off in many directions. I disagree with
almost every statement they make, but it would be of little wider interest to
rebut each one. The key point of their defence, though, is astonishing: They
claim in their seventh paragraph that $V_{1}$ must be proportional to
$b=(33-2n_{f})/6$ and that anything else would somehow spoil the “colour
structure.”

It is easily seen that this is untrue. Ever since the classic work of
Celmaster and Gonsalves (CG) [6] it has been clear that scheme dependence
involves both scale and prescription dependence, so that a general scheme
transformation has the form

$V_{1}=-b\ln(\mu^{\prime}/\mu)+v_{1}.$ (3)

While the scale-change part is proportional to $b$, the prescription-change
part, $v_{1}$, can be any finite number and in QCD could depend arbitrarily
upon $n_{f}$ (though for most prescriptions in the literature $v_{1}$ is
linear in $n_{f}$). CG introduced a class of “momentum subtraction” (MOM)
prescriptions and calculated how they relate to minimal subtraction (MS) in
NLO. Their results, in Eqs. (17, 18), are of the form of Eq. (2) with
$v_{1}$’s that are clearly not proportional to $33-2n_{f}$. (CG also
explicitly checked that their calculations satisfied the Ward identities.) Do
Brodsky et al really believe that all MOM prescriptions are illegitimate?

Further, if the “$V_{1}\propto b$” contention were true, it would need to be
explained why MS and $\overline{\rm MS}$ are correct, while a similar
prescription [6] that defined the trace of the unit Dirac matrix in $d$
dimensions to be $d$ or $2^{d/2}$ (rather than $4$) would somehow be flat
wrong.

The point can also be made as follows. One way to define a renormalization
prescription (as noted by many authors) is through the “effective charge”
associated with a physical quantity. For example, from the $e^{+}e^{-}$ cross-
section ratio one could define a prescription whose couplant is related to the
$\overline{\rm MS}$ one by a $V_{1}$ equal to the $r_{1}$ coefficient for
${\cal R}^{e^{+}e^{-}}$ calculated in $\overline{\rm MS}$:

$r_{1}^{e^{+}e^{-}\,(\overline{\rm
MS})}=b\ln(\mu/Q)+\left(\frac{365}{24}-11\zeta_{3}\right)-n_{f}\left(\frac{11-8\zeta_{3}}{12}\right),$
(4)

which again, is not proportional to $b$. Does the $e^{+}e^{-}$ effective-
charge prescription “violate the colour structure”? (Or does $\overline{\rm
MS}$?) Of course not.

The scale/prescription issue is one that still seems to confuse many
researchers. The key to understanding it is the CG relation [6]:

$\ln(\tilde{\Lambda}^{\prime}/\tilde{\Lambda})=v_{1}/b,$ (5)

which the relates the $\tilde{\Lambda}$ parameters of two prescriptions.

One must first define $\tilde{\Lambda}$ within a given renormalization
prescription. The $\beta$-function equation

$\mu\frac{da}{d\mu}\equiv\beta(a)=-ba^{2}(1+ca+c_{2}a^{2}+\ldots)$ (6)

is integrable and (since we naturally want the range of integration to involve
only $0$ to $a$) the constant of integration can be conveniently specified [8]
by111 For a full discussion see Sect. 6.3 of [9]. CG did not quite arrive at
this way of defining a $\Lambda$ parameter, but their discussion in subsection
II.F of [6] is completely consistent with it. This definition of
$\tilde{\Lambda}$ is not unique, but any satisfactory definition will be
equivalent to it up to the addition of a finite constant to ${\cal
C}(\delta)$; for example, the conventional definition of $\Lambda$ corresponds
to subtracting $(c/b)\ln\mid\\!2c/b\\!\mid$.

$\ln(\mu/{\tilde{\Lambda}})=\lim_{\delta\to
0}\left(\int_{\delta}^{a}\frac{dx}{\beta(x)}+{\cal C(\delta)}\right),$ (7)

with

${\cal C}(\delta)=\int_{\delta}^{\infty}\frac{dx}{bx^{2}(1+cx)}.$ (8)

The $\tilde{\Lambda}$ parameter depends upon the renormalization prescription,
but in the simple and definite way given by Eq. (5) above. That equation is
exact and does not involve any $v_{2},v_{3},\ldots$ coefficients. CG’s proof,
in my notation, is the following. In the primed prescription the counterpart
to Eq. (7) will need primes on $\tilde{\Lambda}$, on $a$, and (beyond NLO) on
$\beta(x)$. (The $\mu$ remains the same since, by definition, a change of
prescription is a change of scheme that keeps the value of $\mu$ unchanged.)
One therefore has

$\ln(\mu/{\tilde{\Lambda}}^{\prime})=\lim_{\delta\to
0}\left(\int_{\delta}^{a^{\prime}}\frac{dx}{\beta^{\prime}(x)}+{\cal
C}(\delta)\right).$ (9)

When taking the difference of Eqs. (7) and (9) the ${\cal C}(\delta)$ terms
cancel leaving

$\ln(\mu/{\tilde{\Lambda}})-\ln(\mu/{\tilde{\Lambda}}^{\prime})=\lim_{\delta\to
0}\left(\int_{\delta}^{a}\frac{dx}{\beta(x)}-\int_{\delta}^{a^{\prime}}\frac{dx}{\beta^{\prime}(x)}\right),$
(10)

On the left-hand side the $\mu$ dependence cancels to leave
$\ln(\tilde{\Lambda}^{\prime}/\tilde{\Lambda})$. The right-hand side must
therefore be independent of $\mu$, so we may choose to take $\mu\to\infty$,
making both $a$ and $a^{\prime}$ tend to zero, by asymptotic freedom. One may
then evaluate the difference of integrals for arbitrarily small $a$ values,
which leads to the result in Eq. (5) above.

CG’s proof is sound, but it does leave a bit of a mystery as to how the right-
hand side of Eq. (10) reduces to $v_{1}/b$ at any $\mu$. An alternative proof,
due to Hugh Osborn, can be found in Sect. 6.5 of [9]. It splits the two
integrations at $\epsilon$ and
$\epsilon^{\prime}=\epsilon(1+v_{1}\epsilon+\ldots)$, respectively. Two terms
cancel exactly, leaving only terms with $x<\epsilon$ or $x<\epsilon^{\prime}$.
Thus it is clear that the non-zero result, $v_{1}/b$, arises solely from the
$\frac{1}{x^{2}}$ singularity and the behaviour of $\beta(x)$ away from the
infinitesimal neighbourhood of $x=0$ plays no role.222 Ref. [7], cited by [5]
in support of their claim, has tacitly assumed that $\Lambda$ is invariant, so
the right-hand side of their equation (18) is missing a
$\ln(\Lambda^{*}/\Lambda)$ term.  333 In ref. [1] I said that “${\cal
C}(\delta)$ involves only $b$ and $c$ and so is RS invariant.” The second part
of that statement is problematic because in different prescriptions one may
choose whether or not to use the same $\delta$. My proof of the CG relation in
Appendix A of [1] used a $\delta^{\prime}=\delta(1+v_{1}\delta+\ldots)$ and
the $v_{1}/b$ result then arose from ${\cal C}(\delta)-{\cal
C}(\delta^{\prime})$. Brodsky it et al seize upon this semantic conundrum, but
it has no bearing on the correctness of the CG result.

The CG relation is the key to the observation [8] that

${\boldsymbol{\rho}}_{1}(Q)\equiv b\ln(\mu/\tilde{\Lambda})-r_{1}$ (11)

is an invariant – independent of both $\mu$ and the prescription. That means
that the NLO coefficient $r_{1}$ of a physical quantity ${\cal R}$ depends on
scheme only through the ratio $\mu/\tilde{\Lambda}$ (which makes sense, since
the $a$ obtained from Eq. (7) at NLO depends only on $\mu/\tilde{\Lambda}$.)

There is no meaningful answer to the questions “What is the right $\mu$?” and
“What is the right prescription?” The only RS variable that matters at NLO is
the ratio of $\mu$ to the $\tilde{\Lambda}$ parameter, so the real question,
for any given quantity ${\cal R}$, is “What is a good choice of
$\mu/\tilde{\Lambda}$?” Methods that “work” supply an answer to that question.

I stress again that here and in [1] I am not concerned with defending my own
“Principle of Minimal Sensitivity” method [8]: That is a separate argument. I
have explained that method in detail in Ref. [9] and anyone who applies it
carefully will find, I am sure, that the method defends itself.

## References

  * [1] P. M. Stevenson, Phys, Lett. B 847 (2023) 138288.
  * [2] S. J. Brodsky, G. P. Lepage, and P. B. Mackenzie, Phys. Rev. D 28 (1983) 228.
  * [3] L. Di Giustino, S. J. Brodsky, P. G. Ratcliffe, X-G Wu, and S-Q Wang, High precision tests of QCD without scale or scheme ambiguities, preprint submitted to Progress in Particle and Nuclear Physics, July 7, 2023. arXiv:2307.03951.
  * [4] W. Celmaster and P. M. Stevenson, Phys. Lett. B 125 (1983) 493 ; J. Chýla, Phys. Rev. D 31 (1985) 2690.
  * [5] S. J. Brodsky, L. Di Giustino, P. G. Ratcliffe, S-Q Wang, and X-G Wu, arXiv:2311.17360.
  * [6] W. Celmaster and R. Gonsalves, Phys. Rev. D 20 (1979) 1420.
  * [7] F. A. Chishtie and D. G. C. McKeon, Phys. Rev. D 95 (2017) 116013.
  * [8] P. M. Stevenson, Phys. Lett. B 100 (1981) 61; Phys. Rev. D 23 (1981) 2916.
  * [9] P. M. Stevenson, Renormalized Perturbation Theory and its Optimization by the Principle of Minimal Sensitivity, World Scientific, 2022. The text is available “open access” from the publisher’s website:
https://www.worldscientific.com/worldscibooks/10.1142/12817#t=aboutBook

## Addendum: Numerical Example

This addendum demonstrates explicitly that the BLM procedure gives different
results in different schemes. The specific example here is for ${\cal
R}_{e^{+}e^{-}}$ with $n_{f}=2$ at $Q=1$ GeV with
$\tilde{\Lambda}_{\overline{\rm MS}}=0.2$ GeV, corresponding to Example 3
($Q/\tilde{\Lambda}_{\overline{\rm MS}}=5$) in Chap. 10 of Ref. [9].

The NLO $\beta$ function is

$\beta^{{\scriptscriptstyle{\rm NLO}}}(a)=-ba^{2}(1+ca),$ (A.1)

with

$b=29/6,\quad\quad c=115/58.$ (A.2)

The $r_{1}$ coefficient ($r_{1}=An_{f}+B$), calculated in the $\overline{\rm
MS}(\mu=Q)$ scheme, has

$\displaystyle A$ $\displaystyle=$
$\displaystyle-\frac{11}{12}+\frac{2}{3}\zeta_{3}=-0.115295,$ $\displaystyle
B$ $\displaystyle=$ $\displaystyle\frac{365}{24}-11\zeta_{3}=1.98571,$ (A.3)

so that for $n_{f}=2$ one has

$r_{1}^{\overline{\rm MS}(\mu=Q)}=2A+B=1.75512.$ (A.4)

The NLO result in the $\overline{\rm MS}(\mu=Q)$ scheme uses an “$a$” that is
the solution to the integrated $\beta$-function equation at NLO for
$\ln(\mu/\tilde{\Lambda})$ equal to $\ln 5$; i.e., it is the root of

$\frac{1}{a}\left(1+ca\ln\left|\frac{ca}{1+ca}\right|\right)-b\ln 5=0,$ (A.5)

which is

$a_{\overline{\rm MS}}(Q)=0.0862557.$ (A.6)

The NLO result in the $\overline{\rm MS}(\mu=Q)$ scheme is then

${\cal R}_{\overline{\rm MS}(\mu=Q)}=a_{\overline{\rm
MS}}(Q)\left(1+r_{1}^{\overline{\rm MS}(\mu=Q)}a_{\overline{\rm
MS}}(Q)\right)=0.0993138.$ (A.7)

To obtain the BLM/PMC result I follow Eqs. (3.4), (3.5) in Ref. [3]444 Though
Eq. (3.4) is not “the NLO $\alpha_{s}$-running formula” but the leading-order
one; it corresponds to the LO $\beta$ function $-ba^{2}$. The NLO running
involves $c$, of course, and requires solving equations such as Eq. (A.5)
above. This error makes matters worse, in that the BLM result is not even the
same as using the $\overline{\rm MS}(\mu=\mu_{r}^{*})$ scheme, but correcting
it would not affect my main point.  These give

$C_{1}^{*}=\frac{33}{2}A+B=0.0833333,$ (A.8)

and

$\mu_{r}^{*}=Q\exp{3A}=0.707593\,{\rm GeV}.$ (A.9)

Then

$a_{BLM}=\frac{a_{\overline{\rm MS}}(Q)}{1+b\,a_{\overline{\rm
MS}}(Q)\ln(\mu_{r}^{*}/Q)}=0.10079,$ (A.10)

so that the NLO BLM result is

${\cal R}_{BLM}=a_{BLM}(1+C_{1}^{*}a_{BLM})=0.101636.$ (A.11)

But suppose we had originally calculated $r_{1}$ in some other renormalization
prescription. That is equivalent to taking the $\overline{\rm MS}$ result and
making a change of renormalization prescription $a^{\prime}=a(1+v_{1}a+...)$.
(Let us choose to keep the scale choice as $\mu=Q$ in both prescriptions.) By
the exact CG relation we have

${\tilde{\Lambda}}^{\prime}=\tilde{\Lambda}_{\overline{\rm MS}}e^{v_{1}/b},$
(A.12)

so that $a^{\prime}$, the NLO couplant of the new scheme, is given by the root
of

$\frac{1}{a^{\prime}}\left(1+ca^{\prime}\ln\left|\frac{ca^{\prime}}{1+ca^{\prime}}\right|\right)-b\ln(Q/{\tilde{\Lambda}}^{\prime})=0,$
(A.13)

and the NLO coefficient in the new scheme is

${r_{1}}^{\prime}=r_{1}-v_{1}.$ (A.14)

Let $v_{1}$ be of the form

$v_{1}=v_{10}\,n_{f}+v_{11}.$ (A.15)

Example (i): $v_{10}=0$, $v_{11}=2$.

First, find the NLO result in this new scheme:

$A^{\prime}=A=-0.115295,\quad B^{\prime}=B-2=-0.0142926,$ (A.16)
${r_{1}}^{\prime}=r_{1}-2=-0.244883,$ (A.17)
${\tilde{\Lambda}}^{\prime}=(0.2\,{\rm GeV})e^{v_{1}/b}=0.302509\,{\rm GeV}.$
(A.18)

Solving for $a^{\prime}=a_{\rm new}(Q)$ from Eq. (A.13) above gives

$a^{\prime}=0.108568,$ (A.19)

so that the NLO result in the new scheme is

${\cal R}^{\prime}=a^{\prime}(1+{r_{1}}^{\prime}a^{\prime})=0.105682.$ (A.20)

Now apply the BLM procedure but starting from the new prescription:

${C_{1}^{*}}^{\prime}=\frac{33}{2}A^{\prime}+B^{\prime}=-1.91667,$ (A.21)
${\mu_{r}^{*}}^{\prime}=Q\exp{3A^{\prime}}=\mu_{r}^{*}=0.707593\,{\rm GeV}.$
(A.22)

Then

${a_{BLM}}^{\prime}=\frac{a^{\prime}}{1+b\,a^{\prime}\ln({\mu_{r}^{*}}^{\prime}/Q)}=0.132643,$
(A.23)

so that the NLO BLM result is

${{\cal
R}_{BLM}}^{\prime}={a_{BLM}}^{\prime}(1+{{C_{1}}^{*}}^{\prime}{a_{BLM}}^{\prime})=0.0989209.$
(A.24)

This is different from the BLM result of Eq. (A.11) above.

Example (ii): $v_{10}=1$, $v_{11}=0$.

Since, with $n_{f}=2$, the net $v_{1}$ is the same as in Example (i), the
results of Eqs. (A.17) to (A.20) are unchanged. Applying the BLM procedure,
however, gives

$A^{\prime}=A-1=-1.115295,\quad B^{\prime}=B=1.98571,$ (A.25)

so that

${C_{1}^{*}}^{\prime}=\frac{33}{2}A^{\prime}+B^{\prime}=-16.4167,$ (A.26)
${\mu_{r}^{*}}^{\prime}=Q\exp{3A^{\prime}}=\mu_{r}^{*}\,e^{-3}=0.035229\,{\rm
GeV}.$ (A.27)

Then

${a_{BLM}}^{\prime}=\frac{a^{\prime}}{1+b\,a^{\prime}\ln({\mu_{r}^{*}}^{\prime}/Q)}=-0.143658,$
(A.28)

so that the NLO BLM result is now

${{\cal
R}_{BLM}}^{\prime}={a_{BLM}}^{\prime}(1+{{C_{1}}^{*}}^{\prime}{a_{BLM}}^{\prime})=-0.482459.$
(A.29)

This result is different again – and is clearly nonsensical, since we are the
wrong side of a Landau pole and the couplant is negative.


# Shortcut-V2V: Compression Framework for Video-to-Video Translation
based on Temporal Redundancy Reduction

Chaeyeon Chung*  KAIST AI  Yeojeong Park  KAIST AI  KT Research & Development
Center, KT Corporation  Seunghwan Choi  KAIST AI  Munkhsoyol Ganbat  KAIST AI
Jaegul Choo  KAIST AI

###### Abstract

Video-to-video translation aims to generate video frames of a target domain
from an input video. Despite its usefulness, the existing networks require
enormous computations, necessitating their model compression for wide use.
While there exist compression methods that improve computational efficiency in
various image/video tasks, a generally-applicable compression method for
video-to-video translation has not been studied much. In response, we present
Shortcut-V2V, a general-purpose compression framework for video-to-video
translation. Shortcut-V2V avoids full inference for every neighboring video
frame by approximating the intermediate features of a current frame from those
of the previous frame. Moreover, in our framework, a newly-proposed block
called AdaBD adaptively blends and deforms features of neighboring frames,
which makes more accurate predictions of the intermediate features possible.
We conduct quantitative and qualitative evaluations using well-known video-to-
video translation models on various tasks to demonstrate the general
applicability of our framework. The results show that Shortcut-V2V achieves
comparable performance compared to the original video-to-video translation
model while saving 3.2-5.7$\times$ computational cost and 7.8-44$\times$
memory at test time. Our code and videos are available at
https://shortcut-v2v.github.io/.

**footnotetext: indicates equal contributions. Figure 1: Overview of
Shortcut-V2V. (a) is an overall framework of Shorcut-V2V, and (b) shows a
detailed architecture of Shortcut block. $\uparrow 2$ and $\downarrow 2$ refer
to upsampling and downsampling by a factor of 2, respectively. G in Offset G
and Offset/Mask G indicates a generator.

##  1 Introduction

Video-to-video translation is a task of generating temporally consistent and
realistic video frames of a target domain from a given input video. Recent
studies on video-to-video translation present promising performance in various
domains such as inter-modality translation between labels and videos [37, 36,
20], and intra-modality translation between driving scene videos [35] or face
videos [1].

Despite enhanced usefulness, video-to-video translation networks usually
require substantial computational cost and memory usage, which limits their
applicability. For instance, multiply–accumulates (MACs) of a widely-used
video translation model, vid2vid [37], is 2066.69G, while the basic
convolutional neural networks, ResNet v2 50 [10] and Inception v3 [31], are
4.12G and 6G, respectively. Furthermore, temporally redundant computations for
adjacent video frames also harm the cost efficiency of a video-to-video
translation network. Performing full inference for every neighboring video
frame that contains common visual features inevitably entails redundant
operations [22, 21].

In this regard, Fast-Vid2Vid [44] proposes a compression framework for vid2vid
[37] based on spatial input compression and temporal redundancy reduction.
However, it cannot be applied to other video-to-video translation models since
it is designed specifically for vid2vid. Moreover, Fast-Vid2Vid does not
support real-time inference since it requires future frames to infer a current
one. Alternatively, one can apply model compression approaches for image-to-
image translation [15, 24, 13] directly to video-to-video translation,
considering video frames as separate images. However, these approaches are not
designed to consider the correlation among adjacent video frames during the
compression. This may result in unrealistic output quality in video-to-video
translation, where the inherent temporal coherence of an input video needs to
be preserved in the outputs. Also, frame-by-frame inference without temporal
redundancy reduction involves unnecessary computations, resulting in
computational inefficiency.

In this paper, we propose Shortcut-V2V, a general-purpose framework for
improving the computational efficiency of video-to-video translation based on
temporal redundancy reduction. Shortcut-V2V allows the original video-to-video
translation model to avoid temporally redundant computations by approximating
the decoding layer features of the current frame with largely reduced
computations. To enable lightweight estimation, our framework leverages
features from the previous frame (_i.e_., reference features), which have high
visual similarity with the current frame. We also exploit current frame
features from the encoding layer to handle newly-appeared regions in the
current frame. Specifically, we first globally align the previous frame
features with the current frame features, and our novel Adaptive Blending and
Deformation block (AdaBD) in Shortcut-V2V blends features of neighboring
frames while performing detailed deformation. AdaBD adaptively integrates the
features regarding their redundancy in a lightweight manner.

In this way, our model significantly improves the test-time efficiency of the
original network while preserving its original performance. Shortcut-V2V is
easily applicable to a pretrained video-to-video translation model to save
computational cost and memory usage. Our framework is also suitable for real-
time inference since we do not require future frames for the current frame
inference. To the best of our knowledge, this is the first attempt at a
general-purpose model compression approach for video-to-video translation. We
demonstrate the effectiveness of our approach using well-known video-to-video
translation models, Unsupervised RecycleGAN [35] (Unsup) and vid2vid [37].
Shortcut-V2V reduces 3.2-5.7$\times$ computational cost and 7.8-44$\times$
memory usage while achieving comparable performance to the original model.
Since there is no existing general-purpose compression method, we compare our
method with Fast-Vid2Vid [44] and the compression methods for image-to-image
translation. Our model presents superiority over the existing approaches in
both quantitative and qualitative evaluations. Our contributions are
summarized as follows:

  * •

We introduce a novel, general-purpose model compression framework for video-
to-video translation, Shortcut-V2V, that enables the original network to avoid
temporally redundant computations.

  * •

We present AdaBD that exploits features from neighboring frames via adaptive
blending and deformation in a lightweight manner.

  * •

Our framework saves up to 5.7$\times$ MACs and 44$\times$ parameters across
various video-to-video translation tasks, achieving comparable performance to
the original networks.

##  2 Related Work

###  2.1 Video-to-Video Translation

Recent video-to-video translation networks are generally classified into
pix2pixHD-based [38] and CycleGAN-based [42] generator. vid2vid [37] proposes
a pix2pixHD-based sequential generation framework that synthesizes a current
output given the previous outputs as additional guidance. As the following
work, few-shot vid2vid [36] achieves few-shot generalization of vid2vid based
on attention modules, and world-consistent vid2vid [20] is proposed to improve
long-term temporal consistency of vid2vid.

While pix2pixHD-based models require paired annotated videos, RecycleGAN [1]
and MocycleGAN [4] propose CycleGAN-based video translation models. They
exploit spatio-temporal consistency losses to generate realistic videos using
unpaired datasets. STC-V2V [23] leverages optical flow for semantic/temporal
consistency to improve the output quality of the existing models. Unsupervised
RecycleGAN [35] achieves state-of-the-art performance among CycleGAN-based
frameworks with a pseudo-supervision by the synthetic flow. Although the
existing video-to-video translation networks achieve decent performance, they
commonly demand a non-trivial amount of computational costs and memory usage.
Also, frame-by-frame inference necessarily causes redundant operations due to
temporal redundancy among adjacent frames.

###  2.2 Model Compression

Model compression for video-related tasks has been actively proposed in
various domains [21, 22, 18, 9, 30, 8, 16, 32], such as object detection,
action recognition, semantic segmentation, and super-resolution. Several
studies [21, 9, 22] exploit temporal redundancy among video frames to improve
efficiency during training or inference. For instance, Habibian _et al_. [9]
distill only the residual between adjacent frames from a teacher model to a
student to speed up the inference. Also, Fast-Vid2Vid [44] firstly proposes a
compression framework for video-to-video translation based on spatial and
temporal compression. However, Fast-Vid2Vid focuses on vid2vid [37], limiting
its application to other video-to-video translation networks. Also, temporal
redundancy reduction via motion compensation in Fast-Vid2Vid requires the
future frame to infer the current frame, which is not suitable for real-time
inference.

Alternatively, model-agnostic compression methods for image-to-image
translation can also be applied to video-to-video translation models. The
existing approaches for image synthesis mainly tackle channel pruning [15],
knowledge distillation [13, 24], NAS [15], etc. For instance, CAT [13]
compresses the teacher network with one-step pruning to satisfy the target
computation budget, while OMGD [24] conducts a single-stage online
distillation in which the teacher generator supports the student generator to
be refined progressively. However, image-based compression methods cannot
consider temporal coherence among neighboring video frames, which may induce
unrealistic results in video translation tasks. Also, performing full model
inference for each video frame still poses computational inefficiency due to
the temporal redundancy.

###  2.3 Deformable Convolution

Deformable convolution [6] is originally proposed to enhance the
transformation capability of a convolutional layer by adding the estimated
offsets to a regular convolutional kernel in vision tasks such as object
detection or semantic segmentation. Besides its original application, recent
studies [33, 7, 39, 2, 12, 17] demonstrate that deformable convolution is also
capable of aligning adjacent video frames. TDAN [33] utilizes deformable
convolution to capture implicit motion cues between consecutive frames by
dynamically predicted offsets in video super-resolution. In addition, EDVR
[39] stacks several deformable convolution blocks to estimate large and
complex motions in video restoration. In this paper, we also leverage a
deformable convolution to adaptively align adjacent video frames in a
lightweight manner.

Algorithm 1 Shortcut-V2V Inference

1:Input: Input video $\\{\mathbf{I}_{t}\\}_{t=0}^{N_{T}-1}$ of length $N_{T}$,
teacher model $T$, layer index of encoder $l_{e}$ and decoder $l_{d}$,
Shortcut block $S$, max interval $\alpha$

2:Output: Output video $\\{\mathbf{O}_{t}\\}_{t=0}^{N_{T}-1}$

3:for $t=0$ to $N_{T}-1$ do

4: $\mathbf{a}_{t}=T_{[:l_{e}]}(\mathbf{I}_{t})$

5: if $t\%\alpha=0$ then

6: $\mathbf{f}_{t}=T_{[l_{e}+1:l_{d}-1]}(\mathbf{a}_{t})$

7: $\mathbf{a}_{\text{ref}}$, $\mathbf{f}_{\text{ref}}$ $\leftarrow$
$\mathbf{a}_{t}$, $\mathbf{f}_{t}$ $\triangleright$ Update the reference
features

8: else

9: $\mathbf{f}_{t}=S(\mathbf{f}_{\text{ref}},\mathbf{a}_{\text{ref}}$,
$\mathbf{a}_{t})$

10: end if

11: $\mathbf{O}_{t}=T_{[l_{d}:]}(\mathbf{f}_{t})$

12:end for

##  3 Shortcut-V2V

In this paper, we propose Shortcut-V2V, a general compression framework to
improve the test-time efficiency in video-to-video translation. As illustrated
in Fig. 1(a), given $\\{\mathbf{I}_{t}\\}_{t=0}^{N_{T}-1}$ as input video
frames, we first use full teacher model $T$ to synthesize the output of the
first frame. Then, for the next frames, our newly-proposed Shortcut block
efficiently approximates $\mathbf{f}_{t}$, the features from the $l_{d}$-th
decoding layer of the teacher model. This is achieved by leveraging the
$l_{e}$-th encoding layer features $\mathbf{a}_{t}$ along with reference
features, $\mathbf{a}_{ref}$ and $\mathbf{f}_{ref}$, from the previous frame.
Here, $l_{d}$ and $l_{e}$ correspond to layer indices of the teacher model.
Lastly, predicted features $\mathbf{\hat{f}}_{t}$ are injected into the
following layers of the teacher model to synthesize the final output
$\mathbf{\hat{O}}_{t}$. To avoid error accumulation, we conduct full teacher
inference and update the reference features at every max interval $\alpha$. We
provide the detailed inference process of Shortcut-V2V in Algorithm 1.

The architecture of our model is mainly inspired by Deformable Convolutional
Network (DCN) [6, 43], which we explain in the next section.

###  3.1 Deformable Convolutional Network

DCN [6, 43] is initially introduced to improve the transformation capability
of a convolutional layer in image-based vision tasks, _e.g_., object detection
and semantic segmentation. In the standard convolutional layer, a $3\times 3$
kernel with dilation 1 samples points over input features using a sampling
position $\mathbf{p}_{k}\in\\{(-1,-1),(-1,0),...,(0,1),(1,1)\\}$. Given input
feature maps $\mathbf{x}$, DCN predicts additional offsets
$\Delta\mathbf{p}\in\mathbb{R}^{2N_{\mathbf{p}}\times H\times W}$ to augment
each sampling position along x-axis and y-axis. Here, $N_{\mathbf{p}}$ is the
number of sampling positions in a kernel, and $H$ and $W$ are the height and
width of output feature maps, respectively. For further manipulation of input
feature amplitudes over the sampled points, DCNv2 [43] introduces a modulated
deformable convolution with $\mathbf{m}\in\mathbb{R}^{N_{\mathbf{p}}\times
H\times W}$ consisting of learnable modulation scalars. The deformed output
feature maps $\mathbf{x}^{\prime}$ by DCNv2 are defined as:

$\mathbf{x}^{\prime}=f_{dc}(\mathbf{w},\mathbf{x},\Delta\mathbf{p},\mathbf{m}),$
(1)

where $f_{dc}$ indicates a deformable convolution. Specifically, a single
point $\mathbf{p_{o}}$ of $\mathbf{x}^{\prime}$ is obtained as:

$\mathbf{x^{\prime}}(\mathbf{p_{o}})=\sum_{k=1}^{N_{\mathbf{p}}}\mathbf{w}(\mathbf{p}_{k})\cdot\mathbf{x}(\mathbf{p_{o}}+\mathbf{p}_{k}+\Delta\mathbf{p}(\mathbf{p}_{k}))\cdot\mathbf{m}(\mathbf{p}_{k}),$
(2)

where $\mathbf{w}(\mathbf{p}_{k})$, $\mathbf{p}(\mathbf{p}_{k})$, and
$\mathbf{m}(\mathbf{p}_{k})$ are convolutional layer weights, offsets, and
modulation scalars between $0$ and $1$, respectively, for the $k$-th sampling
position.

Taking advantage of the enhanced transformation capability, we also leverage
deformable convolution to align features of adjacent frames only with a few
convolution-like operations, instead of using heavy flow estimation networks.

###  3.2 Shortcut Block

As described in Fig. 1(b), Shortcut block $S$ estimates the current frame
features $\mathbf{\hat{f}}_{t}$ given $\mathbf{a}_{t}$ and the reference frame
features $\mathbf{f}_{ref}$ and $\mathbf{a}_{ref}$ as inputs:

$\mathbf{\hat{f}}_{t}=S(\mathbf{f}_{ref},\mathbf{a}_{ref},\mathbf{a}_{t}).$
(3)

Our block effectively obtains rich information from $\mathbf{f}_{ref}$ via
coarse-to-fine alignment referring to alignment between $\mathbf{a}_{ref}$ and
$\mathbf{a}_{t}$. Also, during the fine alignment, our newly-proposed AdaBD
simultaneously performs adaptive blending of the aligned $\mathbf{f}_{ref}$
and the current frame feature $\mathbf{a}_{t}$. Here, $\mathbf{a}_{t}$
supports the synthesis of newly-appeared areas in the current frame.

Coarse-to-Fine Alignment. To handle a wide range of misalignments between the
frames, our model aligns $\mathbf{f}_{ref}$ with the current frame in a
coarse-to-fine manner. Our global/local alignment module consists of an offset
generator to estimate offsets, and deformable convolution layers to deform
features based on the predicted offsets. Following TDAN [33], an offset
generator estimates sampling offsets given the adjacent frame features. For
global alignment, we first downsample the given inputs to enlarge the
receptive fields of the corresponding convolutional layers in a lightweight
manner. The downsampled $\mathbf{a}_{ref}$ and $\mathbf{a}_{t}$ are
concatenated and fed into a global offset generator to generate global offsets
$\Delta\mathbf{p}_{g}\in\mathbb{R}^{2\times\frac{H}{2}\times\frac{W}{2}}$.
Since we only need to capture coarse movement, $\Delta\mathbf{p}_{g}$ includes
a single offset for each kernel, unlike the original DCN. Each offset is
identically applied to all the sampling positions within the kernel. Then, the
deformed features are upsampled back to the original size to obtain
$\mathbf{f}_{ref}^{\prime}$ as follows:

$\mathbf{f}_{ref}^{\prime}=(f_{dc}(\mathbf{w}_{g},(\mathbf{f}_{ref})^{\downarrow
2},\Delta\mathbf{p}_{g},\mathbf{1}))^{\uparrow 2},$ (4)

where $\mathbf{w}_{g}$ denotes weights of global deformable convolution, and
$(\cdot)^{\uparrow 2}$ and $(\cdot)^{\downarrow 2}$ refer to upsampling and
downsampling by a factor of 2 through bilinear interpolation, respectively.
$\mathbf{1}$ indicates a vector filled with 1 so that no modulation is applied
here. While local alignment of the coarsely-aligned feature
$\mathbf{f}_{ref}^{\prime}$ follows the process of global alignment, the
difference lies in that each sampling point of each kernel has a unique
offset, and the alignment operation is conducted in the original resolution.
We leverage $\mathbf{a}_{ref}^{\prime}$ and $\mathbf{a}_{t}$ to estimate local
offsets, where $\mathbf{a}_{ref}^{\prime}$ is the aligned $\mathbf{a}_{ref}$
which is downsampled, deformed, and upsampled with the same weights
$\mathbf{w}_{g}$ used to synthesize $\mathbf{f}_{ref}^{\prime}$.

We estimate the offsets for the decoding layer features $\mathbf{f}$ using the
encoding layer features $\mathbf{a}$ under the assumption that
$\mathbf{a}_{t}$ and $\mathbf{f}_{t}$ have the same structural information. In
video-to-video translation, input and output frames share the same underlying
structure. Thus, it is natural for the network to learn to maintain the
structural information of an input frame throughout the encoding and decoding
process. More details are described in our supplementary materials.

Adaptive Blending and Deformation. During the local alignment, we also take
advantage of the current frame features $\mathbf{a}_{t}$ from the encoding
layer to handle the regions with large motion differences and new objects. To
achieve this in a cost-efficient way, we introduce AdaBD, which simultaneously
aligns $\mathbf{f}_{ref}^{\prime}$ and blends it with $\mathbf{a}_{t}$ in an
adaptive manner, as illustrated in Fig. 1(b) AdaBD.

First, our local offset/mask generator predicts a blending mask
$\mathbf{m}_{b}\in\mathbb{R}^{N_{\mathbf{p}}\times H\times W}$ in addition to
the local offsets $\Delta\mathbf{p}_{l}\in\mathbb{R}^{2N_{\mathbf{p}}\times
H\times W}$. A learnable mask $\mathbf{m}_{b}$ is composed of modulation
scalars ranging from 0 to 1, each of which indicates the blending ratio of the
current features $\mathbf{a}_{t}$ to the aligned reference features. While
DCNv2 [43] originally introduces the modulation scalars to control feature
amplitudes of a single input, we leverage the scalars to adaptively blend the
features from two adjacent frames considering their redundant areas.

We apply deformable convolution by adding local offsets $\Delta\mathbf{p}_{l}$
to sampling positions of the coarsely-aligned reference features
$\mathbf{f}^{\prime}_{ref}$, while the current frame features $\mathbf{a}_{t}$
are fed into standard convolutional operations. Concurrently, blending mask
$\mathbf{m}_{b}$ adaptively combines the two feature maps. In detail, an
output point $\mathbf{p_{o}}$ of $\mathbf{\hat{f}}_{t}$ is calculated as
follows:

$\displaystyle\mathbf{\hat{f}}_{t}(\mathbf{p_{o}})=\sum_{k=1}^{N_{\mathbf{p}}}$
$\displaystyle\mathbf{w}_{l}(\mathbf{p}_{k})\cdot\\{\mathbf{a}_{t}(\mathbf{p_{o}}+\mathbf{p}_{k})\cdot\mathbf{m}_{b}(\mathbf{p}_{k})+$
(5)
$\displaystyle\mathbf{f}_{ref}^{\prime}(\mathbf{p_{o}}+\mathbf{p}_{k}+\Delta\mathbf{p}_{l}(\mathbf{p}_{k}))\cdot(1-\mathbf{m}_{b}(\mathbf{p}_{k}))\\},$

where $\mathbf{w}_{l}$ indicates weights of local deformable convolution.

Intuitively, the higher values of $\mathbf{m}_{b}$ indicate the regions where
current frame features are more required. In other words, Eq. 5 can be
rewritten as a summation of standard convolution and deformable convolution:

$\displaystyle\mathbf{\hat{f}}_{t}=$ $\displaystyle
f_{dc}(\mathbf{w}_{l},\mathbf{a}_{t},\mathbf{0},\mathbf{m}_{b})+f_{dc}(\mathbf{w}_{l},\mathbf{f}_{ref}^{\prime},\Delta\mathbf{p}_{l},1-\mathbf{m}_{b}),$
(6)

In this equation, the convolutional weights $\mathbf{w}_{l}$ are shared
between $\mathbf{a}_{t}$ and $\mathbf{f}_{ref}^{\prime}$. $f_{dc}$ with
$\mathbf{0}$ indicates DCN with zero offsets, illustrating standard
convolutional operation. To save computational costs, we decrease the channel
dimension of all input features and reconstruct the original channel size
before injecting the output features into the remaining layers of the teacher
network.

###  3.3 Training Objectives

To train Shortcut-V2V, we mainly leverage alignment loss, distillation loss,
and GAN losses widely used in image/video translation networks [38, 37, 42].
First, we adopt alignment loss $L_{align}$ to train the deformation layers in
Shortcut-V2V. Since Shortcut-V2V aims to align the reference frame features
$\mathbf{f}_{ref}$ with the current frame, $L_{align}$ computes L1 loss
between the aligned feature $\mathbf{f}^{*}_{ref}$ and the current frame
features $\mathbf{f}_{t}$ extracted from the teacher model. To obtain
$\mathbf{f}^{*}_{ref}$, we align $\mathbf{f}_{ref}$ in a coarse-to-fine manner
without an intervention of $\mathbf{m}_{b}$ or blending with the current
features. The alignment loss $L_{align}$ is formulated as follows:

$\mathbf{f}^{*}_{ref}=f_{dc}(\mathbf{w}_{l},\mathbf{f}_{ref}^{\prime},\Delta\mathbf{p}_{l},1),$
(7) $L_{align}=\left\|\mathbf{f}_{t}-\mathbf{f}^{*}_{ref}\right\|_{1}.$ (8)

Additionally, we employ knowledge distillation losses at the feature and
output levels. A feature-level distillation loss $L_{feat}$ is applied between
the estimated feature $\mathbf{\hat{f}}_{t}$ and the ground truth feature
$\mathbf{f}_{t}$, while an output-level distillation loss $L_{out}$ compares
the approximated output $\mathbf{\hat{O}}_{t}$ to the output $\mathbf{O}_{t}$
generated by the teacher network. The perceptual loss [41] $L_{perc}$ is also
incorporated to distill the high-frequency information of the outputs.

Lastly, we utilize a typical GAN loss $L_{GAN}$ and a temporal GAN loss
$L_{T-GAN}$, following the existing video-based frameworks [37, 3]. Temporal
GAN loss $L_{T-GAN}$ encourages both temporal consistency and realisticity of
the output frames. For the GAN losses, we consider the outputs of the teacher
network as real images.

The overall objective function $L_{total}$ is as follows:

$\displaystyle L_{total}=$
$\displaystyle\lambda_{align}L_{align}+\lambda_{feat}L_{feat}+\lambda_{out}L_{out}$
(9)
$\displaystyle+\lambda_{perc}L_{perc}+\lambda_{GAN}L_{GAN}+\lambda_{T-GAN}L_{T-GAN},$

where $\lambda_{align}$, $\lambda_{feat}$, $\lambda_{out}$, $\lambda_{perc}$,
$\lambda_{GAN}$, and $\lambda_{T-GAN}$ are hyperparameters to control relative
significance among the losses. More details on training objectives are
described in the supplementary materials.

Model | Dataset | Method | FVD | MACs (G) | Param. (M)
---|---|---|---|---|---
|  | Original | 1.253 | 84.61 | 7.84
|  | CAT | 1.310 | 22.83 | 1.48
|  | OMGD | 1.199 | 25.85 | 1.21
| V2C | Ours | 1.180 | 18.19 | 0.32
|  | Original | 1.663 | 42.36 | 7.84
|  | CAT | 1.957 | 12.02 | 1.71
|  | OMGD | 2.467 | 12.93 | 1.21
Unsup | L2V | Ours | 1.754 | 9.09 | 0.32
|  | Original | 0.186 | 2066.69 | 365
|  | CAT | 0.288 | 380.55 | 24.67
|  | OMGD | 0.264 | 485.84 | 35.81
|  | Fast-Vid2Vid | 0.223 | 767.63 | 63.17
| E2F | Ours | 0.209 | 359.99 | 8.29
|  | Original | 0.146 | 1254.17 | 411.34
|  | CAT | 0.304 | 399.36 | 71.11
|  | OMGD | 0.346 | 391.59 | 47.83
|  | Fast-Vid2Vid | 0.310 | 455.29 | 71.8
vid2vid | L2C | Ours | 0.165 | 389.16 | 52.58
|  |  |  |  |

Table 1: Quantitative comparison with baselines.

Dataset | Method | mIoU | AC | MP
---|---|---|---|---
V2L | Original | 12.2 | 16.0 | 62.8
CAT | 5.99 | 8.82 | 44.2
OMGD | 9.76 | 13.0 | 58.7
Ours | 11.5 | 15.2 | 61.4
L2V | Original | 10.0 | 15.6 | 47.3
CAT | 4.10 | 7.81 | 27.6
OMGD | 3.42 | 6.87 | 29.5
Ours | 9.24 | 14.9 | 43.5

Table 2: Comparison of segmentation score for V2L and FCN-score for L2V on
Unsup. Figure 2: Qualitative comparison. The upper rows represent the results
of Unsup V2C, L2V, and V2L. The bottom rows show the results of vid2vid E2F
and L2C.

##  4 Experiments

###  4.1 Experimental Settings

Models and Datasets. To demonstrate the generality of our model, we conduct
experiments on various tasks with widely-used video-to-video translation
models, Unsupervised RecycleGAN [35] and vid2vid [37].

Unsupervised RecycleGAN (Unsup) is the state-of-art video-to-video translation
model among CycleGAN-based [42] ones. We conduct experiments for Unsup on
Viper→Cityscapes (V2C), which involves translating inter-modality from game
driving scenes (Viper [25]) to real-world driving scenes (Cityscapes [5]). The
Viper dataset consists of driving scenes collected from the GTA-V game engine,
with 56 training videos and 21 test videos. Cityscapes is a dataset composed
of 2,975 training videos and 500 validation videos. The input images are
resized into 256$\times$512 for the experiments. We also tackle the
translation of the videos from the Viper dataset to their corresponding
segmentation label maps, Viper→Label (V2L), and vice versa, Label→Viper (L2V).
We resize the images and the labels into 256$\times$256 following the previous
work [1, 35].

vid2vid is a widely-used pix2pixHD-based video-to-video translation network
that serves as a base architecture for various recent video-to-video
translation models [36, 20]. Following vid2vid, we evaluate Shortcut-V2V on
Edge→Face (E2F) and Label→Cityscapes (L2C). E2F translates edge maps into
facial videos from the FaceForensics [26], containing 704 videos for training
and 150 videos for validation with various lengths. The images are cropped and
resized to 512$\times$512 for the experiments. The edge maps are extracted
using the estimated facial landmarks and Canny edge detector. Also, L2C
synthesizes videos of driving scenes from segmentation label maps using the
Cityscapes. We generate segmentation maps using pretrained networks following
the previous studies [37, 44]. The images and the labels are resized into
256$\times$512.

Evaluation Metrics. Primarily, we adopt the Fréchet video distance (FVD) score
[34] to evaluate the performance of Shortcut-V2V quantitatively. The FVD score
measures the Fréchet Inception distance between the distribution of video-
level features extracted from generated and real videos. The lower the FVD
score is, the better visual quality and temporal consistency the generated
video frames have. For V2L and L2V, we follow the measurement of the
evaluation metrics in the teacher model, Unsup [35]. We measure the
segmentation scores, mean intersection over union (mIoU), mean pixel accuracy
(MP), and average class accuracy (AC), to validate the performance of V2L. L2V
is evaluated using FCN-score [19]. For the evaluation, we first estimate label
maps from the generated videos using the FCN model pretrained with the Viper
dataset. Then, we measure how accurately the estimated label maps are mapped
to the ground truth segmentation labels. Higher FCN-scores refer to better
output quality.

Implementation Details. We attach our Shortcut block to the fixed teacher
networks implemented based on the official codes and pretrained by the
authors, except for Unsup V2C. The teacher network of Unsup V2C is trained
from scratch in the same way the original paper described [35].

Also, standard convolutional kernels in the Shortcut block are replaced with
HetConv [29] to further enhance computational efficiency without a performance
drop. For the convenience of implementation and training stability, we intend
$\mathbf{a}$ and $\mathbf{f}$ to have the same spatial size. Furthermore, we
set the max interval $\alpha$ for each dataset considering the factors that
reflect motion differences between frames, such as the frame per second (FPS)
of training videos. Lower FPS usually results in larger motion differences
between the adjacent frames, requiring a shorter max interval $\alpha$ and
vice versa. Specifically, we set $\alpha$ as 3 on V2C, V2L, L2V, and L2C,
since the FPS of Viper [25] and Cityscapes [5] is 15 and 17, respectively.
Also, $\alpha$ of E2F is set as 6, where the FPS of FaceForensics [26] is 30.
Additional details are included in the supplementary materials.

Figure 3: Performance-efficiency trade-off of the original model,
Shortcut-V2V, and the existing compression methods including OMGD [24], CAT
[13], and Fast-Vid2Vid [44]. We measure the FVD score and MACs, where the
lower FVD score indicates better quality. Red points and stars denote ours
with various model sizes.

###  4.2 Comparison to Baselines

To demonstrate the effectiveness of our framework, we conduct qualitative and
quantitative evaluations compared to the original model and other baselines.
Since this is the first work that tackles a generally applicable compression
framework for video-to-video translation, we compare Shortcut-V2V to the
existing compression methods for image-to-image translation, CAT [13] and OMGD
[24], regarding video frames as individual images. In the case of vid2vid, we
additionally conduct a comparison to Fast-Vid2Vid [44], which is the
compression method designed specifically for vid2vid. For a fair comparison,
we compress the student networks of the baselines to have similar or higher
MACs compared to our model.

Qualitative Evaluation. According to Fig. 2, our method presents outputs of
comparable visual quality to the original model with much fewer computations.
In contrast, CAT on Unsup V2C generates undesirable buildings in the sky, and
OMGD on Unsup V2C struggles with noticeable artifacts. For Unsup L2V, CAT and
OMGD generate unrealistic textures for the terrain in the middle or the
vegetation on the right. Moreover, in V2L, CAT estimates inappropriate labels
on the sky and OMGD on the vegetation.

For vid2vid E2F, CAT shows unwanted artifacts on the mouth, and OMGD presents
inconsistent outputs. For vid2vid L2C, the outputs of CAT are blurry,
especially for the trees in the background, and OMGD generates artifacts at
the bottom of the images. Although Fast-Vid2Vid shows reasonable image
quality, the output frames are inaccurately aligned with the input (_e.g_., a
head pose of a person) and suffer from ghost effects (_e.g_., the black car on
the left) due to motion compensation using interpolation.

Quantitative Evaluation. As shown in Table 1 and Table 2, our framework
successfully improves the computational efficiency of the original network
without a significant performance drop. In the case of Unsup, Shortcut-V2V
reduces the MACs by 4.7$\times$ and the number of parameters by 24.5$\times$.
In addition, our approach saves vid2vid’s MACs by 3.2 and 5.7$\times$ and the
number of parameters by 7.8 and 44$\times$ on each task. Fig. 3 visualizes the
performance-efficiency trade-off of our framework and other baselines.

According to Table 1, we also outperform other compression methods for image-
to-image translation, CAT [13] and OMGD [24], even with fewer computations.
Image-based approaches cannot consider temporal coherence among the frames
during the compression, leading to a loss of quality. Meanwhile, we
effectively preserve the original performance by exploiting rich information
in the previous features during inference. Shortcut-V2V even shows superiority
over Fast-Vid2Vid [44], which is a compression method specifically designed
for vid2vid. Table 2 also demonstrates that ours on Unsup surpasses the
existing compression models by a large margin.

Configurations | Unsup V2C | vid2vid E2F
---|---|---
(a) w/o reference features | 1.256 | 0.398
(b) w/o current features | 1.249 | 0.218
(c) Single-stage alignment | 1.195 | 0.213
(d) w/o adaptive blending | 1.208 | 0.244
(e) Ours | 1.180 | 0.209

Table 3: An ablation study on Unsup V2C and vid2vid E2F. We measure the FVD
scores.

###  4.3 Ablation Study

We conduct an ablation study to evaluate the effect of each component of
Shortcut-V2V. As described in Table 3, we compare the FVD scores of 5
different configurations, (a) w/o reference features, (b) w/o current
features, (c) single-stage alignment, (d) w/o adaptive blending, and (e) ours,
on Unsup V2C and vid2vid E2F.

First, while our model originally exploits both the reference features
$\mathbf{f}_{ref}$ and the current frame features $\mathbf{a}_{t}$ to
synthesize $\mathbf{\hat{f}}_{t}$, (a) and (b) are designed to leverage only
either of them. To be specific, (a) synthesizes $\mathbf{\hat{f}}_{t}$ by
processing standard convolutions on $\mathbf{a}_{t}$ without integrating the
deformed $\mathbf{f}_{ref}$. Meanwhile, (b) estimates $\mathbf{\hat{f}}_{t}$
using only $\mathbf{f}_{ref}$ aligned in a coarse-to-fine manner without
blending $\mathbf{a}_{t}$. The results demonstrate that the absence of either
$\mathbf{f}_{ref}$ or $\mathbf{a}_{t}$ leads to performance degradation. Next,
(c) performs a single-stage alignment on $\mathbf{f}_{ref}$ to show the
necessity of a coarse-to-fine alignment. Specifically, (c) leverages a single
offset/mask generator to predict $\Delta\mathbf{p}$ and $\mathbf{m}_{b}$ which
are then used in AdaBD for blending and deformation. The result indicates that
coarse-to-fine alignment effectively assists the estimation of
$\mathbf{\hat{f}}_{t}$ by minimizing misalignment between the features from
adjacent frames. This makes Shortcut-V2V suitable for generating videos with
diverse motion differences. Lastly, (d) blends $\mathbf{f}^{*}_{ref}$ and
$\mathbf{a}_{t}$ simply by element-wise addition instead of adaptive blending
with $\mathbf{m}_{b}$ in AdaBD. The result demonstrates that $\mathbf{m}_{b}$
also encourages better estimation of $\mathbf{\hat{f}}_{t}$ by selectively
exploiting features from temporally adjacent features.

###  4.4 Offset/Mask Visualization

We provide a qualitative analysis of the generated global/local offsets and
blending masks. Fig. 4 visualizes the sampling positions (red points) for each
output position $\mathbf{p}_{o}$ of the current frame (green points), where
the predicted offsets $\Delta\mathbf{p}_{g}$ and $\Delta\mathbf{p}_{l}$ are
added to the original sampling positions. According to the result, global
offsets $\Delta\mathbf{p}_{g}$ effectively reflect the global movement of the
objects such as the bridge. Also, the summation of the global and local
offsets indicates the sampling points refined by local offsets enabling fine
alignment. This shows that the estimated offsets effectively support the
utilization of common features in the reference frame. For the blending masks
$\mathbf{m}_{b}$, Fig. 4 presents that the regions with significant motion
differences (_e.g_., trees) have large mask values compared to regions with
little change (_e.g_., road, sky). That is, our model relies more on the
current features rather than the reference features when the deformation is
challenging, which aligns with our intention and leads to robust performance.

Figure 4: Visualization of global/local offsets and blending masks. The green
points denote output points of the current frame, and the red points around
them are each output point’s sampling positions modified by global offsets
$\Delta\mathbf{p}_{g}$ and local offsets $\Delta\mathbf{p}_{l}$. The values in
a blending mask $\mathbf{m}_{b}$ are averaged by the kernel size for
visualization. The brighter area indicates higher mask values.

Dependence | Unsup V2C | vid2vid E2F
---|---|---
FVD | MACs (G) | Param. (M) | FVD | MACs (G) | Param. (M)
Low | 1.221 | 13.79 | 0.14 | 0.277 | 243.97 | 2.11
Medium | 1.180 | 18.19 | 0.32 | 0.209 | 359.99 | 8.29
High | 1.166 | 27.97 | 1.38 | 0.193 | 475.99 | 32.98

Table 4: Shortcut-V2V performance with different teacher model dependence.
High dependency denotes using more teacher network layers during the
inference.

###  4.5 Performance-Efficiency Trade-off

We present a performance-efficiency trade-off of our framework with respect to
varying model sizes of Shortcut block and teacher model dependence.

Shortcut Block Size. We construct our models of different sizes by reducing
the number of the output channel of 1$\times$1 channel reduction layer by half
(Ours-1/2) and a quarter (Ours-1/4). As shown in Fig. 3, despite the
performance-efficiency trade-off depending on channel dimension, Ours-1/2 and
Ours-1/4 still achieve comparable FVD to the baselines with less computation
costs. On a single RTX 3090 GPU, our models with three different sizes on
Unsup achieve 99.67, 109.69, and 111.80 FPS with an 11.6-21.2% reduction in
inference time compared to the original model’s speed of 88.10 FPS.
Additionally, Ours, Ours-1/2, and Ours-1/4 on vid2vid contribute to a
significant improvement in speed from 5.63 FPS to 12.86, 12.93, and 13.16 FPS,
resulting in a 56.2-57.2% reduction in inference time. Our method is capable
of a real-time inference, unlike the previous work [44] which requires future
frames for the current frame inference due to motion compensation. More
details on channel manipulation are described in the supplementary materials.

Teacher Model Dependence. Since Shortcut-V2V leverages a subset of the teacher
network during inference, the computational costs and memory usage may vary
depending on the amount of teacher network we use. In this regard, we present
an analysis of the computational efficiency and performance of our model with
respect to various levels of teacher model dependence. We categorize the
dependence on the teacher model into three levels, low, medium, and high,
where high dependence indicates using more teacher network layers. According
to Table 4, our model of higher dependence achieves better performance with
larger MACs and the number of parameters. The results also demonstrate that
Shortcut-V2V can leverage temporal redundancy in features extracted from
various layers of the teacher network. Further details for teacher model
dependence and experiments for temporal redundancy are included in the
supplementary materials.

##  5 Discussion and Conclusion

Although Shortcut-V2V significantly improves the test-time efficiency of
video-to-video translation, it still poses several limitations. First, a
constant max interval may induce unsatisfactory outputs when a degree of
temporal redundancy largely varies between frames. Following recent studies
[27, 22, 21], applying an adaptive interval based on frame selection
algorithms or a learnable policy network could be promising future research.
In addition, we need to manually configure various hyperparameters such as
channel dimension, max interval, and teacher model dependence, which can be
automated by NAS [15]. Lastly, our method still has the potential to further
improvement of computational efficiency by compressing the teacher model using
other methods before applying our framework.

Despite the limitations, Shortcut-V2V presents a significant improvement in
test-time efficiency of video-to-video translation networks based on temporal
redundancy reduction, while preserving the original model performance.
Extensive experiments with widely-used video-to-video translation models
successfully demonstrate the general applicability of our framework. To the
best of our knowledge, this is the first work for a general model compression
in the domain of video-to-video translation. We hope our work facilitates
research on video-to-video translation and extends the range of its
application.

Acknowledgments. This work was supported by Institute of Information &
communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government (MSIT) (No.2021-0-02068, Artificial Intelligence Innovation
Hub) and the Ministry of Culture, Sports and Tourism and Korea Creative
Content Agency (Project Number: R2021040097, Contribution Rate: 50).

## References

  * [1] Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh.  Recycle-gan: Unsupervised video retargeting.  Proc. of the European Conference on Computer Vision (ECCV), 2018.
  * [2] Kelvin C. K. Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Understanding deformable alignment in video super-resolution.  Proc. the AAAI Conference on Artificial Intelligence (AAAI), 2021.
  * [3] Ya-Liang Chang, Zhe Yu Liu, Kuan-Ying Lee, and Winston H. Hsu.  Free-form video inpainting with 3d gated convolution and temporal patchgan.  Proc. of the IEEE International Conference on Computer Vision (ICCV), 2019.
  * [4] Yang Chen, Yingwei Pan, Ting Yao, Xinmei Tian, and Tao Mei.  Mocycle-gan: Unpaired video-to-video translation.  In ACM Multimedia (ACM MM), 2019.
  * [5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [6] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  Proc. of the IEEE International Conference on Computer Vision (ICCV), 2017.
  * [7] Jianing Deng, Li Wang, Shiliang Pu, and Cheng Zhuo.  Spatio-temporal deformable convolution for compressed video quality enhancement.  Proc. the AAAI Conference on Artificial Intelligence (AAAI), 2020.
  * [8] Dario Fuoli, Shuhang Gu, and Radu Timofte.  Efficient video super-resolution through recurrent latent space propagation, 2019.
  * [9] Amirhossein Habibian, Haitam Ben Yahia, Davide Abati, Efstratios Gavves, and Fatih Porikli.  Delta distillation for efficient video processing, 2022.
  * [10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [11] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [12] Bangrui Jiang, Zhihuai Xie, Zhen Xia, Songnan Li, and Shan Liu.  Erdn: Equivalent receptive field deformable network for video deblurring.  In Proc. of the European Conference on Computer Vision (ECCV), 2022.
  * [13] Qing Jin, Jian Ren, Oliver J. Woodford, Jiazhuo Wang, Geng Yuan, Yanzhi Wang, and Sergey Tulyakov.  Teachers do more than teach: Compressing image-to-image models.  Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [14] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of StyleGAN.  In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [15] Muyang Li, Ji Lin, Yaoyao Ding, Zhijian Liu, Jun-Yan Zhu, and Song Han.  GAN compression: Efficient architectures for interactive conditional gans.  Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [16] Ji Lin, Chuang Gan, and Song Han.  Temporal shift module for efficient video understanding.  Proc. of the IEEE International Conference on Computer Vision (ICCV), 2019.
  * [17] Jiayi Lin, Yan Huang, and Liang Wang.  Fdan: Flow-guided deformable alignment network for video super-resolution.  arXiv preprint arXiv:2105.05640, 2021.
  * [18] Yifan Liu, Chunhua Shen, Changqian Yu, and Jingdong Wang.  Efficient semantic video segmentation with per-frame inference.  Proc. of the European Conference on Computer Vision (ECCV), 2020.
  * [19] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [20] Arun Mallya, Ting-Chun Wang, Karan Sapra, and Ming-Yu Liu.  World-consistent video-to-video synthesis.  Proc. of the European Conference on Computer Vision (ECCV), 2020.
  * [21] Yue Meng, Rameswar Panda, Chung-Ching Lin, Prasanna Sattigeri, Leonid Karlinsky, Kate Saenko, Aude Oliva, and Rogério Feris.  Adafuse: Adaptive temporal fusion network for efficient action recognition.  Proc. the International Conference on Learning Representations (ICLR), 2021.
  * [22] Bowen Pan, Rameswar Panda, Camilo Fosco, Chung-Ching Lin, Alex Andonian, Yue Meng, Kate Saenko, Aude Oliva, and Rogerio Feris.  Va-red2: Video adaptive redundancy reduction.  Proc. the International Conference on Learning Representations (ICLR), 2021.
  * [23] KwanYong Park, Sanghyun Woo, Dahun Kim, Donghyeon Cho, and In So Kweon.  Preserving semantic and temporal consistency for unpaired video-to-video translation.  ACM Multimedia (ACM MM), 2019.
  * [24] Yuxi Ren, Jie Wu, Xuefeng Xiao, and Jianchao Yang.  Online multi-granularity distillation for gan compression.  Proc. of the IEEE International Conference on Computer Vision (ICCV), 2021.
  * [25] Stephan R. Richter, Zeeshan Hayder, and Vladlen Koltun.  Playing for benchmarks.  In Proc. of the IEEE International Conference on Computer Vision (ICCV), 2017.
  * [26] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner.  FaceForensics++: Learning to detect manipulated facial images.  In Proc. of the IEEE International Conference on Computer Vision (ICCV), 2019.
  * [27] Jonghyeon Seon, Jaedong Hwang, Jonghwan Mun, and Bohyung Han.  Stop or forward: Dynamic layer skipping for efficient action recognition.  In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
  * [28] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Proc. the International Conference on Learning Representations (ICLR), 2015.
  * [29] Pravendra Singh, Vinay Kumar Verma, Piyush Rai, and Vinay P. Namboodiri.  Hetconv: Heterogeneous kernel-based convolutions for deep cnns.  Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [30] Ximeng Sun, Rameswar Panda, Chun-Fu Chen, Aude Oliva, Rogério Feris, and Kate Saenko.  Dynamic network quantization for efficient video inference.  Proc. of the IEEE International Conference on Computer Vision (ICCV), 2021.
  * [31] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [32] Ryan Szeto, Mostafa El-Khamy, Jungwon Lee, and Jason J. Corso.  Hypercon: Image-to-video model transfer for video-to-video translation tasks.  IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021.
  * [33] Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu.  TDAN: temporally deformable alignment network for video super-resolution.  Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [34] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly.  Fvd: A new metric for video generation.  In Proc. the International Conference on Learning Representations Workshop (ICLRW), 2019.
  * [35] Kaihong Wang, Kumar Akash, and Teruhisa Misu.  Learning temporally and semantically consistent unpaired video-to-video translation through pseudo-supervision from synthetic optical flow.  Proc. the AAAI Conference on Artificial Intelligence (AAAI), 2022.
  * [36] Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro.  Few-shot video-to-video synthesis.  Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2019.
  * [37] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  Video-to-video synthesis.  In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2018.
  * [38] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [39] Xintao Wang, Kelvin C. K. Chan, Ke Yu, Chao Dong, and Chen Change Loy.  EDVR: video restoration with enhanced deformable convolutional networks.  Proc. of the IEEE Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2019.
  * [40] Shuai Yang, Liming Jiang, Ziwei Liu, and Chen Change Loy.  Vtoonify: Controllable high-resolution portrait video style transfer.  ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia), 2022.
  * [41] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [42] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Proc. of the IEEE International Conference on Computer Vision (ICCV), 2017.
  * [43] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai.  Deformable convnets v2: More deformable, better results.  Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [44] Long Zhuo, Guangcong Wang, Shikai Li, Wayne Wu, and Ziwei Liu.  Fast-vid2vid: Spatial-temporal compression for video-to-video synthesis.  Proc. of the European Conference on Computer Vision (ECCV), 2022.

Supplementary Material

In supplementary material, we describe temporal redundancy experiments,
analysis of features, implementation details, additional visualization of
offsets and masks, and application of our method to the StyleGAN2-based model.

##  Appendix A Temporal Redundancy Experiments

Layer | CC | Norm. RMSE
---|---|---
$5^{\text{th}}$ ResBlock | 0.80 | 3.917e-4
$6^{\text{th}}$ ResBlock | 0.78 | 4.023e-4
$1^{\text{st}}$ Upsample | 0.80 | 2.454e-4
$2^{\text{nd}}$ Upsample | 0.78 | 1.667e-4
((a)) Unsup V2C

Layer | CC | Norm. RMSE
---|---|---
$8^{\text{th}}$ ResBlock | 0.87 | 2.341e-4
$9^{\text{th}}$ ResBlock | 0.86 | 2.418e-4
$1^{\text{st}}$ Upsample | 0.93 | 1.075e-4
$2^{\text{nd}}$ Upsample | 0.93 | 2.813e-5
$3^{\text{rd}}$ Upsample | 0.93 | 3.014e-5
((b)) vid2vid E2F

Table 5: Feature-level temporal redundancy between adjacent video frames. (a)
and (b) are the results of Unsup V2C and vid2vid E2F, respectively. ‘Layer’
indicates a layer which we extracted the features from. ‘CC’ and ‘Norm. RMSE’
denote correlation coefficient and normalized RMSE, respectively.

In Shortcut-V2V, neighboring video frames are assumed to have redundancy. In
this respect, we quantitatively demonstrate the feature-level temporal
redundancy between adjacent video frames following VA-RED2 [22]. VA-RED2
conducts redundancy analysis to motivate its redundancy reduction framework
for efficient video recognition.

As in VA-RED2, we measure the Pearson correlation coefficient (CC) and
normalized root mean squared error (RMSE) between adjacent frames. CC ranges
from $-1$ to $+1$, where a nonzero value indicates the existence of a positive
or negative correlation, and the higher absolute value means a higher
correlation. That is, a value closer to $+1$ denotes that two variables tend
to move in the same direction. Unlike VA-RED2, we normalize each feature
before calculating RMSE to negate the scale differences between the features
from different frames or different layers.

Since our framework approximates features from a decoding layer using the
previous features from the same layer, we demonstrate the temporal redundancy
of the decoding layers of the teacher networks. Specifically, a generator of
Unsupervised RecycleGAN [35] (Unsup) contains an input layer, 2 downsampling
layers (Downsample), 6 residual blocks (ResBlock), 2 upsampling layers
(Upsample), and an output layer. For Unsup, the features are extracted from
the end of the $5^{\text{th}}$ and $6^{\text{th}}$ ResBlocks, and
convolutional layers of the $1^{\text{st}}$ and $2^{\text{nd}}$ Upsample.
Also, vid2vid [37] generator consists of an input layer, 3 Downsamples, 9
ResBlocks, 3 Upsamples, and an output layer. Accordingly, we extract features
from the end of the $8^{\text{th}}$ and $9^{\text{th}}$ ResBlocks, and
convolutional layers of the $1^{\text{st}}$, $2^{\text{nd}}$, and
$3^{\text{rd}}$ Upsample. We use 100 randomly sampled 30-frame videos and
extract 29 pairs of adjacent frames from each video. We conduct experiments on
Viper→Cityscapes (V2C) for Unsup and Edge→Face (E2F) for vid2vid.

Table 5 presents that feature-level temporal redundancy exists regardless of
the type of teacher network or layer. Positive CC values close to 1 and
normalized RMSEs close to 0 effectively justify the motivation for our
approach based on temporal redundancy reduction. Furthermore, as also
presented in experiments of the performance-efficiency trade-off according to
teacher model dependence (Sec. 4.5 in the main paper), our model can be
applied to various layers of a teacher network.

Figure 5: Visualization of (a) feature-level structural information and (b)
cosine similarity map.

##  Appendix B Analysis of Features

As described in Sec.3.2 of our main paper, AdaBD predicts the offsets of
feature $\mathbf{f}$ from the feature space of $\mathbf{a}$ under the
assumption that $\mathbf{a}_{t}$ and $\mathbf{f}_{t}$ have the same structural
information. In video-to-video translation, input and output frames share the
same underlying structure. Thus, it is natural for the network to learn to
maintain the structural information of an input frame during the encoding and
decoding process. To validate this, we randomly extract 5 channels of
$\mathbf{a}_{t}$ and $\mathbf{f}_{t}$ of a random-sampled frame
$\mathbf{I}_{t}$ from a test set. Fig. 5(a) shows their similar spatial
structure, making it reasonable to apply offsets from $\mathbf{a}$ space to
$\mathbf{f}$ space.

For blending, we note that $\mathbf{a}$ and $\mathbf{f}$ first pass through
separate 1$\times$1 Convs, which serve to convert the features into the same
space while reducing channel dimension. To show the convertibility of the
separate 1$\times$1 Convs, we design a network consisting of two 1$\times$1
Convs and show that it can learn to directly convert $\mathbf{a}_{t}$ to
$\mathbf{f}$-space features $\hat{\mathbf{f}}_{t}$ through $L1$ loss with
$\mathbf{f}_{t}$. Fig. 5(b) shows the average cosine similarity maps for all
channels of $\mathbf{a}_{t}$ and $\mathbf{f}_{t}$, and $\hat{\mathbf{f}}_{t}$
and $\mathbf{f}_{t}$ from 100 randomly sampled 30-frame test videos. The
similarities between $\hat{\mathbf{f}}_{t}$ and $\mathbf{f}_{t}$ notably
increase, particularly for the same channels (_i.e_. diagonal), compared to
$\mathbf{a}_{t}$ and $\mathbf{f}_{t}$.

##  Appendix C Implementation Details

###  C.1 Model Architectures

Coarse-to-Fine Alignment. The global offset generator consists of 2
convolutional layers given two temporally neighboring features but downsampled
by a factor of 2, $(\mathbf{a}_{ref})^{\downarrow 2}$ and
$(\mathbf{a}_{t})^{\downarrow 2}$. The downsampled inputs encourage the global
offset generator to capture relatively coarse movements with a larger
receptive field. Meanwhile, the local offset/mask generator includes 3
convolutional layers given inputs of the original resolution, $\mathbf{a}_{t}$
and $\mathbf{a}^{\prime}_{ref}$ (globally aligned features). While local
offsets contain a distinct offset for every sampling point in each kernel,
global offsets include a single offset per kernel to further support the
capture of coarse motions.

As mentioned in Sec. 3.2 of the main paper, we obtain
$\mathbf{a}_{ref}^{\prime}$ by deforming $\mathbf{a}_{ref}$, the current frame
features from an encoding layer, with the same parameters used for
$\mathbf{f}_{ref}$, the reference features, in Eq. 4 of our main paper, where
$\mathbf{a}_{ref}^{\prime}=(f_{dc}(\mathbf{w}_{g},(\mathbf{a}_{ref})^{\downarrow
2},\Delta\mathbf{p}_{g},\mathbf{1}))^{\uparrow 2}$. Here, $f_{dc}$ indicates a
deformable convolution, $\mathbf{w}_{g}$ is convolutional layer weights for
global alignment, $\Delta\mathbf{p}_{g}$ denotes global offsets, and
$\mathbf{1}$ is a vector filled with the scalar value 1.

AdaBD. AdaBD includes two convolutional layers, one for adaptive blending and
deformation, and the other for the reconstruction of the output channel
dimension. Given a single set of convolutional weights for the adaptive
blending and deformation, we apply deformable convolutions by adding local
offsets $\Delta\mathbf{p}_{l}$ to the sampling position on the coarsely-
aligned reference features $\mathbf{f}^{\prime}_{ref}$, while the current
frame features $\mathbf{a}_{t}$ is fed into standard convolutional operations
without deformation. At the same time, a blending mask $\mathbf{m}_{b}$
adaptively combines the two feature maps. In detail, a point $\mathbf{p_{o}}$
of the estimated current feature $\mathbf{\hat{f}}_{t}$ obtained by Eq. 5 in
our main paper. It can be expanded as follows:

$\displaystyle\mathbf{\hat{f}}_{t}(\mathbf{p_{o}})=$
$\displaystyle\sum_{k=1}^{N_{\mathbf{p}}}\mathbf{w}_{l}(\mathbf{p}_{k})\cdot\\{\mathbf{a}_{t}(\mathbf{p_{o}}+\mathbf{p}_{k})\cdot\mathbf{m}_{b}(\mathbf{p}_{k})\\}+$
(10)
$\displaystyle\sum_{k=1}^{N_{\mathbf{p}}}\mathbf{w}_{l}(\mathbf{p}_{k})\cdot\\{\mathbf{f}^{\prime}_{ref}(\mathbf{p_{o}}+\mathbf{p}_{k}+\Delta\mathbf{p}_{l}(\mathbf{p}_{k}))\cdot(1-\mathbf{m}_{b}(\mathbf{p}_{k}))\\},$

where $N_{\mathbf{p}}$ indicates the number of sampling points in a kernel and
$\mathbf{w}_{l}$ is the convolutional weights. Thus, we implement AdaBD easily
by a summation of convolution and deformable convolution as written in Eq. 6
in our main paper.

Figure 6: Additional offset visualization. (a) presents the results of Unsup
V2C and (b) shows the ones of vid2vid E2F. The cropped images with
$\Delta\mathbf{p}_{g}$ and $\Delta\mathbf{p}_{g}+\Delta\mathbf{p}_{l}$
indicate the area of a red box in reference frames. On the other hand, the
cropped images with $\mathbf{p}_{o}$ are the region of a red box in current
frames. The green points are the output points of the current frame and the
red points are their sampled points. Figure 7: Additional mask visualization.
The third column of (a) and (b) shows blending masks $\mathbf{m}_{b}$ from
Unsup V2C and vid2vid E2F, respectively. The areas with blue boxes represent
the regions with higher mask values compared to other regions.

###  C.2 Details on Training Objectives

We provide details on the training objectives we described in Sec.3.3 in our
main paper. The distillation losses, $L_{feat}$ and $L_{out}$, are computed as
follows:

$L_{feat}=\left\|\mathbf{f}_{t}-\mathbf{\hat{f}}_{t}\right\|_{1}$ (11)
$L_{out}=\left\|\mathbf{O}_{t}-\mathbf{\hat{O}}_{t}\right\|_{1}$ (12)

The perceptual loss $L_{perc}$ between the teacher output and the estimated
output is defined as follows:

$L_{perc}=\left\|VGG_{j}(\mathbf{O}_{t})-VGG_{j}(\mathbf{\hat{O}}_{t})\right\|_{1},$
(13)

where $VGG_{j}$ indicates the $j$-th layer in VGG-Net[28] and
$j\in\\{1,2,3,4\\}$.

Finally, GAN loss $L_{GAN}$ and temporal GAN loss $L_{T-GAN}$ are formulated
as follows:

$L_{GAN}=\mathbb{E}[\log
D(\mathbf{O}_{t})]+\mathbb{E}[\log(1-D(\mathbf{\hat{O}}_{t})]$ (14)
$\displaystyle L_{T-GAN}=$ $\displaystyle\mathbb{E}[\log
D_{T}(\mathbf{O}_{ref};\mathbf{O}_{t})]+$ (15)
$\displaystyle\mathbb{E}[\log(1-D_{T}(\mathbf{O}_{ref};\mathbf{\hat{O}}_{t})],$

where $D$ and $D_{T}$ denote the discriminator and the temporal discriminator,
respectively. Also, $ref$ is the reference timestep used to estimate frames at
$t$ timestep. The reference timestep’s output $\mathbf{O}_{ref}$ is fed into
$D_{T}$ in addition to the current output $\mathbf{\hat{O}}_{t}$ to encourage
temporal consistency between the temporally neighboring outputs. The temporal
discriminator is composed of several 3D convoluational layers based on the
architecture of PatchGAN [11] discriminator. Here, the teacher model outputs
are regarded as real images. For hyperparameters, we set $\lambda_{GAN}$,
$\lambda_{T-GAN}$, $\lambda_{feat}$, $\lambda_{align}$, $\lambda_{out}$, and
$\lambda_{perc}$ as 1, 1, 5, 5, 10, and 10 respectively.

###  C.3 Training and Inference

Shortcut-V2V can be easily applied to a fixed pretrained video-to-video
translation network to improve its test-time efficiency. For the training, we
provide our model with two adjacent video frames (_i.e_., current frame and
reference frame) whose timestep difference is smaller than the maximum
interval $\alpha$. Shortcut-V2V with Unsup randomly samples two frames within
an interval of $\alpha$ from each of the 56 training videos. To train our
model with vid2vid, it adopts a sequential generation following the teacher
model, while updating the reference frame features with the teacher model’s
features every $\alpha$ timestep.

The offsets are zero-initialized and the blending mask is initialized with
0.5. The learning rate of the Shortcut block with Unsup is $2\times 10^{-4}$,
and $2\times 10^{-7}$ for discriminators. Also, the learning rate of the
Shortcut block is set as $1\times 10^{-4}$ and $5\times 10^{-5}$ for vid2vid
E2F and vid2vid L2C, respectively. The learning rate of discriminators is
$2\times 10^{-5}$ for both tasks. We train our model with 3200 epochs for
Unsup and 40 epochs for vid2vid. At the test phase, we perform full model
inference at every $\alpha$ to obtain the output frame and two sorts of
reference features, $\mathbf{f}_{ref}$ and $\mathbf{a}_{ref}$. For the rest
timesteps, we synthesize the outputs by replacing temporally redundant
operations in the original network with our lightweight Shortcut block. In the
case of Unsup, we use our model’s output instead of the teacher’s output at
every $\alpha$ while still extracting the reference features from the teacher
model. This helps to improve the temporal consistency of videos. In addition,
following vid2vid, our model with vid2vid generates the first and second
frames using a pretrained image-to-image translation model.

As in Fast-Vid2Vid [44], we only use the first scale generator of vid2vid.
Moreover, since our model includes alignment modules, we exclude flow
estimation branch of vid2vid during the inference of our model. Unlike vid2vid
on L2C which originally utilizes an additional generator for foreground object
generation, we mainly compress the generators except for the foreground model.

###  C.4 Performance-Efficiency Trade-off Experiments

Channel Dimension. In Sec. 4.5 of our main paper, we construct our model with
varying sizes (_i.e_., Ours, Ours-1/2, Ours-1/4) by manipulating the output
channel dimension of $1\times 1$ channel reduction layer. In particular, for
Unsup, we set the channel dimension as 128 for Ours, 64 for Ours-1/2, and 32
for Ours-1/4, where the channel dimension of the inputs is 256. Also, while
the input channel dimension of vid2vid is 512, we set the channel dimension as
256 for Ours, 128 for Ours-1/2, and 64 for Ours-1/4 for vid2vid.

Teacher Model Dependence. We present the performance-efficiency trade-off with
three different teacher model dependence levels, low, medium, and high, as
described in Sec. 4.5 in our main paper. A higher level indicates using more
layers of the original network during the inference of our model.
Specifically, our Shortcut block replaces the operations from $2^{\text{nd}}$
Downsample in the encoder to $2^{\text{nd}}$ Upsample in the decoder, for the
low level. Medium level indicates the shortcut from $1^{\text{st}}$ Downsample
in the encoder to $1^{\text{st}}$ Upsample in the decoder, which is our
default mode. Finally, we only replace all ResBlocks with Shortcut blocks for
the high level.

##  Appendix D Additional Offset/Mask Visualization

We present further analysis of the offsets and masks estimated by our model.
Fig 6 visualizes the predicted global offsets, $\Delta\mathbf{p}_{g}$, and the
sum of the global offsets and local offsets,
$\Delta\mathbf{p}_{g}+\Delta\mathbf{p}_{l}$. For visualization, we scale the
offsets by the ratio of output frame resolution to feature map resolution. The
green points indicate the point $\mathbf{p}_{o}$ of the current frame output,
and the red points are the sampled points by the offsets. As described in Eq.2
in our main paper, the sampled points are integrated to generate the output
point of the current features.

The results show that the offsets are successfully estimated to sample proper
points for the alignment of the reference frame. For example, the global
offsets capture coarse movements of the objects such as vegetation, car, and
the patterns on the wall as shown in Fig 6(a). Additionally, Fig 6(b)
demonstrates that the predicted global offsets properly align the faces in the
reference frame with the ones in the current frame by identifying the global
movements of eyes, nose, and mouse. Overall, the summation of global and local
offsets presents that the local offsets refine the sampling positions to
further capture detailed movements.

Moreover, Fig 7 illustrates the estimated blending masks, $\mathbf{m}_{b}$.
The mask values are averaged by the kernel size for visualization, where the
brighter areas indicate higher mask values. The results present that the
predicted masks contain higher values for the regions of large motion
differences, occlusions, and new objects, marked with blue boxes. For these
regions, our model exploits more current features than reference features to
synthesize the corresponding output.

Method | MACs | Param. | Style | Content | Temporal | Overall
---|---|---|---|---|---|---
Original | 515.51G | 48.46M | 0.26 | 0.24 | 0.26 | 0.27
CAT | 307.32G | 20.81M | 0.12 | 0.15 | 0.12 | 0.11
Ours | 303.26G | 15.37M | 0.23 | 0.22 | 0.23 | 0.23
Table 6: User study results of Shortcut-V2V on VToonify. Figure 8: Qualitative
comparison on VToonify.

##  Appendix E Application to StyleGAN2-based Network

We apply Shortcut-V2V to a recently-proposed StyleGAN2-based video-to-video
translation model, VToonify [40]. VToonify contains an encoder and a StyleGAN2
[14] generator. Shortcut-V2V reduces computations between an encoder’s final
layer and the generator’s initial layer. Specifically, a single layer within
StyleGAN2 consists of two distinct branches: ‘skip’ and ‘toRGB’. Thus, two
instances of Shortcut-V2V are employed to predict each branch’s features,
resulting in saving 1.7$\times$ MACs and 3.2$\times$ parameters. We reduce
MACs of the original model by 41% with Shortcut-V2V and also compare with CAT
as a baseline. For evaluation, we conduct a user study following VToonify.
Specifically, 20 graduate school students were asked to score the methods from
1 to 3 with 4 criteria: style matching, content preservation, temporal
consistency, and overall quality. In Table 6, ours obtains comparable Mean
Reciprocal Ranks to the original model and significantly outperforms CAT with
respect to all criteria by a large margin, demonstrating applicability to
StyleGAN2-based models. As shown in Fig. 8, our method presents
indistinguishable results from the original model while CAT struggles with
blurry outputs. It is worth noting that application to VToonify, a
StyleGAN2-based model, provides evidence of the wide applicability of our
methods across various architectures.