Input (1910s) |  1930s |  1970s |  2000s
---|---|---|---

Input (2010s) |  1960s |  1940s |  1920s
---|---|---|---

Given an input portrait image, our method generates plausible renditions of
what that portrait might have looked like had it been taken in different
decades spanning over a century. Our framework captures characteristic styles
across decades while maintaining the person’s identity. From top left to
bottom right: J.P. Morgan, Mindy Kaling, Emmeline Pankhurst, Sandra Oh,
Charlie Chaplin, Brad Pitt.

# What’s in a Decade? Transforming Faces Through Time

Eric Ming Chen1, Jin Sun2, Apoorv Khandelwal3, Dani Lischinski4, Noah
Snavely1, Hadar Averbuch-Elor5
1Cornell University, Cornell Tech 2University of Georgia 3Brown University
4The Hebrew University of Jerusalem 5Tel Aviv University

###### Abstract

How can one visually characterize photographs of people over time? In this
work, we describe the _Faces Through Time_ dataset, which contains over a
thousand portrait images per decade from the 1880s to the present day. Using
our new dataset, we devise a framework for resynthesizing portrait images
across time, imagining how a portrait taken during a particular decade might
have looked like had it been taken in other decades. Our framework optimizes a
family of per-decade generators that reveal subtle changes that differentiate
decades—such as different hairstyles or makeup—while maintaining the identity
of the input portrait. Experiments show that our method can more effectively
resynthesizing portraits across time compared to state-of-the-art image-to-
image translation methods, as well as attribute-based and language-guided
portrait editing models. Our code and data will be available at
https://facesthroughtime.github.io. <ccs2012> <concept>
<concept_id>10010147.10010371.10010382</concept_id> <concept_desc>Computing
methodologies Image manipulation</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10010147.10010371</concept_id> <concept_desc>Computing
methodologies Computer graphics</concept_desc>
<concept_significance>100</concept_significance> </concept> <concept>
<concept_id>10010147.10010178.10010224</concept_id> <concept_desc>Computing
methodologies Computer vision</concept_desc>
<concept_significance>100</concept_significance> </concept> </ccs2012>

[500]Computing methodologies Image manipulation [100]Computing methodologies
Computer graphics [100]Computing methodologies Computer vision

††volume: 42††issue: 2

##  1 Introduction

What would photographs of ourselves look like if we were born fifty, sixty, or
a hundred years ago? What would Charlie Chaplin look like if he were active in
the 2020s instead of the 1920s? Such is the conceit of popular diversions like
old-time photography, where we imagine ourselves as we might have looked in an
anachronistic time period like the Roaring Twenties. However, while many
methods for editing portraits have been devised recently on the basis of
powerful generative techniques like StyleGAN [KLA*20, KAH*20, SYTZ20, AZMW21,
WLS21, APC21, OSF*20], little attention has been paid to the problem of
automatically translating portrait imagery in time, while preserving other
aspects of the person portrayed. This paper addresses this problem, producing
results like those shown in What’s in a Decade? Transforming Faces Through
Time.

To simulate such a “time travel” effect, we must be able to model and apply
the characteristic features of a certain era. Such features may include
stylistic trends in clothing, hair, and makeup, as well as the imaging
characteristics of the cameras and film of the day. In this way, translating
imagery across time differs from standard portrait editing effects that
typically manipulate well-defined semantic attributes (_e.g._ , adding or
removing a smile or modifying the subject’s age). Further, while large amounts
of data capturing these attributes are readily available via datasets like
Flickr-Faces-HQ (FFHQ) [KLA19], diverse and high-quality imagery spanning the
history of photography is comparatively scarce.

1990s |  1970s |  1940s |  1910s |  1880s
---|---|---|---|---

Figure 1: Random samples from five decades in the _Faces Through Time_
Dataset.

In this work, we take a step towards transforming images of people across
time, focusing on portraits. We introduce _Faces Through Time_ (FTT), a
dataset containing thousands of images spanning fourteen decades from the
1880s to the present day. _Faces Through Time_ is derived from the massive
public catalog of freely-licensed images and annotations available through the
Wikimedia Commons project. The extensive biographic depth available on
Wikimedia Commons, as well as its organization into time-based categories,
enables associating images with accurate time labels. In comparison to
previous time-stamped portrait datasets, FTT is sourced from a wider
assortment of images capturing notable identities varying in age, nationality,
pose, etc. In contrast, a well-known prior dataset called The Yearbook Dataset
[GRS*15] only contains US yearbook photos. In our work, we demonstrate FTT’s
applicability for synthesis tasks. More broadly, the dataset can allow for
exploration of a variety of analysis tasks, such as estimating time from
images, understanding photographic styles across time, and discovering fashion
trends (see Section 2). Figure 1 shows random samples from five different
decades from FTT.

To transform portraits across time, we build on the success of Generative
Adversarial Networks (GANs) for synthesizing high-quality facial images. In
particular, we finetune the popular StyleGAN2 [KLA*20, KAH*20] generator
network (trained on FFHQ) on our dataset. However, rather than modeling the
entire image distribution of our dataset using a single StyleGAN2 model, we
train a separate model for each decade. We introduce a method to align and map
a person’s image across the latent generative spaces of the fourteen different
decades (see Figure 3). Furthermore, we discover a remarkable linearity in
each model’s generator weights, allowing us to fine-tune images with vector
arithmetic on the model weights. This sets our approach apart from the many
prior works that search for editing directions within a single StyleGAN2
model, [SYTZ20, SZ21, WLS21, AZMW21, APC21, PWS*21]. We find that by using
multiple StyleGAN2 models in this way, our method is more expressive than
these existing approaches. In addition, our classes are separated in a useful
way for style transfer.

We demonstrate results for a variety of individuals from different backgrounds
and captured during different decades. We show that prior methods struggle on
our problem setting, even when trained directly on our dataset. We also
perform a quantitative evaluation, demonstrating that our transformed images
are of high quality and resemble the target decades, while preserving the
input identity. Our approach enables an _Analysis by Synthesis_ scheme that
can reveal and visualize differences between portraits across time, enabling a
study of fashion trends or other visual cultural elements (related to, for
instance, a New York Times article that discusses how artists use digital
tools to imagine what George Washington would have looked like had he lived
today).

In summary, our key contributions are:

  * •

_Faces Through Time_ , a large, diverse and high-quality dataset that can
serve a variety of computer vision and graphics tasks,

  * •

a new task—transforming faces across time—and a method for performing this
task that uses unique vectorial transformations to modify generator weights
across different models,

  * •

and quantitative and qualitative results that demonstrate that our method can
successfully transform faces across time.

##  2 Related Work

###  2.1 Image analysis across time

While common image datasets such as ImageNet [DDS*09] and COCO [LMB*14] do not
explicitly use time as an attribute, those that do show unique
characteristics. Here we focus on image datasets that feature people.

Datasets. The Yearbook Dataset [GRS*15] is a collection of 37,921 front-facing
portraits of American high school students from the 1900s to the 2010s. The
authors design models to predict when a portrait is taken. They also analyze
the prevalence of smiles, glasses, and hairstyles across different eras. The
IMAGO Dataset [SAL*20] contains over 80,000 family photos from 1845 to 2009.
Images are labeled in “socio-historical classes” such as free-time or fashion.
Hsiao and Grauman [HG21] collect news articles and vintage photos and build a
dataset that feature fashion trends in the 20th century. Our dataset contains
portraits from a much wider age range (the Yearbook Dataset focuses on high
school students), from diverse geographic areas (the IMAGO Dataset focuses on
Italian families), and exhibiting rich variation in occupation and styles (not
just fashion images).

Analysis. A standard task applicable to images with temporal information is to
predict when they were taken, i.e., the date estimation problem. Müller-Budack
et al. [MSE17] train two GoogLeNet models, one for classification and one for
regression to predict the date of a photo. Salem et al. [SWZJ16] train
different CNNs for date estimation using the face, torso, and patch of a
portrait image. Other rich information can also be learned from temporal image
collections. In StreetStyle and GeoStyle [MBS17, MMH*19], a worldwide set of
images taken between 2013-2016 were analyzed to discover spatio-temporal
fashion trends and events. In [HG21], topic models and clusterings are used to
discover trends in news and vintage photos. Additionally, in [LEH13], a
weakly-supervised approach is used to discover stylistic trends in cars over
the decades. [LMC*15] learn trends over two centuries of architecture using
Street View images. [MS14, LGZ*20] also disentangle pictures of buildings in a
spatio-temporal way to see how structures have changed over time. Unlike prior
works that focus on analyzing temporal characteristics in the data, we work on
the more challenging task of modifying such characteristics.

###  2.2 Portrait editing

Editing of face attributes has been extensively studied since before the deep
learning era. A classic approach is to fit a 3D morphable model [BV99, EST*20]
to a face image and edit attributes in the morphable model space. Other
methods that draw on classic vision approaches includes Transfiguring
Portraits [Kem16], which can render portraits in different styles via image
search, selection, alignment, and blending. Given the recent success of
StyleGAN (v1 [KLA19], v2 [KLA*20], and v3 [KAL*21]) in high quality face
synthesis and editing, many works focus on editing portrait images using pre-
trained StyleGAN models. In these frameworks, a photo is mapped into a code in
one of StyleGANs latent spaces [XZY*21]. Feeding the StyleGAN generator with a
modified latent code yields a modified portrait [SYTZ20, WLS21]. To find the
latent code of an input image, one can directly optimize the code so that the
StyleGAN can reconstruct the inputs [AQW19, WT20] or train a feed-forward
network such as pSp [RAP*21] and e4e [TAN*21]that directly predicts the latent
code. We adopt an optimization-based procedure to obtain better facial
details.

Once a latent code of an image is obtained, portrait image editing can be done
in the latent space with a pre-trained StyleGAN generator. Directions for
change of viewpoint, aging, and lighting of faces can be found by PCA in the
latent space [HHLP20], or from facial attributes supervision [SGTZ20]. Shen
and Zhou [SZ21] find that editing directions are encoded in the generators’
weights and can be obtained by eigen decomposition. Collins et al. [CBPS20]
perform local editing of portraits by mixing layers from reference and target
images. Alternatively, the StyleGAN generator can also be modified for
portrait editing. StyleGAN-nada [GPM*21] and StyleCLIP [PWS*21] use CLIP
[RKH*21] to guide editing on images with target attributes. Toonify [PA20]
uses layer-swapping to obtain a new generator from models in different
domains. Similar to StyleAlign [WNSL21], we obtain a family of generators by
finetuning a common parent model on different decades. The style change of a
face is achieved by obtaining the latent code of the input image and feeding
it into a modified target StyleGAN generator using PTI [RMBC21]. Our method is
conceptually simple and doesn’t require exploring the latent space.

Age editing on portraits is related to our task since both involve modifying
the temporal aspects of an image. In the work of Or-El, et al. [OSF*20], age
is represented as a latent code and applied to the decoder network of the face
generator. An identity loss is used to preserve identities across ages. Alaluf
et al. [APC21] design an age encoder that takes a portrait and a target age,
produces a style code modification on pSp-coded styles and generates a new
portrait with the target age. In contrast, our editing aims to change the
decade a photo was taken without altering the subject’s age.

GANs can also be used to recolor historic photographs [ZZI*17, Ant19, LZY*21].
In particular, Time-Travel Rephotography [LZY*21] uses a StyleGAN2 model to
project historic photos into a latent space of modern high-resolution color
photos with modern imaging characteristics. Rather than focusing solely on
low-level characteristics like color, our method alters a diverse collection
of visual attributes, such as facial hair and make-up styles. Moreover, our
method can transform images across a wide range of decades, instead of
learning binary transformations between “old” and “modern” as in [LZY*21].

##  3 The _Faces Through Time_ Dataset

Our _Faces Through Time_ (FTT) dataset features 26,247 images of notable
people from the 19th to 21st centuries, with roughly 1,900 images per decade
on average. It is sourced from Wikimedia Commons (WC), a crowdsourced and
open-licensed collection of 50M images.

We automatically curate data from WC to construct FTT (Figure 1) as follows:
(1) The “People by name” category on WC contains 407K distinct people
identities. We query each identity’s hierarchy of people-centric subcategories
(similar to [CKA*21]) and organize retrieved images by identity. (2) We use a
Faster R-CNN model [RHGS15, JL17, RR17] trained on the WIDER Face dataset
[YLLT16] as a face detector. For each detected face, 68 facial landmarks are
found using the Deep Alignment Network [KNT17], and alignments are applied as
in the FFHQ dataset [KLA19] given these landmarks. (3) We devise a clustering
method based on clique-finding in face similarity graphs to group faces by
identities (see appendix). This resolves ambiguities in photos that feature
multiple people. (4) We gather additional samples without biographic
information from the “19th Century Portrait Photographs of Women” and “20th
Century Portrait Photographs of Women” categories. These make up about 15% of
the dataset.

We leverage image metadata, identity labels, and biographic information
available in WC to further assist in balancing and filtering our data. We
discard any photos without time labels or taken before 1880, and sample a
subset of 3,000 faces each for the 2000s and 2010s decades (which tend to
feature many more images than other decades) to maintain a roughly balanced
distribution of images across decades. For images where the identity is known,
we further filter by only keeping images where the identity is between 18 and
80 years old (comparing the image timestamp and identity’s birth year). We
also estimate face pose using Hopenet [RCR18] and remove images with yaw or
pitch greater than $30$ degrees. After these automated collection and
filtering steps, we manually inspected the entire dataset and removed images
with clearly incorrect dates, images that were not cropped properly, images
that were duplicates of other identities, and images featuring objectionable
content. This resulted in a removal of $6\%$ of the assembled data.

The total number of samples from each decade in our curated dataset,
demographic and other biographic distributions, and further implementation
details can be found in our supplementary material. We create train and test
splits by randomly selecting 100 images per decade as a test set, with the
remaining images used for training. Samples from the dataset are shown in
Figure 1.

As detailed in Section 2, the Yearbook dataset [GRS*15] is another dataset
that spans multiple decades and could potentially allow for transforming faces
across time. However, it is grayscale only, contains one age group, and
critically, its images are low resolution ($186\times 171$). In the
supplementary material, we show that high-quality synthesized images cannot be
obtained using this dataset, further highlighting the benefit of our FTT
dataset.

##  4 Transforming Faces Across Time

Given a portrait image from a particular decade, our goal is to predict what
the same person might look like across various decades ranging from 1880 to
2010. The key challenges are: (1) maintaining the identity of the person
across time, while (2) ensuring the result fits the natural distribution of
images of the target decade in terms of style and other characteristics. We
present a novel two-stage approach that addresses these challenges. Figure 2
shows an overview of our approach.

First, rather than training a single generative model that covers all decades
(e.g., [OSF*20]), we train a family of StyleGAN models, one for each decade
(Section 4.1, Figure 3). These are obtained by fine-tuning the same _parent_
model, resulting in a set of _child_ models whose latent spaces are roughly
_aligned_ [WNSL21], as described in Section 4.1. The alignment ensures that
providing the same latent code to different models results in portraits with
similar high-level properties, such as pose. At the same time, the resulting
images exhibit the unique characteristics of each decade. Given a real
portrait from a particular decade, it is first inverted into the latent space
of the corresponding model, and the resulting latent code can then be fed into
the model of any desired target decade. Our approach makes it unnecessary to
search for editing directions in the latent space (e.g., [SYTZ20, HHLP20]).

Next, to better fit the identity of the input individual, we apply single-
image finetuning of the family of per-decade StyleGAN generators (Section
Section 4.2). Specifically, we introduce Transferable Model Tuning (TMT), a
modified PTI (Pivotal Tuning Inversion) [RMBC21] procedure, to obtain an
adjustment for the weights of the source decade generator, and apply the
resulting adjustment to the target generator(s). This input-specific
adjustment is done in the generator’s parameter space, enabling us to better
preserve the input individual’s identity, while maintaining the style and
characteristics of the target decade. We now describe these two stages in more
detail.

Learning Decade Models

Single-image Refinement

Figure 2: Overview of our method. Left: We first train a family of StyleGAN
models, one for each decade, using adversarial losses and an identity loss on
a blended face, which resembles the parent model in its colors. Right:
Afterwards, each real image is projected onto a vector $w$ on the decade
manifold (1960 in the example above). We learn a refined generator
$G^{\prime}_{t}$ and transfer the learned offset to all models (this process
is visualized in Figure 4). To better encourage the refined model to preserve
facial details, we mask the input image and apply all losses in a weighted
manner (further described in the text).  Figure 3:  We finetune a family of
decade generators (child models) from an FFHQ-trained parent model. While each
generator captures unique styles, the generated images from the same latent
code are aligned in terms of high-level properties such as pose.

###  4.1 Learning coarsely-aligned decade models

We are interested in learning a family of StyleGAN2-ADA [KAH*20] generators
$\\{G_{t}\\},\;t\in\\{1880,1890,\cdots,2010\\}$ each of which maps a latent
vector $w\in\mathcal{W}$ to an RGB image. For each decade, we finetune a
separate StyleGAN model with weights initialized from an FFHQ-pretrained
model. We call the FFHQ-pretrained model the _parent model_ $G_{p}$, and the
finetuned network for decade $t$ the _child model_ $G_{t}$. Consistent with
the findings in prior work [WNSL21], we observe that the collection of
generators $\\{G_{t}\\},\;t\in\\{1880,\cdots,2010\\}$ exhibits semantic
alignment of faces generated from the same latent code $w$: they share similar
face poses and shapes. However, various fine facial characteristics such as
eyes and noses, which are important for recognizing a person, often drift from
one another (as evident in Figures 3 and 7).

To better preserve identity across decades when finetuning each child model,
to the standard StyleGAN objective $\mathcal{L}_{\text{GAN}}$ we add an
identity loss. Specifically, we measure the cosine similarity between ArcFace
[DGXZ19] embeddings of images generated by $G_{p}$ and by $G_{t}$:

$\mathcal{L}_{\text{ID}}=1-\texttt{cos\\_sim}(\text{ArcFace}(G_{p}(w)),\text{ArcFace}(G^{B}_{t}(w))).$
(1)

A similar loss is used in [RAP*21]. However, since the ArcFace model was only
trained on modern day images, we found that this raw identity loss performed
poorly on historical images, due to the domain gap. To solve this issue, we
use a blended version $G^{B}_{t}(w)$ instead of the original $G_{t}(w)$. We
create $G^{B}_{t}(w)$ using layer swapping [PA20] to mix $G_{p}(w)$ and
$G_{t}(w)$ at different spatial resolutions: we combine the coarse layers of
the child model $G_{t}(w)$ with the fine layers of the parent model
$G_{p}(w)$. By doing so, we “condition” our input image for the identity loss
by making its colors more similar to the image generated by the parent, and
thus more similar to the distribution of the images used to train the ArcFace
model. Figure 2 (left) shows that the blended (middle) image retains the
structure of the 1900s image, but its colors better resemble those of a modern
day photo. In addition, this technique restricts the identity loss to focus on
layers which generally control head shape, position, and identity. Note that
this blended image is only used to compute the loss, and not in the
transformed results.

2 (1920) 1 (2010)

Input |  $\theta_{1920}$ |  $\theta_{1920}+\Delta\theta_{i}$ |  $\theta_{1970}$ |  $\theta_{1970}+\Delta\theta_{i}$ |  $\theta_{2010}$ |  $\theta_{2010}+\Delta\theta_{i}$
---|---|---|---|---|---|---

Figure 4: Visualization of TMT offsets. We obtain offset vectors
$\Delta\theta_{1}$ (for the image in row 1) and $\Delta\theta_{2}$ (row 2) for
the weights of the source decade generator and apply it to every target decade
generator. On the left we use PCA to visualize the convolutional parameters
for all target generators in 2D. Each dot represents the weights of a single
generator, colored according to decade, and with edges connecting adjacent
decades. We illustrate the offset vectors optimized for the two input images
(colored in gray and red) and the corresponding transformed images for three
different decades. For each decade we show images before (left) and after
(right) applying TMT. Adding these offsets has the effect of improving
identity preservation.

###  4.2 Single-image refinement and transferable model tuning

As previously described, we first train the family of aligned StyleGAN models
with randomly sampled latent codes from $\mathcal{W}$ and our collections of
real per-decade images. Given these coarsely-aligned per-decade models, we are
given a single real face image as input and aim to generate a set of faces
across various decades. In order to better preserve the identity of the input
image across these decades, we introduce Transferable Model Tuning (TMT). TMT
is inspired by PTI [RMBC21], which is a procedure for optimizing a model’s
parameters to better fit to an input image after GAN inversion. TMT extends
PTI from a single generator model to a family of models. Our TMT procedure
produces a set of face images where the identity is preserved in the presence
of changing style over time (Figure 4).

Specifically, given an input image $x$ from decade $t$, we first obtain its
latent code $w\in\mathcal{W}$ using the projection method from StyleGAN2:
$w=\operatorname*{arg\,min}_{w}||x-G_{t}(w;\theta_{t})||$, where
$\theta_{t}\in\Theta$ is the vector of all parameters in $G_{t}$. We only work
with child models in this stage. As the first step of TMT, we fix the obtained
latent code $w$ and optimize over $\theta_{t}$ to obtain a new model
$G^{\prime}_{t}$ with parameters $\theta^{\prime}_{t}$ (Figure 2, right).

$\theta^{\prime}_{t}=\operatorname*{arg\,min}_{\theta_{t}}||x-G_{t}(w;\theta_{t})||.$
(2)

Tuning in the parameter space $\Theta$ of generators, instead of only working
in the latent space $\mathcal{W}$ as in previous work [NBLC20], allows us to
better fit the facial details of the input individual, such as eyes and
expression. Treating $\theta_{t}$ and $\theta^{\prime}_{t}$ as vectors in the
parameter space $\Theta$, this tuning can be thought of as applying an offset
$\Delta\theta=\theta^{\prime}_{t}-\theta_{t}$ to the original model.

We found that this $\Delta\theta$ offset is surprisingly transferable from one
TMT-tuned generator to all other decade generators in the parameter space
$\Theta$. Concretely, to obtain the style-transformed face of $x$ in _any_
target decade $d$, we simply apply:

$x_{d}=G_{d}(w;\theta_{d}+\Delta\theta),$ (3)

where $G_{d}$ is the generator for decade $d$ and $\theta_{d}$ is the vector
of all its parameters. We visualize the learned TMT offsets for two different
input images in Figure 4. As illustrated in the figure, these offsets greatly
improve the identity preservation of synthesized portraits.

Intuitively, $\Delta\theta$ “refines” the parameters of the generator family
to the single input face image to reconstruct better facial details. We
hypothesize that the found $\Delta\theta$ offsets mainly focus on improving
identities. Since the coarsely-aligned family of generators share similar
weights that are responsible for identity-related features, those offsets are
easily transferable. While most prior works, e.g., [SYTZ20, HHLP20, WLS21]
modify images using linear directions in various latent spaces of StyleGAN, we
are the first to apply a linear offset in the generator parameter space, to a
collection of generators. We hope our work will inspire future investigations
on understanding the linear properties in GAN’s parameter spaces. An analysis
of the effects of applying TMT to a family of models can be found in the
supplementary material.

As demonstrated in Figure 2 (right), to focus the loss computation on facial
details instead of hair and background, we apply masks to images before
calculating the losses. We use a DeepLab segmentation network [CZP*18] trained
on CelebAMask-HQ photos [OSF*20, LLWL20]. Empirically we determine it is best
to apply a weight of 1.0 on the face, 0.1 on the hair and 0.0 elsewhere. We
put a small weight on the hair to accurately reconstruct it, as it does
contribute to the image’s stylization. However, we do not want to prioritize
it over facial features. In addition, we find that it is best to keep
StyleGAN’s ToRGB layers frozen. Otherwise, color artifacts are introduced into
the generators. We follow the objectives introduced in [RMBC21] and minimize a
perceptual loss ($\mathcal{L}_{\text{LPIPS}}$), and a reconstruction loss
($\mathcal{L}_{\text{L2}}$). In addition, we add another identity loss
($\mathcal{L}_{\text{id}}$) to further enhance identity preservation for the
generated images.

Using our two-stage approach, for each portrait, we obtain a set of faces that
maintain the identity as well as demonstrate diverse styles across decades.

##  5 Results and Evaluation

We conduct extensive experiments on the _Faces Through Time_ test set. We
compare our approach to several state-of-the-art techniques across a variety
of metrics that quantify how well transformed images depict their associated
target decades and to what extent the identity is preserved. We also present
an ablation study to examine the impact of the different components of our
approach. Additional uncurated results and visualizations of the full test set
(1,400 individuals) are in the supplementary material.

###  5.1 State-of-the-art Image Editing Alternatives

As no prior works directly address our task, we adapt commonly used image
editing models to our setting and perform comprehensive comparisons.

Image-to-image translation. Unpaired image-to-image translation models learn a
mapping between two or more domains. We train a StarGAN v2 [CUYH20] model on
our dataset, where decades are domains. Because the quantitative metrics
between our model and StarGAN are so similar, we show a detailed qualitative
comparison between our model and StarGAN in the supplementary material, over a
set of individuals balanced by ethnicity and gender. We see that StarGAN has a
poor understanding of skin tone and overall identity, which is critical for
our real-world applications. Results on another model, DRIT++ [LTM*20], are in
the supplementary material.

Attribute-based editing. We consider each decade as a facial attribute and
compare against recent works performing attribute-based edits. While many
attributes in prior work are binary [SYTZ20, HHLP20], our decade attribute has
multiple classes. By comparison, age-based transformation is more similar to
our problem setting, as age is often broken up into $K$ bins [OSF*20]. We
compare against SAM [APC21], a recent age-based transformation technique that
also operates in the StyleGAN space.

Language-guided editing. Weakly supervised methods that leverage powerful
image-text representations (_e.g._ CLIP [RKH*21]) have demonstrated impressive
performance in portraits editing. We compare against: (i) StyleCLIP [PWS*21],
which learns latent directions in StyleGAN’s $\mathcal{W+}$ space for a given
text prompt, and (ii) StyleGAN-nada (StyleNADA) [GPM*21], which modifies the
weights of the StyleGAN generator based on textual inputs. For both models, we
used the text prompt “A person from the [XYZ]0s”, where [XYZ]0 is one of FTT’s
14 decades (1880-2010). For StyleCLIP, we use models trained on both FFHQ and
FTT. Because StyleGAN-nada is designed for out-of-domain changes, we
experiment with how well it can modify the generator from the FFHQ space to
various decades in our dataset. We use 100 FFHQ photos. We compare these two
baselines to how well our model can transform FFHQ images to each of the 14
decades.

Conditioning on a single model vs. multiple models. We train one model for
each decade as done in StyleAlign [WNSL21] because it is difficult to
disentangle ecah decade’s style in a single model. The modifications we made
to SAM [APC21], such as adding a decade classifier, can be considered a
straightforward way to condition decade labels on a single model’s latent
space. In addition, StyleCLIP [PWS*21] also uses a single finetuned model to
transform images within a GAN’s latent space. Our results show that these
single-model methods struggle to understand the style of each decade, as
compared to our multiple-model approach.

Time Travel Rephotography. Similar to our method, Time Travel Rephotography
[LZY*21] is designed to imagine what a historical person would have looked
like in another time period (and can only perform a transformation to modern
imagery). However, the authors focus primarily on image restoration as opposed
to changing a person’s style or fashion. We see that Time Travel Rephotography
improves camera quality and lighting instead of changing hairstyles and other
stylistic features as our model does. Technically, [LZY*21] inverts their
photos into a pretrained StyleGAN on FFHQ instead of training new models on
data for each decade. We provide visual comparisons between our model and Time
Travel Rephotography in the supplementary material.

| Method | FID$\downarrow$ | KMMD $\downarrow$ |  $\text{DCA}_{0}$$\uparrow$ |  $\text{DCA}_{1}$$\uparrow$ |  $\text{DCA}_{2}$$\uparrow$ |  $\text{ID}_{acc}$$\uparrow$
---|---|---|---|---|---|---|---
FFHQ | StyleCLIP | 254.39 | 1.87 | 0.08 | 0.18 | 0.36 | 0.99
StyleNADA | 312.06 | 2.03 | 0.10 | 0.30 | 0.38 | 0.96
Ours | 69.46 | 0.43 | 0.50 | 0.81 | 0.91 | 0.93
FTT | StarGAN v2 | 68.05 | 0.40 | 0.38 | 0.75 | 0.89 | 0.97
SAM | 96.52 | 0.72 | 0.51* |  0.85* | 0.89* | 1.00
StyleCLIP | 108.25 | 0.85 | 0.07 | 0.21 | 0.36 | 1.00
Ours | 66.98 | 0.40 | 0.47 | 0.78 | 0.90 | 0.99

Table 1: Quantitative Evaluation. We compare performance against SOTA
techniques on FFHQ (top three rows) and on our test set (bottom four rows).
Our method outperforms others in terms of most metrics. *Note that SAM uses
the decade classifier during training and therefore the DCA metric is skewed
in this case, as we further detail in the text.

StarGAN |  SAM |  StyleCLIP |  Ours
---|---|---|---

StarGAN |  SAM |  StyleCLIP |  Ours
---|---|---|---

StarGAN |  SAM |  StyleCLIP |  Ours
---|---|---|---

StarGAN |  SAM |  StyleCLIP |  Ours
---|---|---|---

Input |  1880s |  1920s |  1940s |  1960s |  1980s |  2010s
---|---|---|---|---|---|---

Input |  1910s |  1930s |  1950s |  1970s |  1990s |  2010s
---|---|---|---|---|---|---

Figure 5: Qualitative Results. Above we compare results generated by baselines
and our technique. The red box indicates the inversion of the original input.
We observe that our approach allows for significant changes across time while
best preserving the input identity. While SAM and StarGAN are able to stylize
images, these changes are mostly limited to color. StyleCLIP struggles to
generate meaningful changes. Please refer to the supplementary material for
qualitative results on the full test set.

###  5.2 Metrics

Visual quality. We use the standard FID [HRU*17, Sei20] metric as well as the
Kernel Mean Maximum Discrepancy Distance (KMMD) [WGB*20] metric. As image
quality varies across decades, we compute scores between real and edited
portraits separately for each decade and then average over all decades.
Because FID can be unstable on smaller datasets [CF19], similar to prior work
[NH19, WGB*20], we measure KMMD [WGB*20] on Inception [SVI*16] features.
Experimentally, we find that these two scores are highly sensitive to an
image’s background. Therefore, we compute the scores on images of size
$256\times 256$ cropped to 160$\times$160 pixels.

Decade style. We evaluate how well the generated samples capture the style of
the target decade using a EfficientNetB0 classifier [TL19] that we trained
separately. Using the classifier, we define the Decade Classification Accuracy
(DCA). We follow prior works [GJ09] and report three metrics:
$\text{DCA}_{0}$, $\text{DCA}_{1}$ and $\text{DCA}_{2}$, where
$\text{DCA}_{p}$ measure the accuracy within a tolerance of $p$ decades.

Identity preservation. We use the Amazon Rekognition service to measure how
well a person’s identity has been preserved in generated portraits. Their
CompareFaces operation outputs a similarity score between two faces. As a
metric, we report $\text{ID}_{acc}$ – the fraction of successful identity
comparisons. We consider a comparison to be successful if its similarity score
is above a certain threshold (set empirically to $1.0$).

Input |  1900s |  1920s |  1940s |  1960s |  1980s |  2010s
---|---|---|---|---|---|---

Figure 6: StyleGAN-nada Results. Although StyleGAN-nada [GPM*21] produces some
style changes across decades, all images are transformed similarly. For
example, all 1980s portraits adopt a frizzy hairstyle.

###  5.3 Results

We present test set performance for all methods in Table 1 and qualitative
results in Figure 5. Results on the _full_ test set (1400 samples) are
provided using an interactive viewer as part of our supplementary material. We
present additional results on StyleGAN-nada’s effect on FFHQ input images in
Figure 6. We see that our method performs well numerically in terms of all
metrics. Although StarGAN also performs well in FID and KMMD, the style
changes by StarGAN are mostly about color; there are few changes to makeup,
hair, and beards, whereas our model can perform such changes. Our method also
has fewer artifacts. As illustrated in Figure 6, modifications from StyleGAN-
nada are more caricature-like than realistic. As a result, StyleGAN-nada
performs poorly with respect to FID, KMMD, and DCA metrics. For StyleCLIP and
SAM, the identity preservation is near 1.0 because their changes are generally
limited to color. Nonetheless, our model still successfully matches input and
transformed images in most cases (for $93\%$ and $99\%$ of samples, for FFHQ
and FTT images, respectively), while generating significant changes in styles.
While SAM performs the best with respect to DCA, we believe this is advantaged
because SAM used the same classifier during training as a loss function. In
fact, in Figure 5, SAM demonstrates little change across decades. We suspect
that the classifier is leading SAM to overfit to noise, instead of truly
changing an image’s style.

From our results we can discern interesting details that provide insight into
style trends across time. For example, as illustrated in he top left example
in Figure 5, we see that the individual adopts a bob haircut, one popularized
by Irene Castle, and strongly associated with the flapper culture of the
Roaring Twenties. Later on, we notice more contemporary hair styles. Finally,
in the 2010s, we see that women tend to adopt longer hairstyles. Despite these
generalizations, the portraits remain well-conditioned on the input,
reflecting an individual’s identity and aspects of their personal style. For
instance, the individual on the left with the long mustache maintains facial
hair across the decade transformations, although in very different styles. In
the bottom left example, we observe glasses that change style over time. Not
only does our model generate realistic transformations, but also captures the
nostalgia of various time periods.

No $\mathcal{L}_{\text{id}}^{\text{(I)}}$ |  No LS |  No TMT |  No $\mathcal{L}_{\text{id}}^{\text{(II)}}$ |  No Mask |  Ours
---|---|---|---|---|---

Input |  1890s |  1910s |  1930s |  1950s |  1970s
---|---|---|---|---|---

Figure 7: Ablations. We see significant improvement in terms of identity
preservation after adding the identity loss and TMT during training. The
masking procedure alleviates artifacts caused by other regions in the image
(such as the hat in the example above) by focusing the model’s attention on
facial details and allowing for larger modifications in other regions. See
Table 2 for descriptions of the ablation labels.

###  5.4 Ablations

We present ablations on components in our approach in Table 2 and Figure 7.
Specifically, we train five ablated models: (1) without the identity loss (for
learning decade models), (2) without the blended image obtained using layer
swapping, (3) without TMT, (4) without the identity loss (during TMT), and (5)
without masking the images. The first ablation is akin to finding the nearest
neighbor of an image in the StyleGAN $\mathcal{W}$ space using latent
projection. As shown in the first row of the table, our baseline method
already captures a decade’s style well. However, the images are not aligned
with respect to an input’s identity, which is reflected in its low
$\text{ID}_{acc}$ score. As a result, $\mathcal{L}_{\text{id}}^{\text{(I)}}$,
$\mathcal{L}_{\text{id}}^{\text{(II)}}$, and TMT are necessary for identity
preservation. In addition, we find that using masks during TMT reduces
artifacts in generated images as it allows for an accurate inversion in the
facial region and larger modifications in other regions in the image.
Empirically, we notice that this reduces noise and improves decade
classification. While FID, KMMD and DCA scores remain similar across the
ablations, our full model shows strong improvement in $\text{ID}_{acc}$, which
is the main objective of $\mathcal{L}_{\text{id}}^{\text{(I)}}$ and our
proposed TMT stage. We also experimentally find that $\mathcal{W+}$ spaces
[RAP*21] are less well aligned than the $\mathcal{W}$ space. More details are
in the supplementary material.

$\mathcal{L}_{\text{id}}^{\text{(I)}}$ | LS | TMT | $\mathcal{L}_{\text{id}}^{\text{(II)}}$ | Mask | FID | KMMD | $\text{DCA}_{0}$ | $\text{DCA}_{1}$ | $\text{DCA}_{2}$ | $\text{ID}_{\mathit{acc}}$
---|---|---|---|---|---|---|---|---|---|---
$\times$ | $\times$ | $\times$ | $\times$ | $\times$ | 69.51 | 0.45 | 0.49 | 0.79 | 0.92 | 0.61
✓ | $\times$ | $\times$ | $\times$ | $\times$ | 69.36 | 0.47 | 0.51 | 0.82 | 0.92 | 0.63
✓ | ✓ | $\times$ | $\times$ | $\times$ | 68.18 | 0.45 | 0.50 | 0.82 | 0.92 | 0.72
✓ | ✓ | ✓ | $\times$ | $\times$ | 67.32 | 0.39 | 0.46 | 0.78 | 0.89 | 0.95
✓ | ✓ | ✓ | ✓ | $\times$ | 67.08 | 0.38 | 0.46 | 0.77 | 0.89 | 0.99
✓ | ✓ | ✓ | ✓ | ✓ | 66.98 | 0.40 | 0.47 | 0.78 | 0.90 | 0.99

Table 2: Ablation study evaluating the effect of the identity loss while
learning decade models ($\mathcal{L}_{\text{id}}^{\text{(I)}}$) and during TMT
($\mathcal{L}_{\text{id}}^{\text{(II)}}$), using a blended image with layer
swapping (LS), TMT, and masking the images during TMT (Mask).

##  6 Ethical Discussion

Face datasets—and the tasks that they enable, such as face recognition—have
been subject to increasing scrutiny and recent work has shed light on the
potential harms of such data and tasks [DRD*20, GNH19]. With awareness of
these issues, our dataset was constructed with attention to ethical questions.
The images in our dataset are freely-licensed and provided through a public
catalog. We will include the source and license for each image in our dataset.
As part of our terms of use, we will only provide our dataset for academic use
under a restrictive license. Furthermore, our dataset does not contain
identity information (and only includes one face per identity), and therefore
cannot readily be used for facial recognition. Nonetheless, our dataset does
inherit biases that are present in Wikimedia Commons. For instance, the data
is gender imbalanced, containing a ratio of roughly $2:1$ male to female
samples (according to the binary gender labels available on Wikimedia Commons,
which are annotated by Wikipedia contributors). While such biases can be
mitigated by balancing the data for training and evaluation purposes, we plan
to continue gathering more diverse data to address this underlying bias in the
data. For additional details on various features of our dataset, please refer
to the accompanying datasheet [GMV*21].

There are also ethical considerations relating to the risks of using portrait
editing for misinformation. Our task is perhaps less sensitive in this regard,
since our explicit goal is to create fanciful imagery that is clearly
anachronistic. That said, any results from such technology should be clearly
labeled as imagery that has been modified.

##  7 Conclusion

We present a dataset and method for transforming portrait images across time.
Our _Faces Through Time_ dataset spans diverse geographical areas, age groups,
and styles, allowing one to capture the essence of each decade via generative
models. By learning a family of generators and efficient tuning offsets, our
two-stage approach allows for significant style changes in portraits, while
still preserving the appearance of the input identity. Our evaluation shows
that our approach outperforms state-of-the-art face editing methods. It also
reveals interesting style trends existing in various decades. However, our
method still has limitations. As with any data-driven technique, our results
are affected by biases that exist in the data. For instance, females with
short hair are less common at the beginning of the 20th century, which may
yield gender inconsistencies when transforming a short-haired modern female
face to these early decades, including unexpected changes in visual features
often associated with gender. In the future, we plan to explore methods that
can improve consistency, perhaps by devising a way to jointly optimize models
for different decades that better enforces consistency among them. Finally, we
envision that future uses of our data could go beyond the synthesis tasks we
consider in our work, and explore the combination of both analysis and
synthesis.

##  8 Acknowledgements

This work was supported in part by the National Science Foundation
(IIS-2008313).

## References

  * [ALS*16] Brandon Amos, Bartosz Ludwiczuk and Mahadev Satyanarayanan  “OpenFace: A general-purpose face recognition library with mobile applications”  In _CMU School of CS_ , 2016
  * [Ant19] Jason Antic  “A deep learning based project for colorizing and restoring old images (and video!)”, 2019  URL: https://github.com/jantic/DeOldify
  * [APC21] Yuval Alaluf, Or Patashnik and Daniel Cohen-Or  “Only a Matter of Style: Age Transformation Using a Style-Based Regression Model”  In _ACM Trans. Graph._ 40.4 Association for Computing Machinery, 2021
  * [AQW19] Rameen Abdal, Yipeng Qin and Peter Wonka  “Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?”  In _Int. Conf. Comput. Vis._ , 2019, pp. 4431–4440
  * [AZMW21] Rameen Abdal, Peihao Zhu, Niloy J. Mitra and Peter Wonka  “StyleFlow: Attribute-Conditioned Exploration of StyleGAN-Generated Images Using Conditional Continuous Normalizing Flows”  In _ACM Trans. Graph._ 40.3 New York, NY, USA: Association for Computing Machinery, 2021  DOI: 10.1145/3447648
  * [BK73] Coen Bron and Joep Kerbosch  “Algorithm 457: finding all cliques of an undirected graph”  In _Communications of the ACM_ 16.9 ACM, 1973, pp. 575–577
  * [BV99] Volker Blanz and Thomas Vetter  “A morphable model for the synthesis of 3D faces”  In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_ , 1999, pp. 187–194
  * [CBPS20] Edo Collins, Raja Bala, Bob Price and Sabine Süsstrunk  “Editing in Style: Uncovering the Local Semantics of GANs”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020
  * [CF19] Min Jin Chong and David Forsyth  “Effectively Unbiased FID and Inception Score and where to find them”  In _arXiv preprint arXiv:1911.07023_ , 2019
  * [CKA*21] Yuqing Cui et al.  “Who’s Waldo? Linking People Across Text and Images”  In _Int. Conf. Comput. Vis._ , 2021, pp. 1374–1384
  * [CUYH20] Yunjey Choi, Youngjung Uh, Jaejun Yoo and Jung-Woo Ha  “StarGAN v2: Diverse Image Synthesis for Multiple Domains”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 8185–8194
  * [CZP*18] Liang-Chieh Chen et al.  “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation”  In _Eur. Conf. Comput. Vis._ , 2018
  * [DDS*09] Jia Deng et al.  “ImageNet: A large-scale hierarchical image database.”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2009, pp. 248–255
  * [DGXZ19] Jiankang Deng, Jia Guo, Niannan Xue and Stefanos Zafeiriou  “Arcface: Additive angular margin loss for deep face recognition”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 4690–4699
  * [DRD*20] Pawel Drozdowski et al.  “Demographic Bias in Biometrics: A Survey on an Emerging Challenge”  In _IEEE Transactions on Technology and Society_ 1.2 Institute of ElectricalElectronics Engineers (IEEE), 2020, pp. 89–103  DOI: 10.1109/tts.2020.2992344
  * [EST*20] Bernhard Egger et al.  “3d morphable face models—past, present, and future”  In _ACM Trans. Graph._ 39.5 ACM New York, NY, USA, 2020, pp. 1–38
  * [GJ09] Lisa Gaudette and Nathalie Japkowicz  “Evaluation methods for ordinal classification”  In _Canadian Conference on Artificial Intelligence_ , 2009, pp. 207–210  Springer
  * [GMV*21] Timnit Gebru et al.  “Datasheets for datasets”  In _Communications of the ACM_ 64.12 ACM New York, NY, USA, 2021, pp. 86–92
  * [GNH19] Patrick Grother, Mei Ngan and Kayee Hanaoka  “Face recognition vendor test (fvrt): Part 3, demographic effects”  National Institute of StandardsTechnology, 2019
  * [GPM*21] Rinon Gal et al.  “StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators”, 2021  arXiv:2108.00946 [cs.CV]
  * [GRS*15] Shiry Ginosar et al.  “A century of portraits: A visual historical record of american high school yearbooks”  In _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2015, pp. 1–7
  * [HG21] Wei-Lin Hsiao and Kristen Grauman  “From Culture to Clothing: Discovering the World Events Behind A Century of Fashion Images”  In _Int. Conf. Comput. Vis._ , 2021
  * [HHLP20] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen and Sylvain Paris  “Ganspace: Discovering interpretable gan controls”  In _arXiv preprint arXiv:2004.02546_ , 2020
  * [HRU*17] Martin Heusel et al.  “GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium”  In _Adv. Neural Inform. Process. Syst._ , 2017
  * [JL17] Huaizu Jiang and Erik Learned-Miller  “Face detection with the faster R-CNN”  In _Int. Conf. on Automatic Face & Gesture Recognition_, 2017
  * [KAH*20] Tero Karras et al.  “Training Generative Adversarial Networks with Limited Data”  In _Adv. Neural Inform. Process. Syst._ , 2020
  * [KAL*21] Tero Karras et al.  “Alias-Free Generative Adversarial Networks”  In _Adv. Neural Inform. Process. Syst._ , 2021
  * [Kar72] Richard M Karp  “Reducibility among combinatorial problems”  In _Complexity of computer computations_ Springer, 1972, pp. 85–103
  * [Kem16] Ira Kemelmacher-Shlizerman  “Transfiguring Portraits”  In _ACM Trans. Graph._ , 2016
  * [KLA*20] Tero Karras et al.  “Analyzing and improving the image quality of stylegan”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 8110–8119
  * [KLA19] Tero Karras, Samuli Laine and Timo Aila  “A style-based generator architecture for generative adversarial networks”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 4401–4410
  * [KNT17] Marek Kowalski, Jacek Naruniec and Tomasz Trzcinski  “Deep alignment network: A convolutional neural network for robust face alignment”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2017
  * [LEH13] Yong Jae Lee, Alexei A. Efros and Martial Hebert  “Style-Aware Mid-level Representation for Discovering Visual Connections in Space and Time”  In _Int. Conf. Comput. Vis._ , 2013, pp. 1857–1864
  * [LGZ*20] Andrew Liu et al.  “Learning to Factorize and Relight a City”  In _Eur. Conf. Comput. Vis._ , 2020
  * [LLWL20] Cheng-Han Lee, Ziwei Liu, Lingyun Wu and Ping Luo  “MaskGAN: Towards Diverse and Interactive Facial Image Manipulation”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020
  * [LMB*14] Tsung-Yi Lin et al.  “Microsoft COCO: Common Objects in Context”, 2014  URL: http://arxiv.org/abs/1405.0312
  * [LMC*15] Stefan Lee et al.  “Linking past to present: Discovering style in two centuries of architecture”  In _IEEE International Conference on Computational Photography (ICCP)_ , 2015
  * [LTM*20] Hsin-Ying Lee et al.  “DRIT++: Diverse Image-to-Image Translation via Disentangled Representations”  In _Int. J. Comput. Vis._ , 2020, pp. 1–16
  * [LZY*21] Xuan Luo et al.  “Time-Travel Rephotography”  In _ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2021)_ 40.6 ACM New York, NY, USA, 2021  DOI: https://doi.org/10.1145/3478513.3480485
  * [MBS17] Kevin Matzen, Kavita Bala and Noah Snavely  “StreetStyle: Exploring world-wide clothing styles from millions of photos”  In _arXiv preprint arXiv:1706.01869_ , 2017
  * [MMH*19] Utkarsh Mall et al.  “GeoStyle: Discovering fashion trends and events”  In _Int. Conf. Comput. Vis._ , 2019
  * [MS14] Kevin Matzen and Noah Snavely  “Scene Chronology”  In _Eur. Conf. Comput. Vis._ , 2014
  * [MSE17] Eric Müller-Budack, Matthias Springstein and Ralph Ewerth  ““When Was This Picture Taken?” – Image Date Estimation in the Wild”  In _Advances in Information Retrieval_ , 2017, pp. 619–625
  * [NBLC20] Yotam Nitzan, A. Bermano, Yangyan Li and D. Cohen-Or  “Face identity disentanglement via latent space mapping”  In _ACM Trans. Graph._ 39, 2020, pp. 1–14
  * [NH19] Atsuhiro Noguchi and Tatsuya Harada  “Image generation from small datasets via batch statistics adaptation”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2019, pp. 2750–2758
  * [NK17] Aaron Nech and Ira Kemelmacher-Shlizerman  “Level Playing Field For Million Scale Face Recognition”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2017
  * [OSF*20] Roy Or-El et al.  “Lifespan age transformation synthesis”  In _Eur. Conf. Comput. Vis._ , 2020, pp. 739–755  Springer
  * [PA20] Justin N.. Pinkney and Doron Adler  “Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains”  In _ArXiv_ abs/2010.05334, 2020
  * [PWS*21] Or Patashnik et al.  “StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery”  In _Int. Conf. Comput. Vis._ , 2021, pp. 2085–2094
  * [RAP*21] Elad Richardson et al.  “Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2021
  * [RCR18] Nataniel Ruiz, Eunji Chong and James M. Rehg  “Fine-Grained Head Pose Estimation Without Keypoints”  In _IEEE Conf. Comput. Vis. Pattern Recog. Worksh._ , 2018
  * [RHGS15] Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun  “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”  In _Adv. Neural Inform. Process. Syst._ , 2015
  * [RKH*21] Alec Radford et al.  “Learning Transferable Visual Models From Natural Language Supervision”  In _ICML_ , 2021
  * [RMBC21] Daniel Roich, Ron Mokady, Amit H Bermano and Daniel Cohen-Or  “Pivotal Tuning for Latent-based Editing of Real Images”  In _arXiv preprint arXiv:2106.05744_ , 2021
  * [RR17] N. Ruiz and J.. Rehg  “Dockerface: an easy to install and use Faster R-CNN face detector in a Docker container”  In _ArXiv e-prints_ , 2017  arXiv:1708.04370 [cs.CV]
  * [SAL*20] Lorenzo Stacchio et al.  “IMAGO: A family photo album dataset for a socio-historical analysis of the twentieth century”  In _CoRR_ abs/2012.01955, 2020  arXiv: https://arxiv.org/abs/2012.01955
  * [Sei20] Maximilian Seitzer  “pytorch-fid: FID Score for PyTorch” Version 0.2.1, https://github.com/mseitzer/pytorch-fid, 2020
  * [SGTZ20] Yujun Shen, Jinjin Gu, Xiaoou Tang and Bolei Zhou  “Interpreting the latent space of gans for semantic face editing”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 9243–9252
  * [SVI*16] Christian Szegedy et al.  “Rethinking the Inception Architecture for Computer Vision”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2016, pp. 2818–2826
  * [SWZJ16] Tawfiq Salem, Scott Workman, Menghua Zhai and Nathan Jacobs  “Analyzing Human Appearance as a Cue for Dating Images”  In _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2016, pp. 1–8
  * [SYTZ20] Yujun Shen, Ceyuan Yang, Xiaoou Tang and Bolei Zhou  “Interfacegan: Interpreting the disentangled face representation learned by gans”  In _IEEE Trans. Pattern Anal. Mach. Intell._ IEEE, 2020
  * [SZ21] Yujun Shen and Bolei Zhou  “Closed-form factorization of latent semantics in gans”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2021, pp. 1532–1540
  * [TAN*21] Omer Tov et al.  “Designing an Encoder for StyleGAN Image Manipulation”  In _arXiv preprint arXiv:2102.02766_ , 2021
  * [TL19] Mingxing Tan and Quoc V. Le  “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”  In _ArXiv_ abs/1905.11946, 2019
  * [WGB*20] Yaxing Wang et al.  “Minegan: effective knowledge transfer from gans to target domains with few images”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2020, pp. 9332–9341
  * [WLS21] Zongze Wu, D. Lischinski and Eli Shechtman  “StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2021, pp. 12858–12867
  * [WNSL21] Zongze Wu, Yotam Nitzan, Eli Shechtman and D. Lischinski  “StyleAlign: Analysis and Applications of Aligned StyleGAN Models”  In _ArXiv_ abs/2110.11323, 2021
  * [WT20] Jonas Wulff and Antonio Torralba  “Improving Inversion and Generation Diversity in StyleGAN using a Gaussianized Latent Space”, 2020  arXiv:2009.06529 [cs.CV]
  * [XZY*21] Weihao Xia et al.  “GAN Inversion: A Survey”, 2021  arXiv:2101.05278 [cs.CV]
  * [YLLT16] Shuo Yang, Ping Luo, Chen Change Loy and Xiaoou Tang  “WIDER FACE: A Face Detection Benchmark”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2016
  * [ZIE*18] Richard Zhang et al.  “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric”  In _IEEE Conf. Comput. Vis. Pattern Recog._ , 2018
  * [ZZI*17] Richard Zhang et al.  “Real-Time User-Guided Image Colorization with Learned Deep Priors”  In _ACM Trans. Graph._ 9.4 ACM, 2017

Appendix

In this document, we present implementation details and additional results
that are supplement to the main paper. Section A reports details about our
_Faces Through Time_ dataset. Section B provides details about the adaptation
of state-of-the-art image editing alternatives as well as additional results
and analysis. Section C provides implementation details such as training
procedures and hardware specifications. A datasheet for our dataset is
provided separately. Note that we also provide an interactive viewer that
demonstrates our results, as well as those obtained using alternative
techniques, on the full test set (as well as a lighter viewer that only
displays results for 100 random test samples).

##  Appendix A _Faces Through Time_ Dataset Details

_Faces Through Time_ contains in total 26,247 portraits over 12 decades from
the 1880s to 2010s. In Table 3, we report the number of distinct identities
per decade. Since our dataset contains only one image per identity, these
numbers also correspond to the number of images per decade. We include a
histogram over the image resolutions for all decades in Figure 8. From
1880–2000, our images have an average resolution of $527\times 527$. From
2000–2020, they have an average resolution of $1403\times 1403$.

By associating the identity from each portrait with the biographic information
on Wikimedia Commons, we show demographic information of _Faces Through Time_
dataset in a set of histograms. Figure 9 shows the 50 most common citizenships
and occupations. For a systematic review on the characteristics of the _Faces
Through Time_ dataset, please refer to the datasheet.

|  | 1880 | 1890 | 1900 | 1910 | 1920 | 1930 | 1940 | 1950 | 1960 | 1970 | 1980 | 1990 | 2000 | 2010
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Identities |  | 525 | 842 | 2049 | 3052 | 2042 | 1710 | 1337 | 1638 | 2611 | 1916 | 1382 | 928 | $3116^{\star}$ | $3099^{\star}$
Table 3: Number of identities per decade in the _Faces Through Time_ dataset.
The symbol ⋆ denotes that these sets were trimmed. Note that our dataset
contains one image per identity, and therefore, these numbers also correspond
to the number of images per decade.

Resolution

1880 – 2000

Resolution

2000 – 2020

Figure 8: Resolution of Aligned Facial Images . A histogram containing each
image’s resolution (measured in pixels) is illustrated above. For images from
1900 – 2000, we only use images of resolution greater than or equal to
$200\times 200$. Due to the abundance of digital images after 2000, we only
use images of resolution greater than or equal to $1000\times 1000$ in the
2000s and 2010s. Note that these resolutions correspond to the images after
alignment and cropping is applied to extract normalized facial regions, and
not to the original images found in Wikimedia Commons.

Occupations

Citizenships

Figure 9: 50 Most Common Occupations and Citizenships. Identities may have
more than one or no associated occupations and citizenships. There are 981
occupations not shown, each with 146 or fewer associated identities.
Similarly, there are 209 citizenships not shown, each with 49 or fewer
associated identities. Citizenship generally refer to historical nations. For
example, individuals from what would be considered modern day China are
categorized into the Qing dynasty, the Republic of China (1912-1949), the
People’s Republic of China, etc.

###  A.1 Assembly: Face Clustering

We elaborate on our face clustering method (see Sec. 3 in the main paper)
here. Although one could naively associate all faces in an identity’s photos
to that identity, a given photo may have multiple faces or may not picture the
target identity at all. We, therefore, approach face assignment as a
clustering problem, where we separately group faces corresponding to each
person $i$. Let $F_{i}$ be the full set of faces extracted from images in
$i$’s category. We wish to cluster $F_{i}$ into groups of people and identify
the actual identity’s group. We propose a hybrid semi-supervised clustering
method for this problem, based on the following observations:

  1. 1.

The number of people represented in a set is unconstrained, so it is difficult
to partition sets with parametric techniques, such as k-means, which assume a
known number of clusters.

  2. 2.

A cluster is most likely to represent a single person if all face pairs are
similar.

  3. 3.

If multiple faces appear in the same image, it is very unlikely that these
faces belong to the same identity. As a corollary, given two images, each face
in one of the images should only belong to the same identity as at most one
face in the other image.

Maximal Clique Clustering. We construct a graph $\mathcal{G}$ and formulate
this as a graph clustering problem. Each face in $F_{i}$ corresponds to a node
$p$ in $\mathcal{G}$. For each face, we also compute its FaceNet embedding
$\mathbf{v}_{p}\in\mathbb{R}^{128}$ using OpenFace [ALS*16]. By observation
(2), our goal is then to find subsets of faces $C\subseteq\mathcal{G}$ such
that every pair $p,q\in C$ is a positive verification pair
($\left\lVert\mathbf{v}_{p}-\mathbf{v}_{q}\right\rVert\leq\epsilon$ for some
threshold $\epsilon$). Hence, we add an edge $(p,q)$ to $\mathcal{G}$ for such
pairs.

However, we can further constrain this construction with observation (3). Let
$I$ and $J$ be a pair of images, containing faces denoted by the sets $I_{F}$
and $J_{F}$. We observe that the corresponding subgraph of $\mathcal{G}$
induced by $I_{F}\cup J_{F}$ must be bipartite, and that at most one edge
should have an endpoint in each node to represent correct identity relations.
For each pair of images, we construct edges for each pair of nodes $(p\in
I_{F},q\in J_{F})$ in increasing order by
$d_{pq}=\left\lVert\mathbf{v}_{p}-\mathbf{v}_{q}\right\rVert$ if
$d_{pq}\leq\epsilon$ and no edge adjacent to $p$ or $q$ already exists. Prior
work shows that setting $\epsilon\approx 1$ is effective in practice [ALS*16],
which we corroborated in our evaluation.

We would then like to search this graph for _cliques_ , subgraphs with an edge
between every pair of nodes (specifically, maximal cliques, which cannot be
further extended). We apply the Bron-Kerbosch [BK73] algorithm to (relatively)
efficiently enumerate all maximal cliques (this is an NP-complete problem
[Kar72]). We sorted the resulting set of cliques in decreasing order by size,
successively selecting those without nodes in any previously selected clique,
until none remained. The selected cliques are our final clusters of $F_{i}$.

We further purified clusters by removing faces with an outlier threshold
$\alpha$. Similar to the approach in MF2 [NK17], we first create a vector $v$
whose elements are the mean pairwise distance for each face in the cluster. We
compute the median absolute deviation
$\textrm{MAD}(v)=\textrm{Median}\left(|v-\textrm{Median}(v)|\right)$. Each
face is an outlier if its corresponding $v_{i}$ satisfies
$|v_{i}-\textrm{Median}(v)|\mathbin{/}\textrm{MAD}(v)>\alpha$ for
$\alpha=3.0$.

Since nearly all identities are annotated with a number of ground-truth
reference images (i.e., their prominent Wikipedia article image or images
listed in WikiData), we can simply assign the cluster containing the largest
number of these reference images to that identity. If no references are
available, we instead assume the largest cluster of faces is the person in
question. By visual inspection of 200 identities, we found over 98% accuracy
between clusters and their automatically assigned identity labels. We
primarily attribute this to having reference images for over 86% of
identities.

##  Appendix B Method Details and Additional Results

In this section, we include details and results that are omitted in the main
paper due to the page limit. Note that these are supplementary to the main
results, not to be viewed as significant new results.

###  B.1 Detailed adaptation of state-of-the-art methods

StarGAN v2 [CUYH20]. StarGAN is well-suited for our decade translation task
because it scales to multiple domains. We ran the training code found in their
official code repository. We used the CelebA-HQ [LLWL20] configuration.

DRIT++ [LTM*20]. DRIT++ is a state-of-the-art image-to-image translation
framework known for generating diverse representations. We train a model using
the code in their official code repository. In practice, DRIT++ scales poorly
to our problem because the model can only be trained on two domains at a time.
91 models are needed to create transformations across all 14 decades. To
visualize results, we ran the model on several pairs of decades. Evaluation
shows that the model suffers from poor image quality compared to other
baselines, as illustrated in Figure 16.

SAM [APC21]. SAM has shown impressive realism with regard to age
transformation. We ran the training code found in their official code
repository. We train SAM on top of a StyleGAN model trained on images from all
decades in our dataset. During training, SAM searches for decade
transformation directions within the model’s $\mathcal{W}+$ space. To modify
SAM, we replace the age regression network with a decade classification
network.

StyleCLIP [PWS*21]. Guided by CLIP [RKH*21], StyleCLIP uses a text prompt to
transform an image in StyleGAN’s $\mathcal{W+}$ space. Similarly to SAM, we
run StyleCLIP on a StyleGAN model trained on all of our dataset’s images.
Although StyleCLIP presents three different training schemes, we used the
latent mapper approach since the authors claim that it is best suited for
complex attributes. We ran the training code found in their official code
repository. For evaluation, we set the target prompt as “A person from the
[decade]s" where decade is an element of $\\{1880,1890,\dots,2010\\}$.

StyleGAN-nada [GPM*21]. Because StyleGAN-nada is designed for out of domain
changes, we started with images from FFHQ and projected them to decades in the
20th century. We used the same text prompt that we used for StyleCLIP. We ran
the training code found in their official code repository. We also
experimented with using exemplar images, and found that the generated images
suffer from a lack of style diversity and are entangled with the identity of
the exemplars, meaning that synthesized images adopted facial characteristics
of the exemplar images.

###  B.2 Decade-classification performance

In our evaluation, we use an EfficientNetB0 classification network trained on
the _Faces Through Time_ dataset to calculate the DCA scores. For reference, a
confusion matrix on the test set of this classifier is in Figure 10. The
classifier has an average accuracy of $45.57\%$ over all decades. Furthermore,
$79.21\%$ of the confusion is captured within a tolerance of $\pm 1$ decade.

Figure 10: Test classification accuracy on the _Faces Through Time_ test set.
Rows indicate the ground truth decade and columns indicate the predicted
decade.

###  B.3 Additional comparisons

Comparisons between our model and state-of-the-art alternatives on the full
test set of _Faces Through Time_ are provided separately using an interactive
viewer. Consistent with the results presented in the main paper, our method
outperforms alternatives in terms of image quality and style changes, while
preserving the identity of the input images.

We highlight differences between our model and StarGAN [CUYH20] in Figure 11.
We present results on CelebAHQ [LLWL20], a dataset of recognizable
celebrities, in Figure 12.We also show comparisons with Time Travel
Rephotography in Figure 13, and the issues we experience with quality using
Yearbook dataset in Figure 14.

Input |  1880 |  1900 |  1920 |  1940 |  1960 |  1980 |  2000
---|---|---|---|---|---|---|---

Input |  1880 |  1900 |  1920 |  1940 |  1960 |  1980 |  2000
---|---|---|---|---|---|---|---

Figure 11: Comparison with StarGAN. We highlight differences between our
method (first row) and StarGAN [CUYH20] (second row) on a selection of
individuals balanced by gender and ethnicity. We show that our method
accentuates differences in style. While StarGAN is able to generate plausible
transformations, it has a poor understanding of skin tone and overall
identity, which is especially critical for real-world applications.

Input |  1880s |  1890s |  1900s |  1910s |  1920s |  1930s |  1940s |  1950s |  1960s |  1970s |  1980s |  1990s |  2000s |  2010s
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---

Figure 12: Results on CelebAHQ. Above we show results on celebrities from
CelebAHQ. We see that the celebrities remain recognizable throughout the
transformations.

Input |  Time Travel Rephotography [LZY*21] |  Ours
---|---|---

Figure 13: Comparison with Time Travel Rephotography. We show our method on
images taken from the supplementary material of Time Travel Rephotography
[LZY*21]. For our method, we transform each face to the 2010s. As illustrated
above, our method modifies the style to simulate what these individuals would
have looked like had they lived today (and also allow for other
transformations through time), whereas [LZY*21] focuses primarily on image
restoration.

Yearbook [GRS*15]

_Faces Through Time_

Input |  1900s |  1920s |  1940s |  1960s |  1980s |  2000s
---|---|---|---|---|---|---

Figure 14: Results on the Yearbook dataset. Given the input images from the
left, which are sourced from the Yearbook dataset [GRS*15], we compare the
quality of transformations between a model trained on the Yearbook dataset and
models trained on _Faces Through Time_. Because the Yearbook dataset photos
are lower resolution, we decided to not include them in the _Faces Through
Time_ dataset.

###  B.4 $\mathcal{W}$ space vs. $\mathcal{W+}$ space

As mentioned in the main paper, our method uses a $\mathcal{W}$ projection to
invert an image into the latent space of a StyleGAN model. Figure 15 shows a
comparison to the alternative $\mathcal{W+}$ space. We find that inverting
images into the $\mathcal{W+}$ space creates more artifacts during training,
which are often amplified after TMT.

$\mathcal{W}$ Inversion

Before TMT |  After TMT
---|---

$\mathcal{W+}$ Inversion

Before TMT |  After TMT
---|---

$\mathcal{W}$ Inversion

Before TMT |  After TMT
---|---

$\mathcal{W+}$ Inversion

Input |  1910s |  1920s |  1940s |  1960s |  1970s |  1980s |  2010s
---|---|---|---|---|---|---|---

Before TMT |  After TMT
---|---

Figure 15: Face inversion and transformation results using the $\mathcal{W}$
vs. $\mathcal{W+}$ space. Above we compare before TMT and after TMT results
obtained using the $\mathcal{W}$ space, which we adopt in our work, with
results obtained using the $\mathcal{W+}$ space. As demonstrated above,
results with the $\mathcal{W+}$ space yield various artifacts, which are often
amplified after TMT.

###  B.5 Additional ablation results

Figure 17 shows additional examples of ablations on essential components in
our approach. Consistent with the results in the main paper, our full model
with all components enabled has the best performance compared to other
variants.

###  B.6 Analysis of TMT offsets

Figure 18 shows the effects of applying the proposed TMT offsets to portraits.
In general, we find that the TMT offsets are distinguishable from directions
between arbitrary pairs of decade generators. This agrees with our intuition
that offsets learned by fine-tuning a generator on an identity should be
independent from the style changes between decades.

Input (1900s) |  1900s |  2010s
---|---|---

DRIT++ |  Ours
---|---

Input (1920s) |  1920s |  1960s
---|---|---

Input (2010s) |  1900s |  2010s
---|---|---

DRIT++ |  Ours
---|---

Input (1900s) |  1900s |  1920s
---|---|---

Figure 16: DRIT++ Results. We show additional qualitative results obtained
using DRIT++ and our method. Compared to our method, DRIT has trouble
reconstructing high quality images. In addition, most of the changes from DRIT
are limited to color.

No $\mathcal{L}_{\text{id}}^{\text{(I)}}$ |  No LS |  No TMT |  No $\mathcal{L}_{\text{id}}^{\text{(II)}}$ |  No Mask |  Ours
---|---|---|---|---|---

No $\mathcal{L}_{\text{id}}^{\text{(I)}}$ |  No LS |  No TMT |  No $\mathcal{L}_{\text{id}}^{\text{(II)}}$ |  No Mask |  Ours
---|---|---|---|---|---

Input |  1880s |  1920s |  1940s |  1960s |  2000s
---|---|---|---|---|---

Input |  1890s |  1910s |  1930s |  1950s |  1970s
---|---|---|---|---|---

Figure 17: Additional Ablation Results. These results further show the
improvement obtained in terms of identity preservation after adding
$\mathcal{L}_{\text{id}}^{\text{(I)}}$, layer swapping, and TMT during
training; and the benefit of incorporating
$\mathcal{L}_{\text{id}}^{\text{(II)}}$ and the masking procedure.

##  Appendix C Training and implementation details

Learning Decade Models. All models were trained for 645k iterations on a
single Nvidia RTX 3090. The codebase is derived from the official
stylegan2-ada-pytorch repository. For training, we use the paper256 config. We
use the PyTorch implementation of InsightFace with a ResNet-100 backbone for
the identity loss. The identity loss was added to the $G_{\texttt{main}}$
phase of StyleGAN training with a weight of $1.0$. We used a regularization
weight of $\gamma=0.5$, which we empirically found was best for the dataset.

Single-image Refinement. We modified the Pivotal Tuning Inversion (PTI) code
base for our task. We added a DeepLab [CZP*18] mask to the input images during
the generator tuning phase of PTI. From experimentation, we found that the L2
loss had little effect on the quality of images. Most of the inversion tuning
is guided by the LPIPS [ZIE*18] loss. Because of this, we set a small LPIPS
threshold of $0.03$ during training. Inference takes 2-3 minutes per image.

|
---|---
Input (1980s) |  Input (1940s)

|  |  |
---|---|---|---
$G_{1900}(w_{1};\theta_{1900})$ |  $G_{1900}(w_{1};\theta_{1900}+\Delta\theta_{1})$ |  $G_{1900}(w_{2};\theta_{1900})$ |  $G_{1900}(w_{2};\theta_{1900}+\Delta\theta_{2})$
|  |  |
$G_{1980}(w_{1};\theta_{1980})$ |  $G_{1980}(w_{1};\theta_{1980}+\Delta\theta_{1})$ |  $G_{1940}(w_{2};\theta_{1940})$ |  $G_{1940}(w_{2};\theta_{1940}+\Delta\theta_{2})$
|  |  |
$G_{2000}(w_{1};\theta_{2000})$ |  $G_{2000}(w_{1};\theta_{2000}+\Delta\theta_{1})$ |  $G_{2000}(w_{2};\theta_{2000})$ |  $G_{2000}(w_{2};\theta_{2000}+\Delta\theta_{2})$

Figure 18: For the two examples shown, we compare the cosine similarity
between the $\Delta\theta$ vectors learned by TMT (colored in orange above)
and the vectors learned by decade transformations $\theta_{t}-\theta_{i}$
(colored in green above). We notice that the cosine similarity is centered
around zero. This implies that in StyleGAN’s parameter space, the TMT offset
has no correlation with the decade transformation direction. To contrast, when
comparing two decade offsets $\theta_{t_{1}}-\theta_{i}$ and
$\theta_{t_{2}}-\theta_{i}$ we see that the vectors have high similarity. This
agrees with our qualitative results, where we observe that TMT offset improves
the identity preservation in each decade independently, without sacrificing
the style of the target decade.