# Towards compact high-efficiency grating couplers for visible wavelength
photonics

Joe A. Smith  Quantum Engineering Technology Labs and Department of Electrical
and Electronic Engineering, University of Bristol, Bristol BS8 1UB, UK
Corresponding author<EMAIL_ADDRESS>Jorge Monroy-Ruz  Quantum
Engineering Technology Labs and Department of Electrical and Electronic
Engineering, University of Bristol, Bristol BS8 1UB, UK  Pisu Jiang  Quantum
Engineering Technology Labs and Department of Electrical and Electronic
Engineering, University of Bristol, Bristol BS8 1UB, UK  John G. Rarity
Quantum Engineering Technology Labs and Department of Electrical and
Electronic Engineering, University of Bristol, Bristol BS8 1UB, UK  Krishna C.
Balram  Quantum Engineering Technology Labs and Department of Electrical and
Electronic Engineering, University of Bristol, Bristol BS8 1UB, UK
<EMAIL_ADDRESS>
###### Abstract

While grating couplers have become the de-facto standard for optical access to
integrated silicon photonics platforms, their performance at visible
wavelengths, in moderate index contrast platforms like silicon nitride, leaves
significant room for improvement. In particular, the index contrast governs
the diffraction efficiency per grating tooth and the resulting overall coupler
length. In this work, we develop two approaches to address this problem: a
dielectric grating that sums multiple optical modes to increase the overall
output intensity, and an embedded metal grating that enhances the attainable
refractive index contrast, and therefore reduces the on-chip footprint. We
present experimental results that can be developed to realise compact
efficient visible wavelength photonic interconnects, with a view towards
cryogenic deployment for quantum photonics, where space is constrained and
efficiency is critical.

††journal: ol

The majority of atom (Rb - 780 nm) and atom-like (spin defects in diamond and
silicon carbide) systems of interest for large-scale quantum technologies have
their transition frequencies in the visible to near-infrared (IR) frequency
range, above silicon’s bandgap [1, 2]. As they start to scale in complexity,
developing robust methods to interface these quantum systems with light become
critical to achieving system-level efficiency. Silicon nitride (SiN), a
material with an established foundry infrastructure and well-developed high
performance optical devices, is a natural candidate for building integrated
quantum photonic devices around solid state defects. While SiN photonics is
well-developed, from the perspective of quantum photonics, there are some key
challenges that still need to be addressed that arise due to the low index
contrast available in an SiN platform.

These problems are best illustrated by considering a grating coupler, a
standard element in silicon photonics, but one whose performance in SiN leaves
a lot to be desired. The low index contrast limits the achievable scattering
efficiency per grating tooth and therefore requires long on-chip couplers that
occupy a significant on-chip footprint [3] and a limited diffraction
efficiency, as obtained from standard foundry processes [4].

Efficient grating couplers have significant advantages for quantum photonics,
with regards to interfacing emitters located at the chip centre and avoiding
routing (and excess loss) to the chip edge. Especially in cryogenic
environments, as will be the case for most quantum photonics experiments,
there are significant space constraints, which are relieved by being able to
interface efficiently in the surface normal to the chip.

Photonics also offers non-dissipative, thermally isolated wiring for classical
interconnects [5, 6]. Electrical wiring is a bottleneck in cryogenic computing
platforms as each wire contributes a thermal budget that offsets the available
cooling power. This will undoubtedly rear its head for large scale quantum
information processing [7] interfaced through thousands of wires to higher
temperature electronics [8]. Considering classical communication at visible
light wavelengths ($\approx$ 637 nm), we can gain a 6$x$ reduction in grating
footprint area compared to 1550 nm, whilst still harnessing established
telecommunication protocols [9, 10]. The mode-matching adibatic taper length
would also decrease by over 2$x$ [11]. In systems with spatial and mechanical
constraints, such as in cryostats, reducing the required fraction of chip area
would free up valuable space for on-chip processing.

With this in mind, in this work, we develop two different coupling approaches.
First we develop a compact free space outcoupler that can funnel emission into
multiple modes to be collected by a high-NA objective, with a view towards
improving the out-coupling efficiency of an emitter located in an SiN
waveguide. In parallel, we use an embedded metal layer to increase the index
contrast (and scattering efficiency) in the SiN platform. We aim to design
sub-100 ${\mu}m^{2}$ footprint components with at least -10 dB coupling power,
considering typical quantum emission efficiencies [12], and to match
comparable classical insertion losses[9, 13].

Figure 1: (a) The far-field scattered power shows 42 % of emission is accepted
within NA = 0.7 (shaded in gray). (b) The 2D plot of power leaving the device,
overlaid on the geometry, shows the multimode grating distributes into four
angular lobes. (c-f) As grating pitch increases, modes are excited leading to
an increased collection $T_{\uparrow}$, maximised at duty $D\sim 0.75$, shown
with purple dot. (g) In/out coupling and waveguiding of 637 nm laser light
demonstrated through brightness of out-grating (right side). Scattered light
off the in-grating (left side) saturates camera. Inset: microscope image of
fabricated grating.

Compared to the moderately efficient small diffractive couplers reported in
suspended gallium arsenide membranes [12, 14], SiN on silica has two
significant issues. Firstly, SiN has a considerably lower refractive index.
Secondly, the silica bottom cladding means that light entering or exiting the
grating will preferentially enter the substrate compared to a suspended
platform, which is air clad on both sides. In addition, when imaging
diffractive couplers in mechanically constrained systems, a long working
distance objective reduces the available Numerical Aperture (NA). The 6.0 mm
MY100X-806 Mitutoyo reduces the available NA = 0.7 or an acceptance angle of
45 degrees. Owing to this, light will not be collected at large diffraction
angles. Here we optimise a diffraction-limited grating coupler design within
these limitations, using a 270 nm nitrogen-rich SiN film ($n$ = 1.89 at 637
nm) [15], with this thickness chosen to maximise refractive index contrast
whilst maintaining single-mode operation.

Under these constraints, an optimal device operating at 637 nm is found by
varying the design pitch and duty cycle of the grating $(\Lambda,D)$ and
maximising the far-field projection of the emitted power (Poynting vector)
that falls within the desired NA, $T_{NA}$, using a particle swarm algorithm
in a numerical finite difference time domain solver (Lumerical FDTD) [16]. The
algorithm converges after 50 generations with the power collected in the top
plane $T_{\uparrow}\sim 0.5$, with $T_{NA}$ = 0.48 for a design accepting
angles up to 60∘ (0.9 NA) or $T_{NA}$ = 0.42 accepting angles up to 45∘ (0.7
NA). The angular out-coupled power distribution of the 0.7 NA design is
depicted in Fig 1(a). Here the design parameters are $(\Lambda,D)$ = (1.41
$\mu$m, 0.77). It can be seen in this plot that there are four discrete lobes
of emission and most of the emission is confined within the 0.7 NA acceptance
angle of the objective (shaded in grey). These distinct lobes arise from the
Bragg condition of the grating and match four emission regions in the
horizontal cross-section in Fig 1(b).

Considering that an air-suspended fully-etched structure would have
$T_{\uparrow}$ = 0.5 by symmetry, our design is near optimal as the silica
substrate reduces the top plane fraction. The resolution required is 300 nm,
within reach of standard lithography in a foundry-compatible platform. We note
related work by Zhu _et al._ on SiN grating couplers, to maximise 633 nm
coupling into an NA = 0.65 objective [17], however this relies on suspended
SiN without a connecting substrate. Unlike this study, we vary the duty cycle
$D$. The clear advantage of this is shown in Fig 1(c-f), as the collected
power saturates for $D<0.75$ as the pitch is increased. Further increase of
$D$ above this has little value.

This device was fabricated using single step e-beam lithography with CSAR-62
resist to obtain a high ICP etch selectivity for the 270 nm SiN film [18]. A
representative image of a fabricated device is shown inset in Fig 1(g). To
characterise the grating, an input laser controlled by a steering mirror (PI -
S334), translated through a $4f$ pair of lenses, sets the angle of entry into
the objective and hence the excitation position at the sample. The sample is
mounted on a stepper motor (PI MiCos) inside a cryostat. A beamsplitter on a
flip mount allows white light illumination to align the laser on the grating.
The collection is transmitted through a 90:10 beamsplitter and focussed onto a
camera. Fig 1(g) is taken with the white light image overlaid with the laser-
only image showing alignment to the grating device. It evidences guiding of
light through the waveguide structure with the right side coupler displaying a
central spot with multiple crescent lobes of emission, as expected from the
design. There is a large component of light scattered through the substrate
from the in-coupler, shown in the broad extent of the input spot, which
contributes to poor signal-to-noise. It was not possible to discriminate the
out-coupled multi-modal signal above the surrounding noise floor, precluding
quantitative analysis of this design. To reduce background scatter, a larger
trench could be etched to separate the two grating couplers from the
surrounding. However, there is a fundamental issue with this multi-mode
grating, which is the cause of the scatter at the input beam. Primarily,
although the device is a good out-coupler into several modes, by definition it
is a poor in-coupler into a single mode waveguide from a focused Gaussian
beam.

In this design, we have increased the amount of light coupled out of the
device by collecting power summed from several different spatial modes, at a
loss of modal purity, which also restricts the coupling efficiency obtainable
into a single mode fibre. Owing to this, there is a limited overlap between a
single Gaussian mode and the spatial profile of the light that couples out of
the device, resulting in large scattering. At most a third of the emission
would couple into a fibre, considering the intensity in a single lobe in Fig
1(a). This will be compounded in a chip with many grating couplers as the
multi-directional emission and scattering will manifest as cross-talk between
devices. Instead of considering only out-coupling, a single-moded design would
equally be suitable for in-coupling. In order to achieve this, we shall
establish an alternative type of grating by considering bi-directional
information transfer.

Figure 2: (a) Transmission in the top ($T_{\uparrow}$) and bottom plane
($T_{\downarrow}$) as the grating height is varied for four symmetry-broken
gratings. In the legend diagrams, SiN is shown in blue and gold in orange on a
grey glass substrate. (b) FDTD simulation showing, as opposed to full and half
etched dielectric gratings, the gold design results in a high proportion of
emission leaving the device within the 630 nm MFD = 4.5 $\mu$m. (c) FDTD
simulation showing input from simulated SM 630 fibre, with the majority of the
input power coupled into the waveguide.

To realise efficient single-moded gratings that remain micron-sized, we must
consider degrees of freedom that would increase the refractive index contrast.
This is to ensure that most of the light leaves in the first few grating teeth
of the device. As well as the need for a small grating length, by symmetry,
emission in the top plane can not be greater than emission in the bottom plane
in fully etched designs, even for an infinite grating length. One way to
achieve a high-contrast broken symmetry design is to use a metallic layer in
combination with a dielectric. Gold is highly reflective, with a low real
component and high imaginary refractive index (here modelled as $n$ = 0.12 and
$k$ = 3.33 at $\lambda$ = 637 nm [19]). We can break the symmetry of the
device by patterning a gold grating before the SiN is deposited. This grating
strongly perturbs the effective index of the SiN waveguide. Evaluating the
effective index of 70 nm gold under 270 nm SiN as $n_{e}$ = 2.15, we see a 300
% increase in index contrast compared to a half-etched grating design ($n_{e}$
= 1.58:1.74).

In Fig 2(a), sweeping the thickness of the gold film, we find a maximal
contrast between the transmission in the top plane ($T_{\uparrow}$) and the
transmission in the substrate ($T_{\downarrow}$) as
$T_{\uparrow}/T_{\downarrow}=4$ with a 70 nm-thick film. Simulations indicate
that optimal contrast is obtained when this gold layer is deposited below the
SiN (dash dotted line), rather than on top (dashed line), which instead
increases emission into the substrate. We believe this is due to the large out
of phase component of the metal which acts as a mirror, reflecting light where
it would otherwise be transmitted. In the simulation, we model perturbations
in the top of the grating due to the conformal nature of the SiN film
deposition (solid line). These have negligible effect, thought due to the
negligible change (<0.05%) in the effective index of the thicker SiN
structure. The taller grating structure has a slight symmetry breaking effect
(dotted line), similar to partial etches, and has been demonstrated before
[20].

Simulating the 70 nm buried gold device, we find that a pitch $\Lambda$ = 320
nm maximises outcoupling $T_{\uparrow}>0.5$. In contrast to the four lobes in
the previous design, this emission leaves the grating design in a nearly mono-
directional 20∘ from the substrate. The transition to this single angle
emission also acts as an effective in-coupler of single mode Gaussian beams
with matched spot sizes, by time reversal symmetry.

As this pitch requires tighter resolution tolerances, we opt for a linear
grating for ease of process optimisation. We target a 5-$\mu$m-wide linear
grating: this allows for smaller footprints than standard 12 $\mu$m
telecommunications gratings, whilst coupling to a broad range of free space
mode beams. The length of the grating is similar in dimension, with a low
number of grating teeth. With this width, we find in Lumerical MODE that a
linear taper can efficiently convert the waveguide mode with $<$ 1 dB loss in
15 $\mu$m compared to $\sim 100\mu$m required for silicon photonics at 1550
nm. Additionally, the device is wide enough to be tested with SM 630 fibre
with a Mode Field Diameter (MFD) of 4.5 $\mu$m to decouple the free space
optics performance from the devices under test. In Fig 2(b), the gold grating
emits over 75 % of its emission within this MFD (dashed gold line).

An alternative method to break symmetry is to employ a partial etch into the
SiN layer. However, this results in a lower refractive index contrast to the
fully etched grating. Therefore, a long grating is required for power in the
waveguide mode to leave the device. As Fig 2(b) shows, a partially etched
grating must be 6 $\mu$m long in order to emit 75 % of its total efficiency (
shown by dashed line).

Further to this, an attractive strategy for most use cases is to modulate the
duty cycle of the grating to tailor emission to fit a Gaussian output mode
[21, 3]. Such phase-profiled apodised gratings in SiN are considerably larger
in size as the initial duty cycle reduces the emitted power per grating tooth.
In Fig 2(b), we see that either a reduced or increased $D$ results in a
significant reduction in the power emitted over the same length, compared to
the $D$ = 0.5 grating. Therefore, duty cycle modulation is not beneficial for
the device efficiency here, due to the limited number of grating teeth within
the desired design length.

Following these results, we simulate the SM 630 fibre facet in FDTD, angled at
14 ∘ in order to inject light into the device refracted through the air
interface at 20 ∘. In Fig 2(c), we can see that a high proportion of the power
(> 0.5) is transmitted into the waveguide through the grating. The grating
metal absorbs 8 % of the incident power. With a cryostat cooling power of 1 W
at 4K, this absorption sets a limit of $10^{5}$ devices for classical
communication, assuming 100 $\mu$W of light per channel [13].

Figure 3: (a) Schematic of gold grating. (b) One half of fabricated grating
pair and connecting silicon nitride waveguide section. Inset: SEM of the 320
nm pitch gold grating. (c) Measured fibre-to-fibre device transmission has a
$S_{21}$ = - 21 dB with a 3 dB bandwidth of 20 nm. (d) Changing the input
polarisation with a half-waveplate, the device exhibits strong selectivity.

The grating coupler was fabricated through a lift-off process using PMMA. Gold
is deposited on the patterned substrate using thermal evaporation and is
lifted-off using a 10 second oxygen plasma de-scum and low power sonication in
acetone. We avoid a chromium adhesion layer to reduce absorption losses and
increase the scattering efficiency. Then, 270 nm of SiN is deposited on the
grating using plasma enhanced chemical vapor deposition [15]. The waveguide
connecting the gratings is formed as in the previous device, with an inter-
grating pitch of 250 um. The grating lines are extended on either side of the
taper section such that the etch stage defining the grating can be overlaid
with a $\pm$ 1 ${\mu}m$ tolerance. A schematic of the design is given in Fig
3(a). Fig 3(b) shows an image of the gold lift-off lines overlaid by the SiN
waveguide.

To test the device, we mount a V-groove fibre array containing 630 PM fibre
(OZ Optics) with a polished facet at 14 ∘ on a 6-axis fibre probing setup
(Maple Leaf Photonics). The gratings are separated by 250 $\mu$m to match the
V-groove spacing by a bend and linear waveguide section shown in Fig 3(b). One
channel is used to excite the grating and the other to readout the power from
the adjacent grating using a Thorlabs PM100 power meter. Light from a
supercontinuum source (NKT SuperK EVO EU-4) is controlled using an acousto-
optic tunable filter (NKT SuperK SELECT) into the excitation channel. In Fig
3(b), at its design wavelength of 637 nm, the best-performing device is
observed to have an $S_{21}$ of -21 dB from fibre to fibre, indicating around
-10 dB efficiency per coupler assuming lossless waveguiding. However, we
observe some scattering of light along the waveguide in the camera indicating
some waveguide loss. It is known that roughness arising from liftoff has a
significant effect in metallic nanostructures and can be replaced by gold
etching in future devices [22]. Residual roughness and e-beam overexposure
also results in a higher effective $D$. We estimate the measured device has a
$D$ = 0.7 and simulate this corresponds to a reduction in efficiency by almost
half to $T=0.28$. Film thickness variation is also significant with 10 nm
error resulting in a further 20 % loss. We would like to note here that
although our measured devices are lower than the predicted efficiency, their
performance is comparable to commercial SiN gratings at 1550 nm [4].

The grating was designed for 637 nm, but our measurement peak is centered at
635 nm. The fibre array set angle is precise to 1∘ and this is likely a cause
of this discrepancy. The measured device is highly broadband with a 3 dB
bandwidth of 20 nm and so operates well at the design wavelength. In Fig 3
(c), a half-waveplate (HWP) was inserted before the first grating to control
the polarisation of the input light. From this measurement, coupling displays
strong polarisation selectivity, as expected from a linear grating. This could
have useful applications for reducing crosstalk by interleaving perpendicular
devices.

We note that embedded metal gratings have been considered before. Wang _et
al._ simulated a gold grating below SiN although they did not consider the
gold thickness [23]. Lamy _et al._ fabricated a gold grating for 1550 nm in
the centre of titanium dioxide, where gold thickness is considered [24]. Our
demonstration experimentally realises a metal grating for visible wavelengths
and specifically makes the connection that high-index contrast gratings are
necessary for small single-moded designs.

Moving forward, the designs presented here could have useful applications in
high-density addressing of quantum emitters [15, 3]. SiN photonics can be
deposited directly on metal when grown using low temperature PECVD, as
presented here. Mature processes of this kind include the foundry platform
developed by IMEC for integration with CMOS imagers [25]. Further work could
look at a hybrid device to combine both structures into a single-mode in-
coupler and multi-mode out-coupler centred around an integrated quantum
emitter. We could extend our designs towards shorter wavelengths to
potentially interface with trapped ions [10]. Arrays should then be fabricated
to measure channel crosstalk, as a step towards high-density photonic
interconnects.

##  1 Backmatter

Funding European Research Council (ERC-StG, SBS 3-5, 758843); Consejo Nacional
de Ciencia y Tecnología (CONACyT); Engineering and Physical Sciences Research
Council (EP/M024458/1, EP/N015126/1, EP/L015730/1); British Council
(352345416).

Acknowledgments We thank A. Murray, M. Cryan and D. Sahin for valuable
suggestions. JAS and JGR were supported by British Council IL6 (352345416) and
JGR’s EPSRC Fellowship EP/M024458/1. JAS and JMR were supported by the EPSRC
Quantum Engineering CDT EP/L015730/1. JMR was supported by Consejo Nacional de
Ciencia y Tecnologia (CONACyT). KCB acknowledges support from the ERC (ERC-
StG, SBS 3-5, 758843). Device fabrication was performed using equipment
acquired through EPSRC QuPIC EP/N015126/1.

Data availability Data presented in this manuscript is available from the
authors on request.

Disclosures The authors declare no conflicts of interest.

## References

  * [1] T. Tiecke, J. D. Thompson, N. P. de Leon, L. Liu, V. Vuletić, and M. D. Lukin, Nature 508, 241 (2014).
  * [2] M. Atatüre, D. Englund, N. Vamivakas, S.-Y. Lee, and J. Wrachtrup, Nature Reviews Materials 3, 38 (2018).
  * [3] K. K. Mehta and R. J. Ram, Scientific reports 7, 2019 (2017).
  * [4] Cornerstone, “Cornerstone SiN process development kit,” .
  * [5] F. Lecocq, F. Quinlan, K. Cicak, J. Aumentado, S. Diddams, and J. Teufel, Nature 591, 575 (2021).
  * [6] D. A. Miller, Journal of Lightwave Technology 35, 346 (2017).
  * [7] J. R. Scott and K. C. Balram, arXiv preprint arXiv:2109.04792 (2021).
  * [8] F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, R. Biswas, S. Boixo, F. G. Brandao, D. A. Buell _et al._ , Nature 574, 505 (2019).
  * [9] S. Rajbhandari, J. J. McKendry, J. Herrnsdorf, H. Chun, G. Faulkner, H. Haas, I. M. Watson, D. O’Brien, and M. D. Dawson, Semiconductor Science and Technology 32, 023001 (2017).
  * [10] A. Minotto, P. A. Haigh, Ł. G. Łukasiewicz, E. Lunedei, D. T. Gryko, I. Darwazeh, and F. Cacialli, Light: Science & Applications 9, 1 (2020).
  * [11] A. Milton and W. Burns, IEEE Journal of Quantum Electronics 13, 828 (1977).
  * [12] M. Arcari, I. Söllner, A. Javadi, S. L. Hansen, S. Mahmoodian, J. Liu, H. Thyrrestrup, E. H. Lee, J. D. Song, S. Stobbe _et al._ , Physical Review Letters 113, 093603 (2014).
  * [13] R. C. Pitwon, K. Wang, J. Graham-Jones, I. Papakonstantinou, H. Baghsiahi, B. J. Offrein, R. Dangel, D. Milward, and D. R. Selviah, Journal of Lightwave Technology 30, 3316 (2012).
  * [14] P. Jiang and K. C. Balram, Optics express 28, 12262 (2020).
  * [15] J. Smith, J. Monroy-Ruz, J. G. Rarity, and K. C. Balram, Applied Physics Letters 116, 134001 (2020).
  * [16] J. Robinson and Y. Rahmat-Samii, IEEE transactions on antennas and propagation 52, 397 (2004).
  * [17] Y. Zhu, J. Wang, W. Xie, B. Tian, Y. Li, E. Brainis, Y. Jiao, and D. Van Thourhout, Optics Express 25, 33297 (2017).
  * [18] S. Thoms and D. S. Macintyre, Journal of Vacuum Science & Technology B, Nanotechnology and Microelectronics: Materials, Processing, Measurement, and Phenomena 32, 06FJ01 (2014).
  * [19] P. B. Johnson and R.-W. Christy, Physical review B 6, 4370 (1972).
  * [20] R. Marchetti, C. Lacava, L. Carroll, K. Gradkowski, and P. Minzioni, Photonics Research 7, 201 (2019).
  * [21] R. Marchetti, C. Lacava, A. Khokhar, X. Chen, I. Cristiani, D. J. Richardson, G. T. Reed, P. Petropoulos, and P. Minzioni, Scientific reports 7, 16670 (2017).
  * [22] A. B. Greenwood, K. C. Balram, and H. Gersen, Nano Letters (2022).
  * [23] L. Wang, Y. Wang, and X. Zhang, Optics express 20, 17509 (2012).
  * [24] M. Lamy, K. Hammani, J. Arocas, C. Finot, and J.-C. Weeber, Optics letters 42, 2778 (2017).
  * [25] M. Malak, T. Claes, V. Garcia-Munoz, H. K. Tyagi, R. Van Hoof, G. Winderickx, S. Severi, W. Lee, D. Kim, S. Ahn _et al._ , 2018 IEEE Micro Electro Mechanical Systems (MEMS) pp. 735–738 (2018).


# Merkel Podcast Corpus: A Multimodal Dataset
Compiled from 16 Years of Angela Merkel’s Weekly Video Podcasts

###### Abstract

We introduce the Merkel Podcast Corpus, an audio-visual-text corpus in German
collected from 16 years of (almost) weekly Internet podcasts of former German
chancellor Angela Merkel. To the best of our knowledge, this is the first
single speaker corpus in the German language consisting of audio, visual and
text modalities of comparable size and temporal extent. We describe the
methods used with which we have collected and edited the data which involves
downloading the videos, transcripts and other metadata, forced alignment,
performing active speaker recognition and face detection to finally curate the
single speaker dataset consisting of utterances spoken by Angela Merkel. The
proposed pipeline is general and can be used to curate other datasets of
similar nature, such as talk show contents. Through various statistical
analyses and applications of the dataset in talking face generation and TTS,
we show the utility of the dataset. We argue that it is a valuable
contribution to the research community, in particular, due to its realistic
and challenging material at the boundary between prepared and spontaneous
speech.
Keywords: multi-modal, German, single-speaker, corpus, speaker diarization,
forced alignment, cross-modal learning

Figure 1: Angela Merkel looking in all directions during video podcasts.

languageresourceLanguage Resources

Merkel Podcast Corpus: A Multimodal Dataset
Compiled from 16 Years of Angela Merkel’s Weekly Video Podcasts

Debjoy Saha1,2, Shravan Nayak1,3, Timo Baumann1,4
---
1Universität Hamburg, 2IIT Kharagpur, 3IIT (BHU) Varanasi, 4OTH Regensburg
<EMAIL_ADDRESS><EMAIL_ADDRESS>timo.baumann@oth-
regensburg.de

Abstract content

## 1\. Introduction

Multi-modal spoken video corpora are becoming more widespread and relevant in
recent years. Given today’s compute power, multi-modal analyses as well as
cross-modal learning between video and audio – and particularly: between
facial mimicry and speech – have become more commonplace. For example, based
on weekly addresses of then-president of the United States, Barack Obama,
ObamaNet [Kumar et al., 2018] produces speech and lip-synchronized facial
videos from text and some intermediate representation.

There are a number of spoken video corpora available for end-to-end audio-
visual machine learning tasks. The LRS 2 Chung_2016 and LRS 3
afouras2018lrs3ted datasets contain thousands of sentence-like units from many
different speakers collected from the BBC and from TED talks, i. e. spoken _in
the wild_.

The broadness and wide coverage of such multi-speaker corpora come with some
limitations. In particular, speech samples are devoid of their original
context as short snippets are available only in isolation. Very little
material is available per-speaker and snippets tend to be short and typically
contain only single phrases. This prohibits modelling long-range phenomena
such as conversational prosody and the focus on short snippets makes it
impossible to model speaker behaviour during short speech pauses.

On the other hand, there is also already a number of single-speaker video
corpora collected from speeches of public figures, such as former US
presidents Obama janssoone:hal-02873020 and Trump ruf-
navarretta-2020-creating. These typically are purely single-speaker and do not
contain any material from other speakers. Also, prepared speeches do not
necessarily contain the full prosodic variability of spontaneous speech.

An important shortcoming of the existing corpora is the lack of multi-lingual
data from the same speakers. For example, to test automated approaches to lip-
synchronous dubbing [Chaume, 2012, Saboo and Baumann, 2019], one needs
training data in one language and data to be dubbed in another. Current
corpora do not provide speech from one speaker speaking multiple languages and
can typically not be extended to cover this: in the case of LRS corpora, the
speakers are not known and finding more material from these speakers is hence
tedious. In the case of corpora from public figures, they typically do not
speak in other languages than their mother tongue (which tends to be English).

Figure 2: Example of semi-automatic labeling for a target speaker (who in this
case is not speaking but listening to the interviewer) in the diarization
pipeline. We first segment the video into scenes. For all scenes, the faces
present are recognised, tracked and cropped into smaller videos. We rate the
faces in each scene using TalkNet (speaking?) and face similarity (target?)
scores. We consider the best candidate face for each scene and combine them to
the full face-cropped video. If no face is found valid for all scenes, the
snippet is discarded (as in this case).

We introduce the Merkel Podcast Corpus. This corpus (a) contains large amounts
of speech from one public figure (former German chancellor Angela Merkel) to
the extent that it can meaningfully be used to train modern single-speaker
deep learning models; (b) also contains further material from many other
speakers (interviewers) which can help generalize to multi-speaker models; (c)
contains material that is both closely aligned (short snippets of video with
the corresponding texts) and consecutive in nature, i. e., snippets can be
contextualized into the overall situation. The data spans 16 years and the
primary speaker’s voice ages correspondingly while recording technology
advances; (d) comes with large amounts of meta-data available, such as date of
recording, speaker names and further written documentation that the speech
material can be related to; (e) is amended with the few publicly available
video recordings of Angela Merkel speaking English, which is particularly
useful for cross-lingual processing tasks such as evaluating lip-synchronous
dubbing.

Our second contribution is a method for semi-automatically differentiating
speakers and tuning in on one target speaker in collections of multi-speaker
videos based on a few examples of the target speaker. The raw material that
our corpus is built from is mostly interview-style video in which the view
often shifts between interviewer and interviewee. Often these shifts do not
match speaker change. Our processing pipeline uses face detection and
recognition, as well as speaking-face detection in order to find video
snippets in which the target speaker is both visible and speaking. This
processing pipeline can be used to create single-speaker corpora from multi-
speaker sources (such as singling out the host or recurring guests from many
episodes of a talk show).

We describe our corpus extraction and preparation pipeline in Section 2 and
then describe the resulting dataset in Section 3. We believe the corpus can be
useful in a variety of ways and we describe some current usages in Section 4
and conclude in Section 5.

## 2\. Dataset Source and Preparation

Angela Merkel decided early on in her tenure to produce weekly video podcasts
to be published via Internet. More than 600 such podcasts have been published
between 2006 and 2021, yielding much more material and covering a longer time
than Obama or Trump. Although always high-quality productions, video quality
and resolution has substantially evolved over time. Since 2013, podcasts
contain lightly edited subtitles; older videos are amended at least with a
written transcript for accessibility. The format changed from semi-spontaneous
speeches to predominantly interviews in 2011-2018.

### 2.1. Scraping, Alignment and Snippeting

We scraped the official video platform to download all videos including
subtitles, transcripts and other alternative formats as available.111See
https://www.bundesregierung.de/breg-de/service/archiv/archiv-podacasts. After
extracting the text from transcript PDFs, we use robust forced alignment
[Baumann et al., 2019] to align text and speech.

Cross-modal learning applications typically require many short audio-video-
text snippets rather than few long files. We built flexible snippeting scripts
that consider linguistic units and use speech pauses as possible cuts and
merge together adjacent material up to a given threshold (to match GPU memory
constraints). It is possible to overlap snippets for data augmentation and the
minimal units (words, inter-pausal-units, sentences) can be set flexibly.
Alternatively, snippets can be created from subtitle files. (See Section 3 for
a comparison of alignments and subtitles.)

### 2.2. Speaker Diarization

Audio-visual corpora should contain video where the face of the target person
speaking is visible and she is actually speaking. In the raw videos, however,
a different person may be speaking (with the target speaker visible or not),
or the target speaker may be speaking but not visible. This is not limited to
interviews but can also be caused by pre-recorded video footage interspersed
in single-speaker podcasts. Furthermore, for multiple faces visible on screen,
the target speaker’s face should be identified.

Our pipeline for identifying snippets of the target person is depicted in
Figure 2 and works as follows: For each snippet, we identify and temporally
crop scene/camera changes using
PySceneDetect222https://github.com/Breakthrough/PySceneDetect and then use
S3FD [Zhang et al., 2017] to detect and crop faces in the extracted
scenes.333These steps are inspired by SyncNet [Chung and Zisserman, 2016]
preprocessing. To get the lips we crop the bottom half of the detected faces.

Snippets must show the target person and that person must be speaking. We
perform two checks on the face-cropped scenes:

  * •

We use the _Talknet_ active speaker detection
tool444https://github.com/TaoRuijie/TalkNet_ASD [Tao et al., 2021] to detect
whether the speaker on-screen is talking. Regardless of who the target
speaker(s) are, this is necessary to ensure that the lip movements visible in
the face-cropped video matches the speech segment.

  * •

We check whether the person on screen is one of the target speakers using a
face recognition tool555https://github.com/ageitgey/face_recognition that
checks similarity with a reference set of images from known target-speaker
videos.666We manually selected 20 snippets of Merkel as samples for face
recognition.

A positive result in both of these tests ensures that our target speaker is
speaking on-screen in the face-cropped scene. If, for some scene, none of the
cropped faces pass the above two checks, the entire snippet is discarded.
Otherwise, the best face-cropped and lip-cropped candidates for each scene are
recombined to give the face-cropped and lip-cropped videos respectively. The
pipeline is demonstrated with an example in Figure 2. On a manually annotated
validation set containing 100 ground truth examples, the pipeline achieves an
accuracy of 94 % with perfect precision, ensuring no false positives in the
processed output.

## 3\. Dataset Statistics

The overall corpus consists of 630 videos totalling 48.0 h of material and
distributed over time as shown in Figure 3. Leading and trailing jingles make
up 2.7 h and there are 2.8 h of (short) silence, leaving 42.5 h of speech
according to the INA speech segmenter777https://github.com/ina-
foss/inaSpeechSegmenter [Doukhan et al., 2018].

Figure 3: Number of podcasts and mean durations (with stddev as error bars)
over time.

Among the podcasts, there are more than 250 interviews, almost all with
different interviewers that at least ask some questions, providing for a rich
multi-speaker background beyond Angela Merkel as the target speaker. The
frequency of podcasts has remained stable over the years; the average duration
increased to about 5-6 minutes during the ‘interview years’ 2012-17, from 3-4
minutes before and after.

All audio is encoded as high-quality AC3 and video resolution was increased to
HD in 2018 (at the same time the audio bitrate reduced from 350kbit/s,
presumably because of adaptive encoding). The signal-to-noise ratio is
excellent for speeches and still good for interviews.

Angela Merkel appears on-screen for approximately 66 % of the time, and was
found to be the on-screen active speaker for around 58 % (statistics obtained
from face recognition and active speaker detection which both are tuned for
precision). As a corollary, the corpus contains 3.8 h of video of Merkel
listening to someone.

Forced alignment (FA) generally is inferior to subtitle timings, although both
have their advantages. Our FA errs on the side of quality (over quantity) and
aligns only about 62 % of all words; frequently, the beginning or end of
sentences is missing (only 54 % of sentences have the first and last words
aligned). Furthermore, we discard sentences where too many words are missing
in the middle. We find that the mean absolute error of subtitle timings is 280
ms and large differences ($>$1 s) are rare. However, we also notice that
subtitles (and also transcripts) are often mildly edited (changes in word
order, repetitions, …) or even completely wrong. FA helps to find and fix
these. Our manually corrected transcripts will be made available with the
corpus.

Figure 4: Target Speaker Dataset Statistics: We show per-snippet histograms of
text length, audio duration and tempo as expressed by the ratio of duration to
text. Figure 5: Violin Plots of the face angles (in degrees) present in the
target-speaker corpus. Two additional peaks in the face yaw-plot belong to the
left and right facing videos.

Based on the analysis, we decide to construct snippets for the single-speaker
corpus based on subtitle files and to back off to alignments for podcasts
before 2013 (where subtitles are not available). The resulting snippets
(Angela Merkel visibly speaking in video, audio and with corresponding text)
make up the single-speaker corpus that can be used for ML applications (see
next section) and is described further in the next paragraphs. Note, though,
that the corpus can also be snippeted in different ways and contextualization
is always possible through the correspondence to the source videos.

Snippets have a mean duration of 7.2 s (std dev 6.2 s) and mean text length of
124 characters (std dev 104 characters). Distributions are shown in Figure 4.
The third subfigure furthermore shows that speech tempo (and pausing) varies
between snippets.

Figure 5 shows violin plots of facial pose. Overall, the variability in the
corpus is similar to LRS 3, although on average, Merkel raises her chin
slightly more, rolls her face more and the interview-style camera positions
result in distinct yaw peaks. Yaw angles are also shown in Figure 1.

### 3.1. Supplementary English Speech

In addition to the podcasts, we acquired three videos of Merkel speaking
English. All three are introductions to speeches with the main content in
German (before the US Congress, the British Parliament and at Stanford). For
some of these, official translations are available.

We manually aligned her English utterances which amount to 86 snippets
containing 250 s of speech (excluding pauses). While too little for machine
learning, it might suffice for cross-lingual evaluation.

## 4\. Machine-learning Applications

In this section, we show different preliminary usages of the single-speaker
corpus that spans a long time period and contains the visual modality. Our
examples are by no means meant to cover the corpus’ usage completely.

### 4.1. Age estimation

Given speech from one speaker over 16 years, we wondered if timing/aging
effects can be observed. We extract pre-trained speaker
embeddings888https://github.com/RF5/simple-speaker-embedding from each snippet
and train a simple softmax logistic regression network to estimate what year
the recording was made. For this 16-class problem, we get a macro-f1 of .63
(and regression coefficient of $R^{2}=.77$). This indicates that the learnt
embeddings – although they are supposed to be speaker-specific – change with
age of the speaker and/or recording. We intend to further research automatic
age estimation which so far has exclusively been performed on multi-speaker
databases.

### 4.2. Lip generation

In lip generation, the task is to re-create lip movements in video when
revoicing it with different speech (e. g. coming from TTS). We re-trained
Wav2Lip [Prajwal et al., 2020] on the single-speaker corpus and we find that
lip renderings (and particularly the movements that result in video) look much
more natural. In particular, Merkel tends to not open her mouth as far as off-
the-shelf Wav2Lip models would and this is fixed by a speaker-specific model.

### 4.3. Visually Grounded Speech Synthesis

The task of synthesizing speech based on text and video (to make the speech
appear in sync with the facial movements) has recently gained traction [Hassid
et al., 2021, Lu et al., 2021, Hu et al., 2021]. This is particularly
important for revoicing video in another language (dubbing) where it avoids
phantom effects (lips moving without speech or vice-versa). It can thus be
seen as an alternative to lip generation to yield synchronized video.

We train a visually grounded text-to-speech system based on Tacotron-2 [Shen
et al., 2018] which we augment by adding an attention mechanism over video. In
preliminary experiments we find that synchrony often improves drastically.
However, the challenges of the Merkel Corpus, in particular the much longer
snippets, the prosodic structures and pauses contained, sometimes lead to
erratic behaviour.

## 5\. Conclusion

We have presented a multi-modal corpus of semi-prepared speeches and more
spontaneous interview-style speech with one primary speaker (speaking for 28 h
excluding silences) and more than 250 secondary speakers (speaking for another
14 h) over 16 years of production of a weekly podcast series. We supplement
this corpus with some English speech from the primary speaker in the hope that
it may be useful for research in lip-synchronous audio-visual translation.

We have presented some preliminary results that show that the corpus can be
used for state-of-the-art ML tasks in which it shows promising results. Beyond
machine-learning applications, the corpus could be useful in the Digital
Social Studies and Political Sciences as it makes available sixteen years of
German federal policy and politics in a way that can easily be automatically
analyzed and searched in a uniform way.

The German government – unlike e. g. the US government – does not place its
works in the public domain. However, recent changes in German copyright law
[Meyer-Sickendiek and Hussein, 2017] allow scraping publicly available data
for research purposes and we may share the data with other researchers. As the
videos and transcripts remain available from the original source, we provide
only the speech-text alignments as well as the snippeting that we used in our
experiments. We have also presented a simple pipeline for multi-modal speaker
diarization in multi-face video which simplifies the conversion of large video
collections (e. g. from talk shows) to single-speaker corpora suitable for
cross-modal machine-learning tasks. We publicly release the scripts required
to download and process the corpus as well as the annotations that we derived
from the original data.999Download scripts and annotations are made available
at https://github.com/deeplsd/Merkel-Podcast-Corpus. The original video data
is available upon e-mail request from the third author.

Finally, ethical implications must be considered when dealing with so much
data from one person – or even trained models for personalized visually
grounded speech synthesis. Given that Angela Merkel is a public figure who is
no longer in office, we believe that the public interest outweighs the
possible impact on her.

## Acknowledgements

We wish to thank Juliane Röscheisen for helping with the scraping scripts used
to download the corpus and Christian Schuler for valuable feedback. The
authors would like to thank the German Academic Exchange Service DAAD for the
internship opportunity that made this work possible.

## 6\. Bibliographical References

## References

  * Baumann et al., 2019 Baumann, T., Köhn, A., and Hennig, F.  (2019).  The Spoken Wikipedia Corpus collection: Harvesting, alignment and an application to hyperlistening.  Language Resources and Evaluation, 53(2):303–329.
  * Chaume, 2012 Chaume, F.  (2012).  Audiovisual translation: Dubbing.  St. Jerome Publishing.
  * Chung and Zisserman, 2016 Chung, J. S. and Zisserman, A.  (2016).  Out of time: automated lip sync in the wild.  In Workshop on Multi-view Lip-reading, ACCV, Taipeh, Taiwan.
  * Doukhan et al., 2018 Doukhan, D., Carrive, J., Vallet, F., Larcher, A., and Meignier, S.  (2018).  An open-source speaker gender detection framework for monitoring gender equality.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5214–5218, Calgary, Canada.
  * Hassid et al., 2021 Hassid, M., Ramanovich, M. T., Shillingford, B., Wang, M., Jia, Y., and Remez, T.  (2021).  More than words: In-the-wild visually-driven prosody for text-to-speech.  CoRR, abs/2111.10139.
  * Hu et al., 2021 Hu, C., Tian, Q., Li, T., Yuping, W., Wang, Y., and Zhao, H.  (2021).  Neural dubber: Dubbing for videos according to scripts.  Advances in Neural Information Processing Systems, 34.
  * Kumar et al., 2018 Kumar, R., Sotelo, J., Kumar, K., de Brébisson, A., and Bengio, Y.  (2018).  ObamaNet: Photo-realistic lip-sync from text.  CoRR, abs/1801.01442.
  * Lu et al., 2021 Lu, J., Sisman, B., Liu, R., Zhang, M., and Li, H.  (2021).  VisualTTS: TTS with accurate lip-speech synchronization for automatic voice over.  CoRR, abs/2110.03342.
  * Meyer-Sickendiek and Hussein, 2017 Meyer-Sickendiek, B. and Hussein, H.  (2017).  The new German copyright law for science and education (UrhWissG): Consequences for DH-projects working with non-academic partners.  In Proceedings of WissOrg, pages 6–11, Berlin, Germany.
  * Prajwal et al., 2020 Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., and Jawahar, C.  (2020).  A lip sync expert is all you need for speech to lip generation in the wild.  In Proceedings of the 28th ACM International Conference on Multimedia, page 484–492, Seattle, USA.
  * Saboo and Baumann, 2019 Saboo, A. and Baumann, T.  (2019).  Integration of dubbing constraints into machine translation.  In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 94–101, Florence, Italy.
  * Shen et al., 2018 Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al.  (2018).  Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779–4783, Calgary, Canada.
  * Tao et al., 2021 Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., and Li, H.  (2021).  Is someone speaking?  Proceedings of the 29th ACM International Conference on Multimedia, Oct.
  * Zhang et al., 2017 Zhang, S., Zhu, X., Lei, Z., Shi, H., Wang, X., and Li, S. Z.  (2017).  S3FD: Single shot scale-invariant face detector.  In Proceedings of the IEEE International Conference on Computer Vision, pages 192–201, Venice, Italy.

## 7\. Language Resource References

lrec2022-bib languageresource