# Host galaxy magnitude of OJ 287 from its colours at minimum light

Mauri J. Valtonen,1,2 Lankeswar Dey,3 S. Zola,4,5 S. Ciprini,6,7 M. Kidger,8
T. Pursimo,9 A. Gopakumar,3 K. Matsumoto,10 K. Sadakane,10 D. B. Caton,11 K.
Nilsson,1 S. Komossa,12 M. Bagaglia,13 A. Baransky,14 P. Boumis,15 D. Boyd,16
A. J. Castro-Tirado,17 B. Debski,4 M. Drozdz,5 A. Escartin Pérez,18 M.
Fiorucci,13 F. Garcia,19 K. Gazeas,20 S. Ghosh,3 V. Godunova,21 J. L. Gomez,17
R. Gredel,22 D. Grupe23 J. B. Haislip,24 T. Henning,22 G. Hurst,25 J. Janík,26
V. V. Kouprianov,24 H. Lehto,2 A. Liakos,15 S. Mathur,27,28 M. Mugrauer,29 R.
Naves Nogues,30 G. Nucciarelli13 W. Ogloza,5 D. K. Ojha,3 U. Pajdosz-
Śmierciak,4 S. Pascolini,13 G. Poyner,31 D. E. Reichart,24 N. Rizzi,13 F.
Roncella,13 D. K. Sahu,32 A. Sillanpää,2 A. Simon,33 M. Siwak,5,34 F. C.
Soldán Alfaro,35,36 E. Sonbas,37,38 G. Tosti,13 V. Vasylenko,33 J. R. Webb,39
and P. Zielinski,40
1Finnish Centre for Astronomy with ESO, University of Turku, Finland
2Tuorla Observatory, Department of Physics and Astronomy, University of Turku,
Finland
3Department of Astronomy and Astrophysics, Tata Institute of Fundamental
Research, Mumbai 400005, India
4Astronomical Observatory, Jagiellonian University, ul. Orla 171, PL-30-244
Krakow, Poland
5Mt Suhora Observatory, Pedagogical University, ul. Podchorazych 2, PL-30-084
Krakow, Poland
6Space Science Data Center - Agenzia Spaziale Italiana, via del Politecnico,
snc, I-00133, Roma, Italy
7Instituto Nazionale di Fisica Nucleare, Sezione di Perugia, Perugia I-06123,
Italy
8European Space Agency European Space Astronomy Centre, 28691 Villanueva de la
Cañada, Madrid, Spain
9Nordic Optical Telescope, Apartado 474, E-38700 Santa Cruz de La Palma, Spain
10Astronomical Institute, Osaka Kyoiku University, 4-698 Asahigaoka,
Kashiwara, Osaka 582-8582, Japan
11Dark Sky Observatory, Department of Physics and Astronomy, Appalachian State
University, Boone, NC 28608, USA
12Max-Planck-Institut für Radioastronomie, Auf dem Hügel 69, D-53121 Bonn,
Germany
13Physics Department, University of Perugia, via A. Pascoli, 06123, Perugia,
Italy
14Astronomical Observatory of Taras Shevshenko National University of Kyiv,
Observatorna str. 3, 04053 Kyiv, Ukraine
15Institute for Astronomy, Astrophysics, Space Applications and Remote
Sensing, National Observatory of Athens,
Metaxa & Vas. Pavlou St., Penteli, Athens GR-15236, Greece
16BAA Variable Star Section, West Challow Observatory, OX12 9TX, UK
17Instituto de Astrofísica de Andalucía (IAA-CSIC), P.O. Box 03004, E-18080,
Granada, Spain
18Aritz Bidea No 8 4B (48100) Mungia Bizkaia, Spain
19Muñas de Arriba La Vara, Valdés (MPC J38) 33780 Valdés, Asturias – Spain
20Section of Astrophysics, Astronomy and Mechanics, Department of Physics,
National and Kapodistrian University of Athens, GR-15784 Zografos, Athens,
Greece
21ICAMER Observatory of NASU, 27, Acad. Zabolotnoho str., 03143 Kyiv, Ukraine
22Max Planck Institute for Astronomy, Koenigstuhl 17, D-69117 Heidelberg,
Germany
23Dept. of Physics, Earth Science, and Space System Engineering, Morehead
State University, 235 Martindale Dr, Morehead, KY 40351, USA
24University of North Carolina at Chapel Hill, Chapel Hill, North Carolina NC
27599, USA
2516 Westminster Close Basingstoke Hampshire RG22 4PP, UK
26Department of Theoretical Physics and Astrophysics, Masaryk University,
Kotlarska 2, 61137 Brno, Czech Republic
27Department of Astronomy, The Ohio State University, 140 West 18th Avenue,
Columbus, OH 43210, USA
28Center for Cosmology and Astroparticle Physics, The Ohio State University,
191 West Woodruff Avenue, Columbus, OH 43210, USA
29Astrophysikalisches Institut und Universitäts-Sternwarte, Schillergässchen
2-3, D-07745 Jena, Germany
30C/Jaume Balmes No 24, Cabrils, Barcelona E-08348, Spain
31BAA Variable Star Section, 67 Ellerton Road, Kingstanding, Birmingham B44
0QE, UK
32Indian Institute of Astrophysics, II Block, Koramangala, Bengaluru 560 034,
India
33Astronomy and Space Physics Department, Taras Shevshenko National University
of Kyiv, Volodymyrska str. 60, 01033 Kyiv, Ukraine
34Konkoly Observatory, Research Centre for Astronomy and Earth Sciences,
Eötvös Loránd Research Network (ELKH), Hungarian Academy of Sciences, Konkoly-
Thege Miklós út 15–17, 1121 Budapest, Hungary
35C/Petrarca 6 1a 41006 Sevilla, Spain
36Astrophysics and Astronomy Introduction, Expert Teacher, Experience Seminar,
Seville University, Seville, Spain
37Department of Physics, University of Adiyaman, Adiyaman 02040, Turkey
38Astrophysics Application and Research Center, Adiyaman University, Adiyaman
02040, Turkey
39Florida International University and SARA Observatory, University Park
Campus, Miami, FL 33199, USA
40Institute of Astronomy, Faculty of Physics, Astronomy and Informatics,
Nicolaus Copernicus University in Toruń, ul. Grudziądzka 5, 87-100 Toruń,
Poland
E-mail<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

OJ 287 is a BL Lacertae type quasar in which the active galactic nucleus (AGN)
outshines the host galaxy by an order of magnitude. The only exception to this
may be at minimum light when the AGN activity is so low that the host galaxy
may make quite a considerable contribution to the photometric intensity of the
source. Such a dip or a fade in the intensity of OJ 287 occurred in November
2017, when its brightness was about 1.75 magnitudes lower than the recent mean
level. We compare the observations of this fade with similar fades in OJ 287
observed earlier in 1989, 1999, and 2010. It appears that there is a
relatively strong reddening of the B$-$V colours of OJ 287 when its V-band
brightness drops below magnitude 17. Similar changes are also seen V$-$R,
V$-$I, and R$-$I colours during these deep fades. These data support the
conclusion that the total magnitude of the host galaxy is $V=18.0\pm 0.3$,
corresponding to $M_{K}=-26.5\pm 0.3$ in the K-band. This is in agreement with
the results, obtained using the integrated surface brightness method, from
recent surface photometry of the host. These results should encourage us to
use the colour separation method also in other host galaxies with strongly
variable AGN nuclei. In the case of OJ 287, both the host galaxy and its
central black hole are among the biggest known, and its position in the black
hole mass-galaxy mass diagram lies close to the mean correlation.

###### keywords:

galaxies: active – BL Lacertae objects: general – BL Lacertae objects:
individual: OJ 287 – galaxies: bulges

††pubyear: 2015††pagerange: Host galaxy magnitude of OJ 287 from its colours
at minimum light–Host galaxy magnitude of OJ 287 from its colours at minimum
light

##  1 Introduction

OJ 287 is a BL Lacertae type quasar (RA: 08:54:48.87, Dec: +20:06:30.6),
situated at a redshift of $z=0.306$. Based on the faintness of its optical
Balmer lines from the broad-line region (, 1985; Nilsson et al., 2010), OJ 287
is classified as a BL Lac object. Given its spectral-energy distribution
(SED), it is of LBL type (, 1995). Optical emission lines from the narrow-line
region (NLR) like [OIII]$\lambda$5007 are only faintly present (Nilsson et
al., 2010). It shows high brightness activity at about 12 yr intervals, most
likely due to a binary black hole (BH) central engine having that orbital
period (Sillanpää et al., 1988; Dey et al., 2019). The binary model consists
of two BHs orbiting each other in an orbit of eccentricity $\sim 0.65$ and
semi-major axis $\sim 0.05$ pc (Dey et al., 2018). The components are unequal:
the mass ratio is $\sim 120$. The secondary cycle of activity with $\sim$
56-year period is also well established and it is understood as the precession
cycle of the binary (Valtonen et al., 2010).

It is less well known that OJ 287 also shows prominent low activity states
called fades, at similar intervals (Takalo et al., 1990; Pietilä et al.,
1999). First such prominent dip in the brightness of OJ 287 was noticed during
1989. Takalo et al. (1990) carried out a detailed study of the 1989 fade and
concluded that at the minimum light, the host galaxy radiated at least the
same amount of energy as the synchrotron source in the V-band. The evidence
for seeing the host galaxy comes in the colours of the object: while the
object’s brightness fades, its spectral energy distribution changes from the
usual power-law spectrum in the optical region to the spectrum typical of an
elliptical galaxy at the redshift of z = 0.3 of OJ 287. In Figure 1, we show
the multicolour data of OJ 287 during the 1989 fade.

Figure 1: The observations of the 1989 fade of OJ 287 with the Nordic Optical
Telescope and the Jacobus Kapteyn Telescope at La Palma, with $2\sigma$
errorbars. Here we plot the magnitudes in three filters B, V and R. The lines
are fifth order polynomial fits through the data points.

Lehto & Valtonen (1996) predicted that the fade should be repeated in 1998,
one binary orbital cycle later, when the underlying binary black hole system
is at the same orbital phase as in 1989. Due to a strong forward precession of
the major axis of the orbit, the orbital phase covers the whole cycle in less
than 12 years. The predicted fade indeed took place, only 6 weeks after the
predicted time (Pietilä et al., 1999). Since then, there have been prominent
fades in 1999 and 2010, while the most recent one took place in November 2017.

In this paper, we collect the data from the fades since 1989 and compare them
with the Takalo et al. (1990) data. We make use of these deepest fades to try
to detect the underlying host galaxy. The host has been difficult to detect
because the radiation from the active galactic nucleus (AGN) is strongly
beamed toward us. During a fade, we have the best opportunity to see the host
galaxy. Further, it is important to determine the host brightness in order to
get another point at the upper end of the BH mass vs. host bulge stellar mass
diagram, where the data are few (Saglia et al., 2016).

The organisation of the paper is as follows: In section 2, we describe the
photometric observations of different fades of OJ 287 used in this paper.
Section 3 uses these and earlier photometric observations to calculate the
changes of colour of OJ 287 as a function of its magnitude. The changes are
modelled by a combination of light from the AGN and the host galaxy. In this
way the two contributions are separated, and the host magnitude is derived. In
section 4, we compare this determination of the host magnitude with the
integrated surface photometry method, and find good agreement with a recent
K-band determination of the host magnitude.

Figure 2: Left panel: The R-band optical light curve of OJ 287 in the 2017/18
season. Right panel: The V-band optical light curve of OJ 287 from 1970 to
2018. The dotted line runs 1.75 magnitudes below the historical mean
brightness of OJ 287.

##  2 Observations

In anticipation of the big flare in OJ 287 on December 2015, an observational
campaign was set up (Valtonen et al., 2016). After the detection of the flare
which peaked on December 5, we continued to monitor the blazar in the wide
band R filter with the Skynet Robotic Network telescopes and at sites in
Japan, Greece, Ukraine, UK, Spain, and Poland. We achieved a daily cadence
except for the periods when OJ 287 was too close to the sun or the moon. On
several occasions, we also took multi-filter data. All photometric
observations were reduced by a single person (S.Z.) to provide as uniform as
possible data set. The reduction is done in a standard way: images were
calibrated for bias, dark and flatfield and we extracted magnitudes using the
aperture method. More details about the procedure were given in Pihajoki et
al. (2013).

The left panel of Figure 2 shows OJ 287 observations which we gathered during
the 2017/18 season. In the right panel of Figure 2, we present how this recent
fade compares with earlier ones. We note that the minimum light at 2017.85 is
1.75 magnitudes below the oscillating historical average shown by the dotted
line (oscillation period is $\sim$ 56 yr (Valtonen et al., 2006)).

Table 1: Perugia VRI observations of the 1998 and 1999 fades. $Year$ | $V$ | $R$ | $I$ | $Year$ | $V$ | $R$ | $I$
---|---|---|---|---|---|---|---
1998.0218 | 16.08 $\pm$ 0.11 | 15.26 $\pm$ 0.05 | 14.56 $\pm$ 0.03 | 1998.9614 | 16.35 $\pm$ 0.19 | 15.85 $\pm$ 0.07 | 15.15 $\pm$ 0.05
1998.0223 | 16.09 $\pm$ 0.09 | 15.57 $\pm$ 0.06 | 14.71 $\pm$ 0.04 | 1998.9616 | — | 15.71 $\pm$ 0.07 | 15.10 $\pm$ 0.064
1998.1312 | 16.40 $\pm$ 0.13 | 16.09 $\pm$ 0.08 | 14.96 $\pm$ 0.04 | 1998.9637 | 15.83 $\pm$ 0.18 | 15.89 $\pm$ 0.06 | 15.10 $\pm$ 0.045
1998.1338 | 15.92 $\pm$ 0.06 | 15.72 $\pm$ 0.04 | 15.27 $\pm$ 0.04 | 1998.9642 | — | 15.68 $\pm$ 0.06 | 15.17 $\pm$ 0.05
1998.1366 | 16.91 $\pm$ 0.13 | 16.01 $\pm$ 0.06 | 15.10 $\pm$ 0.04 | 1998.9748 | — | 15.77 $\pm$ 0.05 | 14.95 $\pm$ 0.04
1998.1394 | 16.53 $\pm$ 0.13 | 16.34 $\pm$ 0.10 | 15.01 $\pm$ 0.04 | 1999.0128 | 16.26 $\pm$ 0.23 | 15.51 $\pm$ 0.05 | 14.87 $\pm$ 0.04
1998.1421 | — | 15.69 $\pm$ 0.07 | 15.08 $\pm$ 0.04 | 1999.0134 | — | 15.68 $\pm$ 0.06 | 14.83 $\pm$ 0.04
1998.1559 | 16.78 $\pm$ 0.14 | 15.79 $\pm$ 0.06 | 15.15 $\pm$ 0.04 | 1999.0136 | — | 15.63 $\pm$ 0.06 | 14.86 $\pm$ 0.04
1998.1586 | 16.63 $\pm$ 0.12 | 15.67 $\pm$ 0.05 | 15.17 $\pm$ 0.04 | 1999.0157 | 15.88 $\pm$ 0.19 | 15.67 $\pm$ 0.05 | 15.07 $\pm$ 0.04
1998.1668 | 16.41 $\pm$ 0.20 | 15.61 $\pm$ 0.08 | 15.00 $\pm$ 0.05 | 1999.0164 | — | 15.76 $\pm$ 0.13 | 14.92 $\pm$ 0.08
1998.1695 | 15.94 $\pm$ 0.10 | 15.59 $\pm$ 0.06 | 15.03 $\pm$ 0.05 | 1999.0239 | — | 15.92 $\pm$ 0.07 | 15.02 $\pm$ 0.04
1998.1777 | 15.96 $\pm$ 0.06 | 15.40 $\pm$ 0.05 | — | 1999.0951 | — | 15.67 $\pm$ 0.09 | 15.25 $\pm$ 0.08
1998.1887 | — | 15.57 $\pm$ 0.06 | 14.86 $\pm$ 0.04 | 1999.1058 | — | 15.82 $\pm$ 0.07 | 15.07 $\pm$ 0.05
1998.1996 | 16.30 $\pm$ 0.11 | 15.78 $\pm$ 0.06 | 14.96 $\pm$ 0.04 | 1999.1196 | — | 15.44 $\pm$ 0.06 | 14.78 $\pm$ 0.04
1998.2049 | 15.72 $\pm$ 0.10 | 15.47 $\pm$ 0.06 | 14.48 $\pm$ 0.04 | 1999.1524 | — | 16.00 $\pm$ 0.07 | 15.34 $\pm$ 0.05
1998.2103 | 15.82 $\pm$ 0.08 | 15.33 $\pm$ 0.05 | 14.75 $\pm$ 0.04 | 1999.1551 | — | 15.53 $\pm$ 0.06 | 15.09 $\pm$ 0.05
1998.2158 | — | 15.40 $\pm$ 0.12 | 14.76 $\pm$ 0.10 | 1999.1688 | — | 16.13 $\pm$ 0.12 | 15.56 $\pm$ 0.07
1998.2350 | — | 15.48 $\pm$ 0.05 | 14.85 $\pm$ 0.04 | 1999.1825 | — | 16.19 $\pm$ 0.10 | 15.56 $\pm$ 0.06
1998.2377 | 16.51 $\pm$ 0.16 | 15.63 $\pm$ 0.06 | 14.92 $\pm$ 0.05 | 1999.1880 | — | 16.22 $\pm$ 0.22 | 15.80 $\pm$ 0.16
1998.2405 | 16.35 $\pm$ 0.21 | 15.82 $\pm$ 0.08 | — | 1999.1907 | — | 16.23 $\pm$ 0.09 | 15.68 $\pm$ 0.07
1998.2432 | 16.07 $\pm$ 0.20 | — | 14.90 $\pm$ 0.05 | 1999.1934 | — | 16.02 $\pm$ 0.08 | 15.41 $\pm$ 0.06
1998.2461 | 16.05 $\pm$ 0.19 | 15.65 $\pm$ 0.13 | 14.81 $\pm$ 0.07 | 1999.1961 | — | 15.88 $\pm$ 0.08 | 15.16 $\pm$ 0.05
1998.2541 | 15.90 $\pm$ 0.11 | 15.41 $\pm$ 0.06 | 14.77 $\pm$ 0.05 | 1999.2016 | — | 16.02 $\pm$ 0.07 | 15.62 $\pm$ 0.06
1998.2568 | 16.01 $\pm$ 0.16 | 15.45 $\pm$ 0.06 | 14.59 $\pm$ 0.04 | 1999.2044 | — | 16.12 $\pm$ 0.11 | 15.37 $\pm$ 0.06
1998.2705 | 15.79 $\pm$ 0.09 | 15.44 $\pm$ 0.05 | 14.84 $\pm$ 0.04 | 1999.2153 | — | 17.02 $\pm$ 0.18 | 15.85 $\pm$ 0.10
1998.3006 | 15.96 $\pm$ 0.08 | 15.29 $\pm$ 0.05 | — | 1999.2155 | — | 16.31 $\pm$ 0.10 | 15.69 $\pm$ 0.07
1998.3034 | 16.00 $\pm$ 0.09 | 15.41 $\pm$ 0.05 | 14.75 $\pm$ 0.04 | 1999.2180 | — | 15.94 $\pm$ 0.08 | 15.44 $\pm$ 0.06
1998.3061 | 15.89 $\pm$ 0.13 | 15.24 $\pm$ 0.05 | 14.78 $\pm$ 0.05 | 1999.2235 | — | 16.78 $\pm$ 0.15 | 15.95 $\pm$ 0.08
1998.3089 | 15.86 $\pm$ 0.09 | 15.42 $\pm$ 0.07 | 14.74 $\pm$ 0.04 | 1999.2453 | — | 16.34 $\pm$ 0.15 | 16.09 $\pm$ 0.11
1998.3144 | 15.39 $\pm$ 0.09 | 15.24 $\pm$ 0.07 | 14.72 $\pm$ 0.05 | 1999.2508 | — | 16.24 $\pm$ 0.12 | 15.73 $\pm$ 0.09
1998.3171 | — | 15.20 $\pm$ 0.09 | 14.58 $\pm$ 0.06 | 1999.2562 | — | 16.21 $\pm$ 0.11 | 15.94 $\pm$ 0.09
1998.9583 | 16.89 $\pm$ 0.38 | 16.01 $\pm$ 0.09 | 14.85 $\pm$ 0.04 | 1999.2727 | — | 16.44 $\pm$ 0.11 | 15.84 $\pm$ 0.08
1998.9610 | — | 15.76 $\pm$ 0.05 | 15.07 $\pm$ 0.04 |  |  |  |
Table 2: Krakow and Mt. Suhora BVRI observations of the 2006, 2010 and 2017 fades. $Year$ | $B$ | $V$ | $R$ | $I$
---|---|---|---|---
2006.7400 | — | 15.85 $\pm$ 0.03 | 15.42 $\pm$ 0.02 | 14.77 $\pm$ 0.01
2006.7455 | — | 15.72 $\pm$ 0.02 | 15.30 $\pm$ 0.02 | 14.66 $\pm$ 0.01
2006.7482 | — | 15.64 $\pm$ 0.01 | 15.21 $\pm$ 0.01 | 14.56 $\pm$ 0.01
2006.7647 | — | 15.54 $\pm$ 0.02 | 15.11 $\pm$ 0.01 | 14.46 $\pm$ 0.01
2006.7730 | — | 15.44 $\pm$ 0.01 | 15.01 $\pm$ 0.01 | 14.36 $\pm$ 0.01
2006.7756 | — | 15.60 $\pm$ 0.01 | 15.15 $\pm$ 0.01 | 14.50 $\pm$ 0.01
2006.7757 | — | — | 15.12 $\pm$ 0.03 | 14.43 $\pm$ 0.02
2006.7783 | — | — | 15.09 $\pm$ 0.01 | 14.45 $\pm$ 0.02
2006.7921 | — | 15.85 $\pm$ 0.02 | 15.40 $\pm$ 0.01 | 14.74 $\pm$ 0.02
2006.7948 | — | 16.01 $\pm$ 0.02 | 15.57 $\pm$ 0.01 | 14.89 $\pm$ 0.01
2006.7940 | 16.61 $\pm$ 0.04 | 16.02 $\pm$ 0.02 | 15.57 $\pm$ 0.01 | 14.85 $\pm$ 0.02
2006.7976 | — | 15.97 $\pm$ 0.04 | 15.50 $\pm$ 0.01 | 14.84 $\pm$ 0.02
2006.8003 | — | 15.68 $\pm$ 0.02 | 15.26 $\pm$ 0.02 | 14.62 $\pm$ 0.01
2006.8524 | 16.56 $\pm$ 0.03 | 16.01 $\pm$ 0.03 | 15.56 $\pm$ 0.02 | 14.92 $\pm$ 0.02
2006.8551 | 16.23 $\pm$ 0.08 | 15.60 $\pm$ 0.11 | 15.18 $\pm$ 0.10 | 14.57 $\pm$ 0.08
2006.8742 | 16.14 $\pm$ 0.04 | 15.59 $\pm$ 0.02 | 15.17 $\pm$ 0.02 | 14.54 $\pm$ 0.02
2006.8800 | 16.16 $\pm$ 0.03 | 15.59 $\pm$ 0.01 | 15.17 $\pm$ 0.01 | 14.52 $\pm$ 0.01
2006.9070 | 15.97 $\pm$ 0.01 | 15.43 $\pm$ 0.02 | 15.02 $\pm$ 0.01 | 14.38 $\pm$ 0.01
2006.9097 | 16.06 $\pm$ 0.01 | 15.49 $\pm$ 0.01 | 15.07 $\pm$ 0.01 | 14.44 $\pm$ 0.01
2006.9150 | 16.09 $\pm$ 0.02 | 15.51 $\pm$ 0.02 | 15.09 $\pm$ 0.02 | 14.44 $\pm$ 0.02
2006.9234 | 16.13 $\pm$ 0.05 | 15.56 $\pm$ 0.01 | 15.10 $\pm$ 0.02 | 14.46 $\pm$ 0.02
2006.9290 | 16.08 $\pm$ 0.01 | 15.53 $\pm$ 0.02 | 15.11 $\pm$ 0.03 | 14.47 $\pm$ 0.03
2006.9452 | 16.41 $\pm$ 0.03 | 15.82 $\pm$ 0.02 | 15.35 $\pm$ 0.02 | 14.70 $\pm$ 0.01
2006.9534 | 16.36 $\pm$ 0.01 | 15.78 $\pm$ 0.01 | 15.32 $\pm$ 0.01 | 14.65 $\pm$ 0.02
2006.9588 | 16.48 $\pm$ 0.01 | 15.88 $\pm$ 0.02 | 15.45 $\pm$ 0.02 | 14.78 $\pm$ 0.02
2006.9863 | 16.32 $\pm$ 0.02 | 15.74 $\pm$ 0.02 | 15.31 $\pm$ 0.01 | 14.65 $\pm$ 0.02
2006.9889 | 16.37 $\pm$ 0.01 | 15.80 $\pm$ 0.01 | 15.36 $\pm$ 0.01 | 14.69 $\pm$ 0.01
2006.9946 | 16.55 $\pm$ 0.02 | 15.98 $\pm$ 0.02 | 15.54 $\pm$ 0.01 | 14.88 $\pm$ 0.01
2007.0270 | 16.43 $\pm$ 0.04 | 15.85 $\pm$ 0.02 | 15.39 $\pm$ 0.02 | 14.72 $\pm$ 0.02
2007.0328 | — | 15.77 $\pm$ 0.06 | 15.33 $\pm$ 0.03 | 14.69 $\pm$ 0.03
2007.0386 | 16.62 $\pm$ 0.02 | 16.03 $\pm$ 0.02 | 15.57 $\pm$ 0.01 | 14.89 $\pm$ 0.02
2007.0442 | 16.60 $\pm$ 0.03 | 16.00 $\pm$ 0.02 | 15.55 $\pm$ 0.03 | 14.94 $\pm$ 0.02
2007.0708 | — | 16.05 $\pm$ 0.02 | 15.61 $\pm$ 0.03 | 14.96 $\pm$ 0.01
2007.1228 | 16.60 $\pm$ 0.01 | 16.04 $\pm$ 0.01 | 15.60 $\pm$ 0.01 | 14.94 $\pm$ 0.01
2007.1315 | 16.36 $\pm$ 0.02 | 15.78 $\pm$ 0.02 | 15.35 $\pm$ 0.01 | 14.71 $\pm$ 0.02
2007.1339 | 16.21 $\pm$ 0.05 | 15.66 $\pm$ 0.02 | 15.23 $\pm$ 0.01 | 14.59 $\pm$ 0.01
2007.1529 | — | 15.63 $\pm$ 0.01 | 15.18 $\pm$ 0.01 | 14.53 $\pm$ 0.01
2007.1750 | 16.19 $\pm$ 0.01 | 15.64 $\pm$ 0.01 | 15.21 $\pm$ 0.01 | 14.56 $\pm$ 0.02
2007.1777 | 16.21 $\pm$ 0.04 | 15.62 $\pm$ 0.03 | 15.20 $\pm$ 0.02 | 14.59 $\pm$ 0.02
2007.1802 | 16.30 $\pm$ 0.02 | 15.72 $\pm$ 0.02 | 15.28 $\pm$ 0.02 | 14.64 $\pm$ 0.02
2007.1939 | 16.20 $\pm$ 0.02 | 15.66 $\pm$ 0.01 | 15.22 $\pm$ 0.01 | 14.60 $\pm$ 0.01
2007.1967 | 16.16 $\pm$ 0.02 | 15.61 $\pm$ 0.01 | 15.20 $\pm$ 0.01 | 14.58 $\pm$ 0.01
2010.6989 | — | 16.23 $\pm$ 0.01 | 15.81 $\pm$ 0.01 | 15.20 $\pm$ 0.01
2010.7290 | — | 16.09 $\pm$ 0.03 | 15.64 $\pm$ 0.02 | 15.04 $\pm$ 0.01
2010.8059 | 16.01 $\pm$ 0.01 | 15.55 $\pm$ 0.01 | 15.12 $\pm$ 0.01 | —
2010.8085 | — | 15.60 $\pm$ 0.01 | 15.18 $\pm$ 0.01 | 14.55 $\pm$ 0.01
2010.8113 | — | 15.63 $\pm$ 0.01 | 15.22 $\pm$ 0.01 | 14.59 $\pm$ 0.02
2010.8278 | — | 15.63 $\pm$ 0.01 | 15.20 $\pm$ 0.01 | —
2011.1555 | 15.86 $\pm$ 0.01 | — | 15.03 $\pm$ 0.01 | —
2017.8717 | 16.60 $\pm$ 0.01 | 16.03 $\pm$ 0.01 | 15.63 $\pm$ 0.01 | 14.99 $\pm$0.01
2017.9182 | 15.82 $\pm$ 0.01 | 15.26 $\pm$ 0.01 | 14.89 $\pm$ 0.01 | 14.29 $\pm$0.01
2018.0221 | 15.32 $\pm$ 0.01 | 14.77 $\pm$ 0.01 | 14.40 $\pm$ 0.01 | 13.82 $\pm$0.01

Older data were obtained from Perugia for the 1998-1999 fade and are presented
in Table 1 (for details see Massaro et al. (2003) and Ciprini et al. (2008)).
The data for more recent 2006, 2010 and 2017 fades are given in Table 2. In
addition to professional observatories, several amateur astronomers also
contributed to the data. Some of these fades happen about six months after the
time when the secondary is close to the jet line (expected jet-line minima at
1988.95 and 1997.61 in the binary cnetral engine model (Valtonen et al.,
2022)) while others do not have such a connection to the orbit. The latest
"jet-line" fade was expected around 2020.53, at the time when OJ 287 was not
observable by ground-based optical telescopes (Valtonen et al., 2022).

##  3 Colours of OJ 287 and its host

Figure 3: The variation of the B-V colour as a function of V-band magnitude in
OJ 287. The points above V$=$16.2 refer to the 1989 fade, the others come from
our Tables 1 and 2, and are averages of ten measurements each. The red dash-
dotted line represents a second order polynomial fit to the data points. The
black dashed line shows the expected colour index variation if a host of
V$=$18.0 is added to the AGN of constant colour. The horizontal dotted lines
represent the expected host galaxy colour and the AGN colour, respectively.

In this section, we compare colours of OJ 287 at various V-magnitude levels
with those corresponding to a hypothetical host galaxy. For a supergiant
elliptical galaxy at the redshift of 0.3, the expected colours are:
$B-V=1.43\pm 0.1$, $V-R=0.91\pm 0.05$, $V-I=1.56\pm 0.1$, and $R-I=0.65\pm
0.05$, for a passive evolutionary model (Kristian et al., 1978; Bruzual, 1983;
, 1987; Guiderdoni & Rocca-Volmerange, 1988; Fioc & Rocca-Volmerange, 1997).

For OJ 287, in addition to the steady light from the host, there is a highly
variable radiation from the AGN. The AGN component radiates synchrotron
radiation with a constant spectral index $\alpha\sim 1.5$ (the flux
$F_{\nu}\sim\nu^{-\alpha}$) over the V-magnitude range from 16.5 to 14,
irrespective of the flux level (Kidger et al., 1995; Hagen-Thorn et al.,
1998). The only confirmed exception to this rule arises during the thermal
flares when the flux rise makes $V\leq 13.5$ and the spectrum becomes
exceptionally blue (Valtonen et al., 2012, 2016). The second possible
exception, discussed in this paper, occurs when brightness in $V$ drops below
about 17 mag. Aside from those special episodes, the colours of OJ 287 are:
$B-V=0.57\pm 0.13$, $V-R=0.44\pm 0.09$ and $V-I=1.11\pm 0.16$ (Efimov et al.,
2002).

In order to model the gradual change of colours due to an underlying galaxy,
we first carried out a second order polynomial fit to all the available
colours as a function of magnitude. The B$-$V, V$-$R, V$-$I and R$-$I colours
were taken from the values shown in Tables 1 and 2 as well as from the
previously published values in Takalo et al. (1990). One such fit is shown in
Figure 3 for example, where we plot the B$-$V at different V intervals of OJ
287 and the red dash-dotted line shows the polynomial fit. The expected
colours of the host galaxy and the AGN are marked by the horizontal lines in
these graphs (see Figure 3). There is an upward curvature of the best fit
which is significant above $3\sigma$ level in all except in the R-I diagram.
We identified the possible location of the host in the magnitude axis as the
point where the fit to the observations meets the host line. At this
brightness, practically all the light comes from the host galaxy.

Determined in this way, we find that the host galaxy magnitude is $V=18.2\pm
0.3$ from the B-V data, $V=17.5\pm 0.5$ from the V$-$R data, and $V=17.25\pm
0.5$ from the V$-$I data. In the R$-$I diagram, the AGN and host galaxy lines
are so close to each other that we cannot derive the host magnitude. The
standard deviation of the fit through points is included in the error
estimate. Also the error limit includes the error in the expected colour of
the host as mentioned above.

Simulations of the colours derived from the optical spectrum (power law plus
host galaxy model) in different states can be used to evaluate the effective
capability of the method. With this method we calculate the theoretical line
more accurately. We add the AGN of the above mentioned colour to a host of a
given V-magnitude. Different host magnitudes were applied until we find one
which gives a good fit to the observations. The result is shown by the black
dashed line in Figure 3. The calculations are carried out in linear flux
units, and subsequently converted to magnitudes. The scatter in the
observational points is similar to earlier studies (Efimov et al., 2002).

From Figure 3, we find that the simulation study supports the host galaxy
magnitude to be $V=18.0\pm 0.3$. However, the faintest R-band value, $17.0\pm
0.2$ (observed in Perugia in 1999), puts an upper limit to the host brightness
of $V\sim 17.7$. All data are thus consistent with $V=18.0\pm 0.3$.

Note that the 2017 high accuracy observations from Krakow and Mt.Suhora
Observatories as well as the UK and Spanish observations, even though with a
larger scatter, agree perfectly with the expected colour behaviour which
arises from adding the host and the AGN components. This would be very
surprising if the underlying reason for the colour changes is something else
than the influence of the host. Remember that the base level of OJ 287 was
about one magnitude fainter in 1989 and 1999 than in 2017. If the colour
changes were related to the relative drop in brightness at the fade, then we
should have seen a prominent upturn of 2017 points at higher V which clearly
is not there.

The best spectral range for the host galaxy study is from B to V. This is
because the expected host galaxy flux turns down strongly below the V-band
wavelength while the AGN flux does not. In the B$-$V diagram, the six points
of highest V value come from Takalo et al. (1990). The minima according to the
fits to the data happen first in the R-band, half-a-day later in the V-band,
and another half-a-day later in the B-band. If we assume that OJ 287 has the
same light curve shape in all channels, but the light curves are shifted as
mentioned, then the scatter in the points around the theoretical line is
reduced. Note that the fade occurred two weeks earlier at 22 GHz and 37 GHz
radio frequencies, supporting the idea of the wavelength dependence of the
timing of fades.

##  4 Discussion and conclusions

There have been several attempts to measure the host galaxy magnitude by
direct imaging, after subtracting the point-like AGN from the image. The task
is difficult since the brightness difference between the host and the AGN is
typically about 2 magnitudes in the K-band, 3 magnitudes in the I-band, and
even greater at shorter optical wavelengths (Nilsson et al., 2020). The most
favourable contrast between the AGN and the host so far has been obtained in
infrared. Nilsson et al. (2020) reported recent infrared observations at the
2.5-meter _Nordic Optical Telescope_ when the AGN was at a low state. Imaging
data were also obtained at the 10.4-meter _Gran Telescopio Canarias_ somewhat
after the 2017 deep fade. They essentially confirm the findings of this paper,
as we will discuss below.

We have determined the magnitude of the host galaxy of the BL Lacertae object
OJ 287 using the change of the colour of the object during its minimum light.
It is interesting to translate the result to the K-band for a direct
comparison with the surface brightness method. Using the distance modulus m -
M = 41, V - K = 3.2 (Fioc & Rocca-Volmerange, 1997; Ho et al., 2011), and the
combined K-correction and evolution correction of -0.3 in the K-band (Kristian
et al., 1978; Bruzual, 1983; , 1987; Guiderdoni & Rocca-Volmerange, 1988; Fioc
& Rocca-Volmerange, 1997; Chilingarian et al., 2010; Nilsson et al., 2020), we
obtain the absolute K-band magnitude of OJ 287 $M_{K}=-26.5\pm 0.3$. This may
be compared with the absolute K-band magnitude $M_{K}=-26.8$ of the nearby
elliptical galaxy NGC 4889; the other nearby supergiant NGC 1600 is a little
fainter (Thomas et al., 2016). These two galaxies are interesting as their
central black hole masses agree with the primary black hole mass in OJ 287
within error limits (Valtonen et al., 2010).

In Figure 4, we place OJ 287 in the BH mass vs. galaxy mass diagram (Saglia et
al., 2016). We see from Figure 4 that OJ 287 is slightly, but not
significantly, above the line for the mean linear correlation for nearby
galaxies. If it were placed in the corresponding graph for galaxies at
redshift 0.5 (Portinari et al., 2012), the result would be very much the same.

Figure 4: The galaxy sample of Saglia et al. (2016) in the black hole mass vs.
galaxy stellar mass diagram (blue points), while the position of OJ 287 is
given as a red star. OJ 287 is taken to be $0.3$ magnitudes fainter than NGC
4889. Three recent additional measurements have been added: NGC 1600 (Thomas
et al., 2016), Abell 1201 central galaxy (, 2017) and Holm 15A (Mehrgan et
al., 2019). We note that OJ 287 lies within the one sigma scatter from the
mean linear correlation (dashed line).

All studies of the magnitude of OJ 287 by the imaging method have been limited
to the distance range 5 - 35 kpc from the center. The light from the rest of
the galaxy has to be extrapolated using models. Nilsson et al. (2020) find the
K-band magnitudes, K=14.25 and K=15.27, where the first one has the higher
signal-to-noise ratio ($\sim 8$) of the two. This translates to
$M_{K}=-26.45\pm 0.1$ with the same K-correction and evolution correction as
above. The second K-band determination would make the host one magnitude
fainter. Either way, the host is found to be very bright also by using the
surface brightness method in the K-band.

Experience with other elliptical galaxies in this brightness class (e.g. M87)
has shown that if we restrict our measurements within 30 kpc distance from the
center, we are missing more than a negligible amount of light from the outer
envelope of the galaxy. For example, the absolute V-magnitude of NGC 4889
(Sandage, 1972) is $M_{V}=-22.2$ (using m - M = 34.7) measured out to roughly
the same 30 kpc linear distance as the K-band data of Nilsson et al. (2020),
while its total magnitude out to 100 kpc distance is about $M_{V}=-23.6$
(Graham & Scott, 2013). Typically the outer envelope contributes about 0.8
magnitudes to the total brightness (De Vaucouleurs & De Vaucouleurs, 1970;
Läsker et al., 2014; Huang et al., 2013).

The colour method measures all the light from the galaxy, including the outer
envelope within the diaphragm of the photometer. In this case, it is roughly
equivalent to 35 kpc linear distance at OJ 287. Therefore some of the light of
the host galaxy is missed by this method. In the imaging method the outer
envelope may be missed altogether if the data are not sufficient to include it
in the model. In OJ 287, the point spread function of the central point source
prevents the determination of the accurate sky background, necessary to
measure the magnitude of the outermost component. Going towards shorter
wavelengths, the signal-to-noise ratio becomes considerably worse due the AGN
component becoming more and more dominant. In the I-band the signal-to-noise
ratio falls below 5 (Nilsson et al., 2020), and at shorter wavelengths the
situation becomes even worse. Probably for this reason, some other surface
brightness measurements seem to have seriously underestimated the true
brightness of the OJ 287 host galaxy (, 1996; Heidt et al., 1999). The Hubble
Space Telescope I-band measurement by Yanny et al. (1997), translated to
K-band, gives the brightness upper limit at $M_{K}\sim-26.3$, in agreement
with our result especially if the missing low brightness outer halo is taken
into account.

Our magnitude determination is based on the assumption that the colour of OJ
287 at low light level is modified by the light of the host galaxy. It is also
possible that there is a "cosmic conspiracy", that is, the colour of OJ 287
evolves at low light levels because the AGN mimics the effects of the host.
However, in this case, one might assume that the colour behaviour should be
related to the relative flux decline, and we should see a similar behaviour in
2017 as in 1989 and 1999. On the other hand, if the colour changes are related
to the absolute brightness levels, such as if the constant host galaxy is
influencing the colours, then the 2017 fade should be quite different from the
earlier deep fades. The latter has clearly been the case.

It appears that the lowering the AGN activity does not in itself cause the
observed colour changes. If indeed the base level brightness variation is due
to the varying Doppler boosting (Valtonen & Pihajoki, 2013), then the current
2017 fade is about as deep as the 1989-1999 fades in relative terms. The
colour changes should be seen now as they were seen on earlier occasions. The
three high accuracy points from Krakow and Mt.Suhora Observatories during the
last fade (see the the first three points on the left hand side of Figure 3)
demonstrate that this is not the case.

Komossa et al. (2020, 2021) confirm this result over a wider spectral range,
using SEDs from the optical to UV (and X-rays as well), taken with the Neil
Gehrels Swift observatory (Swift hereafter) at a cadence of 1–2 days in the
course of the program MOMO (Multiwavelength observations and modelling of OJ
287; see Komossa et al. (2021b)). All optical and UV bands, and the X-rays are
near simultaneous, the UV-optical bands within minutes. In particular, Komossa
et al. (2021) reported that the ratio of UV flux over optical flux ratio is
constant throughout the deep fade, and therefore excluded an extinction event,
because the Swift UV bands are very sensitive to extinction. Table 3 gives the
average B-V colour and the V-band over UV-band W2 flux ratio at 4 brightness
levels. It shows no significant colour trends with brightness. Going to the
other direction on the wavelength axis, Kidger et al. (1998) have found that
between infrared and optical regions, there are no colour changes over the
V-magnitude range from 14.7 to 15.7.

Table 3: Summary of Swift observations. $V$ | $B-V$ | $V/W2$
---|---|---
16.0 | 0.55 $\pm$ 0.03 | 1.47 $\pm$ 0.05
15.6 | 0.60 $\pm$ 0.03 | 1.58 $\pm$ 0.05
15.2 | 0.54 $\pm$ 0.03 | 1.49 $\pm$ 0.05
14.8 | 0.53 $\pm$ 0.03 | 1.46 $\pm$ 0.05

The colour changes are apparently related to the absolute brightness of OJ
287. The influence of the host galaxy naturally explains this fact and seems
to exclude the possibility that there is something in the jet emission
mechanism that changes the colour depending on the blazar flux level.

It is likely that deep fades will happen in OJ 287 also in future. They will
provide an excellent opportunity to probe the outer envelope of this galaxy
further. There is no reason why the photometry during the fade should be
limited only within 6 arcsec from the center. While the direct imaging of the
outer envelope is challenging, the separation of the host from the AGN by the
colour method is rather simple. And on the basis of the excellent agreement of
the magnitudes of the host galaxy by the two different methods we have
demonstrated, the colour separation is a reliable method, and it could be
recommended for the study of host galaxies in other AGN as well. The method is
particularly well suited for the detection of low brightness level components
of the host galaxies. Suitable candidates for such wide aperture photometry
are found among the lowest redshift BL Lacertae objects (Urry et al., 2000).
To our knowledge, such a study has never been attempted. It is understandable,
since it takes a large amount of observing time to sample the BL Lacertae
objects at different light levels, and moreover, the result cannot be deduced
from usual photometry where the aperture is chosen so as to eliminate the
influence of the host on the AGN magnitude.

The fades may arise from a temporary misalignment of the jet with regard to
its average direction (Takalo et al., 1990). The jet appears to wiggle (Gomez
et al., 2021) and in principle these wiggles are predictable (Dey et al.,
2021). The modelling of the time behaviour of the optical polarisation may be
a way to predict also the optical fades. Alternatively, the fades may
represent times of a change in accretion flow affecting (reducing) the jet
emission, and it may also be possible to predict such moments of time
(Valtonen et al., 2022).

## Acknowledgements

We would like to thank Roberto Saglia for providing information on black hole
mass measurements in advance of publication and for enlightening discussions.
S.Z. would also like to acknowledge support of the NCN grant No.
2018/29/B/ST9/01793, and K.M. JSPS KAKENHI grant number 19K03930.

## Data Availability

The data published in this paper is available upon request from the authors.

## References

  * Bruzual (1983) Bruzual,A.G. 1983, ApJ, 273, 105
  * Chilingarian et al. (2010) Chilingarian, I.V., Melchior, A.-L. & Zolotukhin, I.Y. 2010, MNRAS, 405, 1409
  * Ciprini et al. (2008) Ciprini, S., Tosti, G., Nucciarelli, G., Bagaglia, M., Fiorucci, M., Impiombato, D., Luciani, M., Maffei, P., Marchili, N., Pascolini, S. et al. 2008, in Proceedings of the Workshop on Blazar Variability across the Electromagnetic Spectrum. April 22-25, 2008 Palaiseau, France. Published online at http://pos.sissa.it, id.72
  * De Vaucouleurs & De Vaucouleurs (1970) De Vaucouleurs, G., De Vaucouleurs, A., 1970, Astrophys Lett, 5, 219
  * Dey et al. (2019) Dey, L., Gopakumar, A., Valtonen, M., Zola, S., Susobhanan, A., Hudec, R., Pihajoki, P., et al., 2019, Univ, 5, 108. doi:10.3390/universe5050108
  * Dey et al. (2018) Dey, L., Valtonen, M. J., Gopakumar, A., Zola, S., Hudec, R., Pihajoki, P., Ciprini, S., et al., 2018, ApJ, 866, 11. doi:10.3847/1538-4357/aadd95
  * Dey et al. (2021) Dey, L., Valtonen, M. J., Gopakumar, A., Lico, R., Gómez, J. L., Susobhanan, A., Komossa, S. & Pihajoki, P., 2021, MNRAS, 503, 4400
  * Efimov et al. (2002) Efimov, Yu. S., Shakhovskoy, N. M., Takalo, L. O., Sillanpää, A., 2002, A&A, 381, 408
  * Gomez et al. (2021) Gómez, J. L., Traianou, E., Krichbaum, T. P., Lobanov, A., Fuentes, A., Lico, R., Zhao, G.-Y., Bruni, G., Kovalev, Y. Y., Lähteenmäki, A. et al., 2021, arXiv, 211111200
  * Graham & Scott (2013) Graham, A. & Scott, N., 2013, ApJ, 764, 151
  * Fioc & Rocca-Volmerange (1997) Fioc, M. & Rocca-Volmerange, B., 1997, A&A, 326, 950
  * (12) Guiderdoni, B. & Rocca-Volmerange, B. 1987, A&A, 186, 1
  * Guiderdoni & Rocca-Volmerange (1988) Guiderdoni, B. & Rocca-Volmerange, B., 1988, A&AS, 74, 185
  * Hagen-Thorn et al. (1998) Hagen-Thorn, V. A., Marchenko, S. G., Takalo, L. O., Sillanpää, A., Pursimo, T., Boltwood, P., Kidger, M. & Gonzalez-Perez, J. N., 1998, A&AS, 133, 353
  * Heidt et al. (1999) Heidt, J., Nilsson, K., Appenzeller, I., Jäger, K., Seifert, W., Szeifert, T., Gössler, W., Höfner, R. et al. 1999, A&A, 352, L11
  * Ho et al. (2011) Ho, L. C.; Li, Z.-Y., Barth, A. J., Seigar, M. S. & Peng, C. Y. 2011, ApJS, 197, 21
  * Huang et al. (2013) Huang, S., Ho, L. C., Peng, C. Y., Li, Z.-Y. & Barth, A. J. 2013, ApJ, 766, 47
  * Kidger et al. (1995) Kidger, M. R., Gonzalez-Perez, J. N., de Diego, J. A., Zapatero-Osorio, M. R., Hammersley, P. L., Cepa, J., Devaney, N., Sahu, K. & Vidal, I. 1995, A&AS, 113, 431
  * Kidger et al. (1998) Kidger, M., Zola, S., Valtonen, M., Lähteenmäki, A., Järvelä, E., Tornikoski, M., Tammi, J., Liakos, A. & Poyner, G., 2018, A&A, 610, A74
  * Komossa et al. (2020) Komossa, S., Grupe, D., Parker, M. L., Valtonen, M. J., Gómez, J. L., Gopakumar, A., Dey, L. 2020, MNRAS, 498, L35
  * Komossa et al. (2021) Komossa, S., Grupe, D., Gallo, L. C., Gonzalez, A., Yao, S., Hollett, A. R., Parker, M. L. & Ciprini, S. 2021, ApJ, 923, 51
  * Komossa et al. (2021b) Komossa S., Grupe D., Kraus A., Gallo L. C., Gonzalez A. G., Parker M. L., Valtonen M. J., et al., 2021, Univ, 7, 261. doi:10.3390/universe7080261
  * Kristian et al. (1978) Kristian, J., Sandage, A. & Westphal, J. A. 1978, ApJ, 221, 383
  * Läsker et al. (2014) Läsker, R., Ferrarese, L. & van de Ven, G. 2014, ApJ, 780, 69
  * Lehto & Valtonen (1996) Lehto, H.J., & Valtonen, M.J. 1996, ApJ, 460, 207
  * Massaro et al. (2003) Massaro, E., Giommi, P., Perri, M., Tagliaferri, G., Nesci, R., Tosti, G., Ciprini, S., Montagni, F., Ravasio, M., Ghisellini, G. et al. 2003, A&A, 399, 33
  * Mehrgan et al. (2019) Mehrgan, K., Thomas, J., Saglia, R., Mazzalay, X., Erwin, P., Bender, R., Kluge, M. & Fabricius, M. 2019, ApJ, in press
  * Nilsson et al. (2010) Nilsson, K., Takalo, L.O., Lehto & Sillanpää, A., 2010, A&A, 516, A60
  * Nilsson et al. (2020) Nilsson, K., Kotilainen, J., Valtonen, M., Gomez, J.L., Castro-Tirado, A.J., Drozdz, M., Gopakumar, A., Jeong, S., Kidger, M., Komossa, S. et al. 2020, ApJ, 904, 102
  * (30) Padovani P.& Giommi P., 1995, ApJ, 444, 567
  * Pietilä et al. (1999) Pietilä, H., Takalo, L. O., Tosti, G, Benitez, E., Chiattelli, B., Corradi, R. L. M., Cox, G., de Diego, J. A., de Francesco, G., Dultzin-Hacyan, D. et al. 1999, A&A, 345, 760
  * Pihajoki et al. (2013) Pihajoki, P., Valtonen, M., Zola, S., Liakos, A., Drozdz, M., Winiarski, M., Ogloza, W., Koziel-Wierzbowska, D., Provencal, J., Nilsson, K., et al. 2013,ApJ, 764, 5
  * Portinari et al. (2012) Portinari, L., Kotilainen, J., Falomi, R. & Decarli, R. 2012,MNRAS, 420, 732
  * Pursimo et al. (2002) Pursimo, T., Nilsson, K., Takalo, L. O., Sillanpää, A., Heidt, J.& Pietilä, H., 2002, A&A, 381, 810
  * Saglia et al. (2016) Saglia, R. P., Opitsch, M., Erwin, P., Thomas, J., Beifiori, A., Fabricius, M., Mazzalay, X., Nowak, N., Rusli, S. P. and Bender, R., 2016, ApJ, 818, 47
  * Sandage (1972) Sandage, A. 1972, ApJ, 176, 21
  * Sillanpää et al. (1988) Sillanpää, A., Haarala, S., Valtonen, M.J., Sundelius, B. & Byrd, G.G. 1988, ApJ, 325, 628
  * (38) Sitko, M.L., & Junkkarinen, V. T., 1985, PASP, 97, 1158
  * (39) Smith, R. J., Lucey, J. R. & Edge, A. C. 2017, MNRAS, 471, 383
  * Takalo et al. (1990) Takalo, L.O., Kidger, M., de Diego, J.A., Sillanpää, A., Piirola, V. & Teräsranta, H., 1990, A&AS, 83, 459
  * Thomas et al. (2016) Thomas, J., Ma, C.-P., McConnell, N. J., Greene, J. E., Blakeslee, J. P. & Janish, R. 2016, Nature, 532, 340
  * Urry et al. (2000) Urry, C.M., Scarpa, R., O’Dowd, M., Falomo, R., Pesce, J.E. & Treves, A., 2000, ApJ, 532, 816
  * Valtonen et al. (2006) Valtonen, M. J., Lehto, H. J., Sillanpää, A., Nilsson, K., Mikkola, S., Hudec, R., Basta, M., Teräsranta, H., Haque, S. & Rampadarath, H. 2006, ApJ, 646, 36
  * Valtonen et al. (2010) Valtonen, M. J., Mikkola, S., Merritt, D., Gopakumar, A., Lehto, H. J., Hyvönen, T., Rampadarath, H., Saunders, R., Basta, M. & Hudec, R. 2010, ApJ, 709, 725
  * Valtonen et al. (2012) Valtonen, M. J., Ciprini, S. & Lehto, H. J. 2012, MNRAS, 427, 77
  * Valtonen & Pihajoki (2013) Valtonen, M. J. & Pihajoki, P. 2013, A&A, 557, 28
  * Valtonen et al. (2016) Valtonen, M. J., Zola, S., Ciprini, S., Gopakumar, A., Matsumoto, K., Sadakane, K., Kidger, M., Gazeas, K., Nilsson, K., Berdyugin, A. et al. 2016, ApJ, 819, L37
  * Valtonen et al. (2017) Valtonen, M. J., Zola, S., Jermak, H., Ciprini, S., Hudec, R., Dey, L., Gopakumar, A., Reichart, D., Caton, D., Gazeas, K. et al. 2017, Galaxies, 5, 83
  * Valtonen et al. (2022) Valtonen, M. J., Dey, L., Gopakumar, A., Zola, S., Komossa, S., Pursimo, T., Gomez, J. L., Hudec, R., Jermak, H. & Berdyugin, A. V., 2022, Galaxies, 10, 1
  * Yanny et al. (1997) Yanny, B., Jannuzi, B.T. & Impey, C. 1997, ApJ, 484, L113
  * (51) Wurtz, R., Stocke, J. T. & Yee, H. K. C., 1996, ApJS, 103, 109
  * (52) Young, P.J., 1976, AJ, 81, 807


# speaker- and age-invariant training for child acoustic modelling using
adversarial multi-task learning

###### Abstract

One of the major challenges in acoustic modelling of child speech is the rapid
changes that occur in the children’s articulators as they grow up, their
differing growth rates and the subsequent high variability in the same age
group. These high acoustic variations along with the scarcity of child speech
corpora have impeded the development of a reliable speech recognition system
for children. In this paper, a speaker- and age-invariant training approach
based on adversarial multi-task learning is proposed. The system consists of
one generator shared network that learns to generate speaker- and age-
invariant features connected to three discrimination networks, for phoneme,
age, and speaker. The generator network is trained to minimize the phoneme-
discrimination loss and maximize the speaker- and age-discrimination losses in
an adversarial multi-task learning fashion. The generator network is a Time
Delay Neural Network (TDNN) architecture while the three discriminators are
feed-forward networks. The system was applied to the OGI speech corpora and
achieved a 13

Index Terms—  adversarial multitask learning, child speech recognition, age-
invariant, speaker-invariant

##  1 Introduction

Despite the enormous improvement in the acoustic modelling of adult speech
over the last few decades, less progress has been made on the acoustic
modelling of child speech.

Automatic speech recognition systems trained on adult speech have shown a
dramatic degradation in performance when tested on child speech due to
linguistic and acoustic mismatches between adult and child speech [1].
Children have higher fundamental and formant frequencies due to their smaller
vocal cords and shorter vocal tract [2]. Furthermore, the shape of the vocal
tract changes rapidly as children grow up and their ability to correctly
pronounce speech sounds improves. This leads to wider intra- and inter-speaker
variations compared with adult speech [3, 4].

Several approaches were initially proposed to handle variations in child
speech at the feature level such as Vocal Tract Length Normalization (VTLN)
[1, 5, 6, 7], Stochastic Feature Mapping (SFM) [8], and Pitch Adaptive Mel-
Frequency Cepstral Coefficient (PAMFCC) [9]. Recently, deep learning
techniques have become state-of-the-art in speech recognition systems,
however, training such models needs a considerably large amount of speech data
which is not available for child speech. Therefore, different domain
adaptation and data augmentation techniques have been explored that
incorporate both adult and child speech corpora such as teacher-student domain
adaptation [10], transfer learning [11, 12], and Multi-Task Learning (MTL)
[13].

Several works have studied the effect of age on the performance of child
speech recognition. As expected, the performance degraded with a decrease of
age when either an acoustic model trained on adult speech [5] or age-specific
acoustic model [14] was used. Due to the limited availability of child speech
corpora, it is hard to train an accurate acoustic model for each age range. In
[1] acoustic model adaptation was used to adapt an adult acoustic model to the
different age ranges. An age-dependent speaker normalisation technique was
proposed in [15] using subglottal resonances.

Adversarial multi-task learning [16] has been used in literature for speaker
invariant training of adult speech [17, 18]. However, acoustic variations in
child speech are caused by both speaker and age variations due to the rapid,
non-uniform growth of their articulators. In this paper, we thus investigated
whether an adversarial multi-task learning approach can be used to alleviate
the effect of both speaker and age acoustic variations in child speech. To
achieve this, we proposed a system that uses two adversarial tasks, one for
age and one for speaker discrimination, to generate speaker- and age-invariant
features. Unlike most existing adversarial multi-task learning architectures
where only one adversarial task was learnt jointly with the main task, the
proposed architecture uses two adversarial tasks that were simultaneously
trained along with the main phonetic discrimination task. Moreover, this is
the first work to address both age and speaker variations in child speech
using adversarial training. The system is validated using child speech corpus
with a large number of speakers distributed over 11 age groups.

##  2 Method

Since Goodfellow presented the Generative Adversarial Network (GAN) as a novel
method to generate samples from a target distribution [19], a variety of
adversarial learning techniques have been proposed in literature including
adversarial multi-task learning [16]. In traditional multi-task learning, a
second different but relevant task is trained sharing part of the network,
with the primary task to improve the generalization of the primary task [20].
In contrast, in adversarial multi-task learning, the shared network is learnt
adversarially to the secondary task, i.e., to not discriminate between classes
in the secondary task, resulting in representations invariant to the secondary
task. Adversarial training has been successfully utilized to improve the
robustness of speech recognition systems against noisy environments [21],
speaker variations [17, 18], and accent variation [22]. Here we leverage
adversarial multi-task learning to handle the high speaker- and age-variations
in child speech.

Figure 1 depicts our proposed architecture of the adversarial multi-task
learning network. The architecture consists of four subnetworks: the
generative network (G) which is the core network and acts as the feature
extraction network, the phoneme recognition network (P) which is trained to
classify senones, the speaker discrimination network (S) which is trained to
discriminate between speech from different speakers, and the age group
discrimination network (A) which is trained to discriminate between different
age groups. The G network consists of $\eta_{G}$ Time Delay Neural Network
(TDNN) layers similar to the one proposed in [23] with $\theta_{G}$ trainable
parameters, while the networks P, S, and A are formed from $\eta_{A}$,
$\eta_{S}$, and $\eta_{A}$ feed forward layers with $\theta_{A}$,
$\theta_{S}$, and $\theta_{A}$ trainable parameters respectively. Moreover,
the network has three output layers associated with the P, S, and A networks
with output neurons equal to the number of tied-state triphones (senones),
speakers, and age groups respectively.

Fig. 1: Adversarial multi-task learning architecture consisting of four
networks: the generative network (G) of $\eta_{G}$ TDNN layers and trainable
parameters $\theta_{G}$, the phoneme recognition network (P) of $\eta_{P}$
feed forward layers and trainable parameters $\theta_{P}$, the speaker
discrimination network (S) of $\eta_{S}$ feed forward layers and $\theta_{S}$
trainable parameters, and the age group discrimination network (A) of
$\eta_{A}$ feed forward layers and $\theta_{A}$ trainable parameters.

As shown in Figure 1, the G network is shared among the three tasks, while
each of the P, S, and A networks is associated with a separate task.
Therefore, the cross-entropy loss function of each task is computed as follows

$L_{P}(\theta_{G},\theta_{P})=-\sum_{i}P(y_{P}^{(i)}|x^{(i)};\theta_{G},\theta_{P})$
(1)

$L_{S}(\theta_{G},\theta_{S})=-\sum_{i}P(y_{S}^{(i)}|x^{(i)};\theta_{G},\theta_{S})$
(2)

$L_{A}(\theta_{G},\theta_{A})=-\sum_{i}P(y_{A}^{(i)}|x^{(i)};\theta_{G},\theta_{A})$
(3)

where $x^{(i)}$ is the input feature vector of sample $i$, and $y_{P}^{(i)}$,
$y_{S}^{(i)}$, and $y_{A}^{(i)}$ are the associated senones, speaker, and age
group labels. During training, the P, S, and A parameters were updated such
that their associated loss are minimized as follows

$\theta_{P}\leftarrow\theta_{P}-\eta\frac{\partial L_{P}}{\partial\theta_{P}}$
(4)

$\theta_{S}\leftarrow\theta_{S}-\eta\frac{\partial L_{S}}{\partial\theta_{S}}$
(5)

$\theta_{A}\leftarrow\theta_{A}-\eta\frac{\partial L_{A}}{\partial\theta_{A}}$
(6)

The G network is trained to minimize the senones classification loss and
maximize both the speaker and age group discrimination losses allowing the
network to generate features that are invariant to speaker and age variations.
The parameters of the G network are therefore updated as follows

$\theta_{G}\leftarrow\theta_{G}-\eta(\frac{\partial
L_{P}}{\partial\theta_{G}}-\alpha_{S}\frac{\partial
L_{S}}{\partial\theta_{G}}-\alpha_{A}\frac{\partial
L_{A}}{\partial\theta_{G}})$ (7)

where $\eta$ is the learning rate and $\alpha_{S}$ and $\alpha_{A}$ are
positive scalars denoting the portions of the speaker and age group
adversarial tasks’ gradients backpropagated to update the generator’s
parameters. This idea was achieved using the Gradient Reversal Layer (GRL)
proposed by Ganin [24]. As shown in Figure 1, the GRL layer works as an
identity-mapping in the forward path while in the backpropagation path it
reverses and scales gradient by multiplying it with $-\alpha$

Equations (5), (6), and (7) demonstrate that the generator and the speaker and
age discriminators play a minimax game. The generator tries to maximize the
losses of the speaker and age classification tasks ($L_{S}$ and $L_{A}$) while
the speaker and age discriminator networks try to minimize these losses. Most
existing implementations of adversarial multi-task learning in speech acoustic
modelling utilise only one adversarial task [24], In the implementation of our
proposed architecture, two adversarial tasks were trained simultaneously along
with the main task.

Algorithm 1 ADVERSARIAL MULTI-TASK LEARNING

Features: $X={x^{(1)},x^{(2)},\ldots,x^{(n)}}$

Senon Label Sequence: $y_{P}={y_{P}^{(1)},y_{P}^{(2)},\ldots,y_{P}^{(n)}}$

Speaker Label Sequence: $y_{S}={y_{S}^{(1)},y_{S}^{(2)},\ldots,y_{S}^{(n)}}$

Age Label Sequence: $y_{A}={y_{A}^{(1)},y_{A}^{(2)},\ldots,y_{A}^{(n)}}$

for $r=0,1,\ldots,N-1$ do

for $Epoch=1,2,\ldots,m$ do

$L_{P}(\theta_{G},\theta_{P})=-\sum_{i}\log
P(y_{P}^{(i)}|x^{(i)};\theta_{G},\theta_{P})$

Compute $\frac{\partial L_{P}}{\partial\theta_{P}}$ & $\frac{\partial
L_{P}}{\partial\theta_{G}}$

$\theta_{P}\leftarrow\theta_{P}-\eta\frac{\partial L_{P}}{\partial\theta_{P}}$

$\theta_{G}\leftarrow\theta_{G}-\eta\frac{\partial L_{P}}{\partial\theta_{G}}$

end for

for $Epoch=1,2,\ldots,m$ do

$L_{S}(\theta_{G},\theta_{S})=-\sum_{i}\log
P(y_{S}^{(i)}|x^{(i)};\theta_{G},\theta_{S})$

$L_{A}(\theta_{G},\theta_{A})=-\sum_{i}\log
P(y_{A}^{(i)}|x^{(i)};\theta_{G},\theta_{A})$

Compute $\frac{\partial L_{S}}{\partial\theta_{S}}$ & $\frac{\partial
L_{A}}{\partial\theta_{A}}$

$\theta_{S}\leftarrow\theta_{S}-\eta\frac{\partial L_{S}}{\partial\theta_{S}}$

$\theta_{A}\leftarrow\theta_{A}-\eta\frac{\partial L_{A}}{\partial\theta_{A}}$

end for

for $Epoch=1,2,\ldots,m$ do

$L_{P}(\theta_{G},\theta_{P})=-\sum_{i}\log
P(y_{P}^{(i)}|x^{(i)};\theta_{G},\theta_{P})$

$L_{S}(\theta_{G},\theta_{S})=-\sum_{i}\log
P(y_{S}^{(i)}|x^{(i)};\theta_{G},\theta_{S})$

$L_{A}(\theta_{G},\theta_{A})=-\sum_{i}\log
P(y_{A}^{(i)}|x^{(i)};\theta_{G},\theta_{A})$

Compute $\frac{\partial L_{P}}{\partial\theta_{P}}$ & $\frac{\partial
L_{S}}{\partial\theta_{S}}$ & $\frac{\partial L_{A}}{\partial\theta_{A}}$

$\theta_{G}\leftarrow\theta_{G}-\eta(\frac{\partial
L_{P}}{\partial\theta_{G}}-\alpha_{S}\frac{\partial
L_{S}}{\partial\theta_{G}}-\alpha_{A}\frac{\partial
L_{A}}{\partial\theta_{G}})$

end for

end for

The training procedure is summarised in Algorithm 1. Firstly, both generator
(G) and senone classification (P) networks are trained for $m$ epochs to
minimise the senone output loss $L_{P}$ allowing G to generate senone-
discriminative features. In the next $m$ epochs, the network G is frozen, so
that its parameters are not updated, and both the speaker and age group
discrimination networks trained to minimize their associated losses ($L_{S}$
and $L_{A}$). In the final $m$ epochs, the P, S, and A networks are frozen,
and the generator is trained to minimize the senone classification loss
$L_{P}$ and maximise the speaker and age discriminator losses ($L_{S}$ and
$L_{A}$). Therefore, the generator learns to deceive the well-trained speaker
and age discriminators resulting in speaker- and age-invariant features. These
steps are typically repeated $N$ times.

##  3 EXPERIMENTAL SETUP

###  3.1 Dataset

The Oregon Graduate Institute (OGI) kids’ speech corpus [25] was used in this
work. It was collected from 1110 children distributed over 11 age groups based
on their school grade from kindergarten to grad 10. Most of the recordings are
scripted, i.e., each child read a set of prompt words and short sentences.
Additionally, spontaneous speech was elicited by answering a set of open
questions. In this work only the scripted part of the dataset was utilized.
The speech corpus was split into three parts, 794 speakers for training, 158
speakers for developing, and 158 speakers for testing.

###  3.2 Training procedure

Each speech sample was divided into 25 msec frames with 15 msec overlap. Each
frame was first multiplied by a Hamming window and 40 Mel-scale cepstral
coefficients were extracted from each frame.

The tied-state triphones (senones) alignment was obtained using a GMM-HMM
initial acoustic model trained on the same speech corpus. Therefore, three
labels were assigned to each frame, the speaker index, the age group index,
and the senone index. The baseline model consisted of 5 TDNN layers with
delays of -2,-1,0,1,2, -1,2, -3,3, -3,3, and -7,2 each with 1024 neurons
followed by a feed forward fully connected layer of 1024 neurons. The
effective temporal context of the model is 39 frames before and after the
underlying frame (-39, +39). The final output layer had 1360 neurons
representing the number of senones.

For the adversarial multi-task learning architecture depicted in Figure 1, the
G network was similar to the baseline model with the same 5 TDNN layers while
each of the P, S, and A networks had one feed forward fully connected layer in
addition to the last output layer. The numbers of neurons of the output layers
of the P, S, and A discriminators were 1360, 794, and 11 representing the
number of senones, training speakers, and age groups respectively.

As shown in Figure 1, the P discriminator receives input directly from the
output of the last TDNN layer of the G network. On the contrary, both S and A
discriminators receive inputs through GRL layers with scaling factors
$\alpha_{s}$ and $\alpha_{A}$ respectively. The scaling factors $\alpha_{s}$
and $\alpha_{A}$ were gradually increased from 0 to 0.01 as follows

$\alpha_{S}=\alpha_{A}=\frac{r}{N-1}\times\alpha\;\;\;\;r=0,1,\ldots,N-1$ (8)

where $r$,$N$ are the repeat index and the total number of repeats
respectively as explained in Algorithm 1 and $\alpha$ is the maximum value
that is reached at repeat $N-1$.

The effect of each adversarial task separately was further investigated by
training two separate models with each having only one adversarial task. The
acoustic model was evaluated in terms of the Word Error Rate (WER) in an ASR
task. The language model used was a 4-gram model trained on the transcripts of
the OGI data.

##  4 Results

Four different models were trained: the baseline model, the AGE model, the SPK
model and the AGE+SPK model.

In the AGE model, the speaker discriminator (S) was removed, and the G network
was trained adversarially against the age group discriminator (A) only. In
contrast, for the SPK model, only the speaker discriminator was used in the
adversarial update of the G network parameters. Finally, the AGE+SPK model
utilized both discriminators. All four acoustic models were used to achieve
ASR and were tested using the test and development subsets of the OGI dataset.
For all models $\alpha_{max}$ was set to 0.01.

Fig. 2: The effect of the adversarial multi-task learning on the WER of the
ASR trained on the OGI speech corpus. The baseline model was trained with no
adversarial update to the parameters. The AGE and SPK with only the age
discriminator or the speaker discriminator were utilized respectively, whereas
AGE+SPK when both discriminators were used.

Figure 2 shows the WER of the test and development subsets of the four models.
The baseline model achieved WERs of 14.4% and 16% respectively. Training the
model adversarially against the speaker discriminator (SPK) reduced the WER by
8% and 10% to 13.47% and 14.64% for the test and development subsets
respectively. The AGE model gave slightly better improvement compared with the
SPK model, with WERs of 13.26% and 14.42% for the test and development subsets
respectively. The best performance was obtained when both speaker and age
discriminators were used with WER relative reductions of 10% and 13% for the
test and development subsets respectively. These results indicate that the age
variation has more influence on the performance of the acoustic model than the
speaker variation.

To better interpret these results, we broke them down to age level as depicted
in Figure 3. It can be seen that WER reduced significantly in grades K and 1
compared to elder ages. It is noted also that at grade K using only AGE
adversarial training achieved the lower WER than using SPK or AGE+SPK models.
This indicates that there is a high degree of discrepancy between children at
kindergarten and elder ages caused mainly due to the age variations which have
been successfully alleviated using AGE adversarial training.

Fig. 3: Age breakdown of the performance of the four models, Baseline, AGE,
SPK, and AGE+SPK

Figure 4 shows the effect of the $\alpha$ parameter which represents the
portion of the adversarial gradient backpropagated to update the generator
network parameters. Three values of $\alpha$ were tested, namely 0.1, 0.01,
and 0.001. Figure 4 also demonstrates the convergence of the model with the
repeats (see Algorithm 1). The WER of both the validation and testing sets are
computed after each repeat. Note that as formulated in equation (8), at repeat
0 the values of $\alpha_{S}$ and $\alpha_{A}$ equals to 0 for all $\alpha$
values and then increased gradually at each repeat, therefore, the WER of the
three $\alpha$ values starts from the same point at 15.5% and 17.5% WER for
the testing and validation sets respectively.

The results show that when using $\alpha=0.1$, the WER slightly decreased at
first repeat when the effective values of $\alpha_{S}$ and $\alpha_{A}$ were
0.01 and then almost saturated as the effective values of $\alpha_{S}$ and
$\alpha_{A}$ increased. On the other hand, at $\alpha=0.001$ and 0.01, the WER
decreased at each repeat and almost saturated at repeat 5 with 13.5% WER (test
set) and 15% WER (validation), and at repeat 8 with 13% WER (test set), and
14% WER (validation), for $\alpha=0.001$ and $\alpha=0.01$ respectively. These
results demonstrate that selecting the amount of reversed gradient used to
update the feature generator network is crucial in the effectiveness of the
training.

Fig. 4: The WER of the AGE+SPK model at different values of $\alpha$ parameter
over each repeat.

##  5 CONCLUSION

In this paper, a speaker- and age-invariant training approach was proposed
leveraging the adversarial multi-task learning paradigm for child acoustic
modelling to generate the speaker and age acoustic variations. A feature
extraction network was learnt to generate phonetic discriminative features
that are invariant to speaker and age variations. This was achieved by
training the generator to minimize the phoneme discrimination loss and, at the
same time, maximize the speaker and age discrimination losses. This approach
boosted the ASR word recognition accuracy from 85% to 87% when tested on the
Oregon Graduate Institute (OGI) test set.

This shows that the two-way adversarial multi-task learning was effective in
combining data from different ages for the training of age-independent
acoustic model. The age analysis showed that the improvement was significant
in recognizing speech of kindergarten children compared to elder ages.

## References

  * [1] Alexandros Potamianos and Shrikanth Narayanan,  “Robust Recognition of Children’s Speech,”  IEEE Transactions on Speech and Audio Processing, vol. 11, no. 6, pp. 603–616, nov 2003.
  * [2] Ryoko Mugitani and Sadao Hiroya,  “Development of vocal tract and acoustic features in children,”  Acoustical Science and Technology, vol. 33, no. 4, pp. 215–220, 2012.
  * [3] W. Tecumseh Fitch and Jay Giedd,  “Morphology and development of the human vocal tract: a study using magnetic resonance imaging,”  The Journal of the Acoustical Society of America, vol. 106, no. 3 Pt 1, pp. 1511–1522, sep 1999.
  * [4] Sungbok Lee, Alexandros Potamianos, and Shrikanth Narayanan,  “Acoustics of children’s speech: Developmental changes of temporal and spectral parameters,”  The Journal of the Acoustical Society of America, vol. 105, no. 3, pp. 1455, mar 1999.
  * [5] Alexandros Potamianos, Shrikanth Narayanan, and Sungbok Lee,  “Automatic speech recognition for children,”  in Proc. 5th European Conference on Speech Communication and Technology (Eurospeech 1997), 1997, pp. 2371–2374.
  * [6] Andreas Hagen, Bryan Pellom, and Ranald Cole,  “Children’s speech recognition with application to interactive books and tutors,”  2003 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2003, pp. 186–191, 2003.
  * [7] Romain Serizel and Diego Giuliani,  “Vocal tract length normalisation approaches to dnn-based children’s and adults’ speech recognition,”  in 2014 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2014, pp. 135–140.
  * [8] Joachim Fainberg, Peter Bell, Mike Lincoln, and Steve Renals,  “Improving children’s speech recognition through out-of-domain data augmentation.,”  in Interspeech, 2016, pp. 1598–1602.
  * [9] S Shahnawazuddin, Abhishek Dey, and Rohit Sinha,  “Pitch-Adaptive Front-end Features for Robust Children’s ASR,”  2016\.
  * [10] Jinyu Li, Michael L. Seltzer, Xi Wang, Rui Zhao, and Yifan Gong,  “Large-Scale Domain Adaptation via Teacher-Student Learning,”  Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 2017-August, pp. 2386–2390, aug 2017.
  * [11] Romain Serizel and Diego Giuliani,  “Deep neural network adaptation for children’s and adults’ speech recognition,”  .
  * [12] Prashanth Gurunath Shivakumar and Panayiotis Georgiou,  “Transfer learning from adult to children for speech recognition: Evaluation, analysis and recommendations,”  Computer Speech & Language, vol. 63, pp. 101077, sep 2020.
  * [13] Jiarui Wang, Si Ioi Ng, Dehua Tao, Wing Yee Ng, and Tan Lee,  “A study on acoustic modeling for child speech based on multi-task learning,”  2018 11th International Symposium on Chinese Spoken Language Processing, ISCSLP 2018 - Proceedings, pp. 389–393, jul 2018.
  * [14] Gary Yeung and Abeer Alwan,  “On the Difficulties of Automatic Speech Recognition for Kindergarten-Aged Children,”  .
  * [15] Jinxi Guo, Rohit Paturi, Gary Yeung, Steven M Lulich, Harish Arsikere, and Abeer Alwan,  “Age-dependent height estimation and speaker normalization for children’s speech using the first three subglottal resonances,”  .
  * [16] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky,  “Domain-Adversarial Training of Neural Networks,”  Advances in Computer Vision and Pattern Recognition, , no. 9783319583464, pp. 189–209, may 2015.
  * [17] George Saon, Gakuto Kurata, Tom Sercu, Kartik Audhkhasi, Samuel Thomas, Dimitrios Dimitriadis, Xiaodong Cui, Bhuvana Ramabhadran, Michael Picheny, Lynn Li Lim, Bergul Roomi, and Phil Hall,  “English Conversational Telephone Speech Recognition by Humans and Machines,”  Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 2017-August, pp. 132–136, mar 2017\.
  * [18] Zhong Meng, Jinyu Li, Zhuo Chen, Yong Zhao, Vadim Mazalov, Yifan Gong, and Biing-Hwang Juang,  “SPEAKER-INVARIANT TRAINING VIA ADVERSARIAL LEARNING,”  .
  * [19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio,  “Generative adversarial networks,”  Communications of the ACM, vol. 63, no. 11, pp. 139–144, 2020.
  * [20] Rich Caruana, Lorien Pratt, and Sebastian Thrun,  “Multitask Learning,”  Machine Learning 1997 28:1, vol. 28, no. 1, pp. 41–75, 1997.
  * [21] Yusuke Shinohara,  “Adversarial Multi-task Learning of Deep Neural Networks for Robust Speech Recognition,”  2016\.
  * [22] Sining Sun, Ching Feng Yeh, Mei Yuh Hwang, Mari Ostendorf, and Lei Xie,  “Domain adversarial training for accented speech recognition,”  ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, vol. 2018-April, pp. 4854–4858, sep 2018.
  * [23] David Snyder, Daniel Garcia-Romero, and Daniel Povey,  “Time delay deep neural network-based universal background models for speaker recognition,”  2015 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2015 - Proceedings, pp. 92–97, feb 2016.
  * [24] Yaroslav Ganin, Victor Lempitsky, and Lempitsky@skoltech Ru,  “Unsupervised Domain Adaptation by Backpropagation,” jun 2015.
  * [25] Khaldoun Shobaki, John Paul Hosom, and Ronald A. Cole,  “The OGI Kids’ speech corpus and recognizers,”  6th International Conference on Spoken Language Processing, ICSLP 2000, 2000.