The internal Faraday screen of Sagittarius A*

Maciek Wielgus

Sara Issaoun3,4,
Iván Martí-Vidal5,6,
Razieh Emami3,
Monika Moscibrodzka7,

Christiaan D. Brinkerink7,
Ciriaco Goddi8,9,10,11,
Ed Fomalont12

Max-Planck-Institut für Radioastronomie, Auf dem Hügel 69, D-53121 Bonn, Germany

Institute of Physics, Silesian University in Opava, Bezručovo nám. 13, CZ-746 01 Opava, Czech Republic
Center for Astrophysics | Harvard & Smithsonian, 60 Garden Street, Cambridge, MA 02138, USA
NASA Hubble Fellowship Program, Einstein Fellow
Departament d'Astronomia i Astrofísica, Universitat de València, C. Dr. Moliner 50, E-46100 Burjassot, València, Spain
Observatori Astronòmic, Universitat de València, C. Catedrático José Beltrán 2, E-46980 Paterna, València, Spain
Department of Astrophysics, Institute for Mathematics, Astrophysics and Particle Physics (IMAPP), Radboud University, P.O. Box 9010, 6500 GL Nijmegen, The Netherlands
Instituto de Astronomia, Geofísica e Ciências Atmosféricas, Universidade de São Paulo, R. do Matão, 1226, São Paulo, SP 05508-090, Brazil
Dipartimento di Fisica, Universitá degli Studi di Cagliari, SP Monserrato-Sestu km 0.7, I-09042 Monserrato (CA), Italy
INAF - Osservatorio Astronomico di Cagliari, via della Scienza 5, I-09047 Selargius (CA), Italy
INFN, sezione di Cagliari, I-09042 Monserrato (CA), Italy
National Radio Astronomy Observatory, 520 Edgemont Road, Charlottesville, VA 22903, USA

We report on 85-101 GHz light curves of the Galactic Center supermassive black hole, Sagittarius A* (), observed in April 2017 with the Atacama Large Millimeter/submillimeter Array (ALMA). This study of high-cadence full-Stokes data provides new measurements of the fractional linear polarization at a 1-2% level resolved in 4 s time segments, and stringent upper limits on the fractional circular polarization at 0.3%. We compare these findings to ALMA light curves of at 212-230 GHz observed three days later, characterizing a steep depolarization of the source at frequencies below about 150 GHz. We obtain time-dependent rotation measure (RM) measurements, with the mean RM at 85-101 GHz being a factor of two lower than that at 212-230 GHz. Together with the rapid temporal variability of the RM and its different statistical characteristics in both frequency bands, these results indicate that the Faraday screen in is largely of internal character, with about half of the Faraday rotation taking place inside the inner 10 gravitational radii, contrary to the common external Faraday screen assumption. We then demonstrate how this observation can be reconciled with theoretical models of radiatively inefficient accretion flows for a reasonable set of physical parameters. Comparisons with numerical general relativistic magnetohydrodynamic simulations suggest that the innermost part of the accretion flow in is much less variable than what these models predict, in particular, the observed magnetic field structure appears to be coherent and persistent.

The internal Faraday screen of Sagittarius A*
Wielgus et al.

§ INTRODUCTION

Sagittarius A* () is the radio source associated with a 4$\times$10$^6 M_{\odot}$ supermassive black hole (SMBH) located in our Galactic Center [Do et al., 2019, Gravity Collaboration et al., 2022, EHTC et al., 2022].
The source is characterized by a particularly low mass accretion rate of $\sim 10^{-8} M_{\odot}{\rm yr}^{-1}$ [Quataert & Gruzinov, 2000, Yuan et al., 2003, EHTC et al., 2022]. Spectral energy distribution (SED) analysis allowed to identify it as an advection dominated / radiatively inefficient type of accretion flow <cit.>. The SED of exhibits a maximum at a turnover frequency in the range of several hundred GHz that can be attributed to synchrotron emission from the hot thermal electrons in the marginally optically thin innermost region of the accretion disk [Yuan et al., 2003]. Despite significant progress on the theoretical front and multiple observational studies across the electromagnetic spectrum since was first identified in 1974 [Balick & Brown, 1974], our detailed understanding of this object and the accretion flow surrounding it remains incomplete <cit.>.

Overview of the ALMA light curves of obtained in band 3 (left column) and band 6 (right column), each with four 2 GHz-wide sub-bands, spanning 85-101 GHz and 212-230 GHz, respectively. A significant reduction of the fractional polarization and an increase of the EVPA variability in the lower frequency band are visible.

Very long baseline interferometric (VLBI) radio observations indicate that the intrinsic size of decreases with frequency, reaching event horizon scales at $\sim$230 GHz <cit.>. At $\sim$90 GHz the intrinsic diameter of the VLBI image corresponds to about 20 gravitational radii $r_{\rm g} = GM_{\sbullet[1.35]}/c^2$ [Shen et al., 2005, Bower et al., 2006, Lu et al., 2011, Issaoun et al., 2019], hence the emission originates in large part from a layer external to the innermost scales $\lesssim 5 r_{\rm g}$. Therefore, observations at different radio frequencies can be used to probe distinct regions of the accretion flow. Linear (LP) and circular (CP) polarization, rotation measure (RM), as well as temporal variability of these observed quantities, provide additional constraints on models <cit.>.

The CP of was first reported by Bower et al., 1999 at 4.8 GHz, while the first LP detections were obtained by Aitken et al., 2000 in the 150-400 GHz range. Multiple subsequent observations showed LP values of $\sim$5-10% at around 230 GHz [Bower et al., 2003, Marrone et al., 2007, Bower et al., 2018, Wielgus et al., 2022]. The source becomes strongly depolarized at lower frequencies, and there is a very limited number of LP detections below 150 GHz in the literature [Bower et al., 1999, Macquart et al., 2006, Liu et al., 2016]. Measurements of weak negative CP were reported at frequencies below 20 GHz [Bower et al., 1999, Bower et al., 2002]. Above 200 GHz a stronger CP $\sim -1\%$ appears [Muñoz et al., 2012, Bower et al., 2018, Wielgus et al., 2022]. Only weak upper limits were known for intermediate frequencies so far [Tsuboi et al., 2003].

In this paper we present a study of ALMA light curves in the 85-101 GHz range (ALMA band 3), including unambiguous high-time-cadence detections of variable LP, stringent upper limits on CP, and time-resolved measurements of Faraday rotation. We compare these results to the study of 212-230 GHz (ALMA band 6) light curves obtained in a quasi-contemporaneous epoch [Wielgus et al., 2022, Wielgus et al., 2022]. At both frequency bands we build on the results of Goddi et al., 2021, where a preliminary analysis of polarization in the same ALMA observations, reduced under an unphysical static source assumption, was presented. The main consequence of our results for the system is the observationally-demonstrated presence of a significant internal component of the Faraday screen, which must be located within the central 10 $r_{\rm g}$ region (Section <ref>). We discuss the implications of this measurement for the RIAF accretion flow model, utilizing VLBI observations to constrain the radial distribution of the electron temperature (Section <ref>). We also compare the measurements with predictions from general relativistic magnetohydrodynamic (GRMHD) simulations (Section <ref>). A brief summary is given in Section <ref>.

§ OBSERVATIONS AND DATA ANALYSIS

Summary of the light curves observed with ALMA in April 2017

4c Band 3, 2017 Apr 3          4c Band 6, 2017 Apr 6

$\nu_{\rm obs}$ (GHz)     86.3     88.3     98.3     100.3         213.1     215.1     227.1$^a$     229.1

$\mathcal{I}$ (Jy)     $1.91\pm 0.09$     $1.87 \pm 0.09$     $1.91\pm 0.10$     $1.93\pm 0.10$          $2.62 \pm 0.09$     $2.62 \pm 0.09$      $2.62 \pm 0.09$      $2.61 \pm 0.09$

$|\mathcal{P}|$ (mJy)     $23.2\pm 9.5$     $23.9\pm10.3$     $33.4\pm 15.2$     $36.3\pm 16.7$         $194.1\pm 55.5$     $195.4\pm 55.6$     $206.9\pm 55.5$     $208.3\pm 55.1$

$p$ (%)     $1.23\pm 0.51$     $1.28 \pm 0.56$     $1.79\pm 0.81$     $1.89\pm 0.87$
$7.45\pm 2.26$     $7.50\pm 2.25$     $7.93\pm 2.26$     $8.03 \pm 2.28$

$\chi^b$ (deg)     $-19.4 \pm 44.1$     $-15.1 \pm 41.3$     $8.7\pm 37.6$     $13.6\pm 38.5$
$109.9\pm 11.7$     $110.8\pm 11.6$     $116.4\pm 11.2$     $117.3\pm 10.9$

$\mathcal{V}$ (mJy)     $0.4\pm 4.1$     $0.1 \pm 4.0$     $-0.6\pm 4.4$     $-2.3\pm 4.6$
$-39.4\pm 11.1$     $-39.3\pm 11.1$     $-39.9 \pm 11.5$     $-41.2 \pm 10.7$

$v$ (%)$^c$     $0.0 \pm 0.2$     $0.0 \pm 0.2$     $0.0\pm 0.2$     $-0.1 \pm 0.2$
$-1.50 \pm 0.41$     $-1.50 \pm 0.40$     $-1.52 \pm 0.42$     $-1.57 \pm 0.38$

$\alpha_{\mathcal{I}}$     4c $0.10 \pm 0.01$          4c $-0.01 \pm 0.01$

$\alpha_{p}$     4c $2.97 \pm 0.06$          4c $0.91 \pm 0.02$

$\alpha_{v}$     4c —           4c $0.50 \pm 0.02$

RM$^d$     4c $-2.14 \pm 0.51 $         4c $-5.04 \pm 0.83 $

$^a$scaled up by 4% to account for the CN absorption line <cit.>;    $^b$calculated using directional statistics;

$^c$band 3 values are consistent with nondetection, upper limit of $|v| < 0.3\%$;  $^d$ in the units of $10^5$ rad m$^{-2}$

Time-dependent fractional LP measurements obtained at 86.3, 100.3, and 229.1 GHz presented on the $\mathcal{Q}/\mathcal{I}-\mathcal{U}/\mathcal{I}$ plane of linear polarization, compared with the past measurements at the corresponding frequencies. Colors denote the time progression, following the colorbars. Polar coordinates indicate fractional polarization (in %) and EVPA (in degrees; $0^\circ \equiv 180^\circ$ corresponds to EVPA aligned with the North-South axis).

§.§ Data reduction and conventions

The high sensitivity of ALMA enabled detailed recent studies of the rapid variability of [Iwata et al., 2020, Murchikova & Witzel, 2021, Wielgus et al., 2022]. In April 2017, was observed by ALMA during its participation in the VLBI campaigns with the Global mm-VLBI Array <cit.> and with the Event Horizon Telescope <cit.> as a compact phased array [Matthews et al., 2018, Goddi et al., 2019]. Algorithms enabling the extraction of ALMA-only time-dependent light curves of the compact source from these data were developed and extensively discussed in Wielgus et al., 2022. The method is based on the well-motivated assumption of negligible variation of the arc-minute scale radio emission surrounding [Lo & Claussen, 1983, Mus et al., 2022], allowing to self-calibrate to the extended source structure and  extract the time-dependent point source component, reducing the impact of fluctuating amplitude gains. This robust approach, employed following the ALMA QA2 calibration [Goddi et al., 2019], has been applied to band 6 observations at 212-230 GHz, obtained on 2017 April 6, 7, and 11, and presented in Wielgus et al., 2022, Wielgus et al., 2022. In this paper, we employ a very similar calibration algorithm to the data obtained on 2017 April 3 in ALMA band 3, with 4 frequency sub-bands, each 2 GHz wide, centered at 86.3, 88.3, 98.3, and 100.3 GHz. The sub-band depolarization and decorrelation effects are negligible in all cases, hence we work with the data averaged in sub-bands.

The only difference with respect to the procedure of Wielgus et al., 2022 was motivated by the wide total fractional bandwidth in band 3 observations $\sim$15%, as opposed to $\sim$7% in band 6. In order to account for the frequency dependence of the extended source structure across a wider frequency coverage we tested two approaches: carrying out an independent CLEAN deconvolution of each sub-band and using a CLEAN multi-frequency-synthesis first-order expansion [Conway et al., 1990, McMullin et al., 2007]. Both variants of the extended emission modeling resulted in very consistent light curves of the compact component, fitted individually in each sub-band. The latter approach was adopted for the final data set presented in this paper.

The data set contains all Stokes parameters, that is, total intensity $\mathcal{I}$, linear polarization $\mathcal{P} = \mathcal{Q} + i\mathcal{U}$, and circular polarization $\mathcal{V}$, observed with a time cadence of 4 s. We define the fractional LP $p$ and fractional CP $v$ as

\begin{equation}
p = \frac{|\mathcal{P}|}{\mathcal{I}} = \frac{\sqrt{\mathcal{Q}^2 + \mathcal{U}^2}}{\mathcal{I}}   \ \ ; \ \ v = \frac{\mathcal{V}}{{\mathcal{I}}} \ .
\end{equation}

Furthermore, we define the electric vector position angle (EVPA) as $\chi = 0.5 {\rm Arg}(\mathcal{Q} + i\mathcal{U})$.

§.§ Intensity and polarization

The obtained light curves are summarized in Fig. <ref>, with mean values and standard deviations reported in Tab. <ref>. The reported uncertainties are strongly dominated by the intrinsic source variability, with the effective signal-to-noise ratio (S/N) of each 4 s ALMA measurement $\sim$200 for the Stokes $\mathcal{I}$ component [Wielgus et al., 2022]. We focus on comparisons between band 3 observations on 2017 April 3 and band 6 observations on 2017 April 6, given that these light curves are closest in time and of similar total duration. Generally similar mean parameters were found on other days of the band 6 observations <cit.>, with some discrepancies highlighted in Appendix <ref>. We employed an identical flagging procedure for each data set presented in Fig. <ref>, in order to remove points for which the calibration procedures did not converge. The sparser time coverage after $\sim$12:30 UT reflects the decreasing elevation of at ALMA by the end of the observing epoch. We verified that flagging these data does not appreciably impact the reported results. Our physically-motivated analysis (Figs. <ref>-<ref>) confirms that there is a remarkable change in the degree of polarization between the two ALMA bands, first reported under a static source assumption by Goddi et al., 2021. Furthermore, the EVPA is significantly more variable at 85-101 GHz than at 212-230 GHz on the same timescales (Figs. <ref>-<ref>).

Spectral dependence of fractional polarization, fitted with a power law $p(\nu) \propto \nu^\alpha$. Both LP and CP measurements indicate a rapid depolarization with decreasing frequency at about 100 GHz (2017 Apr 3), in contrast to a weak dependence of fractional polarization at observing frequencies above 200 GHz (2017 Apr 6). Error bars shown represent the standard deviation in the samples and are dominated by the intrinsic source variability, while the formal uncertainties of the spectral indices fits are very small, see Tab. <ref>.

Left center: Time-resolved RM measurements in ALMA band 3 (85-101 GHz, blue) and band 6 (212-230 GHz, orange) in April 2017, and corresponding histograms of the observed RM. Dashed lines represent mean values. The rapid variability of RM hints at the internal character of the Faraday screen, which is further supported by a significant discrepancy between the measurements obtained for the two bands, also including historical data, and a joint histogram of the 2017 Apr 6, 7, and 11 data (dashed gray line). Right: SF analysis of the RM variability, indicating more variability at larger frequencies, with relatively more variation on the shortest timescales. A 0.5 h variability decorrelation timescale is found for the 221 GHz RM data, which is absent at 93 GHz.

We also present time-dependent fractional LP vector measurements obtained at 86.3, 100.3, and 229.1 GHz <cit.> in Fig. <ref>. We compare these measurements with the results reported by Macquart et al., 2006 at 86.3 GHz (low S/N measurements not shown), Liu et al., 2016 at 105-107 GHz, Marrone et al., 2007 at 230.6-231.9 GHz, Bower et al., 2018 at 226.0 GHz, and Wielgus et al., 2022 at 229.1 GHz. We observe consistency between these historical measurements and our light curves at 86.3 and 100.3 GHz. The large temporal variations of the EVPA that we observe on a few hours timescale explain EVPA changes reported by Macquart et al., 2006 between subsequent observing days. These variations appear as a full counter-clockwise loop on the plane of linear polarization executed between 8:00 and 11:30 UT on 2017 Apr 3, see the EVPA panel of Fig. <ref> and the first two panels of Fig. <ref>. If we attempted to explain this loop in the framework of a coherent orbital motion, we would conclude Keplerian orbit of $\sim$20$r_{\rm g}$ radius. However, the direction of the loop's direction is opposite to the one observed by Wielgus et al., 2022 and The GRAVITY Collaboration et al., 2023, challenging that interpretation. At 230 GHz, we generally see a slightly increased degree of LP in 2017 with respect to the previous measurements, which can be related to the lower total compact flux density reported in April 2017 <cit.>. The 230 GHz EVPA, while variable, appears to have a preference for values $\chi \sim 120^\circ$ (derotated EVPA $\chi_0 = \chi - {\rm RM}\lambda^2 \sim 170^\circ$), and almost never takes values in the range between $-20^\circ$ and $80^\circ$, which constitutes a potentially powerful constraint on theoretical models, particularly for the on-sky position angle of the system. As an example, for a toy model in which the observed non-zero net LP is a consequence of Doppler boosting of the approaching side of the inclined accretion disk, we would expect the on-sky projected disk spin axis to align with the intrinsic EVPA $\chi_0$ for the predominantly azimuthal magnetic field, and to be perpendicular to $\chi_0$ if the magnetic field is predominantly vertical.

§.§ Depolarization at lower frequencies

A significant change of the fractional polarization with the observing frequency is summarized in Fig. <ref>, where we fitted spectral indices of $p$ and $|v|$, as reported in Tab. <ref>, across ALMA sub-bands. In band 3, we measured a steep depolarization toward lower frequencies $p \propto \nu^3$, inconsistent with the milder depolarization $p \propto \nu$ seen in band 6. Similarly, fractional CP has a weak dependence on the frequency in band 6, inconsistent with the upper limits on $|v|$ that we report in band 3, which we give as $|v| < 0.3\%$, based on the time-resolved analysis. An increase of LP fraction with frequency has been observed in active galactic nuclei (AGN), e.g., Agudo et al., 2014 reported a mean factor of $\sim$1.5 change between 86 and 229 GHz, corresponding to a fractional LP spectral index $\alpha_p \approx 0.5$. A plausible explanation is that the higher frequency emission originates from a more compact region, with a more ordered magnetic field, or that the changes are related to the optical depth variation. However, the effect observed in is far more extreme, hinting at a prominent transition in the system in its innermost part, at around $5-10\,r_{\rm g}$, possibly separating strong and ordered magnetic fields near the event horizon [Johnson et al., 2015] from a weaker and more chaotic component further away. The depolarization could also be caused by the optical depth $\tau$ increasing at lower frequencies, as polarization from the optically thick thermal synchrotron emission is suppressed exponentially with $\tau$ [Pacholczyk, 1970]. Hence, an increase in the optical depth by $\Delta \tau \lesssim 2$ would suffice to explain the change in fractional polarization. However, the opacity interpretation is only straightforward under the assumption of a uniform optical depth in the emission zone, which is likely an oversimplification.

§.§ Faraday rotation

Linearly polarized radiation undergoes a change in EVPA as it propagates through a Faraday screen – the magnetized plasma located between the emitter and the observer. This effect is quantified with rotation measure (RM), which can be defined as

\begin{equation}
{\rm RM} = \frac{{\rm d} \chi }{ {\rm d} \lambda^2 } \approx \frac{\Delta \chi}{ \Delta \lambda^2} = \frac{ \chi(\lambda_2)- \chi(\lambda_1) }{ \lambda^2_2 - \lambda^2_1 }
\label{eq:RM_basic}
\end{equation}

for two observing wavelengths $\lambda_1$, $\lambda_2$. If the Faraday depth is small and the Faraday screen is external with respect to the emitting region, the RM is independent of the observing wavelength and the approximation in Eq. <ref> turns into a strict equality. The intrinsic (derotated) EVPA of the emission may then be calculated as

\begin{equation}
\chi_0 = \chi(\lambda) - {\rm RM} \lambda^2 \ .
\label{eq:intrinsicEVPA}
\end{equation}

On the other hand, measuring inconsistent values of RM at different wavelengths indicate a deviation from the $\lambda^2$ relation, implying either a complex unresolved source structure involving multiple Faraday screens, or an internal Faraday screen overlapping with the emission zone at some wavelengths [Burn, 1966, Brentjens & de Bruyn, 2005]. Such effects have been observed, as an example, in the quasar  3C 273 [Hovatta et al., 2019]. In the case of , VLBI observations reveal a persistent, simple, compact, single-component source morphology across frequencies, hence the internal Faraday screen interpretation is favored.

The RM toward is well established through measurements at frequencies near 230 GHz ($\lambda = c/\nu \approx 1.3$ mm), with ${\rm RM} \approx -5 \times 10^5$ rad m$^{-2}$ [Bower et al., 2003, Marrone et al., 2007, Bower et al., 2018, Wielgus et al., 2022], which is a rather large value when compared to what is typically observed toward AGN sources <cit.>. The RM was shown to fluctuate significantly [Bower et al., 2018], even on very short sub-hour timescales [Wielgus et al., 2022]. This poses a difficulty in estimating the RM based on non-simultaneous EVPA measurements, with both RM and EVPAs fluctuating in time, and the short associated timescales generally point toward the Faraday screen compactness.

There are not many RM measurements in available at different frequencies. They involve estimates at 345 GHz by Marrone et al., 2007, who found a mean RM in excess of $-10^6$ rad m$^{-2}$, but large measurement uncertainties did not allow to exclude consistency with the values observed at 230 GHz. At longer wavelengths, Macquart et al., 2006 calculated RM between 86 GHz and 230 GHz, but concluded that consistency with 230 GHz measurements depended on shifting the (180$^\circ$-periodic) EVPA measurement by 180$^\circ$, and significantly lower RM was estimated without the shift. A preliminary discussion of ALMA bands 3 and 6 RM measurements (under the static source assumption) and their implications was presented in Goddi et al., 2021.

Following our time-dependent calibration of the ALMA observations, we estimated RM as a function of time by fitting a linear model for the EVPA as a function of squared wavelength across 4 sub-bands in band 3, independently for each timestamp. Subsequently, we compared these measurements with the band 6 results, already discussed in Appendix A of Wielgus et al., 2022. While the time-dependent model linear in $\lambda^2$ provides a good fit quality across each ALMA band individually, the resulting RM measurements significantly differ between the two bands. This is shown in Fig. <ref>, where high time cadence RM measurements in $\sim$6 h duration observing windows are presented for the mean band 3 frequency of 93 GHz (2017 April 3) and mean band 6 frequency of 221 GHz (2017 April 6). RM values observed at the lower frequency band are incompatible with both quasi-contemporaneous and historical measurements near 230 GHz, demonstrating a deviation from the $\lambda^2$ relation defined in Eq. <ref>. This implies that about half of the Faraday rotation occurs internally with respect to the 93 GHz emission region, that is, at most several $r_{\rm g}$ away from the SMBH's event horizon. The two measurements give an approximate scaling relation ${\rm RM} \propto r^{-1}$.

Furthermore, in the EVPA panels of Fig. <ref> we show the intrinsic EVPA values $\chi_0(t)$, derotated using the time-dependent RM measurements and Eq. <ref>. If the Faraday screen was external, but variable on short timescales, derotating $\chi(t)$ should reduce its variability. Since the shape of derotated $\chi_0(t)$ and its measured variability remain overall very similar to those of the observed $\chi(t)$, we conclude that the intrinsic variability of the emitter dominates over the variability in the external Faraday screen, consistent with the dominant character of the internal Faraday screen component.

Simultaneous measurements of LP at distinct frequencies are necessary in order to provide an ultimate and bulletproof argument for the $\lambda^2$ relation violation, otherwise one could still attempt to explain our results with an unusually low, but $\lambda$-independent RM on 2017 April 3. Nonetheless, these findings constitute by far the most convincing observational demonstration of an internal Faraday screen component in to date.

§.§ Quantifying the RM variability

We investigate the RM variability with the structure function (SF) approach [Simonetti et al., 1985], which is the time-domain analog of the power spectrum analysis, defined as

\begin{equation}
{\rm SF}(\tau) = \langle ( {\rm RM}(t) - {\rm RM}(t- \tau) )^2 \rangle \ ,
\end{equation}

where the averaging is taken over all times $t$. Hence, the SF effectively splits the observed variation across timescales $\tau$ (lags). The results of the SF analysis are shown in the right panel of Fig. <ref>, where we compare 2017 Apr 3 and 6 analysis, as well as the joint analysis of band 6 RM calculated for Apr 6, 7, and 11, with a total of $\sim$ 20 h of data. We confirm significantly less variation in the lower frequency band across the sampled timescales in the absolute sense. Relative to the mean value, RM fluctuates by $\sim$24% at the lower frequency and by $\sim$16% at the higher one, Tab. <ref>. Furthermore, the short timescale variability slope in band 3 is significantly steeper than the one revealed by the joint analysis of band 6, implying less power at the shortest timescales in band 3 data, consistent with the Faraday rotation occurring on larger physical scales at the lower frequency. We observe a similar dependence in the SF of RM in a numerical GRMHD simulation of (see also Section <ref>), with the lower frequency slope $\alpha = 0.5$ steeper than the higher frequency one $\alpha = 0.2$. However, both slopes in the simulation are significantly less steep than the observations, indicating a relatively larger contribution from variability occurring on the shortest timescales in the numerical model than in the real source. In the joint analysis of the Apr 6, 7,11 data, an SF maximum occurs at around 0.2-0.3 h, and for timescales longer than 0.5 h SF becomes flat, indicating uncorrelated variability structure for larger time lags. Hence, we identify 0.5 h as a characteristic timescale for the 221 GHz Faraday screen variability. A similar flattening is not apparent at 93 GHz, but the reason is most likely related to the short total duration of the observed light curve. We predict that a longer decorrelation timescale will be identified in band 3 with additional observations, allowing to contrast the characteristic timescales at different frequencies. The frequency dependence of the statistical characteristics of the RM variability is yet another hint that the Faraday screen is different for the two observed bands, and at least partly cospatial with the compact emission region.

§ IMPLICATIONS FOR THE RIAF MODEL

§.§ Electrons temperature profile

VLBI measurements of the intrinsic morphology and brightness temperature of in April 2017.

Reference     Date (2017)     $\nu$ (GHz)     $S_\nu$ (Jy)     $\theta_{\rm maj}$ (mas)     $\theta_{\rm maj}$ (mas)     $r_{\rm e}\,(r_{\rm g})$     $T_{\rm B, obs}$ ($10^{10}$ K)     $(1+z)$      $T_{\rm B, int}$ ($10^{10}$ K)

Cho et al., 2022     3 Apr     22.2     1.0     0.80     0.60     66.9     0.51     1.02     0.52

Cho et al., 2022     4 Apr     43.1     1.3     0.30     0.23     24.8     1.24     1.04     1.29

Issaoun et al., 2019     3 Apr     86.3     1.9     0.10     0.12     9.7     2.58     1.12     2.89

EHTC et al., 2022, EHTC et al., 2022     6-7 Apr     228.1     2.4     0.052     0.052     4.1     2.08     1.40     2.91

In order to study the implications of our findings for the RIAF model [Yuan et al., 2003], we first use the quasi-simultaneous VLBI observations of from April 2017 to constrain the radial distribution of the electron temperature. These data sets are summarized in Tab. <ref>. The VLBI observations measure flux density $S_{\nu}$ and allow to estimate the intrinsic (descattered, see Johnson et al., 2018) angular size of the source $\theta_{\rm maj} \times \theta_{\rm min}$, modeled as an elliptical Gaussian. Hence, they allow to estimate the brightness temperature of , a proxy for the temperature of the emitting electrons under the assumption of a thermal energy distribution

\begin{equation}
T_{\rm B, obs} = 1.22 \times 10^{12} \frac{S_\nu}{\nu^2 \theta_{\rm maj} \theta_{\rm min}} \ \ {\rm [K]} \ ,
\end{equation}

where flux density is given in Jy, frequency in GHz, and angular dimensions in mas.

Fitting intrinsic brightness temperature $T_{\rm B,int}$ as a function of the emission radius $r_{\rm e}$. $T_{\rm B, int}$ is a proxy for the temperature of the emitting electrons $T_{\rm e}$. The measurements are based on the gravitational-redshift-corrected VLBI observations performed on 2017 Apr 3-7, with 20% errorbars assumed on both $T_{\rm B}$ and $r_{\rm e}$ estimates.

Furthermore, the observed intrinsic size of the source allows to estimate the characteristic emission radius $r_{\rm e}$ as

\begin{equation}
r_{\rm e}/r_{\rm g} = 0.5(\theta_{\rm maj} \theta_{\rm min})^{0.5}/\theta_{\rm g} - 1 \ ,
\label{eq:emission_radius}
\end{equation}

where $\theta_{\rm g} = r_{\rm g}/D$ is the angular size of the source gravitational radius $r_{\rm g}$ viewed from a distance $D$, and following Gravity Collaboration et al., 2022 we take $\theta_{\rm g} = 5.1\, \mu$as. The subtraction of $1$ in Eq. <ref> approximately accounts for the geometric lensing around the black hole regardless of the detailed spacetime geometry [Gralla & Lupsasca, 2020, Wielgus, 2021]. The emission from the vicinity of the SMBH's event horizon is affected by the gravitational redshift, which scales the
intrinsic brightness temperature by a factor of $(1+z) = [-g_{ tt}(r_{\rm e})]^{-0.5} = (1-2M/r_{\rm e})^{-0.5}$, $T_{\rm B, int} = (1+z) T_{\rm B, obs}$, see Fig. <ref> and Tab. <ref>.

The EHT measurement deviates from a power law characterising $T_{\rm B, int}$ at lower frequencies, which can be clearly seen in Fig. <ref>. We verify that the choice of elliptical Gaussian model rather than a ring model with parameters reported in EHTC et al., 2022 does not impact the $T_{\rm b}$ estimates by more than $\sim 20\%$. A possible explanation for the lower $T_{\rm B, int}$ is either that the emission at 230 GHz, corresponding to $r_{\rm e} \approx 4 r_{\rm g}$, is optically thin and non-thermal, violating assumptions of the $T_{\rm b}$ calculation, and/or that the emission is produced by plasma plunging into the black hole with a relativistic radial velocity component and thus experiencing Doppler deboosting. Hence, we fit a power law $T_{\rm e}(r) = T_{0} (r/r_{\rm g})^\gamma$ only to the three 22-86 GHz data points, obtaining $\gamma = -0.87 \pm 0.20$ and $T_{0} \approx 2.0 \times 10^{11}$ K. The estimated power law index is consistent with the values commonly assumed for $T_{\rm e}(r)$ in RIAF models <cit.> based on spectral energy density fitting [Yuan et al., 2003]. Additionally, based on the four VLBI measurements reported in Tab. <ref>, we model the intrinsic source size dependence on frequency, finding a scaling of $r_{\rm e} \propto \nu^{-1.2}$, in between the values reported by Shen et al., 2005 and Bower et al., 2006.

§.§ RM in a RIAF model

We study the properties of a simple spherically symmetric RIAF power law model to see whether the observed wavelength-dependent RM can be reconciled with the theoretical expectations. In this framework, we can compute RM with the integral along the line of sight from the emitter to the observer <cit.>

\begin{equation}
{\rm RM}= 10^4 \frac{e^3}{ 2 \pi m_{\rm e}^2 c^4 } \int\limits^{\rm r_{\rm obs}}_{r_{\rm e}} n_{\rm e} B_{\parallel} f_{\rm e}(\Theta_{\rm e}) \text{d}r \ [{\rm rad \,m}^{-2}] \ ,
\label{eq:RM}
\end{equation}

for the electron number density $n_{\rm e}$ measured in cm$^{-3}$ and magnetic field component along the line of sight $B_\parallel$ measured in G. The additional dimensionless multiplier $f_{\rm e}(\Theta_{\rm e})$ is related to the reduced impact of hot relativistic electrons, and can be expressed as a function of dimensionless electron temperature $\Theta_{\rm e} = kT_{\rm e}/ m_{\rm e} c^2$ [Quataert & Gruzinov, 2000, Ressler et al., 2023]

\begin{equation}
f_{\rm e}(\Theta_{\rm e}) = \begin{cases}
1 &\text{if $\Theta_{\rm e}  \le 1$} \ ,\\
\Theta_{\rm e}^{-2} \left[ 0.5 \log(\Theta_{\rm e}) \left( 1 - \Theta_{\rm e}^{-1} \right)  + 1\right] &\text{if $\Theta_{\rm e} > 1$ \ .}
\end{cases}
\end{equation}

The presence of this correction factor was raised as an argument against the internal Faraday screen in a RIAF system, since the contribution from the very hot innermost region would be strongly reduced <cit.>. The argument appears reasonable, as the suppression factor from $f_{\rm e}(\Theta_{\rm e})$ reaches $0.008$ at $T_{\rm e} = 10^{11}$K ($\Theta_{\rm e} = 16.8$). Moreover, the characteristic linear scale of a compact system is small in comparison to what the thickness of the external Faraday screen could be. Nevertheless, we will show that constructing a RIAF flow model with significant internal Faraday rotation occurring inside the $10 r_{\rm g}$ radius is feasible regardless of the $f_{\rm e}(\Theta_{\rm e})$ factor.

Predicted value of RM as a function of the emission radius for a RIAF toy-model fitted to the ALMA data. The background RM corresponding to the value observed toward the magnetar J1745-2900 is assumed. Accounting for the relativistic correction $f_{\rm e}$ decreases the RM significantly, by a factor of 20 for $r_{\rm e}= 4 r_{\rm g}$, but does not preclude the dominant contribution to the Faraday rotation from the innermost part of the flow.

For the RIAF model we assume the electron temperature distribution estimated in Section <ref>

\begin{equation}
T_{\rm e}(r) = 2 \times 10^{11} \left(\frac{r}{r_{\rm g}} \right)^{-0.87}  {\rm [K]} \ .
\end{equation}

We also assume power law distributions for the electron number density $n_{\rm e}$ and magnetic field parallel to the line of sight $B_{\parallel}$

\begin{align}
& n_{\rm e}(r) = n_0 \left(\frac{r}{r_{\rm g}} \right)^{\alpha}  {\rm [cm}^{-3}] \ , \\
& B_{\parallel}(r) = B_0 \left(\frac{r}{r_{\rm g}} \right)^{\beta}  {\rm [G]} \ .
\end{align}

To simplify the problem further, we assume the emission at each separate frequency band to originate from a sphere located at the radius $r_{\rm e}$, as estimated in Subsection <ref> – an onion-like model of the radio source. Particularly in case of a low optical depth these radii may only have an effective, approximate sense. We then request for the RM, integrated between the respective $r_{\rm e}$ and the distant observer ($r_{\rm obs} = 10^5 r_{\rm g}$ for practical purposes) using Eq. <ref>, to match the ALMA measurements. We assume a background RM value of $-7 \times 10^4$ rad m$^{-2}$, following the measurement of RM toward the Galactic Center magnetar J1745-2900 [Eatough et al., 2013], located $\sim10^6 r_{\rm g}$ away from (projected distance of $\sim$0.1 pc), with a caveat that the RM toward the magnetar is itself variable, and may involve a contribution from its compact vicinity. Since the sign of RM depends only on the polarity of the uniform magnetic field, we work with absolute values of the RM. With two RM measurements corresponding to different $r_{\rm e}$, we can then fit for the two model parameters, $\alpha + \beta$, and $n_0 B_0$. In Fig. <ref> we show a solution, corresponding to

\begin{align}
& \alpha + \beta = -3.5  \ , \\
& \left(\frac{n_0}{1 \times 10^7 {\rm cm}^{-3}} \right) \left(\frac{B_0}{500 G} \right) = 1 \ .
\end{align}

Comparing our results to the thermal synchrotron one-zone emission model given by EHTC et al., 2022, with $n_{\rm e, zone} = 10^6$ cm$^{-3}$ and $B_{\rm zone} = 29$ G, we can reproduce these numbers reasonably well as $n_{\rm e}(r_{\rm e})$ and $B_\parallel(r_{\rm e})$ for $\alpha = -1.5$, $\beta = -2$, and $r_{\rm e} = 4 r_{\rm g}$. In Fig. <ref> we also show the RM from the same model, computed neglecting the $f_{\rm e}(\Theta_{\rm e})$ factor. Clearly, accounting for this correction has a big impact on the results, but regardless of its presence Faraday rotation may be dominated by the plasma in the innermost part of the accretion flow.

To reproduce the observed ratio of the two RM measurements, around a factor of 2, we generally need a high value of $\alpha+\beta$ in the RIAF model. As an example $\alpha + \beta = -2.5$ assumed by Vos et al., 2022 appears insufficient, yielding a ratio of 1.4. Steeper radial decay favors the interpretation involving vertical, rather than azimuthal, magnetic fields, but on the other hand it may also correspond to a partially inhomogeneous magnetic field, where contributions from different emission regions cancel one another. If we attribute the large contribution to the Faraday rotation from the innermost region of the flow to the partially ordered [Johnson et al., 2015] vertical magnetic field then it also suggests a relatively low viewing angle, so that a significant component of the magnetic field is oriented along the line of sight. Both vertical magnetic field and low inclination are consistent with the conclusions of Gravity Collaboration et al., 2018, Wielgus et al., 2022, and EHTC et al., 2022. Low viewing angle and dominant RM contribution from compact scales were also concluded by Sharma et al., 2007 based on the analysis of global MHD simulations of accretion in .

As an additional sanity check we may estimate the locations of photospheres $r_{\rm ph}$ for the observing frequencies in the model with the parameters estimated above, following calculations of Mahadevan et al., 1996 and integrating the resulting thermal synchrotron opacities inwards. Interestingly, we obtain $r_{\rm ph}$ around 4$r_{\rm g}$ at 228.1 GHz and $r_{\rm ph}$ around 7$r_{\rm g}$ at 86.3 GHz, which are not terribly inconsistent with the estimated emission radii $r_{\rm e}$. At lower frequencies $r_{\rm ph}$ are too low and generally smaller than $20 r_{\rm g}$. We do not attempt to fit the locations of photospheres by tuning the model parameters.

Since the presented model constitutes an extreme simplification of reality, our findings should be considered primarily as a demonstration that an internal Faraday screen in , dominated by the contribution from very compact scales, is to be expected for a reasonable set of physical parameters in a RIAF system. Any more quantitative conclusions should be taken with a sizeable grain of salt.

§ COMPARISONS TO GRMHD

Models more physically self-consistent than a power law RIAF are obtained through numerical GRMHD simulations. RM in GRMHD simulations was studied, among others, by Mościbrodzka et al., 2017 and Ricarte et al., 2020 in the context of M 87* and recently by Ressler et al., 2023 for . EHTC et al., 2022 considered a large library of GRMHD models in order to identify ones that fulfil observational constraints. Here we investigate one of the models favored by the EHT analysis (the "best-bet" model), corresponding to a magnetically arrested disk <cit.> system, viewed at low inclination of 30 deg. High magnetization and small viewing angle are also supported by the analysis of Gravity Collaboration et al., 2018 and Wielgus et al., 2022. The model considered is also characterized by a moderate SMBH spin $a_* = 0.5$ and an ion-to-electron temperature ratio parametrized by $R_{\rm high} = 160$ <cit.>. While the model passed the majority of observational total intensity constraints, it has been reported [EHTC et al., 2022] that much like other GRMHD MAD models it significantly overproduces the variability in total intensity light curves when compared to observations [Wielgus et al., 2022]. The selected GRMHD simulation was performed using the KHARMA code <cit.> and consecutively ray-traced in a curved Kerr spacetime using  [Mościbrodzka & Gammie, 2018], see Wong et al., 2022 for details of the simulation generation pipeline. During ray-tracing a scaling of the plasma density was selected in order to match the 2.4 Jy total compact flux density observed at 228 GHz in April 2017 [Wielgus et al., 2022]. Ray-traced images were then averaged over the entire field of view for each Stokes component at every time step to obtain full-Stokes simulated light curves.

In order to study the polarimetric properties of the simulated GRMHD light curves, we performed ray-tracing at four frequencies, 86.3, 100.3, 213.1, and 229.1 GHz, to be able to mimic the the RM measurements at ALMA band 3 and band 6. Each light curve corresponds to the same GRMHD output of 1000 snapshots with a cadence of 5 $r_{\rm g}/c \approx 100$ s. Since the light curves observed on 2017 Apr 3 and 6 have a duration of about $1000\,r_{\rm g}/c$ (about 6 h), we effectively analyze five different realizations of the theoretical GRMHD model.

The GRMHD light curves from the selected best-bet model generally do not match the  observed polarimetric properties of . The modeled fractional LP is $p = 4 \pm 2$% across frequencies, with no sign of depolarization in the lower band. Compared to the values presented in Tab. <ref>, the model's polarization is too low in band 6, and too high in band 3. The EVPA in the simulation wanders much more than it does in the band 6 data.
The model fractional CP is strongly variable and while the model mean values at 213-229 GHz are around -1%, similarly to the observed quantities, there is about seven times more variability in the simulation than is observed. Hence, the modeled CP sign is flipping on a $\sim$1 h timescale, while it appears to never change in the observed light curves <cit.>.

Smoothed histograms of RM extracted from the GRMHD simulation (sign-flipped and shifted by the Galactic Center magnetar RM $= -7\times 10^4$ rad m$^{-2}$) compared to the ranges inferred from observations on 2017 Apr 3 and 6, discussed in Section <ref> (vertical bands). Shaded and hatched regions denote intervals containing 68% of the distribution around the median value.

The RM in the simulation is evaluated from the EVPA differences between 86.3 and 100.3 GHz, as well as between 213.1 and 229.1 GHz, following Eq. <ref>. We only ray-trace the innermost $300\,r_{\rm g}$ zone of the GRMHD simulation, hence there is no large scale Faraday screen present. Since we find significantly different values of RM at each band for the same time snapshots, the Faraday screen in the simulations is necessarily largely internal (located between the 93 and 221 GHz emission region), consistently with our interpretation of the observations. However, while the observed RMs vary by $\sim20-30$% around the median values and (almost) never change sign, in the simulations the RM sign flips often, on a $\sim$1 h timescale. In order to maximize the consistency, in Fig. <ref> we flip the sign of the model RM and shift the RM values by the Galactic Center magnetar RM, to roughly represent the contribution from the Faraday rotation occurring outside of the GRMHD simulation domain. Accounting for the external Faraday rotation is sufficient for the model band 3 RM to mostly remain negative, although they are about a factor of two lower than the observed values. Recently Ressler et al., 2023 suggested, based on multiscale MHD/GRMHD modeling of the accretion flow fed by stellar winds, that the constant observed RM sign is related to a large and stable external contribution, dominating over the rapidly varying RM component from the compact region. This interpretation is challenged by our results. While it is easy to imagine a larger RM bias related to an external Faraday screen, allowing to match the 93 GHz observations, a more challenging endeavour is to reproduce the large and strictly negative difference between RM at 221 and 93 GHz, necessarily caused by an intrinsic Faraday screen component. Not only are the RM values in our GRMHD simulation at band 6 smaller (in a mean sense) than observed, they are also dramatically more variable with respect to the median value, which is close to zero in the simulation, see also Sharma et al., 2007 and Pang et al., 2011. This discrepancy can be understood in the framework of the magnetic field variability, which is the only signed quantity contributing to the RM. It appears that the magnetic field is far more turbulent and variable in the simulations than in reality, which echoes the total intensity light curve variability discrepancy of EHTC et al., 2022.

MAD models with lower $R_{\rm high}$ parameter ($R_{\rm high}=160$ is the largest value considered in the EHT simulation library) will have hotter electrons than the best-bet model studied here. Hotter electrons would only further decrease the mean value of RM with little impact on its stability because the emission would still emerge from the disk with turbulent magnetic fields. An interesting alternative is the observed mm radiation originating predominantly in the jet sheath region, which is typically threaded by a more stable, nearly-vertical magnetic field in GRMHD simulations [Mościbrodzka & Falcke, 2013]. A subsequent detailed comparison of the simulations from the EHT library (and beyond it) with polarimetric observations at different wavelengths should provide more insight in the future.

§ SUMMARY AND CONCLUSIONS

Using high sensitivity ALMA observations, we characterize the full-Stokes light curves of in the 85-101 GHz range, and compare them with a complementary data set obtained at 212-230 GHz. Both data sets were obtained just three days apart, on 2017 April 3 and 6, respectively. We provide new measurements of linear polarization at 85-101 GHz as well as stringent upper limits on circular polarization. The fractional polarization of decreases rapidly below 150 GHz, which we interpret as a transition in the accretion flow, possibly in magnetic field geometry, strength, or coherence, occurring at around $5-10\,r_{\rm g}$, but it could also be related to the transition to an optically thin flow at higher frequencies. Our observations yield time-dependent measurements of the RM in the 85-101 GHz band, which we find to be lower than the established measurements at higher frequencies by a factor of two. Together with the rapid temporal variability of the RM, lack of variability reduction in the derotated EVPA, and different statistical characteristics of RM temporal variability in the two frequency bands, these results show that the Faraday screen in is most likely largely of internal character, cospatial with the compact region of the synchrotron emission. We demonstrate how these findings can be reproduced using a simple theoretical model of a radiatively inefficient accretion flow. Finally, we demonstrate that the Faraday screen is largely internal in the numerical GRMHD simulations of Sgr A$^*$. However, the particular simulation that we considered, while mostly consistent with the observational total intensity constraints, did not quantitatively reproduce the observed parameters of polarization, and indicated significantly more variability in polarization fractions and in rotation measure than what is observed in .

§ ACKNOWLEDGEMENTS

We thank Geoff Bower, Ilje Cho, Michael D. Johnson, Thomas Krichbaum, Dan Marrone, Ue-Li Pen, Venkatessh Ramakrishnan, Sean Ressler, Angelo Ricarte, Pablo Torne, Sebastiano von Fellenberg, and Guang-Yao Zhao for their helpful comments and enlightening discussions, as well as Vedant Dhruv, Charles Gammie, Abhishek Joshi, Ben Prather, and George Wong for performing the GRMHD simulation that we analyzed. We also thank Alexandra Elbakyan for her contributions to the open science initiative. This paper makes use of the following ALMA data: ADS/JAO.ALMA#2016.1.01404.V and ADS/JAO.ALMA#2016.1.00413.V ALMA is a partnership of ESO (representing its member states), NSF (USA) and NINS (Japan),together with NRC (Canada), NSC and ASIAA (Taiwan), and KASI (Republic of Korea), in cooperation with the Republic of Chile. The Joint ALMA Observatory is operated by ESO, AUI/NRAO and NAOJ. This research is supported by the European Research Council advanced grant “M2FINDERS - Mapping Magnetic Fields with INterferometry Down to Event hoRizon Scales” (Grant No. 101018682). SI is supported by Hubble Fellowship grant HST-HF2-51482.001-A awarded by the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Inc., for NASA, under contract NAS5-26555. IMV acknowledges partial support from Generalitat Valenciana (GenT Project CIDEGENT/2018/021), the MICINN Research Project PID2019-108995GB-C22 and the ASTROVIVES FEDER infrastructure IDIFEDER-2021-086. MM acknowledges support by the NWO grant No. OCENW.KLEIN.113 and support by the NWO Science Athena Award.
RE acknowledges the support from grant numbers 21-atp21-0077, NSF AST-1816420, and HST-GO-16173.001-A as well as the Institute for Theory and Computation at the Center for Astrophysics. CG was supported by FAPESP (Fundação de Amparo á Pesquisa do Estado de São Paulo) under grant 2021/01183-8.

[Agudo et al., 2014]
Agudo, I., Thum, C., Gómez, J. L., & Wiesemeyer, H. 2014, ,
566, A59

[Aitken et al., 2000]
Aitken, D. K., Greaves, J., Chrysostomou, A., et al. 2000, , 534,

[Balick & Brown, 1974]
Balick, B. & Brown, R. L. 1974, , 194, 265

[Bower et al., 2018]
Bower, G. C., Broderick, A., Dexter, J., et al. 2018, , 868, 101

[Bower et al., 1999]
Bower, G. C., Falcke, H., & Backer, D. C. 1999a, ,
523, L29

[Bower et al., 2002]
Bower, G. C., Falcke, H., Sault, R. J., & Backer, D. C. 2002, ,
571, 843

[Bower et al., 2006]
Bower, G. C., Goss, W. M., Falcke, H., Backer, D. C., & Lithwick, Y.
2006, , 648, L127

[Bower et al., 1999]
Bower, G. C., Wright, M. C. H., Backer, D. C., & Falcke, H.
1999b, , 527, 851

[Bower et al., 2003]
Bower, G. C., Wright, M. C. H., Falcke, H., & Backer, D. C. 2003,
, 588, 331

[Brentjens & de Bruyn, 2005]
Brentjens, M. A. & de Bruyn, A. G. 2005, , 441, 1217

[Broderick et al., 2011]
Broderick, A. E., Fish, V. L., Doeleman, S. S., & Loeb, A. 2011, ,
735, 110

[Burn, 1966]
Burn, B. J. 1966, , 133, 67

[Cho et al., 2022]
Cho, I., Zhao, G.-Y., Kawashima, T., et al. 2022, , 926, 108

[Conway et al., 1990]
Conway, J. E., Cornwell, T. J., & Wilkinson, P. N. 1990, , 246,

[Do et al., 2019]
Do, T., Hees, A., Ghez, A., et al. 2019, Science, 365, 664

[Doeleman et al., 2008]
Doeleman, S. S., Weintroub, J., Rogers, A. E. E., et al. 2008, ,
455, 78

[Eatough et al., 2013]
Eatough, R. P., Falcke, H., Karuppusamy, R., et al. 2013, , 501,

[EHTC et al., 2022]
EHTC, Akiyama, K., Alberdi, A., Alef, W., et al. 2022a,
, 930, L12

[EHTC et al., 2022]
EHTC, Akiyama, K., Alberdi, A., Alef, W., et al. 2022b,
, 930, L13

[EHTC et al., 2022]
EHTC, Akiyama, K., Alberdi, A., Alef, W., et al. 2022c,
, 930, L14

[EHTC et al., 2022]
EHTC, Akiyama, K., Alberdi, A., Alef, W., et al. 2022d,
, 930, L16

[Goddi et al., 2021]
Goddi, C., Martí-Vidal, I., Messias, H., et al. 2021, , 910,

[Goddi et al., 2019]
Goddi, C., Martí-Vidal, I., Messias, H., et al. 2019, , 131,

[Gralla & Lupsasca, 2020]
Gralla, S. E. & Lupsasca, A. 2020, , 101, 044031

[Gravity Collaboration et al., 2022]
Gravity Collaboration, Abuter, R., Aimar, N., et al. 2022, , 657,

[Gravity Collaboration et al., 2018]
Gravity Collaboration, Abuter, R., Amorim, A., et al. 2018, , 618,

[Hovatta et al., 2019]
Hovatta, T., O'Sullivan, S., Martí-Vidal, I., Savolainen, T., &
Tchekhovskoy, A. 2019, , 623, A111

[Issaoun et al., 2019]
Issaoun, S., Johnson, M. D., Blackburn, L., et al. 2019, , 871, 30

[Iwata et al., 2020]
Iwata, Y., Oka, T., Tsuboi, M., Miyoshi, M., & Takekawa, S. 2020,
, 892, L30

[Johnson et al., 2015]
Johnson, M. D., Fish, V. L., Doeleman, S. S., et al. 2015, Science,
350, 1242

[Johnson et al., 2018]
Johnson, M. D., Narayan, R., Psaltis, D., et al. 2018, , 865, 104

[Liu et al., 2016]
Liu, H. B., Wright, M. C. H., Zhao, J.-H., et al. 2016, , 593, A107

[Lo & Claussen, 1983]
Lo, K. Y. & Claussen, M. J. 1983, , 306, 647

[Lu et al., 2011]
Lu, R. S., Krichbaum, T. P., Eckart, A., et al. 2011, , 525, A76

[Macquart et al., 2006]
Macquart, J.-P., Bower, G. C., Wright, M. C. H., Backer, D. C., &
Falcke, H. 2006, , 646, L111

[Mahadevan et al., 1996]
Mahadevan, R., Narayan, R., & Yi, I. 1996, , 465, 327

[Marrone et al., 2006]
Marrone, D. P., Moran, J. M., Zhao, J.-H., & Rao, R. 2006, , 640,

[Marrone et al., 2007]
Marrone, D. P., Moran, J. M., Zhao, J.-H., & Rao, R. 2007, , 654,

[Matthews et al., 2018]
Matthews, L. D., Crew, G. B., Doeleman, S. S., et al. 2018, , 130,

[McMullin et al., 2007]
McMullin, J. P., Waters, B., Schiebel, D., Young, W., & Golap, K.
2007, in Astronomical Society of the Pacific Conference Series, Vol. 376,
Astronomical Data Analysis Software and Systems XVI, ed. R. A. Shaw,
F. Hill, & D. J. Bell, 127

[Morris, 2023]
Morris, M. R. 2023, arXiv e-prints, arXiv:2302.02431

[Mościbrodzka et al., 2017]
Mościbrodzka, M., Dexter, J., Davelaar, J., & Falcke, H. 2017,
, 468, 2214

[Mościbrodzka & Falcke, 2013]
Mościbrodzka, M. & Falcke, H. 2013, , 559, L3

[Mościbrodzka et al., 2016]
Mościbrodzka, M., Falcke, H., & Shiokawa, H. 2016, , 586, A38

[Mościbrodzka & Gammie, 2018]
Mościbrodzka, M. & Gammie, C. F. 2018, , 475, 43

[Muñoz et al., 2012]
Muñoz, D. J., Marrone, D. P., Moran, J. M., & Rao, R. 2012, ,
745, 115

[Murchikova & Witzel, 2021]
Murchikova, L. & Witzel, G. 2021, , 920, L7

[Mus et al., 2022]
Mus, A., Martí-Vidal, I., Wielgus, M., & Stroud, G. 2022, ,
666, A39

[Narayan et al., 2003]
Narayan, R., Igumenshchev, I. V., & Abramowicz, M. A. 2003, , 55,

[Narayan et al., 1995]
Narayan, R., Yi, I., & Mahadevan, R. 1995, , 374, 623

[Pacholczyk, 1970]
Pacholczyk, A. G. 1970, Radio astrophysics. Nonthermal processes in galactic
and extragalactic sources

[Pang et al., 2011]
Pang, B., Pen, U.-L., Matzner, C. D., Green, S. R., &
Liebendörfer, M. 2011, , 415, 1228

[Prather et al., 2021]
Prather, B., Wong, G., Dhruv, V., et al. 2021, The Journal of Open
Source Software, 6, 3336

[Quataert & Gruzinov, 2000]
Quataert, E. & Gruzinov, A. 2000, , 545, 842

[Ressler et al., 2023]
Ressler, S. M., White, C. J., & Quataert, E. 2023, , 521, 4277

[Ricarte et al., 2020]
Ricarte, A., Prather, B. S., Wong, G. N., et al. 2020, , 498,

[Sharma et al., 2007]
Sharma, P., Quataert, E., & Stone, J. M. 2007, , 671, 1696

[Shen et al., 2005]
Shen, Z.-Q., Lo, K. Y., Liang, M. C., Ho, P. T. P., & Zhao, J. H.
2005, , 438, 62

[Simonetti et al., 1985]
Simonetti, J. H., Cordes, J. M., & Heeschen, D. S. 1985, , 296, 46

[The GRAVITY Collaboration et al., 2023]
The GRAVITY Collaboration, Abuter, R., Aimar, N., et al. 2023, arXiv
e-prints, arXiv:2307.11821

[Tsuboi et al., 2003]
Tsuboi, M., Miyahara, H., Nomura, R., Kasuga, T., & Miyazaki, A.
2003, Astronomische Nachrichten Supplement, 324, 431

[Vincent et al., 2022]
Vincent, F. H., Gralla, S. E., Lupsasca, A., & Wielgus, M. 2022, ,
667, A170

[Vos et al., 2022]
Vos, J., Mościbrodzka, M. A., & Wielgus, M. 2022, , 668, A185

[Wielgus, 2021]
Wielgus, M. 2021, , 104, 124058

[Wielgus et al., 2022]
Wielgus, M., Marchili, N., Martí-Vidal, I., et al.
2022a, , 930, L19

[Wielgus et al., 2022]
Wielgus, M., Moscibrodzka, M., Vos, J., et al. 2022b,
, 665, L6

[Wong et al., 2022]
Wong, G. N., Prather, B. S., Dhruv, V., et al. 2022, , 259, 64

[Yuan & Narayan, 2014]
Yuan, F. & Narayan, R. 2014, , 52, 529

[Yuan et al., 2003]
Yuan, F., Quataert, E., & Narayan, R. 2003, , 598, 301

§ 2017 APR 7 AND 11 DATA

Summary of the properties of light curves observed with ALMA in April 2017

4c Band 6, 2017 Apr 7          4c Band 6, 2017 Apr 11

$\nu_{\rm obs}$ (GHz)     213.1     215.1     227.1$^a$     229.1         213.1     215.1     227.1$^a$     229.1

$\mathcal{I}$ (Jy)     $2.35\pm 0.14$     $2.35 \pm 0.14$     $2.35\pm 0.15$     $2.35\pm 0.15$          $2.36 \pm 0.29$     $2.34 \pm 0.30$      $2.32 \pm 0.29$      $2.32 \pm 0.30$

$|\mathcal{P}|$ (mJy)     $168.9\pm 56.3$     $170.7\pm57.3$     $181.4\pm 63.0$     $181.1\pm 63.6$         $190.0\pm 50.5$     $188.8\pm 50.4$     $195.7\pm 52.1$     $199.1\pm 54.9$

$p$ (%)     $7.15\pm 2.32$     $7.23 \pm 2.37$     $7.65\pm 2.57$     $7.67\pm 2.62$
$8.02\pm 1.78$     $8.01 \pm 1.80 $     $8.40 \pm 1.98$     $8.53 \pm 2.05$

$\chi^b$ (deg)     $111.9 \pm 6.2$     $112.8 \pm 6.2$     $118.0\pm 6.9$     $118.6\pm 7.2$
$128.8\pm 8.9$     $129.5\pm 9.1$     $133.5\pm 9.3$     $133.9\pm 9.2$

$\mathcal{V}$ (mJy)     $-27.6\pm 9.8$     $-27.5 \pm 9.8 $     $-27.9\pm 9.9$     $-27.7\pm 10.1$
$-29.7\pm 5.8$     $-27.7\pm 6.0 $     $-26.0 \pm 5.8$     $-24.7 \pm 5.4$

$v$ (%)     $-1.18 \pm 0.43$     $-1.18 \pm 0.43$     $-1.19\pm 0.42$     $-1.18 \pm 0.43$
$-1.30 \pm 0.38$     $-1.22 \pm 0.39$     $-1.16 \pm 0.37$     $-1.10 \pm 0.35$

$\alpha_{\mathcal{I}}$     4c $0.02 \pm 0.01$          4c $-0.20 \pm 0.01$

$\alpha_{p}$     4c $1.02 \pm 0.03$          4c $0.78 \pm 0.02$

$\alpha_{v}$     4c $-0.07 \pm 0.03$           4c $-1.87 \pm 0.03$

RM$^c$     4c  $-4.50 \pm 1.16 $         4c$-3.19 \pm 0.72 $

$^a$scaled up by 4% to account for the CN absorption line <cit.>;

$^b$calculated using directional statistics;   $^c$ in the units of $10^5$ rad m$^{-2}$

We present a summary of the analysis of ALMA light curves obtained on 2017 April 7 and 11, in the same fashion as presented in Tab. <ref> for the 2017 Apr 3 and 6 data. On April 11 an X-ray flare occurred shortly before the ALMA observations [EHTC et al., 2022, Wielgus et al., 2022]. On this day the 230 GHz light curves were significantly more variable, with a more negative spectral index and lower RM. These changes are broadly consistent with a hotter, more optically thin accretion disk on April 11 <cit.>.


# Representation Mixing for TTS Synthesis

###### Abstract

Recent character and phoneme-based parametric TTS systems using deep learning
have shown strong performance in natural speech generation. However, the
choice between character or phoneme input can create serious limitations for
practical deployment, as direct control of pronunciation is crucial in certain
cases. We demonstrate a simple method for combining multiple types of
linguistic information in a single encoder, named _representation mixing_ ,
enabling flexible choice between character, phoneme, or mixed representations
during inference. Experiments and user studies on a public audiobook corpus
show the efficacy of our approach.

Index Terms—  Text-to-speech, deep learning, recurrent neural network,
attention, sequence-to-sequence learning.

##  1 Introduction

TTS synthesis [1] focuses on building systems which can generate speech (often
in the form of an audio feature sequence $a$), given a set of linguistic
sequences, $l$. These sequences, $l$ and $a$, are of different length and
dimensionality thus it is necessary to find the alignment between the
linguistic information and the desired audio. We approach the alignment
problem by jointly _learning_ to align these two types of information [2],
effectively translating the information in the linguistic sequence(s) into
audio through learned transformations for effective TTS synthesis [3, 4, 5, 6,
7].

###  1.1 Data Representation

We employ _log mel spectrograms_ as the audio feature representation, a well-
studied time-frequency representation [8] for audio sequence $a$. Various
settings used for this transformation can be seen in Table 2.

Linguistic information, $l$, can be given at the abstract level as graphemes
(also known as characters when using English) or at a more detailed level
which may include pronunciation information, such as phonemes. Practically,
character-level information is widely available in open data sets though this
representation allows ambiguity in pronunciation [9].

###  1.2 Motivating Representation Mixing

In some cases, it may be impossible to fully realize the desired audio without
being given direct knowledge of pronunciation. Take as a particular example
the sentence _”All travelers were exhausted by the wind down the river”_. The
word _”wind”_ in this sentence can be either noun form such as _”The wind blew
swiftly on the plains”_ , or verb form for traveling such as _”… and as we
wind on down the road”_ , both of which have different pronunciation.

Without external knowledge (such as additional context, or an accompanying
video) of which pronunciation to use for the word, a TTS system which operates
on character input will always have ambiguity in this situation. Alternate
approaches such as grapheme to phoneme methods [10] will also be unable to
resolve this problem. Such cases are well-known in both TTS and linguistics
[11, 12], motivating our desire to flexibly combine grapheme and phoneme
inputs in a single encoder. Our method allows per example control of
pronunciation during inference _without requiring_ this information for all
desired inputs, and similar methods have also been important for other systems
[13].

Fig. 1: Visualization of embedding computation

##  2 Representation Mixing Description

The input to the system consists of one data sequence, $l_{j}$, and one mask
sequence, $m$. The data sequence, $l_{j}$, consists of a mixture between a
character sequence, $l_{c}$, and a phonetic sequence, $l_{p}$. The mask
sequence describes which respective sequence the symbol came from with an
integer ($0$ or $1$). Each time a training sequence is encountered, it is
randomly mixed at the word level and we assign all spaces and punctuation as
characters.

For example, a pair ”$the$ $cat$”, ”$@d@ah$ $@k@ah@t$” (where @ is not a used
symbol and serves only as an identifier to mark phoneme boundaries) can be
resampled as ”$the$ $@k@ah@t$”, with a corresponding mask of
$[0,0,0,0,1,1,1]$. On the next occurrence, it may be resampled as ”$@d@ah$
$cat$”, with mask $[1,1,0,0,0,0]$. This resampling procedure can be seen as
data augmentation, as well as training the model to smoothly process both
character and phoneme information without over-reliance on either
representation. We demonstrate the importance of each aspect in Section 4.

###  2.1 Combining Embeddings

The full mixed sequence, $l_{j}$, separately passes through two embedding
matrices, $e_{c}$ and $e_{p}$, and is then combined using the mask sequence to
form a joint mixed embedding, $e_{j}$. For convenience, $e_{c}$ and $e_{p}$,
are set to a vocabulary size that is the _max_ of the character and phoneme
vocabulary sizes. This mixed embedding is further combined with an embedding
of the mask sequence itself, $e_{m}$, for a final combined embedding, $e_{f}$.
This embedding, $e_{f}$, is treated as the standard input for the rest of the
network. A diagram describing this process is shown in Figure 1.

$\displaystyle e_{j}=$ $\displaystyle\>(1-m)*e_{c}+m*e_{p}$ (1) $\displaystyle
e_{f}=$ $\displaystyle\>e_{m}+e_{j}$ (2)

###  2.2 Stacked Multi-scale Residual Convolution

In the next sections, we describe the network architecture for transforming
the mixed representation into a spectrogram and then an audio waveform. A full
system diagram of our neural network architecture can be seen in Figure 2.

Fig. 2: Encoding, attention, and one step of mel decoder

The final embedding, $e_{f}$, from the previous section is used as input to a
stacked multi-scale residual convolutional subnetwork (SMRC). The SMRC
consists of several layers of multi-scale convolutions, where each multi-scale
layer is in turn a concatenation of $1\times 1$, $3\times 3$, and $5\times 5$
layers concatenated across the channel dimension. The layers are connected
using residual bypass connections [14] and batch normalization [15] is used
throughout. After the convolutional stage, the resulting activations are input
to a bidirectional LSTM layer [16, 17]. This ends the _encoding_ part of the
network.

###  2.3 The Importance of Noisy Teacher Forcing

All audio information in the network passes through a multilayer pre-net with
dropout [18, 4, 5], in both training and evaluation. We found that using pre-
net corrupted audio in every layer (including the attention layer) greatly
improves the robustness of the model at evaluation, while corruption of
linguistic information made the generated audio sound less natural.

###  2.4 Attention-based RNN Decoder

The encoding activations are attended using a Gaussian mixture (GM) attention
[19] driven by an LSTM network (conditioned on both the text and pre-net
activations), with a softplus activation for the step of the Gaussian mean as
opposed to the more typical exponential activation. We find that a softplus
step substantially reduces instability during training. This is likely due to
the relative length differences between linguistic input sequences and audio
output.

Subsequent LSTM decode layers are conditioned on pre-net activations,
attention activations, and the hidden state of the layer before, forming a
series of skip connections [19, 20]. LSTM decode layers utilize cell dropout
regularization [21]. The final hidden state is projected to match the
dimensionality of the audio frames and a mean squared error loss is calculated
between the predicted and true next frames.

###  2.5 Truncated Backpropagation Through Time (TBPTT)

The network uses truncated backpropagation through time (TBPTT) [17] in the
decoder, only processing a subsection of the relevant audio sequence while
reusing the linguistic sequence until the end of the associated audio
sequence. TBPTT training allows for a particular iterator packing scheme not
available with full sequence training, which continuously packs subsequences
resetting only the particular elements of the sequence minibatches (and the
associated RNN state) when reaching the end of an audio sequence. A simplified
diagram of this approach can be seen in Figure 3.

Single minibatch

Truncated minibatches with resets

Fig. 3: Comparison of standard minibatching versus truncation.

###  2.6 Converting Features Into Waveforms

After predicting a log mel spectrogram sequence from linguistic input, further
transformation must be applied to get an audible waveform. A variety of
methods have been proposed for this inversion such as the Griffin-Lim
transform [22], and modified variants [23]. Neural decoders such as the
conditional WaveNet decoder [24, 5] or conditional SampleRNN [25] can also act
effectively as an inversion procedure to transform log mel spectrograms into
audible waveforms.

Optimization methods also work for inversion and we utilize an L-BFGS based
routine. This process optimizes randomly initialized parameters (representing
the desired waveform) passed through a log mel spectrogram transform, to make
the transform of these parameters match a given log mel spectrogram [26].

This results in a set of parameters (which we now treat as a fixed waveform)
with a lossy transform that closely matches the given log mel spectrogram. The
overall procedure closely resembles effective techniques in other generative
modeling settings [27].

_Inversion Method_ | _Seconds per Sample_ | _STRTF_
---|---|---
100 L-BFGS | 1.49E-4 | 3.285
1000 L-BFGS | 1.32E-3 | 29.08
Modified Griffin-Lim | 2.81E-4 | 6.206
100 L + GL | 4.11E-4 | 9.063
1000 L + GL | 1.6E-3 | 35.28
WaveNet | 7.9E-3 | 174.7
100 L + GL + WN | 8.3E-3 | 183.7
1000 L + GL + WN | 9.5E-3 | 210.0
Table 1: Comparison of various log mel spectrogram to waveform conversion
techniques. STRTF stands for ”slower than real time factor” (calculated
assuming output sample rate of $22.05$ kHz), where $1.0$ would be real-time
generation for a single example.

###  2.7 Inversion Pipeline

This work uses a combined pipeline of L-BFGS based inversion, followed by
modified Griffin-Lim for waveform estimation [23]. The resulting waveform is
of moderate quality, and usable for certain applications including speech
recognition. Using either L-BFGS or Griffin-Lim in isolation did not yield
usable audio in our experiments. We also found that this ordering (L-BFGS then
Griffin-Lim) in our two-stage pipeline works much better than the reverse
setting.

To achieve the highest quality output, we then use the moderate quality
waveform output from the two stage pipeline, converted back to log mel
spectrograms, for conditional WaveNet synthesis. Though other work [5] clearly
shows that log mel spectrogram conditioned WaveNet can be used directly, we
find the two stage pipeline allows for quicker quality checking during
development, as well as slightly higher quality in the resulting audio after
WaveNet sampling. We suspect this is because the pre-trained WaveNet [28] that
we use is trained on ground-truth log mel spectrograms rather than predicted
outputs. The synthesis speed of each setting is shown in Table 1.

##  3 Related Work

Representation mixing is closely related to the ”mixed-character-and-phoneme”
setting described in Deep Voice 3 [13], with the primary difference being our
addition of the mask embedding $e_{m}$. We found utilizing a mask embedding
alongside the character and phoneme embeddings further improved quality, and
was an important piece in the text portion of the network. The focused user
study in this paper also highlights the advantages of this kind of mixing
independent of high-level architecture choices, since our larger system is
markedly different from that used in Deep Voice 3.

The SMRC subnetwork is closely related to the CBHG encoder [4], differing in
the number and size of convolutional scales used, no max-pooling, use of
residual connections, and multiple stacks of multi-scale layers. We use a
Gaussian mixture attention, first described by Graves [19] and used in several
speech related papers [3, 29, 30], though we utilize _softplus_ activation for
the mean-step parameter finding that it improved stability in early training.

Our decoder LSTMs utilize cell dropout [21] as seen in prior work [31, 32].
Unlike Tacotron [4], we do not use multi-step prediction, convolutional layers
in the mel-decoding stage, or Griffin-Lim inside the learning pathway. Audio
processing closely resembles Tacotron 2 [5] overall, though we precede
conditional WaveNet with a computationally efficient two stage L-BFGS and
Griffin-Lim inference pipeline for quality control, and as an alternative to
neural inversion.

##  4 Experiments

The model is trained on LJSpeech [33], a curated subset of the LibriVox corpus
[9]. LJSpeech consists of $13,100$ audio files (comprising a total time of
approximately $24$ hours) of read English speech, spoken by Linda Johnson. The
content of these recordings is drawn from scientific, instructional, and
political texts published between $1893$ and $1964$. The recordings are stored
as $22.05$ kHz, $16$ bit WAV files, though these are conversions from the
original $128$ kbps MP3 format.

Character level information is extracted from audio transcriptions after a
normalization and cleaning pipeline. This step includes converting all text to
lowercase along with expanding acronyms and numerical values to their spelled-
out equivalents. Phonemes are then extracted from the paired audio and
character samples, using a forced alignment tool such as Gentle [34]. This
results in character and phoneme information aligned along word level
boundaries, so that randomly _mixed_ linguistic sequences can be sampled
repeatedly throughout training. When using representation mixing for training,
we choose between characters and phonemes with probability $.5$ for each word
in all experiments.

Audio Processing |  $22.05$ kHz $16$ bit input, scale between $(-1,1)$, log mel spectrogram $80$ mel bands, window size $512$, step $128$, lower edge $125$ Hz, upper edge $7.8$ kHz, mean and std dev normalized per feature dimension after log mel calculation.
---|---
Embeddings |  vocabulary size (v) $49$, $49$, $2$, embedding dim $15$, truncated normal $\frac{1}{\sqrt{v}}$
SMRC |  $3$ stacks, stack scales $1\times 1$, $3\times 3$, $5\times 5$, $128$ channels each, batch norm, residual connections, _ReLU_ activations, orthonormal init
Enc Bidir LSTM |  hidden dim $1024$ $(128\times 4\times 2)$, truncated normal init scale $0.075$
Pre-net |  $2$ layers, hidden dim $128$, dropout keep prob $0.5$, linear activations, orthonormal init
Attention LSTM |  num mixes $10$, _softplus_ step activation, hidden dim $2048$ $(512\times 4)$, trunc norm init scale $0.075$
Decoder LSTMs |  $2$ layers, cell dropout keep prob $0.925$, truncated normal init scale $0.075$
Optimizer |  mse, no output masking, Adam optimizer, learning rate $1E-4$, global norm clip scale $10$
Other |  TBPTT length $256$, batch size $64$, training steps $500$k, $1$ TitanX Pascal GPU, training time $7$ days
Table 2: Architecture hyperparameter settings

###  4.1 Log Mel Spectrogram Inversion Experiments

We use a high quality implementation of WaveNet, including a pre-trained model
directly from Yamamoto et. al. [28]. This model was also trained on LJSpeech,
allowing us to directly use it as a neural inverse to the log mel spectrograms
predicted by the attention-based RNN model, or as an inverse to log mel
spectrograms extracted from the network predictions after further processing.
Ultimately combining the two stage L-BFGS and modified Griffin-Lim pipeline
with a final conditional WaveNet sampling pass demonstrated the best quality,
and is what we used for the user study shown in Table 3.

###  4.2 Preference Testing

Given the overall system architecture, we pose three primary questions
(referenced in column Q in Table 3) as a user study:

1) Does representation mixing (RM) improve character-based inference over a
baseline which was trained only on characters?

2) Does RM improve phoneme with character backoff for unknown word (PWCB)
inference over a baseline trained on fixed PWCB?

3) Does PWCB inference improve over character-based inference in a model
trained with representation mixing?

To answer these questions, we pose each as a preference test by presenting
study participants with $20$ paired choices. Each user was instructed to
choose the sample they preferred 111 https://s3.amazonaws.com/representation-
mixing-site/index.html . The $20$ tests were chosen randomly in a pool of
$123$ possible tests covering all three question types, from $41$ possible
sentences. Presentation order was randomized for each question and every user.
The overall study consisted of $22$ users and $429$ responses across all
categories (some users didn’t select any preference for some questions).

_Q_ | _Model A_ | _Model B_ | _C. A_ | _Total_ | _% A_
---|---|---|---|---|---
1 | RM (char) | Char | 101 | 137 | 73.7%
2 | RM (PWCB) | PWCB | 106 | 145 | 73.1%
3 | RM (PWCB) | RM (char) | 86 | 147 | 58.5%
Table 3: A / B user preference study results. RM uses approach in parenthesis
at inference time. Columns ” _C. A_ ” and ” _% A_ ” indicate the number and
percentage of users which preferred _Model A_.

We see that users clearly prefer models trained with representation mixing,
even when using identical information for inference. This highlights the data
augmentation aspects of representation mixing, as regardless of information
type representation mixing (RM) gives clearly preferable results compared to
static representations (Char, PWCB). The preference of representation mixing
over static PWCB also means that introducing phoneme and character information
without mixing is less beneficial than full representation mixing.

It is also clear that for representation mixing models, pronunciation
information (PWCB) at inference gives preferable samples compared to character
information. This is not surprising, but further reinforces the importance of
using pronunciation information where possible. Representation mixing enables
choice in input format, allowing the possibility to use many different
inference representations for a given sentence with a single trained model.

##  5 Conclusion

This paper shows the benefit of _representation mixing_ , a simple method for
combining multiple types of linguistic information for TTS synthesis.
Representation mixing enables inference conditioning to be controlled
independently of training representation, and also results in improved quality
over strong character and phoneme baselines trained on a publically available
audiobook corpus.

## References

  * [1] A. J. Hunt and A. W. Black,  “Unit selection in a concatenative speech synthesis system using a large speech database,”  in Acoustics, Speech, and Signal Processing, 1996. ICASSP-96. Conference Proceedings., 1996 IEEE International Conference on. IEEE, 1996, vol. 1, pp. 373–376.
  * [2] D. Bahdanau, K. Cho, and Y. Bengio,  “Neural machine translation by jointly learning to align and translate,”  in International Conference on Learning Representations (ICLR 2015), 2015.
  * [3] J. Sotelo, S. Mehri, K. Kumar, J. F. Santos, K. Kastner, A. Courville, and Y. Bengio,  “Char2wav: End-to-end speech synthesis,”  2017\.
  * [4] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, et al.,  “Tacotron: A fully end-to-end text-to-speech synthesis model,”  arXiv preprint, 2017.
  * [5] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerry-Ryan, et al.,  “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,”  arXiv preprint arXiv:1712.05884, 2017.
  * [6] H. Tachibana, K. Uenoyama, and S. Aihara,  “Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention,”  in ICASSP, 2018.
  * [7] W. Ping, K. Peng, and J. Chen,  “ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech,”  ArXiv e-prints, July 2018.
  * [8] J. O. Smith,  Spectral Audio Signal Processing,  online book, 2011 edition.
  * [9] “Librivox,” http://librivox.org/.
  * [10] K. Rao, F. Peng, H. Sak, and F. Beaufays,  “Grapheme-to-phoneme conversion using long short-term memory recurrent neural networks,”  in ICASSP, 2015.
  * [11] A. W. Black, K. Lenzo, and V. Pagel,  “Issues in building general letter to sound rules,”  1998\.
  * [12] C. M. Eddington and N. Tokowicz,  “How meaning similarity influences ambiguous word processing: The current state of the literature,”  Psychonomic bulletin & review, vol. 22, no. 1, pp. 13–37, 2015\.
  * [13] W. Ping, K. Peng, A. Gibiansky, S. O. Arik, A. Kannan, S. Narang, J. Raiman, and J. Miller,  “Deep voice 3: Scaling text-to-speech with convolutional sequence learning,”  in International Conference on Learning Representations (ICLR 2018), 2018.
  * [14] K. He, X. Zhang, S. Ren, and J. Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2016.
  * [15] S. Ioffe and C. Szegedy,  “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”  2015\.
  * [16] M. Schuster and K. K. Paliwal,  “Bidirectional recurrent neural networks,”  IEEE Transactions on Signal Processing, vol. 45, no. 11, pp. 2673–2681, 1997.
  * [17] S. Hochreiter and J. Schmidhuber,  “Long short-term memory,”  Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  * [18] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov,  “Dropout: A simple way to prevent neural networks from overfitting,”  The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014.
  * [19] A. Graves,  “Generating sequences with recurrent neural networks,”  arXiv:1308.0850 [cs.NE], Aug. 2013.
  * [20] S. Zhang, Y. Wu, T. Che, Z. Lin, R. Memisevic, R. R. Salakhutdinov, and Y. Bengio,  “Architectural complexity measures of recurrent neural networks,”  in Advances in Neural Information Processing Systems, 2016, pp. 1822–1830.
  * [21] S. Semeniuta, A. Severyn, and E. Barth,  “Recurrent dropout without memory loss,”  in Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, 2016, pp. 1757–1766.
  * [22] D. Griffin and J. Lim,  “Signal estimation from modified short-time fourier transform,”  IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 2, pp. 236–243, 1984.
  * [23] M. Slaney, D. Naar, and R. Lyon,  “Auditory model inversion for sound separation,”  in Acoustics, Speech, and Signal Processing, 1994. ICASSP-94., 1994 IEEE International Conference on. IEEE, 1994, vol. 2, pp. II–77.
  * [24] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu,  “WaveNet: A Generative Model for Raw Audio,”  ArXiv e-prints, Sept. 2016.
  * [25] S. Mehri, K. Kumar, I. Gulrajani, R. Kumar, S. Jain, J. Sotelo, A. Courville, and Y. Bengio,  “Samplernn: An unconditional end-to-end neural audio generation model,”  arXiv preprint arXiv:1612.07837, 2016.
  * [26] C. Thomé,  “Personal communication,”  2018\.
  * [27] L. A. Gatys, A. S. Ecker, and M. Bethge,  “Image style transfer using convolutional neural networks,”  in CVPR, 2016, pp. 2414–2423.
  * [28] R. Yamamoto, M. Andrews, M. Petrochuk, W. Hy, cbrom, O. Vishnepolski, M. Cooper, K. Chen, and A. Pielikis,  “r9y9/wavenet_vocoder: v0.1.1 release,” Oct. 2018,  https://github.com/r9y9/wavenet_vocoder.
  * [29] R. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. J. Weiss, R. Clark, and R. A. Saurous,  “Towards end-to-end prosody transfer for expressive speech synthesis with tacotron,”  arXiv preprint arXiv:1803.09047, 2018.
  * [30] Y. Taigman, L. Wolf, A. Polyak, and E. Nachmani,  “Voice synthesis for in-the-wild speakers via a phonological loop,”  arXiv preprint arXiv:1707.06588, 2017.
  * [31] D. Ha, A. Dai, and Q. V. Le,  “Hypernetworks,”  arXiv preprint arXiv:1609.09106, 2016.
  * [32] D. Ha and D. Eck,  “A neural representation of sketch drawings,”  arXiv preprint arXiv:1704.03477, 2017.
  * [33] K. Ito,  “The lj speech dataset,” https://keithito.com/LJ-Speech-Dataset/, 2017.
  * [34] R. M. Ochshorn and M. Hawkins,  “Gentle forced aligner [computer software],” 2017,  https://github.com/lowerquality/gentle.