∎

11institutetext:  1: Department of Mechanical Systems Engineering, Tokyo
University of Agriculture and Technology, Koganei, Tokyo, 184-8588, Japan
2: Department of Industrial Technology and Innovation, Tokyo University of
Agriculture and Technology, Koganei, Tokyo, 184-8588, Japan
3: Department of Electrical and Mechanical Engineering, Nagoya Institute of
Technology, Nagoya, Aichi, 466-8555, Japan
∗Corresponding author. 11email<EMAIL_ADDRESS>

# Flow birefringence of cellulose nanocrystal suspensions in three-dimensional
flow fields: revisiting the stress-optic law

Kento Nakamine1 Yuto Yokoyama1 William Kai Alexander Worby2
Masakazu Muto3 Yoshiyuki Tagawa1∗

(Received: date / Accepted: date)

###### Abstract

This study systematically investigates the flow birefringence of cellulose
nanocrystal (CNC) suspensions. The aim is to clarify the importance of the
stress component along the camera’s optical axis in the stress-optic law
(SOL), which describes the relationship between birefringence, the retardation
of transmitted polarized light, and the stress field. More than 100 datasets
pertaining to the retardation of CNC suspensions (concentrations of 0.1, 0.3,
0.5, and 1.0 wt%) in a laminar flow field within a rectangular channel (aspect
ratios of 0.1, 1, and 3) are systematically obtained. The measured retardation
data are compared with the predictions given by the conventional SOL excluding
the stress component along the camera’s optical axis and by the SOL including
these components as second-order terms (2nd-order SOL). The results show that
the 2nd-order SOL gives a significantly better agreement with the
measurements. Based on the 2nd-order SOL, the retardation at the center of the
channel, where the effect of the stress component along the camera’s optical
axis is most pronounced, is predicted to be proportional to the square of the
flow rate, which agrees with the experimental data. The results confirm the
importance of considering the stress component along the camera’s optical axis
in the flow birefringence of CNC suspensions at high flow rates, even for
quasi-two-dimensional channel flow.

###### Keywords:

Flow birefringence Cellulose nanocrystals Stress-optic law Polarization camera
Rectangular channel

††journal:

##  1 Introduction

Cellulose nanocrystals (CNCs) are rod-shaped crystals with a high aspect ratio
(Beck-Candanedo et al., 2005). These rigid, highly crystalline nanoparticles
are made from wood and plants (Habibi et al., 2010), and have characteristics
including a low density and high strength (Saito et al., 2013), as well as
interesting thermodynamic and rheological properties (Buffa et al., 2019).
Many researchers have discussed the potential applications of such properties
(Thiruganasambanthan et al., 2022; Mu et al., 2019; Lagerwall et al., 2014).

One property of CNCs that we are interested in is birefringence (Kádár et al.,
2021; Detert et al., 2023). When a suspension containing particles with a
large aspect ratio, such as CNCs, is at rest, the particles within the fluid
are randomly oriented, resulting in optical isotropy. As the fluid flows, the
shear forces orient the particles in a preferred direction, resulting in the
fluid exhibiting birefringence (Aben and Puro, 1997; Kádár et al., 2021). This
birefringence property of a fluid (or flow birefringence) appears when the
particles are under a mechanical force (corresponding to shear in our case)
and are aligned in a uniform direction (Hausmann et al., 2018; Calabrese et
al., 2021b). Flow birefringence strongly and directly depends on the stress
applied to the bulk fluid; details are given in Sec. 2.1. Therefore,
quantitative measurements of flow birefringence provide a direct estimation of
the fluid stress field, without the need for velocity field measurements
through techniques such as particle image velocimetry.

If light is passed through a fluid while it is subject to shear-induced
birefringence, the light is elliptically polarized with a retardation $\Delta$
that depends on the magnitude of the birefringence and an orientation $\phi$
that depends on the direction of the particle. The relationship between
$\Delta$, $\phi$, and the stresses in the material is called the stress-optic
law (SOL) (Aben and Guillemet, 1993; Ramesh, 2021). Using the SOL, the stress
can be estimated from the retardation $\Delta$ and orientation $\phi$ of the
emitted polarized light obtained by optical measurements (Prabhakaran, 1975;
Janeschitz-Kriegl, 2012; Knight Jr and Pih, 1976; Noto et al., 2020). Studies
using CNC suspensions have reported the applicability of the SOL in quasi-two-
dimensional flow fields, such as rectangular channels, in which the channel
depth is greater than the width, or in Taylor–Couette flows (Lane et al.,
2022a; Calabrese et al., 2021a, 2022; Lane et al., 2023), as a means of
understanding the flow birefringence of such fluids. Studies on flow
birefringence in three-dimensional flow fields are limited compared with those
on two-dimensional flows.

In the three-dimensional case, previous studies have suggested that the SOL
exhibits a nonlinear relationship between the retardation and stresses (McAfee
and Pih, 1974; Doyle, 1982; Aben and Puro, 1997). McAfee and Pih (1974)
observed a three-dimensional flow in a channel and stated that “the isoclinic
[retardation in this paper] was dependent on shear strain rates both normal
and parallel [camera’s optical axis in this paper] to the light ray”, but they
did not derive the direct relation between the SOL and their observations.
Doyle (1982) extended the SOL to three-dimensional flows by including the
stress component along the camera’s optical axis as second-order terms. This
study highlighted the importance of the stress component along the camera’s
optical axis, although the theory was not validated against experiments. The
effect of the stress component along the camera’s optical axis in the SOL has
been neglected in the birefringence of solids, i.e., photoelasticity. Aben and
Puro (1997) discussed the flow birefringence by referring to Doyle’s work,
although their extended SOL neglected the stress component along the camera’s
optical axis. Other significant works (Ainola, 1998; Kim et al., 2017; Ober et
al., 2011; Clemeur et al., 2004) have also omitted the stress component along
the camera’s optical axis. Thus, the influence of the stress component along
the camera’s optical axis on flow birefringence has not been incorporated into
many previous studies, although its importance has been identified. The
negative effects of this omission seem to have been avoided by using a quasi-
two-dimensional flow field, as described above. Nevertheless, the results of
several studies using quasi-two-dimensional channels highlight the
discrepancies between the retardation predicted by the SOL for the two-
dimensional case and the experimental data, especially in the center of the
channel (Kim et al., 2017; Ober et al., 2011; Calabrese et al., 2023). Thus,
to clarify the effect of the stress component along the camera’s optical axis
on flow birefringence, it is necessary to revisit the extended SOL described
by Doyle (1982) and verify it experimentally. To the best of the authors’
knowledge, the SOL considering the stress component along the camera’s optical
axis has never been experimentally validated, although fluid flows are always
strictly three-dimensional.

This study determines the effect of the stress component along the camera’s
optical axis on the flow birefringence of CNC suspensions by revisiting
Doyle’s SOL (Doyle, 1982). For this purpose, we performed systematic
experiments measuring the flow birefringence of CNC suspensions in steady
laminar flow in rectangular channels. The retardation field measured during
the experiments was compared with that predicted by the SOL proposed by Doyle
(1982). The methodology used in this study is shown in Fig. 1. A steady
laminar flow of a Newtonian fluid in a rectangular channel is considered
because analytical solutions for the stress field are available for such
cases. The aspect ratio of the rectangular channel was varied to change the
stress component’s contribution along the camera’s optical axis.

Figure 1: Schematic of the concept of this study. The theoretical retardation
field, calculated using an SOL from the analytical solution of the flow field,
is compared with the experimentally obtained retardation field to verify the
SOL.

##  2 Methodology: Polarization measurements in rectangular channel flow of
CNC suspensions

In this section, we explain the SOL, which is the relationship between
retardation and stress component. We then explain how retardation is measured
and how the theoretical retardation of a fluid is calculated.

###  2.1 Stress-optic law

In this section, we carefully review the SOLs described by Aben and Doyle
(Aben and Puro, 1997; Doyle, 1982). We consider which formulations include
three-dimensional effects, i.e., the effect of the stress component along the
camera’s optical axis, and refer to 1st-order SOLs for those that do not
include the stress component along the camera’s optical axis and 2nd-order
SOLs for those that do include the stress component along the camera’s optical
axis.

The retardation of light causes birefringence due to the anisotropic
refractive index tensor of a birefringent material when loaded by stress. The
material’s refractive index is determined by the relative permittivity, also
known as the dielectric constant. If $z$ is set as the camera’s optical axis
in Cartesian coordinates, the dielectric tensor $\boldsymbol{\varepsilon}$ can
be written as

$\boldsymbol{\varepsilon}=\begin{bmatrix}\varepsilon_{xx}&\varepsilon_{xy}&\varepsilon_{xz}\\\
\varepsilon_{yx}&\varepsilon_{yy}&\varepsilon_{yz}\\\
\varepsilon_{zx}&\varepsilon_{zy}&\varepsilon_{zz}\end{bmatrix}.$ (1)

The retardation $\Delta$ and orientation $\phi$ of the polarized light through
the material are related to the components of the dielectric tensor in
Cartesian coordinates as follows (Aben and Puro, 1997; Doyle, 1982):

$\displaystyle V_{1}\equiv\Delta\cos 2\phi$ $\displaystyle=$
$\displaystyle\frac{1}{2n_{0}}\int\left(\varepsilon_{yy}-\varepsilon_{xx}\right)dz,$
(2) $\displaystyle V_{2}\equiv\Delta\sin 2\phi$ $\displaystyle=$
$\displaystyle\frac{1}{n_{0}}\int\varepsilon_{xy}dz,$ (3)

where $n_{0}$ is the initial refractive index of the material. Therefore,
$\Delta$ and $\phi$ can be expressed as

$\displaystyle\Delta=\sqrt{V_{1}^{2}+V_{2}^{2}},\qquad\phi=\frac{1}{2}\tan^{-1}\frac{V_{2}}{V_{1}}.$
(4)

If $\Delta$ is less than $\lambda/4$ and the rotation of the principal stress
direction does not exceed $\pi/6$, Eqs. (2) and (3) are valid (Aben et al.,
1989; Aben and Guillemet, 1993). This approximation holds for the conditions
considered in this study because the measured retardation is always less than
$\lambda/4$. The optical effect is a function of the strain rate
$\dot{e}_{jk}$ (Aben and Puro, 1997; Doyle, 1982):

$\frac{1}{2n_{0}}\varepsilon_{jk}=f(\dot{e}_{jk}).$ (5)

Using the Cayley–Hamilton theorem (Aben and Puro, 1997; Doyle, 1982), the
optical effect can be written as

$\frac{1}{2n_{0}}\varepsilon_{jk}=\alpha_{0}\delta_{jk}+\alpha_{1}\dot{e}_{jk}+\alpha_{2}\dot{e}_{jl}\dot{e}_{lk},$
(6)

where $\alpha_{0}$, $\alpha_{1}$, and $\alpha_{2}$ are material-specific
constants and $\delta_{jk}$ is the Kronecker delta. Here, $j$, $k$, and $l$
are Einstein’s notation indices. As described in previous studies (Aben and
Puro, 1997; Doyle, 1982), the relationship between the retardation,
orientation, and the components of the strain rate tensor can be expressed
using Eqs. (2) and (3) as:

$\begin{split}V_{1}&=\Delta\cos
2\phi=\int\left\\{\alpha_{1}\left(\dot{e}_{yy}-\dot{e}_{xx}\right)\right.\\\
&\left.+\alpha_{2}\left[\left(\dot{e}_{yy}+\dot{e}_{xx}\right)\left(\dot{e}_{xx}-\dot{e}_{yy}\right)+\dot{e}_{yz}^{2}-\dot{e}_{xz}^{2}\right]\right\\}dz,\end{split}$
(7) $\begin{split}V_{2}=\Delta\sin
2\phi&=\int\left\\{2\alpha_{1}\dot{e}_{xy}\right.\\\
&\left.+\alpha_{2}\left[2\left(\dot{e}_{yy}+\dot{e}_{xx}\right)\dot{e}_{xy}+2\dot{e}_{yz}\dot{e}_{xz}\right]\right\\}dz.\end{split}$
(8)

For a Newtonian fluid, the viscous stress is proportional to the strain rate,
i.e., $\sigma_{jk}=\eta\dot{e}_{jk}$, where $\eta$ is the liquid viscosity.
Thus, these equations become

$\begin{split}V_{1}&=\Delta\cos
2\phi=\int\left\\{C_{1}\left(\sigma_{yy}-\sigma_{xx}\right)\right.\\\
&\left.+C_{2}\left[\left(\sigma_{yy}+\sigma_{xx}\right)\left(\sigma_{yy}-\sigma_{xx}\right)+\sigma_{yz}^{2}-\sigma_{xz}^{2}\right]\right\\}dz,\end{split}$
(9) $\begin{split}V_{2}=\Delta\sin
2\phi&=\int\left\\{2C_{1}\sigma_{xy}\right.\\\
&\left.+C_{2}\left[2\left(\sigma_{yy}+\sigma_{xx}\right)\sigma_{xy}+2\sigma_{yz}\sigma_{xz}\right]\right\\}dz.\end{split}$
(10)

Note that $C_{1}=\alpha_{1}/\eta$ and $C_{2}=\alpha_{2}/\eta^{2}$.

As described in Section 1, the second-order terms of the stress component
along the camera’s optical axis in Eqs. (9) and (10), i.e., $\sigma_{xz}$ and
$\sigma_{yz}$, have been neglected in previous studies of flow birefringence,
which assumed that $C_{2}=0$ (Aben and Puro, 1997; Ober et al., 2011). This
approach leads to the following well-known equations of integrated
photoelasticity for solids (Aben and Guillemet, 1993; Yokoyama et al., 2023a):

$\displaystyle V_{1}$ $\displaystyle=$ $\displaystyle\Delta\cos
2\phi=C_{1}\int\left(\sigma_{yy}-\sigma_{xx}\right)dz,$ (11) $\displaystyle
V_{2}$ $\displaystyle=$ $\displaystyle\Delta\sin
2\phi=2C_{1}\int\sigma_{xy}dz.$ (12)

Hereafter, we refer to Eqs. (11) and (12) as the “1st-order SOL”, often simply
called the SOL in conventional studies. This formulation excludes the stress
component along the camera’s optical axis. In the remainder of this paper, we
call Eqs. (9) and (10), in which $C_{2}\neq 0$, the “2nd-order SOL”. This
formulation includes the stress component along the camera’s optical axis.

Based on 1st- and 2nd-order SOLs, we can calculate the retardation $\Delta$ if
the stress field is known a priori.

###  2.2 Working fluids

The working fluids disperse CNCs (CNC-HSFD, Cellulose Lab, Ltd.) in ultra-pure
water. The CNCs are mixed with ultra-pure water using a magnetic stirrer
(CHPS-170DF, ASONE Co., Ltd.) at 25 ∘C and 600 rpm for at least 1 h. The CNC
suspensions are then sonicated using an ultrasonic processor (UX-300, Mitsui
Electric Co. Ltd.) for 10 min, which reduces the error between the
experimental and theoretical orientation (Lane et al., 2022a; Ober et al.,
2011). CNC suspensions with concentrations of 0.1, 0.3, 0.5, and 1.0 wt% are
used in the experiments. Figure 2 shows the shear viscosity $\eta$ versus the
shear rate $\dot{\gamma}$ of the CNC suspensions and ultra-pure water, as
measured using a rheometer (MCR302, Anton Paar Co., Ltd.). The shear
viscosities at $\dot{\gamma}=10^{3}$ 1/s are 1.01, 1.06, 1.27, and 1.58
mPa$\cdot$s for 0.1, 0.3, 0.5, and 1.0 wt% CNC suspensions, respectively.
These values are used when the theoretical flow field is calculated. There is
negligible interparticle interaction in the CNC suspensions if the
concentration is less than 0.5 wt% (Lane et al., 2022a; Bertsch et al., 2019).
The shear rheology of the 1.0 wt% CNC suspension exhibits a slight shear-
thinning viscosity. We evaluated the effect of this weak shear-thinning
property on the flow field through numerical simulations using commercial
software (COMSOL Multiphysics). As described in detail in Appendix A, its flow
field is very similar to that of a Newtonian fluid. Therefore, all CNC
suspensions used in this study can be considered as Newtonian fluids.

Figure 2: Shear viscosity $\eta$ versus shear rate $\dot{\gamma}$ of CNC
suspensions with concentrations $c$ of 0.1, 0.3, 0.5, and 1.0 wt% and ultra-
pure water, as measured by a rheometer. The dashed line indicates the low-
torque limit of the rheometer.

###  2.3 Experiments

A schematic diagram of the experimental setup is shown in Fig. 3, along with
the coordinates of the system. A linear polarizer, a quarter-wave plate, and a
rectangular channel are placed between a light source (SOLIS-565C, Thorlabs),
which generates green light with a typical wavelength $\lambda$ of 540 nm, and
a high-speed polarization camera (CRYSTA PI-5WP, Photron, temporal resolution:
250 fps, spatial resolution: 848 $\times$ 680 pixels for an 8-bit light
intensity image). The light passes through the linear polarizer at $0^{\circ}$
and the quarter-wave plate at $45^{\circ}$ with respect to the $y$-axis, and
becomes circularly polarized. As the light passes through the stress field, it
accumulates a retardation and orientation corresponding to the stress states
encountered along the camera’s optical axis. After passing through the stress
field, the light becomes elliptically polarized and is recorded by the
polarization camera, which contains “super-pixels” (Lane et al., 2022b). Each
super-pixel of the image sensor within the polarization camera consists of
four linear polarizers with four different directions: $0^{\circ}$,
$45^{\circ}$, $90^{\circ}$, and $135^{\circ}$. The light intensity values
measured by the camera’s sensor through each linear polarizer are denoted as
$I_{0^{\circ}},I_{45^{\circ}},I_{90^{\circ}}$, and $I_{135^{\circ}}$,
respectively, as shown in Fig. 3(b).

Using the four-step phase-shifting method (Ramesh, 2021; Otani et al., 1994;
Onuma and Otani, 2014; Yokoyama et al., 2023b; Miyazaki et al., 2021), the
retardation $\Delta$ is obtained from the four intensity values of the super-
pixels as follows:

$\displaystyle\Delta=\frac{\lambda}{2\pi}\sin^{-1}{\frac{\sqrt{\left(I_{90^{\circ}}-I_{0^{\circ}}\right)^{2}+\left(I_{45^{\circ}}-I_{135^{\circ}}\right)^{2}}}{I/2}},$
(13)

where $I=I_{0^{\circ}}+I_{45^{\circ}}+I_{90^{\circ}}+I_{135^{\circ}}$. An
example of the measured retardation calculated using software (Photron Ltd.,
CRYSTA Stress Viewer) is shown in Fig. 4(a). The spatial resolution of the
retardation data is 424 $\times$ 340 pixels, which is a quarter of 848
$\times$ 680 pixels. The background noise is subtracted before measuring the
retardation field during fluid flow. To reduce the shot noise of the camera,
the measured retardation is spatially averaged in the flow direction
($x$-direction) and temporally averaged over $0.4$ s, resulting in a line
profile (Fig. 4(b)). As discussed later, the flow field is effectively
constant in the flow direction and time. We verified the lowest measurable
retardation of the polarization camera used in this study by measuring the
retardation field of the rectangular channel filled by a non-flowing CNC
suspension. Figure 5 shows the temporal evolution of the spatially averaged
retardation $\Delta_{\rm Mean}$, which exhibits retardation of approximately
$1.27$ nm. This is considered to be the lowest measurable retardation of the
polarization camera.

Figure 3: (a) Schematic diagram of the experimental setup. (b) Raw image taken
by the polarization camera. The inset shows that the neighboring pixels are
polarizers with different directions. The retardation is calculated from the
intensity values of the four pixels next to each other. Figure 4:  (a) Spatial
distribution of the measured retardation field $\Delta$ of CNC 0.5 wt% in the
case of $Q=30$ ml/min with an aspect ratio of $w/d=1$. (b) Line profile of the
spatiotemporally averaged retardation of (a) over a period of 0.4 s. Figure 5:
Temporal evolution of spatially averaged retardation at rest for 0.5 wt% CNC
suspension and aspect ratio of 1. Error bars indicate the standard deviation
of the calculated average retardation over the entire image.

We prepared channels with three aspect ratios $w/d$, i.e., $w/d=0.1,1$, and
$3$. The aspect ratio $w/d=0.1$ was produced using a channel width of $2w=1$
mm and a channel depth of $2d=10$ mm. The aspect ratio $w/d=1$ was produced
using a channel width of $2w=2$ mm and a channel depth of $2d=2$ mm. The
aspect ratio $w/d=3$ was produced using a channel width of $2w=6$ mm and a
channel depth of $2d=2$ mm. The rectangular channel with $w/d=1$ was made of
quartz glass. The rectangular channels with aspect ratios of $w/d=0.1$ and $3$
were prepared using a three-dimensional printer (Form3, Formlabs; material:
clear resin). To create a flat surface, the front and back sides of the
channels were polished. The flow channels were sealed with two glass plates
level with the surfaces. The flow of the channel with $w/d=0.1$ can be
considered quasi-two-dimensional, as mentioned above.

The coordinate system is set up as shown in Fig. 3. The length of the channel
is around 50 mm. The fluid was supplied using a syringe pump (Pump 11 Elite,
Harvard Apparatus) to produce a steady laminar flow. The flow rates in the
experiments ranged from 1–100 ml/min. The Reynolds number $Re=\rho
U_{mean}R_{H}/\eta$ reaches a maximum value of approximately 200 when the flow
rate is 100 ml/min using the channel with $w/d=1$. Here, $U_{mean}$ is the
mean velocity, defined by $U_{mean}=Q/(4wd)$, and $R_{H}$ is the hydraulic
radius, defined by $R_{H}=wd/(w+d)$. The measurement area extends over $x\sim
45$ mm. The entrance length $L_{e}$, which is defined as $L_{e}=0.06ReR_{H}$,
is sufficiently smaller than the distance between the flow inlet and the
measurement position in all experimental conditions. Therefore, the flow field
is effectively fully developed.

###  2.4 Theoretical flow field

Figure 6: (a,b,c) Velocity fields for different aspect ratios $w/d$. (d,e,f)
Distributions of shear stress $\sigma_{xy}$ for different aspect ratios $w/d$.
(g,h,i) Distributions of shear stress $\sigma_{xz}$ for different aspect
ratios $w/d$. All quantities calculated using $Q=30$ ml/min, $\eta=1.12$
mPa$\cdot$s.

This section describes the theoretical flow field in a rectangular channel to
calculate the retardation and orientation field to be compared with
experiments. First, the velocity distribution of a steady laminar flow in a
rectangular channel is derived from the Navier–Stokes equations as follows
(Delplace, 2018; Nagamoto et al., 2006):

$\displaystyle\left(\frac{\partial^{2}}{\partial
y^{2}}+\frac{\partial^{2}}{\partial z^{2}}\right)u_{x}=-\frac{4Q}{wd^{3}K},$
(14) $\displaystyle u_{y}=u_{z}=0,$ (15)

where

$K=\frac{16}{3}-\frac{1024}{\pi^{5}}\frac{d}{w}\sum_{n=0}^{\infty}\frac{1}{(2n+1)^{5}}\tanh{\frac{\left(2n+1\right)\pi
w}{2d}}.$ (16)

Here, $Q$ is the flow rate and $w$, $d$ are the channel width and channel
depth, respectively. $u_{y}$ and $u_{z}$ are the velocities in the $y$\- and
$z$-directions, respectively. The velocity distribution $u_{x}(y,z)$, which
satisfies Eqs. (14)–(16), and the no-slip boundary condition at the wall
($y=\pm w$ or $z=\pm d$) are defined as follows:

$\begin{split}u_{x}(y,z)&=\frac{64Q}{wd\pi^{3}K}\sum_{n=0}^{\infty}\frac{(-1)^{n}}{(2n+1)^{3}}\\\
&\left[1-\dfrac{\cosh\dfrac{(2n+1)\pi y}{2d}}{\cosh\dfrac{(2n+1)\pi
w}{2d}}\right]\cos\dfrac{(2n+1)\pi z}{2d},\end{split}$ (17) $u_{y}=u_{z}=0.$
(18)

Examples of the theoretical velocity distributions with different aspect
ratios $w/d$ are shown in Figs. 6(a,b,c). From Eqs. (17) and (18), the viscous
stresses are calculated as follows:

$\begin{split}\sigma_{xy}&(y,z)=\eta\frac{\partial u_{x}}{\partial
y}=\frac{64Q\eta}{wd\pi^{3}K}\sum_{n=0}^{\infty}\dfrac{(-1)^{n}}{(2n+1)^{3}}\\\
&\dfrac{(2n+1)\pi}{2d}\dfrac{\sinh\dfrac{(2n+1)\pi
y}{2d}}{\cosh\dfrac{(2n+1)\pi w}{2d}}\cos\dfrac{(2n+1)\pi z}{2d},\end{split}$
(19) $\begin{split}&\sigma_{xz}(y,z)=\eta\frac{\partial u_{x}}{\partial
z}=\frac{64Q\eta}{wd\pi^{3}K}\sum_{n=0}^{\infty}\dfrac{(-1)^{n}}{(2n+1)^{3}}(-1)\\\
&\dfrac{(2n+1)\pi}{2d}\left[1-\dfrac{\cosh\dfrac{(2n+1)\pi
y}{2d}}{\cosh\dfrac{(2n+1)\pi w}{2d}}\right]\sin\dfrac{(2n+1)\pi
z}{2d},\end{split}$ (20) $\sigma_{xx}=\sigma_{yy}=\sigma_{zz}=\sigma_{yz}=0.$
(21)

Examples of the theoretical stress fields in the channel are shown in Figs.
6(d,e,f) and 6(g,h,i). Therefore, Eqs. (9) and (10) in this flow field become

$\displaystyle V_{1}=\Delta\cos 2\phi=-C_{2}\int\sigma_{xz}^{2}dz,$ (22)
$\displaystyle V_{2}=\Delta\sin 2\phi=2C_{1}\int\sigma_{xy}dz.$ (23)

In particular, the stress component at the wall ($y=\pm w$) and at the channel
center ($y=0$) are

$\begin{split}\sigma_{xy}&(\pm
w,z)=\frac{64Q\eta}{wd\pi^{3}K}\sum_{n=0}^{\infty}\dfrac{(-1)^{n}}{(2n+1)^{3}}(\pm
1)\\\ &\dfrac{(2n+1)\pi}{2d}\tanh\dfrac{(2n+1)\pi w}{2d}\cos\dfrac{(2n+1)\pi
z}{2d},\end{split}$ (24) $\displaystyle\sigma_{xz}(\pm w,z)=0,$ (25)
$\displaystyle\sigma_{xy}(0,z)=0,$ (26)
$\begin{split}&\sigma_{xz}(0,z)=\dfrac{64Q\eta}{wd\pi^{3}K}\sum_{n=0}^{\infty}\dfrac{(-1)^{n}}{(2n+1)^{3}}(-1)\\\
&\dfrac{(2n+1)\pi}{2d}\left[1-\dfrac{1}{\cosh\dfrac{(2n+1)\pi
w}{2d}}\right]\sin\dfrac{(2n+1)\pi z}{2d}.\end{split}$ (27)

Thus, the theoretical retardation and orientation can be calculated using Eqs.
(4), (20)–(23) and compared with the experimental data. In particular, the
scaling of the retardation at the wall ($y=\pm w$), $\Delta_{\rm wall}$, and
the retardation at the center ($y=0$), $\Delta_{\rm center}$, can be
calculated as follows:

$\displaystyle\Delta_{\rm wall}=\left|2C_{1}\int\sigma_{xy}(\pm
w,z)dz\right|\propto Q,$ (28) $\displaystyle\Delta_{\rm
center}=\left|-C_{2}\int\sigma_{xz}^{2}(0,z)dz\right|\propto Q^{2}.$ (29)

These equations suggest that the retardation at the wall increases in
proportion to the 1st power of $Q$ and the retardation at the center increases
in proportion to the 2nd power of $Q$. Furthermore, if we use the 1st-order
SOL, i.e., $C_{2}=0$, the retardation is expected to be zero at the center of
the channel.

In terms of the effects of stress components along the camera’s optical axis
on flow birefringence, i.e., $C_{2}$ term in the SOL, the effect is most
pronounced at the center of the channel. The effect is negligible at the wall.
In other words, if the effect is not ignored, the retardation at the center of
the channel will not be zero, although the retardation at the wall may not be
affected. Therefore, the retardation at the wall and at the center of the
channel are the main targets of discussion in this study.

Figure 7:  Line profiles of the retardation for different aspect ratios, CNC
concentrations, and flow rates. The horizontal axis is normalized by the width
of the channel $w$. The error bars show the standard deviation calculated when
the retardation is temporally averaged over 250 frames, but the size of the
marker tends to obscure these quantities.

##  3 Results and discussion

Figure 8:  (Left column) Relationship between retardation at $y=\pm w$
$\Delta_{\rm wall}$ and flow rate $Q$ with different CNC concentrations: (a)
0.1 wt%, (c) 0.3 wt%, (e) 0.5 wt%, and (g) 0.1 wt%. (Right column)
Relationship between retardation at the center $\Delta_{\rm wall}$ and flow
rate $Q$ with different CNC concentrations: (a) 0.1 wt%, (c) 0.3 wt%, (e) 0.5
wt%, and (g) 0.1 wt%. Solid lines in (a,c,e,g) have a slope of 1, indicating
$\Delta_{\rm wall}\propto Q$ predicted by Eq. (28). Dashed lines in (b,d,f,h)
have a slope of 2, indicating $\Delta_{\rm center}\propto Q^{2}$ predicted by
Eq. (29). The error bars show the standard deviation calculated when the
measured retardation is temporally averaged over 250 frames. Figure 9:
Comparison between (a) experimental and (b) theoretical retardation profiles
for the 0.5 wt% CNC suspension and different aspect ratios $w/d$. The
retardation distributions $\Delta(y)$ are normalized by the retardation at the
wall $\Delta_{\rm wall}$. The dashed and solid lines in (b) are calculated
using 1st- and 2nd-order SOLs, respectively. For $w/d=0.1$, the flow rate is
$Q=20$ ml/min and we use $C_{1}=1.51\times 10^{-5}$ 1/Pa and $C_{2}=4.19\times
10^{-4}$ 1/Pa2 to calculate the theoretical retardation profile. For the case
of $w/d=1$, the flow rate is $Q=70$ ml/min and we use $C_{1}=1.52\times
10^{-5}$ 1/Pa and $C_{2}=1.11\times 10^{-5}$ 1/Pa2 to calculate the
theoretical retardation profile. For the case of $w/d=3$, the flow rate is
$Q=50$ ml/min and we use $C_{1}=1.35\times 10^{-5}$ 1/Pa and $C_{2}=2.8\times
10^{-5}$ 1/Pa2 to calculate the theoretical retardation profile.

###  3.1 Retardation distributions

The retardation fields were systematically measured using three rectangular
channels and four CNC suspensions with different concentrations. We collected
more than 100 data. Figure 7 shows the line profiles of the measured
retardation at each flow rate $Q$. In all cases, the highest retardation
values appear at the channel wall ($y=\pm w$) and the lowest values occur at
the channel center ($y=0$). The retardation at each $y$-position increases
with the flow rate. At the wall ($y=\pm y$), the retardation for the same flow
rate increases with the concentration $c$ of CNC. This means that the
concentration of CNC is related to the degree of birefringence, i.e., the
stress-optic coefficient $C_{1}$ of Eq. (28), that occurs at a given flow
rate. The spatial distributions of the retardation vary with the aspect ratio
of the channel. The rectangular channel with an aspect ratio of 0.1 exhibits a
V-shaped retardation distribution with respect to $y/w$ because of the quasi-
two-dimensional flow field. The retardation distribution deviates from the
V-shape as the aspect ratio increases. The rectangular channel with an aspect
ratio of 3 displays a plateau around $y=0$ when the concentration $c$ or the
flow rate is low. For example, in Fig. 7(c), the minimum retardation is
approximately 1 nm. This value is comparable to the retardation at rest, as
shown in Fig. 5. Therefore, the retardation may actually be lower than this
value, but the camera noise prevents us from obtaining accurate measurements.
These results indicate that the use of CNC suspensions enables the measurement
of flow birefringence distributions that reflect the aspect ratio $w/d$, i.e.,
the channel geometry, even at low CNC concentrations.

Equation (29) suggests that the retardation at the center of the channel,
$\Delta_{\rm center}$, should be zero if there is no effect of the stress
component along the camera’s optical axis on the flow birefringence, i.e.,
$C_{2}=0$. However, when the flow rate is sufficiently high, $\Delta_{\rm
center}$ clearly shows a finite value and increases with the flow rate $Q$,
even for the quasi-two-dimensional channel ($w/d=0.1$) and at low CNC
concentrations. This result indicates that the term relating to the stress
component along the camera’s optical axis in the SOL must be considered in the
flow birefringence, except when the flow rate is sufficiently low and when the
channel flow is quasi-two-dimensional.

###  3.2 Effect of stress component along the camera’s optical axis on flow
birefringence

To quantitatively assess the effect of the stress component along the camera’s
optical axis on flow birefringence, the retardation values at the wall
$\Delta_{\rm wall}$ and center $\Delta_{\rm center}$ of the channel are
plotted against the flow rate $Q$ in Fig. 8. The left-hand side of Fig. 8
shows that the retardation at the wall increases in proportion to the 1st
power of $Q$. This result agrees with the expectation of both the 1st- and
2nd-order SOLs described in Eq. (28). Although a previous study suggested that
retardation will reach a saturation level because of the saturated alignment
of CNC (Detert et al., 2023; Calabrese et al., 2023) at sufficiently high flow
rates, the relationship between retardation and flow rate remains linear
within the range considered in this study. The right-hand side of Fig. 8 shows
that the retardation at the center increases in proportion to the 2nd power of
$Q$ if $Q$ is sufficiently large ($Q\geq 10$ ml/min in our study). This result
does not agree with the 1st-order SOL in Eqs. (11) and (12), but does agree
with the expectation of the 2nd-order SOL in Eqs. (9) and (10), for which the
stress component along the camera’s optical axis, i.e., $C_{2}$, is nonzero.
These results indicate that the flow birefringence of CNC suspensions in a
three-dimensional flow field can be better understood using the 2nd-order SOL,
which considers the effect of the stress component along the camera’s optical
axis, rather than the 1st-order SOL that has conventionally been used.

As the lowest measurable retardation of the polarization camera used in this
study is around $1.27$ nm, retardation values of less than around $1$ nm can
be considered as noise. For example, almost all $\Delta_{\rm center}$ for
$w/d=3$ are at the noise level because the stress along the $z$-axis and its
integrating length are small for this case. This region can be further
verified using a long channel with a larger cross-section to gain a longer
integrating length. Additionally, $\Delta_{\rm center}$ is at the noise level
for $w/d=0.1$ with the 1.0 wt% CNC suspension because of the low flow rate,
$Q\leq 9$ ml/min.

To illustrate the practicality of the 2nd-order SOL, we now reexamine the
retardation distribution. Figure 9 compares the experimental and theoretical
retardation line profiles for all channel aspect ratios. The retardation
profiles are normalized by the retardation at the wall, $\Delta_{\rm wall}$.
In calculating the line profiles of the retardation using the 2nd-order SOL,
$C_{1}$ and $C_{2}$ match the retardation at the wall and center of the
channel with the respective experimental values. The values of $C_{2}$ and
$C_{1}$ for all cases are described in Appendix B. Figure 9 clearly shows
that, for any aspect ratio, the retardation profiles calculated using the 2nd-
order SOL, i.e., $C_{2}\neq 0$, are in better agreement with the experimental
results than those calculated using the 1st-order SOL, i.e., $C_{2}=0$.

##  4 Conclusion

This study has attempted to clarify the effect of the stress component along
the camera’s optical axis on the flow birefringence of CNC suspensions by
revisiting the SOL described by Doyle (1982). This effect has often been
neglected in conventional 1st-order SOLs. The 2nd-order SOL includes the
stress component along the camera’s optical axis. In this study, systematic
experiments were carried out to measure the flow birefringence of CNC
suspensions in steady laminar flow through rectangular channels, and more than
100 datasets were assessed. The retardation distributions were measured for
various CNC suspensions (with 0.1, 0.3, 0.5, and 1.0 wt% CNC concentrations)
flowing through rectangular channels with aspect ratios of 0.1, 1, and 3. The
measured retardation values were compared with those predicted theoretically
by the 1st- and 2nd-order SOLs.

The results show that the retardation distribution varies with the channel
aspect ratio. The 1st-order SOL holds in flow fields where the stress
component along the camera’s optical axis ($z$-axis) is negligible, such as in
a quasi-two-dimensional flow channel with a small aspect ratio and low flow
rate. However, in cases where the stress component along the camera’s optical
axis cannot be ignored, such as in the case of a high flow rate, the 1st-order
SOL cannot be applied, even for quasi-two-dimensional channel flows. In such
cases, the 2nd-order SOL predicts the retardation distribution with reasonable
accuracy based on experimental data. Focusing on the retardation at the
channel center, where the effect of the stress component along the camera’s
optical axis is most pronounced, the 2nd-order SOL predicts the retardation to
be proportional to the 2nd power of the flow rate $Q$, in agreement with the
experimental retardation results. To conclude, when measuring flow
birefringence in the three-dimensional flow of CNC suspensions, the stress
component along the camera’s optical axis must be considered. The results of
this study will be of great importance in studies of flow birefringence in
three-dimensional channels for CNCs and other materials.

##  Appendix Appendix A Non-Newtonian effect of 1.0 wt% CNC Suspension

Figure 10:  (a) Viscosity of 1.0 wt% CNC suspension. The solid line shows the
power law model of $\eta=K\dot{\gamma}^{n-1}$ with $n\simeq 0.9$ and $K\simeq
4.0$. (b,c,a) are the velocity profiles along the $z$-axis at $y=0$ when
$Q=30$ ml/min for aspect ratios $w/d$ of 0.1, 1, and 3, respectively. The
dashed lines show the theoretical velocity profile calculated by Eq. (17)
using a constant viscosity of 1.58 $\rm mPa\cdot s$. The solid lines show the
numerically calculated velocity profile of the power law fluid.

We now verify the non-Newtonian effect of the 1.0 wt% CNC suspension. The
slight shear-thinning viscosity of the suspension was fitted by a power law
model described by the following equation:

$\eta=K\dot{\gamma}^{n-1},$ (30)

where $K$ is the flow consistency index and $n$ is the power exponent. Figure
10(a) shows the viscosity of the 1.0 wt% CNC suspension and the fitted power
law model using $K\simeq 4.0$ and $n\simeq 0.9$. The velocity profiles of the
1.0 wt% CNC suspension flowing in rectangular channels with different aspect
ratios were numerically calculated using commercial software (COMSOL
Multiphysics) with the power law model and these constants.

Figures 10(b,c,d) show the measured velocity profiles (solid lines) for aspect
ratios of 0.1, 1, and 3, respectively, and the theoretical distribution
(dashed lines) calculated by Eq. (17) using a constant viscosity of 1.58 $\rm
mPa\cdot s$. The difference between the velocity distributions of non-
Newtonian and Newtonian fluids is particularly small because the power
exponent of 0.9 is close to 1. Therefore, the non-Newtonian effect of the 1.0
wt% CNC suspension is negligible.

##  Appendix Appendix B Stress-optic coefficients

Figure 11:  (a) $C_{1}$ and (b) $C_{2}$ adjusted to minimize the difference
between the calculated theoretical retardation distribution and the
experimental values at the wall and center of the channel, denoted as
$\Delta_{\rm wall}$ and $\Delta_{\rm center}$, respectively. The solid line in
(a) shows the linear relationship between $C_{1}$ and $c$, $C_{1}\propto
c^{1}$.

Figure 11 shows the adjusted $C_{1}$ and $C_{2}$ such that the retardation at
the wall and the center of the channel, $\Delta_{\rm wall}$ and $\Delta_{\rm
center}$, are closest to the experimental value when calculating the
theoretical retardation distribution. $C_{1}$ is linearly proportional to the
CNC concentration $c$ and is nearly independent of the aspect ratio $w/d$. The
value of $C_{2}$ also increases with the CNC concentration, but varies with
the aspect ratio. This means that the magnitude of the effect of the stress
component along the camera’s optical axis on flow birefringence changes with
the flow condition, because $C_{2}$ is the contribution level of the stress
component along the camera’s optical axis in the SOL. Additionally, the
procedure performed in this study provides a method for determining $C_{1}$
and $C_{2}$ in birefringent fluids.

## Statements and Declarations

## Acknowledgments

We thank Dr. T. Shikata, Dr. T. Rösgen, and Dr. C. Lane for their valuable
discussions and suggestions.

## Funding

This work was funded by Japan Society for the Promotion of Science KAKENHI
(Grant Numbers: JP19K23483, JP20K14646, JP20H00222, JP20H00223, JP22KJ1239),
Japan Science and Technology Agency PRESTO (Grant Number: JPMJPR21O5), and
Japan Agency for Medical Research and Development (Grant Number:
JP22he0422016).

## Author contributions

Y.T. and M.M. conceptualized and supervised the study. K.N., Y.Y., W.W., and
M.M. developed the methodology. K.N., Y.Y., and W.W. conducted the formal
analysis and investigation. K.N., Y.Y., and W.W. prepared the original draft
manuscript. All authors reviewed and edited the manuscript.

## Competing interest

The authors have no conflicts of interest to declare that are relevant to the
content of this article.

## Data availability

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * Aben and Guillemet (1993) Aben H, Guillemet C (1993) Photoelasticity of glass. Springer Berlin Heidelberg, Berlin, Heidelberg, DOI 10.1007/978-3-642-50071-8
  * Aben and Puro (1997) Aben H, Puro A (1997) Photoelastic tomography for three-dimensional flow birefringence studies. Inverse Problems 13(2):215–221, DOI 10.1088/0266-5611/13/2/002
  * Aben et al. (1989) Aben HK, Josepson JI, Kell KJE (1989) The case of weak birefringence in integrated photoelasticity. Optics and Lasers in Engineering 11(3):145–157, DOI 10.1016/0143-8166(89)90029-8
  * Ainola (1998) Ainola L (1998) Application of integrated photoelasticity to the determination of viscous flow velocity in closed conduits. Proceedings of the Estonian Academy of Sciences, Physics and Mathematics 47(1):48–55
  * Beck-Candanedo et al. (2005) Beck-Candanedo S, Roman M, Gray DG (2005) Effect of reaction conditions on the properties and behavior of wood cellulose nanocrystal suspensions. Biomacromolecules 6(2):1048–1054
  * Bertsch et al. (2019) Bertsch P, Sánchez-Ferrer A, Bagnani M, Isabettini S, Kohlbrecher J, Mezzenga R, Fischer P (2019) Ion-induced formation of nanocrystalline cellulose colloidal glasses containing nematic domains. Langmuir 35(11):4117–4124, DOI 10.1021/acs.langmuir.9b00281
  * Buffa et al. (2019) Buffa JM, Casado U, Mucci V, Aranguren MI (2019) Cellulose nanocrystals in aqueous suspensions: rheology of lyotropic chiral liquid crystals. Cellulose 26(4):2317–2332, DOI 10.1007/s10570-019-02278-3
  * Calabrese et al. (2021a) Calabrese V, Haward SJ, Shen AQ (2021a) Effects of shearing and extensional flows on the alignment of colloidal rods. Macromolecules 54(9):4176–4185
  * Calabrese et al. (2021b) Calabrese V, Varchanis S, Haward SJ, Tsamopoulos J, Shen AQ (2021b) Structure-property relationship of a soft colloidal glass in simple and mixed flows. Journal of Colloid and Interface Science 601:454–466
  * Calabrese et al. (2022) Calabrese V, Varchanis S, Haward SJ, Shen AQ (2022) Alignment of colloidal rods in crowded environments. Macromolecules 55(13):5610–5620, DOI 10.1021/acs.macromol.2c00769
  * Calabrese et al. (2023) Calabrese V, Santos TP, Lopez CG, Lettinga MP, Haward SJ, Shen AQ (2023) Extensibility governs the flow-induced alignment of polymers and rod-like colloids. DOI 10.48550/arXiv.2311.13093, 2311.13093
  * Clemeur et al. (2004) Clemeur N, Rutgers RPG, Debbaut B (2004) Numerical evaluation of three dimensional effects in planar flow birefringence. Journal of Non-Newtonian Fluid Mechanics 123(2):105–120, DOI 10.1016/j.jnnfm.2004.07.002
  * Delplace (2018) Delplace F (2018) Laminar flow of Newtonian liquids in ducts of rectangular cross-section a model for both physics and mathematics. Open Access J Math Theor Phys 1:198–201
  * Detert et al. (2023) Detert M, Santos TP, Shen AQ, Calabrese V (2023) Alignment–rheology relationship of biosourced rod-like colloids and polymers under flow. Biomacromolecules DOI 10.1021/acs.biomac.3c00347
  * Doyle (1982) Doyle JF (1982) On a nonlinearity in flow birefringence. Experimental Mechanics 22(1):37–38
  * Habibi et al. (2010) Habibi Y, Lucia LA, Rojas OJ (2010) Cellulose Nanocrystals: Chemistry, Self-Assembly, and Applications. Chemical Reviews 110(6):3479–3500, DOI 10.1021/cr900339w
  * Hausmann et al. (2018) Hausmann MK, Rühs PA, Siqueira G, Läuger J, Libanori R, Zimmermann T, Studart AR (2018) Dynamics of cellulose nanocrystal alignment during 3D printing. ACS Nano 12(7):6926–6937, DOI 10.1021/acsnano.8b02366
  * Janeschitz-Kriegl (2012) Janeschitz-Kriegl H (2012) Polymer melt rheology and flow birefringence, vol 6. Springer Science & Business Media
  * Kádár et al. (2021) Kádár R, Spirk S, Nypelö T (2021) Cellulose nanocrystal liquid crystal phases: progress and challenges in characterization using rheology coupled to optics, scattering, and spectroscopy. ACS Nano 15(5):7931–7945, DOI 10.1021/acsnano.0c09829
  * Kim et al. (2017) Kim J, Michelin S, Hilbers M, Martinelli L, Chaudan E, Amselem G, Fradet E, Boilot JP, Brouwer AM, Baroud CN, et al. (2017) Monitoring the orientation of rare-earth-doped nanorods for flow shear tomography. Nature nanotechnology 12(9):914–919
  * Knight Jr and Pih (1976) Knight Jr CE, Pih H (1976) Orthotropic stress-optic law for plane stress photoelasticity of composite materials. Fibre Science and Technology 9(4):297–313
  * Lagerwall et al. (2014) Lagerwall JPF, Schütz C, Salajkova M, Noh J, Hyun Park J, Scalia G, Bergström L (2014) Cellulose nanocrystal-based materials: from liquid crystal self-assembly and glass formation to multifunctional thin films. NPG Asia Materials 6(1):e80–e80, DOI 10.1038/am.2013.69
  * Lane et al. (2022a) Lane C, Rode D, Rösgen T (2022a) Birefringent properties of aqueous cellulose nanocrystal suspensions. Cellulose 29(11):6093–6107, DOI 10.1007/s10570-022-04646-y
  * Lane et al. (2022b) Lane C, Rode D, Rösgen T (2022b) Calibration of a polarization image sensor and investigation of influencing factors. Applied Optics 61(6):C37–C45, DOI 10.1364/AO.437391
  * Lane et al. (2023) Lane C, Baumann F, Rode D, Rösgen T (2023) Two-dimensional strain rate imaging study using a polarization camera and birefringent aqueous cellulose nanocrystal suspensions. Experiments in Fluids 65(1):8, DOI 10.1007/s00348-023-03730-8
  * McAfee and Pih (1974) McAfee WJ, Pih H (1974) Scattered-light flow-optic relations adaptable to three-dimensional flow birefringence. Experimental Mechanics 14(10):385–391, DOI 10.1007/BF02324941
  * Miyazaki et al. (2021) Miyazaki Y, Usawa M, Kawai S, Yee J, Muto M, Tagawa Y (2021) Dynamic mechanical interaction between injection liquid and human tissue simulant induced by needle-free injection of a highly focused microjet. Scientific Reports 11(1):14,544, DOI 10.1038/s41598-021-94018-6
  * Mu et al. (2019) Mu R, Hong X, Ni Y, Li Y, Pang J, Wang Q, Xiao J, Zheng Y (2019) Recent trends and applications of cellulose nanocrystals in food industry. Trends in Food Science & Technology 93:136–144, DOI 10.1016/j.tifs.2019.09.013
  * Nagamoto et al. (2006) Nagamoto H, Kobayashi Y, Otomo J, Oshima E (2006) Laminar flow characteristics in a rectangular microchannel. Kagaku Kougaku Ronbunshu 32(3):293–296, DOI 10.1252/kakoronbunshu.32.293
  * Noto et al. (2020) Noto D, Tasaka Y, Hitomi J, Murai Y (2020) Applicability evaluation of the stress-optic law in Newtonian fluids toward stress field measurements. Physical Review Research 2(4):043,111
  * Ober et al. (2011) Ober TJ, Soulages J, McKinley GH (2011) Spatially resolved quantitative rheo-optics of complex fluids in a microfluidic device. Journal of Rheology 55(5):1127–1159
  * Onuma and Otani (2014) Onuma T, Otani Y (2014) A development of two-dimensional birefringence distribution measurement system with a sampling rate of 1.3mhz. Optics Communications 315:69–73, DOI 10.1016/j.optcom.2013.10.086
  * Otani et al. (1994) Otani Y, Shimada T, Yoshizawa T, Umeda N (1994) Two-dimensional birefringence measurement using the phase shifting technique. Optical Engineering 33(5):1604–1609, DOI 10.1117/12.168435
  * Prabhakaran (1975) Prabhakaran R (1975) On the stress-optic law for orthotropic-model materials in biaxial-stress fields. Experimental Mechanics 15(1):29–34
  * Ramesh (2021) Ramesh K (2021) Developments in photoelasticity. IOP Publishing, DOI 10.1088/978-0-7503-2472-4
  * Saito et al. (2013) Saito T, Kuramae R, Wohlert J, Berglund LA, Isogai A (2013) An Ultrastrong Nanofibrillar Biomaterial: The Strength of Single Cellulose Nanofibrils Revealed via Sonication-Induced Fragmentation. Biomacromolecules 14(1):248–253, DOI 10.1021/bm301674e
  * Thiruganasambanthan et al. (2022) Thiruganasambanthan T, Ilyas RA, Norrrahim MNF, Kumar TSM, Siengchin S, Misenan MSM, Farid MAA, Nurazzi NM, Asyraf MRM, Zakaria SZS, Razman MR (2022) Emerging Developments on Nanocellulose as Liquid Crystals: A Biomimetic Approach. Polymers 14(8):1546, DOI 10.3390/polym14081546
  * Yokoyama et al. (2023a) Yokoyama Y, Ichihara S, Tagawa Y (2023a) High-speed photoelastic tomography for axisymmetric stress fields in a soft material: temporal evolution of all stress components. DOI 10.48550/arXiv.2311.10107, 2311.10107
  * Yokoyama et al. (2023b) Yokoyama Y, Mitchell BR, Nassiri A, Kinsey BL, Korkolis YP, Tagawa Y (2023b) Integrated photoelasticity in a soft material: phase retardation, azimuthal angle, and stress-optic coefficient. Optics and Lasers in Engineering 161:107,335, DOI 10.1016/j.optlaseng.2022.107335


# Audio-visual talker localization in video for spatial sound reproduction

###### Abstract

Object-based audio production requires the positional metadata to be defined
for each point-source object, including the key elements in the foreground of
the sound scene. In many media production use cases, both cameras and
microphones are employed to make recordings, and the human voice is often a
key element. In this research, we detect and locate the active speaker in the
video, facilitating the automatic extraction of the positional metadata of the
talker relative to the camera’s reference frame. With the integration of the
visual modality, this study expands upon our previous investigation focused
solely on audio-based active speaker detection and localization. Our
experiments compare conventional audio-visual approaches for active speaker
detection that leverage monaural audio, our previous audio-only method that
leverages multichannel recordings from a microphone array, and a novel audio-
visual approach integrating vision and multichannel audio. We found the role
of the two modalities to complement each other. Multichannel audio, overcoming
the problem of visual occlusions, provides a double-digit reduction in
detection error compared to audio-visual methods with single-channel audio.
The combination of multichannel audio and vision further enhances spatial
accuracy, leading to a four-percentage point increase in F1 score on the
Tragic Talkers dataset. Future investigations will assess the robustness of
the model in noisy and highly reverberant environments, as well as tackle the
problem of off-screen speakers.

##  1 Introduction

In 3D audio-visual production, meticulous attention is required to accurately
align sound sources with the visual events they accompany. To produce and
author an immersive experience, audio sources are generally treated as objects
and manually placed in the virtual space by the producer. This approach to
production is often referred to as object-based media (OBM) production [1, 2].
In OBM, each object, spanning audio, video, graphics, text, or other forms of
media, is accompanied by its metadata. The metadata associated with an
individual object describes specific attributes or desired behaviors of the
object, such as its content or position in space over time. OBM is valued for
its adaptability and interactivity, allowing for tailored experiences based on
user preferences or device configuration. For example, in the context of
producing immersive spatial audio, object-based audio can theoretically be
rendered on any loudspeaker configuration, unlike traditional channel-based
mixes, which lack such flexibility.

Figure 1: Pipeline for speech signals objectification proposed by Mohd Izhar
_et al_. [3] and Schweiger _et al_. [4]. Positional metadata are automatically
predicted by leveraging video and microphone array data. These predictions are
not only used as final positional information for the spatialization of the
objects but also to drive a spatial beamformer. Filtered signals extracted
with the beamformer are associated with, and replaced by, the high-quality
speech data recorded with Lavalier microphones. This paper focuses on the
audio-visual prediction of the speaker’s positional metadata.

However, while it is relatively trivial to manually spatialize synthetic
audio-visual assets, this is not true for real scenes. Ideally, the ultimate
spatialization should truthfully reflect the positioning and movements of the
events. An example of a common audio-visual event is a moving talker. To
tackle this problem, Mohd Izhar _et al_. [3] and Schweiger _et al_. [4]
proposed the speaker objectification pipeline depicted in Figure 1. Automated
predictions regarding the locations of active speakers are generated using an
audio-visual tracker, which utilizes both video data and audio captured
through a microphone array. These predictions drive the spatial listening
directivity of a beamformer employed to filter the signals from the microphone
array. The resulting filtered speech signals are then associated with, and
substituted by, the high-quality speech recordings obtained through Lavalier
microphones.

This paper addresses the automatic extraction of the speaker positional
metadata, corresponding to the green block of the pipeline presented in Figure
1. Specifically, we focus on video-based horizontal active speaker detection
and localization (ASDL) as videos are a widely consumed form of media, and
subjective studies suggest that humans tend to be more spatially sensitive
across the azimuth direction than elevation [5]. The audio-visual localization
is performed on the Tragic Talkers dataset [6] and it expands upon our
previous audio-only studies [7, 8]. Our previous works highlighted the
benefits of employing multichannel audio captured with a microphone array,
outperforming traditional video-based audio-visual approaches for active
speaker detection that employ monaural audio and rely on visual face
detection. Multichannel audio provides a higher detection accuracy as it does
not suffer from visual occlusions. This paper demonstrates that partnering
multichannel audio with the visual modality will improve spatial accuracy
while preserving the high detection rate enabled by multichannel audio.

The remainder of this document provides an overview of the related work on
video-based active speaker detection and localization, describes the proposed
method and the audio-visual network architecture, presents and discusses our
experimental results, concludes the paper and suggests future directions for
the field.

##  2 Related Work

ASDL can be addressed through two distinct phases. Initially, the localization
subtask is undertaken, wherein a visual face detector is utilized to pre-
select a set of candidate speakers. Subsequently, the detected faces undergo
classification into active or inactive. In computer vision, this second
classification process is usually referred to as Active Speaker Detection
(ASD) [9, 10, 11, 12, 13]. Researchers usually partner the video stream with
the respective (mono) audio signal. Pioneering the work on video-based active
speaker detection, Cutler _et al_. [14] proposed to observe the correlation
between mouth motion and audio data. Haider _et al_. [15] combined lip
tracking and voice activity detection (VAD) to predict who is speaking in
multiparty dialogue videos. Chakravarty _et al_. [16] proposed to also include
head and upper-body motion as additional visual cues to detect the active
speaker. They adopted a self-supervised solution to perform ASD by training a
visual network under the supervision of its audio counterpart. Subsequent
works [13, 17], adopted a two-step audio-visual co-training for speaker
detection and identification. They exploited the co-occurrence of speech and
faces in videos to associate clusters generated from speech features with
clusters generated from facial features. With the advance of deep learning
techniques and the availability of larger datasets [18, 19], different yet
related audio-visual tasks have been introduced, such as audio-visual lip
reading [19, 20], lip-voice synchronization [21], and audio-visual speaker
separation [22]. What is probably the first, large, annotated dataset for ASD
was released for the ActivityNet Challenge (Task B) at CVPR 2019: the AVA-
ActiveSpeaker dataset [9]. It provides 38.5 hours of audio-visual face tracks
(sequences of consecutive face crops) labeled for speech activity. Since the
face tracks are provided, the challenge task consists of classifying each face
as active or inactive, leveraging audio and video signals. At the 2019
challenge, the first [23] and second [24] positions were achieved by
leveraging 3D convolutional neural networks (CNNs). After that, Alcázar et al.
proposed a model called Active Speaker in Context (ASC) [12]. Instead of
compute-intensive 3D convolutions or large-scale audio-visual pre-training,
ASC leverages context: in assessing the speech activity of a candidate
speaker, it looks at any other available faces. Zhang et al. [10] also tackled
the ASD task by leveraging contextual information and proposed the UniCon
network. Tao et al. introduced TalkNet [11], an ASD model that leverages
short- and long-term features. Additionally, motivated by the call for an ASD
system that works properly outside the AVA-ActiveSpeaker dataset domain, they
formed a second ASD dataset based on LRS3 [25] and VoxCeleb2 [26] called
TalkSet [11]. Recently, Alcázar et al. [27] proposed an end-to-end ASD that
unifies audio-visual feature extraction and spatio-temporal context
aggregation.

However, these ASD solutions, focusing solely on the audio-visual
classification of the provided face tracks, depend on visual face detection
for the localization subtask. In practice, the speaker can be occluded or
facing away from the camera, leading to face detection failures and subsequent
degradation of the overall system performance. When this happens, monaural
audio is insufficient to compensate for the visual failure, as it lacks the
necessary spatial cues required to locate audio sources. In other words, the
active speaker is detected only when visible. In previous studies [8, 7], we
overcame this problem by leveraging multichannel audio to simultaneously
address the detection and localization aspects of ASDL. Our model was able to
locate the active speaker in the video frames solely from audio inputs,
achieving better detection and recall rates.

Some other recent studies partnered multichannel audio with vision for audio-
visual ASDL. For example, Qian et al. in [28] and [29] employed visual feature
vectors encoding face bounding box coordinates to improve spatial accuracy.
The detected face bounding boxes are represented as horizontal and vertical
Gaussian-like vectors. Similarly, Wu _et al_. [30] adopted Gaussian-like
vectors as visual input features and a transformer-based network [31] for
their predictions. Zhao _et al_. [32] proposed a self-supervised student-
teacher knowledge distillation approach to train a multichannel audio network
from visual supervision. In a different study, they explored the audio-visual
speaker localization from egocentric views [33]. That is, the prediction is
performed from the point of view of the device “wearer”, who is free to move
in space and the sensors are therefore not stationary. Research groups from
Meta Reality Labs released the EasyCom dataset [34]. It consists of a set of
videos captured from custom AR glasses with a camera and a microphone array
integrated. Jiang et al. [35] employed EasyCom to perform audio-visual speaker
localization in an egocentric setting. Concurrently, Gurvich _et al_. [36]
performed the real-time ASDL with a microphone array in 360° videos.
Similarly, in this paper, we integrate multichannel audio and visual data.
However, instead of leveraging visual features such as Gaussian-like vectors,
we focus on the integration of audio and visual embeddings extracted with pre-
trained audio and visual encoders [37].

##  3 Method

The proposed method extracts audio and visual feature embeddings with an audio
and a visual encoder, respectively. The embeddings are then concatenated and
fed to an attention-based unit. The output of the attention-based unit
represents a joint latent audio-visual representation of the input signals,
which is used to generate the final prediction through a feed-forward network.
A schematic representation of the proposed model is presented in Figure 3.

###  3.1 Dataset

Figure 2: (a) Schematic of camera (blue circles) and microphone (red dots)
positions on the AVA Rig. The green square highlights the reference
microphone. (b) Photo of an AVA Rig.

To train and evaluate the proposed method, we employed the Tragic Talkers
dataset [6]. It offers sequences captured by two Audio-Visual Array (AVA)
Rigs. Each AVA Rig is a light-field and sound-field sensing platform
consisting of a 16-element microphone array and 11 cameras fixed on a flat
perspex baffle as depicted in Figure 2. Therefore, each sensor is located in a
fixed relative position and orientation with respect to the other sensors. The
microphone array has a horizontal aperture of 450 mm and a vertical aperture
of only 40 mm. This results in higher resolution when localizing audio sources
along the azimuth direction than elevation, which is consistent with human
perception [5]. Horizontally, the microphones are log-spaced for broad
frequency coverage from 500 Hz to 8 kHz, to better support the horizontal
speech band resolution. Tragic Talkers was captured in an acoustically treated
studio with an average reverberation time of 0.3s in the mid 0.5-2 kHz
frequency range and minimal background noise floor (SNR $\geq$ 30 dB). The
dataset does not contain sequences in which the speakers talk simultaneously,
off-screen talkers, or external sources of sound other than speech, making it
ideal for audio-visual speaker diarization applications too. Tragic Talkers
was specifically designed for OBM production research. Its content allows the
implementation of the speaker objectification pipeline presented in Figure 1
[3, 4]. In fact, it provides high-quality speech signals recorded with
Lavalier microphones. Additionally, the scenes are captured against a blue
background that facilitates the actors’ silhouette extraction, empowering the
producer to extract and position the actors as desired while retaining the
natural motion of the performance.

The method for ASDL proposed in this paper necessitates a single video feed
with the microphone array. Leveraging the multiple views available enables us
to extend the network training by choosing the relevant camera perspective.
Consequently, a one-hot vector denoting the selected view is appended to the
input data, augmenting the dataset with diverse camera perspectives and
enabling the network to learn the correct mapping to the desired viewpoint.
The sequences include one or two actors positioned at a distance of about 3–4
m, engaging in monologues, conversations, and interactive scenes involving
movement and occlusion. Tragic Talkers comprises 30 scenes captured with two
AVA Rigs. Each rig’s audio-visual stream is employed independently, i.e.,
16-channel audio is used to predict the speaker’s position within any of the
11 camera perspectives of the rig. So the dataset’s 30 scenes provide 60 rig
sequences, each offering 11 viewpoints. The dataset is partitioned into a
50-sequence development set and a 10-sequence test set. TragicTalkers provides
ground truth (GT) labels for voice activity and 2D face bounding box. The test
set used for evaluation also includes 3D mouth coordinates.

###  3.2 Network Architecture

We tested the proposed architecture in a recent study where we tackled the
sound event localization and detection (SELD) task [37]. Here, we aim to
explore whether a similar network can be extended to the more specific ASDL
tasks too. As depicted in Figure 3, the network presents an audio and a visual
encoder to extract audio and visual feature embeddings. The embeddings are
then concatenated and fed to an audio-visual Conformer [38]. Conformer models
were originally proposed for speech recognition but lately, they have achieved
state-of-the-art performance in tasks such as SELD too [39]. They present an
architecture similar to the Transformer proposed by Vaswani _et al_. [31],
however, they integrate a convolution module that performs pointwise and
depthwise convolutions. Inspired by Wang _et al_. [39], 4 layers are employed
with 8 heads each. The size of the kernel for the depthwise convolutions is
set to 51, and the dimension of the hidden layer in the feed-forward networks
of each Conformer layer is 1024 [39]. The output of the Conformer, i.e., the
latent audio-visual representation, is then fed to a feed-forward unit
consisting of three fully-connected layers to make the final predictions. A
one-hot vector encoding the desired camera view of the AVA Rig is concatenated
to the latent audio-visual vector after the second fully-connected layer. It
allows mapping the final prediction to the desired camera of the rig. In
output, the network predicts the speaker’s horizontal location and voice
activity confidence at a temporal resolution that matches the label and video
frame rate (i.e., a position-confidence pair is generated for each video
frame).

Figure 3: Proposed network architecture for audio-visual ASDL. An audio
encoder based on a CNN extracts an audio embedding from the audio input
features. Similarly, an encoder consisting of ResNet50 [40] followed by a
Conformer unit [38] extracts a visual embedding from the video frames.
$\otimes$ denotes the concatenation operation. After concatenation, the audio-
visual features and processed by a second Conformer unit. A feed-forward
network generates the final prediction. A camera ID one-hot vector is used to
regress the speaker’s position to the desired camera view.

####  3.2.1 Audio Encoder

The audio encoder takes as input spatial features extracted from the
multichannel audio signals (a detailed description of the audio input features
will be provided in 3.3) with shape $Ch_{in}\times T_{in}\times F_{in}$, where
$Ch_{in}$ corresponds to the number of channels of the input spatial features,
$T_{in}$ the number of temporal bins, and $F_{in}$ the frequency bins. The
audio encoder presents a CNN architecture. The CNN architecture consists of
four convolutional blocks, each consisting of two 3$\times$3 convolutional
layers followed by average pooling, batch normalization [41], and ReLU
activation. The average pooling layer is applied with a stride of 2, halving
the temporal and frequency dimension at each block. The resulting tensor of
shape $512\times T_{in}/16\times F_{in}/16$ is then reshaped and frequency
average pooling is applied to achieve a $T_{in}/16\times 512$ dimensional
feature embedding. $T_{in}$ is chosen so that $T_{in}/16$ matches the label
frame rate of the Tragic Talkers dataset (30 labels per second).

####  3.2.2 Visual Encoder

As a visual encoder, the ResNet50 model [40] followed by a Conformer [38] is
tested. Each video frame is fed to ResNet50. Since the video streams of Tragic
Talkers are captured at 30 fps, ResNet50 extracts a number of frame embeddings
that match the label frame rate as well as the audio embedding temporal
resolution. The frame embeddings extracted by ResNet50 are further processed
by the Conformer module, which presents the same hyper-parameters utilized for
the one employed for the audio-visual fusion. The video frames used as inputs
to the visual encoders are resized to 224x224p. We employed the ResNet50 model
available with the torchvision
library111https://pytorch.org/vision/stable/index.html, which is pre-trained
on image classification on the ImageNet dataset [42]. Before being fed to the
Conformer, the frame embeddings are resized from the original 2048-dimensional
vectors generated from ResNet50 to 512 dimensions employing a fully-connected
layer to match the size of the audio embeddings.

The remainder of this paper will often refer to the visual encoder as ResNet-
Conformer and to the attention-based audio-visual fusion as AV-Conformer.

###  3.3 Audio Input Feature

This work adopts log-mel spectrograms concatenated with generalized cross-
correlation with phase transform (GCC-PHAT) features in log-mel space [43, 44]
extracted from the microphone array signals. These audio features were chosen
because they achieved good performance and robustness on the large microphone
array of the Tragic Talker dataset [45].

The GCC-PHAT is employed to estimate the time difference of arrival (TDOA) of
a sound source at two microphones [43]. The idea is to find the lag time that
maximizes the cross-correlation function between the signals sensed by the two
microphones. The generalized cross-correlation (GCC) is computed through the
inverse Fast-Fourier Transform (inverse-FFT) of their cross-power spectrum.
Phase-transformed GCC, namely the GCC-PHAT, eliminates the influence of the
amplitude by leaving only the phase [44]. The GCC-PHAT between the i-th and
the j-th microphone is defined at each audio frame t as:

$GCC_{ij}(t,\tau)=\mathcal{F}^{-1}_{f\rightarrow\tau}\frac{\mathbf{X}_{i}(t,f)\mathbf{X}^{*}_{j}(t,f)}{|\mathbf{X}_{i}(t,f)\mathbf{X}^{*}_{j}(t,f)|},$
(1)

where $\mathbf{X}_{i}(t,f)$ is the Short-Time Fourier Transform (STFT) of the
i-th channel, $\mathcal{F}^{-1}_{f\rightarrow\tau}$ the inverse-FFT from the
frequency domain $f$ to the lag-time domain $\tau$, and $(.)^{*}$ denotes the
complex conjugate. The TDOA can be estimated as the lag-time $\Delta\tau$ that
maximizes $GCC_{ij}(t,\tau)$.

With time bins ($t$) on the $x$-axis and time-lags ($\tau$) on the $y$-axis,
the GCC-PHAT can be concatenated with the log-mel spectrograms extracted from
the channels of the microphone array, as indicated by Cao _et al_. [44]. The
concatenation of GCC-PHAT features and log-mel spectrograms provides the
network with a unified representation that enables both detection and
localization subtasks. We compute the GCC-PHAT between a reference microphone
and the other microphones of the array. As the reference microphone, we select
the first channel of the lower sub-array, as highlighted in Figure 2 (a). From
the same channel, we also extract a single log-mel spectrogram for the
concatenation.

###  3.4 Loss Function

For each input segment, the loss function $\mathcal{L}$ is determined as the
sum of the individual frame losses. A sum-squared error loss [46] is computed
at each output frame and is comprised of a regression and a voice activity
confidence loss:

$\mathcal{L}=\sum_{i=1}^{T_{in}/16}\mathbbm{1}_{i}(x_{i}-\hat{x}_{i})^{2}+(C_{i}-\hat{C}_{i})^{2}$
(2)

where $x_{i}$ and $\hat{x}_{i}$ are respectively the predicted and target
positions of the speaker along the horizontal axis of the $i$-th video frame,
while $C_{i}$ and $\hat{C}_{i}$ are the predicted and target confidences. The
voice activity confidence loss is trivially achieved using the voice activity
annotations: $\hat{C}_{i}$ is set to 1 when the frame is active and 0 when
silent. The masking term $\mathbbm{1}_{i}$ is 1 only when voice activity GT is
true. It is set to 0 otherwise. So, when the frame is silent, the network is
only penalized by the voice activity confidence loss and not by the regression
loss. The target position of the speaker, $\hat{x}_{i}$, corresponds to the
horizontal position of the center of the face bounding box of the active
speaker, normalized by the size of the video frame to be in the range [0, 1].

##  4 Experiments

###  4.1 Implementation Details Evaluation Metrics

The network is trained with a 5-fold cross-validation approach: each
validation fold sets aside 10 unseen sequences from the 50 sequences of the
development set. This cross-validation approach is used to find suitable
hyper-parameters for the network. Once found, the model is retrained using the
entire 50-sequence training set with these values. The network is trained for
50 epochs using batches of 32 audio feature inputs and Adam optimizer. The
learning rate is fixed for the first 30 epochs, then reduced by 10% each
epoch, as in [44]. The initial learning rate determined in the cross-
validation is ${10}^{-4}$.

The audio stream of the Tragic Talkers dataset is sampled at 48 kHz. The
dataset is discretized into audio-visual segments of 2 seconds. The label
frame rate is consistent with the video frame rate (30 fps). To align the
output temporal resolution ($T_{out}=T_{in}/16$) with the labels frame rate,
i.e., generating 60 activity-regression pair predictions for the 2-second
input, an STFT with Hann window is applied at hop steps of 100 samples. Thus,
the 2-second (96k-sample) audio chunk is discretized into 960 temporal bins
(96k$/$100), which correspond to $T_{in}$. The Hann window presents a size of
512 samples, as in [47]. To compute the log-mel spectrogram used in the
concatenation with the GCC-PHAT features, the frequency resolution of the
spectrogram is down-sampled over 64 mel-frequency bins and the logarithm
operation is applied. The number of time-lags for the GCC-PHAT is also set to
64 to enable the concatenation.

The evaluation is performed on the TragicTalkers test set, labeled for 2D
speaker mouth positions. A frame prediction is considered positive, i.e. the
network predicts the presence of speech, when the predicted voice activity
confidence is above a threshold $Th$, and a positive detection is true when
the localization error is within a predefined spatial tolerance $S$. The
precision and recall rates are computed by varying the confidence threshold
$Th$ from 0% to 100% sampling the thresholds from a Sigmoid-spaced
distribution to provide more data points for high and low confidence values.
The average precision (AP) was computed as the numerical integration of the
precision-recall curve, as indicated in [48]. We set a spatial tolerance $S$
of $\pm$2° along the azimuth according to human auditory perception [5], the
minimum audible angle (MAA), corresponding to $\pm$89 pixels on the image
plane. From the precision and recall rates, the F1 score is computed too. To
independently evaluate the localization and the speaker detection subtasks, we
define the average distance (aD) and the detection error (Det Err %) metrics.
The former represents the average distance error between the active detections
and the GT speaker locations. It is computed in pixels on the image frame and
then converted to angle units leveraging the camera calibration data. The
detection error corresponds to the percentage of frames incorrectly classified
as active or inactive when a threshold $Th=0.5$ is set.

###  4.2 Methods and Baselines

The proposed audio-visual method with multichannel audio is compared with two
traditional audio-visual approaches for active speaker detection that employ
single-channel audio and rely on face detection: Active Speaker in Context
(ASC) [12] and TalkNet [11]. We also report the results achieved in our
previous multichannel audio-only study [7]. The audio-only approach proposed
in [7] leverages a convolutional recurrent neural network (CRNN) with an
architecture similar to the audio encoder proposed in the present paper.
However, instead of the Conformer unit, the CRNN presents two bidirectional
gated recurrent units (biGRUs). Additionally, as a baseline system to
highlight the advantages introduced by multichannel audio, we report the
results achieved with a single-channel audio-only network (Mono). The input to
the Mono baseline is a log mel spectrogram extracted from the central
microphone of the array.

###  4.3 Results

Table 1: ASDL results on the test set of the Tragic Talkers dataset and modality potential. The results for the proposed approach are achieved with audio-visual inputs and multichannel audio (AV-M). The table includes the results achieved by a single-channel audio-only network (A-S), two audio-visual systems that employ single-channel audio (AV-S), and an audio-only method that leverages multichannel audio (A-M).  Method | Mod | DetErr | aD | AP | F1
---|---|---|---|---|---
Mono | A-S | 2.7% | 210p, 4.7° | 10% | 30.0
ASC [12] | AV-S | 43% | 50p, 1.1° | 59% | 67.6
TalkNet[11] | AV-S | 14% | 35p, 0.79° | 82% | 84.9
CRNN [7] | A-M | 3.2% | 39p, 0.88° | 87% | 90.9
Proposed | AV-M | 2.7% | 32p, 0.72° | 92% | 94.9

Figure 4: Comparison of precision versus recall curves for different ASDL
methods. The plot includes audio-visual methods with single-channel audio
(AV-S), i.e., ASC [12] and TalkNet [11], the multichannel audio-only CRNN
(M-S) [7], and the proposed audio-visual approach with multichannel audio
(AV-M). The combination of precision and recall rates that achieves the
highest F1 score is marked on each curve.

The experimental results are presented in Table 1. The audio-visual
multichannel (AV-M) method significantly improves the performance of the
audio-only systems across nearly all metrics.

In the Mono baseline, the detection subtask is accomplished with a detection
error of only 2.7%. However, the position of the speaker is not accurately
predicted due to the absence of spatial cues. To minimize the error, the model
locates the speaker in the central area of the frame.

Conventional active speaker detectors, such as ASC [12] and TalkNet [11],
employ the audio modality only to classify the pre-extracted faces. The
localisation subtask is performed by the visual face detector, yielding high
spatial accuracy. However, since the average distance is achieved as the
average of the predicted positions, the final average includes true and false
positive predictions (false positive predictions happen when the detector
classifies the silent actor as active). False positive detections are mainly
present in the detections of ASC [12], penalizing its spatial accuracy as well
as its detection. In the audio-visual methods with single-channel audio, the
horizontal coordinate of the center of the bounding box is used as prediction,
while the ground truth used for evaluation refers to the actual mouth position
of the speaker. Therefore, the aD achieved is slightly overestimated due to
the offset between the two representations. For example, when the speaker is
captured in profile and his/her mouth is closer to the edge of the bounding
box. Additionally, the AV-S methods fail when the face of the active speaker
is not detected by the face detector. This causes a higher detection error
and, consequently, a lower recall rate, as shown in Figure 4. At its best
precision-recall pair TalkNet presents a recall rate of 79.7%, while ASC only
of 58.5%. As a consequence, their overall F1 scores are affected too. In
contrast, the multichannel audio method achieves a lower detection error as
speech activity can be sensed even when the speaker is visually occluded. In
fact, the double-digit detection errors of the AV-S methods are reduced to
3.2% with the A-M approach. Figure 4 shows how the gap in recall rate
generated by TalkNet is halved with the multichannel audio method (90.6%
recall rate). This produces an AP and F1 score higher than the AV-S systems.

When multichannel audio is partnered with vision, beneficial effects involve
both detection and localization accuracy. The detection error decreases by 0.5
percentage points to 2.7%, while the aD outperforms even the audio-visual
TalkNet model. The F1 score is 4 percentage points higher than the A-M
approach and 10 points greater than TalkNet.

The residual error in the F1 score for the proposed AV-M method is only 5%. In
the future, this error might be further narrowed by implementing visually
guided predictions post-processing [49]. For example, pose detection could
rectify spatial predictions using the mouth key-point coordinates. This would
further improve the localization accuracy and consequently increase the number
of true positive detections that fall within the spatial tolerance. Another
aspect to consider is the consistency between training and testing labels. The
GT labels employed to train the model correspond to center of the face
bounding boxes, whereas the evaluation is based on GT mouth positions. This
disparity introduces a subtle domain bias between the training and inference
phases, potentially resulting in residual errors regardless of the quality of
the model.

##  5 Conclusions

This paper proposes an audio-visual approach for active speaker detection and
localization that leverages multichannel audio on the Tragic Talkers dataset.
The approach extracts audio and visual embedding leveraging audio and visual
encoders. Then, the embeddings are concatenated and processed by an AV-
Conformer. The proposed method outperforms conventional audio-visual
approaches for active speaker detection that rely on visual face detection as
well as our previous audio-only multichannel work. This highlights the
importance of the input modalities. Active speaker detection and localization
can be employed for the automatic extraction of speaker positional metadata
useful in immersive audio productions.

##  6 Acknowledgments

This research was funded by EPSRC-BBC Prosperity Partnership ‘AI4ME: Future
personalised object-based media experiences delivered at scale anywhere’
(EP/V038087/1). For the purpose of open access, the authors have applied a
Creative Commons Attribution (CC BY) license to any Author Accepted Manuscript
version arising. Data supporting this study are available from
https://cvssp.org/data/TragicTalkers.

## References

  * [1] Philip Coleman, Andreas Franck, Jon Francombe, Qingju Liu, Teofilo de Campos, Richard J. Hughes, Dylan Menzies, Marcos F. Simón Gálvez, Yan Tang, James Woodcock, Philip J. B. Jackson, Frank Melchior, Chris Pike, Filippo Maria Fazi, Trevor J. Cox, and Adrian Hilton,  “An audio-visual system for object-based audio: From recording to listening,”  IEEE Transactions on Multimedia, vol. 20, no. 8, pp. 1919–1931, 2018.
  * [2] Chris Pike, Richard Taylor, Tom Parnell, and Frank Melchior,  “Object-based 3d audio production for virtual reality using the audio definition model,”  Journal of the Audio Engineering Society, september 2016.
  * [3] Mohd Azri Mohd Izhar, Marco Volino, Adrian Hilton, and Philip J. B. Jackson,  “Tracking sound sources for object-based spatial audio in 3D audio-visual production,”  in Forum Acusticum, 2020, pp. 2051–2058.
  * [4] Florian Schweiger, Chris Pike, Tom Nixon, Matt Firth, Bruce Weir, Paul Golds, Marco Volino, Craig Cieciura, Mohd Izhar, Nick Graham-Rack, Philip J. B. Jackson, and Alex Ang,  “Tools for 6-DoF immersive audio-visual content capture and production,”  in International Broadcasting Convention, 2022.
  * [5] Thomas Strybel and Ken Fujimoto,  “Minimum audible angles in the horizontal and vertical planes: Effects of stimulus onset asynchrony and burst duration,”  Journal of the Acoustical Society of America, vol. 108 6, pp. 3092–5, 2000.
  * [6] Davide Berghi, Marco Volino, and Philip J. B. Jackson,  “Tragic Talkers: A Shakespearean sound- and light-field dataset for audio-visual machine learning research,”  in European Conference on Visual Media Production, 2022.
  * [7] Davide Berghi and Philip J. B. Jackson,  “Leveraging visual supervision for array-based active speaker detection and localization,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 984–995, 2024.
  * [8] Davide Berghi, Adrian Hilton, and Philip J. B. Jackson,  “Visually supervised speaker detection and localization via microphone array,”  in IEEE International Workshop on Multimedia Signal Processing, 2021.
  * [9] Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia Schmid, Zhonghua Xi, and Caroline Pantofaru,  “AVA active speaker: An audio-visual dataset for active speaker detection,”  in IEEE International Conference on Acoustics, Speech and Signal Processing, 2020, pp. 4492–4496.
  * [10] Yuanhang Zhang, Susan Liang, Shuang Yang, Xiao Liu, Zhongqin Wu, Shiguang Shan, and Xilin Chen,  “UniCon: Unified context network for robust active speaker detection,”  in ACM Multimedia, 2021.
  * [11] Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, and Haizhou Li,  “Is someone speaking? Exploring long-term temporal features for audio-visual active speaker detection,”  in ACM International Conference on Multimedia, 2021, p. 3927–3935.
  * [12] Juan León Alcázar, Fabian Caba, Long Mai, Federico Perazzi, Joon-Young Lee, Pablo Arbeláez, and Bernard Ghanem,  “Active speakers in context,”  in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12462–12471.
  * [13] Punarjay Chakravarty, Jeroen Zegers, Tinne Tuytelaars, and Hugo Van hamme,  “Active speaker detection with audio-visual co-training,”  in International Conference on Multimodal Interaction, 2016, p. 312–316.
  * [14] Ross Cutler and Larry Davis,  “Look who’s talking: Speaker detection using video and audio correlation,”  in IEEE International Conference on Multimedia and Expo, 2000, pp. 1589–1592.
  * [15] Fasih Haider and Samer Al Moubayed,  “Towards speaker detection using lips movements for human-machine multiparty dialogue,”  in Fonetik, 2012.
  * [16] Punarjay Chakravarty, Sayeh Mirzaei, Tinne Tuytelaars, and Hugo Van hamme,  “Who’s speaking? Audio-supervised classification of active speakers in video,”  in International Conference on Multimodal Interaction, 2015, p. 87–90.
  * [17] Ken Hoover, Sourish Chaudhuri, Caroline Pantofaru, Ian Sturdy, and Malcolm Slaney,  “Using audio-visual information to understand speaker activity: Tracking active speakers on and off screen,”  in IEEE International Confonference on Acoustics, Speech and Signal Processing, 2018, pp. 6558–6562.
  * [18] Arsha Nagrani, Joon Son Chung, Weidi Xie, and Andrew Zisserman,  “VoxCeleb: Large-scale speaker verification in the wild,”  Computer Speech & Language, vol. 60, pp. 101027, 2020.
  * [19] Joon Son Chung and Andrew Zisserman,  “Lip reading in the wild,”  in Asian Conference on Computer Vision, 2016.
  * [20] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman,  “ASR is all you need: Cross-modal distillation for lip reading,”  in IEEE International Conference on Acoustics, Speech and Signal Processing, 2020, pp. 2143–2147.
  * [21] Soo-Whan Chung, Joon Son Chung, and Hong-Goo Kang,  “Perfect match: Improved cross-modal embeddings for audio-visual synchronisation,”  in IEEE International Conference on Acoustics, Speech and Signal Processing, 2018, pp. 3965–3969.
  * [22] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman,  “The conversation: Deep audio-visual speech enhancement,”  in INTERSPEECH, 2018.
  * [23] Joon Son Chung,  “Naver at ActivityNet Challenge 2019 - Task B Active speaker detection (AVA),”  ArXiv, vol. abs/1906.10555, 2019.
  * [24] Yuan-Hang Zhang, Jing-Yun Xiao, Shuang Yang, and Shiguang Shan,  “Multi-task learning for audio-visual active speaker detection,”  in Techical Report AVA-ActiveSpeaker Challenge, 2019.
  * [25] Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman,  “LRS3-TED: a large-scale dataset for visual speech recognition,”  ArXiv, vol. abs/1809.00496, 2018.
  * [26] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman,  “VoxCeleb2: Deep speaker recognition,”  in INTERSPEECH, 2018.
  * [27] Juan León Alcázar, Moritz Cordes, Chen Zhao, and Bernard Ghanem,  “End-to-end active speaker detection,”  in European Conference on Computer Vision, 2022, p. 126–143.
  * [28] Xinyuan Qian, Maulik Madhavi, Zexu Pan, Jiadong Wang, and Haizhou Li,  “Multi-target DoA estimation with an audio-visual fusion mechanism,”  in IEEE International Conference on Acoustics, Speech and Signal Processing, 2021, pp. 4280–4284.
  * [29] Xinyuan Qian, Zhengdong Wang, Jiadong Wang, Guohui Guan, and Haizhou Li,  “Audio-visual cross-attention network for robotic speaker tracking,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 550–562, 2023.
  * [30] Yulin Wu, Ruimin Hu, Xiaochen Wang, and Shanfa Ke,  “Multi-speaker DoA estimation using audio and visual modality,”  Neural Processing Letters, vol. 55, no. 7, pp. 8887–8901, 2023.
  * [31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in International Conference on Neural Information Processing Systems, 2017, p. 6000–6010.
  * [32] Jinzheng Zhao, Peipei Wu, Shidrokh Goudarzi, Xubo Liu, Jianyuan Sun, Yong Xu, and Wenwu Wang,  “Visually assisted self-supervised audio speaker localization and tracking,”  in European Signal Processing Conference, 2022, pp. 787–791.
  * [33] Jinzheng Zhao, Yong Xu, Xinyuan Qian, and Wenwu Wang,  “Audio visual speaker localization from egocentric views,”  ArXiv, vol. abs/2309.16308, 2023.
  * [34] Jacob Donley, Vladimir Tourbabin, Jung-Suk Lee, Mark Broyles, Hao Jiang, Jie Shen, Maja Pantic, Vamsi Krishna Ithapu, and Ravish Mehra,  “Easycom: An augmented reality dataset to support algorithms for easy communication in noisy environments,”  arXiv, vol. 2107.04174, 2021.
  * [35] Hao Jiang, Calvin Murdock, and Vamsi Krishna Ithapu,  “Egocentric deep multi-channel audio-visual active speaker localization,”  IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10534–10542, 2022.
  * [36] Ilya Gurvich, Ido Leichter, Dharmendar Reddy Palle, Yossi Asher, Alon Vinnikov, Igor Abramovski, Vishak Gopal, Ross Cutler, and Eyal Krupka,  “A real-time active speaker detection system integrating an audio-visual signal with a spatial querying mechanism,”  ArXiv, vol. abs/2309.08295, 2023.
  * [37] Davide Berghi, Peipei Wu, Jinzheng Zhao, Wenwu Wang, and Philip J. B. Jackson,  “Fusion of audio and visual embeddings for sound event localization and detection,”  in IEEE International Conference on Acoustics, Speech and Signal Processing, 2024.
  * [38] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang,  “Conformer: Convolution-augmented transformer for speech recognition,”  ArXiv, vol. abs/2005.08100, 2020.
  * [39] Qing Wang, Jun Du, Hua-Xin Wu, Jia Pan, Feng Ma, and Chin-Hui Lee,  “A four-stage data augmentation approach to resnet-conformer based acoustic modeling for sound event localization and detection,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1251–1264, 2023.
  * [40] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
  * [41] Sergey Ioffe and Christian Szegedy,  “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”  in International Conference on Machine Learning, 2015, vol. 37, pp. 448–456.
  * [42] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li,  “ImageNet: A large-scale hierarchical image database,”  IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009.
  * [43] Charles Knapp and G. Clifford Carter,  “The generalized correlation method for estimation of time delay,”  IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 24, no. 4, pp. 320–327, 1976.
  * [44] Yin Cao, Qiuqiang Kong, Turab Iqbal, Fengyan An, Wenwu Wang, and Mark D. Plumbley,  “Polyphonic sound event detection and localization using a two-stage strategy,”  in Detection and Classification of Acoustic Scenes and Events Workshop, 2019.
  * [45] Davide Berghi and Philip J. B. Jackson,  “Audio inputs for active speaker detection and localization via microphone array,”  in IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2023.
  * [46] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi,  “You only look once: Unified, real-time object detection,”  in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016, pp. 779–788.
  * [47] Thi Ngoc Tho Nguyen, Karn N. Watcharasupat, Ngoc Khanh Nguyen, Douglas L. Jones, and Woon-Seng Gan,  “SALSA: Spatial cue-augmented log-spectrogram features for polyphonic sound event localization and detection,”  IEEE/ACM Trans. on Audio, Speech, and Language Processing, vol. 30, pp. 1749–1762, 2022.
  * [48] Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman,  “The PASCAL visual object classes challenge: A retrospective,”  International Journal of Computer Vision, vol. 111, no. 1, pp. 98–136, 2015.
  * [49] Qing Wang, Ya Jiang, Shi Cheng, Maocheng Hu, Zhaoxu Nian, Pengfei Hu, Zeyan Liu, Yuxuan Dong, Mingqi Cai, Jun Du, and Chin-Hui Lee,  “The NERC-SLIP system for sound event localization and detection of DCASE2023 challenge,”  in Techical Report of DCASE Challenge, 2023.