# Calibration of Wideband LFM Radars based on Sliding Window Algorithm

Hyung-Woo Kim, Jin-woo Kim, Jin-ha Kim, JaeYoung Choi, Sangpyo Hong and
Byungkwan Kim

###### Abstract

This paper addresses the challenges of wideband signal beamforming in radar
systems and proposes a new calibration method. Due to operating conditions,
the frequency dependent characteristics of the system can be changed, and
amplitude, phase, and time delay error can be generated. The proposed method
is based on the concept of sliding window algorithm for linear frequency
modulated (LFM) signals. To calibrate the frequency-dependent errors from
transceiver and the time delay error from true time delay elements, the
proposed method utilizes the characteristic of the LFM signal. The LFM signal
changes its frequency linearly with time, and the frequency domain
characteristics of the hardware are presented in time. Therefore, by applying
matched filter to a part of the LFM signal, the frequency dependent
characteristics can be monitored and calibrated. The proposed method is
compared with the conventional matched filter based calibration results and
verified by simulation results and beampatterns. Since the proposed method
utilizes LFM signal as calibration tone, the proposed method can be applied to
any beamforming systems, not limited to LFM radars.

###### Index Terms:

Phased Array Radar; Radar Calibration; Wideband Signal; True Time Delay;
Beamforming

Corresponding Author: Byungkwan Kim<EMAIL_ADDRESS>
##  I Introduction

Radar performs various operations like surveillance, imaging, target detection
tracking in remote location. To perform these operations with high resolution
and accuracy, radar system requires wideband signal and large number of
transmit and receive modules (TRM) for beamforming[1]. The beamforming can be
performed in time domain by true-time delay element (TTD) or in frequency
domain by phase shifter. The phase shifter based beamforming, frequency-domain
beamforming, is widely used in phased array radar system because of its lower
cost than TTD elements[2]. However, the frequency-domain beamforming has a
limitation on the radar waveform, in terms of bandwidth and pulse duration, as
follows.

$BW\times\tau_{sig}\leq 1$ (1)

If this limitation is not satisfied, the phase shifter based beamforming forms
the beam in the undesired direction at some frequencies and the effective beam
width is broadened. Therefore, the time domain beamforming is required for the
wideband beamforming radar systems.[longbrake2011true] The required time delay
for the beamforming is proportional to the distance between the phase center
of antenna elements. For example, an S-band radar system with center frequency
of 3.25GHz, the minimum delay between antenna is equivalent to 130ps.
Therefore, precise control of time delay between TRMs are required and true
time delay elements are adopted for those requirements.

Recent phased array radar systems are designed to operate for multiple
functions with various waveforms. Based on the target or operation mode, the
radar system selects its waveform including bandwidth or additional binary
phase modulation [3]. This means that the radar system should be able to
control its phase using the phase shifter, even it performs beamforming in
time domain with TTD elements. On the other hand, the radar system can apply
both time delay and phase shift for the beamforming by considering total cost
and performance of the system [4].

The TRMs are composed of various components such as power amplifier, phase
shifter, and many passive elements that have frequency-dependent
characteristics[5]. In narrowband assumption, the frequency-dependent
characteristics are negligible and the TRMs are assumed to have constant
characteristics in the frequency domain[6]. However, for wideband signal, the
characteristics of the TRM are not constant in the frequency domain and the
characteristics of the TRMs are different from each other.

Therefore, beamforming system requires precise control of the transmitted or
received signal to have a desired differences in time, amplitude, phase in all
frequencies. To achieve this, many calibration methods have been proposed and
implemented[7, 8, 9, 10, 11, 12]. The PN gating method is widely used
technique for the calibration of the remote sensing radar system, which is
based on the orthogonal code[13, 14, 15]. The calibration method can monitor
characteristics of each TRMs by applying orthogonality of the waveform during
the calibration process. This method is simple and easy to implement, and can
measure realistic characteristics of the TRMs since it monitors the
characteristics of the TRMs when all TRMs are operating simultaneously.
However, the PN gating method is still targeted on the narrowband system and
the calibration is performed with single value of amplitude, phase, and
delay[16].

In this paper, we propose a monitoring and calibration method for wideband
phased array radar system by application of sliding window algorithm to the
chirp-pulsed signal. The proposed method can monitor the characteristics of
the TRMs in the frequency domain, by amplitude and phase of the TRMs, and in
the time domain, by time delay of the TRMs. This method does not alter any
hardware of the radar system and can be applied to the existing radar system
by proposed processing method. The proposed method can be applied to the radar
system with both TTD and phase shifter based beamforming system.

The rest of this paper is organized as follows. In Section 2, the principles
of the calibration of wideband phased array radar system with linear frequency
modulation is described. In Section 3, the proposed error monitoring and
calibration method is presented. In Section 4, the simulation setup,
parameter, and results are presented. Finally, the conclusion is given in
Section 5.

##  II Calibration of Beamforming System with Linear Frequency Modulation

The linear frequency modulation (LFM), also known as pulse compression
technique, is a widely used waveform for the radar system to achieve high
range resolution. The LFM waveform is defined as follows.

$x(t)=e^{j2\pi(f_{0}t+kt^{2})}$ (2)

where $f_{0}$ is the center frequency of the LFM waveform and $k$ is the chirp
rate of the LFM waveform.

The calibration of the beamforming system with LFM waveform is performed by
the matched filter output of the received signal, to monitor the
characteristics of the transmitter, receiver, cables, passive and active
components. The matched filter output represents relative delay, amplitude
attenuation, and phase shift of the received signal compared to the reference
signal[16]. This information is utilized to detect the target and to estimate
the target location, or to generate the remote images of the target. By using
same principle, we utilize the matched filter output of the received signal to
monitor the characteristics of the TRMs.

Since the beamforming is implemented by the relative difference in time and
frequency domain of the signal, the calibration is performed by setting a
reference TRM and comparing the characteristics of the other TRMs to the
reference TRM. The signal delay can be measured by the index differences of
the matched filter. The amplitude attenuation and phase shift differences can
be measured from the maximum value of the matched filter output, which means
the calibration signal’s amplitude and phase. Further details of the
calibration with LFM signal can be found in[17].

The conventional calibration of the beamforming system with LFM waveform is
performed by the single value of amplitude, phase, and delay. This means that
if the TRM and any component has frequency dependent characteristics, the
calibration result cannot be accurate. To overcome this problem, we propose a
new calibration method for wideband phased array radar system by adopting
sliding window algorithm.

##  III Proposed Method

###  III-A Frequency Dependent Error Model

As described in the previous section, the conventional calibration can monitor
and generate precise beampattern of the phased array radar system for
narrowband signal. However, if wideband signals are applied, the performance
of the beamforming system can be different in frequency for various
conditions. The frequency-dependent performance variation of TRMs, phase
shifter, TTDs can be found in literatures [18].

Before describing the proposed method, we define our error model for the
beamforming system. Since the radar system considering in this study is based
on the LFM waveform, the frequency error model can be defined also in time
domain. The frequency-dependent amplitude and phase can be represented in time
valued function, since the LFM waveform changes its frequency linearly in
time.

$E_{N}(t)=A_{N}(t)*exp(j2\pi*\phi_{N}(t))$ (3)

where $A_{N}(t)$ and $\phi_{N}(t)$ are the amplitude and phase error of the
$N$th TRM of beamforming system in frequency domain, respectively. The time
variable t can be converted to frequency variable $f$ by the linear
relationship of LFM signal. On the other hand, the relative delay error for
any signal $s(t)$ can be represented as follows.

$s_{delayed}(t)=s(t-\tau_{N})$ (4)

where $\tau_{N}$ is the relative delay error of the $N$th TRM of beamforming
system. Therefore, the frequency dependent error model of the beamforming
system can be expressed as follows.

$x_{N}(t)=A_{N}(t-\tau_{N})*exp(j2\pi*\phi_{N}(t-\tau_{N}))*x(t-\tau_{N})$ (5)

where $x_{N}(t)$ is received signal including time delay, frequency dependent
amplitude and phase errors.

In the simulation, the delay can be applied by shifting index of the signal,
therefore, the precise delay error requires oversampling and decimation
process of the signal.

###  III-B Calibration with Sliding Window Algorithm

The proposed method is based on the sliding window concept for matched
filtering, which is used to analyze the time-frequency characteristics of the
signal. The sliding window algorithm performs matched filter of the received
LFM signal with a part of the calibration signal. The main concept of the
sliding window algorithm is the matched filtering of the LFM signals with
different length, by dividing the LFM signal to represent a narrowband
characteristic of the system. The Fig. 1 shows the concept of the proposed
method with sliding window algorithm. The narrowband segmentation of the LFM
signal is processed by the matched filter with same part of the reference
signal. By sliding the matched filter window in time domain, the
characteristic of the LFM signal can be monitored in the frequency domain
because the LFM signal is linearly changing its frequency.

Figure 1: The conceptual diagram of the proposed method.

The total process of the proposed method is shown in Fig. 2. When the
calibration process is started, the wideband LFM signal which includes the
characteristic of each TRM is gathered. The signal is processed with matched
filter by the reference signal. A reference TRM is selected, and the coarse
delay is estimated by the index difference of the maximum of matched filter
output. The coarse delay is compensated by shifting the index of the total
signal. After coarse delay compensation, the proposed method is started. The
sliding window algorithm is performed to monitor the frequency dependent
characteristics of the TRMs. The amplitude and phase error for a certain
frequency can be measured by the maximum value of the matched filter output
from the LFM segmentation by a window. Based on the measured error values, the
proposed method constructs a error matrix in frequency domain. The calibration
of the TRMs can be performed by applying the error matrix to the received
signal.

The calibration of the wideband LFM signal is performed by the same principle
of the error model described in the previous section. First, the time delay
calibration is implemented by the shifting the total signal index by the
measured delay error. However, this calibration is not enough to compensate
the precise time delay of the TRMs, since the time delay error is represented
in the units of sample index of the signal. For example, by supposing 1GHz
bandwidth of the LFM signal with 10GHz center frequency, the sampling rate of
the signal processing unit should be larger than 2GHz[4]. Assuming that the
sampling rate of analog-digital converter is 2GHz, then the sampling time of a
sample is 0.5ns, which means that the time delay error can be represented in
the units of 0.5ns. However, the beamforming radar will control the time delay
in units of few picoseconds to generate high resolution beampattern, which
means that the time delay error in the units of picoseconds should also be
calibrated. Therefore, the index based delay calibration is named as coarse
delay calibration, and precise time delay calibration is separately performed
in this study. The coarse delay calibration can be enhanced by increasing the
sampling rate of the analog-digital converter, but it is not practical in the
real system since the beamforming system should process large amount of
digital signal in real time.

Figure 2: The procedure of proposed calibration method. The highlighted block
is the proposed method, while normal blocks are conventional methods.

After coarse delay calibration, the signal still contains frequency dependent
amplitude and phase errors, and precise time delay error. The frequency
dependent amplitude and phase errors can be calibrated with the same method of
the error model described in the previous section. The phase error in this
stage is consist of two parts, which are true frequency dependent phase error
of the TRM and the linear phase term due to precise time delay. The Fig. 3
shows an example for the phase error consist of two parts.

The calibration of precise time delay can be done with two different
approaches. First method is the calibration of true-time delay element with
the precise delay error by applying linear regression of measured phase error.
This is effective only when the frequency-dependent phase error is very
smaller than the additional linear phase due to time delay error.
($\phi_{N}<<\tau_{N}*f(t)$)

If the beamforming radar system works fully in digital domain, the precise
time delay can be considered as another phase error source that needs
calibration. In this situation, the second method can be introduced, the
predistortion of the LFM signal with phase error, for the precise time delay
and phase error of TRMs. This can be implemented without additional
computational cost, since the calculation is equivalent with proposed
amplitude and phase calibration.

Figure 3: An example of frequency dependent phase error. Due to the physical
nature of the microwave circuits, frequency dependent phase errors are
generated. The linear phase error is from the time delay error.

The proposed method can calibrate this phase error and precise time delay
simultaneously since both errors are represented in the measured phase error.
Therefore, frequency dependent amplitude, phase error, and precise time delay
of the $N$th TRM can be calibrated by the following equation.

$x_{calibrated}(t)=x_{N}(t)*\frac{1}{A_{N}(t)}*exp(-j2\pi*\phi_{N}(t))$ (6)

##  IV Simulation & Results

###  IV-A Simulation Parameters

To verify the proposed method, the simulation is performed by considering
realistic simulation parameters and beamforming system. The parameters for
error models are summarized in the following Table 1. The wideband beamforming
radar system which is based on both phase shifter and true time delay is
considered in the simulation, as presented in [4]. The beamforming system
consists of 64 TRMs, and the LFM signal is generated with 0.5GHz bandwidth,
with 8us pulse length and 3.25GHz center frequency.

TABLE I: The waveform and beamforming system parameters. Parameters | Value | Unit
---|---|---
Central Frequency | 3.25 | GHz
Bandwidth | 0.5 | GHz
Pulsewidth | 8 | us
No. of TRMs | 64 |
No. of TTDs | 8 |

###  IV-B Error model

The error model of the beamforming system is generated by referencing measured
error values of amplitude, phase, and time delay error from literatures. The
frequency dependent amplitude and phase error is modeled with interpolation
from random values from the reported typical error range[19]. The example of
amplitude and phase error versus frequency for a TRM is shown in Fig. 4. Each
frequency dependent error model are generated and assigned to each TRMs by a
frequency-amplitude matrix and frequency-phase matrix, as presented in the
Fig. 11.

TABLE II: The amplitude, phase, and time error values for the simulation. The error values are selected to simulate the error levels similar with single bit range of phase shifter, true-time delay element.  Parameters | Value | Unit
---|---|---
Amplitude Error | -1.1 $\sim$ 0 | dB
Phase Error | -7 $\sim$ 7 | degree
Coarse Time Delay Error Unit | 100 | ps
Precise Time Delay Error Unit | 20 | ps

The time delay error is also referred typical error range of true time delay
elements for the beamforming system[19]. In digital domain, the time delay can
be assigned to a signal by shifting the index of the signal. Therefore, the
oversampling and decimation process is introduced to represent the time delay
error lower than the sample time. In this simulation, we assigned time delay
with 20ps step by 10GHz oversampling, and all simulations are performed after
decimating the signal with 5. Therefore, the coarse delay error can be
represented with 0.1ns step, and the precise time delay error is represented
with 20ps step in the simulation.

By applying these error model to reference LFM signal, the error signal with
frequency-dependent and time delay can be generated.

###  IV-C Scenario and Results

####  IV-C1 Error Measurement

In this simulation, we assign errors to selected TRMs to verify the error
measurement performance of proposed method. To measure the various types of
error, we assigned following different error model to the TRMs. Conventional
and Proposed calibration method are performed to the error signal, and the
error values are compared with the assigned error values.

Three different types of error are prepared for the simulation to verify the
proposed method and compare with the conventional method. The first type of
error is the frequency constant amplitude and phase error, which is the most
common error model for the narrowband beamforming system. The second type of
error is the frequency dependent amplitude and phase errors from the
electromagnetic components, and the third type of error is the frequency
dependent amplitude and phase error with time delay error from the true time
delay element.

For clear comparison, the error values are assigned to the TRMs as follows.
The type 1 error model is assigned to the TRM 10, and the type 2 error model
is assigned to the TRM 20, and the type 3 error model is assigned to the TRM
30.

  * •

Frequency constant amplitude and phase error

Figure 4: The amplitude and phase of frequency constant error applied signal.

The amplitude error is assigned as -3dB, and the phase error is assigned as 5
degree. The error signal is generated by multiplying the error model to the
reference signal, and the amplitude and phase of the error signal is shown in
Fig. 4.

Figure 5: The assigned amplitude and phase error are presented with the
results from the conventional method and the proposed method.

After calibration with the conventional and proposed method, the result error
values are compared with the assigned error values for the frequency of LFM
signal, as shown in Fig. 5. Since the conventional method presents the error
values with the same value for all frequency, the error values are presented
as a line. Both method presents the error values with the same value for the
amplitude and phase, which means that the conventional method can calibrate
the LFM signal when the error is constant for all frequency. In this case, the
proposed method can also calibrate the error, but the computational complexity
is higher than the conventional method since it generates same values for all
frequency.

  * •

Frequency dependent amplitude and phase error

Figure 6: The amplitude and phase of frequency dependent amplitude and phase
error applied signal.

The frequency depedent error is assigned as the function of frequency, which
is generated by interpolating the random values from the reported typical
error range. An example of the error applied signal is shown in Fig. 6.

Figure 7: The assigned amplitude and phase error are presented with
conventional and proposed calibration results.

The calibration result of error signal presented in Fig. 6 is shown in Fig. 7.
In the same manner, the error values from the conventional calibration are
presented as a line because it does not consider frequency dependent error. On
the other hand, the proposed method presents the error values for each
frequency bin. The calculated calibration accuracy of amplitude error of
0.016dB, and the phase error of 3.523degree. The proposed method can calibrate
the frequency dependent error, but the selection of window size and sliding
step size is an important factor to the calibration accuracy.

Figure 8: The proposed calibration result with different window size and
overlapping ratio. The smaller window size and larger overlapping ratio shows
higher calibration accuracy.

To show the effect of window size and sliding step size, the calibration
results are shown in Fig. 8. The window size is set to 500 and 4000, which
slices 50ns and 400ns length of the LFM signal. The sliding step is determined
by the overlapping ratio, which is set to 85% and 30%, 75 and 2800 samples for
500ns and 4000ns window size, respectively. Both parameters determines the
number of error values versus frequency, which is directly related to the
calibration accuracy. This is due to the nature of the LFM signal, which has
the linear frequency increasement in time. The smaller window size will
represent the narrowband characteristics, and the larger overlapping ratio
will contribute the changing characteristics of the error values in frequency
domain. Therefore, the larger window size and smaller overlapping ratio shows
lower calibration accuracy as shown in Fig. 8.

  * •

Frequency dependent amplitude and phase error with time delay error

As decribed in the proposed calibration process, the coarse delay error is
calibrated first, and the precise delay error is calibrated after the coarse
delay error is calibrated. Therefore, the phase error from the precise delay
error is included in the error signal as presented in Fig. 3, and it needs to
be calibrated with the proposed method. In this simulation, the total delay
error is set to 100 ps and calibrated coarsely, and the precise delay error is
set to 20 ps. The error signal with the precise delay error is shown in Fig.
9. The Fig. 9 shows that the phase difference of the error signal is
increasing with time, which is from the precise delay error.

Figure 9: The amplitude and phase of frequency dependent amplitude, phase
error and time delay error applied signal example. We applied 100ns delay for
better presentation of time delayed signal. Due to the time delay error, the
phase difference of the error signal is increasing with time.

The calibration result of error signal with precise time delay is presented in
Fig. 10.

Figure 10: The assigned amplitude and phase error are presented with
conventional and proposed calibration results.

From the calibration result, the precise time delay does not affect the
calibration accuracy of amplitude error, but it affects the calibration
accuracy of phase error. As presented in Fig. 10, the applied phase error is
added with a linear function of frequency. By applying simple linear
regression model to the phase error, the slope of the phase error is
calculated as -3 degree/GHz, equivalent to 16 ps of precise delay. The
difference of the precise delay error applied and obtained is 2 ps, which is
below the requirement of true time delay element resolution.

In summary, the proposed calibration method is compared with the applied error
model and the conventional calibration method for three different error types,
presented in Fig. 11.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 11: The TRM number 10, 20, and 30 are assigned with the constant
amplitude and phase error, frequency dependent amplitude and phase error, and
frequency dependent amplitude and phase error with precise time delay error,
respectively. The conventional and proposed calibration method are performed
to the find out error from reference LFM signal.

Therefore, the proposed calibration method can calibrate the frequency
dependent amplitude and phase error with precise time delay error. In next
section, the calibration result of the conventional and proposed method is
utilized to generate the beamforming result.

####  IV-C2 Calibration Effects on Beamforming

In this scenario, all TRMs are assigned with randomly generated errors, and
the conventional and proposed calibration method are performed. The Fig. 12
shows the error matrix from the conventional and proposed calibration method
for all TRMs. The assigned error values are randomly generated as described in
Table. II.

(a)

(b)

(c)

(d)

Figure 12: The calibration result with randomly generated errors for all TRMs.
The error values are randomly generated as described in Table. II.

Supposing that the TRMs are performing beamforming in the direction of -35
degree in azimuth plane, the expected beamforming result is presented in the
Fig. 13a. Based on the measured error value from conventional method and
proposed method, the beamforming result is presented in Fig. 13b, and 13c.

(a)

(b)

(c)

Figure 13: (a) The desired beamforming result with ideal steering vector. (b)
The beamforming result with steering vector calibrated with the conventional
method. Due to frequency dependent amplitude and phase error described in Fig.
12, the beamforming result is distorted for all LFM frequencies. (c) The
beamforming result with steering vector calibrated with the proposed method.
Compared to the conventional method, the proposed method can calibrate the
frequency dependent amplitude and phase error with time delay error.

From the beamforming result, the proposed method can calibrate the frequency
dependent amplitude and phase error with time delay error, and the beamforming
result is not distorted for all LFM frequencies. However, the conventional
method cannot calibrate the frequency dependent amplitude and phase error, and
the beamforming result is distorted for all LFM frequencies. The main
contribution for the beam distortion is the frequency dependent phase error
which is from the characteristic of TRM and precise delay error.

###  IV-D Discussions

The proposed method calibrates both precise time delay error and phase error
after coarse time delay calibration. The impact of the precise time delay
error is more significant than the phase error when the sampling rate is low.
This means that if the sampling rate can be increased, the precise time delay
error will be reduced because the time delay error can be calibrated at the
coarse time delay calibration step. The higher sampling rate will make coarse
time delay calibration more accurate. However, the increased sampling rate
will also increase the number of samples to be processed, which will increase
the computational complexity of the calibration process.

The proposed method generates the error matrix by applying matched filter with
sliding window to the error signal. The computational complexity may be
problem when the number of TRMs is large. In this case, the sub-band concept
can be applied to reduce the computational complexity while maintaining the
frequency dependent error monitoring capability, as reported in [17].

The proposed method is compared with the sub-band approach, as shown in Fig.
14. The number of sub-band is selected as 20, for 25MHz bandwidth. The figure
shows that the sub-band matched filtering follows tendency of the frequency
dependent error, however, the values are not accurate as the proposed method.
In case that the frequency dependent errors are not changing dramatically,
sub-band approach can be applied to reduce the computational complexity.
However, at frequency band where the frequency dependent errors are changing
rapidly, the sub-band approach cannot be applied because the errors are not
accurately measured. By lowering overlapping ratio of sliding window
algorithm, the proposed method can be converted to sub-band approach. Based on
system parameters and signal processing capability, the window size and
frequency resolution can be adjusted.

Figure 14: Calibration result comparison between proposed method and sub-band
calibration concept. The sub-band matched filtering follows tendency of the
frequency dependent error, however, the values are not accurate as the
proposed method.

##  V Conclusion

The multifunctional and high performance radar system requires the wideband
signal to utilize the high range resolution. The beamforming of wideband
signal requires the precise time delay calibration of TTD elements and the
frequency dependent amplitude and phase error from the TRM. As known in
literatures, the wideband beamforming requires precise time delay control for
each TRM.

This paper presents a new calibration method for the wideband LFM beamforming
radar system. The proposed method can calibrate the frequency dependent
amplitude and phase error with precise time delay error without any additional
hardware. By applying the stretched processing to the calibration signal, the
proposed method can measure and calibrate the LFM signal with frequency
dependent amplitude and phase error with precise time delay error. The error
measurement accuracy and its impact on the beamforming result are analyzed.
The proposed method is compared with true error values and conventional
calibration results, and it verified its calibration performance by the
simulation results. The accuracy of the proposed method is determined by
following factors: the sampling rate, and the window size and sliding step
size of the stretched processing.

The proposed method can be applied to with and without the orthogonal code
based calibration method. This is because the main process of the proposed
method that enables frequency dependent monitoring is the sliding window
algorithm, which can be applied to any LFM based radar transceivers. Also,
other types of transceivers can adopt the proposed method by generating the
LFM signal for calibration purpose. The proposed method can be extended to the
optimization of the parameter selection of the stretched processing for
reducing the computational complexity.

## Acknowledgments

This work is supported by HANWHA Systems.

## References

  * [1] M. I. Skolnik, “Introduction to radar systems,” _New York_ , 1980.
  * [2] K. Sreenivasulu, U. Pandey, P. Kumar, S. Pal, and K. Ray, “Design consideration of wide bandwidth active phased array antenna,” in _2021 6th International Conference for Convergence in Technology (I2CT)_. IEEE, 2021, pp. 1–4.
  * [3] S. Wang and P. He, “Research on low intercepting radar waveform based on lfm and barker code composite modulation,” in _2018 International Conference on Sensor Networks and Signal Processing (SNSP)_. IEEE, 2018, pp. 297–301.
  * [4] P. Guccione, D. Mapelli, D. Giudici, and A. R. Persico, “Design of f-scan acquisition mode for synthetic aperture radar,” _Remote Sensing_ , vol. 14, no. 20, p. 5283, 2022.
  * [5] M. Maffei, A. Aubry, A. De Maio, and A. Farina, “Spaceborne radar sensor architecture for debris detection and tracking,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 59, no. 8, pp. 6621–6636, 2020.
  * [6] R. Rau and J. H. Mcclellan, “Analytic models and postprocessing techniques for uwb sar,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 36, no. 4, pp. 1058–1074, 2000.
  * [7] M. Fujita, T. Masuda, Y. Fujino, and M. Satake, “Polarimetric calibration of the sir-c c-band channel using active radar calibrators and polarization selective dihedrals,” _IEEE transactions on geoscience and remote sensing_ , vol. 36, no. 6, pp. 1872–1878, 1998.
  * [8] C. Buck, J. Suchail, R. Torres, and M. Zink, “Asar instrument calibration,” in _Proceedings of ERS-ENVISAT Symposium_ , 2000, pp. 16–20.
  * [9] D. Hounam, M. Schwerdt, and M. Zink, “Active antenna module characterisation by pseudo-noise gating,” 2002.
  * [10] J. P. M. Boncori and G. Schiavon, “Signal processing issues for the exploitation of pulse-to-pulse encoding sar transponders,” _IEEE transactions on geoscience and remote sensing_ , vol. 46, no. 4, pp. 1048–1057, 2008.
  * [11] M. Schwerdt, B. Brautigam, M. Bachmann, B. Doring, D. Schrank, and J. H. Gonzalez, “Final terrasar-x calibration results based on novel efficient methods,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 48, no. 2, pp. 677–689, 2009.
  * [12] S. Kim, J. Sung, and A. Torre, “In-orbit antenna pattern extraction method for active phased-array sar antennas,” _IEEE Antennas and Wireless Propagation Letters_ , vol. 15, pp. 317–320, 2015.
  * [13] M. Schwerdt, B. Bräutigam, M. Bachmann, and B. Döring, “Terrasar-x calibration results,” 07 2008, pp. 1 – 4.
  * [14] D. Vollbracht, “System specification for dual polarized low power x-band weather radars using phased array technology,” in _2014 International Radar Conference_. IEEE, 2014, pp. 1–6.
  * [15] B. Brautigam, M. Schwerdt, and M. Bachmann, “An efficient method for performance monitoring of active phased array antennas,” _IEEE transactions on geoscience and remote sensing_ , vol. 47, no. 4, pp. 1236–1243, 2009.
  * [16] S. Wang, H. Qi, and W. Yu, “Polarimetric sar internal calibration scheme based on t/r module orthogonal phase coding,” _IEEE transactions on geoscience and remote sensing_ , vol. 47, no. 12, pp. 3969–3980, 2009.
  * [17] Y. Deng, H. Zheng, R. Wang, J. Feng, and Y. Liu, “Internal calibration for stepped-frequency chirp sar imaging,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 8, no. 6, pp. 1105–1109, 2011.
  * [18] V. Kumar, D. S. Rao, P. Kumar, K. Sreenivasulsu, K. Beenamole, R. K. Gangwar _et al._ , “Demonstration of x-band wideband scanning using hybrid beam steering components,” in _2022 IEEE International Symposium on Phased Array Systems & Technology (PAST)_. IEEE, 2022, pp. 1–5.
  * [19] J.-C. Jeong, I.-B. Yom, J.-D. Kim, W.-Y. Lee, and C.-H. Lee, “A 6–18-ghz gaas multifunction chip with 8-bit true time delay and 7-bit amplitude control,” _IEEE transactions on microwave theory and techniques_ , vol. 66, no. 5, pp. 2220–2230, 2018.


# Spoken language change detection inspired by speaker change detection

Jagabandhu Mishra<EMAIL_ADDRESS>S. R. Mahadeva Prasanna
<EMAIL_ADDRESS>Department of Electrical Engineering, IIT Dharwad,
Dharwad 580011, Karnataka, India.

###### Abstract

Spoken language change detection (LCD) refers to identifying the language
transitions in a code-switched utterance. Similarly, identifying the speaker
transitions in a multispeaker utterance is known as speaker change detection
(SCD). Since tasks-wise both are similar, the architecture/framework developed
for the SCD task may be suitable for the LCD task. Hence, the aim of the
present work is to develop LCD systems inspired by SCD. Initially, both LCD
and SCD are performed by humans. The study suggests humans require (a) a
larger duration around the change point and (b) language-specific prior
exposure, for performing LCD as compared to SCD. The larger duration
requirement is incorporated by increasing the analysis window length of the
unsupervised distance-based approach. This leads to a relative performance
improvement of $29.1\%$ and $2.4\%$, and a priori language knowledge provides
a relative improvement of $31.63\%$ and $14.27\%$ on the synthetic and
practical codeswitched datasets, respectively. The performance difference
between the practical and synthetic datasets is mostly due to differences in
the distribution of the monolingual segment duration.

††preprint: Mishra $et~{}al.$, LCD inspired from SCD

##  I  Introduction

Spoken language diarization (LD) is a task to automatically segment and label
the monolingual segments in a given multilingual speech signal. The existing
works towards LD are very few Sitaram _et al._ (2019). The majority of them
use phonotactic (i.e. the distribution of sound units) based approaches Chan
_et al._ (2004); Lyu _et al._ (2013); Spoorthy _et al._ (2018). The
development of LD using a phonotactic-based approach requires transcribed
speech utterances. The same is difficult to obtain as most of the languages
present in the code-switched multilingual utterances are resource-scare in
nature Sitaram _et al._ (2019); Spoorthy _et al._ (2018). Even though, there
exist some transfer learning approaches that adapt the phonotactic models of
the high resource language to obtain the models for the low resource language,
may end up with performance degradation if both the languages are not from the
same language group Sitaram _et al._ (2019). Further, LD is effortless for
humans, especially for known languages, and challenging for machines. Hence
there is a need for exploring alternative approaches for LD.

Speaker diarization (SD) is a task to automatically segment and label the
mono-speaker segments for a given multispeaker utterance, which is well
explored in the literature. Though there exist differences in the information
that needs to be captured to perform LD and SD tasks, there exist many
similarities like the features approximating the vocal tract resonances that
have been successfully used for the modeling of both speaker and language-
specific phonemes Carrasquillo _et al._ (2002); Li _et al._ (2013); Liu _et
al._ (2021). Furthermore, most of the approaches used for spoken language
identification (LID) are inspired by the approaches used for the speaker
identification/verification (SID/SV) task Snyder _et al._ (2018); Richardson
_et al._ (2015). In addition to that most of the successful LID systems that
are borrowed from SID/SV literature do not require transcribed speech data Li
_et al._ (2013); Snyder _et al._ (2018). Alternatively LID systems developed
using the phonotactic approach require transcribed speech data. This motivates
a close association study between the LD and SD tasks and may be exploited to
come up with approaches for LD.

The SD field has evolved mainly in two ways: (1) change point detection
followed by clustering and boundary refinement, and (2) fixed duration
segmentation followed by i-vector/ embedding vector extraction, clustering,
and boundary refinement Park _et al._ (2022); Moattar and Homayounpour (2012);
Tranter and Reynolds (2006). Dawalatabad _et al._ (2020); Bredin _et al._
(2017); Hogg _et al._ (2019); Park _et al._ (2022) reported that initial
change point detection improved overall SD performance. Thus this study
focuses on the development of spoken language change detection (LCD) through a
comparative analysis between LCD and speaker change detection (SCD). The
available SCD approaches can be broadly classified into two groups: (1)
distance-based unsupervised approach and (2) model-based supervised approach
Moattar and Homayounpour (2012); Park _et al._ (2022). The distance-based
approach applies hypothesis testing (either coming from a unique speaker or
not) for predicting the speaker change to the speaker’s specific features
extracted from the speech signal with sliding consecutive windows Moattar and
Homayounpour (2012); Park _et al._ (2022). Following this approach, many
feature extraction techniques like excitation source Dhananjaya and
Yegnanarayana (2008); Sarma _et al._ (2015), fundamental frequency contour
Hogg _et al._ (2019), etc., and distance metrics like Kullback–Leibler (KL)
divergence Siegler _et al._ (1997), Bayesian information criteria (BIC) Chen
_et al._ (1998), KL2 Siegler _et al._ (1997), generalized likelihood ratio
(GLR) Gish _et al._ (1991) and information bottleneck (IB) Dawalatabad _et
al._ (2020) are proposed in the literature. Generally, the performance of the
distance-based unsupervised approach degrades with variation in environment
and background noise (it may predict false changes), hence to resolve the
issue supervised model-based approaches are proposed in the literature Moattar
and Homayounpour (2012); Park _et al._ (2022). In the early days, the proposed
approaches model individual speakers using the Gaussian mixture model and
universal background model (GMM-UBM) Barras _et al._ (2006); Moattar and
Homayounpour (2012), hidden Markov model (HMM) Meignier _et al._ (2006), etc,
but nowadays, using the deep learning framework the approach predicts the
speaker change by discriminating between the speaker change segments
(neighborhood of the speaker change point) with no change segments Moattar and
Homayounpour (2012); Park _et al._ (2022). However, the model-based approach
smooths the output evidence and may lead to miss detection of the change
points Moattar and Homayounpour (2012). In addition to that training of the
supervised model requires labeled speech data from a similar
environment/recording condition, speaking style, language, etc., making the
system development complicated. Therefore the distance-based unsupervised
approaches are more popular and widely used for SCD tasks Dawalatabad _et al._
(2020); Park _et al._ (2022); Moattar and Homayounpour (2012).

Even though the available SCD frameworks look simple to adopt, there are
challenges in doing so. Fig. 1 (a) and (b), show the time domain speech
signals corresponding to the utterance having a speaker change and a language
change, respectively. By listening and observing the time domain
representation of both utterances, the identified speaker/language change
points are manually marked. From the time domain signal, it is very difficult
to locate both the speaker and language change points. Fig. 1 (c) and (d) show
the spectrogram of both utterances. From the spectrogram, it can be observed
that around the speaker change the formant structure shows significant
variation, whereas around language change the structure is intact. When the
speaker changes, the vocal tract system information changes and hence the
variation in the formant structure. However, the structure of the formant
frequencies remains intact during language change as the single speaker is
speaking both languages. It is interesting to note that humans discriminate
between spoken languages without knowing the detailed lexical rules and
phonemic distribution of the respective languages. Of course, humans need to
have prior exposure to the languages Li _et al._ (2013). Humans may exploit
the long-term phoneme dynamics to discriminate between languages. Therefore,
the language change may be detected by capturing the long-term language-
specific spectral-temporal dynamics. This may represent valid phoneme
sequences and their combinations to form syllables and subwords of a language.

Figure 1: (a) and (c) Two speaker time domain speech signal and its
spectrogram, respectively. (b) and (d) Two languages (Bilingual) time domain
speech signal and its spectrogram, respectively.

Based on the need to exploit the long-term spectro-temporal evidence, it can
be hypothesized that the LCD by human/machine may require more neighborhood
duration around the change point than the SCD. In addition, LCD may also
benefit from prior exposure to respective languages. A human subjective study
that focuses on language/speaker change detection is set up for validating the
same.

For automatic detection of language change, the initial studies are performed
using the available unsupervised distance and the supervised model-based SCD
approaches. The model-based approaches include GMM-UBM, i-vector, and
x-vector. Based on the experimental results for LCD and SCD, appropriate
modifications will be done to each framework for improving the performance of
the LCD task.

The main contribution of this work are summarized as follows: (a) by observing
the spectro-temporal representation around the speaker and language change, it
is hypothesized that detecting language change, requires a larger duration
around the change point and a priori knowledge of the language as compared to
detecting a speaker change. The same hypothesis is confirmed by the human
subjective study, (b) the SCD frameworks are used as initial baselines to
perform LCD and their performances are analyzed, and (c) these frameworks are
further refined to improve the performance of LCD.

##  II Database setup

This section provides a brief description of the database used in this study.
For performing the LCD/SCD task among humans, we have selected $32$ and $15$
utterances for the language and speaker change study, respectively. All the
utterances have only one language/speaker change point and have approximately
$6-8$ syllables on either side of the change point. For the language change
study, we have selected $32$ utterances from the publicly available sources
(mostly from Youtube), whereas we have chosen $15$ utterances from the IITG-MV
phase 3 and DIHARD datasets for the speaker change study Haris _et al._
(2012); Ryant _et al._ (2018). The $32$ utterances used for the language
change study are from the $10$ language pairs and have $4,4,4,4,4,2,2,4,2,2$
utterances, respectively from, (1) Hindi-English (HIE), (2) Bengali-English
(BEE), (3) Telugu-English (TEE), (4) Tamil-English (TAE), (5) Bengali-Assamese
(BEA), (6) Bengali-Bengali (BEB), (7) Assamese-Assamese (ASA), (8) Tamil-
Malayalam (TAM), (9) Tamil-Tamil (TAT) and, (10) Malayalam-Malayalam (MAM),
respectively. It is difficult to get the utterances having language pairs,
Bengali-Assamese, and Tamil-Malayalam spoken by a single speaker. Hence, these
language pairs with and without having a language change are considered along
with a speaker change. The selected utterances for both LCD and SCD tasks
along with their change point annotations are available at
https://github.com/jagabandhumishra/HUMAN-SUBJECTIVE-STUDY-FOR-LCD-and-SCD.

Initially, the studies have been performed with synthetically generated code-
switch and multi-speaker utterances. For generating the utterances, we have
used the Indian institute of technology Madras text-to-speech (IITM-TTS)
corpus Baby _et al._ (2016). The IITM-TTS corpus consists of speech data
recordings from native speakers of $13$ Indian languages. For each native
language, two speakers (a male and a female) recorded their utterances in
their native language and English. In this study for synthesizing the code-
switch utterances, a female speaker speaking her native language Hindi, and
her second language English is considered. For each language, the first $5$
hours of data are used for training purposes. The rest of the monolingual
utterances are stitched randomly for generating code-switched utterances.
Altogether, $4000$ utterances are generated having one to five language change
points. The average monolingual segment duration of the generated code-switch
utterances for Hindi and English languages are approximately $6.5$ and $5.2$
secs, respectively. The generated dataset is termed as TTS female language
change (TTSF-LC) corpus. Similarly, for generating speaker change utterances
by keeping the language identical, we have used English speech utterances from
native Hindi and Assamese female speakers. The average mono-speaker segment
duration of the generated utterances are $5.19$ and $4.86$ secs, respectively.
The generated dataset is termed as TTS female speaker change corpus (TTSF-SC).

Finally, for generalizing the obtained observations, the experiments are
performed on the standard LCD corpus. Microsoft code-switched challenge task-B
(MSCSTB) dataset is used. The dataset has development and training partitions
that consist of code-switched utterances and language tags (each $200$ msec)
from three language pairs: Gujarati-English (GUE), Tamil-English (TAE), and
Telugu-English (TEE). The approximate duration of each language in the
training and development set is $16$ and $2$ hours, respectively. The detail
about the database can be found at Diwan _et al._ (2021).

##  III Human subjective study for language and speaker change detection

An experimental procedure has been set up, where each human subject is exposed
to a pool of utterances that may or may not have a language/speaker change.
The human subjects are asked to mark, if there exists a language/speaker
change or not. The utterances are classified into five groups. Each group is
represented with approximate duration considered in terms of the number of
voiced frames (NVF) taken around the true/false change point. The true change
point refers to the actual change points of the selected utterances. The
selected utterances are split around the change point to generate the mono-
language/speaker utterance. The false change point represents the centered
voiced frame’s start location of the given mono-language/speaker utterance.
The voiced frame is decided by taking $6\%$ of the average short time frame
energy (computed with a frame size of $20$ msec and a frameshift of $10$ msec)
of a given utterance as a threshold Rabiner (1978). The $30$ mono-speaker
utterances are generated by splitting the selected $15$ utterances around the
true change point. Out of $30$, with respect to duration, the largest $15$ has
been chosen for this study. The same procedure has been followed to generate
the mono-lingual utterances using the selected code-switched utterances
belonging to the HIE, BEE, TAE, and TEE language pairs. However, there is an
exception for the utterances belonging to BEA and TAM, as the utterances have
a speaker change along with the language change. Hence for a fair comparison,
the mono-lingual utterances for these cases are synthesized, such that they
also have a speaker change, i.e. BEB, ASA, MAM, and TAT, respectively. After
that, each utterance $S(n)$ is masked by considering $x$ number of voiced
frames (NVF-$x$) from the left and right of the true/false change point.
According to the value of $x$, the masked utterances are grouped into five
different groups, termed NVF-$10$, NVF-$20$, NVF-$30$, NVF-$50$, and NVF-$75$.
To avoid abrupt masking, a Gaussian mask $G(n)$ with appropriate parameters is
multiplied with the utterances to obtain the masked utterance
$S_{m}(n)=S(n)\times G(n)$. The masked signal is passed through an energy-
based endpoint detection algorithm to obtain the final masked utterance
Rabiner (1978). The detailed procedure of the masked utterance generation is
attached in the supplementary111Supplementary material for the data generation
procedure for the LCD and SCD study among humans is available at [AIP will
insert URL], and also the generated utterances are available at
https://github.com/jagabandhumishra/HUMAN-SUBJECTIVE-STUDY-FOR-LCD-and-SCD.

The listening experiment is conducted with $18$ subjects. Out of them, $13$
number of the subjects are male and $5$ are female. The selected subjects are
from the $20-30$ years age group. The subjects have no prior exposure to the
voice samples of the speakers used in this study. However, the subjects are
comfortable with English, and for other languages, the comfortability varies.
To know the language comfortability, each of the subjects is asked to provide
a language comfortability score (LCS) from zero to three for each pair of
languages.

The listening study is conducted with $390$ utterances (i.e $240$ for LCD and
$150$ for SCD). The LCD task is separate from SCD, hence conducted in two
different sessions, and also the subjects are well rested so that they don’t
have listener fatigue. A graphical user interface (GUI) has been designed to
perform the listening study. For a specific LCD/SCD study, all the masked
utterances are presented to the listener in a random order, irrespective of
their segment duration. If a listener is unable to provide the response for
one-time playing, s/he is allowed to play the utterance multiple times. Our
objective here is to observe, how correctly humans recognize the speaker and
language change by listening to the utterances coming from the five different
groups. Hence, the responses recorded in Sharma _et al._ (2019) for analyzing
the talker change detection ability of humans are used here. Three kinds of
responses have been recorded, these are (1) language/speaker change detected
or not (2) the number of times replayed (NR), and (3) response time (RT). RT
is the time duration taken by a subject to provide his/her response, after
listening to the full utterance. The RT is computed by subtracting the
respective utterance duration (UD) from the total duration (TD) (i.e. $RT=TD-
UD$). The TD is the duration taken by a subject (i.e. from pressing the play
button to pressing the yes/no button) to provide his/her response.

$DER=\frac{(FA+FR)}{N}$ (1)

For a given subject, there are three kinds of performance measures computed in
this study: (1) average detection error rate ($DER$) (2) average number of
times replayed ($NR$), and (3) average response time ($RT$). The $DER$ is
defined in Eq. 1, where $N$ is the total number of trials, $FA$ is the number
of false language/speaker change utterances, marked as true by the subject and
$FR$ is the number of true language/speaker change utterances, marked as false
by the subject, respectively. The $DER$ measure defines the inability of the
subject to detect language/speaker change. The $NR$ provides an estimation of
the average number of replays required for the subject to mark their response
comfortably. Similarly, the $RT$ provides an estimation of the average
duration required for the subject to perceive the language/speaker change,
after listening to the respective utterances. A higher value of the
performance measures indicates the inability of the human subject to perceive
the language/speaker change and vice versa.

Figure 2: (a) $DER$ distributions of the subjects, (b) F-Statistics (F Stat)
values of the ANOVA test between the $DER$ distributions of LCD (L) and SCD
(S) study, respectively . Figure 3:  Median values of the (a) $NR$ and (b)
$RT$ distributions for the LCD and SCD.

After performing both the LCD and SCD experiments, the subject-specific,
$DER$, $NR$, and $RT$ are computed with respect to NVF. The distributions of
the obtained $DER$ with respect to the NVF are depicted in Fig. 2(a). It can
be seen that the $DER$ values are smaller for the SCD than for the LCD,
regardless of the NVF. This suggests that human subjects are more comfortable
with detecting the switching of speakers than language. Furthermore, as the
NVF increases from $10$ to $75$, the DER decreases for both SCD and LCD. The
differences between the DER distribution of the LCD and SCD decrease with an
increase in the NVF. This suggests that human subjects’ comfortability in
detecting language change increases and becomes at par with speaker change,
with the increase in the NVF. To further validate fact, a statistical test
called an analysis of variance (ANOVA) has been performed between the $DER$
distribution (after removing the outliers) of LCD and SCD. The obtained
F-statistics values are depicted in Fig. 2(b). The higher F-statistics value
suggests having better discrimination between the two distributions and vice-
versa. From the figure, it can be observed the F-statistics values reduced
with an increase in NVF. This justifies the claim that humans’ language
discrimination ability improves and goes closure to the speaker discrimination
ability with an increase in NVF. The median values of the recorded $NR$ and
$RT$ values are depicted in Fig. 3. It can be observed from the figure that,
like $DER$, the median value of $NR$, and $RT$ reduces with an increase in
NVF. The median values of $NR$ and $RT$ are also smaller for SCD than LCD.
This concludes that human subjects require a larger duration around the change
point to detect language than the speaker change comfortably.

For observing the effect of language comfortability on detecting language
change, the responses of the human subjects are considered for the group
NVF-$50$ and NVF-$75$ that have the median of $DER$ lesser than $0.25$
(assuming sufficient duration from either side). With respect to the LCS, the
responses are segregated into four groups. The group segregation with respect
to language comfortability is done as $0$: very low, $1$: lower medium, $2$:
medium, and $3$: excellent, respectively. The obtained $DER$ distribution with
respect to LCS is depicted in Fig. 4. From the figure, it can be observed that
the $DER$ values are decreases with an increase in LCS. This concludes that a
priori knowledge of languages helps people to better discriminate between
languages.

Figure 4: DER vs. language comfortability score (LCS) for LCD with NVF-50 and
NVF-75.

##  IV LCD and SCD using unsupervised distance-based approach

The objective of this section is to perform LCD tasks inspired by the existing
unsupervised distance-based SCD framework. In general, the SCD task is
performed by computing and threshold the distance contour obtained between the
features of the sliding analysis window with a fixed length $N$. The basic
block diagram of the approach is depicted in Fig. 5. First feature vectors are
extracted from the speech signal and then energy-based voice activity
detection (VAD) is performed to obtain the voiced frame indices. The voiced
frame indices are stored for future reference and the feature vectors
corresponding to the voiced frames are used for further processing. The voiced
feature vectors are used with two consecutive windows having a fixed length to
model two different Gaussian distributions ($g_{a}$ and $g_{b}$). The
divergence distance contour is obtained through the entire scan of the given
test utterance by sliding the analysis window with a frame, as mentioned in
Eq. 2. The evidence contour is then smoothed with the hamming window with
length ($h_{l}$). The smoothed contour is then used for peak detection, with a
peak-picking algorithm having a minimum peak distance parameter called
$\gamma$. The higher value of $\gamma$ reduces the number of detected peaks
and vice-versa. For reducing the number of false change points, an approach of
deriving a threshold counter proposed in Lu and Zhang (2002) and mentioned in
Eq 3 is used here. Finally, the change frame is obtained by comparing the
strength of the detected peaks with the threshold contour. The change point’s
actual frame index and sample location are obtained by using the stored voiced
frame locations.

Figure 5: Basic block diagram of the change detection framework for
unsupervised distance-based approach

$D(i)=KL(g_{A}|g_{B})+KL(g_{B}|g_{A}),\\\ $ (2)

$Th(i)=\alpha.\frac{1}{N}\sum_{n=0}^{N}D(i-n-1,i-n)$ (3)

Initially, we used the TTSF-SC dataset for designing and tuning the
hyperparameters of the SCD system. Out of $4000$ test utterances, the first
$100$ utterances are used to tune the hyperparameters. It has been observed
that the performance is optimal by considering $\alpha=1$, $\gamma$ equal to
$0.9$ times the analysis window length, and $150$ as the analysis window
length. Keeping the methodology and hyperparameters identical, the TTSF-LC and
MSCSTB dataset is used to perform the LCD task. For evaluating the
performance, the commonly used performance measures for event detection tasks,
i.e. identification rate (IDR), false acceptance rate (FAR), miss rate (MR),
and mean deviation ($D_{m}$) are used here Murty and Yegnanarayana (2008);
Mishra _et al._ (2021). The performances of both tasks are tabulated in Table
1.

From the results, it can be observed that the performance of the SCD in terms
of IDR is $84.1\%$, whereas the performance of the LCD in terms of IDR is
$51.2\%$. The reduction in performance may be due to two reasons, (1) the used
MFCC features may fail to capture language-specific discriminative evidence,
and (2) the hyperparameters, mostly the analysis window length, are tuned for
SCD and may not be appropriate for LCD. Hence to understand the issue a study
is carried out by varying the features and analysis window length around the
change point. The most used features in literature for language identification
(LID) tasks, i.e. MFCC, LPCC, SDC, and PLP are considered here. The objective
here is to observe the language discriminative ability of the features by
considering a fixed number of voiced frames (NVF), $x$ around the change point
and compare it with the speaker discrimination ability of the MFCC feature.
This study will help us to reason out the performance degradation of LCD as
compared to SCD. Further, the observation will also help us to optimally
decide the feature and analysis window length for performing LCD.

Figure 6: Distance computation around the true and false change point of an
utterance, (a) true change point and (b), (c) false change points, fl_d, and
tr_d are false and true distances, respectively.

For performing the study, the TTSF-SC and TTSF-LC dataset is considered. Out
of $4000$ test utterances, the utterances having only one change point are
selected. The number of utterances selected for speaker change and language
change is $799$ and $836$, respectively. For observing the discrimination
ability, the idea here is to observe the distributional difference between the
true and false distances. The true distances are the KL divergence distance
between the $x$ number feature vectors from either side of the ground truth
change point. Similarly, the false distance is computed by placing the change
point randomly anywhere in the mono-language/ speaker segments. The procedure
of computing the true and false distances is also depicted in Fig. 6. For
observing the duration effect on the discrimination, the value $x$ is
considered as $10$, $20$, $30$, $50$, $75$, $100$, $150$, $200$, $250$, and
$300$, respectively. For a given $x$ value, the ANOVA test is conducted
between the obtained true and false distances. The obtained F-statistics
values of the ANOVA test are depicted in Fig. 7.

Figure 7: ANOVA test F-statistics (F Stat) values obtained between the true and false KL divergence distances for speaker/language change study with varying the number of voiced frames (NVF). Figure 8: True (T) and false (F) distance distribution (D-dist) (a) with MFCC feature for speaker change, (b) with MFCC feature for language change, (c) with PLP feature for language change, and (d) shows the corresponding F-statistics (F Stat) values. Table 1: Perforance of LCD and SCD with the unsupervised distance-based approach. A: with $N=150$ (tuned for SCD) and B: with the optimal $N$ value (tuned for LCD). | TTSF-SC | TTSF-LC | MSCSTB
---|---|---|---
GUE | TAE | TEE
MFCC | MFCC | PLP | MFCC
A | A | B | B | A | B | A | B | A | B
IDR | 84.1 | 51.2 | 66.1 | 64.06 | 42.9 | 44.07 | 47.4 | 48.75 | 44.5 | 45.24
FAR | 10.7 | 41.3 | 20.91 | 22.58 | 8.1 | 9.24 | 8.2 | 8.57 | 7.7 | 7.70
MDR | 5.2 | 7.5 | 12.98 | 13.36 | 49 | 46.69 | 44.4 | 42.68 | 47.8 | 47.06
Dm | 0.19 | 0.45 | 0.51 | 0.57 | 0.5 | 0.49 | 0.5 | 0.56 | 0.5 | 0.56

From the figure, it can be observed that the F-statistics values increase with
an increase in NVF and saturate after a certain number of voiced frames, and
started decreasing after that. A similar observation has also been observed in
the case of the LCD and SCD study by humans. However, in case humans’
performance doesn’t degrade with an increase in NVF. This may be due to the
inability of the Gaussian (assumption of statistical independence) to model
the speaker and language spectral dynamics and leading to the increase of the
class-specific variance in the distance distribution. Using the MFCC feature,
the F-statistic values of the SCD are higher than the LCD irrespective of the
NVF. Further, it can also be observed that the discrimination ability (in
terms of F-statistics) of the LCD follows the SCD with an increase in the NVF.
Furthermore, it has also been observed that the highest F-statistics values
obtained for speaker and language change study are at $150$ and $200$,
respectively.

In addition to this, for language change study, the MFCC features provide
better F-statistics value, followed by PLP, LPCC, and SDC. For clear
observation, the distance distribution of the MFCC feature to perform SCD and
the MFCC and PLP features to perform LCD with NVF of $50$, $150$, $200$, and
$250$ is depicted through box plots in Fig. 8. From the box plots, it can also
be noticed that the speaker and language discrimination saturates at NVF $150$
and $200$, respectively. Though the boxplots look to have better
discrimination, the increase in inter-class variance leads to a decrease of
the F-statistics values. Furthermore, the discrimination ability of the MFCC
is better compared to PLP, as the separation between the true and false
distance distribution of the MFCC feature is higher than the PLP feature for
LCD at NVF equal to 200. This motivates us to consider the MFCC feature with
the analysis window length of $200$ for performing LCD for the TTSF-LC
dataset. The performance of the LCD task with modified analysis window length
is tabulated in Table 1.

The table shows that the performance in terms of IDR, FAR, and MDR follows the
observations noticed with respect to the F-statistics. The performance
obtained for the TTSF-LC dataset with MFCC feature (considering analysis
window length $200$) is $66.1\%$ in terms of IDR, providing a relative
improvement of $29.1\%$ and followed by the IDR of $64.06\%$ using PLP
feature. Similar observations also have been reported using the MSCSTB
dataset, where the performance in terms of IDR improved relatively with
$2.72\%$, $2.85\%$, and $1.63\%$ by considering the analysis window length of
$160$, $180$, and $170$ for GUE, TAE, and TEE language pairs, respectively.
The analysis window length $160$, $180$, and $170$ are decided greedily by
evaluating the performance by considering the analysis window length from
$100$ to $250$ with a shift of $10$ on the first $100$ test trails. Hence,
this justifies the hypothesis that the requirement of relatively higher
duration information to perform LCD than SCD.

##  V Language Change Detection by Model-based Approach

The SCD and LCD by human suggest that prior exposure to the language make
human more efficient in detecting language change. This motivates extracting
the statistical/embedding vectors from the trained machine learning (ML)/ Deep
learning (DL) framework and using them to perform change detection tasks. The
detailed procedure is explained in the following subsections.

###  V.1 Model-based change detection framework

The block diagram of the model-based change detection framework is depicted in
Fig. 9. From the training data, initially, MFCC$+\Delta+\Delta\Delta$ are
computed, and voiced feature vectors are selected for further processing by
using VAD. The voiced feature vectors are used to train the statistical models
like the universal background model (UBM), adaptation model, Total variability
matrix (T matrix), and DL model like TDNN-based x-vector models. The
statistical vectors like u/a/i-vectors are extracted using trained UBM/adapt
model/ T-matrix, respectively. The u-vector and a-vectors are computed by
computing the zeroth order statistics from the UBM and adapt model,
respectively. The zeroth order statistics are computed using Equation 4, where
$i$ ranges from $1\leq i\leq M$, $M$ is the number of mixture components,
$x_{j}$ are the MFCC features and $T$ is the number of voiced frames. The
u-vectors are the $M$ dimensional vectors extracted using the UBM model,
whereas the a-vectors are the concatenation of the $M$ dimensional vectors,
extracted from the class-specific adapt models. The i-vectors are extracted as
mentioned in Dehak _et al._ (2010). Similarly, the x-vectors are extracted
from the trained TDNN-based x-vector model. Both the statistical/ embedding
vectors are computed by considering $N$ number voiced feature vectors as
analysis window length. The extracted vectors are then used to train the
linear discriminate analysis (LDA), within class covariance normalization
(WCCN) matrix, and the probabilistic LDA (PLDA) model.

Figure 9: Block diagram for the model-based change detection study.

$N(i)=\frac{1}{T}\sum_{j=1}^{T}P(i|x_{j})$ (4)

During testing, the feature vectors are extracted from the code-switched
utterances. After that using the VAD labels, with a fixed number of voiced
frames the statistical/embedding (S/E) vectors are extracted using the trained
models. The S/E vector extraction and the distance contour for each test
utterance are computed using Eq. 5. Where $x_{i}$s’ are the voiced feature
vectors, $\psi(.)$ is the distance computation function and $\mathbb{F}(.)$ is
the mapping function from the feature space to S/E vector space.

$D(i)=\psi(\mathbb{F}(x_{i-N},\ldots,x_{i}),\mathbb{F}(x_{i+1},\ldots,x_{i+N}))$
(5)

The distance contour is then smoothed using a hamming window with length
($h_{l}$). The $h_{l}$ is considered as $1/\delta$ times $N$. The peaks of the
smoothed contour are computed and the magnitude of peaks greater than the
threshold contour is considered as the change points.

###  V.2 Experimental Setup

The TTSF-SC dataset is used for SCD, whereas TTSF-LC and MSCSTB are used for
performing LCD tasks. The $39$ dimensional MFCC$+\Delta+\Delta\Delta$ feature
vectors are computed from the speech signal with $20$ msec and $10$ msec as
window and hop duration, respectively. The voiced frames are decided by
considering the frame energy that is greater than the $6\%$ of the utterance’s
average frame energy. The UBM and adapt models are trained with a cluster size
of $32$. The dimensions of the u/a/i-vectors are $32$, $64$ and $50$,
respectively. The recipe from the speech brain is used to train and extract
the $512$ dimension x-vectors Ravanelli _et al._ (2021). For the speaker-
specific study, the x-vectors are trained without dropout and L2
normalization, whereas for the language-specific study, dropouts of $0.2$ in
the second, third, fourth, and sixth layers are used along with L2
normalization.

During training, the speaker/language-specific voiced feature vectors are used
to extract the S/E vectors dis-jointly with a fixed $N$, whereas during
testing the S/E vectors are extracted with a sample frameshift. All the models
have been trained for $20$ epochs. For TTSF-LC the optimal $N$ is decided
experimentally as $200$ and for TTSF-SC $N$ is considered as $50$. After
training, by observing the validation loss and accuracy the model
corresponding to the $15^{th}$ and $11^{th}$ epoch is chosen for the language
and speaker-specific study, respectively. Similarly, for MSCSTB, x-vector
models for each language pair are trained. After training for $100$ epochs, by
observing the validation loss and accuracy the model belonging to the
($54^{th}$, $29^{th}$, and $26^{th}$) epochs for $N=200$ and ($25^{th}$,
$80^{th}$, and $18^{th}$) epochs for $N=50$ are chosen for GUE, TAE, and TEE
language pairs, respectively.

For TTSF-LC and TTSF-SC, the extracted embedding vectors are normalized
without having LDA and WCCN. The normalized vectors are used for modeling the
PLDA and computing the distance contour for LCD and SCD tasks. Using the
MSCSTB dataset, it is observed that performing LDA, and WCCN along with using
cosine kernel distance instead of PLDA distance contour improves the change
detection performance. This may be due to the nature of the datasets. The
TTSF-LC and TTSF-SC are the studio recording of read speech, whereas the
MSCSTB is the conversation recording in the office environment.

For the SCD task, after extracting the i/x vectors, the change points are
detected for each test utterance using the hyperparameters $\alpha$, $\delta$,
and $\gamma$ as $2.6$, $1.3$ and $0.9$, respectively. The hyper-parameters are
decided greedily by observing the change detection performance on the first
$100$ test trails. For the LCD task (using TTSF-LC), the hyper-parameters are
decided as $3.2$, $1.3$, and $0.9$, respectively. Similarly, for MSCSTB
($N=200$), the optimal hyperparameters for GUE, TAE, and TEE are ($0.3$,
$4.5$, and $1.1$), ($0.3$, $4.5$, and $1.1$) and ($0.3$, $3.9$, and $1.1$),
respectively. For $N=50$, the optimal hyperparameters are ($0.3$, $0.9$, and
$1.1$), ($0.3$, $0.9$, and $1.3$) and ($0.3$, $0.5$, and $1.3$), respectively.

###  V.3 Language discrimination by statistical/embedding vectors

The aim here is to observe the discrimination ability of the extracted S/E
vectors for language discrimination, by synthetically emulating the CS
scenario. The TTSF-LC, where the same speaker is speaking two languages is
considered for this study. The training partition is used to train the UBM,
adapt, T-matrix, and TDNN-based x-vector model. From the test partitions, two
utterances are selected, one from each language, spoken by a speaker. Using
the selected utterances the MFCC$+\Delta+\Delta\Delta$ features and the S/E
vectors are extracted and projected in two dimensions using t-SNE Maaten and
Hinton (2008).

Figure 10: t-SNE feature distribution between the Hindi (H) and English (E)
(a) MFCC features, (b) u-vector, (c) a-vector, (d) i-vector, and (e) x-vector.
Within and Between language PLDA score distribution, with EER of (f) $28.5$,
(g) $17.35$,(h) $12.55$, and (i) $3.6$ for u, a, i, and x-vector,
respectively.

The two-dimensional representations are depicted in Fig. 10(a-e). From the
figure, it can be observed that the overlapping between the languages reduces
by moving from the feature space to the S/E vector space. This shows, like
human subjects, prior exposure to the languages through ML/DL models helps in
better discrimination. Furthermore, among the S/E vectors, the overlap between
the languages is least in the x-vector space, followed by the i-vector, adapt,
and UBM posterior space. This is due to the ability of the modeling techniques
to capture the language-specific feature dynamics.

For strengthening the observation, the features are extracted from the test
utterances and pooled together with respect to a given language. The pooled
feature vectors are randomly segmented with a context of $200$ and used to
extract the S/E vectors. The extracted S/E vectors are paired to form $2000$
within a language (WL) and $2000$ between language (BL) trails. The WL and BL
vector pairs are compared using the PLDA scores. Fig. 10 (f-i) shows boxplots
of the PLDA score distribution of the WL and BL pairs. From the box plot
distribution, it can be observed that, between the WL and BL, the overlap of
PLDA scores distribution reduces with improvement in the modeling techniques
from UBM to x-vector.

In the change point detection task, the aim is to get a sudden change in the
distance contour, when there exists a change in language. That can be achieved
if the contour (negative of PLDA score) variation is less in WL and provide a
sudden change in the contour for BL pairs. Hence for ensuring this, the PLDA
score distribution between the WL and BL should be maximized. Keeping this
into account, the equal error rate (EER) has been used as an objective
measure, where the WL and BL trials are termed false scores and true scores,
respectively. The obtained EER for UBM/adapt/i-vector and x-vector are $28.5$,
$17.35$, $12.55$, and $3.6$, respectively. Hence as per the discrimination
ability, the change point detection study has been carried out using
i/x-vectors as the representations of the speaker and language.

Table 2: Performance of LCD and SCD by model-based approaches, S: statistical i-vector, E: embedding based x-vector, N: analysis window length. | TTSF-SC | TTSF-LC | MSCSTB | MSCSTB
---|---|---|---|---
GUE | TAE | TEE | GUE | TAE | TEE
S | E | S | E | E | E
N | 50 | 50 | 200 | 200 | 200 | 50
IDR | 87.75 | 92.27 | 80.58 | 87.01 | 46.56 | 49.91 | 47.13 | 54.74 | 52.19 | 50.84
FAR | 5.42 | 3.96 | 8.8 | 8.84 | 5.95 | 10.42 | 6.50 | 13.10 | 27.82 | 19.34
MDR | 6.83 | 3.76 | 10.57 | 4.41 | 47.49 | 39.67 | 46.36 | 32.16 | 19.99 | 29.83
Dm | 0.05 | 0.03 | 0.33 | 0.28 | 0.51 | 0.56 | 0.56 | 0.35 | 0.30 | 0.34

###  V.4 Experimental Results

Initially, the change detection study is conducted with TTSF-SC and TTSF-LC
using i/x-vectors as the speaker/language representation. The discrimination
ability and the LCD/SCD study suggest that the x-vector is a better
representation of the speaker/language than the i-vector. Therefore, the LCD
task on the MSCSTB dataset is conducted by considering x-vectors as language
representations.

The experimental results are tabulated in Table 2. The performance obtained in
terms of IDR on SCD task using i-vector and x-vector is $87.75\%$ and
$92.27\%$, respectively. Similarly, for LCD tasks the performances on TTSF-LC
are $80.58\%$ and $87.01\%$, respectively. As evidenced by the language
discrimination study, the performance of LCD provides a relative improvement
of $21.9\%$ and $31.63\%$ using i-vectors and x-vectors, over the best
performance achieved on the unsupervised distance-based approach,
respectively. This justifies the claim that, like humans, the performance of
the LCD can be improved by incorporating language-specific prior information
through computational models.

Figure 11:  Distribution of the mono-lingual segment duration of (a) TTSF-LC’s
HE, and MSCSTB’s (b) GUE, (c) TAE, and (d) TEE language pairs, respectively.

The performance of the LCD task on the MSCSTB dataset using x-vectors as
language representation with considering N as $200$ (same as TTSF-LC) is
$46.56\%$, $49.91\%$ and $47.13\%$ in terms of IDR for the GUE, TAE, and TEE
partitions, respectively. The performance provides a relative improvement of
$5.6\%$, $2.3\%$, and $4.2\%$. However, the improvement is small as compared
to the improvement achieved using TTSF-LC data. This may be due to the
distributional difference in the monolingual segment duration in the TTSF-LC
and MSCSTB datasets. A boxplot showing the distribution of the monolingual
segments of TTSF-LC’s and MSCSTB’s test set is depicted in Fig. 11. From the
figure, it can be observed that the median of the monolingual segment duration
in the case of TTSF-LC for primary and secondary language are ($5.54$ and
$4.9$) seconds, and for MSCSTB is ($1.46$ and $0.51$), ($1.54$ and $0.41$),
($1.61$ and $0.41$) seconds for GUE, TAE, and TEE partition, respectively.
Further, it has been observed that language discrimination is better by
considering $N$ equal to $200$ (i.e. approx. $2$ seconds). Hence, due to the
monolingual segment duration of the MSCSTB dataset being smaller than the
considered analysis window duration resulting in smoothing on the resultant
distance contour, and leads to an increase in the MDR. Therefore, the
alternative is to reduce the analysis window length, but that may affect the
language discrimination ability of the x-vectors.

Table 3: Perforance of LCD by varying the analysis window length. | MSCSTB
---|---
GUE
N | 200 | 150 | 100 | 75 | 50
IDR | 46.56 | 48.12 | 50.12 | 51.78 | 54.74
FAR | 5.95 | 7.38 | 12.54 | 11.01 | 13.10
MDR | 47.49 | 44.50 | 37.35 | 37.21 | 32.16
Dm | 0.51 | 0.48 | 0.37 | 0.33 | 0.35

A study is performed for observing the trade-off between the analysis window
length and language discrimination ability. The language discrimination test
and the LCD task are performed using the GUE partition of the MSCSTB dataset
by reducing the analysis window length from $200$ to $50$. The results of the
LCD task are tabulated in Table 3. The cosine score distribution of the
x-vectors’ WL and BL pairs after the LDA and WCCN projection with varying the
analysis window length are depicted in Fig. 12. From the Table, it can be
observed that with decreasing in $N$, the performance of the LCD task
improves, and achieved the best performance of $54.74\%$ at $N$ equals to
$50$. Hence the change detection performance is computed with $N$ equal to
$50$ for GUE, TAE, and TEE language pairs and tabulated in Table 2. However,
the relative performance improvement by incorporating language-specific prior
exposure through the x-vector model is not as expected as in the TTSF-LC
dataset. This is due to the language discrimination ability of the x-vectors
reducing with the decrease in $N$. From Fig. 12, it can be observed that the
overlap between the WL and BL score distribution increases with a decrease in
the value of $N$. As an objective measure, the computed EER for $N$ equals to
$200$, $150$, $100$, $75$ ,and $50$ are $7.1$, $9.8$, $12.8$, $19.8$ and
$29.2$, respectively.

Figure 12:  Within and Between language cosine score distribution, with EER of
(a) $7.1$, (b) $9.8$,(c) $12.8$, (d) $19.8$ and (i) $29.2$ for the analysis
window length of $200$, $150$, $100$, $75$ and $50$, respectively.

##  VI Discussion

The human-based LCD and SCD study suggests that the language requires more
neighborhood information as compared to the speaker for comfortable
discrimination. Further, prior exposure to the languages helps humans to
better discriminate between the languages. Motivated by this, it is
hypothesized that the performance of LCD by machine can be improved with the
(a) incorporation larger duration analysis window ($N$) and (b) language-
specific exposure through computational models.

In the unsupervised distance-based approach, it has been observed that the
performance of the LCD improves by increasing the value of $N$. The optimal
$N$ value for the SCD study is $150$. Considering the same value of $N$, the
LCD task is carried out for both TTSF-LC and MSCSTB datasets, and performances
are tabulated in Table 4. In the case of the MSCSTB dataset, the average IDR
values with respect to all three language pairs are tabulated. Motivating by
the LCD/SCD study by humans, the $N$ value is increased and the obtained
optimal $N$ value for the LCD with TTSF-LC is $200$. Similarly, the optimum N
value for MSCSTB is $160$, $180$, and $170$ for the GUE, TAE, and TEE ,
respectively. The performance with the optimal $N$ value for TTSF-LC and
MSCSTB is $66.1\%$ and $46.02\%$, which provides a relative improvement of
$29.1\%$, and $2.4\%$, respectively. These observations justify the claim that
the performance of the LCD by machines can be improved by increasing the
analysis window duration.

Table 4: Performance comparison, RI: relative improvement, A: with $N=150$ (tuned for SCD), B: with the optimal $N$ value (tuned for LCD), and C: x-vector based approach. Dataset | Approach | IDR | RI
---|---|---|---
TTSF-SC | A | 84.1 | -
C | 92.27 | 9.71
TTSF-LC | A | 51.2 | -
B | 66.1 | 29.1
C | 87.01 | 31.63
MSCSTB | A | 44.93 | -
B | 46.02 | 2.4
C | 52.59 | 14.27

Furthermore, as hypothesized from the subjective study, the incorporation of
language-specific exposure through computational models improves LCD
performance. The i/x-vector models have been trained, which essentially
capture the language-specific cepstral dynamics. It has been observed that
with the x-vector approach, the obtained performance is $87.01\%$ for TTSF-LC
and $52.59\%$ in terms of IDR, which provides a relative improvement of
$31.63\%$ and $14.27\%$ over the performance of the unsupervised distance-
based approach. Similarly, for the SCD task using the TTSF-SC dataset, the
performance provides a relative improvement of $9.71\%$. Comparing the
performance of LCD and SCD on synthetic data, it can be observed that the
improvement is more significant on LCD than the SCD. This concludes, like
human subjective study, in an ideal condition (only speaker/language variation
and keeping other variations limited), the requirement model-based approach is
more significant on LCD than the SCD.

It is also observed that in the LCD task, the performance improvement on
MSCSTB data is limited as compared to the improvement achieved on the
synthetic TTSF-LC dataset. This is due to the difference in the mono-lingual
segment duration. The trade-off between the analysis window duration and the
language discrimination ability shows that the discrimination ability improves
with an increase in analysis window duration. At the same time during change
detection, as the mono-lingual segment duration can possibly be lesser than
$500$ msec (approx. $50$ voiced frames), considering a larger analysis window
leads to degrading in performance by smoothening the evidence contour (leads
to an increase in MDR). Hence to overcome this issue, (1) need to achieve
significant language discrimination with the $N$ value as small as possible,
and (2) need to develop a framework whose performance will be least
affected/independent with the variations of the analysis window duration.

##  VII Conclusion

In this work, we performed LCD using the available frameworks for SCD. From
the subjective study, it is observed that humans require comparatively larger
neighborhood information around the change point as compared to the speaker.
It is also observed that prior language-specific exposure improves the
performance of the LCD task. In the unsupervised distance-based approach, the
incorporation of larger neighborhood information improves the LCD performance
by relatively $29.1\%$ and $2.4\%$ on the synthetic TTSF-LC and the practical
MSCSTB dataset, respectively. Similarly, incorporating language-specific prior
information through the computational models provides a relative improvement
of $31.63\%$ and $14.27\%$ over the unsupervised distance-based approach.

It has also been observed that the practical data set does not perform as
expected like synthetic data. This is due to the distributional difference in
the monolingual segment duration on both datasets. The MSCSTB dataset consists
of the monolingual segments having a duration lesser than $0.5$ secs, and for
better language discrimination the required duration is about $2$ secs (about
$200$ voiced frames). Hence it is challenging to decide on the analysis window
duration. The larger duration smooths the evidence contour and increases the
MDR, whereas a smaller duration of $0.5$ secs is not able to provide
appropriate language discrimination.

Therefore, our future attempts will try to develop a better framework, which
can provide better language discrimination on a small duration, and also plan
to come up with a change detection framework, whose performance should be
independent/less affected by the variations of the analysis window duration.

###### Acknowledgements.

The authors like to acknowledge ”Anatganak”, high-performance computation
(HPC) facility, IIT Dharwad, for enabling us to perform our experiments. And
the Ministry of Electronics and Information Technology (MeitY), Govt. of
India, for supporting us through different projects.

## References

  * Baby _et al._ (2016) Baby, A., Thomas, A. L., Nishanthi, N., Consortium, T. _et al._ (2016). “Resources for indian languages,” in _Proceedings of Text, Speech and Dialogue_.
  * Barras _et al._ (2006) Barras, C., Zhu, X., Meignier, S., and Gauvain, J.-L. (2006). “Multistage speaker diarization of broadcast news,” IEEE Transactions on Audio, Speech, and Language Processing 14(5), 1505–1512.
  * Bredin _et al._ (2017) Bredin, H., Barras, C. _et al._ (2017). “Speaker change detection in broadcast tv using bidirectional long short-term memory networks,” in _Interspeech 2017_ , ISCA.
  * Carrasquillo _et al._ (2002) Carrasquillo, P., Singer, E., Kohler, M., Greene, R., Reynolds, D., and Deller, J. (2002). “Approaches to language identification using gaussian mixture models and shifted delta cepstral features,” Proc of ICSLP2002-INTERSPEECH2002 16–20.
  * Chan _et al._ (2004) Chan, J. Y., Ching, P., Lee, T., and Meng, H. M. (2004). “Detection of language boundary in code-switching utterances by bi-phone probabilities,” in _2004 International Symposium on Chinese Spoken Language Processing_ , IEEE, pp. 293–296.
  * Chen _et al._ (1998) Chen, S., Gopalakrishnan, P. _et al._ (1998). “Speaker, environment and channel change detection and clustering via the bayesian information criterion,” in _Proc. DARPA broadcast news transcription and understanding workshop_ , Virginia, USA, Vol. 8, pp. 127–132.
  * Dawalatabad _et al._ (2020) Dawalatabad, N., Madikeri, S., Sekhar, C. C., and Murthy, H. A. (2020). “Novel architectures for unsupervised information bottleneck based speaker diarization of meetings,” IEEE/ACM Transactions on Audio, Speech, and Language Processing 29, 14–27.
  * Dehak _et al._ (2010) Dehak, N., Kenny, P. J., Dehak, R., Dumouchel, P., and Ouellet, P. (2010). “Front-end factor analysis for speaker verification,” IEEE Transactions on Audio, Speech, and Language Processing 19(4), 788–798.
  * Dhananjaya and Yegnanarayana (2008) Dhananjaya, N., and Yegnanarayana, B. (2008). “Speaker change detection in casual conversations using excitation source features,” Speech communication 50(2), 153–161.
  * Diwan _et al._ (2021) Diwan, A., Vaideeswaran, R., Shah, S., Singh, A., Raghavan, S., Khare, S., Unni, V., Vyas, S., Rajpuria, A., Yarra, C., Mittal, A., Ghosh, P. K., Jyothi, P., Bali, K., Seshadri, V., Sitaram, S., Bharadwaj, S., Nanavati, J., Nanavati, R., Sankaranarayanan, K., Seeram, T., and Abraham, B. (2021). “Multilingual and code-switching asr challenges for low resource indian languages,” Proceedings of Interspeech .
  * Gish _et al._ (1991) Gish, H., Siu, M.-H., and Rohlicek, J. R. (1991). “Segregation of speakers for speech recognition and speaker identification.,” in _icassp_ , Vol. 91, pp. 873–876.
  * Haris _et al._ (2012) Haris, B. C., Pradhan, G., Misra, A., Prasanna, S., Das, R. K., and Sinha, R. (2012). “Multivariability speaker recognition database in indian scenario,” International Journal of Speech Technology 15(4), 441–453.
  * Hogg _et al._ (2019) Hogg, A. O., Evers, C., and Naylor, P. A. (2019). “Speaker change detection using fundamental frequency with application to multi-talker segmentation,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , IEEE, pp. 5826–5830.
  * Li _et al._ (2013) Li, H., Ma, B., and Lee, K. A. (2013). “Spoken language recognition: from fundamentals to practice,” Proceedings of the IEEE 101(5), 1136–1159.
  * Liu _et al._ (2021) Liu, H., Perera, L. P. G., Zhang, X., Dauwels, J., Khong, A. W., Khudanpur, S., and Styles, S. J. (2021). “End-to-end language diarization for bilingual code-switching speech,” in _22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021_ , International Speech Communication Association, Vol. 2\.
  * Lu and Zhang (2002) Lu, L., and Zhang, H.-J. (2002). “Speaker change detection and tracking in real-time news broadcasting analysis,” in _Proceedings of the tenth ACM international conference on Multimedia_ , pp. 602–610.
  * Lyu _et al._ (2013) Lyu, D. C., Chng, E. S., and Li, H. (2013). “Language diarization for conversational code-switch speech with pronunciation dictionary adaptation,” in _Signal and Information Processing (ChinaSIP), 2013 IEEE China Summit and International Conference on_ , IEEE, pp. 147–150.
  * Maaten and Hinton (2008) Maaten, L. v. d., and Hinton, G. (2008). “Visualizing data using t-SNE,” Journal of machine learning research 9(Nov), 2579–2605.
  * Meignier _et al._ (2006) Meignier, S., Moraru, D., Fredouille, C., Bonastre, J.-F., and Besacier, L. (2006). “Step-by-step and integrated approaches in broadcast news speaker diarization,” Computer Speech & Language 20(2-3), 303–330.
  * Mishra _et al._ (2021) Mishra, J., Agarwal, A., and Prasanna, S. M. (2021). “Spoken language diarization using an attention based neural network,” in _2021 National Conference on Communications (NCC)_ , IEEE, pp. 1–6.
  * Moattar and Homayounpour (2012) Moattar, M. H., and Homayounpour, M. M. (2012). “A review on speaker diarization systems and approaches,” Speech Communication 54(10), 1065–1103.
  * Murty and Yegnanarayana (2008) Murty, K. S. R., and Yegnanarayana, B. (2008). “Epoch extraction from speech signals,” IEEE Transactions on Audio, Speech, and Language Processing 16(8), 1602–1613.
  * Park _et al._ (2022) Park, T. J., Kanda, N., Dimitriadis, D., Han, K. J., Watanabe, S., and Narayanan, S. (2022). “A review of speaker diarization: Recent advances with deep learning,” Computer Speech & Language 72, 101317.
  * Rabiner (1978) Rabiner, L. R. (1978). _Digital processing of speech signals_ (Pearson Education India).
  * Ravanelli _et al._ (2021) Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., Zhong, J. _et al._ (2021). “Speechbrain: A general-purpose speech toolkit,” arXiv preprint arXiv:2106.04624 .
  * Richardson _et al._ (2015) Richardson, F., Reynolds, D., and Dehak, N. (2015). “Deep neural network approaches to speaker and language recognition,” IEEE signal processing letters 22(10), 1671–1675.
  * Ryant _et al._ (2018) Ryant, N., Church, K., Cieri, C., Cristia, A., Du, J., Ganapathy, S., and Liberman, M. (2018). “First dihard challenge evaluation plan,” 2018, tech. Rep. .
  * Sarma _et al._ (2015) Sarma, M., Gadre, S. N., Sarma, B. D., and Prasanna, S. M. (2015). “Speaker change detection using excitation source and vocal tract system information,” in _2015 Twenty First National Conference on Communications (NCC)_ , IEEE, pp. 1–6.
  * Sharma _et al._ (2019) Sharma, N. K., Ganesh, S., Ganapathy, S., and Holt, L. L. (2019). “Talker change detection: A comparison of human and machine performance,” The Journal of the Acoustical Society of America 145(1), 131–142.
  * Siegler _et al._ (1997) Siegler, M. A., Jain, U., Raj, B., and Stern, R. M. (1997). “Automatic segmentation, classification and clustering of broadcast news audio,” in _Proc. DARPA speech recognition workshop_ , Vol. 1997\.
  * Sitaram _et al._ (2019) Sitaram, S., Chandu, K. R., Rallabandi, S. K., and Black, A. W. (2019). “A survey of code switching speech and language processing,” arXiv:1904.00784 [cs.CL] .
  * Snyder _et al._ (2018) Snyder, D., Garcia-Romero, D., McCree, A., Sell, G., Povey, D., and Khudanpur, S. (2018). “Spoken language recognition using x-vectors.,” in _Odyssey_ , pp. 105–111.
  * Spoorthy _et al._ (2018) Spoorthy, V., Thenkanidiyoor, V., and Dinesh, D. A. (2018). “SVM Based Language Diarization for Code-Switched Bilingual Indian Speech Using Bottleneck Features,” in _Proc. The 6th Intl. Workshop on Spoken Language Technologies for Under-Resourced Languages_ , pp. 132–136, http://dx.doi.org/10.21437/SLTU.2018-28, 10.21437/SLTU.2018-28.
  * Tranter and Reynolds (2006) Tranter, S. E., and Reynolds, D. A. (2006). “An overview of automatic speaker diarization systems,” IEEE Transactions on audio, speech, and language processing 14(5), 1557–1565.