# Crop Row Switching for Vision-Based Navigation: A Comprehensive Approach for
Efficient Crop Field Navigation

Rajitha de Silva1, Grzegorz Cielniak2 and Junfeng Gao3 This work was supported
by Lincoln Agri-Robotics as part of the Expanding Excellence in England (E3)
Programme.1Rajitha de Silva, 2Grzegorz Cielniak and 3Junfeng Gao are with
Lincoln Agri-Robotics Centre, Lincoln Institute for Agri-Food Technology,
University of Lincoln, UK (correspondence author: Junfeng Gao)
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Vision-based mobile robot navigation systems in arable fields are mostly
limited to in-row navigation. The process of switching from one crop row to
the next in such systems is often aided by GNSS sensors or multiple camera
setups. This paper presents a novel vision-based crop row-switching algorithm
that enables a mobile robot to navigate an entire field of arable crops using
a single front-mounted camera. The proposed row-switching manoeuvre uses deep
learning-based RGB image segmentation and depth data to detect the end of the
crop row, and re-entry point to the next crop row which would be used in a
multi-state row switching pipeline. Each state of this pipeline use visual
feedback or wheel odometry of the robot to successfully navigate towards the
next crop row. The proposed crop row navigation pipeline was tested in a real
sugar beet field containing crop rows with discontinuities, varying light
levels, shadows and irregular headland surfaces. The robot could successfully
exit from one crop row and re-enter the next crop row using the proposed
pipeline with absolute median errors averaging at 19.25 cm and 6.77°for linear
and rotational steps of the proposed manoeuvre.

##  I Introduction

The integration of robotics and autonomous systems in agriculture has enabled
precision farming operations leading to effective use of time and resources.
Achieving autonomous navigation in agricultural robots is an enabling
technology that must be optimised for the deployment of autonomous robots for
precision agriculture. The existing autonomous navigation solutions for in-
field navigation, albeit being efficient, often relies on expensive sensors
such as Real-Time Kinematic Global Navigation Satellite System (RTK-GNSS)
sensors. Camera-based agricultural robot navigation systems are a popular
alternative to these expensive sensors [1]. Implementation of such vision
systems is often limited to crop row following behaviour [2]. In most such
systems, row switching is achieved by the aid of GNSS sensors or multiple
cameras to identify row end and re-entry to the next row [1, 3, 4].

Figure 1: Crop row switching state machine. $d_{r}$: Distance between current
and next crop row, A: Initial Detection of the EOR, B: Robot is at the EOR, C:
Robot traversed a distance equal to its length into the headland, D: Robot
turn 90°towards the next row direction from state C, E: Robot traverse $d_{r}$
distance forward from state D, F: Robot turn 90°towards the next row direction
from state E, G: Robot re-enter the next crop row.

The premise of vision-based navigation in agricultural robots is to reduce the
cost of the overall robotic system with the aim of increased adoption of such
technologies. To this end, a vision-based navigation system that uses a single
front-mounted camera to perform in-row navigation and row switching would be
in line with this objective. In our previous work, we have developed a vision-
based crop row detection algorithm [5] and an in-row navigation framework [6]
that then uses the crop row detection to guide the robot through a single crop
row only based on RGB images. The crop row switching algorithm presented in
this paper relates to our previous work on vision-based crop row detection [5]
and navigation [6] in arable fields. The newly proposed algorithm serves as a
vital bridge, seamlessly connecting with the established in-row navigation
algorithm to dexterously integrate into a comprehensive, fully autonomous
field-scale navigation behaviour. This row-switching algorithm could also
integrated seamlessly with any other existing in-row navigation methods to
eliminate the need for GNSS sensors in row-switching. This existing system can
follow a crop row based on RGB image input and it can also identify the
location of the end of crop rows when the robot is reaching towards the
headland area. The crop row switching algorithm presented is be triggered upon
the detection of the end of row (EOR) and navigates the robot towards the
entry point of the next crop row to be traversed.

The main contributions of this work are as follows:

  * •

A vision-based re-entry point detection algorithm that identifies the relative
3D position of the entry point of the adjacent crop row.

  * •

A crop row switching pipeline based on vision and robot wheel odometry to
navigate the robot towards the entry point of the next crop row to be
traversed.

  * •

A comprehensive evaluation and validation of an autonomous navigation system
in a real arable field with a mobile robot deployed the proposed system.

##  II Related Work

Research into vision-based navigation systems for crop row following is an
extensively explored subject area [7, 8]. Existing robot navigation
technologies for agri-robotics research include GNSS, inertial navigation
systems (INS) and light detection and ranging (LiDAR) [9]. Each of these
existing systems presents a cost-benefit trade-off which results in a lack of
reliable navigation options for an affordable solution in crop row navigation.
The vision-based infield navigation technologies available for arable field
navigation explores the usage of RGB and depth images to identify the crop
rows [1, 10]. The existing systems for such infield navigation mainly uses
image segmentation, object detection or image matching methods to identify the
crop rows for robot navigation [7]. Despite the considerable interest in
vision-based crop row following systems, the presence of reliable vision-based
crop row switching algorithms remains limited [2]. Most of the existing
vision-based navigation systems depend completely or partially on GNSS, INS or
Lidar-based solutions for crop row switching rather than a fully vision-based
solution [7]. The crop row switching algorithms proposed in [2, 11, 12]
completely rely on the RTK-GNSS-based sensors to perform the crop row
switching manoeuvre. However, GNSS-based systems are not considered a simple
and straightforward solution for agricultural robot navigation since they also
need multiple redundancies in place for effective operation due to multi-path
reflections and signal blockage[13]. Some systems also use manual control to
perform the headland turn [14]. Fully vision-based solutions in agricultural
robot navigation depend on multiple cameras on the robot for maintaining the
localisation during the row switching process [15, 16]. The methods that use a
single camera also demand special requirements such as variable field of view
[17].

End of row (EOR) detection is an important step for any crop row switching
algorithm since it serves as the starting point for any crop row switching
manoeuvre [6]. The EOR detection scheme proposed in [18] employs image
binarisation using classic computer vision methods to calculate the pixel
count in binary masks to determine the EOR. However, the pixel count
thresholds may vary depending on the stage of the plant growth and these
thresholds must be matched to each growth stage when used for the long term.
The pixel counts of binary image representing the crop would not yield the
spatial localisation of EOR within a given image, while it only generates an
EOR trigger based on the image. In contrast, the EOR detection method proposed
by authors of [19] uses $Cr$ channel of $YCbCr$ colour space to calculate the
position of the EOR within a given image. Such methods pose the advantage of
early detection and potential failures due to noisy images in the method
presented in [18]. However, such colour-based EOR detection methods are highly
susceptible to distortions caused by external field variations. The advantages
of deep learning-based methods in EOR detection outperform such colour-based
methods [5, 6]. A 3D point cloud-based row-end detection was introduced in
[14] where the EOR is identified by detection of height drop between the plant
and the ground within the point cloud. This approach is mostly limited to
crops with noticeable height differences relative to the ground level or crops
at later growth stages. The novel EOR detection algorithm presented in this
paper is based on the deep learning-based crop row detection model from our
earlier work [5], considering the limitations of the existing EOR detection
methods. Identification of the relative distance between two crop rows is also
vital for accurate crop row switching. The distance between two crop rows
often referred to as ”inter-row distance” is considered as a fixed distance in
existing crop row switching methods [3, 20]. The relative position between the
current robot pose and the next crop row could vary due to imperfections in
planting or due to a slight offset during robot navigation. Therefore, active
perception of the relative position of the re-entry point to the next crop row
to be traversed is a useful attribute. The re-entry point detection algorithm
proposed in this work estimates the relative distance to the next crop row
based on crop row segmentation mask from Triangle Scan Method (TSM) [5] and
depth data. Such active perception of inter-row space helps to eliminate any
potential row-switching failures caused by varying inter-row space or
inaccurate positioning of the robotic platform within the currently traversing
crop row. Four most common headland turning patterns are semi-circular turn,
U-turn, light bulb turn (a.k.a $\Omega$ turn) and switch-back turn [11]. The
semi circular turn describes a half circle with a constant radius while U-turn
describes two quarter circle turns (90°) centered with a linear traversal
stage. These methods are typically used in smaller robots with tighter turning
radii relative to the inter-row distance [18, 15, 20]. The $\Omega$ turn and
the switch-back turn are mostly used in robots with constrained
maneuverability such as Ackerman steering robots or tractors [11, 21, 22].
Considering the skid-steering robot configuration of the robotic platform used
in this work, U-turn pattern was chosen to execute the row switching
manoeuvre.

##  III Methodology

Figure 1 portrays a state machine representing the process of crop row
switching, spanning across a total of seven states. Table I describe each
state of the process given in Figure 1 encountered during row switching
manoeuvre. The robot was driven within the crop rows using Triangle Scan
Method (TSM) based crop row navigation framework [6] we have developed in our
previous work up to state $A$. The switching manoeuvre is composed of three
steps: row exit, U-turn and re-entry. The transitions $A\rightarrow
B\rightarrow C$ belong to the step of row exit. The U-turn step contains to
the transitions $C\rightarrow D\rightarrow E\rightarrow F$ while the
transition $F\rightarrow G$ is considered as re-entry step. The methods and
techniques used in each of these three steps are explained in sections III-B,
III-C and III-D respectively.

TABLE I: States encountered during crop row switching manoeuvre. State |  Description
---|---
$A$ |  Position of the robot during initial EOR detection.
$B$ |  Robot reaches the EOR position.
$C$ |  Robot enters the headland area completely passing EOR position.
$D$ |  Robot turns 90°towards the direction of next crop row from state $C$.
$E$ |  Robot aligns itself with the next crop row traversing forward from state $D$.
$F$ |  Robot turns 90°facing itself towards the starting point of the next crop row while residing within the headland buffer.
$G$ |  Robot enters the next crop row.

###  III-A Robotic Platform

A Mark-1 robot from Hexman Robotics was used to test the proposed crop row
switching algorithm. Figure 2 shows the robotic setup with a front mounted
Intel RealSense D435i camera and a Reach RS+ RTK GNSS receiver. A NVIDIA
Jetson AGX Orin developer kit was used as the onboard computer for the robot.
Mark-1 is a skid steer robot with a ground clearance of 128 mm. The dimensions
of the robot are $526\times 507\times 244$ mm.

Figure 2: Hexman Mark-1 robot in the Sugar Beet Field.

###  III-B Row Exit Step

Row exit is the process in which the robot drives itself completely out of the
currently traversing crop row after detecting the EOR. The EOR is initially
detected at state $A$ where the robot estimates the relative 3D coordinate of
the starting point of the next crop row robot would enter. The Y value of this
relative 3D coordinate would represent the inter-row distance $d_{r}$ between
current and the next crop row. The $d_{r}$ value is useful when robot is
traversing during the $D\rightarrow E$ transition. The robot transits through
the sates $A\rightarrow B\rightarrow C$ after the re-entry point detection
using a combination of visual feature matching and odometry as explained in
following subsections.

####  III-B1 Re-entry Point Detection

The re-entry point detection module is extended based on the TSM crop row
detection pipeline from our previous work [5] on crop row detection. Similar
to TSM, a deep learning based skeleton segmentation of crop rows was used as
the input for the re-entry point detector. As illustrated in Figure 3, the ROI
$AL_{2}L_{3}B$ was defined if the next intended turn is to the left
($AR_{2}R_{3}C$ for right). Points $A,B$ and $C$ were determined using the
”Anchor scans” step of TSM [5]. The horizontal green line in Figure 3 was
obtained using the EOR detector [6]. Equation 1 yields a point $P_{t}$ by
scanning for the pixel sum along $AP$ line where point $P$ is an arbitrary
point on $L_{2}L_{3}B$ path. Similarly, equation 2 yields a point $A_{t}$ by
scanning the pixel sum along $\overline{A}P_{t}$ line where point
$\overline{A}$ is an arbitrary point on $AL_{1}$ line. The intersection
between the EOR line and $A_{t}P_{t}$ is identified as the re-entry point $R$
for the next crop row. The depth information from the corresponding depth
image was used to determine the 3D coordinate corresponding to the point $R$
which was then used to determine $d_{r}$.

Figure 3: Regions of interests (ROI) for re-entry point scanning. Red: Left
side ROI, Blue: Right side ROI, Green: Detected EOR line.

$P_{t}=\operatorname*{arg\,max}\Biggl{[}\sum_{I_{xy}=A}^{P}I(x,y)\Biggr{]}_{P=L_{1}\rightarrow
L_{2}}^{P=L_{2}\rightarrow B}$ (1)

$A_{t}=\operatorname*{arg\,max}\Biggl{[}\sum_{I_{xy}=P_{t}}^{\overline{A}}I(x,y)\Biggr{]}_{\overline{A}=A}^{\overline{A}=L_{1}}$
(2)

####  III-B2 A to B Transition

The TSM based crop row navigation framework from our previous work [6] was
proven to be able to maintain the average heading of the robot relative to the
crop row under 1°. Therefore, the relative heading angle between the robot and
the crop row was assumed to be under 1°at state $A$. The detected EOR line in
Figure 3 demarcates the headland area and the crop row region within the RGB
image obtained from the front mounted camera. The RGB image was cropped below
the EOR line and saved as a reference image $I_{R}$ at state $A$. The robot
was then moved towards the EOR with a constant forward linear velocity while
calculating the local feature (Scale Invariant Feature Transform [23] was
used) similarity score in each new image captured by the robot camera with
$I_{R}$. The robot was stopped and assumed to reach state $B$ when the feature
similarity score dropped below a experimentally determined threshold. The
threshold value was determined by observing the feature similarity score while
driving the robot to the actual EOR position using teleoperation.

####  III-B3 B to C Transition

The frontmost edge of the robot is coincident with EOR at state $B$. The
minimum distance the robot must move in order to completely exit the crop row
is the length of the robot ($L_{R}$) itself. The wheel odometry of the robot
was used as feedback to move the robot forward towards the headland. The
length of the Hexman Mark 1 robotic platform used in during the field trials
was 526 mm. The wheel odometry of the robot was assumed to be accurate enough
to navigate the robot forward into the headland at such small distances.

###  III-C U-Turn Step

The U-turn step involves the robot taking two 90°turns with a linear
navigation stage ($D\rightarrow E$) in between. ”Headland buffer” region is
the space in the headland that is directly in front of a given crop row bound
by the EOR, edge of the field and the two centre lines of the inter-row space
between adjacent crop rows. The goal of the U-turn step is to bring the robot
into the headland buffer region of the next crop row while facing towards the
next crop row to be traversed. It was experimentally verified that the TSM can
resume it’s normal crop row navigation framework from this point(state $F$)
onward to traverse in to the crop row. The 90°turns and $D\rightarrow E$
transition is executed with wheel odometry feedback of the robot. The
practical behaviour of the robot during rotation stages ($C\rightarrow D$ and
$E\rightarrow F$) ensured that the robot would reach the headland buffer when
the $D\rightarrow E$ transition distance was set to $d_{r}$.

###  III-D Re-Entry Step

The robot is within the headland buffer region at state $F$ and state $G$ is
reached when the robot enters the next crop row to be traversed. The re-entry
step is the transition from state $F$ to state $G$ where the robot moves from
the headland buffer into the crop row in front of it. This transition was
realized by launching the TSM based crop row following framework at state $F$.
The TSM was able to detect the crop row in front of the robot and navigate the
robot into the crop row. This behaviour of TSM based re-entry navigation is
verified by the experiment outlined in Section IV-C.

##  IV Results and Discussion

The row switching manoeuvre presented in this paper is a three-step process
that involves 6 state transitions as illustrated in Figure 1. The accuracy and
the performance of each transition is individually evaluated during the
experiments. The Hexman Mark-1 robot was programmed to follow a crop row and
automatically detect the EOR and re-entry positions. The proposed row-
switching manoeuvre is automatically started based on the EOR detection
trigger. The path of the robot during the row-switching manoeuvre was tracked
using the onboard RTK GNSS tracker with sub-centimeter accuracy.

An experiment was set up in a selected area of 10 crop rows within a real
sugar beet field. The GNSS coordinates of the 10 crop rows were recorded with
sub-centimetre accuracy by driving the robot through each crop row at very
slow speeds by an expert human driver. These ground truth GNSS coordinates of
each crop row was then used to generate a regression line which would be used
as a reference to calculate errors in autonomous navigation during row
switching manoeuvre of the robot. The robot was allowed to autonomously
execute the row switching manoeuvre among the 10 crop rows turning in both
directions (left and right turns). A total of 18 row-switching trials were
conducted during this experiment as plotted in Figure 5. The GNSS coordinates
were converted to Universal Transverse Mercator (UTM) coordinates for plotting
and error calculations. The errors during each state transition of each row
switching manoeuvre trial are illustrated in Figure 4.a where distance errors
and angular errors are normalized within a scatter plot. There are 9 x-ticks
in Figure 4.a between each pair of adjacent states which represents a pair of
consecutive real-world crop rows where the trial took place. A box and whisker
plot of the errors for each transition is denoted in Figure 4.b. This
normalised representation provides a comparative illustration of the error
magnitudes in each state transition. The plots corresponding to translational
steps stands above the zero line while box and whisker plots of angular
transitions are centered close to zero. Table II presents the median errors of
traversal during each stage of the row-switching manoeuvre. Normalized median
percentage error ($\alpha=E_{median}/E_{max,T}$) was calculated where
$E_{max,T}$ is the maximum absolute error for the type $T$ (T being a distance
or an angle) across the entire manoeuvre and $E_{median}$ is the median error
for each transition. The vision based transition $A\rightarrow B$ records the
highest $\alpha$ value representing the transition with highest error. The
remaining transitions record relatively lower $\alpha$ values representing
accurate navigation relative to the vision-based transition.

TABLE II: Linear and angular errors during each transition of the row switching maneuver. Transition | Median Errors | $\alpha$
---|---|---
Error | Absolute Error
$A\rightarrow B$ | 23.40 cm | 31.63 cm | 40.20 %
$B\rightarrow C$ | 8.87 cm | 8.87 cm | 15.24 %
$C\rightarrow D$ | -1.09° | 6.91° | -2.88 %
$D\rightarrow E$ | 12.47 cm | 17.26 cm | 21.41 %
$E\rightarrow F$ | 2.51° | 6.62° | 6.64 %
Figure 4: Normalized state transition errors during the row switching
manoeuvre. a: Scatter plot, b: Box and whisker plot. Figure 5: UTM projections
of the GNSS trajectories from row switching experiments (Black: Regression
lines and ground truth coordinates). Figure 6: GNSS trajectories from re-entry
experiment (Black: Ground truth coordinates).

###  IV-A Crop row exit and headland entry

The row exit step ($A\rightarrow B\rightarrow C$ ) in the proposed row
switching manoeuvre could be identified with the densely distributed GNSS
coordinates in Figure 5 leading towards headland from within the crop row in
each trial. The dense distribution of GNSS coordinates attributes to the
slower speed of row switching manoeuvre relative to the in-row navigation
speed of the crop row navigation algorithm.

The row crop row exit ($A\rightarrow B$ transition) is a vision-based
navigation stage where the headland entry ($B\rightarrow C$ transition) uses
the wheel odometry to guide the robot into the headland area. This difference
in feedback modalities is reflected in the distance errors of each transition.
The visual feedback in $A\rightarrow B$ transition ensures that the robot has
successfully reached the EOR position with visual confirmation. Therefore, it
is vital for successfully reaching the EOR despite the higher error margins
compared to wheel odometry. Majority of the errors in $A\rightarrow B$ and
$B\rightarrow C$ transitions are positive, which indicates that the robot
always travel further into the headland area beyond desired positions at
states $B$ and $C$. This trend doesn’t have significantly adverse impact on
the overall row switching manoeuvre since such extra distance traversed into
the headland wouldn’t cause the robot to damage crops during the U-turn step.

The robot would stop at 52.6 cm ($L_{robot}$) away from the actual EOR
position in an ideal $C$ state. However, the overall maximum error in the row
exit step ($E_{ABC,max}$) was recorded as 64.27 cm, a distance which the robot
would move further away from the desired position at state C. Based on these
observations, the minimum width $W_{H,min}$ for the headland space was
calculated to be 143.17 cm using Equation 3. The coefficient of $L_{robot}$
term in Equation 3 was set to $1.85$ since the RTK-GNSS receiver used to
measure the robot motion was mounted at 45 cm behind the front of the robot.
This coefficient must be changed to $(1+R)$ where $R$ depends on the position
of the RTK-GNSS receiver on the robot, if this experiment is being repeated on
a different robot. The robot was expected to be aligned with the crop row at
state $A$ within a heading error margin of 2°. A heading error beyond 2°at
state $A$ leads the robot to cross into the headland buffer of an adjacent
crop row at state $C$, which would cause it to skip 1 crop row during
switching or re-enter to the same crop row it traversed as seen in the
unsuccessful attemps in Figure 5.

$W_{H,min}=1.85\times L_{robot}+E_{ABC,max}$ (3)

###  IV-B U-turn towards next crop row

The U-turn step of the row switching manoeuvre is represented by the state
transitions $C\rightarrow D\rightarrow E\rightarrow F$. The angular error for
$C\rightarrow D$ transition was calculated based on the angle between
$\overrightarrow{AC}$ and $\overrightarrow{DE}$ vectors. The angle between
$\overrightarrow{DE}$ and $\overrightarrow{FF_{N}}$ vectors was considered to
calculate the angular error for $E\rightarrow F$ transition where $F_{N}$ is a
point on the GNSS trajectory which is $N$(=5) points after the GNSS coordinate
of state F. Distance error for $D\rightarrow E$ transition was calculated by
comparing the $DE$ distance with the inter-row distance between the adjacent
crop rows which the robot is being switched, using regressed ground truth
lines.

The angular errors in both rotational transitions are evenly distributed with
a near zero mean. The absolute median angular errors during rotational
transitions were less than 7°. These error margins could be considered as
acceptable for this application scenario since such small angular errors
wouldn’t incur significant deviations to $\overrightarrow{DE}$ and
$\overrightarrow{FG}$ vectors. The robot would stay at the same place without
any translational motion in an ideal rotational transition. However, it was
evident from some of the recorded trajectories in Figure 5 that these
rotations also incur some translational motion within the trajectory. Such
motions push the robot towards or away from the direction of next crop row.
This would cause the robot to skip the next crop row to be traversed or turn
towards the same row it came in, despite achieving the desired
$\overrightarrow{DE}$ distance. This unintended translational motion is often
caused by the uneven terrain in the headland area which is not detected by the
wheel odometry. The headland buffer area of $4^{th}$ crop row had such uneven
terrain which lead to failure of both left and right turns originated from
that crop row.

###  IV-C TSM based Re-Entry Validation

The median error for $D\rightarrow E$ transition which led to successful re-
entry was always below 30 cm. This indicate that the robot could execute a
successful re-entry when it faces the next crop row with a perpendicular
offset of 30 cm or below at state $F$. An experiment was setup to validate
this hypothesis where the robot was placed facing towards the crop row at
different angles within the headland buffer of a given crop row. The TSM
algorithm was executed on the robot such that it would detect the crop row in
front of it and gradually drive the robot into the crop row in front. The path
of the robot was recorded in GNSS coordinates as plotted in Figure 6. All the
recorded trajectories could successfully enter the row in front of it since
the robot was initiated in the headland buffer and facing towards the general
direction of the crop row in front of it.

The re-entry failures in Figure 5 could be explained by the main findings of
this experiment. There are two key factors governing the success of a re-entry
to the next crop row during $F\rightarrow G$ transition. First requirement is
that the robot must be positioned within the headland buffer of the crop row
it intends to enter. If the perpendicular offset between the crop row and
robot position at state $F$ is beyond 30 cm, a re-entry failure occurs. The
second factor is that the robot must be oriented towards the general direction
of the crop row it intends to enter. Maximum deviation angle of the robot
heading from crop row was 26°in the experiment illustrated in Figure 6.
Although the errors in individual state transitions of the row switching
manoeuvre are minimal, the overall outcome of all transitions would not lead
to success when these two requirements are not met at state $F$.

##  V Conclusion and Future Work

The proposed row-switching manoeuvre could navigate the robot from one crop
row to another without needing RTK-GNSS sensors or multiple cameras while
using a single front mounted camera on the robot. Individual steps of the row-
switching manoeuvre demonstrated excellent results within the context of each
state transition and its functionality. The vision-based $A\rightarrow B$
transition was the transition with highest errors in the proposed manoeuvre.
The rotational transitions of the manoeuvre yielded smaller percentage errors
relative to the translational transitions. The success rate of the conducted
row switching experiment was 55.5% while the re-entry experiment yielded 100%
success rate. The row switching manoeuvre is also crop agnostic since it
doesn’t rely on plant specific visual features.

Two main unexpected behaviour patterns in the row exit and U-turn steps lead
to the failure of row switching: large heading error at state $A$ and
translational motion during rotational state transitions. These two
shortcomings of the proposed row switching manoeuvre could be corrected by
introducing a heading correction step at state $A$ and inertial measurement
unit (IMU) based sensor fusion to track the unintended translational motion
during the rotational transitions correcting the intended
$\overrightarrow{DE}$ distance.

## References

  * [1] S. Bonadies and S. A. Gadsden, “An overview of autonomous crop row navigation strategies for unmanned ground vehicles,” _Engineering in Agriculture, Environment and Food_ , vol. 12, no. 1, pp. 24–31, 2019.
  * [2] P. Huang, Z. Zhang, and X. Luo, “Feedforward-plus-proportional–integral–derivative controller for agricultural robot turning in headland,” _International Journal of Advanced Robotic Systems_ , vol. 17, no. 1, p. 1729881419897678, 2020.
  * [3] A. Ahmadi, L. Nardi, N. Chebrolu, and C. Stachniss, “Visual servoing-based navigation for monitoring row-crop fields,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 4920–4926.
  * [4] S. Kanagasingham, M. Ekpanyapong, and R. Chaihan, “Integrating machine vision-based row guidance with gps and compass-based routing to achieve autonomous navigation for a rice field weeding robot,” _Precision Agriculture_ , vol. 21, no. 4, pp. 831–855, 2020.
  * [5] R. de Silva, G. Cielniak, G. Wang, and J. Gao, “Deep learning-based crop row detection for infield navigation of agri-robots,” _Journal of Field Robotics_ , 2023.
  * [6] R. de Silva, G. Cielniak, and J. Gao, “Vision based crop row navigation under varying field conditions in arable fields,” _arXiv preprint arXiv:2209.14003_ , 2022.
  * [7] T. Wang, B. Chen, Z. Zhang, H. Li, and M. Zhang, “Applications of machine vision in agricultural robot navigation: A review,” _Computers and Electronics in Agriculture_ , vol. 198, p. 107085, 2022.
  * [8] N. Shalal, T. Low, C. McCarthy, and N. Hancock, “A review of autonomous navigation systems in agricultural environments,” _SEAg 2013: Innovative agricultural technologies for a sustainable future_ , 2013.
  * [9] Z. Man, J. Yuhan, L. Shichao, C. Ruyue, X. Hongzhen, and Z. Zhenqian, “Research progress of agricultural machinery navigation technology,” _Nongye Jixie Xuebao/Transactions of the Chinese Society of Agricultural Machinery_ , vol. 51, no. 4, 2020.
  * [10] Y. Bai, B. Zhang, N. Xu, J. Zhou, J. Shi, and Z. Diao, “Vision-based navigation and guidance for agricultural autonomous vehicles and robots: A review,” _Computers and Electronics in Agriculture_ , vol. 205, p. 107584, 2023.
  * [11] Z. He, Y. Bao, Q. Yu, P. Lu, Y. He, and Y. Liu, “Dynamic path planning method for headland turning of unmanned agricultural vehicles,” _Computers and Electronics in Agriculture_ , vol. 206, p. 107699, 2023.
  * [12] D. S. Paraforos, R. Hübner, and H. W. Griepentrog, “Automatic determination of headland turning from auto-steering position data for minimising the infield non-working time,” _Computers and electronics in agriculture_ , vol. 152, pp. 393–400, 2018.
  * [13] F. Rovira-Más, I. Chatterjee, and V. Sáiz-Rubio, “The role of gnss in the navigation strategies of cost-effective agricultural robots,” _Computers and electronics in Agriculture_ , vol. 112, pp. 172–183, 2015\.
  * [14] J. Kneip, P. Fleischmann, and K. Berns, “Crop edge detection based on stereo vision,” _Robotics and Autonomous Systems_ , vol. 123, p. 103323, 2020.
  * [15] A. Ahmadi, M. Halstead, and C. McCool, “Towards autonomous visual navigation in arable fields,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 6585–6592.
  * [16] E. Liu, J. Monica, K. Gold, L. Cadle-Davidson, D. Combs, and Y. Jiang, “Vision-based vineyard navigation solution with automatic annotation,” _arXiv preprint arXiv:2303.14347_ , 2023.
  * [17] J. L. Xue and T. E. Grift, “Agricultural robot turning in the headland of corn fields,” _Applied Mechanics and Materials_ , vol. 63, pp. 780–784, 2011\.
  * [18] P. Huang, L. Zhu, Z. Zhang, and C. Yang, “Row end detection and headland turning control for an autonomous banana-picking robot,” _Machines_ , vol. 9, no. 5, p. 103, 2021.
  * [19] Z. Zhang, R. Cao, C. Peng, R. Liu, Y. Sun, M. Zhang, and H. Li, “Cut-edge detection method for rice harvesting based on machine vision,” _Agronomy_ , vol. 10, no. 4, p. 590, 2020.
  * [20] L. Guevara, M. M. Michałek, and F. A. Cheein, “Headland turning algorithmization for autonomous n-trailer vehicles in agricultural scenarios,” _Computers and Electronics in Agriculture_ , vol. 175, p. 105541, 2020.
  * [21] H. Wang and N. Noguchi, “Adaptive turning control for an agricultural robot tractor,” _International Journal of Agricultural and Biological Engineering_ , vol. 11, no. 6, pp. 113–119, 2018.
  * [22] J. T. Evans IV, S. K. Pitla, J. D. Luck, and M. Kocher, “Row crop grain harvester path optimization in headland patterns,” _Computers and electronics in agriculture_ , vol. 171, p. 105295, 2020.
  * [23] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _International journal of computer vision_ , vol. 60, pp. 91–110, 2004.


# A Further Study of Unsupervised Pre-training for Transformer Based Speech
Recognition

###### Abstract

Building a good speech recognition system usually requires large amounts of
transcribed data, which is expensive to collect. To tackle this problem, many
unsupervised pre-training methods have been proposed. Among these methods,
Masked Predictive Coding achieved significant improvements on various speech
recognition datasets with BERT-like Masked Reconstruction loss and Transformer
backbone. However, many aspects of MPC have not been fully investigated. In
this paper, we conduct a further study on MPC and focus on three important
aspects: the effect of pre-training data speaking style, its extension on
streaming model, and how to better transfer learned knowledge from pre-
training stage to downstream tasks. Experiments reveled that pre-training data
with matching speaking style is more useful on downstream recognition tasks. A
unified training objective with APC and MPC provided 8.46% relative error
reduction on streaming model trained on HKUST. Also, the combination of target
data adaption and layer-wise discriminative training helped the knowledge
transfer of MPC, which achieved 3.99% relative error reduction on AISHELL over
a strong baseline.

Index Terms: unsupervised pre-training, transformer, speaking style, streaming
speech recognition, knowledge transfer

##  1 Introduction

Current industrial end-to-end automatic speech recognition (ASR) systems rely
heavily on large amounts of high quality transcribed audio data. However,
transcribed data take substantial effort to obtain in industrial applications,
while at the same time, a lot of un-transcribed data exist in online systems
and cost little to collect. It is worthwhile to explore how to effectively use
un-transcribed data to improve the performance of speech recognition systems
when labeled data are limited.

Recently, unsupervised pre-training has shown promising results in several
areas, including Computer Vision (CV) [1], Natural Language Processing (NLP)
[2] and so on. One work that stands out among these methods is Bidirectional
Encoder Representations from Transformers (BERT) [2], which used a Masked
Language Model (MLM) pre-training objective and obtained new state-of-the-art
results on eleven NLP benchmarks.

In speech area, researchers also proposed many unsupervised pre-training
algorithms. Contrastive Predictive Coding (CPC) [3] extracts representation
from data using contrastive loss that requires distinguishing a true future
audio sample from negatives. There are multiple concurrent approaches that
generalize this approach [4, 5] and applied it in learning speaker
representation [6], extracting speech representation [7, 8], performing
various speech-related tasks like speech recognition [7, 9, 10], speech
emotion recognition [11] and so on. [12, 13, 14] proposed Autoregressive
Predictive Coding (APC) objective that predicts unseen future frame given past
frames and also achieved good results on phonetic classification, speech
recognition, and speech translation. Some other work [15, 16, 17, 18] got
motivation from NLP and applied similar methods on speech tasks.

Among these methods, Masked Predictive Coding (MPC) [18] achieved significant
improvements on state-of-the-arts Transformer based speech recognition models
on various datasets without introducing any additional parameters for the
speech recognition model. However, many aspects of MPC have not been fully
explored:

  * •

Speaking style has a strong impact on performance of ASR systems [19, 20].
Whether speaking style of pre-training data would affect the performance of
downstream tasks has not been fully investigated.

  * •

The ability to perform streaming recognition is important for ASR systems. MPC
may not work for streaming recognition because it requires bidirectional
context for predictive coding. How MPC can contribute to streaming models is
worth exploring.

  * •

Despite abundant work [21, 22, 23] on NLP about knowledge transfer between
pre-trained model and downstream task, there are few work exploring how to
perform better knowledge transfer in the area of speech.

In this paper, we investigate these aspects of MPC and discuss how we can
extend MPC for better speech recognition.

##  2 Masked Predictive Coding

Masked Predictive Coding (MPC) [18] uses Masked Reconstruction objective to
perform predictive coding on Transformer based models. As depicted in Fig. 1,
the model structure of MPC is essentially the encoder part of Transformer
based speech recognition model plus a single fully-connected projection layer.
During training, masks are applied on input FBANK features before feeding into
the encoder. The training objective is L1 loss computed between masked input
FBANK features and projected encoder output at corresponding position. The
local smoothness of speech makes the task of predicting adjacent frames too
easy. So, in the current setup of MPC, we divide input features into chunks of
four frames and apply mask on chunks with a probability of 15%. Unlike BERT,
MPC adopted dynamic masking proposed in [24] where the masking pattern is
generated every time a sequence is fed to the model.

One unique characteristic of sequence-to-sequence with attention ASR models is
it usually applies downsampling in the encoder. Previous research showed
temporal pooling encourages effective encoding in different temporal
resolution and makes alignments in the decoding easier [25]. When temporal
downsampling is applied, with input feature $X$ $\in$ $\mathbb{R}^{t\times d}$
and temporal downsampling rate $r$, Transformer encoder projects the output of
last encoder layer to dimension $X_{0}$ $\in$ $\mathbb{R}^{t/r,d\times r}$. We
then reshapes it back to same shape as input feature for MPC loss computation.

Figure 1: Masked Predictive Coding with four-fold downsample

The overall training procedure of MPC consists of two stages, pre-training on
unsupervised data and fine-tuning on supervised data. In the pre-training
stage, MPC performs predictive coding directly on FBANK input and encoder
output. After the unsupervised pre-training procedure, we remove the
additional projection layer for predictive coding and add Transformer decoder
for fine-tuning on downstream ASR tasks. All model parameters are end-to-end
trainable in the fine-tuning stage. The work perhaps most similar to ours is
Mockingjay [15], which also employed Transformer encoder with Masked
Reconstruction loss. But their work mainly used pre-trained model as a feature
extractor while MPC works more like BERT and focuses on obtaining a good
parameter initialization. Also, unlike [15] and other previous work on
predictive coding [7, 12], our setup does not introduce any additional
parameters into speech recognition model.

##  3 Methods

###  3.1 MPC for streaming models

To apply MPC in streaming models, the Transformer encoder needs to be
restricted to only use information that has appeared before. Though some
previous work [26, 27] employed chunk-wise splitting for streaming models, in
this paper, we simply changed self-attention mask on Transformer encoder to
make the whole model stream-able. Specifically, we use a triangular matrix for
self-attention mask M in encoder, where the upper triangular part is set to
${-\infty}$, and the other elements to 0.

Recent work on APC [13] got impressive results on downstream tasks with
Transformer decoder backbone. Inspired by [28], we also propose to use a
unified training objective that combines MPC and APC. During training, with
probability p, we apply triangular matrix on Transformer encoder and use APC
objective, with probability 1 - p, we use the Transformer encoder as-is with
MPC objective. This parameter sharing framework has the advantage of making
the learned speech representations more general because they are jointly
optimized for different pre-training objectives where context is utilized in
different ways.

###  3.2 Knowledge transfer for MPC

For speech recognition task in a specific domain, its data distribution may be
a lot different from data used for MPC pre-training. Directly using MPC model
in fine-tuning stage might cause degradation in performance, even catastrophic
forgetting [29, 30]. To deal with this problem, we followed previous work [21,
23] and adopted target data adaption to fine-tune MPC model on data of target
task before the fine-tuning stage.

It is well-known that different layers of neural network capture different
types of information, and the transferability of different layers also varies
a lot [31, 32]. Previous work [21] made use of these findings by assigning
lower learning rates to layers that are more general for downstream tasks and
achieved promising results. To adapt their findings to MPC, we first used
probing task [33, 34] on pre-trained model to find out which layers of
Transformer encoder are more useful downstream speech recognition tasks. After
that, layer-wise discriminative training [21] is used to assign different
learning rates to each layer of Transformer encoder and adapt them to
different extents for better knowledge transfer.

Transfer Learning with single-step auxiliary loss [22] is yet another transfer
learning approach that adopted multi-task learning perspective via the
addition of pre-training objective in the fine-tuning stage. With the same
idea, we also added MPC loss in the fine-tuning stage. This way, the joint
loss in the fine-tuning stage became the weighted sum of task-specific loss
$L_{Attn}$, $L_{CTC}$ and auxiliary MPC loss $L_{MPC}$:

$L=\alpha_{attn}*L_{Attn}+\beta_{ctc}*L_{CTC}+\gamma_{mpc}*L_{MPC},$ (1)

For convenience, we name this method multi-task with MPC through out this
paper. Masked input and output for MPC are also added during training.

##  4 Experiments

###  4.1 Data

For Mandarin pre-training, we first collected reading type dataset
OpenMandarin as described in [18]. Note unlike [18], we did not include HKUST
and AISHELL in OpenMandarin dataset in this work because we found self pre-
trained MPC also improves performance. OpenMandarin contains about 1150 hours
of speech in total. To understand the impact of pre-training data speaking
style, our internal reading type dataset Didi Dictation and spontaneous type
dataset Didi Callcenter were also included. Didi Dictation is collected from
our internal mobile dictation application while Didi Callcenter is collected
from phone calls between our user and customer service staff. We randomly
selected 5000 hours of Didi Callcenter and 5000 hours of Didi Dictation for
MPC pre-training. For fair comparison with Open Mandarin, we also added
another pre-training dataset by randomly selecting 1150 hours of Didi
Callcenter and name it Didi Callcenter - 1K. For English pre-training data,
reading type dataset Librispeech [35] and spontaneous type dataset Fisher [36]
are used. To provide a fair comparison on data size, we created a new pre-
training dataset by randomly selecting only 960 hours of Fisher dataset and
name it Fisher - 1K. Detailed information of pre-training corpora is provided
in Table 1.

Fine-tuning experiments were conducted on HKUST, AISHELL-1 and Switchboard.
The speaking style of HKUST and Switchboard is spontaneous while the speaking
style of AISHELL is reading. Speed perturbation of 0.9 and 1.1 was used on the
training data. Mandarin characters are used as modeling units for HKUST and
AISHELL like described in [37], while 2000 BPE subwords [38] is used for
experiments on Switchboard.

The sample rate of HKUST and Switchboard is 8000 Hz, which is lower than some
pre-training data. To alleviate the possible influence, we kept the sample
rate of target dataset unchanged and downsampled pre-training data with higher
sample rate to 8000 Hz when needed.

###  4.2 Experimental setups

For Transformer based full-sequence models, we followed model structure of
previous work [39] with $e=12$, $d=6$, $d_{model}=256$, $d_{ff}=1280$ and
$d_{head}=4$. A source sequence X is first fed into a prenet consisting of
two-layer CNN with 256 channels, stride size 2 and kernel size 3 and
transformed to subsampled sequence $X_{0}$ $\in$ $\mathbb{R}^{n^{sub}\times
d^{attn}}$ before feeding into Transformer. For Transformer based streaming
models, two representative work with CTC [40] and transducer loss [41] are
slightly modified and used in this work. We used the same prenet and encoder
structure as Transformer based models. A two-layer Transformer decoder is used
as prediction network for transducer.

In pre-training stage, all models are trained with a total batch size of 256
for about 100 epochs. We used Adam optimizer and varied learning rate with
warmup schedule [42] according to the formula:

$lrate=k*d_{model}^{0.5}*min(n^{-0.5},n*warmup\\_n^{-1.5}),$ (2)

where $n$ is the step number. $k=0.5$ and $warmup\\_n=5000$ were chosen for
all pre-training experiments.

In the fine-tuning stage, all models are trained with a total batch size of
128. The same warmup schedule is used except we changed $n$ to 1.0 and
$warmup\\_n$ to 25000. For Transformer based models, the Attention-CTC multi-
task training objective is used, with weights determined on development set.
Models for HKUST and AISHELL is trained for 50 epochs while model for
Switchboard is trained for 100 epochs. Label smoothing of 0.1 and weight decay
of $1e^{-5}$ are applied for all fine-tuning experiments. SpecAugument [43] is
applied when training for Switchboard.

In the decoding stage, we selected 10 models with lowest error rates on the
validation set and averaged their parameters. Beam search with Attention-CTC
joint decoding and RNN language model [44] is used for Transformer based
models, with weights determined using grid search on development set. The beam
size is 10 for HKUST and AISHELL for Transformer based models and 20 for
Switchboard. WFST with word-level language model is used for decoding with
CTC.

We made our code publicly available for reproducibility at
https://github.com/athena-team/athena/tree/mpc_improvement.

Table 1: Details of pre-training corpora Datasets | Hours | Speaking Style
---|---|---
Open Mandarin | 1150 | Reading
Didi Callcenter - 1K | 1150 | Spontaneous
Didi Callcenter | 5000 | Spontaneous
Didi Dictation | 5000 | Reading
Librispeech | 960 | Reading
Fisher - 1K | 960 | Spontaneous

###  4.3 Effect of pre-training data speaking style

The results on HKUST and AISHELL with different pre-training data are listed
in Table 2. Our baseline result without MPC matches the strong baseline in
[39]. Comparing relative error reduction of HKUST with same amounts of pre-
training data, it is obvious MPC models pre-trained with matching speaking
style data achieved lower error rates for downstream tasks.

Open Mandarin and Didi Dictation are both used in pre-training for HKUST and
AISHELL. From Table 2, we can also find the relative error reduction using the
same pre-training data is bigger with matching fine-tuning data.
Interestingly, the error reduction Didi Callcenter - 1K brings on HKUST is
even bigger than Didi Dictation, which suggests speaking style has a bigger
impact than pre-training data size in some cases.

Table 2: Character Error Rates(%) and Relative Error Rates Reduction(%) on HKUST and AISHELL test set Task | Pre-training Data | Hours | CER | RERR
---|---|---|---|---
HKUST | - | - | 23.3 | -
HKUST | 170 | 22.8 | 2.14
Open Mandarin (8k) | 1150 | 22.7 | 2.58
Didi Callcenter - 1K | 1150 | 22.0 | 5.58
Didi Callcenter | 5000 | 21.5 | 7.73
Didi Dictation (8k) | 5000 | 22.1 | 5.15
AISHELL | - | - | 6.82 | -
AISHELL | 178 | 6.61 | 3.07
Open Mandarin | 1150 | 6.38 | 6.45
Didi Dictation | 5000 | 6.26 | 8.21
Table 3: Word Error Rates(%) and Relative Error Rates Reduction(%) on Switchboard and CallHome test set Pre-training Data | Hours | WER | RERR
---|---|---|---
SWB | CH | SWB | CH
- | - | 8.8 | 17.8 | - | -
Switchboard | 260 | 8.5 | 17.4 | 3.41 | 2.25
Fisher - 1K | 960 | 8.0 | 16.2 | 9.09 | 8.99
Librispeech (8k) | 960 | 8.4 | 17.2 | 4.55 | 3.37

Experiments were also conducted on English ASR database Switchboard. Our
baseline results on Switchboard is also on par with previous work [39]. As
shown in Table 3, Switchboard got better results with spontaneous type pre-
training data Fisher - 1K than with Librispeech, which further confirms our
findings on speaking style.

###  4.4 MPC for streaming model

We first tested the effect of directly initializing streaming model with MPC.
As shown in Table 4, pre-trained MPC brings consistent gains on streaming
models for HKUST and AISHELL. MPC model pre-trained on more data obtained
better results on streaming models, which is also in line with the conclusion
for full-sequence models. Though not specifically optimized, MPC pre-training
is also useful for streaming models, which suggests the parameter
initialization obtained by MPC is helpful for both uni-directional and
bidirectional models.

Table 4: Character Error Rates(%) and Relative Error Rates Reduction(%) for uni-directional CTC and uni-directional RNN-T with pre-trained MPC Task | Model | Pre-training Data | CER | RERR
---|---|---|---|---
HKUST | CTC | - | 29.3 | -
Didi Callcenter - 1K | 28.0 | 4.56
Didi Callcenter | 27.6 | 5.80
RNN-T | - | 28.1 | -
Didi Callcenter - 1K | 26.9 | 4.43
Didi Callcenter | 26.6 | 5.20
AISHELL | CTC | - | 9.91 | -
Open Mandarin | 9.42 | 4.91
RNN-T | - | 9.43 | -
Open Mandarin | 8.88 | 5.83

To further improve performance of MPC on streaming models, we tried to combine
APC with MPC in pre-training stage. Previous work on APC showed a future
prediction step of 5 gave best results on Transformer decoder [13]. We
implemented APC with the same future prediction step and set switching
probability p to 0.5. Experiments were conducted on HKUST dataset with
Transformer CTC model and the results of different pre-training objectives are
presented in Table 5. When used alone, APC and MPC got similar improvements.
Combining APC with MPC results in consistent gains over them, which echoes
with the findings in [28].

Table 5: MPC + APC is the model pre-trained with APC 50% of time and MPC 50% of time. Relative Error Rates Reduction(%) is calculated with HKUST baseline without MPC Pre-training Data | Objective | CER | RERR
---|---|---|---
Didi Callcenter - 1K | MPC | 28.0 | 4.56
APC | 27.8 | 5.12
MPC + APC | 27.2 | 7.32
Didi Callcenter | MPC | 27.6 | 5.80
APC | 27.9 | 4.72
MPC + APC | 26.8 | 8.46

###  4.5 Knowledge transfer for MPC

Each layer of pre-trained MPC encoder captures different features of the input
speech. To get a quantitative measure of how important layer l is, for each
layer, we create a new Transformer with l layers of encoder and 6 layers of
decoder. The encoder for the new Transformer is frozen and initialized from
pre-trained MPC model while the decoder is trainable. As shown in Fig 2,
features from the middle layers of pre-trained Transformer encoder are
generally more helpful than features from top and bottom layers. In the fine-
tuning stage, we propose to set the learning rates of encoder layers
discriminatively by multiplying the learning rate of layer l with
${\lambda}^{\mathinner{\\!\left\lVert\textit{l}-{\theta}\right\rVert}}$, where
$\lambda$ $\in$ (0, 1). To fit the accuracy curve above, we set $\lambda$ to
be 0.95 and $\theta$ to be 5.5. This way, the parameters of middle layers are
updated more slowly and knowledge from pre-trained MPC model is best retained.
Different $\lambda$ and $\theta$ values have also been tested and no
significant difference is observed. Using this schedule on baseline systems
without MPC doesn’t lead to better accuracy either.

Figure 2: CER(%) with each layer of pre-trained Transformer encoder. Results
on HKUST is pre-trained with Didi Callcenter and results on AISHELL is pre-
trained with Open Mandarin

The results for target data adaptation and layer-wise discriminative training
are listed in Table 6. Two baseline results we used are HKUST pre-trained with
Didi Callcenter and AISHELL pre-trained with Open Mandarin. Using target data
adaption or layer-wise discriminative training alone doesn’t help the
knowledge transfer of MPC very much. But when combined together, they provide
consistent gains on downstream tasks.

For multi-task with MPC, the MPC loss should contribute more to the joint loss
to facilitate knowledge transfer in the first few epochs. As training
proceeds, the task-specific component of the loss function becomes more
important and $\gamma_{mpc}$ should be decreased. In this work, we empirically
found it work well to set initial $\gamma_{mpc}$ to 0.2 and decrease it by
half every 5 epochs. The result in Table 6 shows multi-task with MPC gets
slight improvements over baseline.

Table 6: Results on HKUST and AISHELL with different knowledge transfer methods. HKUST is pre-trained with Didi Callcenter. AISHELL is pre-trained with Didi Dictation. Target Data + Layer-wise is the combination of target data adaption and layer-wise discriminative training Task | Knowledge Transfer Methods | CER | RERR
---|---|---|---
HKUST | - | 21.5 | -
Target Data Adaption | 21.2 | 1.40
Layer-wise Discriminative | 21.3 | 0.93
Target Data + Layer-wise | 20.8 | 3.26
Multi-Task MPC | 21.1 | 1.86
AISHELL | - | 6.26 | -
Target Data Adaption | 6.23 | 0.48
Layer-wise Discriminative | 6.21 | 0.80
Target Data + Layer-wise | 6.01 | 3.99
Multi-Task MPC | 6.13 | 2.08

##  5 Conclusion

In this work, we investigated three important aspects of MPC. Pre-training
data with matching speaking style was found to be more useful on downstream
recognition tasks. Using MPC directly on streaming models helps, but combining
MPC with APC brings further improvements on streaming models. Also, the
combination of target data adaption and layer-wise discriminative training
provides consistent gains on knowledge transfer to downstream tasks.

## References

  * [1] C. Doersch, A. Gupta, and A. Efros, “Unsupervised visual representation learning by context prediction,” in _ICCV_ , 2015, pp. 1422–1430.
  * [2] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in _NAACL-HLT (1)_ , 2019, pp. 4171–4186.
  * [3] O. A. van den, Y. Li, and V. Oriol, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018.
  * [4] A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” _arXiv preprint arXiv:1910.05453_ , 2019\.
  * [5] A. Baevski, M. Auli, and A. rahman Mohamed, “Effectiveness of self-supervised pre-training for speech recognition,” _arXiv preprint arXiv:1911.03912_ , 2019.
  * [6] R. Mirco and B. Yoshua, “Learning speaker representations with mutual information,” _Interspeech_ , Sep 2019.
  * [7] S. Steffen, B. Alexei, C. Ronan, and A. Michael, “wav2vec: Unsupervised pre-training for speech recognition,” _Interspeech_ , Sep 2019.
  * [8] P. Santiago, R. Mirco, S. Joan, B. Antonio, and et al, “Learning problem-agnostic speech representations from multiple self-supervised tasks,” _Interspeech_ , Sep 2019.
  * [9] K. Kawakami, L. Wang, C. Dyer, P. Blunsom, and A. van den Oord, “Learning robust and multilingual speech representations,” _arXiv preprint arXiv:2001.11128_ , 2020.
  * [10] M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux, “Unsupervised pretraining transfers well across languages,” _arXiv preprint arXiv:2002.02848_ , 2020.
  * [11] Z. Lian, Y. Li, J.Tao, and J. Huang, “Improving speech emotion recognition via transformer-based predictive coding through transfer learning,” _arXiv preprint arXiv:1811.07691_ , 2018.
  * [12] C. Yu-An, H. Wei-Ning, T. Hao, and G. James, “An unsupervised autoregressive model for speech representation learning,” _Interspeech_ , Sep 2019.
  * [13] Y.-A. Chung and J. Glass, “Generative pre-training for speech with autoregressive predictive coding,” _arXiv preprint arXiv:1910.12607_ , 2019\.
  * [14] C. Yu-An and G. James, “Improved speech representations with multi-target autoregressive predictive coding,” _arXiv preprint arXiv:2004.05274_ , 2020\.
  * [15] A. T. Liu, S. Yang, P.-H. Chi, P.-C. Hsu, and et al, “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,” _arXiv preprint arXiv:1910.12638_ , 2019.
  * [16] W. Wang, Q. Tang, and K. Livescu, “Unsupervised pre-training of bidirectional speech encoders via masked reconstruction,” _arXiv preprint arXiv:2001.10603_ , 2020.
  * [17] X. Song, G. Wang, Z. Wu, Y. Huang, and et al, “Speech-xlnet: Unsupervised acoustic model pretraining for self-attention networks,” _arXiv preprint arXiv:1910.10387_ , 2019.
  * [18] D. Jiang, X. Lei, W. Li, N. Luo, and et al, “Improving transformer-based speech recognition using unsupervised pre-training,” _arXiv preprint arXiv:1910.09932_ , 2019.
  * [19] B. Mohamed, D. Renato, D. Olivier, D. Stephane, and et al, “Automatic speech recognition and speech variability: A review,” _Speech communication_ , vol. 49, no. 10-11, pp. 763–786, 2007.
  * [20] W. Mitch, T. Kelsey, H. Kate, and S. Amy, “Effect of speaking style on lvcsr performance,” in _Proc. ICSLP_ , vol. 96, 1996.
  * [21] J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,” in _ACL_ , 2018.
  * [22] A. Chronopoulou, C. Baziotis, and A. Potamianos, “An embarrassingly simple approach for transfer learning from pretrained language models,” _Proceedings of the 2019 Conference of the North_ , 2019.
  * [23] C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune bert for text classification?” _arXiv preprint arXiv:1905.05583_ , 2019.
  * [24] Y. Liu, O. Myle, G. Naman, J. Du, and et al, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.
  * [25] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in _ICASSP_ , 2016, pp. 4960–4964.
  * [26] T. N. Sainath, C.-C. Chiu, R. Prabhavalkar, A. Kannan, and et al, “Improving the performance of online neural transducer models,” in _ICASSP_ , 2018, pp. 5864–5868.
  * [27] H. Miao, G. Cheng, C. Gao, P. Zhang, and Y. Yan, “Transformer-based online ctc/attention end-to-end speech recognition architecture,” _arXiv preprint arXiv:2001.08290_ , 2020.
  * [28] L. Dong, N. Yang, W. Wang, F. Wei, and et al, “Unified language model pre-training for natural language understanding and generation,” in _NeurIPS_ , 2019.
  * [29] I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and et al, “An empirical investigation of catastrophic forgetting in gradient-based neural networks,” _arXiv preprint arXiv:1312.6211_ , 2013.
  * [30] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, and et al., “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the National Academy of Sciences_ , vol. 114, no. 13, p. 3521–3526, Mar 2017.
  * [31] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” in _NIPS_ , 2014.
  * [32] N. F. Liu, M. Gardner, Y. Belinkov, M. E. Peters, and et al, “Linguistic knowledge and transferability of contextual representations,” _arXiv preprint arXiv:1903.08855_ , 2019.
  * [33] X. Shi, I. Padhi, and K. Knight, “Does string-based neural mt learn source syntax?” in _EMNLP_ , 2016.
  * [34] Y. Adi, E. Kermany, Y. Belinkov, O. Lavi, and et al, “Fine-grained analysis of sentence embeddings using auxiliary prediction tasks,” _arXiv preprint arXiv:1608.04207_ , 2017.
  * [35] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An asr corpus based on public domain audio books,” in _ICASSP_ , 2015, pp. 5206–5210.
  * [36] C. Cieri, D. Miller, and K. Walker, “The fisher corpus: a resource for the next generations of speech-to-text,” in _LREC_ , 2004.
  * [37] W. Zou, D. Jiang, S. Zhao, G. Yang, and et al, “Comparable study of modeling units for end-to-end mandarin speech recognition,” in _ISCSLP_ , 2018, pp. 369–373.
  * [38] T. Zenkel, R. Sanabria, F. Metze, and A. H. Waibel, “Subword and crossword units for ctc acoustic models,” in _Interspeech_ , 2018.
  * [39] K. Shigeki, C. Nanxin, H. Tomoki, H. Takaaki, and et al, “A comparative study on transformer vs rnn in speech applications,” _arXiv preprint arXiv:1909.06317_ , 2019.
  * [40] J. Salazar, K. Kirchhoff, and Z. Huang, “Self-attention networks for connectionist temporal classification in speech recognition,” in _ICASSP_ , 2019, pp. 7115–7119.
  * [41] C.-F. Yeh, J. Mahadeokar, K. Kalgaonkar, Y. Wang, and et al, “Transformer-transducer: End-to-end speech recognition with self-attention,” _arXiv preprint arXiv:1910.12977_ , 2019.
  * [42] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, and et al, “Attention is all you need,” in _NIPS_ , 2017, pp. 5998–6008.
  * [43] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, and et al, “Specaugment: A simple data augmentation method for automatic speech recognition,” in _Interspeech_ , 2019.
  * [44] K. Suyoun, H. Takaaki, and W. Shinji, “Joint ctc-attention based end-to-end speech recognition using multi-task learning,” in _ICASSP_ , Mar 2017.