# Measurements of nematic susceptibility with phase sensitive nuclear magnetic
resonance in pulsed strain fields

C. Chaffey  Department of Physics and Astronomy, University of California
Davis, Davis, CA  C. Williams  Department of Physics and Astronomy, University
of California Davis, Davis, CA  M. A. Tanatar  S. L. Bud’ko  P. C. Canfield
Ames Laboratory U.S. DOE and Department of Physics and Astronomy,Iowa State
University, Ames, Iowa 50011, USA  N. J. Curro<EMAIL_ADDRESS>Department
of Physics and Astronomy, University of California Davis, Davis, CA

###### Abstract

We present nuclear magnetic resonance data in BaFe2As2 in the presence of
pulsed strain fields that are interleaved in time with the radiofrequency
excitation pulses. In this approach, the precessing nuclear magnetization
acquires a phase shift that is proportional to the strain and pulse time. The
sensitivity in this approach is limited by the homogeneous decoherence time,
$T_{2}$, rather than the inhomogeneous linewidth. We measure the nematic
susceptibility as a function of temperature, and demonstrate three orders of
magnitude improvement in sensitivity. This approach will enable studies of the
strain response in a broad range of materials that previously were
inaccessible due to inhomogeneous broadening.

##  I Introduction

In recent years there has been growing attention to the role of electronic
nematicity in strongly correlated electron materials, where the electronic
degrees of freedom spontaneously break an underlying discrete symmetry of the
lattice [1]. This phenomenon is exemplified in the parent compounds of many
iron-based superconductors, where the the Fe $3d$ orbitals undergo ferro-
orbital ordering, accompanied by a tetragonal to orthorhombic structural
transition and often the onset of long-range antiferromagnetic order [2, 3].
Below this temperature in the nematic, or orthorhombic, phase, the Fe $d_{xz}$
and $d_{yz}$ orbitals become nondegenerate, with an energy splitting on the
order of 40 meV, and different occupation levels [4]. Doping tends to suppress
the long-range nematic order and stabilizes unconventional superconductivity.
Elastoresistance measurements of the nematic susceptibility [5, 6, 7] revealed
the presence of a nematic quantum phase critical point that may be responsible
for driving the superconductivity in this system [8, 9, 10, 11].

Several techniques have been developed to probe the nematic degrees of
freedom. Anisotropic resistivity [12, 13], elastoresistance [5], elastic
constants [14, 15, 16, 17], thermopower [18], electronic Raman scattering
[19], polarized light image color analysis [20, 21] and optical conductivity
[22] probe bulk anisotropies. Nuclear magnetic resonance (NMR) and neutron
scattering have been utilized to investigate the effect of nematicity on the
spin fluctuations [23, 24, 25, 26, 27]. NMR studies of quadrupolar nuclei are
sensitive to charge degrees of freedom, and have also been utilized to study
both nematic fluctuations [28], and changes to the Fe $3d$ orbital occupations
in response to external strain fields [29]. The latter provides a direct
microscopic measure of the nematic susceptibility, and has the advantage that
it can be measured in the superconducting state, where other measurements such
as elastoresistance are unable to operate.

NMR measurements of the nematic susceptibility probe the response of the NMR
resonance frequencies as a function of applied strain. Strain can couple to
the nuclear spins via either the Knight shift or the electric field gradient
(EFG) tensors. These quantities inherit the point group symmetry of the
lattice, and if this symmetry is lowered by strain, then these tensors can
acquire new asymmetries or off-diagonal components. Note that the Knight shift
tensor depends on both the hyperfine coupling and the electronic spin
susceptibility, and both could change in response to strain. In the case of
FeSe only the latter appears to become anisotropic in the nematic phase [30].
In BaFe2As2, evidence to date says that the Knight shift tensor does not
respond to strain, although the EFG tensor changes dramatically. The NMR
resonance frequencies depend sensitively on these parameters, and thus one can
probe the nematic susceptibility of a material by observing the linear
response of the resonance frequency to strain.

A straightforward approach to measuring the nematic suscepibility with NMR is
to observe a shift in the spectrum while applying a static strain field.
However in order to detect a response, the frequency shift must be on the
order of the spectral linewidth, $(T_{2}^{*})^{-1}$, which often is
inhomogeneously broadened. This quantity can exceed the intrinsic decoherence
rate, $T_{2}^{-1}$, by orders of magnitude, especially in the presence of
doping which creates an inhomogeneous distribution of local EFGs. In such
cases the sensitivity of the susceptibility measurements can be severely
limited. On the other hand, if the strain field is pulsed while the nuclear
spins are coherently precessing, then the shift of the resonance frequency
will alter the phase of the precessing magnetization and can be observed via
quadrature detection. This process forms the basis of “spin-warp” imaging
widely used in magnetic resonance imaging, in which a magnetic field gradient
is pulsed during the evolution times of a spin echo sequence [31, 32]. In this
case it is $T_{2}^{-1}$, rather than the inhomogeneous linewidth, that limits
the sensitivity of the response measurement. A pulsed-strain approach should
thus enable measurements in systems with inhomogeneous broadening or
manifesting smaller intrinsic suscepetibilities. Indeed, strain pulses were
used to study the strain response of color centers in silicon carbide using
optically detected magnetic resonance [33].

The approach outlined here is similar to AC strain measurements that have been
utilized to measure elastoresistance and elastocaloric effects [34, 35], but
an important difference is that the strain fields we apply are pulsed rather
than sinusoidal. In this case the response function is a convolution of the
frequency dependent nematic susceptibility with the square-wave driving
voltage, giving rise to an exponential rise or decay of the nematicity with
time constant $\tau_{nem}$. This time scale is determined by the electronic
degrees of freedom, thus $\tau_{nem}\sim 10^{-9}$ s. Another important time
scale, $\tau_{s}$, is determined by the speed of sound as the strain pulse
traverses the length of the sample, which we estimate to be $\sim 10^{-7}$ s.
On the other hand, the shortest time scale in our measurements is at least a
microsecond, which is larger than either $\tau_{nem}$ or $\tau_{s}$, therefore
we are operating in the quasi-static limit.

##  II Experiments

In order to examine the feasibility of a pulsed-strain approach to measuring
the nematic susceptibility, we studied the response in a well-characterized
system, BaFe2As2 [2]. A single crystal grown by self-flux was cut to
dimensions $\sim 1.6$ mm long by $\sim 0.3$ mm wide, with the long axis
parallel to the (110) direction (in the tetragonal basis), and mounted to a
commercial piezoelectric strain cell (CS100, Razorbill Instruments) [36]. The
crystal was mounted using epoxy (UHU plus endfest 300) in order to orient the
magnetic field in-plane, as shown in Fig. 1 and described in [37]. Uniaxial
stress was applied along the long-axis by piezoelectric stacks giving rise to
a strain $\varepsilon=(\varepsilon_{xx}-\varepsilon_{yy})/2$, with $B_{2g}$
symmetry. Because the Poisson ratio is non-zero, there will also be strain
fields with $A_{1g}$ symmetry, however the nematic order in this material
couples primarily to the $B_{2g}$ channel and will dominate the response [38,
6]. The static sample length displacement, $\Delta L$ was measured by a
capacitive dilatometer, and strain, $\varepsilon=\Delta L/L_{0}$, was
determined based an unstressed length of $L_{0}=0.55$ mm at room temperature.

###  II.1 Pulsed Strain

Figure 1:  (Upper panel) Circuit diagram illustrating the creating of the
strain pulses. TTL pulses control analog switches with DC voltages set to $\pm
V_{P}$, which control the inner stack of the CS100 strain device. The outer
stack is held at constant displacement using the PID feedback control and the
capacitive dilatometer. (Lower panel) Measured voltage across the inner stack
as a function of time for three different values of $V_{P}$ (corresponding
strain values shown on the right side axis). The light orange vertical bands
indicate the NMR radiofrequency pulses, and the light blue vertical band
indicates the position of the echo. The lower inset shows the crystal mounted
in the device with the surrounding coil, and the direction of the applied
field. The upper insert shows a blow up of the mounted crystal.

Th strain device utilizes two sets of piezos, the inner and the outer stacks,
as illustrated in Fig. 1. In order to implement the strain pulses, we used the
outer stack to hold a static strain field, while applying voltage pulses to
the inner stack. These strain pulses were created by utilizing TTL pulses from
an NMR spectrometer to drive analog switches (CD4066BE, Texas Instruments)
that were timed to turn on immediately following the $\pi/2$ and $\pi$
radiofrequency pulses, as illustrated as in Fig. 2. The time dependent voltage
across the inner stacks is shown in the lower panel of Fig. 1. The resistance
$R=100$ $\Omega$ shown in Fig. 1 is in parallel with the piezo stack, which
has an intrinsic capacitance ($\sim 0.2$ $\mu$F) and was chosen to provide a
rise time that is shorter than $T_{2}$, so that the strain pulse can be
applied between the NMR radiofrequency pulses.

Static displacements are measured via a capacitance bridge, however the time
constant for the bridge precludes any measurements during a short strain
pulse. We estimate the strain during the pulses using a temperature dependent
calibration factor, $\alpha=\varepsilon/V_{P}$, shown in Fig. 3(c), which was
determined by measurements of the displacement during static applied voltage
levels. This approach was utilized in previous AC strain measurements [34,
35], and can lead to overestimates of the strain applied due to the frequency
dependence of the calibration factor. As discussed below, the NMR response
with pulsed and static strain give similar values, suggesting that the
calibration approach is reasonable. The dynamic strain can also give rise to
DC heating effects of the strain device on the order of a few Kelvin at finite
frequencies, depending on the magnitude of the driving voltage, because the
large capacitances and low thermal conductivity of the piezo stacks [34]. To
minimize such effects we operated with low driving voltages ($\leq 2$ V).

Figure 2:  Strain pulses are applied during the times of free precession,
between the two radiofrequency pulses. The phase of the echo in the rotating
frame will change by an amount that is proportional to the strain pulse width,
$t_{p}$, times the strain pulse magnitude, $V_{p}$. $t_{P}$ was varied between
200 and 670 $\mu$s. The pulse length for a 90 degree pulse was 2.5 $\mu$s, and
$V_{P}$ varied from -2 to +2 V.

A potential issue with the application of voltage pulses to the strain cell is
that natural vibration resonances could be excited in the device. We observed
resonances in the range of $\sim 2-30$ kHz for our device suggesting that time
scales faster than 1 ms could be problematic. In practice, however, the low
values of $V_{P}$ we utilized did not appear to introduce any oscillations
measured in $V_{inner}$, as shown in Fig. 1. We speculate that low mass of the
strain cell ensures that these resonances lie outside the bandwidths of our
pulses.

Figure 3:  (a) Quadrupolar splitting $\nu_{yy}=|f_{+}-f_{-}|/2$ as a function
of temperature, compared with that in a free-standing sample (from [39].) (b)
Residual strain versus temperature, based on $V_{yy}$ and using the nematic
susceptibility reported in [29]. (c) The strain calibration factor,
$\alpha=d\varepsilon/dV$, versus temperature as discussed in the text.

###  II.2 NMR phase sensing

The goal of the experiments is to probe the strain response of the spin
$I=3/2$ 75As nuclei in this material. The resonance frequencies are determined
by the eigenvalues of the nuclear spin Hamiltonian:

$\mathcal{H}=\gamma\hbar{\mathbf{I}}\cdot(1+\mathbf{K})\cdot\mathbf{H}_{0}+\frac{h\nu_{zz}}{6}[3{I}_{z}^{2}-\hat{I}^{2}+\eta\left(\hat{I}_{x}^{2}-{I}_{y}^{2}\right)]$
(1)

where $\gamma=0.7292$ kHz/G is the gyromagnetic ratio, ${I}_{\alpha}$ are the
nuclear spin operators, $\mathbf{K}$ is the Knight shift tensor, $\nu_{zz}$ is
the largest eigenvalue of the electric field gradient (EFG) tensor, and
$\eta=(\nu_{yy}-\nu_{xx})/(\nu_{xx}+\nu_{yy})$ is the asymmetry parameter. To
second order in perturbation theory, the resonance frequencies of the
satellites and central transitions for $c\perp\mathbf{H}_{0}$ are given by:

$\displaystyle f_{\pm}$ $\displaystyle=$ $\displaystyle\gamma
H_{0}(1+K_{\perp}+\Delta K)\mp\frac{\eta+1}{2}\nu_{zz}$ (2) $\displaystyle
f_{0}$ $\displaystyle=$ $\displaystyle\gamma H_{0}(1+K_{\perp}+\Delta
K)+\frac{(\eta-3)^{2}\nu_{zz}^{2}}{48\gamma H_{0}}$

where $K_{\perp}=(K_{xx}+K_{yy})/2$ and $\Delta K=K_{yy}-K_{xx}$. If any of
the three parameters $\eta$, $\nu_{zz}$ or $\Delta K$ changes in response to
strain, then there will be shifts in $f_{0}$ and $f_{\pm}$.

Figure 4:  Normalized spectra versus frequency for the (a) upper, (b) central
and (c) lower transitions of the 75As. Solid and dotted lines correspond to
the projection along the $x$ and $y$ axes in the rotating frame. The frequency
axes have been offset by the corresponding resonance frequencies, and the
spectra have been offset vertically by the value of $V_{p}$. Plots (d), (e)
and (f) show the same data in which $S_{y}$ is plotted versus $S_{x}$, with
frequency as an implicit parameter. The color scale corresponds to $V_{p}$ and
is the same for all plots. (g) Echo phase over strain pulse time (in cycles
per second) versus applied strain at 140 K. The dotted lines are linear fits
using least orthogonal distance method, with slopes $\partial
f_{-}/\partial\varepsilon=78\pm 8$ MHz/strain, $\partial
f_{0}/\partial\varepsilon=-1.5\pm 0.1$ MHz/strain, and $\partial
f_{+}/\partial\varepsilon=-79\pm 8$ MHz/strain.

For the pulse sequence shown in Fig. 2, the $\pi/2$ pulse creates a
magnetization that precesses in the plane perpendicular to $\mathbf{H}_{0}$ at
the resonance frequency of the particular transition. In the absence of any
perturbing strain pulses, the magnetization would remain static in the
rotating frame during the evolution time, $\tau$, between the $\pi/2$ and
$\pi$ pulses. However, if a strain pulse during this period shifts the
resonance frequency by an amount, $\Delta f$, then the magnetization
accumulates an extra phase that is proportional to the product of $\Delta f$
and $t_{P}$, the duration of the strain pulse. The second strain pulse is
inverted so that phase accumulation continues even though the direction of
precession in the rotating frame is reversed after the $\pi$ pulse. The end
result is that the echo acquires a net phase $\theta=2\Delta ft_{p}$. This
phenomenon is illustrated in Fig. 4, where spectra are shown for both
quadrature channels, $S_{x}$ and $S_{y}$, for all three transitions of the As.
The phase rotation is evident in the changes of the relative intensities of
$S_{x}$ and $S_{y}$ components as a function of frequency as the magnitude of
the piezo voltage, $V_{p}$, changes, as well as in plots of $S_{x}$ versus
$S_{y}$ in the lower panels. The effect is greatest for the two quadrupolar
satellites, and smaller for the central transition. Moreover, the lower
satellite rotates counterclockwise ($\theta$ increases with $V_{p}$), whereas
the upper and central transition rotate clockwise ($\theta$ decreases with
$V_{p}$). This behavior is summarized in Fig. 4(g), which shows $\Delta
f=\theta/(2t_{p})$ as a function of $V_{p}$. Here we determine the angle
$\theta=\arctan(I_{y}/I_{x})$ (in cycles) where $I_{x,y}$ are the integrated
intensities of the spectra $S_{x,y}$.

We confirmed the behavior of the echo in response to strain pulses via
numerical simulations of the full time-dependent Schrodinger equation using
the QuTIP package [40, 41] with realistic pulse widths similar to those used
in experiment, and modeling strain as a time dependent $\eta$. The simulated
echo phase exhibit behavior nearly identical to the experimental observations,
and justifies the use of perturbation theory to express the frequencies of the
transitions in Eq. II.2.

###  II.3 Susceptibility

There are three distinct response functions that can give rise to the behavior
in Fig. 4: $\chi_{\eta}=\partial\eta/\partial\varepsilon$,
$\chi_{K}=\partial\Delta K/\partial\varepsilon$, and
$\chi_{\nu}=\partial\nu_{zz}/\partial\varepsilon$. The linear response of the
resonance frequencies are then given by ${\partial
f_{i}}/{\partial\varepsilon}=\sum_{j}M_{ij}\chi_{j}$, where
$M_{ij}=\frac{\partial f_{i}}{\partial x_{j}}$ evaluated at zero strain and
$x_{j}=\eta$, $\Delta K$ or $\nu_{zz}$. By measuring $f_{0}$ and $f_{\pm}$, we
can disentangle the strain susceptibility of each parameter individually:
$\chi_{j}=\sum_{j}{M}^{-1}_{ij}{\partial f_{i}}/{\partial\varepsilon}$. Using
the fitted slopes in Fig. 4(g), we find

$\displaystyle\chi_{\eta}$ $\displaystyle=$ $\displaystyle 92\pm
2/\textrm{strain}$ $\displaystyle\chi_{\nu}$ $\displaystyle=$
$\displaystyle-54\pm 1\textrm{MHz/strain}$ $\displaystyle\chi_{K}$
$\displaystyle=$ $\displaystyle-0.336\pm 0.004\textrm{\%/strain}$

at 140 K. The value of $\chi_{\eta}$ is comparable to that reported previously
[24], however no change to the Knight shift has been previously observed. The
value we observe for $\chi_{K}$ is too small to detect via static strain
techniques: for a maximum static strain $\varepsilon=10^{-3}$, $\Delta K$
corresponds to a shift of $\sim 0.5$ kHz, which is below the static linewidth
of $\sim 1.2$ kHz.

Figure 5:  (a) Frequency shift of the lower satellite versus strain for
several different temperatures. Solid lines are linear fits to the data. (b)
Nematic susceptibility, $\chi_{\eta}$, as a function of temperature. The solid
and dashed lines are fits with and without the residual strain. The inset
shows the temperature dependence of the inverse of the susceptibility, as well
as the corresponding fits.

The nonzero response $\chi_{\nu}$ is surprising, because $\nu_{zz}$ should
only couple quadratically to strain with $B_{2g}$ symmetry. We speculate that
the linear response we observe reflects a finite $\varepsilon_{zz}$ strain
field that is induced by the applied stress because of the finite Poisson
ratio of our crystal. It is reasonable that $\chi_{\nu}<0$ because $\nu_{zz}$
decreases with increasing $c$-axis length in the AFe2As2 (A = Ca, Ba, Sr)
family [42].

Note that $\chi_{\eta}$ is the dominant source of the shift of the satellite
resonance frequencies (Eq. II.2). Therefore it is possible to approximate
$\chi_{\eta}\approx 2(\partial f_{-}/\partial\varepsilon)/\nu_{zz}(0)$, rather
than measure the response of all three transitions at each temperature. Fig.
5(a) displays the response of the lower transition for a series of
temperatures, and Fig. 5(b) displays the temperature dependence of
$\chi_{\eta}$.

For these experiments the sample displacement was held constant as the
temperature was lowered. A consequence of this condition is that thermal
contraction of the crystal gives rise to a positive residual strain. We
estimated this residual strain by observing the EFG splitting of the satellite
resonances in the absence of strain pulses. As shown in Fig. 3(a), the EFG
splitting, $\nu_{yy}$, is temperature dependent and larger than in an the
unstrained case. The residual strain can be determined using the nematic
susceptibility measured previously using static strain [29]:
$\varepsilon_{res}=(2|\nu_{yy}|/\nu_{zz}-1)/(\chi_{0}+C/(T-T_{0}))$, and is
shown in Fig. 3(b) as a function of temperature using parameters reported
previously. The peak at 150 K is due to a software error, in which PID control
was temporarily lost.

The residual strain field will tend to suppress the true nematic
susceptibility and can alter the temperature dependence of $\chi_{\eta}$. To
model this behavior, we assume that the nematic order parameter, $\phi$, can
be described by a Ginzburg-Landau theory, and find the mean-field solution in
the presence of a finite strain field. Expanding the differential
susceptibility as a series in $\varepsilon_{res}$ gives:

$\left[\frac{\partial\phi}{\partial\varepsilon}\right]_{\varepsilon=\varepsilon_{res}}=\frac{C}{T-T_{0}}-\frac{C_{2}^{3}\varepsilon_{res}^{2}}{(T-T_{0})^{3}}.$
(3)

The solid line in Fig. 5(b) is a fit to this expression with $C=(1.93\pm
0.24)\times 10^{3}$ K and $C_{2}=(5.3\pm 3.1)\times 10^{3}$ K and $T_{0}=116$
K. The dotted line shows the result with $C_{2}=0$, which should be the
response in the absence of $\varepsilon_{res}$. The value of $C$ is
approximately 40% of that reported previously [29]. The reason for reduction
could be errors in the strain-to-voltage calibration factor, $\alpha$. In
previous studies using a strain gauge to estimate $\alpha$, this quantity was
observed to decrease by up to 8% at frequencies up to 10 kHz [34], therefore
it is possible that the strain levels reported here are overestimated, and the
magnitude of the nematic susceptibility is underestimated.

##  III Discussion

Figure 6 compares the NMR frequency shifts measured with pulsed strain and
static strain. Despite several orders of magnitude difference in the strain,
the slopes are approximately the same for the different temperatures. This
result confirms that our approach using the calibration factor $\alpha$ for
pulsed strain captures the nematic susceptibility reasonably well, despite
utilizing strain levels that are 2-3 orders of magnitude smaller.

Figure 6:  Comparison of the frequency offset, $|\Delta f|$, versus strain,
$\varepsilon$, measured by static ($\blacklozenge$, reproduced from [29]) and
pulsed ($\bullet$) techniques. The pink and green lines correspond to
$T_{2*}^{-1}$ and $T_{2}^{-1}$ for BaFe2As2, respectively, and the blue line
corresponds to the minimum detection level, as discussed in the text.

The green and red horizontal lines in Fig. 6 indicate the homogeneous and
inhomogeneous linewidths in this material, which set the scale for the
sensitivity limits for the measured frequency shift, $\Delta f$. It is clear
that the pulsed strain approach enables measurements of the nematic
susceptibility with much smaller levels of applied strain by taking advantage
of the fact that $T_{2}^{-1}\ll(T_{2}^{*})^{-1}$. For the 75As, inhomogeneous
strain in the crystal due to imperfections or dopants gives rise to
inhomogeneous broadening via the nuclear quadrupole interaction. Although it
is possible to detect a frequency shift that is smaller than the linewidth,
such an approach requires careful measurements of the full spectrum. On the
other hand, the minimum detectable frequency shift using pulsed strain is
determined by the precision of the phase angle of the echo. The minimum
$\Delta f$ we measured was 76 Hz at $\varepsilon=3.4$ ppm, which required 3.5
hours of measurement time. The precision of our measurements of $\Delta f$ was
$\sim 0.5^{\circ}$ for a spectrum with signal to noise ratio $\sim 5\times
10^{6}$. For $T_{2}=692$ $\mu$s, this corresponds to a minimum detectable
frequency shift of $\sim 1$ Hz, shown as the solid blue line in Fig. 6. If the
frequency shift is entirely due to $\chi_{\eta}$, this corresponds to a
sensitivity of $\Delta\eta\sim 1$ ppm.

The pulsed strain technique enables measurements of nematic susceptibilities
via NMR in a broader range of materials than were previously accessible. This
is particularly true for materials with inhomogeneous linewidths, such as the
hole or electron-doped iron-based superconductors, e.g. Ba(Fe1-xCox)2As2.
Although the nematic susceptibility has been measured in the normal state of
these systems previously, pulsed-strain NMR measurements can enable
measurements of the nematic response below $T_{c}$ where elastoresistance
studies are not feasible. Below $T_{c}$, the NMR signal is usually reduced in
a single crystal because the penetration depth is much smaller than the skin
depth, requiring longer signal averaging times. In such cases it is difficult
to detect spectral shifts in the presence of static strain, however pulsed
strain measurements should be straightforward. Another advantage of this
approach is that it may also be useful for materials that exhibit a large
Young’s modulus, where DC strain approaches are unable to resolve any
linewidth changes. Alternatively, because the high sensitivity of this
approach does not require large strain values in order to determine the
response function, it will be useful for measurements of the nematic
susceptibility in materials that are particularly fragile. For example, this
technique would be ideal to investigate CaKFe4As4, where $T_{c}$ is high and
the material is nominally pure, yet elastoresistance measurements indicate an
enhanced nematic susceptibility in the normal state [43, 44]. Observations of
the how the nematic susceptibility changes in the superconducting state may
shed important light on whether nematic fluctuations play a role in
stabilizing the superconducting state [11]. More broadly, knowledge of the
response of the EFG tensor to strain may provide important insights into the
electronic structure calculations in strongly correlated materials [45, 46].

###### Acknowledgements.

We thank A. Chakraborty, J. Barraclough, R. Fernandes, I. Fisher and A. Ward
for stimulating discussions. NJC greatly benefited from the International
Workshop on the Experimental Advances in the Use of Pressure and Strain to
Probe and Control Quantum Matter, PSQM, held in Ames Iowa in May of 2022; many
of the key ideas associated with this paper were developed during the workshop
time. Work at UC Davis was supported by the NSF under Grants No. DMR-2210613
and PHY-1852581. Work at the Ames National Laboratory was supported by the
U.S. Department of Energy, Office of Science, Basic Energy Sciences, Materials
Sciences and Engineering Division. Ames National Laboratory is operated for
the U.S. Department of Energy by Iowa State University under Contract No. DE-
AC02-07CH11358.

## References

## References

  * Fradkin  _et al._ [2010] E. Fradkin, S. A. Kivelson, M. J. Lawler, J. P. Eisenstein, and A. P. Mackenzie, “Nematic Fermi fluids in condensed matter physics,” Annu. Rev. Condens. Matter Phys. 1, 153–178 (2010).
  * Canfield and Bud’ko [2010] P. C. Canfield and S. L. Bud’ko, “FeAs-based superconductivity: A case study of the effects of transition metal doping on BaFe2As2,” Annu. Rev. Condens. Matter Phys. 1, 27–50 (2010).
  * Fernandes  _et al._ [2012] R. M. Fernandes, A. V. Chubukov, J. Knolle, I. Eremin, and J. Schmalian, “Preemptive nematic order, pseudogap, and orbital order in the iron pnictides,” Phys. Rev. B 85, 024534 (2012).
  * Yi  _et al._ [2012] M. Yi, D. H. Lu, R. G. Moore, K. Kihou, C.-H. Lee, A. Iyo, H. Eisaki, T. Yoshida, A. Fujimori, and Z.-X. Shen, “Electronic reconstruction through the structural and magnetic transitions in detwinned NaFeAs,” New J. Phys. 14, 073019 (2012).
  * Chu  _et al._ [2012] J.-H. Chu, H.-H. Kuo, J. G. Analytis, and I. R. Fisher, “Divergent nematic susceptibility in an iron arsenide superconductor,” Science 337, 710–712 (2012).
  * Kuo  _et al._ [2016] H.-H. Kuo, J.-H. Chu, J. C. Palmstrom, S. A. Kivelson, and I. R. Fisher, “Ubiquitous signatures of nematic quantum criticality in optimally doped Fe-based superconductors,” Science 352, 958–962 (2016).
  * Worasaran  _et al._ [2021] T. Worasaran, M. S. Ikeda, J. C. Palmstrom, J. A. W. Straquadine, S. A. Kivelson, and I. R. Fisher, “Nematic quantum criticality in an Fe-based superconductor revealed by strain-tuning,” Science 372, 973–977 (2021).
  * Fernandes, Chubukov, and Schmalian [2014] R. M. Fernandes, A. V. Chubukov, and J. Schmalian, “What drives nematic order in iron-based superconductors?” Nat. Phys. 10, 97–104 (2014).
  * Lederer  _et al._ [2015] S. Lederer, Y. Schattner, E. Berg, and S. A. Kivelson, “Enhancement of superconductivity near a nematic quantum critical point,” Phys. Rev. Lett. 114, 097001 (2015).
  * Maier and Scalapino [2014] T. A. Maier and D. J. Scalapino, “Pairing interaction near a nematic quantum critical point of a three-band ${\mathrm{CuO}}_{2}$ model,” Phys. Rev. B 90, 174510 (2014).
  * Lederer  _et al._ [2017] S. Lederer, Y. Schattnerb, E. Berg, and S. A. Kivelson, “Superconductivity and non-Fermi liquid behavior near a nematic quantum critical point,” PNAS 114, 4905–4910 (2017).
  * Tanatar  _et al._ [2010] M. A. Tanatar, E. C. Blomberg, A. Kreyssig, M. G. Kim, N. Ni, A. Thaler, S. L. Bud’ko, P. C. Canfield, A. I. Goldman, I. I. Mazin, and R. Prozorov, “Uniaxial-strain mechanical detwinning of CaFe2As2 and BaFe2As2 crystals: Optical and transport study,” Phys. Rev. B 81, 184508 (2010).
  * Chu  _et al._ [2010] J.-H. Chu, J. G. Analytis, K. De Greve, P. L. McMahon, Z. Islam, Y. Yamamoto, and I. R. Fisher, “In-plane resistivity anisotropy in an underdoped iron arsenide superconductor,” Science 329, 824–826 (2010).
  * Fernandes  _et al._ [2010] R. M. Fernandes, L. H. VanBebber, S. Bhattacharya, P. Chandra, V. Keppens, D. Mandrus, M. A. McGuire, B. C. Sales, A. S. Sefat, and J. Schmalian, “Effects of nematic fluctuations on the elastic properties of iron arsenide superconductors,” Phys. Rev. Lett. 105, 157003 (2010).
  * Böhmer  _et al._ [2015] A. E. Böhmer, T. Arai, F. Hardy, T. Hattori, T. Iye, T. Wolf, H. v. Löhneysen, K. Ishida, and C. Meingast, “Origin of the tetragonal-to-orthorhombic phase transition in FeSe: A combined thermodynamic and NMR study of nematicity,” Phys. Rev. Lett. 114, 027001 (2015).
  * He  _et al._ [2016] M. He, L. Wang, F. Ahn, F. Hardy, T. Wolf, P. Adelmann, J. Schmalian, I. Eremin, and C. Meingast, “Dichotomy between in-plane magnetic susceptibility and resistivity anisotropies in extremely strained BaFe2As2,” Nat. Commun. 8, 504 (2016).
  * Böhmer and Meingast [2016] A. E. Böhmer and C. Meingast, “Electronic nematic susceptibility of iron-based superconductors,” C. R. Phys. 17, 90 – 112 (2016).
  * Jiang  _et al._ [2013] S. Jiang, H. S. Jeevan, J. Dong, and P. Gegenwart, “Thermopower as a sensitive probe of electronic nematicity in iron pnictides,” Phys. Rev. Lett. 110, 067001 (2013).
  * Gallais  _et al._ [2013] Y. Gallais, R. M. Fernandes, I. Paul, L. Chauvière, Y.-X. Yang, M.-A. Méasson, M. Cazayous, A. Sacuto, D. Colson, and A. Forget, “Observation of incipient charge nematicity in Ba(Fe1-xCox)2As2,” Phys. Rev. Lett. 111 (2013), 10.1103/PhysRevLett.111.267001.
  * Blomberg  _et al._ [2012] E. C. Blomberg, A. Kreyssig, M. A. Tanatar, R. M. Fernandes, M. G. Kim, A. Thaler, J. Schmalian, S. L. Bud’ko, P. C. Canfield, A. I. Goldman, and R. Prozorov, “Effect of tensile stress on the in-plane resistivity anisotropy in BaFe2As2,” Phys. Rev. B 85, 144509 (2012).
  * Tanatar  _et al._ [2016] M. A. Tanatar, A. E. Böhmer, E. I. Timmons, M. Schütt, G. Drachuck, V. Taufour, K. Kothapalli, A. Kreyssig, S. L. Bud’ko, P. C. Canfield, R. M. Fernandes, and R. Prozorov, “Origin of the resistivity anisotropy in the nematic phase of FeSe,” Phys. Rev. Lett. 117, 127001 (2016).
  * Mirri  _et al._ [2015] C. Mirri, A. Dusza, S. Bastelberger, M. Chinotti, L. Degiorgi, J.-H. Chu, H.-H. Kuo, and I. R. Fisher, “Origin of the resistive anisotropy in the electronic nematic phase of BaFe2As2 revealed by optical spectroscopy,” Phys. Rev. Lett. 115, 107001 (2015).
  * Ning  _et al._ [2014] F. L. Ning, M. Fu, D. A. Torchetti, T. Imai, A. S. Sefat, P. Cheng, B. Shen, and H.-H. Wen, “Critical behavior of the spin density wave transition in underdoped Ba(Fe1-xCox)2As2 ($x\leq 0.05$): 75As NMR investigation,” Phys. Rev. B 89, 214511 (2014).
  * Kissikov  _et al._ [2018] T. Kissikov, R. Sarkar, M. Lawson, B. T. Bush, E. I. Timmons, M. A. Tanatar, R. Prozorov, S. L. Bud’ko, P. C. Canfield, R. M. Fernandes, and N. J. Curro, “Uniaxial strain control of spin-polarization in multicomponent nematic order of BaFe2As2,” Nat. Commun. 9, 1058 (2018).
  * Dai [2015] P. Dai, “Antiferromagnetic order and spin dynamics in iron-based superconductors,” Rev. Mod. Phys. 87, 855–896 (2015).
  * Man  _et al._ [2015] H. Man, X. Lu, J. S. Chen, R. Zhang, W. Zhang, H. Luo, J. Kulda, A. Ivanov, T. Keller, E. Morosan, Q. Si, and P. Dai, “Electronic nematic correlations in the stress-free tetragonal state of BaFe2-xNixAs2,” Phys. Rev. B 92, 134521 (2015).
  * Lu  _et al._ [2014] X. Lu, J. T. Park, R. Zhang, H. Luo, A. H. Nevidomskyy, Q. Si, and P. Dai, “Nematic spin correlations in the tetragonal state of uniaxial-strained BaFe2-xNixAs2,” Science 345, 657 – 660 (2014).
  * Dioguardi  _et al._ [2016] A. P. Dioguardi, T. Kissikov, C. H. Lin, K. R. Shirer, M. M. Lawson, H.-J. Grafe, J.-H. Chu, I. R. Fisher, R. M. Fernandes, and N. J. Curro, “NMR evidence for inhomogeneous nematic fluctuations in BaFe2(As1-xPx)2,” Phys. Rev. Lett. 116, 107202 (2016).
  * Kissikov  _et al._ [2017a] T. Kissikov, R. Sarkar, M. Lawson, B. T. Bush, E. I. Timmons, M. A. Tanatar, R. Prozorov, S. L. Bud’ko, P. C. Canfield, R. M. Fernandes, W. F. Goh, W. E. Pickett, and N. J. Curro, “Local nematic susceptibility in stressed BaFe2As2 from NMR electric field gradient measurements,” Phys. Rev. B 96, 241108(R) (2017a).
  * Baek  _et al._ [2015] S.-H. Baek, D. V. Efremov, J. M. Ok, J. S. Kim, J. van den Brink, and B. Büchner, “Orbital-driven nematicity in FeSe,” Nat. Mater. 14, 210–214 (2015).
  * Edelstein  _et al._ [1980] W. A. Edelstein, J. M. S. Hutchison, G. Johnson, and T. Redpath, “Spin warp NMR imaging and applications to human whole-body imaging,” Phys. Med. Biol. 25, 751–756 (1980).
  * Johnson  _et al._ [1983] G. Johnson, J. Hutchison, T. Redpath, and L. Eastwood, “Improvements in performance time for simultaneous three-dimensional nmr imaging,” Journal of Magnetic Resonance (1969) 54, 374–384 (1983).
  * Falk  _et al._ [2014] A. L. Falk, P. V. Klimov, B. B. Buckley, V. Ivády, I. A. Abrikosov, G. Calusine, W. F. Koehl, Á. Gali, and D. D. Awschalom, “Electrically and mechanically tunable electron spins in silicon carbide color centers,” Phys. Rev. Lett. 112, 187601 (2014).
  * Hristov  _et al._ [2018] A. T. Hristov, J. C. Palmstrom, J. A. W. Straquadine, T. A. Merz, H. Y. Hwang, and I. R. Fisher, “Measurement of elastoresistivity at finite frequency by amplitude demodulation,” Rev. Sci. Instrum. 89, 103901 (2018).
  * Ikeda  _et al._ [2019] M. S. Ikeda, J. A. W. Straquadine, A. T. Hristov, T. Worasaran, J. C. Palmstrom, M. Sorensen, P. Walmsley, and I. R. Fisher, “AC elastocaloric effect as a probe for thermodynamic signatures of continuous phase transitions,” Rev. Sci. Instrum. 90, 083902 (2019).
  * Hicks  _et al._ [2014] C. W. Hicks, M. E. Barber, S. D. Edkins, D. O. Brodsky, and A. P. Mackenzie, “Piezoelectric-based apparatus for strain tuning,” Rev. Sci. Instrum. 85, 065003 (2014).
  * Kissikov  _et al._ [2017b] T. Kissikov, R. Sarkar, B. T. Bush, M. Lawson, P. C. Canfield, and N. J. Curro, “Nuclear magnetic resonance probe head design for precision strain control,” Rev. Sci. Instrum. 88, 103902 (2017b).
  * Curro  _et al._ [2022] N. J. Curro, T. Kissikov, M. A. Tanatar, R. Prozorov, S. L. Bud’ko, and P. C. Canfield, “Nematicity and glassy behavior probed by nuclear magnetic resonance in iron-based superconductors,” Front. Phys. 10 (2022), 10.3389/fphy.2022.877628.
  * Kitagawa  _et al._ [2008] K. Kitagawa, N. Katayama, K. Ohgushi, M. Yoshida, and M. Takigawa, “Commensurate itinerant antiferromagnetism in BaFe2As2: 75As-NMR studies on a self-flux grown single crystal,” J. Phys. Soc. Jpn. 77, 114709 (2008).
  * Johansson, Nation, and Nori [2012] J. Johansson, P. Nation, and F. Nori, “QuTiP: An open-source python framework for the dynamics of open quantum systems,” Comput. Phys. Commun. 183, 1760–1772 (2012).
  * Johansson, Nation, and Nori [2013] J. Johansson, P. Nation, and F. Nori, “QuTiP 2: A python framework for the dynamics of open quantum systems,” Comput. Phys. Commun. 184, 1234–1240 (2013).
  * Dioguardi  _et al._ [2013] A. P. Dioguardi, J. Crocker, A. C. Shockley, N. apRoberts Warren, C. Lin, K. R. Shirer, D. Nisson, A. Lodhia, P. Klavins, and N. J. Curro, “Angular dependent 75As NMR study of the electric field gradient in CaFe2As2,” Supercond. Sci. Technol. 26, 025012 (2013).
  * Cui  _et al._ [2017] J. Cui, Q.-P. Ding, W. R. Meier, A. E. Böhmer, T. Kong, V. Borisov, Y. Lee, S. L. Bud’ko, R. Valentí, P. C. Canfield, and Y. Furukawa, “Magnetic fluctuations and superconducting properties of CaKFe4As4 studied by 75As NMR,” Phys. Rev. B 96, 104512 (2017).
  * Meier  _et al._ [2016] W. R. Meier, T. Kong, U. S. Kaluarachchi, V. Taufour, N. H. Jo, G. Drachuck, A. E. Böhmer, S. M. Saunders, A. Sapkota, A. Kreyssig, M. A. Tanatar, R. Prozorov, A. I. Goldman, F. F. Balakirev, A. Gurevich, S. L. Bud’ko, and P. C. Canfield, “Anisotropic thermodynamic and transport properties of single-crystalline ${\mathrm{cakfe}}_{4}{\mathrm{as}}_{4}$,” Phys. Rev. B 94, 064501 (2016).
  * Stoll, Meier, and Claxton [2002] E. P. Stoll, P. F. Meier, and T. A. Claxton, “Electric field gradients from first-principles and point-ion calculations,” Phys. Rev. B 65, 064532 (2002).
  * Rusz  _et al._ [2008] J. Rusz, P. M. Oppeneer, N. J. Curro, R. R. Urbano, B.-L. Young, S. Lebègue, P. G. Pagliuso, L. D. Pham, E. D. Bauer, J. L. Sarrao, and Z. Fisk, “Probing the electronic structure of pure and doped CeMIn5 (M = Co,Rh,Ir) crystals with nuclear quadrupolar resonance,” Phys. Rev. B 77, 245124 (2008).


# TartanDrive 2.0: More Modalities and Better Infrastructure to Further Self-
Supervised Learning Research in Off-Road Driving Tasks

Matthew Sivaprakasam1, Parv Maheshwari2, Mateo Guaman Castro1, Samuel Triest1,
Micah Nye3,
Steve Willits1, Andrew Saba1, Wenshan Wang1, and Sebastian Scherer1 * This
work was supported by ARL awards #W911NF1820218 and #W911NF20S0005.1 Robotics
Institute, Carnegie Mellon University,
<EMAIL_ADDRESS>Department of
Mathematics, Indian Institute of Technology Kharagpur.
<EMAIL_ADDRESS>Department of Mechanical Engineering,
University of Pittsburgh<EMAIL_ADDRESS>
###### Abstract

We present TartanDrive 2.0, a large-scale off-road driving dataset for self-
supervised learning tasks. In 2021 we released TartanDrive 1.0, which is one
of the largest datasets for off-road terrain. As a follow-up to our original
dataset, we collected seven hours of data at speeds of up to 15m/s with the
addition of three new LiDAR sensors alongside the original camera, inertial,
GPS, and proprioceptive sensors. We also release the tools we use for
collecting, processing, and querying the data, including our metadata system
designed to further the utility of our data. Custom infrastructure allows end
users to reconfigure the data to cater to their own platforms. These tools and
infrastructure alongside the dataset are useful for a variety of tasks in the
field of off-road autonomy and, by releasing them, we encourage collaborative
data aggregation. These resources lower the barrier to entry to utilizing
large-scale datasets, thereby helping facilitate the advancement of robotics
in areas such as self-supervised learning, multi-modal perception, inverse
reinforcement learning, and representation learning. The dataset is available
at https://theairlab.org/TartanDrive2.

Figure 1: We provide a new off-road driving dataset for self-supervised
learning tasks. Multi-modal data is collected as the robot is driven through
several types of terrain that present challenging scenarios to perception,
planning, and control algorithms.

##  I Introduction

As the state of the art in autonomous driving improves, robots are expected to
handle increasingly complex tasks. In off-road driving, this translates to
situations such as a robot knowing whether the dark brown path in front of it
is dry dirt or thick mud, whether it is worth it to take a shortcut through
tall grass, and when to preemptively increase velocity in order to make it up
a hill. Navigating these decisions requires behaviors that are extremely
difficult to robustly design and tune by hand. Many of the currently-
researched solutions to this challenge involve the adoption of large neural
networks and data-driven models. For the task of on-road driving in urban
scenarios, there are several large datasets available [1, 2, 3, 4]. These
datasets were feasible not only because of the resources present at the
institutions that collected them, but also because of the inherent everyday
nature of on-road driving. There exist fewer datasets for off-road driving,
primarily because the nature of off-road terrain makes gathering enough data
uniquely difficult. Collecting samples in simulation is often insufficient due
to the complexity of calculating the dynamics of terrain in complex
environments. In fact, most real-world scenarios that are hard to accurately
simulate are the same ones which produce complex situations for autonomous
off-road agents, such as dense foliage and slippery surfaces. However,
collecting data in real-life presents logistics challenges such as preserving
driver safety and dealing with frequent vehicle damage and wear. These
difficulties have resulted in a scarcity of available data for off-road
driving.

While the number of off-road datasets has been growing [5, 6, 7, 8, 9, 10, 11,
12], many of them are still limited in sample size, modality, or difficulty
[13]. This is often due to their focus on specific tasks such as semantic
segmentation, which requires manual labeling effort. Scaling up these datasets
would come either at the cost of time or label quality. It is possible to use
multiple datasets together to train a model, but the variability of off-road
terrain often causes inconsistency in labels across datasets [14]. The
difficulty of generating large amounts of manually labeled data in off-road
terrain suggests that self-supervised learning is essential to outperforming
prior methods. Many works have shown that self-supervised methods can be used
to learn strong representations [15, 16, 17], and some works have already
shown how it can be used for various tasks in off-road driving environments
[18, 19, 20, 21]. We argue that in order to scale up the amount of data for
off-road tasks, datasets should be designed to be task-agnostic and with self-
supervised learning in mind.

In 2021, we released TartanDrive, a multi-modal off-road driving dataset
designed for dynamics modeling [5]. In this work we present TartanDrive 2.0, a
larger dataset geared towards self-supervised learning methods. The
contributions of this dataset over the previous generation are as follows:

  1. 1.

More sensors and data: In our original work, we argue that multiple modalities
are essential for learning models for off-road terrain. To that end, we add
LiDAR as an additional modality. We also collect seven hours over the five
hours of data present in the original dataset, at higher average speeds and
including some new areas, as well as areas present in the previous dataset
that have dramatically changed over the past two years.

  2. 2.

Better infrastructure for end-users: We have improved the infrastructure in
order to improve ease-of-use and utility of the dataset, such as the ability
to re-process the data based on user-specified configurations. We also
implement a metadata system that filters and groups data by a variety of
properties.

  3. 3.

Open-source data collection tooling: Our data collection process has been
constructed in a way that allows us to continually release more data over time
as the seasons change and as we add more modalities. By releasing our tools
and framework, others can collect their data and easily merge it with our own
to create larger datasets.

##  II Related Work

Collecting general driving datasets is a common pursuit, the largest ones
coming from companies gathering data in urban scenarios. For example, nuScenes
by Motional is a dataset containing 1.4 million images and 390,000 LiDAR
sweeps, with corresponding 3D object annotations [3, 4]. The Waymo Open
Dataset is another example with over a million images and LiDAR pointclouds
[1, 2]. Both datasets include infrastructure and tools to query and process
these massive amounts of data in ways that support various downstream learning
tasks and benchmarks that are not often seen in smaller datasets. There are
also other efforts in collecting data for on-road environments, such as the
RACECAR dataset that includes raw and processed sensor data from autonomous
racecars driving at high speeds [22].

A number of earlier works have tried to circumvent the issues due to the lack
of real-world off-road driving data by utilizing simulation environments.
Tremblay et al. show how training a neural network on multiple modalities in
simulation can allow a robot to predict its dynamics in unseen real-world data
[23]. Sivaprakasam et al. collect data in simulation of a robot driving over
obstacles in order to train a model that predicts the difference between a
desired path and its resulting path [24].

Now, more off-road datasets have been collected. RUGD contains over 7,000
images in a variety of off-road terrain with manually-labeled semantic
segmentation masks [7]. Rellis-3D includes annotations for 6,235 images and
13,556 scans from two different LiDARs, as well as the bags that they
originally recorded which also contain IMU, GPS, and stereo image data. Sharma
et al. created an off-road image dataset where, rather than creating semantic
masks, they label regions in an image based on their traversability by
different types of vehicles [8]. The first version of TartanDrive has also
been available for two years and has been used for a number of tasks even
outside the domain of off-road driving. Shah et al. have used this data as
part of a bigger dataset that was used to train a large foundation model
designed with vision-based robotic navigation in mind [25].

Figure 2: The ATV used for data collection (left); The primary sensor payload
on the vehicle (right) Figure 3: The 3D scan of the ATV. The rear half of the
scan is cropped in the left picture to increase clarity of the front payload
in the image.

##  III The Dataset

The original TartanDrive dataset was designed for the task of dynamics
prediction. When designing TartanDrive 2.0 we aimed to support a broader set
of tasks in perception, planning, and control. Thanks to more sensor
modalities and better infrastructure, the new data that we have collected
driving through diverse terrain and unique scenarios is more broadly
applicable than before.

###  III-A The Platform

We use the same Yamaha Viking All-Terrain Vehicle (ATV) as before (Fig. 2),
first modified by Mai et al. [26]. We have modified it again in order to equip
it with three LiDAR sensors. There are two Velodyne VLP-32 sensors mounted to
the front of the roof of the vehicle, with one tilted downwards to increase
coverage of the ground closer to the vehicle. There is also a Livox Mid-70,
mounted under the MultiSense camera, which provides more information on
objects directly in front of the vehicle. Using a Faro Focus Scanner, we
generated a 3D pointcloud model of the ATV as shown in Fig. 3 and provide it
with the dataset so that users can take their own measurements (e.g. distance
between the wheel axle and some arbitrary point on the vehicle, or distance
between the GPS antenna and the ground) for their own specific experiments.

###  III-B Data Collection

The site for collecting data is the same location in western Pennsylvania as
TartanDrive 1.0, consisting of terrain such as narrow paths, dense foliage,
rocky terrain, dirt paths, and steep hills. It is worth noting that some areas
of the site have changed significantly due to natural causes such as erosion
or overgrowth as shown in Fig. 4. All the data included was acquired by a
human tele-operating the vehicle. Each sequence of data is annotated with the
following metadata:

  * •

Driver ID and robot

  * •

Number of people in the vehicle

  * •

Date/time

  * •

Context (e.g. data collection)

  * •

Weather conditions (e.g. dry, damp, snow)

  * •

Lighting conditions (e.g. sunny, overcast, sunset)

  * •

Course ID or general location (from a pre-defined list)

as well as any other information relevant to that specific run. We emphasize
that recording this metadata makes the dataset significantly easier to use,
especially as the dataset becomes larger. At the end of the collection, a
post-processor records information such as top speed, average speed, duration,
and sensors present. During data collection, a co-pilot takes time-stamped
annotations of relevant or unique events such as sensor failure or something
uncommon such as a deer spotting in front of the vehicle. Additionally, in
some runs the co-pilot periodically annotates a weak driving score ranging
from 1-5 where 5 signifies ideal driving. Recording this metadata provides
more structure in the data which significantly improves the utility of the
dataset when used in combination with our post-processing pipelines.

Figure 4: The data collection site has changed significantly since our last
dataset released in 2021. For example, some older dirt paths (top, shown in
red) are now covered in tall grass (bottom). Figure 5: Example coverage
provided by our LiDAR sensors. Purple and yellow points come from the two
Velodynes, and cyan points from the Livox. Figure 6: The high-level flow of
information in our data collection process, with software in red and hardware
in blue.

###  III-C Raw Data

Most of the sensors on our platform have already been detailed [5, 27], but
all raw data is briefly summarized below:

####  III-C1 Pointclouds

We record incoming pointclouds from two Velodyne VLP-32 LiDAR sensors and a
Livox Mid-70. We also provide extrinsics so that they can be merged into a
single pointcloud. An example of the coverage they provide is shown in Fig. 5.
All LiDARs are configured to run at 10Hz.

####  III-C2 Images

A Carnegie Robotics MultiSense S21 is used to provide stereo images,
specifically greyscale images from both cameras as well as RGB images from the
left camera at 10Hz each.

####  III-C3 IMU and Pose

A NovAtel PROPAK-V3-RT2i GNSS provides IMU data at 100Hz. This is fused with
incoming GPS data to provide a pose estimate at 50Hz. The MultiSense also
provides IMU data at 400Hz.

####  III-C4 Teleoperation

The driver uses a joystick controller to send steering commands, which are
then recorded alongside the values of the current steering angle (actuation on
steering commands is not instantaneous). A Racepak G2X Pro Data Logger is used
to provide the positions of the acceleration and brake pedals.

####  III-C5 Proprioceptive Information

The Racepak is also used to record RPMs for each wheel and suspension shock
travel for the rear two wheels. Shock travel data for the front two wheels is
omitted due to frequent damage to the sensors during aggressive driving
maneuvers, but will be included in future data releases.

###  III-D Post-Processed Data

In order to increase utility and ease-of-use of the dataset, we provide the
raw data and the outputs from some existing modules that have been integrated
into our software stack (Fig. 6) and commonly use as inputs to our own
algorithms:

####  III-D1 TartanVO

We run TartanVO [28] on the platform, which takes in the stereo images from
the MultiSense as input. It outputs an odometry estimate, a predicted
pointcloud, and top-down height and RGB maps, all at 10Hz.

####  III-D2 Roughness Cost

We produce a roughness cost that describes the bumpiness of terrain as we
drive over it, derived from a sliding window of Z-axis linear acceleration
values from the IMU as described in Guaman Castro et al. [20].

####  III-D3 Odometry and Registered Pointcloud

In addition to the odometry provided by the Novatel system, we also provide an
estimated output by Super Odometry [29]. The high accuracy and frequency of
this output allows for cleaner results in tasks such as pointcloud
registration which is in turn important for other downstream tasks.

####  III-D4 Local LiDAR Maps

Using the registered pointcloud provided by Super Odometry, we generate a
birds-eye-view feature map 200x200m wide at .5m resolution. This map provides
geometric information about the environment (Fig. 7), with features consisting
of the following:

  * •

Min/Max/Mean Height of Points

  * •

Roughness

  * •

SVD Features

  * •

Estimated Ground Height

  * •

Estimated Ground Slope (X, Y, Magnitude)

Figure 7: Using the registered pointcloud, we provide local maps that contain
various geometric features. Figure 8: Our dataset covers over 255 acres
collected over seven hours. Using the accompanying metadata, the it can easily
be split into groups by aspects such as speed, GPS bounding box, and time of
day.

###  III-E Data Pipelines

####  III-E1 Formatting

We provide the data in two main formats, the first one being the rosbags that
they were originally recorded as. This allows users to test how their
algorithms might perform in different environments in real-time. The second
format is as a set of sequences similar to the KITTI format [30]. For each
bag, a folder is created and a subfolder for each modality is initialized. The
samples across all modalities are then timesynced and then placed in their
respective subfolders (e.g. ’pointcloud_1/0000.bin’ and ’image/0000.jpg’ are
associated with each other).

####  III-E2 Reconfiguration

We have post-processed our data in a way that makes sense for our own
algorithms, but our ATV platform is somewhat unique. Across different robots,
the optimal parameters for elements such as sample frequency, map size, and
map resolution vary. For example, a smaller robot with better agility might
require a map with finer than .5m resolution. In order to make our data more
directly applicable to other platforms, we provide scripts that allow users to
regenerate the dataset with different parameters as they see fit.

####  III-E3 Utilities

As previously mentioned, we collect metadata and some annotations during data
collection. We provide scripts to take advantage of this metadata in order to
filter the data by various elements and therefore increase the utility of the
dataset for different learning tasks. For example, given the whole dataset, a
user can easily create subsets grouped by components such as speed, GPS
bounding box (we provide a GUI for easily generating bounding boxes), lighting
conditions, or driver (Fig. 8). This is especially useful for testing model
generalizability by training on one location/condition, and testing on
another.

####  III-E4 Common Framework

The infrastructure we have developed has streamlined our data collection
procedures to make it easy to train models on processed datasets as well as
test our algorithms in real-time. By continuing to follow the same procedures,
the data we collect in the future can easily be merged into our existing
datasets. Likewise, other researchers can use our tools when collecting their
own off-road driving data which could eventually lead to a larger multi-site,
and multi-vehicle dataset.

Figure 9: In cases where conditions like poor lighting affect camera
performance, LiDARs can allow a robot to still understand its surroundings.
Figure 10: An example of the overlap between a single scan from all 3 LiDARS
and our camera. Points are colored by forward distance from the vehicle.

##  IV Impact on Current and Future Research

While we do not provide any explicit supervision labels (e.g. segmentation
masks, classification labels) outside of annotations, the several modalities
and actions that we provide facilitate a number of self-supervised learning
tasks. Our original TartanDrive has already benefited off-road research
immensely, and we believe the categories listed in Table I are only a subset
of the research topics TartanDrive 2.0 can bolster.

Dataset | State | Action | Image | Pointcloud | Heightmap | RGBmap | IMU | Wheel RPM | Shocks | Metadata | Labels
---|---|---|---|---|---|---|---|---|---|---|---
RUGD [7] | No | No | Yes | No | No | No | No | No | No | No | Yes
Rellis 3D [6] | Yes | Yes | Yes | Yes | No | No | Yes | No | No | No | Yes
Wild-Places [11] | Yes | No | No | Yes | No | No | No | No | No | No | Yes
Montmorency [31] | Yes | Yes | Yes | Yes | Yes | No | Yes | No | No | No | Yes
Verti-Wheelers [10] | Yes | Yes | Yes | No | No | No | Yes | Yes | No | No | No
TartanDrive 1.0 [5] | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | No | No
TartanDrive 2.0 (Ours) | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No
TABLE I: Overview and comparison of various off-road driving datasets Figure
11: In TartanDrive 2.0 we collect a higher concentration of data at higher
speeds than in TartanDrive 1.0, reaching up to 15m/s.

###  IV-A Perception - Cross-Modal Supervision:

Our data was used by Guaman et al. to visually predict a bumpiness cost
supervised by an IMU-derived metric [20]. With the addition of LiDAR in our
dataset, we can advance algorithms by providing information in scenarios where
cameras fail (Fig. 9) and learning from a much more accurate source of
odometry and depth (Fig. 10). For example, Chen et al. accumulates near-range
LiDAR measurements to learn long-range visual traversability model [19]. Meng
et al. train a model that uses images to predict feature maps supervised by
lidar inputs [18]. However, these works rely on off-road driving datasets that
are not publicly available. By releasing data in a similar domain with the
same modalities, we lower the barrier to entry for expanding on this field of
research, and allow a common point of comparison.

###  IV-B Perception - Map Completion:

Prediction of occluded and sparsely sensed areas enables faster safe
navigation in occluded area [18, 32]. Our generated accumulated maps are being
used as a supervisory signal in our ongoing research in map completion and can
also serve as a common benchmark for other approaches.

###  IV-C Perception - Ground Height Estimation:

An understanding of the support ground surface in off-road terrain is
essential for successful navigation. We enable methods of learning this
feature by providing multiple modalities as inputs and a 3D model of the car
that can be used alongside odometry estimates to provide supervision of the
true ground surface.

###  IV-D Planning - Learning from Demonstration:

The actions derived from human teleoperation coupled with sensor inputs can be
used to learn models that can reason about what areas are more traversable
than others. Triest et al. treats teleoperated driving as expert
demonstrations and use inverse reinforcement learning to birds-eye-view
costmaps from the LiDAR feature maps [21].

###  IV-E Controls - Aggressive Maneuver Driving:

The unstructured nature of complex terrain makes it difficult to drive
aggressively at high speeds without losing control. Our new dataset contains a
higher proportion of speeds beyond 7m/s than before. Some of our ongoing
research on learning vehicle dynamics models is supported by this data.

##  V Conclusions and Future Work

We have presented TartanDrive 2.0, which provides significant improvements
over the original TartanDrive. We emphasize how the inclusion of more
modalities can benefit research in self-supervised learning, and how more data
and better infrastructure allow it to scale up to the needs of large models.
The tools that we release with the dataset have streamlined our data
collection, facilitating our plans of collecting more data on our existing
site as the seasons change and then releasing that data over time.

Despite the major improvements in this generation of our dataset, there
remains a number of directions to pursue in order to further increase its
quality and utility. For example, while LiDAR sensors are useful in much of
modern robotics research, there has also been an increasing amount of research
in using passive sensors for perception [33, 34]. To that end it would be
helpful to add sensors like thermal cameras in the near future.

We also plan to include audio and language in our data. There is an increasing
amount of research in foundation models and vision-language models [35, 36],
some of which have already demonstrated improvement in some robotics
applcations [37, 38, 39, 40]. We hypothesize that learning the relationship
between driver dialogue and their actions and/or the resulting environmental
interactions might be supplemental in training large neural networks. Apart
from recording dialogue, we also plan to record audio of the interactions
between tires and the ground as we drive through different terrains, as Zurn
et al. have shown that this information can be used to learn an understanding
of complex terrain [41].

Additionally, our test site is very large and complex, but we have found that
our own algorithms are close to enabling the ATV to successfully traverse much
of the terrain. This brings up the issue of overfitting, and whether or not
our algorithms would be able to reach the same level of performance in unseen,
challenging terrain. By releasing our tools alongside the dataset we hope that
others will contribute data from other locations in order to collaboratively
form an even larger dataset. However, regardless of external contributions, we
plan to collect data in other off-road sites with different types of
challenging terrain that will allow ourselves and others to test
generalizability and online adaptation of our algorithms.

## References

  * [1] P. Sun, H. Kretzschmar, X. Dotiwalla _et al._ , “Scalability in perception for autonomous driving: Waymo open dataset,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [2] S. Ettinger, S. Cheng, B. Caine _et al._ , “Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset.”
  * [3] H. Caesar, V. Bankiti, A. H. Lang _et al._ , “nuscenes: A multimodal dataset for autonomous driving,” in _CVPR_ , 2020.
  * [4] J. H. L. Z. H. C. O. B. A. V. W. Fong, R. Mohan, “Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking,” in _ICRA_ , 2022.
  * [5] S. Triest, M. Sivaprakasam, S. J. Wang _et al._ , “Tartandrive: A large-scale dataset for learning off-road dynamics models,” in _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 2546–2552.
  * [6] P. Jiang, P. Osteen, M. Wigness, and S. Saripalli, “Rellis-3d dataset: Data, benchmarks and analysis,” 2020.
  * [7] M. Wigness, S. Eum, J. G. Rogers _et al._ , “A rugd dataset for autonomous navigation and visual perception in unstructured outdoor environments,” in _International Conference on Intelligent Robots and Systems (IROS)_ , 2019.
  * [8] S. Sharma, L. Dabbiru, T. Hannis _et al._ , “Cat: Cavs traversability dataset for off-road autonomous driving,” _IEEE Access_ , vol. 10, pp. 24 759–24 768, 2022.
  * [9] D. Maturana, P.-W. Chou, M. Uenoyama, and S. Scherer, “Real-time semantic mapping for autonomous off-road navigation,” in _Field and Service Robotics_ , M. Hutter and R. Siegwart, Eds. Cham: Springer International Publishing, 2018, pp. 335–350.
  * [10] A. Datar, C. Pan, M. Nazeri, and X. Xiao, “Toward wheeled mobility on vertically challenging terrain: Platforms, datasets, and algorithms,” 2023.
  * [11] J. Knights, K. Vidanapathirana, M. Ramezani _et al._ , “Wild-places: A large-scale dataset for lidar place recognition in unstructured natural environments,” 2023.
  * [12] A. Reinke, M. Palieri, B. Morrell _et al._ , “Locus 2.0: Robust and computationally efficient lidar odometry for real-time 3d mapping,” _IEEE Robotics and Automation Letters_ , pp. 1–8, 2022.
  * [13] A. Shaban, X. Meng, J. Lee _et al._ , “Semantic terrain classification for off-road autonomous driving,” in _Proceedings of the 5th Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, A. Faust, D. Hsu, and G. Neumann, Eds., vol. 164. PMLR, 08–11 Nov 2022, pp. 619–629. [Online]. Available: https://proceedings.mlr.press/v164/shaban22a.html
  * [14] Z. Yang, Y. Tan, S. Sen _et al._ , “Uncertainty-aware perception models for off-road autonomous unmanned ground vehicles,” _CoRR_ , vol. abs/2209.11115, 2022. [Online]. Available: https://doi.org/10.48550/arXiv.2209.11115
  * [15] J.-B. Grill, F. Strub, F. Altché _et al._ , “Bootstrap your own latent - a new approach to self-supervised learning,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell _et al._ , Eds., vol. 33. Curran Associates, Inc., 2020, pp. 21 271–21 284. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf
  * [16] M. Oquab, T. Darcet, T. Moutakanni _et al._ , “Dinov2: Learning robust visual features without supervision,” 2023.
  * [17] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _Proceedings of the 37th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 13–18 Jul 2020, pp. 1597–1607. [Online]. Available: https://proceedings.mlr.press/v119/chen20j.html
  * [18] X. Meng, N. Hatch, A. Lambert _et al._ , “Terrainnet: Visual modeling of complex terrain for high-speed, off-road navigation,” 2023.
  * [19] E. Chen, C. Ho, M. Maulimov _et al._ , “Learning-on-the-drive: Self-supervised adaptation of visual offroad traversability models,” 2023.
  * [20] M. G. Castro, S. Triest, W. Wang _et al._ , “How does it feel? self-supervised costmap learning for off-road vehicle traversability,” 2023.
  * [21] S. Triest, M. G. Castro, P. Maheshwari _et al._ , “Learning risk-aware costmaps via inverse reinforcement learning for off-road navigation,” 2023.
  * [22] A. Kulkarni, J. Chrosniak, E. Ducote _et al._ , “Racecar – the dataset for high-speed autonomous racing,” 2023.
  * [23] J.-F. Tremblay, T. Manderson, A. Noca _et al._ , “Multimodal dynamics modeling for off-road autonomous vehicles,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 1796–1802.
  * [24] M. Sivaprakasam, S. Triest, W. Wang _et al._ , “Improving off-road planning techniques with learned costs from physical interactions,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 4844–4850.
  * [25] D. Shah, A. Sridhar, N. Dashora _et al._ , “ViNT: A Foundation Model for Visual Navigation,” _arXiv pre-print_ , 2023. [Online]. Available: https://arxiv.org/abs/2306.14846
  * [26] J. Mai, “System design, modelling, and control for an off-road autonomous ground vehicle,” Master’s thesis, Carnegie Mellon University, Pittsburgh, PA, July 2020.
  * [27] M. Sivaprakasam, S. Triest, M. G. Castro _et al._ , “Tartandrive 1.5: Improving large multimodal robotics dataset collection and distribution,” in _ICRA2023 Workshop on Pretraining for Robotics (PT4R)_ , 2023. [Online]. Available: https://openreview.net/forum?id=7Y1pnhFJUT
  * [28] W. Wang, Y. Hu, and S. Scherer, “Tartanvo: A generalizable learning-based vo,” 2020.
  * [29] S. Zhao, H. Zhang, P. Wang _et al._ , “Super odometry: Imu-centric lidar-visual-inertial estimator for challenging environments,” 09 2021, pp. 8729–8736.
  * [30] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _International Journal of Robotics Research (IJRR)_ , 2013.
  * [31] J.-F. Tremblay, M. Béland, F. Pomerleau _et al._ , “Automatic 3d mapping for tree diameter measurements in inventory operations,” _arXiv preprint arXiv:1904.05281_ , 2019.
  * [32] R. Schmid, D. Atha, F. Schöller _et al._ , “Self-supervised traversability prediction by learning to reconstruct safe terrain,” 2022.
  * [33] A. Srivastav and S. Mandal, “Radars for autonomous driving: A review of deep learning methods and challenges,” 2023.
  * [34] M. Brenner, N. H. Reyes, T. Susnjak, and A. L. C. Barczak, “RGB-d and thermal sensor fusion: A systematic literature review,” _IEEE Access_ , vol. 11, pp. 82 410–82 442, 2023. [Online]. Available: https://doi.org/10.1109%2Faccess.2023.3301119
  * [35] OpenAI, “Gpt-4 technical report,” 2023.
  * [36] A. Radford, J. W. Kim, C. Hallacy _et al._ , “Learning transferable visual models from natural language supervision,” in _Proceedings of the 38th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 8748–8763. [Online]. Available: https://proceedings.mlr.press/v139/radford21a.html
  * [37] A. Brohan, N. Brown, J. Carbajal _et al._ , “Rt-1: Robotics transformer for real-world control at scale,” in _arXiv preprint arXiv:2212.06817_ , 2022.
  * [38] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor, “Chatgpt for robotics: Design principles and model abilities,” Microsoft, Tech. Rep. MSR-TR-2023-8, February 2023. [Online]. Available: https://www.microsoft.com/en-us/research/publication/chatgpt-for-robotics-design-principles-and-model-abilities/
  * [39] M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in _Conference on Robot Learning_. PMLR, 2022, pp. 894–906.
  * [40] D. Shah, B. Osiński, S. Levine _et al._ , “Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action,” in _Conference on Robot Learning_. PMLR, 2023, pp. 492–504.
  * [41] J. Zürn, W. Burgard, and A. Valada, “Self-supervised visual terrain classification from unsupervised acoustic feature learning,” _IEEE Transactions on Robotics_ , vol. 37, no. 2, pp. 466–481, 2021.