# Sensitivity of non-radiative cloud-wind interactions to the hydrodynamics
solver

Joey Braspenning,1 Joop Schaye,1 Josh Borrow2 and Matthieu Schaller3,1
1Leiden Observatory, Leiden University, PO Box 9513, NL-2300 RA Leiden, The
Netherlands
2Department of Physics, Kavli Institute for Astrophysics and Space Research,
Massachusetts Institute of Technology, Cambridge, MA 02139, USA
3Lorentz Institute for Theoretical Physics, Leiden University, PO Box 9506,
NL-2300 RA Leiden, the Netherlands E-mail<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

Cloud-wind interactions are common in the interstellar and circumgalactic
media. Many studies have used simulations of such interactions to investigate
the effect of particular physical processes, but the impact of the choice of
hydrodynamics solver has largely been overlooked. Here we study the cloud-wind
interaction, also known as the “blob test”, using seven different
hydrodynamics solvers: Three flavours of SPH, a moving mesh, adaptive mesh
refinement and two meshless schemes. The evolution of masses in dense gas and
intermediate-temperature gas, as well as the covering fraction of
intermediate-temperature gas, are systematically compared for initial density
contrasts of 10 and 100, and four numerical resolutions. To isolate the
differences due to the hydrodynamics solvers, we use non-radiative simulations
without physical conduction. We find large differences between these methods.
SPH methods show slower dispersal of the cloud, particularly for the higher
density contrast, but faster convergence, especially for the lower density
contrast. Predictions for the intermediate-temperature gas differ particularly
strongly, also between non-SPH codes, and converge most slowly. We conclude
that the hydrodynamical interaction between a dense cloud and a supersonic
wind remains an unsolved problem. Studies aiming to understand the physics or
observational signatures of cloud-wind interactions should test the robustness
of their results by comparing different hydrodynamics solvers.

###### keywords:

hydrodynamics – instabilities – turbulence – ISM: clouds – Galaxy: kinematics
and dynamics – galaxies: evolution

††pubyear: 2021††pagerange: Sensitivity of non-radiative cloud-wind
interactions to the hydrodynamics solver–Sensitivity of non-radiative cloud-
wind interactions to the hydrodynamics solver

##  1 Introduction

Interactions between cold, dense clouds moving at high velocity through a hot
and dilute medium are common in turbulent layers within the circumgalactic
medium (CGM) (Tumlinson et al., 2017). Such layers are hypothesized to
originate in cold, dense clouds finding themselves entrained in a hot, dilute
flow. Such a configuration could, for example, arise from a hot, feedback-
driven wind interacting with pre-existing or newly formed clouds, or if gas
clumps fall through a hot, hydrostatic halo. The turbulent, intermediate-
temperature gas created when the cloud is stripped may find itself, under the
right conditions, near the peak of the cooling curve and can condense rapidly
to the cold phase. This process allows for the cloud’s total cold gas mass to
grow (e.g. Gronke & Oh, 2020a, 2018).

The precise fractions of cold and intermediate-temperature gas formed in the
cloud-wind interactions are crucial towards making observational predictions
(e.g. Shelton et al., 2012; de la Cruz et al., 2020). The robustness of these
predictions is not aided by the theoretical uncertainty about whether clouds
fragment into a mist (McCourt et al., 2018) or whether a few large clouds
contribute the bulk of gas flux (e.g. Gronke & Oh, 2020b; Vijayan et al.,
2020).

In the past few years a tremendous amount of work has gone into improving the
physical models included in simulations of cloud-wind interactions to study
the effects of magnetic fields (e.g. Sparre et al., 2020, 2019), conduction
(Kooij et al., 2021; Armillotta et al., 2017, e.g), rich multiple cloud
systems (e.g. Banda-Barragán et al., 2020, 2021) and cosmic ray driven winds
(e.g. Brüggen & Scannapieco, 2020). The fractions of cold gas as a function of
time differ with the specific choice of model, but generalised analytical
considerations gave rise to criteria governing the long-term survival of the
cold cloud (e.g. Li et al., 2020; Kanjilal et al., 2021). An intuitively
reasonable conclusion arose: if the gas cools faster than it gets disrupted,
the cloud will survive. While different authors would agree on this general
statement, the precise cloud survival criteria they obtain differ, and are in
some cases mutually exclusive. We will argue that many of the differences
could actually originate from the diffent hydrodynamics solvers used.

When solving a hydrodynamics problem without an analytical solution, such as
the cloud-wind problem (also called the “blob test” in the hydrodynamics
literature), astrophysicists have to rely on numerical simulations. Though
these simulations should, in principle, all solve the same hydrodynamics
equations, their methods differ widely. The underlying assumption when
comparing relatively small, idealized simulations such as those of the cloud-
wind problem, is that the methods will produce the same result given a
sufficiently high resolution. This expectation is reasonable, since common
methods are all rigorously tested and are known to be well behaved.

However, the cloud-wind interaction poses some unique challenges which can
give rise to different problems for different methods. For example, pressure-
confined clouds with high density contrasts are particularly challenging for
smooth particle hydrodynamics (SPH) methods, whereas the symmetry inherent to
the problem may pose a problem for grid methods. Furthermore, SPH and grid
codes tend to respectively under- and overestimate the mixing rate of
different gas phases (Hopkins, 2015).

Notable work has been done specifically on the comparison between SPH and grid
methods (e.g. Agertz et al., 2007). Motivated by those findings, many
improvements have been made to SPH methods to reduce spurious surface tension
using artificial diffusion and conduction. As the cloud-wind interaction
involves many phenomena for which no analytical solution is available, there
has been a considerable effort to deconstruct it into simpler, better
understood constituent pieces. For example, extensive comparisons for the
Kelvin-Helmholtz (KH) instablility test using well-posed initial conditions
have shown that, when using higher order kernels, SPH, grid and spectral
methods agree both in their resulting morphology and quantitative evolution
(e.g. Tricco, 2019; Lecoanet et al., 2016; McNally et al., 2012).

Moreover, new hydrodynamics methods have been adopted with the aim of bridging
the gap between a static Eulerian grid and Lagrangian SPH. Codes that have
become widely used in astrophysics are Arepo (Springel, 2010) and gizmo
(Hopkins, 2015). Arepo uses Voronoi tiling to create cells around quasi-
Lagrangian cell generating points, with the goal of combining the flux
exchange of grid methods with the conservative properties of SPH methods. The
two gizmo methods, meshless finite mass (mfm) and meshless finite volume
(mfv), in effect build on the Voronoi principle by introducing smoothed
boundaries between neighbouring cells, much like in SPH (Vila, 1999). The mfm
flavour does this by conserving the mass of volume elements at the time of
flux exchange, whereas the mfv methods conserves their volume.

Rapid development in the SPH community has also led to a large number of new
flavours, all using slightly different diffusion and viscosity prescriptions
in order to perform well on a suite of hydrodynamics tests. For example,
gasoline-2 (Wadsley et al., 2017) uses non-conservative viscosity terms which
mimick flux exchange in grid methods. However, even though modern
hydrodynamics methods are converging on a standard set of analytically
solvable test problems, the cloud-wind interaction has only rarely been
compared beyond approximate morphological similarity. Despite the growing
literature using the cloud-wind interaction to model additional physics and
make observational predictions, the uncertainties in the non-radiative
hydrodynamic result have received little attention.

To investigate the uncertainty in the results of simulations of the cloud-wind
problem, and give quantitative measures of their magnitude, both in terms of
theoretical quantities and observational characteristics, we carefully study
seven different hydrodynamics methods with the same initial conditions. These
seven cover all methods used in existing studies into the wind-cloud problem
we are aware of and should hence provide a comprehensive picture of the
differences and similarities. They are:

  1. 1.

Modern Smooth Particle Hydrodynamics (SPH; three flavours)

  2. 2.

SPH-ALE (two flavours: mfm, mfv)

  3. 3.

Moving Mesh (mm)

  4. 4.

Static grid with adaptive mesh refinement (amr)

Each of these methods has been used in large cosmological simulations and
extensively tested on a range of hydrodynamics test cases. All have advantages
and disadvantages, which are especially apparent in difficult problems such as
those with large amounts of rotation or vorticity.

Because the interaction between a dilute wind and a dense cloud has no
analytic solution, there is no independent benchmark to compare the results
with. Instead, parallels to other, simpler, problems are drawn and signs of
such behaviours are sought. For example, one looks for Kelvin-Helmholtz
instabilities at the edges of the cloud, even though the cloud is curved and
being pushed along which makes the comparison difficult

Here we use non-radiative hydrodynamics simulations to isolate differences due
to the hydrodynamics solvers. As our simulations are non-radiative and do not
include physical conduction, they should, in the limit of infinite resolution,
only show evolution in the gas densities and temperatures due to the pressure
gradients induced by the ram pressure exerted by the wind. Non-adiabatic
energy exchange between different phases only takes place at the microscopic
level, which is not modelled by any of these methods.

Volume elements in finite resolution simulations represent averages over many
microscopic phases, which can artificially give the appearance of a single
mixed phase. Such artificial mixing is inherent to grid and mesh methods, but
also occurs in SPH methods where artificial conduction is invoked to suppress
spurious surface tension. It is often implicitly assumed that the level of
mixing predicted by reasonably converged high-resolution simulations is
realistic. However, this may be optimistic given that the mixing is still
numerical as long as microscopic conduction is not modelled.

An interesting new avenue that we will explore, is to compare the level of
mixing in a lower resolution simulation to that in a higher resolution
simulation after coarse-graining the latter onto a lower-resolution grid for
analysis. The hope is then that as one increases the resolution, the coarse-
grained results converge even if the highest resolution simulation is still
not converged. This would indicate convergence of the volume-averaged phase
structure. However, we note that observational diagnostics such as emission
and absorption signatures depend on the microscopic rather than the coarse-
grained phase structure.

We study the level of deviation between these different methods and find that
it is large enough to have a major impact on predictions for observations, as
well as on the theoretical quantities typically measured in cloud-wind
interactions.

In section 2 we introduce the hydrodynamics methods and the set-up used in
this study. In section 3 we then compare the simulations using different
methods using a number of different metrics. First we compare the evolution of
dense mass in section 3.1, then we look at the amount of intermediate-
temperature gas in section 3.2, we consider what covering fraction this gas
produces in projection in section 3.3, and finally study the effect of coarse-
graining in sections 3.4. We conclude and make some recommendations in section
4.

Figure 1: Density slices through the midplane of the simulation box at
$t\approx 5t_{\text{cc}}$, for initial density contrasts $\chi=10$ (left) and
$\chi=100$ (right), at the highest resolution ($n=128^{3}$). Large
morphological differences can be observed.

##  2 Methods

In this work we simulate cloud-wind interactions using seven different
hydrodynamics solvers. All simulations are non-radiative and use identical
initial conditions in which the cold cloud and hot wind are in thermal
pressure balance. This relates their temperatures and densities as

$\frac{T_{\text{wind}}}{T_{\text{cloud}}}=\frac{\rho_{\text{cloud}}}{\rho_{\text{wind}}}\equiv\chi\,.$
(1)

where $\chi$ is the density contrast. The wind is set to have a Mach number
$\mathcal{M}=\frac{v_{\text{wind}}}{c_{\text{s,wind}}}=1.5$, with
$v_{\text{wind}}$ the wind speed and $c_{\text{s,wind}}$ the wind’s sound
speed. The wind and the cloud are initialised on a grid with respectively _N_
and $\chi^{1/3}$_N_ resolution elements on a side. The region into which the
cloud is inserted is then removed from the wind’s grid. Our simulations are
wind tunnels that are four times longer than they are wide and high. All
boundary conditions are periodic, a choice that we found to have no
significant effect on the results presented here. In our default set-up the
cloud has a radius of one-tenth of the box width. As we use pure
hydrodynamics, i.e. radiative cooling is disabled, the problem is scale
invariant. In the unit system used throughout we choose

$R_{\text{cloud}}=0.1\,.$ (2)

Two important time scales used throughout the results section are the cloud
crushing time and the drag time. The cloud crushing time

$t_{\text{cc}}=\frac{\sqrt{\chi}R_{\text{cloud}}}{v_{\text{wind}}}\,,$ (3)

with $v_{\text{wind}}$ the wind speed, is the time it would take an external
shock hitting the cloud’s edge to travel to the cloud centre, taking into
account the lower sound speed by a factor of $\sqrt{\chi}$ inside the cloud
due to its higher density. Though the cloud is not expected to have fully
disappeared by this time, it is a useful measure. In practice, clouds survive
for $\approx$10 $t_{\text{cc}}$.

The drag time

$t_{\text{drag}}=\frac{\chi R_{\text{cloud}}}{v_{\text{wind}}}\,,$ (4)

is the time it would take a cubical cloud to sweep up its own mass in wind
particles. For a spherical cloud this time is larger by $\frac{4}{3}$, which
is commonly ignored.

In this study we use three modern versions of SPH:

  1. 1.

The new density-energy implementation of Sphenix sph (Borrow et al., 2022)
integrated in the SWIFT simulation code (Schaller et al., 2016; Schaller et
al., 2018) , which was particularly designed to work well with sub-grid
physics modules for galaxy formation and includes limiters to achieve that.

  2. 2.

The SPH method gdf sph (Geometric Density Force) as implemented in SWIFT. This
implementation is based on the equations presented in the gasoline-2 methods
paper (Wadsley et al., 2017), using the same parameter values.

  3. 3.

The anarchy-pu sph pressure-energy method as implemented in SWIFT, which is
based on Schaller et al. (2015) (see also appendix A of Schaye et al. (2015)),
but using energy instead of entropy as its thermodynamics variable (Hopkins,
2013).

The mfm (meshless finite mass) and mfv (meshless finite volume) methods are
also used as implemented in SWIFT as the SPH-ALE method (Vila, 1999; Lanson &
Vila, 2008), based on the equations presented in the gizmo methods paper
(Hopkins, 2015). The mm (moving mesh) method in Arepo (Springel, 2010) and the
amr (static grid with adaptive mesh refinement) method in Athena++ (Stone et
al., 2020) complete our set of seven hydrodynamics solvers.

The mfm, mfv and amr methods all use the HLLC Riemann solver, while the mm
method uses an exact Riemann solver. The amr method uses a second order
Piecewise Linear Method (PLM) reconstruction of the primitive variables, and
the second-order accurate van Leer predictor-corrector scheme. To ensure
results independent of the chosen smoothing kernel, all SPH flavours use the
same kernel (Quintic spline (M6)) and the same number of neighbours
($\eta=1.23$, 83.49 $N_{\text{ngb}}$) (see Dehnen & Aly (2012)).

There are three major differentiators between these three SPH methods. The
first is the equation of motion that they choose to solve. Sphenix sph and
anarchy-pu use different equations of motion derived from the same Lagrangian
(Hopkins, 2013), with different choices for the smoothed field. gdf sph
explicitly symmetrises the equation of motion to remove spurious forces
generated at contact discontinuities, yielding yet another equation of motion.

The choice of the smoothed field, and hence the way the density is calculated,
is the second differentiator. For Sphenix sph the density is computed as a
smoothed quantity as a sum over neighbours. Instead, for anarchy-pu sph the
pressure is smoothed. In comparison to the Sphenix sph equation of motion,
this smoothed pressure formulation ensures that at points where there is an
instantaneous jump in the internal energy (i.e. at the edge of the cloud) the
pressure forces vary smoothly. If there is an associated increase in density,
such that the pressure is intended to be uniform, then a smoothed density SPH
can produce spurious repulsive forces (also known as surface tension) (e.g. Hu
et al., 2014). In contrast, the smoothed pressure formulation does not create
these as it ensures that information is used from both inside and outside of
the cloud to construct the pressure field. Although this can lead to
unintended time-stepping issues (Borrow et al., 2021), it is highly effective
at removing these instabilities.

The approach taken in gdf sph is to modify the equation of motion from the
smoothed density formulation to reduce the impact of contact discontinuities.
Their new equation of motion has been shown to reduce surface tension
significantly (Wadsley et al., 2017, Fig. 1), but is no longer explicitly
derived from a Lagrangian.

The third differentiator is the choice for artificial conduction and
viscosity. Several prescriptions are present in the literature, and each of
these methods chooses a different combination which they find to work best
with their implementation.

Throughout this work, all codes are run using their fiducial time-stepping,
and where applicable the same CFL number has been used.

We have striven to make all initial conditions as similar as possible between
all methods. All methods that are integrated in SWIFT use the same initial
condition files, and the relevant parts of the initial condition generation
code for Arepo are directly copied from the SWIFT equivalent. Because of the
spherical shape of our cloud, there are some small and unavoidable differences
at the interface of the cloud between the amr method and all others. In the
amr method we allow two levels of adaptive mesh refinement based on the mass
in each cell. This introduces some amount of splitting and merging of
resolution elements that is not possible in any of the other methods. However,
the existence of adaptive smoothing lengths and cell sizes should make such
differences small. In the mm method, we also allow cell refinement and de-
refinement based on cell mass, but in practice this is only relevant at very
early times on the cloud boundaries. The density contrast of the cloud in the
mm method is generated by having a larger number of cells, rather than by
changing the mass in each cell. This is entirely analogous to the approach in
the sph, mfm and mfv methods.

We model cloud-wind interactions with two different initial density contrasts
each run at four different resolutions. Both these density contrasts,
$\chi=10$ and $\chi=100$, may be high compared to those produced by realistic
physical processes for clouds with sharp edges, but they are representative of
the literature. We indicate the resolution by the number of wind particles or
volume elements ($n$) per short side length of the simulation volume. In the
elongated direction, four times that number is present. This brings the total
number of wind particles or volume elements to $4n^{3}$. In the case of non-
particle based methods, the initial centres of volume elements or mesh
generating points are placed at the same locations as the particles in SPH. We
make use of the resolutions $n=\\{16,32,64,128\\}$. Given the cloud radius,
this translates to $nR_{\text{cloud}}=\\{1.6,3.2,6.4,12.8\\}$ wind particles
per cloud radius111It is not clear that choosing a fixed number of particles
or volume elements results in the same effective resolution. However, the
resolution steps are the same for all methods, and hence the relative
differences when changing the resolution are not affected by this
uncertainty..

##  3 Results

Figure 1 shows the disruption of the cloud in progress for the seven different
hydrodynamics solvers at the highest resolution. The lower density contrast
($\chi=10$) is shown in the left column and the higher density contrast
($\chi=100$) in the right column. Large differences between the different
methods can be readily observed. All images are made at time $t\approx
5t_{\text{cc}}$. It is clear that the problem is not scale free with respect
to $\chi$.

We observe that the morphological evolution of the gdf sph cloud is different
from that presented in Wadsley et al. (2017). However, those authors impose a
much faster wind ($\mathcal{M}=2.7$), causing ram pressure to dominate, and
use a kernel (Wendland $\mathrm{C^{4}}$) with more than double the number of
neighbours. Our results for mfm and mfv are smoother, though otherwise similar
in shape to those presented in (Hopkins, 2015) (who also use
$\mathcal{M}=2.7$), especially the vertical fragmentation in the mfv method is
reproduced. However, due to a different mach-number those results are not
directly comparable. Borrow et al. (2022) also present a cloud exposed to a
$\mathcal{M}=2.7$ wind. Compared with us, they find a qualitatively similar
evolution with a central cloud holding together and only limited fragmentation
or tail formation. Though all these code presentation papers used the higher
Mach number $\mathcal{M}=2.7$, we choose to adopt a lower value of
$\mathcal{M}=1.5$ in line with recent work exploring cloud-wind interactions
in more detail (e.g. McCourt et al., 2018; Gronke & Oh, 2018, 2020a; Sparre et
al., 2019, 2020; Kanjilal et al., 2021)

###  3.1 Mass of dense gas

Figure 2: Evolution of the mass of dense ($\rho>\rho_{\text{cloud}}/3$) gas
for different hydro schemes (colours and rows) and different resolutions
(linestyles). Initial density contrasts are $\chi=10$ (left) and $\chi=100$
(right) Figure 3: Evolution of the mass of intermediate-temperature gas for
different hydro schemes (colours and rows) and different resolutions
(linestyles). Initial density contrasts are $\chi=10$ (left) and $\chi=100$
(right)

The mass of dense gas in the simulation volume is defined as

$M_{\text{dense}}=\sum_{i}m_{i}\left[\rho_{i}>\frac{\rho_{\text{cloud}}}{3}\right]\,,$
(5)

where $\rho_{\text{cloud}}$ is the initial density of the cloud, $\rho_{i}$
the density of a particle or volume element and $m_{i}$ its mass. Figure 2
shows large differences in the evolution of the mass of dense gas between the
different hydrodynamics methods for the two different initial density
contrasts. The left and right columns show respectively the lower ($\chi=10$)
and higher ($\chi=100$) density contrasts. The top row compares all
hydrodynamics methods at the highest resolution ($n=128$). Each of the
remaining rows compares one method at all four resolutions. We remind the
reader that because none of the simulations model physical conduction, mixing
is numerical. However, because of the ram pressure from the wind, pressure
equilibrium is violated and the density may change adiabatically.

It immediately stands out that for the lower density contrast the cold cloud
disperses much more slowly for the simulation using gdf sph than for the other
methods. The results for simulations with the Sphenix sph, anarchy-pu sph,
mfm, amr and mm methods are all very similar, while mfv is an intermediate
case. The mfv method gives a slower decline in cloud mass, especially when the
cloud has already lost more than 50% of its original mass. Strikingly, this
much slower dispersal at late times only occurs for the two highest
resolutions. We note that for the gdf sph method the morphology of the dense
gas is extremely clumpy (see Fig. 1), with the clumps seemingly holding
together indefinitely, becoming co-moving with the wind. Again, this effect is
amplified by increasing the resolution, suggesting that more of the gas is
retained in dense clumps when the resolution is increased.

Interestingly, the convergence properties differ substantially between the
methods for the lower density contrast. Sphenix sph, anarchy-pu sph, amr and
mm seem quite well converged at the highest resolution, with Sphenix sph even
showing good convergence at lower resolutions. However, mfm, mfv and gdf sph
show much poorer convergence. These three methods show a continuous increase
in the mass of dense gas with increasing resolution. At the higher density
contrast, which one would assume to be more difficult, the convergence is
distinctly better.

For the higher density contrast it is apparent that simulations with the three
SPH methods have a much slower dispersal of the cold cloud. Whereas for the
other four methods the cloud is completely destroyed within $\approx 7$ cloud
crushing times, by that time the cloud (remnants) in the simulations using SPH
methods still retain 40-70% of their original mass. Though results from the
other four methods seem similar when compared to SPH, the differences are non-
negligible. The mfm simulation disperses the cloud the quickest, having lost
half the cloud mass 25% faster than the amr simulation, which has the longest
cloud survival. The mfv and mm simulations are very well matched, which is
expected given that the mfv method should be similar to the mm method bar
having smoothed cell boundaries. However, this level of similarity contrasts
with the large difference at late times between the mfv and mm simulations for
the lower density contrast. We note that for the lower density contrast the
threshold for dense gas is much closer to the wind density, making it more
likely that fluctuations in the background density contribute to the mass of
the dense gas.

It is well known that amr methods suffer from artificical mixing due to gas
being exchanged across cell boundaries and mixed at the resolution scale (e.g.
Springel, 2010). This is especially relevant near large density contrasts as
they can be quickly diluted with mass being pushed from one cell to the next.
This is confirmed by this method dispersing the cloud more quickly for the
higher density contrast and more slowly for the lower density contrast,
compared to the three methods with moving cells. Though the mm method has a
Voronoi mesh which moves with the bulk flow, flux exchanges across cell
boundaries are still expected in turbulent regions (e.g. Mandelker et al.,
2021). Since the mfm and mfv methods are similar to the mm method, but have
smoothed boundaries, one would expect the mm method to suffer more from
artificial mixing and hence to disperse the cloud more quickly. We do not see
this reflected in the results for the higher density contrast. The results
from the mm method are well matched to the mfv method and cloud dispersal is
slightly slower than for the mfm method.

For both density contrasts, the differences between the amr and mm methods at
the highest resolution are surprisingly small. Since the mm method has a
moving mesh, one would expect the artifical mixing and hence cloud dispersal
to be slower compared to the amr method. It should also be noted that the two
methods both seem well converged for this metric.

It is worthwhile to compare the behaviour seen for the higher density contrast
with the images in Fig. 1, where it can be seen that the morphological
evolution of the clouds is very different, even for simulations that look very
similar in Fig. 2. We postulate that the slower dispersal of the cloud in the
three SPH methods is due to the very sharp and high density contrast, which is
hard to resolve for these methods and leads to a much slower decline in cloud
mass. Having the SPH method in the fold for the lower density contrast
strengthens our conviction that it is simply unable222The term ”unable” is a
misnomer in this context, because in the absence of conduction the formally
correct solution is zero mixing. to handle a high and sharp density contrast.
We note that the anarchy-pu sph method shows cloud destruction morphologies
more akin to Eulerian methods.

The gdf sph method, which for the higher density contrast is positioned
between Sphenix sph and all others, gives a much longer dispersal time for the
lower density contrast. The difference with the other methods also increases
with increasing resolution. This might be due to the many cloudlets produced
in this method (see Fig. 1) being above $\rho_{\text{cl}}/3$ for the lower
density contrast, but below it for the higher density contrast where the
initial cloud density is higher. Hence, they would only be included in the
lower density contrast’s definition of dense gas.

We have tested sphenix sph with higher values for the artificial conduction
and viscosity parameters. This made cloud dispersal faster by diffusing the
edges of the confined cloud to the background density. The morphology remained
unchanged and the dense mass evolution was, for the higher density contrast,
still distinctly different from the non-SPH methods. In an effort to alleviate
any potential problems with neighbour finding in the SPH codes near the edge
of the cloud, we also experimented with an increased particle number in the
wind while keeping its density fixed, which implies lower-mass wind particles.
Even when using the same SPH particle number density for the wind as for the
cloud’s interior, differences compared to the default set-up were minimal.

Finally, we note some morphological differences evident from Figure 1. For the
higher density contrast the mfm and mfv simulations create a noticeably smooth
density distribution with long wispy tails. The amr simulation shows features
associated with a regular grid, being fully symmetric and having a horizontal
component aligned with the cloud’s original centre. We find the cloud in the
mm simulation to be the most disturbed. Even though it retains a dense core of
gas, the filamentary tails are much less smooth and show significant
substructure. The very different morphologies between the three SPH methods
demonstrate the large effect of different artificial viscosity and conduction
prescriptions. For the Sphenix sph simulation the cloud slowly diffuses at the
edges without the clear break-up seen in most other methods. The gdf sph
simulation does break the cloud, but yields a very clumpy distribution, unlike
the smoother tails seen in the mfm, mfv, mm and amr methods. The anarchy-pu
sph method does show smoother tails, but also retains a relatively large dense
cloud which seemingly only sheds small amounts of mass through its tails
without any shattering.

###  3.2 Mass of intermediate-temperature gas

The mass at intermediate temperatures is defined as

Figure 4: Evolution of the covering fraction of intermediate-temperature gas
for different hydro schemes (colours and rows) and different resolutions
(linestyles). Initial density contrasts are $\chi=10$ (left) and $\chi=100$
(right)

$\begin{split}M_{\text{mix}}=\sum_{i}m_{i}\Bigl{[}\log_{10}(T_{\text{mix}})-&\frac{1}{4}\left(\log_{10}(\chi)\right)\\\
&<\log_{10}(T_{i})<\\\
&\log_{10}(T_{\text{mix}})+\frac{1}{4}\left(\log_{10}(\chi)\right)\Bigr{]}\,,\end{split}$
(6)

where $T_{i}$ is the temperature of a particle or volume element, and
$T_{\text{mix}}$ the geometric mean temperature

$T_{\text{mix}}=\left(T_{\text{cloud}}T_{\text{wind}}\right)^{1/2}=T_{\text{wind}}\chi^{-1/2}=T_{\text{cloud}}\chi^{1/2}\,.$
(7)

This corresponds to the mass within half the logarithmic temperature range
between the cold cloud and the hot wind centred on the geometric mean
temperature.

Large differences in the mass of gas with intermediate temperatures would
indicate possible large discrepancies in predictions for observable
diagnostics of mixing layers.

As our setup is in thermal pressure balance, there is a direct correspondence
between the density and the temperature contrasts. Gas with a density
$\hat{\chi}$ higher than the initial wind density will have a temperature of
$\frac{1}{\hat{\chi}}$ times the initial wind temperature. For the higher
density contrast case studied here $(\chi=100)$, our intermediate-temperature
range yields temperatures that do no overlap with the density range which
counted towards the mass of dense gas in §3.1, however, for the lower density
contrast $(\chi=10)$ the upper half of this range overlaps with the dense gas
density range.

In the absence of radiative cooling, the only mechanisms that can produce this
intermediate-temperature gas are numerical mixing and adiabatic expansion.
Hence, the peak occurs when the dense gas mass decreases rapidly.

In Figure 3, similar to Fig. 2, we see the dichotomy between the lower and
higher density contrasts. For $\chi=10$ there is a large amount of variation,
whereas for $\chi=100$ all methods are in reasonable agreement except the
three SPH simulations.

For the sphenix sph, anarchy-pu sph, mfm and amr methods, the mass of
intermediate-temperature gas peaks at roughly the same time, but the peak is
much higher for the amr method. The mm simulation peaks earlier and the mfv
later. gdf sph is an outlier, showing a slow but persistent increase in the
mass of intermediate-temperature gas.

Figure 5: Projection of the mass at intermediate temperatures for initial
density contrast $\chi=10$ at time $t\approx 9\mathrm{t_{\text{cc}}}$.
Morphologies differ widely, ranging from fuzzy balls and inky blotches to
symmetric grid-like artifacts.

For the higher density contrast the four non-SPH methods produce similar
results. The many cloudlets seen in the gdf sph simulations are mostly in this
intermediate-temperature regime, giving rise to a large amount of mixed gas,
far greater than produced in simulations using any other method. The other two
SPH methods match well except at late times when anarchy-pu sph shows a drop
in intermediate-temperature gas, whereas the Sphenix sph method maintains a
constant level. This is due to anarchy-pu sph having lost most of the cold
cloud by this time, ending the creation of new intermediate-temperature gas.

The convergence with resolution is worse than for the dense gas mass, but
still reasonable between the two highest levels. Indicative of the articifical
mixing in the amr and mm methods is the decrease in peak height with
increasing resolution, especially apparent for the higher density contrast. A
higher resolution should give rise to less artificial mixing, and hence allow
a sharp density contrast to survive for longer.

For the lower density contrasts the two highest resolutions are very similar
for the gdf sph and amr simulations and still quite similar for the Sphenix
sph, anarchy-pu sph, mfm and mfv methods. Only the mm method seems poorly
converged.

Figure 6: Projection of the mass at intermediate temperatures for initial
density contrast $\chi=100$ at time $t\approx 5\mathrm{t_{\text{cc}}}$. Very
different morphologies are apparent, even among those methods for which the
covering fractions are similar (e.g. compare mm and mfv). The grid-like
artifacts from the amr method are clearly visible.

###  3.3 Covering fraction of intermediate-temperature gas

Figure 7: Evolution of the mass of dense gas (top) and intermediate-
temperature gas (bottom) for the Sphenix sph hydrodynamics method. Initial
density contrasts are $\chi=10$ (left) and $\chi=100$ (right). The black lines
show the evolution of the different gas phases after coarse-graining to the
common resolution of $32^{3}$ grid-cells.

Whereas Figure 3 showed the mass in the intermediate-temperature gas phase, we
will now look at the projected distribution of that gas. This is an
observationally relevant metric of cloud-wind interactions. Some tentative
efforts have already been made to translate the rather idealised cloud-wind
interaction to expectations for observable absorption lines (e.g. de la Cruz
et al., 2020). Given the large variety of evolutionary paths taken by the
cloud depending on the hydrodynamics method, we explore, in a simplified
setting, what impact the hydrodynamics solver might have on observable
statistics such as absorption lines. To this end, we project all intermediate-
temperature gas (defined by eq. 6) along the long axis of our simulation box.
Counting the fraction of pixels that have a column density of intermediate-
temperature gas higher than 1% of the original cloud column density333This is
an arbitrary choice and does not reflect the sensitivity limits encountered in
particular observations., we present the results as a function of time,
resolution and hydrodynamics method in Fig. 4.

We observe a large variety in covering fractions for the different methods,
with very different convergence characteristics. For the higher density
contrast, methods which are similar in terms of the evolution of the mass of
intermediate temperature gas, namely mfm, mfv, mm, and amr, produce very
different covering fractions. Not only does the height of the peak vary, the
peaks are also offset by considerable amounts.

The important metric in this figure is the combination of peak width and
height. A higher peak covering fraction indicates a larger likelihood of
detecting any material in a hypothetical absorption line study, but for a
narrow peak the time-window would be short. We remind the reader that our
initial cloud occupies only 3% of the projected area of the box. With some
methods reaching peak covering fractions of well over 50%, these must be due
to low-density fragments as is also evident from Fig. 1.

Representative images of the projected clouds after $6t_{\text{cc}}$ are shown
in Fig. 5 for $\chi=10$ and Fig. 6 for $\chi=100$. The morphology of the
projected intermediate-temperature gas differs significantly. The amr method
has very symmetric grid-like characteristics. gdf sph and anarchy-pu sph
clearly show that additional methods to reduce artificial surface tension
produce more diffuse projected distributions compared to Sphenix sph. However,
gdf sph also shows signs of retaining a relatively denser core. mfv and mm are
quite comparable in the total extent of their gas distribution, with mfm
showing more clumping, and for $\chi=100$ mm retaining a denser core. Except
for the Sphenix sph and amr methods, none of the projected distributions are
symmetric. The amr method shows a grid symmetry, and Sphenix sph shows the
effects of artificial surface tension retaining a near perfect spherical
symmetry, but none of the other methods shows either of these symmetries.
Given that the initial conditions are set up on a grid, this is unexpected. We
tried to explicitly enforce symmetry by only generating half of the initial
conditions and creating the other half by mirroring the first, but this did
not lead to appreciable more symmetric morphologies.

Figure 8: Evolution of the mass of dense gas (top) and intermediate-
temperature gas (bottom) for the gdf sph hydrodynamics method. Initial density
contrasts are $\chi=10$ (left) and $\chi=100$ (right). The black lines show
the evolution of the different gas phases after coarse-graining to the common
resolution of $32^{3}$ grid-cells. When coarse-grained, the many cloudlets
produced by gdf sph contributing to both the dense gas (at their cores) and
intermediate-temperature gas (at their edges) average out with the hot wind,
leading to an evolution more comparable to that predicted by other methods.

###  3.4 Coarse-graining to a common resolution

A new avenue in exploring the evolution of idealised hydrodynamics simulations
is to compare the level of mixing in lower-resolution simulations to that in
higher-resolution simulations after coarse-graining the latter onto a lower-
resolution grid. To this end we have devised a 3D grid which is comoving with
the median position of the dense gas in a simulation. On this 3D grid, we
compute mass-weighted quantities for two different grid resolutions. These two
resolutions correspond to the two lowest resolutions employed in this work,
$16^{3}$ and $32^{3}$ grid cells. No smoothing is applied in this procedure,
volume elements whose centres fall within a grid cell are fully counted in
that single grid cell.

We find that in general, the coarse-grained results differ from the
corresponding lower-resolution simulation, though they are more similar to
them than the non-coarse-grained higher-resolution simulations. As a
representative comparison with coarse-grained results, we show in Fig. 7 a
comparison between Sphenix sph at all resolutions and when coarse-grained onto
a regular grid with $32^{3}$ grid cells. Though there are clear differences
between the coarse-grained and original resolution results, they agree
qualitatively. For the higher density contrast ($\chi=100$), we find that
there is a consistent offset between the converged full resolution simulation
and the coarse-grained results, which show similar convergence. The coarse-
grained mass of dense gas is slightly lower at all times, whereas the coarse-
grained mass of intermediate-temperature gas is slightly higher compared to
the full resolution simulations. This is in line with expectations, where
dense gas averaged over a finite volume might drop below the density
threshold, and likewise hot and cold gas artificially mix on the coarse-
grained grid resolution, yielding more intermediate-temperature gas. Similar
small deviations are found for all but the gdf sph hydrodynamics method.

For gdf sph we find differences which change the interpretation of the
results. As shown in Fig. 8, for the lower density contrast ($\chi=10$), the
simulation employing this hydrodynamics method retain a large amount of dense
gas at the highest resolution, but after coarse-graining this onto the lower-
resolution grid, the dispersal time is in line with the other hydrodynamics
methods. We attribute this to the large amount of tiny cloudlets formed using
this method (see Fig. 1). When averaging over a grid cell containing a
cloudlet and dilute gas, the average density will be below the dense gas
threshold.

For intermediate-temperature gas it is not just the time scale which changes,
but also the shape of the curve. Whereas the original gdf sph shows a
continuous increase of intermediate-temperature gas with time, unlike the
peaked time evolution seen for other methods, after coarse-graining peaks do
appear (see Fig. 8). The time at which the amount of intermediate-temperature
gas peaks now also coincides with the time at which dense gas is disappearing
the fastest. Just as for the dense mass, we attribute this to the survival of
many small cloudlets. These fall into the intermediate-temperature regime at
late times, but when averaged over a finite volume, their temperature no
longer satisfies the criterion for intermediate-temperature gas.

These coarse-grained evolutionary tracks would be important for hypothetical
observations measuring volume-averaged densities or temperatures. However,
because the ionisation balance and molecule abundances depend on the resolved
density and temperature, it is the original non-coarse-grained result that is
most relevant for observations. This is an important distinction when making
predictions for observational diagnostics, since we have shown that the two
approaches can yield qualitatively different results.

##  4 Conclusions

We conducted a systematic study of the non-radiative 3D cloud-wind problem
using seven different hydrodynamics solvers. Three SPH methods: sphenix sph ,
gdf sph, anarchy-pu sph; two meshless methods: meshless finite mass (mfm), and
meshless finite volume (mfv); a static grid with adaptive mesh refinement
(amr); and a moving mesh (mm). For all methods we conducted simulations for
two initial density contrasts: $\chi=\\{10,100\\}$, and four resolutions:
$nR_{\text{cloud}}=\\{1.6,3.2,6.4,12.8\\}$ wind particles per cloud radius. We
simulate a static spherical cloud that is initially in pressure equilibrium
and is impacted by a wind with Mach number $\mathcal{M}=1.5$. As we do not
include physical conduction, mixing is numerical. However, because the ram
pressure from the wind breaks pressure equilibrium, densities and temperatures
can evolve adiabatically.

We compared the mass of dense gas and the mass of intermediate-temperature gas
as a function of time, as well as the covering fraction of the intermediate-
temperature gas. We find that the differences between hydrodynamics methods
are substantial, and are highly dependent on the quantity of interest.
Summarised, we find:

  1. 1.

For $\chi=100$ the differences are much more pronounced compared to $\chi=10$.
For the lower density contrast most methods stretch the cloud into a thin
shell before fragmenting it, and all tails are relatively short (Fig. 1). For
$\chi=100$ in the simulations using the mfm, mfv, mm and amr methods the cloud
fragments after an initial compression, but the resulting morphologies differ
widely. The mfm and mfv methods predict tails that are longer than for mm and
amr. The three SPH methods show a very different evolution, with a more
pronounced core of dense gas at $t=5t_{\mathrm{cc}}$. For Sphenix sph the
cloud survives as a single entity, only losing mass through slow diffusion.
For gdf sph the cloud fragments into many cloudlets that experience only
limited further evolution. For anarchy-pu sph the cloud ablates into a smooth
tail similar to the four non-SPH methods.

  2. 2.

The evolution of the mass of dense gas (Fig. 2) differs greatly with the
initial density contrast and hydrodynamics method. For $\chi=10$ the mass of
dense gas is reduced by an order of magnitude within $\approx 10$ cloud
crushing times, except for gdf sph which predicts a much slower decline. At
the highest resolution level, the other six methods agree, with only mfv
deviating slightly at late times. The cloud lifetime increases with
resolution. The amr, mm and anarchy-pu sph methods converge at the one-but-
highest resolution level, Sphenix sph already converges at lower resolution.
The two meshless methods (mfm and mfv) and gdf sph do not show any signs of
convergence. For the higher initial density contrast ($\chi=100$) the
differences in lifetime are even more pronounced. For mfm, mfv, amr and mm the
lifetime is half that for the lower density contrast, and the convergence is
better. The convergence of the sph methods also improves compared with the
lower density contrast, especially for gdf sph. However, whereas the gdf sph
lifetime decreases with density contrast, for anarchy-pu sph and especially
Sphenix sph the cloud survives much longer.

  3. 3.

The mass of intermediate-temperatures gas peaks at the time when the cloud
loses mass most rapidly (Fig. 3). As the time of destruction differs between
the methods, the peak in this quantity occurs at different times. For the
higher density contrast ($\chi=100$) the height of the peak is similar for
mfm, mfv, mm and amr, but significantly lower for sphenix and anarchy-pu sph
due to the tardy disappearance of the cloud in those methods. gdf sph shows a
continuous rise in the mass of intermediate-temperature gas, which is ascribed
to the fragmentation into many long-lived cloudlets. However, after coarse-
graining the higher-resolution gdf sph simulations to a lower common
resolution, a peak in the mass of intermediate-temperature gas becomes
apparent (Fig. 8), which is more in line with the other hydrodynamics methods.
For $\chi=10$ the longer survival of dense gas for the mfv method translates
into a significant offset in this metric. The amr method produces more
intermediate-temperature gas than any other method, which might be explained
by artificial mixing in grid-based simulations. Whereas amr and mm converged
to the same answer for the dense gas, they are very different in this metric,
with mm producing less intermediate-temperature gas and for a shorter time.

  4. 4.

The evolution of the covering fraction of intermediate-temperature gas, a
metric relevant for the probability of observing intermediate-ionisation
stages in absorption, differs greatly both in terms of the time and width of
the peak (Fig. 4). Methods which yield similar masses of intermediate-
temperature gas can have very different covering fractions. For the lower
density contrast the covering fractions decline relatively slowly with time,
compared to the much faster changes in the mass contributing to this covering
fraction. mfv and gdf sph are again the outliers, predicting a much slower
decline. For the higher density contrast, the peaks in the covering fraction
are more pronounced, but peak heights differ by more than 50%, something that
is not visible in the mass of intermediate-temperature gas. The covering
fraction peaks after the time when most intermediate-temperature gas is
present. The three sph methods predict much larger covering fractions at late
times. For the lower density contrast all methods except mm and amr are well
converged at the one-but-highest resolution level, with Sphenix sph and mfv
showing convergence at lower resolution. The convergence is generally better
for the higher density contrast, except for gdf sph and mm.

Fifteen years after Agertz et al. (2007) provided the comparison between an
SPH and a grid code for the cloud-wind interaction, we conclude that at a
lower density contrast ($\chi=10$) most methods now agree well on the
evolution of dense gas, though at a higher density contrast ($\chi=100$)
significant differences remain, also between non-SPH methods. We introduced
new metrics to study the properties of mixed intermediate-temperature gas and
concluded that for those quantities predictions from cloud-wind interaction
studies are still marred by numerical uncertainties. Even among the three
methods which yield reasonably similar results, i.e. mfm, mm and amr, the
relevant timescales differ by a factor two. For $\chi=10$ mfv and gdf sph
predict a much slower dispersal of the dense gas, and a corresponding later
peak in the mass of intermediate-temperature gas. For $\chi=100$ all sph
methods show a slower dispersal, though they also differ strongly from each
other. The covering fractions of intermediate-temperature gas differ even
more, with all methods showing different peak times and evolution. Hence, we
conclude that the effects of the inclusion of additional physics, such as
radiative cooling, MHD or physical conduction, should be studied by comparing
simulations using both the same and different hydrodynamics methods. Ideally,
different hydrodynamics methods would be used for any cloud-wind interaction
study to both quantify uncertainties and offer greater understanding.

## Acknowledgements

The authors thank Bert Vandenbroucke for technical help in realising this
project. The authors gratefully acknowledge funding from the Netherlands
Organization for Scientific Research (NWO) through Vici grant number
639.043.409. This work used the DiRAC@Durham facility managed by the Institute
for Computational Cosmology on behalf of the STFC DiRAC HPC Facility
(www.dirac.ac.uk). The equipment was funded by BEIS capital funding via STFC
capital grants ST/K00042X/1, ST/P002293/1, ST/R002371/1 and ST/S002502/1,
Durham University and STFC operations grant ST/R000832/1. DiRAC is part of the
National e-Infrastructure. The research in this paper made use of the SWIFT
open-source simulation code (http://www.swiftsim.com, (Schaller et al., 2018))
version 0.9.0. This paper made use of the following python packages:
swiftsimio (Borrow & Borrisov, 2020), numpy (Harris et al., 2020), yt (Turk et
al., 2011), and matplotlib (Hunter, 2007).

## Data Availability

All code and initial conditions used to generate the simulations within swift
are made part of swift. As the simulations presented in this paper are
relatively small simulations that can easily be repeated using the publicly
available initial conditions, the data is not made immediately available.

## References

  * Agertz et al. (2007) Agertz O., et al., 2007, Monthly Notices of the Royal Astronomical Society, 380, 963
  * Armillotta et al. (2017) Armillotta L., Fraternali F., Werk J. K., Prochaska J. X., Marinacci F., 2017, Monthly Notices of the Royal Astronomical Society, 470, 114
  * Banda-Barragán et al. (2020) Banda-Barragán W. E., Brüggen M., Federrath C., Wagner A. Y., Scannapieco E., Cottle J., 2020, Monthly Notices of the Royal Astronomical Society, 499, 2173
  * Banda-Barragán et al. (2021) Banda-Barragán W. E., Brüggen M., Heesen V., Scannapieco E., Cottle J., Federrath C., Wagner A. Y., 2021, Monthly Notices of the Royal Astronomical Society
  * Borrow & Borrisov (2020) Borrow J., Borrisov A., 2020, Journal of Open Source Software, 5, 2430
  * Borrow et al. (2021) Borrow J., Schaller M., Bower R. G., 2021, Monthly Notices of the Royal Astronomical Society, 505, 2316
  * Borrow et al. (2022) Borrow J., Schaller M., Bower R. G., Schaye J., 2022, Monthly Notices of the Royal Astronomical Society, 511, 2367
  * Brüggen & Scannapieco (2020) Brüggen M., Scannapieco E., 2020, The Astrophysical Journal, 905, 19
  * Dehnen & Aly (2012) Dehnen W., Aly H., 2012, Monthly Notices of the Royal Astronomical Society, 425, 1068
  * Gronke & Oh (2018) Gronke M., Oh S. P., 2018, Monthly Notices of the Royal Astronomical Society, 480, L111
  * Gronke & Oh (2020a) Gronke M., Oh S. P., 2020a, Monthly Notices of the Royal Astronomical Society, 492, 1970
  * Gronke & Oh (2020b) Gronke M., Oh S. P., 2020b, Monthly Notices of the Royal Astronomical Society, 494, L27
  * Harris et al. (2020) Harris C. R., et al., 2020, Nature, 585, 357
  * Hopkins (2013) Hopkins P. F., 2013, Monthly Notices of the Royal Astronomical Society, 428, 2840
  * Hopkins (2015) Hopkins P. F., 2015, Monthly Notices of the Royal Astronomical Society, 450, 53
  * Hu et al. (2014) Hu C.-Y., Naab T., Walch S., Moster B. P., Oser L., 2014, Monthly Notices of the Royal Astronomical Society, 443, 1173
  * Hunter (2007) Hunter J. D., 2007, Computing in Science Engineering, 9, 90
  * Kanjilal et al. (2021) Kanjilal V., Dutta A., Sharma P., 2021, Monthly Notices of the Royal Astronomical Society, 501, 1143
  * Kooij et al. (2021) Kooij R., Grønnow A., Fraternali F., 2021, Monthly Notices of the Royal Astronomical Society, 502, 1263
  * Lanson & Vila (2008) Lanson N., Vila J.-P., 2008, SIAM Journal on Numerical Analysis, 46, 1912
  * Lecoanet et al. (2016) Lecoanet D., et al., 2016, Monthly Notices of the Royal Astronomical Society, 455, 4274
  * Li et al. (2020) Li Z., Hopkins P. F., Squire J., Hummels C., 2020, Monthly Notices of the Royal Astronomical Society, 492, 1841
  * Mandelker et al. (2021) Mandelker N., van den Bosch F. C., Springel V., van de Voort F., Burchett J. N., Butsky I. S., Nagai D., Oh S. P., 2021, arXiv e-prints, p. arXiv:2107.03395
  * McCourt et al. (2018) McCourt M., Oh S. P., O’Leary R., Madigan A.-M., 2018, Monthly Notices of the Royal Astronomical Society, 473, 5407
  * McNally et al. (2012) McNally C. P., Lyra W., Passy J.-C., 2012, The Astrophysical Journal Supplement Series, 201, 18
  * Schaller et al. (2015) Schaller M., Dalla Vecchia C., Schaye J., Bower R. G., Theuns T., Crain R. A., Furlong M., McCarthy I. G., 2015, Monthly Notices of the Royal Astronomical Society, 454, 2277
  * Schaller et al. (2016) Schaller M., Gonnet P., Chalk A. B. G., Draper P. W., 2016, in Proceedings of the Platform for Advanced Scientific Computing Conference. PASC ’16. Association for Computing Machinery, New York, NY, USA, pp 1–10, doi:10.1145/2929908.2929916
  * Schaller et al. (2018) Schaller M., Gonnet P., Draper P. W., Chalk A. B. G., Bower R. G., Willis J., Hausammann L., 2018, Astrophysics Source Code Library, p. ascl:1805.020
  * Schaye et al. (2015) Schaye J., et al., 2015, Monthly Notices of the Royal Astronomical Society, 446, 521
  * Shelton et al. (2012) Shelton R. L., Kwak K., Henley D. B., 2012, The Astrophysical Journal, 751, 120
  * Sparre et al. (2019) Sparre M., Pfrommer C., Vogelsberger M., 2019, Monthly Notices of the Royal Astronomical Society, 482, 5401
  * Sparre et al. (2020) Sparre M., Pfrommer C., Ehlert K., 2020, Monthly Notices of the Royal Astronomical Society, 499, 4261
  * Springel (2010) Springel V., 2010, Monthly Notices of the Royal Astronomical Society, 401, 791
  * Stone et al. (2020) Stone J. M., Tomida K., White C. J., Felker K. G., 2020, The Astrophysical Journal Supplement Series, 249, 4
  * Tricco (2019) Tricco T. S., 2019, Monthly Notices of the Royal Astronomical Society, 488, 5210
  * Tumlinson et al. (2017) Tumlinson J., Peeples M. S., Werk J. K., 2017, Annual Review of Astronomy and Astrophysics, 55, 389
  * Turk et al. (2011) Turk M. J., Smith B. D., Oishi J. S., Skory S., Skillman S. W., Abel T., Norman M. L., 2011, The Astrophysical Journal Supplement Series, 192, 9
  * Vijayan et al. (2020) Vijayan A., Kim C.-G., Armillotta L., Ostriker E. C., Li M., 2020, The Astrophysical Journal, 894, 12
  * Vila (1999) Vila J. P., 1999, Mathematical Models and Methods in Applied Sciences, 09, 161
  * Wadsley et al. (2017) Wadsley J. W., Keller B. W., Quinn T. R., 2017, Monthly Notices of the Royal Astronomical Society, 471, 2357
  * de la Cruz et al. (2020) de la Cruz L. M., Schneider E. E., Ostriker E. C., 2020, arXiv e-prints, p. arXiv:2012.02212


# StreetSurfaceVis: a dataset of crowdsourced street-level imagery with semi-
automated annotations of road surface type and quality

Alexandra Kapp1
Edith Hoffmann1
Esther Weigmann1
Helena Mihaljević1

1 Hochschule für Technik und Wirtschaft Berlin (HTW Berlin)
Corresponding author<EMAIL_ADDRESS>

###### Abstract

Road unevenness significantly impacts the safety and comfort of various
traffic participants, especially vulnerable road users such as cyclists and
wheelchair users. This paper introduces StreetSurfaceVis, a novel dataset
comprising 9,122 street-level images collected from a crowdsourcing platform
and manually annotated by road surface type and quality. The dataset is
intended to train models for comprehensive surface assessments of road
networks. Existing open datasets are constrained by limited geospatial
coverage and camera setups, typically excluding cycleways and footways. By
crafting a heterogeneous dataset, we aim to fill this gap and enable robust
models that maintain high accuracy across diverse image sources. However, the
frequency distribution of road surface types and qualities is highly
imbalanced. We address the challenge of ensuring sufficient images per class
while reducing manual annotation by proposing a sampling strategy that
incorporates various external label prediction resources. More precisely, we
estimate the impact of (1) enriching the image data with OpenStreetMap tags,
(2) iterative training and application of a custom surface type classification
model, (3) amplifying underrepresented classes through prompt-based
classification with GPT-4o or similarity search using image embeddings. We
show that utilizing a combination of these strategies effectively reduces
manual annotation workload while ensuring sufficient class representation.

_Keywords_ Street-level Imagery, Road Surface Type and Quality, OpenStreetMap,
Mapillary

##  1 Introduction

Road damages have a significant impact on the comfort and safety of all
traffic participants and have been identified as a major cause of traffic
accidents [14]. This is particularly true for vulnerable road users such as
cyclists [7, 18], wheelchair users [23, 6, 2] and individuals employing inline
skates [15], cargo bikes [1], scooters [27], or strollers. This issue has
sparked a large body of research on methods that apply deep learning models to
street-level imagery for road surface condition assessment [3, 25, 12]. Yet,
road damage may not reflect the full range of factors that influence a
driver’s experience. For example, the smoothness of sett (regular-shaped
cobblestone) is rather determined by the flatness of the utilized stones.
Rateke et al. [26] developed a hierarchical vision-based approach, first
predicting the surface type and then employing specific models for each type
to classify quality. However, the utilized datasets are too restricted to
obtain models that generalize well to new data [26]. Typical street-level
image datasets are commonly collected in good weather conditions, using only a
single vehicle and camera setup within a limited geographic boundary, e.g.,
KITTI [8], CaRINA [28], or Oxford RoboCar [16]. Cityscapes [5] is a dataset of
20,000 images of street scenes recorded in 50 German cities tailored for
autonomous driving applications. Perspectives of sidewalks and cycleways are
not considered and labels consist of semantic segmentation, while surface type
and quality information are unavailable.

We aim to create a street-level image dataset containing a substantial amount
of images for each pertinent surface type and quality, fostering the training
of robust classification models. We utilize the crowdsourcing platform
Mapillary [17] to gather images, as these are contributed by individuals from
various regions using different devices, camera angles, and modes of
transportation, resulting in a heterogeneous dataset. According to data from
the crowdsourcing geographic database OpenStreetMap (OSM), distributions of
surface type and quality are highly skewed in Germany: For instance, asphalt
is the predominant road type, accounting for 47% of tagged road segments,
while only about 3% are made of sett. Similarly, for 54% of asphalted roads,
the quality is considered ‘good’, while only 1% is rated ‘bad’. Consequently,
the difficulty lies in gathering sufficient images for every relevant class
without an infeasible manual labeling effort.

This paper introduces StreetSurfaceVis, a new street-level image dataset
comprising 9,122 images from the crowdsourcing platform Mapillary, labeled for
road surface type and quality by human
experts111https://doi.org/10.5281/zenodo.11449977
The referenced repository contains the dataset, a description, a labeling
guide, and a datasheet documenting the dataset. . Additionally, we present and
evaluate different strategies for semi-automated annotation to efficiently
amplify underrepresented classes in the dataset. These strategies include pre-
filtering using OSM tags, iterative training and application of type
classification models, prompt-based image classification with GPT-4 models,
and similarity-based search using image embeddings.

##  2 Dataset construction

#### Image base.

Our dataset is based on images from Mapillary [17] limited to the geographical
boundaries of Germany. Launched in 2013, the crowdsourcing platform Mapillary
provides openly available street-level images. Contributors can use, among
others, the Mapillary smartphone app to capture georeferenced image sequences
during their trips by, e.g., car, bicycle, or on foot. Thus, the dataset
encompasses not only images from roadways but also cycleways and footways. As
of January 2024, Mapillary contains about 170 million images in Germany, with
more than 50% captured within the last three years. The geographic coverage
depends on the contributors within each region, resulting in hundreds of
thousands of images for every major city in Germany. As any contributor can
upload images, the dataset exhibits varying quality levels, influenced by
factors such as the device used, its mounting (e.g., the car dashboard is
visible in the frame), or prevailing lighting and weather conditions. Hence,
some images are dark, blurred, or contain motifs other than street-level
scenes, such as traffic signs, cars, or trees.

#### Image selection.

To increase the dataset’s heterogeneity, we restrict the number of images for
each location and sequence222During recording, the Mapillary app captures
images every few seconds, with all images from a trip belonging to the same
sequence. per class. This reduces the number of images taken by the same
person on one trip and thus increases spatial diversity, camera
specifications, environmental conditions, and photographic perspectives.
Specifically, we limit the number of images per geographic unit (XYZ style
spherical mercator tiles on zoom level 14333We thereby adhere to the same
geographic unit as utilized by the Mapillary API for computational
feasibility., which is roughly equivalent to $\sim 1.5$ x $1.5km$ grid cells)
to 5 and the number of images per sequence to 10.

For modeling, we recommend using a train-test split where the test data
includes geospatially distinct areas, thereby ensuring the model’s ability to
generalize to unseen regions is tested. We propose five cities varying in
population size and from different regions in Germany for testing, comprising
776 images tagged accordingly.

#### Labeling scheme.

Our labels for surface type and quality primarily align with the OSM road
segment tags surface and smoothness, respectively. While surface [22]
describes the surface type such as ‘asphalt’, smoothness [21] reflects the
physical usability of a road segment for wheeled vehicles, particularly
regarding its regularity or flatness [21]. Our labeling scheme includes those
classes that are important from a traffic perspective and represent a relevant
portion of street types in Germany. This results in the type labels asphalt,
concrete, paving stones, sett, and unpaved. For the quality label, we restrict
to five of eight proposed levels, ranging from excellent (suitable for
rollerblades), good (racing bikes), intermediate (city bikes and wheelchairs),
bad (normal cars with reduced velocity) to very bad (cars with high-
clearance). The final scheme comprises 18 classes of type and quality
combinations, displayed in Table 1, as not all quality labels are suitable for
all surface types. See Figure 1 for example images and labels.

(a) paving st. - excel.
strubbl|3044550732526999

(b) paving st. - inter.
carlheinz|789444188806153

(c) unpaved - inter.
macsico|148771344320493

(d) sett - good
hubert87|812903276289693

(e) sett - bad
carlheinz|129178263353193

(f) asphalt - bad
zoegglmeyr|293245499014573

Figure 1: Example images of different surface types and qualities, with
Mapillary contributor names and image IDs.

#### Manual annotation.

After conducting a thorough explorative analysis of Mapillary images, the
first three authors developed an annotation guide containing quality level
descriptions with example images and underwent self-organized training to
manually label surface type and quality. The instructions include labeling the
focal road located in the bottom center of the street-level image. In cases
where the focus is ambiguous, such as when two parts of the road (e.g., the
cycleway and footway) are depicted equally, or when the surface could not be
classified due to factors such as snowy roads, blurry images, or non-road
images, the image is sorted out. If the surface quality falls between two
categories, annotators are directed to select the lower quality level.
Annotators are encouraged to consult each other for a second opinion when
uncertain. For annotation, we use Labelstudio [30], as it allows to preset
labels from pre-labeling strategies.

To evaluate inter-rater reliability, 180 heterogeneous images were
independently rated by all three annotators. Twelve images were marked for
revision by at least one annotator, and a further 49 were discarded by at
least one annotator for the above reasons, all of which were excluded for the
calculation of inter-rater reliability. Krippendorff’s $\alpha$ [13] for
surface type is calculated at 0.96, indicating a high level of agreement.
Surface quality, treated as an ordinal scale variable, achieves a
Krippendorff’s $\alpha$ of 0.74. While this is generally deemed an acceptable
level of agreement, it reflects the fluid class transitions of quality.

An overview of the final annotated dataset is given in Table 1.

Table 1: Final dataset size by type-quality class. | excellent | good | intermediate | bad | very bad
---|---|---|---|---|---
asphalt | 971 | 1,696 | 821 | 246 | -
concrete | 314 | 350 | 250 | 58 | -
paving stones | 385 | 1,063 | 519 | 70 | -
sett | - | 129 | 694 | 540 | -
unpaved | - | - | 326 | 387 | 303

##  3 Annotation strategies

Using the frequency distribution of OSM tags as a baseline estimate444The OSM
distribution is likely an overestimation of the frequency of underrepresented
classes, as main roads with good quality are typically more frequented, and
thus presumably have more images., manual labeling of randomly sampled images
would be highly inefficient. For example, to obtain 7 images of asphalt-bad,
1,000 images would need to be checked (cf. Table 2). Thus, our goal is to
employ pre-selection strategies that yield samples for manual labeling where
underrepresented classes have a higher frequency than indicated by the OSM
baseline. We evaluate three strategies: (1) enriching the image dataset with
OSM tags; (2) iterative training and application of a model classifying
surface type; and (3) amplifying underrepresented type-quality classes
prompting with GPT-4o or through a similarity search using image embeddings.
Table 2 presents the improvements through the first two sub-strategies in
terms of precision*100, based on 19,747 images processed according to the
strategy shown in Figure 2 (marked in green and yellow).

Table 2: Precision*100 of (1) OSM pre-label and (2) type pseudo-label sub-strategies based on 19,747 images vs. OSM frequency distribution as a baseline. class | OSM distr. | OSM pre-l. | OSM pre-l.
\+ type pseudo-l.
---|---|---|---
asphalt-excellent | 20.31 | 27.83 | 43.46
asphalt-good | 29.29 | 28.54 | 50.17
asphalt-intermediate | 4.05 | 14.73 | 20.31
asphalt-bad | 0.69 | 5.64 | 8.80
concrete-excellent | 0.36 | 4.23 | 28.48
concrete-good | 0.94 | 23.32 | 55.03
concrete-intermediate | 0.78 | 22.34 | 44.58
concrete-bad | 0.23 | 9.12 | 20.51
paving stones-excellent | 2.98 | 10.87 | 30.27
paving stones-good | 10.62 | 15.35 | 46.36
paving stones-intermediate | 1.94 | 3.05 | 9.95
paving stones-bad | 0.22 | 1.00 | 3.83
sett-good | 0.56 | 2.19 | 12.92
sett-intermediate | 1.97 | 19.37 | 47.80
sett-bad | 1.26 | 11.92 | 46.65
unpaved-intermediate | 5.75 | 35.09 | 60.87
unpaved-bad | 7.67 | 28.61 | 44.68
unpaved-very bad | 7.38 | 18.70 | 26.89
Figure 2: Proposed strategy for selecting, pre-labeling, and annotating the
training dataset.

#### Pre-labeling via OSM tags

OSM [19] contains surface tags for 49% and smoothness tags for 16% of road
segments in Germany, as of January 2024. We incorporate this information by
spatially intersecting with geolocations of Mapillary images555For
computational feasibility, we refrain from intersecting all Mapillary images
and use a sample of Mercantile tiles where the desired classes occur
particularly frequently. and assigning the surface and smoothness labels of
the closest OSM road segment within a maximum distance of two meters. To
eliminate ambiguous street intersections, we cut off 10% of the start and end
of each road segment beforehand.

We test the accuracy of this approach based on a sample of 100 images for each
of the 18 classes in terms of OSM labels (Batch 1 in Figure 2). OSM labels and
manual annotation agree on 69% of surface type labels, and among these, the
quality is correct for 55% of the images. However, the errors are not equally
distributed: while 94% of asphalt images are correctly pre-labeled, only 54%
of paving stones and sett are correct, both being underrepresented surface
types. This strategy thus increases the precision for each class
substantially, albeit with varying impact. For example, to obtain 7 images
from the underrepresented class asphalt-bad, around 100 instead of 1,000
images need to be reviewed (cf. Table 2). We refer to the labels resulting
from this strategy as OSM pre-labels.

Incorrect type labels mainly result from mixing up adjacent road parts, as OSM
sometimes lacks separate geometries for cycleways and footways. Differences in
quality labels are likely due to varying subjective assessments by OSM
contributors. Additionally, GPS inaccuracies as well as time differences
between image capturing and road segment tagging in OSM are plausible sources
of discrepancies, for both type and quality.

#### Pseudo-labeling with a type classifier

To further increase pre-selection precision, we iteratively train a
classification model to predict the type and use its predictions as pseudo-
labels. Since type is easier to classify than quality, a smaller amount of
data should be sufficient to obtain valuable pseudo-labels. More precisely, we
fine-tune EfficientNetV2-S [29], pre-trained on ImageNet, on the first
annotated batch to predict the type. The model is then applied to the next
batch of images. All images where the prediction matches the OSM pre-label are
selected for manual annotation; this combination of labels achieves an average
precision of 95% for the surface type, with the lowest precision of 89% for
paving stones over all batches. To reduce bias towards easy-to-classify
examples, a random sample of 10% from the excluded images is manually
annotated. In the next iteration, the training set is extended with the
manually annotated images from the previous round. The batch composition is
also adjusted towards underrepresented classes, aiming for 300-400 images per
class. This procedure leads to significant improvements for certain classes.
For example, paving stones-excellent achieves a precision of 30% with a total
of 342 images, while asphalt-good shows an excess of images totaling 1,334.
However, the precision remains very low for some classes, e.g., less than 4%
(cf. Table 2) for paving stones-bad with only 30 images gathered at this
point. Thus, continuing this procedure would be infeasible.

#### Expanding underrepresented classes through prompting and similarity
search

We evaluate two approaches to efficiently enlarge classes that remain
underrepresented. Our first approach uses prompt-based image classification
with OpenAI’s GPT-4V [9] and GPT-4o [10] models, which can generate textual
output from image-text input. Previous studies have demonstrated the potential
of GPT-4V for automated image labeling in various application domains,
including street intersections [11] and traffic scenery [32]. GPT-4o, released
in May 2024, is expected to be similarly effective at half the cost (at the
time of release). As an alternative, we explore similarity search using image
embeddings [4] from OpenAI’s CLIP [24], DINOv2 [20], and our fine-tuned
EfficientNet-based type classifier. Specifically, annotated images from the
class of interest are used as the query, and all images with a cosine
similarity score above a certain threshold are pre-labeled as class members.

For both approaches, we restrict the search space using the previously
described strategy combining OSM tags and type classification. Due to cost and
time constraints, we limit the experiments to three underrepresented classes.
Using a validation dataset of randomly sampled manually annotated images (50
for the classes asphalt-bad and paving stones - intermediate and 30 for paving
stones - bad), we systematically evaluate base models and hyperparameters of
both approaches. Note that in both cases, a higher precision implies a lower
estimated effort in human post-annotation, while a higher recall correlates
with a larger class increase and, in the prompting scenario, with a lower
monetary cost.

Prompting configurations include two different image cropping styles, varying
levels of detail in class definitions, zero-shot versus one-shot prediction,
and processing a batch versus one image per prompt. While cropping shows only
minimal effect, one-shot outperforms the zero-shot setting, albeit with a
larger monetary cost, resulting in a similar price per hit. GPT-4o achieves
notably better results than GPT-4V for all classes. The best configuration in
terms of F1 score and cost-effectiveness consists of GPT-4o, lower half-center
cropping, shortened definitions, one-shot prediction, and one request per
prompt (see Appendix B for the utilized final prompt). To decide on the
similarity threshold, we compute the optimal ROC cut-off value for each
embedding model and class. All three embeddings give similar and reasonable
results for the two larger classes with optimal cut-off values around 0.85 for
CLIP and 0.6 for the other two models. The type-classifier slightly
outperforms DINOv2 and CLIP on average and is therefore selected for the
following experiment. The smallest class paving stones - bad remains difficult
for all three models, e.g., our embedding returns nearly all images given the
optimal threshold.

The models are then applied in their optimal configurations to a dataset
compiled as follows: around 20M Mapillary images are used for image selection
as described in Section 2 and pre-labeled using OSM tags and type pseudo-
labels. From the remaining data, up to 1,000 images per pre-label are drawn at
random. Note that for paving stones-bad only 210 images are available. As
shown in Table 3, both approaches yield a substantial improvement. However,
prompting with GPT-4o achieves substantially higher precision than the
similarity search, with values between 40% and 65%.

Table 3: Evaluation of prompt-based classification (GPT-4o) and similarity search (SimS) on three underrepresented pre-label classes, in terms of precision*100 (recall*100), vs. OSM pre-label + type pseudo-label as a baseline. class | OSM pre-l.
\+ type pseudo-l.
| OSM pre-l.
\+ type pseudo-l.
\+ GPT-4o | OSM pre-l.
\+ type pseudo-l.
\+ SimS
---|---|---|---
asphalt-bad | 10.06 | 39.46 (81.11) | 20.38 (47.78)
paving st.-bad | 10.00 | 64.71 (52.38) | 10.29 (100.00)
paving st.-inter. | 22.75 | 40.29 (83.03) | 30.74 (47.88)

While the described strategy has shown to have a notable impact (cf. Table 3),
it substantially limits the search space (for both GPT-4o and the similarity
search) due to the sparsity of OSM tags. Moreover, the reliance on OSM tags
yields a potential selection bias as the distribution can be assumed to depend
on factors such as urbanity and OSM community. To estimate the efficacy of
prompting with GPT-4o without OSM-based filtering, we apply only the type
classification model to a random sample of 20,000 Mapillary images to obtain
pseudo-labels and thus define the search space. From the resulting 15,100
images classified as asphalt, we prompt GPT-4o with a random sample of 2,000
images, as well as all 712 images pseudo-labeled as paving stones. The results
show a reduced precision of 18%-31%, thus higher manual labeling effort.
Nevertheless, there remains a relevant increase to the OSM pre-label baseline,
providing a viable method if the OSM tag pre-labeled search space is
exhausted. However, costs substantially rise: with about $0.01 per GPT-4o
prompt, obtaining a correctly labeled image without pre-labeling via OSM costs
$0.12 to $3.45, depending on the class, compared to $0.07 and $0.18 in the
previous experiment.

##  4 Conclusion and discussion

We presented StreetSurfaceVis, a new street-level imagery dataset labeled
according to surface type and quality. To avoid inefficient manual labeling
work, we tested different strategies, in particular, to increase the pre-
selection of under-represented classes for further manual annotation. As the
most feasible strategy, we identified a combination of intersecting OSM tags,
predicting surface type with a classification model, and prompting with
GPT-4o. We hope that our results will be useful for other researchers and
practitioners to efficiently create similar datasets, e.g., for other regions.

Generally, performance increases of GPT-4o and similar models are to be
expected in the future, further amplifying the viability of this approach. To
reduce dependency on OpenAI and monetary cost, future work should evaluate
open-source alternatives. Similarity search is more efficient in computational
and monetary terms, therefore currently remaining a viable alternative despite
inferior results. The results varied between classes and showed the worst
performance for paving stones - bad, where there was no improvement to the
baseline. Further experiments, especially with more images for the smallest
class, as well as advancements, could be explored, e.g., by incorporating
different clustering strategies [31].

We envision extensions of the dataset to include road-type information as well
as unsuitable images, such as blurred ones or those without a road in focus.
Including these will enable training models to automatically check image
suitability, thus improving classification robustness and enhancing the GPT-4o
prompting strategy.

## References

  * [1] Konstantinos Athanasopoulos et al.  “Integrating cargo bikes and drones into last-mile deliveries: Insights from pilot deliveries in five greek cities” Publisher: MDPI  In _Sustainability_ 16.3, 2024, pp. 1060
  * [2] Linda Beale et al.  “Mapping for wheelchair users: Route navigation in urban spaces” Publisher: Taylor & Francis  In _The Cartographic Journal_ 43.1, 2006, pp. 68–81
  * [3] Minh-Tu Cao, Quoc-Viet Tran, Ngoc-Mai Nguyen and Kuan-Tsung Chang  “Survey on performance of deep learning models for detecting road damages using multiple dashcam image resources”  In _Advanced Engineering Informatics_ 46, 2020, pp. 101182  DOI: 10.1016/j.aei.2020.101182
  * [4] Cody Coleman et al.  “Similarity Search for Efficient Active Learning and Search of Rare Concepts” Number: 6  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 36.6, 2022, pp. 6402–6410  DOI: 10.1609/aaai.v36i6.20591
  * [5] Marius Cordts et al.  “The Cityscapes Dataset for Semantic Urban Scene Understanding”, 2016, pp. 3213–3223
  * [6] Jonathan Duvall et al.  “Development of Surface Roughness Standards for Pathways Used by Wheelchairs”  In _Transportation Research Record_ 2387.1 SAGE Publications Inc, 2013, pp. 149–156  DOI: 10.3141/2387-17
  * [7] April Gadsby, James Tsai and Kari Watkins  “Understanding the Influence of Pavement Conditions on Cyclists’ Perception of Safety and Comfort Using Surveys and Eye Tracking” Publisher: SAGE Publications Inc  In _Transportation Research Record_ 2676.12, 2022, pp. 112–126  DOI: 10.1177/03611981221090936
  * [8] A Geiger, P Lenz, C Stiller and R Urtasun  “Vision meets robotics: The KITTI dataset”  In _The International Journal of Robotics Research_ 32.11, 2013, pp. 1231–1237  DOI: 10.1177/0278364913491297
  * [9] “GPTV_System_Card”, 2024  URL: https://cdn.openai.com/papers/GPTV_System_Card.pdf
  * [10] “Hello GPT-4o”, 2024  URL: https://openai.com/index/hello-gpt-4o/
  * [11] Hochul Hwang, Sunjae Kwon, Yekyung Kim and Donghyun Kim  “Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing”  In _arXiv [cs]_ arXiv, 2024  arXiv:2402.06794 [cs]
  * [12] Young-Mok Kim et al.  “Review of Recent Automated Pothole-Detection Methods” Number: 11 Publisher: Multidisciplinary Digital Publishing Institute  In _Applied Sciences_ 12.11, 2022, pp. 5320  DOI: 10.3390/app12115320
  * [13] Klaus Krippendorff  “Content Analysis: An Introduction to Its Methodology”  SAGE Publications, Inc., 2019  DOI: 10.4135/9781071878781
  * [14] Joseph Kurebwa, Tawanda Mushiri and others  “A study of damage patterns on passenger cars involved in road traffic accidents” Publisher: Hindawi  In _Journal of Robotics_ 2019, 2019
  * [15] Stephen W Lorimer and Stephen Marshall  “Beyond walking and cycling: scoping small-wheel modes”  In _Proceedings of the institution of civil engineers-engineering sustainability_ Thomas Telford Ltd, 2015
  * [16] Will Maddern, Geoffrey Pascoe, Chris Linegar and Paul Newman  “1 year, 1000 km: The oxford robotcar dataset” Publisher: SAGE Publications Sage UK: London, England  In _The International Journal of Robotics Research_ 36.1, 2017, pp. 3–15
  * [17] “Mapillary”  In _Mapillary_ , 2024  URL: https://www.mapillary.com/
  * [18] Peter Nyberg, Ulf Björnstig and L-O Bygren  “Road characteristics and bicycle accidents” Publisher: SAGE Publications  In _Scandinavian Journal of Social Medicine_ 24.4, 1996, pp. 293–301  DOI: 10.1177/140349489602400410
  * [19] “OpenStreetMap”  In _Key:surface_ , 2024  URL: https://openstreetmap.org/
  * [20] Maxime Oquab et al.  “DINOv2: Learning Robust Visual Features without Supervision”  In _Transactions on Machine Learning Research_ , 2023
  * [21] “OSM Wiki Smoothness”  In _Key:smoothess_ , 2024  URL: https://wiki.openstreetmap.org/wiki/Key:smoothness
  * [22] “OSM Wiki Surface”  In _Key:surface_ , 2024  URL: https://wiki.openstreetmap.org/wiki/Key:surface
  * [23] Jonathan Pearlman, Rory Cooper, Jonathan Duvall and Ryan Livingston  “Pedestrian Pathway Characteristics and Their Implications on Wheelchair Users”  In _Assistive Technology_ 25.4, 2013, pp. 230–239  DOI: 10.1080/10400435.2013.778915
  * [24] Alec Radford et al.  “Learning Transferable Visual Models From Natural Language Supervision”  In _Proceedings of the 38th International Conference on Machine Learning_ 139, Proceedings of Machine Learning Research  PMLR, 2021, pp. 8748–8763
  * [25] Fahim Ur Rahman et al.  “A Comparative Study on Road Surface State Assessment Using Transfer Learning Approach”  In _2022 13th International Conference on Computing Communication and Networking Technologies (ICCCNT)_ , 2022, pp. 1–6  DOI: 10.1109/ICCCNT54827.2022.9984393
  * [26] Thiago Rateke, Karla Aparecida Justen and Aldo Von Wangenheim  “Road Surface Classification with Images Captured From Low-cost Camera - Road Traversing Knowledge (RTK) Dataset”  In _Revista de Informática Teórica e Aplicada_ 26.3, 2019, pp. 50–64  DOI: 10.22456/2175-2745.91522
  * [27] Caroline Rodier, Susan A Shaheen and Stephanie Chung  “Unsafe at any speed?: what the literature says about low-speed modes”, 2003
  * [28] Patrick Y. Shinzato et al.  “CaRINA dataset: An emerging-country urban scenario benchmark for road detection systems”  In _2016 IEEE 19th international conference on intelligent transportation systems (ITSC)_ , 2016, pp. 41–46  DOI: 10.1109/ITSC.2016.7795529
  * [29] Mingxing Tan and Quoc Le  “Efficientnetv2: Smaller models and faster training”  In _International conference on machine learning_ PMLR, 2021, pp. 10096–10106
  * [30] Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk and Nikolai Liubimov  “Label Studio: Data labeling software”, 2020  URL: https://github.com/heartexlabs/label-studio
  * [31] Huy V. Vo et al.  “Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach”  In _arXiv [cs]_ arXiv, 2024  arXiv:2405.15613 [cs]
  * [32] Licheng Wen et al.  “On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving”  In _arXiv [cs]_ arXiv, 2023  arXiv:2311.05332 [cs]

##  Appendix A Labeling scheme

###  A.1 Asphalt

Quality |  Description
---|---
excellent |  As good as new asphalt, on which a skateboard or rollerblades will have no problem.
good |  Asphalt showing the first signs of wear, such as narrow, smaller than 1.5 cm cracks, or wider cracks filled up with tar, shallow dents in which rainwater may collect, which may cause trouble for rollerblades but not for racing bikes.
intermediate |  Asphalt roads that show signs of maintenance, such as patches of repaired surface, wider cracks larger than 2cm. Asphalt sidewalks may contain potholes, but these are small, shallow (<3cm deep) and can be easily avoided. Asphalt driving lanes show damage due to subsidence (depressions of a scale >50 cm) or heavy traffic (shallow ruts in asphalt caused by trucks in summer). This means that the road can be used by normal city bikes, wheelchairs and sports cars, but not by a racing bike.
bad |  Damaged asphalt roads that show clear signs of maintenance: This might include potholes, some of them quite deep, which might decrease the average speed of cars. However, it isn’t so rough that ground clearance becomes a problem. Meaning that the street causes trouble to normal city bikes but not a trekking bike and a car.

###  A.2 Concrete

Quality |  Description
---|---
excellent |  As good as new concrete, on which a skateboard or rollerblades will have no problem.
good |  Concrete road showing the first signs of wear, such as narrow, smaller than 1.5 cm cracks, or wider cracks filled up with tar, shallow dents in which rainwater may collect, which may cause trouble for rollerblades but not for racing bikes.
intermediate |  Concrete roads that show signs of maintenance, such as patches of repaired surface, wider cracks larger than 2cm. Concrete sidewalks may contain potholes, but these are small, shallow (<3cm deep) and can be easily avoided. Concrete driving lanes show damage due to subsidence (depressions of a scale >50 cm) or heavy traffic (shallow ruts in concrete caused by trucks in summer). This means that the road can be used by normal city bikes, wheelchairs and sports cars, but not by a racing bike.
bad |  Heavily damaged concrete roads that badly need maintenance: many potholes, some of them quite deep. The average speed of cars is less than 50% of what it would be on a smooth road. However, it isn’t so rough that ground clearance becomes a problem. Meaning that the street causes trouble to normal city bikes but not a trekking bike and a car.

###  A.3 Paving stones

Quality |  Description
---|---
excellent |  Newly installed and regularly laid paving stones that show no signs of wear. Gaps may be visible, but they are small and uniform and do not significantly affect the driving experience.
good |  Paving stones showing first signs of wear or newly installed stones with visible but uniform gaps between them. While still suitable for most activities, these surfaces may pose minor challenges for rollerblades and skateboards but remain navigable for racing bikes.
intermediate |  Characterized by paving stones exhibiting multiple signs of wear, such as shifted heights, potholes, or cracks. This grade allows for the comfortable passage of normal city bikes and standard vehicles but may prove challenging for racing bikes.
bad |  Heavily uneven or damaged paving stones in dire need of maintenance, featuring significant height disparities and numerous deep potholes. While ground clearance remains sufficient for most vehicles, the surface severely impedes travel, particularly for standard city bikes.

###  A.4 Sett

Quality |  Description
---|---
good |  The best sett roads with flattened stones and gaps that are at most small or filled up. The surface might cause trouble for rollerblades but not for racing bikes.
intermediate |  The surfaces of the sett stones are not completely flat, or there may be slightly larger gaps between the stones, this causes problems for racing bikes and slows down city bikes and cars.
bad |  Sett stones with large and possibly uneven gaps or uneven stone surfaces or damaged stones, resulting in an overall bumpy surface: This results in a highly uncomfortable driving experience for city bikes. The average speed of cars is less than 50% of what it would be on a smooth road. However, it isn’t so rough that ground clearance becomes a problem.

###  A.5 Unpaved

Quality |  Description
---|---
intermediate |  The best unpaved roads that have a compacted surface. This grade allows for the passage of normal city bikes and standard vehicles but may prove challenging for racing bikes.
bad |  Unpaved roads that do not have a smooth and compacted surface but ones that can still be used with a car or trekking bike but not with a city bike. This category also includes hiking paths, which are too narrow for cars.
very_bad |  Unpaved roads with potholes, ruts or generally a highly uneven surface not safely passable with a regular passenger car but still passable with an average SUV with higher ground clearance. This category also includes highly uneven hiking paths, which are too narrow for cars.

##  Appendix B GPT 4-o Prompt

The following shows the utilized prompt for one-shot asphalt image prediction
using GPT 4-o. Prompts for other surface types are adjusted according to type-
specific labeling schemes (see Appendix A).

    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": "You are a data annotation expert trained to
                classify the quality level of road surfaces in images."
                },
        ]
    },
    {
        "role": "user",
        "content": [
                {
                "type": "text",
                "text": f"""
                        You need to determine the quality level of the road
                        surface depicted in the image, following this defined scale:

                        Asphalt surfaces are graded from excellent to bad
                        according to the following scale:
                        1) excellent: As good as new asphalt, on which a
                        skateboard or rollerblades will have no problem.

                        2) good: Asphalt showing the first signs of wear, such as
                        narrow, smaller than 1.5 cm cracks, or wider cracks filled
                        up with tar, shallow dents in which rainwater may collect,
                        which may cause trouble for rollerblades but not for
                        racing bikes.

                        3) intermediate: Asphalt roads that shows signs of
                        maintenance, such as patches of repaired surface, wider
                        cracks larger than 2cm. Asphalt sidewalks may contain
                        potholes, but these are small, shallow (<3cm deep) and can
                        be easily avoided, asphalt driving lanes shows damage due
                        to subsidence (depressions of a scale >50 cm) or heavy
                        traffic (shallow ruts in asphalt caused by trucks in
                        summer). This means that the road can be used by normal
                        city bikes, wheelchairs and sports cars, but not by a
                        racing bike.

                        4) bad: Damaged asphalt roads that show clear signs of
                        maintenance: This might include potholes, some of them
                        quite deep, which might decrease the average speed of
                        cars.  However, it isnt so rough that ground clearance
                        becomes a problem. Meaning that the street causes trouble
                        to normal city bike but not a trekking bike and a car.

                        Please adhere to the following instructions:
                        1) Step 1: If you detect multiple surface types, only
                        consider the path, driving lane, cycleway or sidewalk in
                        the focus area.

                        2) Step 2: Check if the road surface is worn off and if
                        you can find any damages, like cracks.

                        3) Step 3: Check the quantity and the size of the damages.

                        3) Step 4: Then decide if you could ride on the surface
                        with  a skateboard, rollerblades, racing bikes, city bike,
                        or a normal car.

                        4) Step 5: If you detect characteristics of two classes,
                        choose the worse class.

                        How would you rate this image using one of the four
                        options of the defined scale:
                        1) excellent
                        2) good
                        3) intermediate
                        4) bad

                        Provide your rating in one word disregarding the bullet
                        point numbers and brackets as a string using the four
                        levels of the scale provided.
                        Make sure you have the same number of image urls as input
                        as you have output values.

                        Do not provide any additional explanations for your
                        rating; focus solely on the road surface quality."""
                },

                {
                "type": "image_url",
                "image_url": {
                            "url": f"data:image/jpeg;base64,{excellent_encoded_image}"
                            },
                },

                {
                "type": "text",
                "text": "This was an example for ’excellent’"
                },

                {
                "type": "image_url",
                "image_url": {
                            "url": f"data:image/jpeg;base64,{good_encoded_image}"
                            },
                },

                {
                "type": "text",
                "text": "This was an example for ’good’"
                },

                {
                "type": "image_url",
                "image_url": {
                            "url": f"data:image/jpeg;base64,{intermediate_encoded_image}"
                            },
                },

                {
                "type": "text",
                "text": "This was an example for ’intermediate’"
                },

                {
                "type": "image_url",
                "image_url": {
                            "url": f"data:image/jpeg;base64,{bad_encoded_image}"
                            },
                },

                {
                "type": "text",
                "text": "This was an example for ’bad’"
                },

                {
                "type": "text",
                "text": "Please decide now in one word which category the
                following picture belongs to as instructed in the beginning of the
                prompt. Then compare this image to the previous ones and decide
                whether this category is correct."
                },

                {
                "type": "image_url",
                "image_url": {
                            "url": f"data:image/jpeg;base64,{image}"
                            },
                },

                    ],
    },

    ]