[datatype=bibtex] [fieldsource=adsurl,fieldtarget=iswc]
[fieldsource=gbkurl,fieldtarget=iswc]

# Carving out the low surface brightness universe with NoiseChisel

Mohammad Akhlaghi1,2 1Instituto de Astrofísica de Canarias, C/ Vía Láctea,
38200 La Laguna, Tenerife, Spain.
email<EMAIL_ADDRESS>
2Facultad de Física, Universidad de La Laguna, Avda. Astrofísico Fco. Sánchez
s/n, 38200
La Laguna, Tenerife, Spain.

(2019)

###### Abstract

NoiseChisel is a program to detect very low signal-to-noise ratio (S/N)
features with minimal assumptions on their morphology. It was introduced in
2015 and released within a collection of data analysis programs and libraries
known as GNU Astronomy Utilities (Gnuastro). Over the last ten stable releases
of Gnuastro, NoiseChisel has significantly improved: detecting even fainter
signal, enabling better user control over its inner workings, and many bug
fixes. The most important change may be that NoiseChisel’s segmentation
features have been moved into a new program called Segment. Another major
change is the final growth strategy of its true detections, for example
NoiseChisel is able to detect the outer wings of M51 down to S/N of $0.25$, or
$28.27$ mag/arcsec2 on a single-exposure SDSS image (r-band). Segment is also
able to detect the localized HII regions as “clumps” much more successfully.
Finally, to orchestrate a controlled analysis, the concept of a “reproducible
paper” is discussed: this paper itself is exactly reproducible (snapshot
v4-0-g8505cfd).

###### keywords:

galaxies: halos, galaxies: photometry, galaxies: structure, methods: data
analysis, methods: reproducible, techniques: image processing, techniques:
photometric

††volume: 355††journal: The Realm of the Low-Surface-Brightness
Universe††editors: D. Valls-Gabaud, I. Trujillo & S. Okamoto, eds.

##  1 Introduction

Signal from the low surface brightness universe is buried deep in the
dataset’s noise and thus requires accurate detection methods. In [2]
(henceforth AI15) a new method was introduced to detect such very low signal-
to-noise ratio (S/N) signal from the images in a non-parametric manner. It
allows accurate detection of the diffuse outer features of galaxies (that
often have a different morphology from the centers). The software
implementation of this method (NoiseChisel) is released as part of a larger
collection of data analysis software known as GNU Astronomy
Utilities111https://www.gnu.org/s/gnuastro (Gnuastro). It was the first
professional astronomical software to be independently refereed by an
independent panel (GNU Evaluation committee) and fully conforms with the GNU
Coding Standards222https://www.gnu.org/prep/standards.

Since its release, NoiseChisel has been used in many studies. For example [3]
used it to identify objects that were missed by [13] (henceforth R15), who
used a combination of six SExtractor [4] runs with different configurations to
avoid deblending problems, but still missed many sources with significant
signal, see Figure 1. [5, 12, 14] used it for accurate flat field and Sky
subtraction to create deeper co-added images in galaxy fields for optimal
detection of the low surface brightness features. [6] used it to find
Lyman-$\alpha$ emitters in spectra. For future studies, [11, 8] have proposed
using NoiseChisel to maximize LSST’s low surface brightness features (in extra
galactic and solar system science outputs).

Figure 1:  Example targets (central object in top row) found in MUSE datacubes
[10] (I17) that did not correspond to anything in the [13] (R15) catalog which
is based on HST images. They had footprints in the combined SExtractor
segmentation map of R15 (bottom row), so they are ‘detectable’. However, they
were missed due to incorrect labeling: a segmentation/deblending problem. The
top row is the HST/ACS F775W image from the XDF survey [9]. From left to
right, their IDs in I17 are 6313, 6460, 6331, 6338, 6341, 6431. In case (2),
the central detection has the same label as the large detection that the red
arrow is pointing to. Note that the R15 segmentation shown is a combination of
six SExtractor runs with different deblending parameters. For more, see
Section 7.3 of [3].

With ten stable releases until now, NoiseChisel and Gnuastro have greatly
evolved. In Sections 2 and 3 a review of the major changes and improvements in
Gnuastro’s detection features since its first release is discussed and Section
4 demonstrates these new features on an example image of M51. Finally,
understanding the systematics (i.e., reproducibility) is particularly
important for low surface brightness science, therefore AI15 was also built in
a reproducible manner. A short review of the evolved work in this frontier is
given in Section 5.

##  2 NoiseChisel improvements

Over the last ten stable releases of Gnuastro (version 0.10), as with many of
Gnuastro’s programs, NoiseChisel has also greatly improved: some of the
original features have been removed and many new features have been added. In
Subsections 2.1 and 2.2 the removed and new features of NoiseChisel are
respectively discussed. Note that NoiseChisel will continue to evolve after
this paper also, so the best reference to consult is always its own
manual/book, in particular the “NoiseChisel changes after publication”
section.

The overall direction of Gnuastro’s evolution (including NoiseChisel) has been
modularity and more user control. In terms of modularity, the most notable
change has been that NoiseChisel doesn’t do segmentation any more.
Segmentation of the signal into its substructure is fundamentally a separate
kind of operation than detection (or separating the Sky from the signal).

In terms of more user control NoiseChisel has gradually moved in the direction
of not having any number/configuration hard-coded into it. This gives the user
a very detailed control over how it operates at every step. As a scientific
software, it is critically important for the user to be able to tweak any step
and understand the systematics of their final choice. In the latter two
complete tutorials of the Gnuastro manual/book the logic behind customizing
them to fit different noise properties is fully described in real analysis
scenarios.

###  2.1 NoiseChisel removed features

  * •

Segmentation has been wholly moved to a new program; Segment. From Gnuastro
0.6, NoiseChisel is only in charge on detection. Therefore its output is now
just a binary/two-value image (0: Sky, 1: Signal). Identifying the sub-
structure over the detected regions (for example true clumps) is a
fundamentally different problem. Changes in the original segmentation steps
are discussed in Section 3. This spin-off allows much greater
modularity/creativity and is in the spirit of Gnuastro’s modular design.

For example in some scenarios there may be no Sky (the whole dataset is
covered by targets) and detection would thus not be necessary. This happens
when imaging parts of dense resolved objects for example globular clusters or
the wings of M31. In other cases, segmentation isn’t necessary, for example to
estimate the Sky or flat-field level in individual exposures during data
reduction like [5].

  * •

\--skysubtracted: This option was used to account for the extra noise that is
added if the Sky value has already been subtracted. However, NoiseChisel
estimates the Sky standard deviation based on the input data, not exposure
maps or other theoretical considerations. Therefore the standard deviation of
the undetected pixels also contains the errors due to any previous sky
subtraction. This option is therefore no longer present in NoiseChisel.

  * •

\--dilate: During detection, the outer layers of true detections are eroded.
Until version 0.5, after the true detections were found, the were dilated to
recover the eroded pixels. However, the simple 8- or 4-connected dilation
produced boxy/diamond shapes respectively. They were thus were not realistic
and would miss diffuse flux. The final dig into the noise is now done by
“grow”ing the true detections. Similar to how true clumps were grown, see the
description of \--detgrowquant below.

###  2.2 NoiseChisel new features

The new features in NoiseChisel are listed below. Note that their order is
based on NoiseChisel’s functionality: which step is done before which.

  * •

The quantile difference to identify tiles with no significant signal is
measured between the _mean_ and median. Until Gnuastro 0.8, it was between the
_mode_ and median. The quantile of the mean is more sensitive to skewness (the
presence of signal), so it is preferable to the quantile of the mode.

  * •

\--widekernel: By default, NoiseChisel convolves the input once and estimates
the proper tiles for quantile estimations on the convolved image. The same
convolved image is later used for quantile estimation. A larger kernel has the
advantage that it increases the skewness (and thus difference between the mean
and median, therefore helps in detecting the presence signal). However, a
larger kernel disfigures the shapes/morphology of the objects.

The new \--widekernel option (and a corresponding \--wkhdu option to specify
its HDU) were added in Gnuastro 0.5 to allow benefiting from the advantage and
not being affected by the problem. When its given, the input will be convolved
with both the sharp (given through the \--kernel option) and wide (given
through this option) kernels. The mean and median are calculated on the
dataset that is convolved with the wider kernel and good tiles are chosen.
Finally, quantiles are estimated on the good tiles based on the convolution
with the sharper kernel.

  * •

Rejection of outlier tiles used in estimating the quantile threshold: when
there are large galaxies or bright stars in the image, their gradient may be
on a smaller scale than the selected tile size. In such cases, those tiles
will be identified as tiles with no signal, over subtracting the Sky on the
wings of bright sources, and creating artificial dark halos in the most
extreme cases. An outlier identification algorithm has been added to
NoiseChisel to mitigate this problem since Gnuastro 0.8. It can be configured
with the following options: \--outliersigma and \--outliersclip.

If we assume there are $N$ elements (useful tiles), the values are first
sorted. Searching for the outlier starts on element $N/2$ (integer division).
Let’s take $v_{i}$ to be the $i$-th element of the sorted input (with no blank
values) and $m$ and $\sigma$ as the $\sigma$-clipped median and standard
deviation from the distances of the previous $N/2-1$ elements (not including
$v_{i}$). If the value given to \--outliersigma is displayed with $s$, the
$i$-th element is considered as an outlier when the condition below is true.

${(v_{i}-v_{i-1})-m\over\sigma}>s$

Since $i$ begins from the median, the outlier has to be larger than the
median. You can use the check images (for example \--checkqthresh,
\--checkdetsky and \--checksky options in NoiseChisel) to inspect the steps
and see which tiles have been discarded as outliers prior to interpolation.

  * •

\--blankasforeground: allows blank pixels to be treated as foreground in
NoiseChisel’s binary operations: the initial erosion (\--erode) and opening
(\--open) as well as the filling holes and opening step for defining pseudo-
detections (\--dthresh). Until Gnuastro 0.8, NoiseChisel treated blank pixels
as foreground by default. But this could create false positives near
blank/masked regions, blank pixels are now considered to be in the background
by default. This option will re-create the old behavior.

  * •

\--skyfracnoblank: To reduce the bias caused by undetected wings of galaxies
and stars in the Sky measurements, NoiseChisel only uses tiles that have a
sufficiently large fraction of undetected pixels. Until Gnuastro 0.8, the
reference for this fraction was the whole tile size. With this option, it is
now possible to ask for ignoring blank pixels when calculating the fraction.
This is useful when blank/masked pixels are distributed across the image.

  * •

\--dopening: Number of openings after applying \--dthresh, introduced in
Gnuastro 0.9.

  * •

\--dopeningngb: Type of connectivity to define neighbors (4-, or 8-connected)
in \--dopening, introduced in Gnuastro 0.9.

  * •

\--holengb: Type of connectivity to define neighbors (4-, or 8-connected) to
define (and thus, fill) holes after applying \--dthresh to find pseudo-
detections. Introduced in Gnuastro 0.8.

  * •

\--pseudoconcomp: Type of connectivity to define neighbors (4-, or
8-connected) to find individual pseudo-detections. For example if connectivity
is defined by 8-connected neighbors, pseudo-detections that are touching on
the corner will be identified as one. This was introduced in Gnuastro 0.9.

  * •

\--snthresh: Manually set the S/N of true pseudo-detections and thus avoid the
need to automatically identify this value. Introduced in Gnuastro 0.9.

  * •

\--detgrowquant: is used to grow the final true detections until a given
quantile into the noise since Gnuastro 0.5. It replaces the old \--dilate
option in the paper and older versions of Gnuastro. Dilation is a blind growth
method which causes objects to be boxy or diamond shaped when too many layers
are added. However, with the growth method that is defined now, we can follow
the signal into the noise with any shape.

It uses a modified the Watershed algorithm, very similar to the way that
clumps are grown during segmentation (compare columns 2 and 3 in Figure 10 of
the paper). The difference with the Watershed algorithm is that when a pixel
isn’t touching any previously labeled pixel, it is not given a new label, it
is just sent to the end of the sorted queue to be checked again in the next
round. The best quantile to grow the quantile depends on your dataset’s
correlated noise properties and how cleanly it was Sky subtracted. The new
\--detgrowmaxholesize can also be used to specify the maximum hole size to
fill as part of this growth.

This new growth process can be much more successful in detecting diffuse flux
around true detections compared to dilation and give more realistic results.
For example see Figures 2 and 3. Figure 2 shows how decreasing this option
(going deeper into the noise) effects the final detection map. It can be seen
that beyond a certain point, it will just start to follow correlated noise,
emanating from the initial detections. See the discussion in Section 4 on how
to address this problem while its not too significant.

Figure 2:  Growing true detections with various values to the \--detgrowquant
option in NoiseChisel. The top-left image is the input image. Each subsequent
binary image is the grown detection map after the growth has been completed to
the given quantile. All other NoiseChisel parameters are identical.

  * •

\--cleangrowndet: After dilation, if the signal-to-noise ratio of a detection
is less than the derived pseudo-detection S/N limit, that detection will be
discarded. In an ideal/clean noise, a true detection’s S/N should be larger
than its constituent pseudo-detections because its area is larger and it also
covers more signal. However, on false detections (especially at lower
\--snquant values), the increase in size can cause a decrease in S/N below
that threshold.

When this option is called, if the grown S/N of a detected region is below the
pseudo-detection S/N, it is discarded. Because a true detection has flux in
its vicinity and dilation will catch more of that flux and increase the S/N.
This option will therefore improve purity and not effect the completeness (a
true detection will not be discarded).

However, in many real images bad processing creates artifacts that cannot be
accurately removed by the Sky subtraction. In such cases, this option will
decrease the completeness (will artificially discard true detections). So this
feature is not default and should to be explicitly called when you know the
noise is clean.

##  3 Segment program for Segmentation

Prior to Gnuastro version 0.6 (released in June 2018), one program
(NoiseChisel) was in charge of detection _and_ segmentation. To increase
creativity and modularity, NoiseChisel’s segmentation features were spun-off
into a separate program (Segment). Segment’s main algorithm and working
strategy were initially defined and introduced in Section 3.2 of AI15.

  * •

Since the spin-off from NoiseChisel, the default kernel to smooth the input
for convolution has a FWHM of 1.5 pixels (still a Gaussian). This is slightly
less than NoiseChisel’s default kernel (which has a FWHM of 2 pixels). This
enables the better detection of sharp/point-like clumps: as the kernel gets
wider, the lower S/N (but sharp/small) clumps will be washed away into the
noise. Gnuastro’s MakeProfiles can be used to build any custom kernel if this
is too sharp/wide for a certain purpose.

The ability to use a different convolution kernel for detection and
segmentation is one example of how separating detection from segmentation into
separate programs can increase productivity. The purpose of detection is
separating diffuse and extended emission from noise, but in segmentation,
sharp/localized peaks are desired.

  * •

The criteria to select true from false clumps is the peak significance. It is
defined to be the difference between the clump’s peak value ($C_{c}$) and the
highest valued river pixel around that clump ($R_{c}$). Both are calculated on
the convolved image (signified by the $c$ subscript). To avoid absolute values
(differing from dataset to dataset), $C_{c}-R_{c}$ is then divided by the Sky
standard deviation under the river pixel used ($\sigma_{r}$) as shown below
(when \--minima is given, the nominator becomes $R_{c}-C_{c}$):

$C_{c}-R_{c}\over\sigma_{r}$

The input Sky standard deviation dataset (\--std) is assumed to be for the
unconvolved image. Therefore a constant factor (related to the convolution
kernel) is necessary to convert this into an absolute peak significance333To
get an estimate of the standard deviation correction factor between the input
and convolved images, you can take the following steps: 1) Mask (set to NaN)
all detections on the convolved image with the where operator or Gnuastro’s
Arithmetic program. 2) Calculate the standard deviation of the undetected
(non-masked) pixels of the convolved image with the \--sky option of
Gnuastro’s Statistics program (which also calculates the Sky standard
deviation). Just make sure the tessellation settings of Statistics and
NoiseChisel are the same (you can check with the -P option). 3) Divide the two
standard deviation datasets to get the correction factor.. However, as far as
Segment is concerned, the absolute value of this correction factor is
irrelevant: a purity level is found based on the ambient noise (undetected
regions) to find the numerical threshold of this fraction. The user doesn’t
directly specify this value by default (unless \--clumpsnthresh is given).

A distribution’s extremum (maximum or minimum) values, used in the new
criteria, are strongly affected by scatter. On the other hand, the convolved
image has much less scatter444For more on the effect of convolution on a
distribution, see Section 3.1.1 of AI15. Therefore $C_{c}-R_{c}$ is a more
reliable (with less scatter) measure to identify signal than $C-R$ (on the
unconvolved image).

Initially, the total clump signal-to-noise ratio of each clump was used, see
Section 3.2.1 of AI15. Therefore its completeness decreased dramatically when
clumps were present on gradients for example see Figure 4. In tests, this
measure proved to be more successful in detecting clumps on gradients and on
flatter regions simultaneously.

  * •

With the new \--minima option, it is now possible to detect inverse clumps
(for example absorption features).

##  4 Demonstration: M51 outskirts on a shallow image

As a small demonstration of the detection and segmentation power of Gnuastro,
a single exposure (exposure time of 54 seconds), SDSS [1] image of M51 and
NGC5195 in the _r_ -band is used. It is a raw image directly downloaded when
searching for NGC5195 in the SDSS “simple field search” online tool.

As a reference for readers who may be more familiar with SExtractor, it is run
on the image first (version 2.25.0). The borders of SExtractor’s detected
pixels that overlap with M51 and NGC5195 are marked with red in Figure 3. The
default value of all the SExtractor parameters were used (generated with the
-d option) except for DETECT_THRESH=0.5 and ANALYSIS_THRESH=0.5. The default
thresholds of 1.5 (multiple of $\sigma$) were far too high and useless in this
scenario. Such a low threshold is rarely used because it creates many false
positives (see Sections 5.2 and B.1.3 in AI15).

Figure 3:  Detection of M51’s outer wings on a single-exposure SDSS image. (a)
Input image overlayed with the border of detected regions. The lower-S/N
regions of the image are shown in a logarithmic scale, the brighter parts are
shown in linear scale to simultaneously show the familiar bright structure and
outer low surface brightness wings. The red boundary shows SExtractor’s
detection (default parameters, except thresholds that are set to $0.5\sigma$).
The light-blue boundary shows the outer edge of the detection with
NoiseChisel’s default configuration, and has an average S/N of $0.59$ (or
$27.34$ mag/arcsec2). The green boundary shows the result after optimizing the
configuration for the noise of single-exposure SDSS images. The green boundary
reaches an average S/N of $0.25$ (or $28.27$ mag/arcsec2). For more on the
customization of SExtractor and NoiseChisel, see Section 4. (b) Optimized
NoiseChisel Sky image, showing that there is still signal outside its detected
regions. (c) Much deeper, processed image from [15] showing the outer signal
at much higher S/N, reproduced by permission of the AAS.

However, even with such a decreased signal, it can only detect the very bright
parts of the signal. This is primarily because of poor Sky subtraction in the
vicinity of diffuse signal, see Figure 16 and Section A.6 of AI15. All the
small detections also had the same label as the large detections, similar to
Figure 1(2). This is a big problem in a general catalog, where all these
background galaxies will be missed in the final catalog.

NoiseChisel (from Gnuastro 0.10) is also run on this image. It is first run
with its default parameters (no run-time options), producing the light-blue
boundary in Figure 3 (S/N=$0.59$, or $27.34$ mag/arcsec2). However, the
default parameters are only for a generic noise pattern. Once NoiseChisel is
slightly tweaked for the single-exposure SDSS images (command below), it is
able to detect the wings of M51 much more deeply as shown in the green border
(S/N=$0.25$, or $28.27$ mag/arcsec2, almost one magnitude deeper than the
default):

astnoisechisel m51-r.fits -h0 --tilesize=75,75 --meanmedqdiff=0.001 \

\--interpnumngb=6 --snquant=0.95 --detgrowquant=0.65 \

\--detgrowmaxholesize=10000 --minskyfrac=0.9

The displayed boundaries and surface brightness measurements of Figure 3
correspond to the NoiseChisel output after being eroded two times. The reason
for this can be seen in Figure 2. As \--detgrowquant is decreased, when there
is no signal, NoiseChisel starts following correlated noise that usually
manifests as thin connections between smaller detections (like the three small
detections that start touching beyond \--detgrowquant=0.65 in Figure 2).
However, when there is signal (like the outer parts of the central galaxy), it
is clear that NoiseChisel is detecting very low S/N signal and such thin
branches aren’t created. Erosion removes the thin connections while not
affecting the large contiguous diffuse detections.

Figure 3(b) shows the Sky image (average of undetected pixels over a grid) for
the optimized NoiseChisel run. Even though $28.27$ mag/arcsec2 seems very deep
for a short-exposure image, the Sky image still resembles the shape of M51,
showing that there is still fainter signal in this image. In the tutorial, the
readers are encouraged to optimize NoiseChisel even further to obtain a better
result. Finally, Figure 3(c) shows a rotated crop from the much deeper study
of M51 by [15], confirming the physical reality of this signal with the same
morphology.

The logic behind finding the best parameters used in the command above is
described in the “Detecting large extended targets” tutorial of the Gnuastro
manual. This configuration is good for any single-exposure SDSS image. When
dealing with a new type of dataset (images with different cameras, or reduced
differently), for an optimal result, its always good to tweak it a little and
optimize it for that particular noise-pattern.

Figure 4 shows the result of Segment (with its default configuration) on the
brighter part of the M51 image. It clearly shows the effectiveness of the new
criteria to identify true peaks discussed in Section 3. Each one of the HII
regions in Segment’s clumps can then be cataloged with MakeCatalog to be
studied individually, or masked to study the diffuse region.

Figure 4:  Identification of HII regions over the M51 spiral arm in Figure 3
using Segment. (a) Input image, (b) all detected clumps masked, (c) the
detected clumps.

##  5 Reproducible data analysis

As shown in Figure 3, when NoiseChisel is optimized for a special noise
pattern (in this case, any single-exposure SDSS image), it can operate
significantly better. But this isn’t peculiar to NoiseChisel, scientific
software need to allow their users to modify every step of their analysis.
Some software authors try to decrease the number of tweak-able parameters by
hard-coding the parameters within the code. But this decreases the ability of
the user (a scientist) to understand the details of the analysis and its
effect on their scientific interpretation. It is therefore critical for the
integrity of the result that the authors report the exact details of all their
software’s configuration options, what order they were run, how they were
installed and etc. Simply reporting the name of a software is not enough. But
the format of a traditional paper like this also doesn’t allow going into too
much detail.

Generally, given the large volume of data already in surveys like HSC and DES
(which will increase dramatically with LSST), and the many configuration
options (parameters to all the software used, software versions, and their
operating environment), it is becoming harder to test new configurations,
methods or software and thus understand the limits of the final choice in the
low S/N regime. This is a major hurdle in using such studies for the low
surface brightness universe. Such tests can be facilitated/encouraged in a
design where previous results can be exactly reproduced, on different
machines, independent of the host operating system.

To address this major problem, AI15 also publicly released all the analysis
scripts and configuration files that generated its results with the paper’s
LaTeX source on arXiv. Since that paper, that solution has grown into a
“reproducible paper template”555https://gitlab.com/makhlaghi/reproducible-
paper, which provides a robust framework to start reproducible research
projects. It installs fixed versions of all the necessary software from source
and cites them where necessary (see Appendix A). Combined with all the scripts
that produced every number or figure in this paper, this allows exact
reproducibility of this paper by any random reader. This paper has been
written using this template and can be exactly reproduced/verified/modified by
retrieving version v4-0-g8505cfd of its
history666https://gitlab.com/makhlaghi/iau-symposium-355. All necessary files
(reproducible source, its Git history and software tarballs) are also archived
on Zenodo (https://doi.org/10.5281/zenodo.3408481). The reproducible source
includes links to download the input images from their respective server, as
well as their MD5 checksum to confirm that the downloaded file is the same
that was used here (has not been updated on the server).

##  6 Acknowledgements

This research was done using GNU Astronomy Utilities (Gnuastro,
ascl.net/1801.009), and the reproducible paper template v4-0-g8505cfd. Work on
Gnuastro and the reproducible paper template has been funded by the Japanese
Ministry of Education, Culture, Sports, Science, and Technology (MEXT)
scholarship and its Grant-in-Aid for Scientific Research (21244012, 24253003),
the European Research Council (ERC) advanced grant 339659-MUSICOS, European
Union’s Horizon 2020 research and innovation programme under Marie Sklodowska-
Curie grant agreement No 721463 to the SUNDIAL ITN, and from the Spanish
Ministry of Economy and Competitiveness (MINECO) under grant number
AYA2016-76219-P.

This research was done using data from SDSS. Funding for the SDSS and SDSS-II
has been provided by the Alfred P. Sloan Foundation, the Participating
Institutions, the National Science Foundation, the U.S. Department of Energy,
the National Aeronautics and Space Administration, the Japanese
Monbukagakusho, the Max Planck Society, and the Higher Education Funding
Council for England. The SDSS Web Site is http://www.sdss.org.

## References

  * [1] K.. Abazajian et al.  “The Seventh Data Release of the Sloan Digital Sky Survey”  In _ApJS_ 182, 2009, pp. 543  DOI: 10.1088/0067-0049/182/2/543
  * [2] M. Akhlaghi and T. Ichikawa  “Noise-based Detection and Segmentation of Nebulous Objects”  In _ApJS_ 220, 2015, pp. 1  DOI: 10.1088/0067-0049/220/1/1
  * [3] R. Bacon et al.  “The MUSE Hubble Ultra Deep Field Survey. I. Survey description, data reduction, and source detection”  In _A &A_ 608, 2017, pp. A1  DOI: 10.1051/0004-6361/201730833
  * [4] E. Bertin and S. Arnouts  “SExtractor: Software for source extraction.”  In _A &AS_ 117, 1996, pp. 393  DOI: 10.1051/aas:1996164
  * [5] Alejandro Borlaff et al.  “The missing light of the Hubble Ultra Deep Field”  In _A &A_ 621, 2019, pp. A133  DOI: 10.1051/0004-6361/201834312
  * [6] R. Calvi et al.  “MOS spectroscopy of protocluster candidate galaxies at z= 6.5”  In _MNRAS_ 489, 2019, pp. 3294  DOI: 10.1093/mnras/stz2177
  * [7] M. Frigo and S.. Johnson  “The Design and Implementation of FFTW3”  In _IEEE Proc._ 93.2, 2005, pp. 216  DOI: 10.1109/JPROC.2004.840301
  * [8] Henry H. Hsieh et al.  “Maximizing LSST Solar System Science: Approaches, Software Tools, and Infrastructure Needs”  In _Preprint_ , 2019, pp. arXiv:1906.11346  arXiv:1906.11346 [astro-ph.EP]
  * [9] G.. Illingworth et al.  “The HST eXtreme Deep Field (XDF): Combining All ACS and WFC3/IR Data on the HUDF Region into the Deepest Field Ever”  In _ApJS_ 209.1, 2013, pp. 6  DOI: 10.1088/0067-0049/209/1/6
  * [10] H. Inami et al.  “The MUSE Hubble Ultra Deep Field Survey. II. Spectroscopic redshifts and comparisons to color selections of high-redshift galaxies”  In _A &A_ 608, 2017, pp. A2  DOI: 10.1051/0004-6361/201731195
  * [11] Seppo Laine et al.  “LSST Cadence Optimization White Paper in Support of Observations of Unresolved Tidal Stellar Streams in Galaxies beyond the Local Group”  In _Preprint_ , 2018, pp. arXiv:1812.04897  arXiv:1812.04897 [astro-ph.GA]
  * [12] Oliver Müller et al.  “A tidal tale: detection of several stellar streams in the environment of NGC 1052”  In _A &A_ 624, 2019, pp. L6  DOI: 10.1051/0004-6361/201935463
  * [13] Marc Rafelski et al.  “UVUDF: Ultraviolet Through Near-infrared Catalog and Photometric Redshifts of Galaxies in the Hubble Ultra Deep Field”  In _AJ_ 150.1, 2015, pp. 31  DOI: 10.1088/0004-6256/150/1/31
  * [14] Ignacio Trujillo et al.  “A distance of 13 Mpc resolves the claimed anomalies of the galaxy lacking dark matter”  In _MNRAS_ 486.1, 2019, pp. 1192  DOI: 10.1093/mnras/stz771
  * [15] Aaron E. Watkins et al.  “Deep Imaging of M51: a New View of the Whirlpool’s Extended Tidal Debris”  In _ApJL_ 800.1, 2015, pp. L3  DOI: 10.1088/2041-8205/800/1/L3

##  Appendix A Software acknowledgement

This research was done with the following free software programs and
libraries: Bzip2 1.0.6, CFITSIO 3.47, CMake 3.15.3, cURL 7.65.3, Discoteq
flock 0.2.3, FFTW 3.3.8 [7], File 5.36, Git 2.23.0, GNU Astronomy Utilities
0.10 [2], GNU AWK 5.0.1, GNU Bash 5.0.11, GNU Binutils 2.32, GNU Compiler
Collection (GCC) 9.2.0, GNU Coreutils 8.31, GNU Diffutils 3.7, GNU Findutils
4.7.0, GNU Grep 3.3, GNU Gzip 1.10, GNU Integer Set Library 0.18, GNU libiconv
1.16, GNU Libtool 2.4.6, GNU M4 1.4.18, GNU Make 4.2.90, GNU Multiple
Precision Arithmetic Library 6.1.2, GNU Multiple Precision Complex library,
GNU Multiple Precision Floating-Point Reliably 4.0.2, GNU NCURSES 6.1, GNU
Readline 8.0, GNU Scientific Library 2.6, GNU Sed 4.7, GNU Tar 1.32, GNU
Texinfo 4.7, GNU Wget 1.20.3, GNU Which 2.21, GPL Ghostscript 9.27, Libbsd
0.9.1, Libgit2 0.28.2, Libjpeg v9b, Libtiff 4.0.10, Lzip 1.20, Metastore
(forked) 1.1.2-23-fa9170b, OpenBLAS 0.3.5, OpenSSL 1.1.1a, PatchELF 0.10, pkg-
config 0.29.2, Sextractor 2.25.0 [4], Unzip 6.0, WCSLIB 6.4, XZ Utils 5.2.4,
Zip 3.0 and Zlib 1.2.11. The LaTeX source of the paper was compiled to make
the PDF using the following packages: biber 2.13, biblatex 3.13a, caption
2019-09-01, courier 2016-06-24, csquotes 5.2e, datetime 2.60, ec 1.0, etoolbox
2.5f, fancyhdr 3.10, fmtcount 3.05, fontaxes 1.0d, footmisc 5.5b, fp 2.1d,
kastrup 2016-06-24, logreq 1.0, newtx 1.554, pgf 3.1.4b, pgfplots 1.16,
preprint 2011, setspace 6.7a, tex 3.14159265, texgyre 2.501, times 2016-06-24,
titlesec 2.11, txfonts 2016-06-24, ulem 2016-06-24, xcolor 2.12 and xkeyval
2.7a. We are very grateful to all their creators for freely providing this
necessary infrastructure. This research (and many others) would not be
possible without them.


# The Fault in the Stars: Understanding the Underground Market of Amazon
Reviews

Rajvardhan Oak
University of California, Berkeley
<EMAIL_ADDRESS>The wok was done while the author was a graduate studnt.
The author presently works at Microsoft. This research is not affiliated to,
sponsored or endorsed by Microsoft in any way.

###### Abstract

In recent times, the Internet has been plagued by a tremendous amount of
misinformation. Online markets such as Amazon are also not free from
misinformation. In this work, we study the misinformation propagated to
consumers through the form of Amazon reviews. There exists a vast underground
market where reviews by real Amazon users are purchased and sold. While such a
practice violates Amazon’s terms of service, we observe that there exists a
complex network consisting of thousands of sellers and agents, who provide a
rebate to consumers for leaving positive reviews to over $5000$ products.
Based on interviews with members involved in the reviews market, we understand
the working of this market, and the tactics used to avoid detection by Amazon.
We also present a set of recommendations of features that Amazon and similar
online markets can take into consideration to detect such reviews.

##  1 Introduction

In recent times, the Internet has been riddled with disinformation in the form
of fake news articles and deep fakes. Social media platforms have been plagued
by hoax articles, with false claims seemingly made by experts. Online markets
such as Amazon have not been left unaffected by misinformation. Misinformation
exists on such platforms through customer reviews, an important factor driving
product sales. In recent times, a massive, complex underground market,
spanning across multiple countries has developed which works hard at
manipulating reviews on Amazon.

The goal of this market is to obtain a large number of verified, positive,
five-star rated reviews for products. A review is a verified review if the
person writing the review has purchased the product through Amazon. As a
tactic to promote sales, Amazon always shows the ’top reviews’ for any product
while viewing product information. Large number of positive verified reviews
also help improve the product’s ranking. More sales and five-star reviews can
also result in the product being marked as Amazon’s Choice or Best Seller in a
particular category.

Reviews have become an increasingly important parameter for judging a product
quality before making a purchase. $82\%$ of adults in the United States check
product reviews before buying a product online [10]. Studies [4] have shown
that positive customer reviews strongly influence a buyer’s decision to
purchase a new product.

Amazon sellers (not manufacturers) are at the heart of this underground market
for fake reviews. These are third-party entities who can list products for
sale. Such sellers recruit ’review agents’, who help them increase their sales
numbers and gain positive reviews. The agents contact buyers, convince them to
buy the product, leave a review, and then refund them via PayPal or Amazon
gift cards. In this underground market, buyers are often referred to as
’jennies’. The seller gains a large number of organic, verified reviews, and
the buyer gets to keep the product for free.

This can help sellers and their products to bootstrap themselves by gaining
sales and reviews for new items. These activities, and any other activities
which manipulate reviews are against Amazon’s terms of service. Offering
discounts for reviews, or giving away free products for positive reviews is
banned.

For products which need bootstrapping in order to promote sales and gain
reviews, Amazon offers a program known as the Amazon Vine Program
111https://www.amazon.com/gp/vine/help. Highly rated reviewers are allowed to
select products, get them for free and honestly review them. Sellers, however,
prefer to go the underground route; they are guaranteed to get positive, five-
star reviews irrespective of how the product is. The seller market is heavily
saturated; according to a study [3], $50$ new products were uploaded on Amazon
every second from Chinese sellers.

This network of sellers, agents, and reviewers spans across multiple
countries. Our investigations revealed that nearly all of the sellers were
from China. The agents worked out of India, Pakistan, China and Bangladesh.
These agents reach out to Amazon buyers in the United States, Canada, Germany,
UK and France. Buyers are contacted via targeted advertisements and dedicated
groups on Facebook.

Contributions. In this work, we study this complex underground market and aim
to understand it’s functioning. We reach out to agents and sellers to learn
how the market works. Our investigation uncovers several evasion techniques
used in order to avoid detection by Amazon.

We summarize our contributions as follows:

  1. 1.

We study the intricacies, functioning and caveats of the underground market
for incentivized Amazon reviews. To the best of our knowledge, this study is
the first of its kind.

  2. 2.

We interview agents and buyers involved in this market and study their
motivations and incentives. In addition, we also identify key tactics used by
reviewers to avoid detection by Amazon.

  3. 3.

Through our research, we demonstrate how Facebook and Instagram, with their
targeted advertisement program further fuel this market and attract more and
more users.

  4. 4.

We present recommendations of features that online markets can take into
consideration for increased fidelity in detection of such incentivized
reviews.

##  2 Related Work

The issue of illegal incentivized reviews has received wide media coverage [8]
[9] [2]. Journalists have studied the market by speaking to sellers, buyers,
agents, and even Amazon. These journalistic reports reveal that Amazon is
aware of the incentivized reviews and says that they make for less than $1\%$
of the total reviews on the platform [8]. They also reveal that the market
operates through all social media communication channels: facebook, Slack,
Discord and even Reddit [9]. One report [8] finds that incentivized reviews
adversely hurt genuine sellers. According the report, a family run business
for bedsheet fasteners suddenly suffered a 50% loss in revenue because of
counterfeit products; these sellers manufactured duplicate products, and
obtained incentivized reviews for them which ranked them higher in a short
period. Another report reveals that Amazon is trying to crack down on these
incentivized reviews using machine learning systems in combination with social
media to detect them and stop them at source [2]. A third report explores the
seller perspectives in the market; sellers spend as much as $\$15000$ in order
to increase their product ratings. The report also reveals that the
underground review market has given rise to several other businesses; those
which claim to understand the parameters Amazon uses in order to detect
whether a review is fake or not [7].

We were unable to find substantial academic works studying the underground
review market. A prior work [6] studied fake reviews and their effects on
travel websites TripAdvisor and Expedia. The closest and most relevant work to
ours was He et al. [5]. In this work, the authors study the underground market
and how it functions, but focus more on the kind of the products that solicit
reviews and characterizing product outcomes before and after reviews. Their
study on $1500$ products shows that these fake reviews lead to a significant
increase in average rating and sales rank but the effect disappears after
around a month. The authors also observe that as much as a third of the
reviews are deleted by Amazon and most of them are five-star reviews. The
dates of review deletion falls roughly a month or two after the campaign on
Facebook has ended. The fact that Amazon aggressively deletes reviews is
surprising; a recent study [1] shows that flagging reviews as fake but leaving
them on the platform can increase consumer trust.

Our work is different from prior works. First, we study the underground review
economy in a structured, qualitative manner by interviewing both agents and
buyers/reviewers who participate. Second, we focus on how the reviewers
attempt to evade detection by Amazon.

##  3 Market Functioning

###  3.1 Overview

Most sellers cover the PayPal fees incurred as part of the refund. Many of
them also cover the sales taxes that buyers have to pay. A small number of
products also offer additional commission on top of the PayPal fees and taxes.

###  3.2 Understanding the Process

Through our discussions with the buyers and agents, we were able to understand
how the review-refund process works in detail.

  1. 1.

The agent/seller engages with a potential buyer via Facebook or Instagram.
This is generally through the buyer clicking on a targeted advertisement.

  2. 2.

The buyer is asked for their Amazon profile. This is to look at whether the
particular account will be able to leave a review or not. Buyers are also
asked how old the account is, and if any reviews have been removed by Amazon.

  3. 3.

The buyer is asked their PayPal email. This will be used to refund the buyer
later on.

  4. 4.

The agent shares the search keywords and product images. The buyer searches
for the product, and sends a screenshot to confirm.

  5. 5.

Once confirmed, the buyer places the order. They send a screenshot of the
order consisting of the order number and total amount to the seller/agent.

  6. 6.

After the order is received, the buyer leaves a review. Once the review is
live, they send a screenshot to the seller/agent.

  7. 7.

Upon checking the verified review, the seller/agent refund the buyer the full
amount of the product via PayPal.

##  4 Research Methodology

In order to understand the underground market for reviews, we conducted a
qualitative research study. We interviewed agents as well as buyers who are
involved in these activities.

###  4.1 Participant Recruitment

We were able to find $37$ groups on Facebook which served as forums for
sellers and agents to post about the products they had. From the posts and
comments, we were able to infer whether a particular member was a buyer or a
seller. We contacted $55$ agents and asked them to be part of our study. Out
of these, $39$ responded positively and answered our questions.

Recruiting buyers for our study wasn’t as simple. We observed a lack of
response from the first several buyers we contacted. We created a dummy
account and posed as buyers, and we realized that buyer inboxes are flooded
with messages from sellers and agents. Through Facebook groups, we contacted
$85$ buyers out of which $30$ responded. We were able to identify other buyers
by reading the reviews of the advertised products on Amazon; we tried to track
them down on Facebook, and were able to recruit $5$ more. Thus, overall, we
interviewed $39$ agents and $35$ buyers.

###  4.2 Interview Methodology

All of our interviews were conducted online in form of chat via Facebook
messenger or email. Since engaging in activities related to incentivized
reviews is against Amazon’s terms of service, participants did not agree to
talk to us on the phone or via video calls.

Target Group | Category |  Questions
---|---|---
Agents | Communication and Outreach |  How do the sellers get in contact with you? Can you share how you got involved in this market?
How do you look for potential buyers? How do you communicate with them?
What media do you use to communicate with buyers and sellers?
How do you protect your identity on social media?
Process Overview |  How does the entire refund process work?
Are the product taxes and PayPal fees covered by the seller?
How do you share product information with buyers?
How do you make sure that reviews are not removed?
How do you keep track of orders and refunds?
Incentives |  Are you aware that this is against Amazon’s TOS?
How much do you earn per review?
What do sellers gain through this activity, since they refund the full price?
How many products and sellers do you deal with?
Buyers | General Questions |  How did you get introduced to these opportunities for reviews in exchange for products?
How do you keep track of items purchased and reviewed?
How do you search for the correct product?
What has been your experience in writing reviews and receiving the refund?
How do you make sure that your account is not flagged by Amazon for fraud?
Incentives |  How many items do you review per week?
Are you aware that these activities violate Amazon’s TOS?
How much bonus do you earn on an average per week?
Table 1: Questionnaire of Interview Questions asked to participants.

###  4.3 Ethical Considerations

We did not collect any personally identifying information during the course of
this study. Since all of our interactions with the participants were online,
we are not even aware of the real identities of most people. Only $2$ out of
the $39$ agents and $5$ out of the $34$ buyers used their real-world
identities; the rest of them used a fake online alias. As a result, we do not
have any information about sensitive or protected attributes.

All the participants were aware that the answers they provide will be reported
as part of a research study. Participants were recruited only after they
provided us consent for participation. All participants had the choice to
decline to answer any of the questions, or withdraw their participation at any
time.

Throughout the course of our study, we identified several sellers and brands
which engage in the illegal activity of buying reviews on Amazon. However, we
do not reveal their identities. If we were to do so, the sellers could trace
the source back to our participants, which would jeopardize their anonymity.

##  5 Agent and Reviewer Mechanics

###  5.1 Incentives

As part of our study, we seek to understand the motivation and incentives for
both buyers and sellers in participating in such activities. Our interviews
reveal that both buyers and agents are aware that writing incentivized reviews
is against Amazon terms of service. In fact, $20$ of the reviewers we
interviewed had a backup Amazon account with different payment methods. The
incentive, however, is financial.

Agent Incentives. Agents are paid a commission on every review they secure.
The commission ranges anywhere from $4$ USD to $10$ USD. This is still a
significant amount in many developing countries; as an agent A3, who is based
in Pakistan, tells us:

"I get at least $4 per review, which is roughly 660 PKR. I can buy at least
five whole meals here for that amount. And I can get this in a matter of
minutes, without much work."

The commissions add up quickly, leading to a significant amount at the end of
the month. Most agents work on getting reviews as a second, night-shift job;
this serves them well, as their target customers are mainly in western
countries (A6). According to agent A28, based in Bangladesh:

"One month I earned almost $250; my regular job pays me around $100 a month.
And it requires no additional expense; just an internet connection."

Buyer Incentives. A clear motivation for the buyers is the opportunity to get
products for free [9]. A reviewer R20 says:

"I know it’s wrong….but when I moved into my new place, I was able to get a
chair, desk, humidifier, kitchen utensils, storage boxes, vacuum cleaners and
a lot more for free. I saved so much by just writing reviews."

Other buyers go a step ahead. They get their products for free by writing
reviews, and then sell them online. Reviewer R1 tells us:

"The product is basically free. Once I get my refund, I sell it on Facebook
marketplace for around $75\%$ of the price. I made over $300 last month."

###  5.2 Communication Tactics

Reviewers are contacted primarily via Facebook. There are two mediums by which
agents recruit reviewers: Facebook groups and targeted advertisements.

Facebook Groups. We were able to find over $100$ groups on Facebook which
served as a hub for reviewers and sellers to contact each other. The largest
of these groups had close to two hundred thousand members. On these groups,
agents post information about the products which they have. Many agents post
wide calls for action (see Fig. 1. Reviewers can also ask for any products
they want. Agents are aware that what they are doing is illegal, and hence,
always operate with an alias. They also use minor perturbations on the
keywords so that they may not be flagged by Facebook (see Fig. 3). Agent A25
says:

"I always share secondary contact information like email or WeChat, as FB
often deletes our [fake] accounts. I also avoid posting sentences which convey
that we are offering refund in return for review. I always use words like
re3view, r*fuund, amz*n, etc."

Figure 1: Agents soliciting reviewers on Facebook groups Figure 2: Targeted
Advertisements on Facebook Figure 3: Agents modifying keywords in order to
avoid detection

Targeted Advertisements. This is primarily how people are exposed to this
underground market for reviews. Agents have pages on Facebook, which can then
deliver targeted advertisements to users. These advertisements commonly tout
their business as ’looking for influencers’, or ’looking for testers’ for
their product. A few sample sponsored posts can be seen in Fig. 2. Clicking on
the advertisement leads to opening a chat with the page via Facebook
Messenger. There is a chatbot which asks users for their Amazon account links,
PayPal accounts, and explains the process. A particular page can deal with
hundreds of products; see Fig. 4.

Facebook does seem to be on the lookout for such pages; a reviewer R18 shared
that he received an email from one of the pages saying how their Facebook page
has been blocked. However, agents simply create another page with a different
account and start operating and delivering advertisements from that page. See
Fig. 5 for a sample email that R18 received.

Figure 4: Chat with bot contacted through a targeted advertisement Figure 5:
Agents on their pages being blocked

###  5.3 Malicious Actors

The underground market is not without it’s fair share of malicious actors.
Such actors exists among both reviewers and agents. $30$ of the reviewers said
that they had been scammed at least once. Reviewer R15 tells us:

"It happens often. Some agents submit your review to the seller and give their
own PayPal for refund, which means that they get the money. But in that case,
we can just return the product and get the refund from Amazon itself. So, not
our loss."

Dishonest actors among reviewers seem to be fewer; only $10$ agents noted that
they had been scammed at least once. Dishonest reviewers get the refund, and
then return the product to Amazon within the 30-day return window; thus
earning the full amount of the product without spending anything. This affects
agents adversely; as agent A4 says:

"We are paid our commission at the end of the month. If an order is returned,
the seller does not pay us the commission for that review."

There is also a system in place to identify such malicious actors; there are
Facebook groups in which reviewers post agents who have scammed them, and
agents post the reviewers who scammed them. $24$ of the reviewers we talked to
said that they first check for the agent names on these groups before placing
an order.

###  5.4 Role of Social Media

Our research reveals that Facebook plays an important role in the continued
operation of the underground reviews market. Facebook facilitates the delivery
of advertisements to a targeted audience; therefore the pages (as described in
Section 5.2 are able to leverage search history, marketplace activity and
other data available to Facebook to identify users who might be interested in
writing such incentivized reviews. Further, a direct fallout of the targeted
advertisements is the rabbit hole phenomenon; once a user clicks or interacts
positively with an advertisement for free products, he goes down the rabbit
hole and sees several more advertisements of similar pages. Reviewer R32 says:

"I just chanced upon this side-hustle one day; I clicked on the advertisement,
and then saw a few more. Pretty soon, my news feed was full of such
advertisements; every third or fourth post I saw was for free products in
exchange for reviews."

Thus, targeted advertisements facilitate identifying users who might be
complicit with writing incentivized reviews, and gradually sucks them deeper
into the underground market. Further, since Instagram is also owned by
Facebook, targeted advertisements can work across the platforms. The pages can
leverage users Instagram activity data to target them. Once a user clicks on
an advertisement on Facebook, he is shown similar advertisements on Instagram
as well.

##  6 Evasion Tactics

In this section, we discuss the suggestions and tactics that reviewers are
asked to follow. Agents and sellers provide these guidelines during the
purchase and before writing a review. The goal of these is to minimize the
probability of your purchase and the following review being considered a
fraudulent activity, and subsequently being deleted by Amazon.

###  6.1 Finding the Right Product

In order for the process to work smoothly, buyers must purchase the exact
product shown. The brand name and the seller must be the same. Sometimes,
buyers may be forced to buy the item at a higher price in order to match the
seller name. A reviewer(R1) notes:

"I once found the product with the correct brand after going to page 5 of
Amazon search results, but the price shown was around $3 less than what the
agent shared with me. I asked if it was okay and they declined. I had to go to
the list of sellers available for that product and choose the specific one.
That gave me the correct price."

Agents reveal that even if the same brand is purchased, which seller it was
purchased from matters. Agents are hired by sellers. An agent(A7) tells us:

"We are paid our commission by sellers, not the brands. If the buyer chooses a
different seller, our seller does not gain anything and his rank does not
improve. Thus, he will not refund for such an order."

###  6.2 Organic Search for Product

Sellers always recommend buyers to search for the product with keywords,
rather than provide a direct link or the brand name. Buyers are then asked to
scroll through the products and find the correct item. According to agents,
this is because Amazon can track that you landed on that product from a link,
and can consider it to be suspicious. An agent (A3) notes:

"It is for security of your account. Sellers don’t allow us to share direct
links with buyers, as Amazon will detect this, and may remove the review that
you write."

Similarly, buyers are advised never to search for a product with the brand
name in the search term. As far as possible, agents never directly disclose
the brand name. A reviewer(R2) describes this:

"They never show us the brand. In the image they send us, the brand name is
blurred out. I try to find the correct product and send them a screenshot or
link to confirm."

###  6.3 Platform Engagement

Buyers are encouraged to engage with the Amazon platform while searching for
the product. They are asked to look at similar products, browse through
images, and read reviews. The goal of this is to simulate an organic product
discovery experience. An agent(A8) says:

"We always ask buyers to spend at least 1 minute browsing similar products,
and at least 15 seconds on checkout page since if search and purchase is done
too quickly, it is suspicious to Amazon."

Buyers generally follow the guidelines, since their accounts are at risk. A
reviewer(R1) also concurs, saying:

"They want us to convince amazon that we actually searched for, found and
liked this product. I am also concerned for my account; I do not want to get
banned from the site. So even when I have found the product, I browse as I
would normally do. I read reviews, mark them as helpful and watch product
videos."

###  6.4 Payment Restrictions

Buyers are advised not to pay using gift cards, and always make the full
payment. In addition, Amazon at times displays coupons using which buyers can
get a discount on the product; buyers are cautioned not to use these either.
The reason behind this is that the seller has designated a fixed number of
coupons, and these are to attract other, genuine customers to buy the product.
An agent (A4) says:

"Most sellers tell us not to ask buyers to use coupons. If they do, they still
refund them, as it is not a loss. But the coupons are mainly there for other
customers (who we will not be refunding) to think that they are getting a
discount."

At this time, we do not know why sellers do not want buyers to use gift card.
None of the agents we spoke to were aware of the reasoning behind the
instruction. However, one reviewer (R11) notes:

"….I generally try not to use the gift card balance. One time I accidentally
used $5 from my gift card balance, and they refunded me $5 less. Now I have to
manually uncheck any gift card balance available, since Amazon uses it by
default."

###  6.5 Review Timing

Buyers are recommended to wait at least $10$ days before submitting a review
on Amazon. Reviews which are submitted too soon arouse suspicious. According
an agent (A32):

"…Please submit reviews 10-15 days after you receive the shipment. Use the
product for a few days and then review. Else, your review will be removed by
Amazon."

Buyers are, of course, eager to receive the refund as soon as possible.
However, they understand the reasoning. Reviewer R4 says:

"It makes sense. I need to have experienced the product before writing a
review, otherwise it is suspicious. I maintain a spreadsheet of items I
purchased along with their dates, so I know when 10 days have passed to write
the review."

Furthermore, agents prefer those accounts who have not written more than $3$
reviews in a week, and not more than $10$ per month. Agent A22 tells us:

"Sellers do not want reviews from accounts who write too many reviews, such
reviews always get removed by Amazon. We decline if we see too many recent
reviews, and all of them five-star."

###  6.6 Review Content

Agents provide detailed instructions to buyers on the content of the reviews.
According to agents, factors like the length of the review, and presence of
supplementary media (images and videos) affects whether Amazon will remove the
review. All ratings have to be five-star. Agents always recommend writing a
review of at least $100$ words. A reviewer (R19) tells us:

"Picture and video reviews are really important to them. Once, an agent
offered me additional commission of $5 if I wrote a review of more than 200
words. And $5 more if I attached a picture or video to it."

In addition, they are cautioned never to strongly emphasize any particular
seller. An agent (A30) notes:

"Never ever mention ’always buy from this seller’ in your review. We want the
product from our seller to be highly ranked. If you write about a seller,
Amazon moves the review to seller feedback, which sellers don’t want, because
customers don’t view seller feedback often."

Agents seem to know a lot about the internal workings of amazons detection
models. An agent A27 tells us:

"There are two computer programs that Amazon uses to detect reviews. One looks
at content and the other looks at timing, search behavior, etc. We know that
the review content program checks if you have shared your opinion and not
simply information about the product."

###  6.7 Seller, Agent and Product Similarity

In general, buyers are not permitted to review multiple products from the same
seller within a certain period of time. According to agent A21:

"Very suspicious if you buy too many products from same store. I never return
to the same buyer before 2 months of the first review."

Reviewers have another thumb rule which they follow, and that is avoiding
buying too many products offered by the same agent. Reviewer R15 shares his
experience:

"I purchased a vacuum cleaner and sent the order screenshot to the agent. She
confirmed my order, and showed me some more products to buy. When I went to
amazon, I saw the SAME product as a suggestion, under the heading ’people with
similar purchases to you also bought….’. It would have been very suspicious.
Clearly, a pattern had been captured."

We observed that product similarity was not a concern amongst both buyers and
agents. We expected that the same item purchased and reviewed from various
brands from a single account was bound to be suspicious. However, that was not
the case. Many buyers purchased the same (or similar item) from different
brands and stores, and reviewed them. A reviewer R9 revealed that he had
reviewed $13$ different pairs of headphones in the past $6$ months. Similarly,
$28$ other reviewers reported having reviewed more than $10$ of the same
product within the past $6$ months.

###  6.8 Account Information

Agents and sellers also take into consideration account metadata while
analyzing whether a particular buyer’s review is likely to be removed or not.
Of our $39$ participating agents, $31$ reported that they strongly prefer
Amazon prime account holders. In addition, the account should be able to leave
reviews on Amazon; there is a requirement to have spent a minimum amount in
the past $6$ months in order to be eligible to write reviews. Agent A24 says:

"Amazon moderates all reviews before they go live on the product page. Reviews
by prime accounts are approved in around $36$ hours, but it takes around $96$
hours (4 days) for reviews from non-prime accounts. "

According to agents, reviews from accounts who have Amazon Prime subscriptions
are less likely to get removed. $24$ of our participating agents reported that
when some products had some reviews removed, all of them were from non-Prime
accounts. In addition, agents also check if a prospective buyer is an easy
rater, that is, he has too many five-star reviews and no three or four-star
reviews.

Agents also recommend that reviewers write reviews for items that they have
purchased organically (that is, without rebate). They also recommend that not
all reviews be five-star. An agent $A2$ tells us:

"I always make sure that the profile has some three and four star reviews as
well. Otherwise, Amazon might flag the person to be an easy rater. If he
easily gives five stars to all purchases, they find it suspicious. "

##  7 Recommendations

Through our research, we identified key tactics used by buyers and agents to
avoid detection. We also studied how the review market functions and what
aspects are important for it. Based on this, we recommend some features that
online markets like Amazon can use in order to detect such incentivized
reviews. These features may help improve machine learning models and increase
the fidelity of detections.

Emphasis on Sellers. Our investigations show that having the correct seller is
important for getting a refund. Amazon generally shows users the product from
the seller which is least expensive. Genuine buyers do not generally care
which seller the product comes from, since Amazon always provides a guarantee
for returns. Therefore, if a user deliberately changes the seller (as
described in Section 6.1), it may indicate that the review he writes
subsequently is incentivized and not a genuine one.

Unnecessary Media. As described in Section 6.6, agents believe that reviews
with photos and videos attached to them are less likely to be removed. This
leads to buyers overcompensating and attaching pictures even when not
necessary. $13$ of our reviewers noted adding pictures and videos even when it
did not make sense. For example, in products such as nutritional supplements,
USB chargers, and memory cards, photos and videos do not really add anything
to the review; everyone already knows what the product looks like in real
life. Thus, adding multimedia when not required may indicate that the review
is incentivized.

Ignoring Discounts. A reasonable consumer will try to get the best price for a
particular product he is buying. However, as discussed in Section 6.4, buyers
who are receiving a rebate are instructed to not use any coupons offered by
Amazon. A reviewer R33 noted that he was to buy a robot vacuum cleaner, but
the agent asked him not to select the $70 savings coupon that was offered.
Therefore, when a buyer does not elect to apply the coupons displayed on the
product page, it may indicate that the subsequent review is incentivized.

Avoiding Gift Card Balance. If there is any existing gift card balance in an
account, Amazon automatically applies that to any purchase you make. In order
to not use that balance, the user needs to select the payment methods menu,
and manually uncheck the box corresponding to the gift card balance. If a
buyer deliberately avoids using any existing points and gift card balance on
his account, it may indicate that the purchase and review is incentivized.

Unreasonable Choice. Most of the buyers we interviewed reported that
oftentimes the product they had to find was not displayed on the first page,
and they had to scroll till they reached the fourth pr even fifth page. This,
they noted, was not something they did for their genuine purchases; they
almost always chose the products on the first page. In addition, buyers reveal
that they often had to buy items which had little to no reviews, as opposed to
competing products with ratings above $4$ and hundreds of reviews. Again, this
was not something they did for their genuine purchases; they chose the well-
rated products. Therefore, if a buyer carries out an excessive search into the
fourth or fifth page, or chooses a product with few reviews even when
competing products are available, it may indicate that the purchase is
incentivized.

Review and Return. As noted in Section 5.3, buyers may attempt to return the
product even after having a written a five-star review for it because they are
being cheated, or are trying to scam the agents themselves. Such activity can
be evidence of incentivized reviews. This can also manifest itself as a
downgrading of the review after a few days.

##  8 Conclusion

In this work, we have studied the underground market of fake Amazon reviews.
We interview several agents and buyers and discover that buyers are targeted
via customized advertisements and Facebook groups. Although they are aware
that these activities violate Amazon’s terms of service, incentives for both
buyers and agents are monetary. Buyers are instructed to simulate organic
search for a product, engage with content on Amazon, not use a gift card or
discount codes, and write a positive, five-star review after a week in order
to evade detection by Amazon. Our research shows the various evasion tactics
used, and we present a set of recommendations that can help increase the
fidelity of detections by Amazon.

## References

  * [1] Uttara M Ananthakrishnan, Beibei Li, and Michael D Smith.  A tangled web: Should online review portals display fraudulent reviews?  Information Systems Research, 2020.
  * [2] Zachary Crockett.  5-star phonies: Inside the fake amazon review complex, April 2019.  https://thehustle.co/amazon-fake-reviews [Online; accessed 04-September-2020].
  * [3] John Emont.  Amazon’s heavy recruitment of chinese sellers puts consumers at risk, November 2019.  https://www.wsj.com/articles/amazons-heavy-recruitment-of-chinese-sellers-puts-consumers-at-risk-11573489075 [Online; accessed 27-August-2020].
  * [4] Junpeng Guo, Xiaopan Wang, and Yi Wu.  Positive emotion bias: Role of emotional content from online customer reviews in purchase decisions.  Journal of Retailing and Consumer services, 52:101891, 2020.
  * [5] Sherry He, Brett Hollenbeck, and Davide Proserpio.  The market for fake reviews.  Available at SSRN, 2020.
  * [6] Dina Mayzlin, Yaniv Dover, and Judith Chevalier.  Promotional reviews: An empirical investigation of online review manipulation.  American Economic Review, 104(8):2421–55, 2014.
  * [7] Leticia Miranda.  Some amazon sellers are paying $10,000 a month to trick their way to the top, April 2019.  https://www.buzzfeednews.com/article/leticiamiranda/amazon-marketplace-sellers-black-hat-scams-search-rankings?bfsource=relatedmanual [Online; accessed 21-September-2020].
  * [8] Nicole Nguyen.  Inside amazon’s fake review economy, May 2018.  https://www.buzzfeednews.com/article/nicolenguyen/amazon-fake-review-problem [Online; accessed 04-September-2020].
  * [9] Nicole Nguyen.  Her amazon purchases are real. the reviews are fake, November 2019.  https://www.buzzfeednews.com/article/nicolenguyen/her-amazon-purchases-are-real-the-reviews-are-fake[Online; accessed 24-August-2020].
  * [10] Aaron Smith and Monica Anderson.  Online shopping and e-commerce.  Pew Research Center, 19, 2016.