# Performance testing of a novel short axis photomultiplier tube for the HUNT
project

Yijiang Peng  Zike Wang  Bo Gao  Yiyue Tang  Mingjun Chen  Kai Li  Ling Ren
Xiaohao You11footnotetext: Corresponding author. and Maoyuan Liu

###### Abstract

Photomultiplier tubes (PMTs) with large-area cathodes are increasingly being
used in cosmic-ray experiments to enhance detection efficiency. The optical
modules (OMs) of the High-Energy Underwater Neutrino Telescope (HUNT) have
employed a brand new N6205 20-inch microchannel plate photomultiplier tube
(MCP-PMT) developed by the North Night Vision Science & Technology (Nanjing)
Research Institute Co. Ltd. (NNVT). In order to make the 20-inch PMT fit into
the 23-inch diameter pressure-resistant glass sphere, NNVT improved the
internal structure of PMT and shortened the height of PMT by more than 10 cm.
The first batch of these PMTs has been delivered for preliminary research
work. This paper describes a specific PMT testing platform built for the first
batch of 15 MCP-PMTs, and some performance parameters of PMT, such as P/V
ratio, TTS and nonliniearity, are measured.The measurement results show that
the new PMT still has good performance and can meet the requirements of HUNT
project.

##  1 Introduction

Since Moisey Markov in 1960 proposed deploying large-volume detectors in deep
lakes or seas to detect Cherenkov radiation generated by high-energy neutrinos
[1], many large Cherenkov high-energy neutrino telescope experiments have
emerged internationally, such as DUMAND [2], AMANDA [3], ANTARES [4], IceCube
[5], and Baikal-GVD [6]. The High-Energy Underwater Neutrino Telescope (HUNT)
is a next-generation high-energy neutrino detector proposed by the Institute
of High Energy Physics (IHEP) of the Chinese Academy of Sciences. Its primary
goal is to observe high-energy neutrinos in association with the discovery of
a large number of ultra-high-energy gamma photons in the Milky Way by LHAASO
[7], thereby helping to solve the mystery of the origin of cosmic rays. The
HUNT project plans to deploy over 55,000 Optical Modules (OMs) in an array
within a volume of 30 $km^{3}$, with each OM housing a 20-inch PMT.

The PMT is the most commonly used photosensitive detector, and can provide the
function of detecting Cherenkov light for OMs of high-energy neutrino
telescopes. In order to ensure the highest possible photon detection
efficiency, the photosensitive device must have the largest possible
collection area and the highest possible optical sensitivity. In order to
achieve good angular resolution, it should have good time accuracy. Utilizing
a PMT with unprecedented large photocathode area for the detection of high-
energy neutrinos provides HUNT with high sensitivity, low dark counting rate,
and wide dynamic range, which has advantages, especially in large-area light-
signal detection and position resolution. The performance of a single PMT will
directly affect the detection capability of the entire array. Considering the
PMT manufacturing process and based on the physical requirements of HUNT, the
PMT needs to meet the performance indicators specified in table 1.

Each PMT will undergo at least one performance test after delivery, including
single photoelectron (SPE) charge spectrum (gain, SPE resolution, and peak-to-
valley ratio), transit time spread (TTS), dark noise rate (DNR), Q-T effect
curve, anode charge nonlinearity, and after pulse ratio. These test results
will be used not only to evaluate the quality of the PMT itself, but also to
provide data support for optimizing the HUNT detection array. This paper
introduces the basic information about N6205, describes the construction of
the testing platform, and discusses in detail the testing process and
calculated results for each feature.

Table 1: Performance requirements for 20-inch PMT in HUNT Parameter | Specification
---|---
Gain | 5 $\times$ 106
High voltage |  $<$2000 V, Mean $\pm$ 120 V
Peak-to-valley |  $>$2
QE |  $>$25%
Rise time |  $<$3 ns, typically 1.4 ns
TTS |  $<$7 ns
DNR |  $<$25 kHz, threshold 1/3 PE
Afterpulse ratio |  $<$2%, 0.5 $\mu$s - 10 $\mu$s, typically 1%
Anode charge non-linearity |  $<$1800 PEs (10%), mean $>$2000 PEs
Stability | variation gain $<$5% & DNR $<$10%, 3-12 h

##  2 N6205 MCP-PMT

The pressure vessel designed for HUNT is a spheroidal glass chamber with a
maximum diameter of 23 inches (585 mm). Therefore, based on NNVT’s previously
developed 20-inch PMT [8], IHEP and NNVT jointly designed a new 20-inch MCP-
PMT (N6205) with an axial height of 470 mm, allowing it to be installed in the
glass chamber. N6205 uses a dual alkali photocathode with high quantum
efficiency. The unique lotus shaped expansion type focusing electrode greatly
improves the efficiency of photoelectrons reaching the microchannel plate. PMT
uses a negative high-voltage circuit ,connected via a 50 $\Omega$ coaxial
cable simultaneously transmits high voltage and signals. It has passed
temperature aging tests. Under the gain condition of 5 $\times$ 106, the
working voltage is less than 2000 V.

The specific dimensions and appearance of N6250 are shown in figure 1. The
circuit design of the voltage divider used in N6205 is shown in figure 2; it
has a total resistance of 10 M$\Omega$.

Figure 1: Left: the appearance of N6205. Right: the specific dimensions in mm.
Figure 2: Circuit design diagram of the voltage divider.

##  3 Platform

The testing platform, as shown in figure 3, consists of three parts: darkroom,
light source, and DAQ system. This section provides a detailed introduction to
the three parts.

Figure 3: Schematic diagram of testing platform. The yellow line represents
the optical fiber, the red line represents the HV line, the gray line
represents the trigger signal, and the blue line represents the signal line.
Different types of blue lines represent different test proseses.

###  3.1 Dark room

In the laboratory, a separate room was set up as a darkroom, where two barrels
were installed for testing the PMTs. The walls of the darkroom and the outer
wall of each barrel were pasted with materials for shielding the magnetic
field, resulting in a residual magnetic field at the PMT cathode position of
less than 5 $\mu$T. The magnetic shielding material is black, and a black
cloth is pasted inside each barrel to eliminate the influence of stray light.
The size of a barrel is shown in figure 4. It has an inner diameter of 540 mm
and a height of 900 mm. Three 135° brackets were evenly installed on the inner
wall of each barrel at a height of 670 mm for PMT placement. For these
setting, the top of the PMT cathode is 500 mm above the base, and the PMT
equator is 670 mm above the base.

Figure 4: Dimensions of the barrel placed in the darkroom to support PMT
testing.

###  3.2 Light source

The testing system uses a continuous optical attenuator integrated with a
multi-channel signal generator. This system can not only output light pulse
signals, it can also allow continuous adjustment of light power by three
orders of magnitude through software control [9, 10].

The signal generator features 10 channels with SMA connectors, each offering
independent adjustment of pulse width and delay. The output signal has an
amplitude range of 0 to 6.6 V, and after calibration, the relative timing
jitter between channels is maintained below 50 ps. The triggering signal is
input to a 405 nm LED module to drive its emission; the calibrated emission
time is shown in figure 5. The optical signal is then input to a continuous
optical attenuator via a quartz optical fiber.

Figure 5: Left: The emission time of the 405-nm LED used for testing. Right:
The relationship between step size and light relative transmittance when the
motor rotates one revolution.

The core components of the continuous optical attenuator consist of a neutral-
density continuous attenuation plate and a deceleration stepper motor that
drives the rotating attenuation plate. The base material of the attenuation
plate is custom-made quartz glass with a pinhole diameter of 10 mm, an outer
diameter of 100 mm, and a thickness of 1.0 mm. A semi-transparent film with a
variable thickness is coated on the surface of the glass to achieve continuous
variation of light transmittance with angle. The attenuation plate is sealed
in a black aluminum dark chamber, with the pinhole fixed on the shaft of the
stepper motor. On both sides of the dark chamber, quartz lenses with SMA-905
interfaces are fixed; the optical axes of the two lenses coincide. After
connecting the lenses to the quartz op tical fiber, the relative light
transmittance is calibrated with a 0.1%-accuracy optical power meter under
room temperature conditions, and the relationship between the relative light
transmittance and the number of motor rotation steps is obtained. As shown in
figure 5, these data provide assistance for Q-T effect testing and anode
charge nonlinearity testing.

The light source device serves three principal functions: Channels 1 and 2 act
as trigger windows for QDC and TDC to generate signals; Channels 3-9 can all
serve as trigger signals for the LED module; and the optical attenuator
continuously attenuates the optical signal, enabling adjustable control of
light intensity.

The output light is split into two parts by a UV-transmitting quartz optical
fiber, and each part is input into two beam expanders. A fiber beam expander
is installed at the central position inside the barrel to expand the output
light, forming a spot with a diameter of approximately 450 mm on the cathode
surface of the PMT, covering over 80% of its area.

###  3.3 DAQ system

Figure 3 shows the setup method of the DAQ system. A divider distributes HV to
the anode pin of the PMT and receives pulse signals from the anode, which are
then transmitted to a custom-made selector switch. Each channel of the switch
has one input and three outputs, with the output signals connected to
different modules in the VME crate for data processing in various test
projects. The detailed procedure is explained in Section 4. All tests were
conducted using commercial instruments, with the various models and parameters
listed in Table 2. The processed data are transmitted to the control computer
for storage and analysis via a USB 3.0 communication cable.

Instrument | Model | Characteristics
---|---|---
Oscilloscopes | PICOSCOPE 6824E | 8 Channels, 500 MHz bandwidth
Power Supply | CAEN DT8033M | 8 Channels, 4 kV/3 mA (6 W)
QDC | CAEN V792N | 16 Channels, 100 fC/LSB
TDC | CAEN V775N | 16 Channels, 35-300 ps/LSB
TDC | CIQTEK TDC0110 | 2 Channels, 8 ps/LSB
Attenuator | CAEN V859 | 0-89 dB attenuator
LED | CAEN N841 | -1 to -255 mV (1 mV step)
Amplifier | CAEN N978 | 1-10 variable gain
Amplifier | CAEN N979 | 10 fixed gain
Table 2: Devices and specific performance used during testing (1 fC = 1
$\times$ 10-15, LSB means Least Significant Bit).

##  4 Test process

The entire testing process is divided into seven following stages: waveform
inspection, cooling, SPE charge spectrum, TTS, Q-T effect, anode charge
nonlinearity, and afterpulse ratio.

###  4.1 Waveform inspection

To inspect the pulse signal output by the PMT, the light intensity was
adjusted to a level of SPE, with a signal trigger frequency of 1 kHz. The PMT
signal was acquired using an oscilloscope, PICOSCOPE 6824E, with results shown
in figure 6. The blue waveform is the PMT signal. When checking the signal, it
was found that there was overshoot in the PMT signal. Since directly
connecting to the QDC would cause instrument damage, the baseline was biased
to -2 mV, keeping the PMT signal below 0 mV.

Figure 6: The waveform under SPE conditions after amplified tenfold, showing
an amplitude of 60 mV.

###  4.2 Cooling

When cooling tests are initially conducted, it is essential to ensure that the
tests are performed under SPE conditions. Therefore, before running the test
script, a debugging script is executed to determine the HV and trigger ratio
during testing. During debugging, the high voltage is set initially according
to the values provided in the NNVT factory test table, which calls for taking
a 500,000 trigger signal, and repeating four times. The output includes the
statistical average and error of the trigger ratio, gain, and charge. The high
voltage and other settings are adjusted to achieve trigger ratios and gains at
levels of 10% and 5×106, respectively, before running the formal test script.

The cooling test includes monitoring four parameters: trigger ratio, gain,
charge, and DNR. This serves as an evaluation data set and is conducted over a
period of at least 2 hours. The signal output from the PMT is routed through a
switch, with the switch set to the first gear. The combined signal is then
amplified tenfold using an N979 amplifier before being processed by the V792N.

1,000,000 trigger signals are collected to measure the trigger ratio, gain,
and charge. Subsequently, the LED is turned off, and the TDC is configured for
the lowest precision (300 ps/LSB). 5,000,000 trigger windows are measured to
determine statistically the signal count under fixed-width conditions. The
count is then divided by the effective time to obtain the DNR (with a trigger
threshold of 2 mV, 1/3 PE). The above process is repeated, with an execution
efficiency of about 140 s/time. In order to ensure that the PMT has sufficient
cooling time, based on the results of multiple tests, this needs to be
executed at least 50 times, which takes about 2 hours. The final monitoring
results are obtained and plotted in figure 7 (using PAB2308-9022 as an
example).

The instability during testing comes from the environment, leakage current,
residual gases in the PMT, etc. Therefore, consistent working conditions need
to be ensured during the testing process. The cooling tests are considered
complete when the trigger ratio variation is less than 2%, gain variation is
less than 3%, charge variation is less than 2%, and DNR variation is less than
10%.

Figure 7: A prolonged cooling test conducted in SPE state for the monitor
trigger ratio, gain, charge, and DNR. The mean and fluctuation of the first
three are given.

###  4.3 SPE charge spectrum

The SPE charge spectrum test includes measuring four quantities: gain, trigger
ratio, SPE resolution, and peak-to-valley ratio. These parameters are tested
together because they all need to be measured in the SPE state. The trigger
ratio reflects the SPE state of the test, and the SPE resolution and peak-to-
valley ratio together represent the PMT’s ability to distinguish single
photoelectrons. The switch gear remains in first gear, and the signal is
amplified 10 times after being input into N979, synchronously outputting two
signals. One signal is still input into V792N, QDC to process the signal.

The setup includes a 20-kHz signal trigger rate, with 2,000,000 trigger
signals collected. The V792N is used to record the charge, and from the charge
distribution, the trigger ratio, peak-to-valley ratio, and SPE resolution are
derived, as shown in figure 9. The SPE charge spectrum includes pedestal,
gain, SPE resolution, trigger ratio, and peak-to-valley ratio.

###  4.4 TTS

At the same switch gear as described in Section 4.3, one signal is output to
QDC, while the other signal is input to the leading-edge discriminator N841
and output to V775N. TDC processes the signal.

The transition time of the PMT is defined as the time difference between the
light reaching the PMT and the signal being received; it follows a Gaussian
distribution. The FWHM is defined as the transit time spread, also known as
TTS. Due to signal delays caused by signal lines and electronic systems, the
directly calculated time difference is commonly represented by relative
transit time (RTT). Additionally, the TDC is configured for precision (35
ps/LSB), and 2,000,000 trigger windows are measured to record the TTS. From
this distribution, the RTT and TTS are derived, as shown in figure 9. After
the PMT signal is processed by TDC, the TTS is obtained. The peak position,
which is the RTT, is marked in the graph, and the half width, which is the
TTS, is also indicated.

Figure 8: The SPE charge spectrum.

Figure 9: The SPE TTS.

###  4.5 Q-T effect

Variation in the signal amplitude from the PMT causes fluctuations in the
measurement time, leading to an inconsistency between charge and time, known
as the Q-T effect. Therefore, it is necessary to measure the Q-T effect of the
PMT and correct the experimental data to eliminate timing errors caused by the
leading-edge threshold. The Q-T effect test involves testing both charge and
time information. After completing the SPE charge spectrum and TTS test, the
command interface of the control computer will prompt to change the switch
position to the second setting. Upon completing this change, the signal is
input to the amplifier N978 for a fourfold amplification. The synchronized
output is split into two paths. One path enters the V792N, while the other
path enters the N841 and then outputs to the V775N. Both the QDC and TDC
process the signals.

A 2-kHz signal trigger rate is set, and 10,000 trigger signals are collected
for both QDC and TDC. The light transmission rate of the attenuator is varied,
covering a total of 30 points, and at each point, measurements of both time
and charge are made. The amount of charge and time for each transmittance
point are calculated, with a time error of $\text{FWHM}/{\sqrt{12}}$. The FWHM
is obtained by fitting the time distribution with a Gaussian, as shown in the
figure 10.

Figure 10: Q-T effect. The relationship between charge and time, with error
bars in red.

###  4.6 Anode charge nonlinearity

The anode charge nonlinearity test is primarily aimed at obtaining charge
information. Upon completion of the Q-T effect test, the command interface of
the control computer prompts to change the switch position to the third
setting. After completing the switch operation, the signal is input to the
attenuator V859, attenuating the signal by a factor of 10, and then input to
the V792N for processing.

At a certain light intensity, the PMT nonlinearity is $L_{att}$, and its
output linear charge is $Q_{linearity}$. Under this light intensity, the
charge $Q_{test}$ that is obtained is related to the linear charge
$Q_{linearity}$ as follows:

$Q_{test}=Q_{linearity}\times(1+L_{att})$ (4.1)

$Q_{linearity}=I_{LED}\times E_{\lambda}\times E_{C}\times G_{HV}\times\tau$
(4.2)

In the equation (4.2), $I_{LED}$ is the light intensity of the LED emission.
During testing, the LED emission state remains constant; hence $I_{LED}$ is
constant. $\tau$ is the relative light transmittance of the neutral-density
filter, which can be obtained from the figure 5. $E_{\lambda}$ is the quantum
efficiency for the given wavelength, and $E_{C}$ is the collection efficiency
of cathode photoelectrons. $G_{HV}$ is the operating gain of the PMT under the
test high voltage. During the test, the high voltage and the peak wavelength
of the light source remain constant; thus $E_{\lambda}$, $E_{C}$, and $G_{HV}$
are constants. This means that $Q_{i}$ is positively correlated only with
$\tau$.

Typically, at low light intensities, the output of the PMT is close to linear.
In that case, the PMT output charge is $Q_{0}$, and the relative transmittance
of the attenuator is called $\tau_{0}$. Based on the relative transmittance
and the motor steps, the linear charge quantity $Q_{i}$ at any relative
transmittance $\tau_{i}$ can be obtained. This can be expressed as

$\frac{Q_{i}}{\tau_{i}}=\frac{Q_{0}}{\tau_{0}}$ (4.3)

The nonlinearity $L_{att}$ at the relative light transmittance $\tau_{i}$ can
be obtained from equations (4.1) and (4.3):

$L_{att}=\left(\frac{Q_{i}\times\tau_{0}}{Q_{0}\times\tau_{i}}-1\right)\times
100\%$ (4.4)

Figure 11 shows the nonlinearity test results obtained using the light
attenuation method. The horizontal axis shows the number of photoelectrons nPE
calculated based on the actual output charge of the PMT, while the vertical
axis shows the nonlinearity calculated using equation (4.4). Considering that
the charge nonlinearity of the MCP-PMT depends on the signal frequency, three
different frequencies (1, 2 and 5 kHz) were used for each measurement to
evaluate the nonlinearity.

Figure 11: Nonlinear results measured using the optical attenuation method at
1, 2 and 5 kHz of light source.

###  4.7 Afterpulse ratio

The afterpulse ratio test measures the slow afterpulse components in the PMT
signal. After completing the anode nonlinearity test, the light intensity is
adjusted to the 30 PEs level. The PMT signal is then input into the N979 and
amplified tenfold. It is next input into the CIQTEK TDC0110. This TDC module
can adjust the threshold through PC driver software, with the threshold set to
1/3 PE. The plotting window is set to 10 $\mu$s, and the acquisition time is
set to 4 minutes. Clicking the start button to begins the test; the results
are shown in figure 12. The afterpulse ratio is calculated using equation
(4.5). The count from 0.5 $\mu$s to 10 $\mu$s after the main pulse is
determined statistically and called $N_{A}$. Under the same conditions, DNR
measured during the same time window when the light source is turned off is
$D_{R}$ $\times$ 9500 $\times$ 10-9. $N_{0}$ is the count of the main pulse.
The afterpulse ratio, given by equation (4.5) is 0.86%.

$\text{Ratio}=\frac{N_{A}-D_{R}\times 9500\times 10^{-9}}{30\times
N_{0}}\times 100\%$ (4.5)

Figure 12: Time distribution of the afterpulse.

##  5 Summary

We set up a set of PMT test platform and established the test process of PMT
performance. The test results of HV, peak-to-valley ratio, DNR, TTS, and SPE
resolution and Afterpulse ratio are summrized in figure 13. These parameters
will be used to evaluate the performance of PMTs for the HUNT experiment. All
PMTs operate at a gain of 5 $\times$ 106; the high voltages range vary from
1650 to 1800 V, peak-to-valley ratios all are larger than 2.0, and SPE
resolutions are smaller than 50%. After a certain cooling period, the DNRs
become below 25 kHz. And TTSs’ values are less than 7 ns. Afterpulse ratios of
the tested PMT are all below 0.1%. Therefore ,all above test results have met
the requirements.

Figure 13: The test results of HV, peak-to-valley ratio, DNR, TTS, and SPE
resolution and Afterpulse ratio.

This article mainly focuses on testing the performance of a new type of
20-inch PMT developed for the HUNT experiment, in preparation for the
prototype deployment plan in March 2024. Besides, the testing system
established in this work will provide the foundation for testing a larger
batch of PMTs for the HUNT experiment in the future. From the first batch of
PMTs delivered by NNVT, 15 PMTs with qualified performance have been selected
and are awaiting assembly into the OMs.

## Acknowledgments

This work is supported by the Science and Technology Department of Sichuan
Province (Grant No. 2021YFSY0030), and the IHEP funds (Grant No. E42982U8 &
E35452U2).

## References

  * [1] M. Markov and I. Zheleznykh, _On high energy neutrino physics in cosmic rays_ , _Nuclear Physics_ 27 (1961) 385.
  * [2] A. Roberts, _The birth of high-energy neutrino astronomy: A personal history of the dumand project_ , _Rev. Mod. Phys._ 64 (1992) 259.
  * [3] AMANDA collaboration, _The AMANDA neutrino telescope: Principle of operation and first results_ , _Astropart. Phys._ 13 (2000) 1 [astro-ph/9906203].
  * [4] ANTARES collaboration, _ANTARES: The first undersea neutrino telescope_ , _Nuclear Instruments and Methods in Physics Research A_ 656 (2011) 11 [1104.1607].
  * [5] IceCube collaboration, _The icecube neutrino observatory: instrumentation and online systems_ , _Journal of Instrumentation_ 12 (2017) P03012.
  * [6] Baikal-GVD collaboration, _The baikal underwater neutrino telescope: Design, performance, and first results_ , _Astroparticle Physics_ 7 (1997) 263.
  * [7] LHAASO collaboration, _Ultrahigh-energy photons up to 1.4 petaelectronvolts from 12 $\gamma$-ray Galactic sources_, _Nature_ 594 (2021) 33.
  * [8] Q. Wu et al., _Summary of the R &D of 20-inch MCP-PMTs for neutrino detection_, _Journal of Instrumentation_ 16 (2021) C11003.
  * [9] S. Yin, L. Liu, B. Gao, M. Chen, X. You, K. Li et al., _Study on nonlinearity measurement of photomultiplier tubes by continuous neutral optical attenuator_ , in _International Conference on Optoelectronic Information and Functional Materials (OIFM 2023)_ , vol. 12781, pp. 61–71, SPIE, 2023.
  * [10] X. Wang and S. Cui, _Measurement of pmt charge nonlinearity using the continuous light attenuation method_ , _Journal of Instrumentation_ 18 (2023) T12010.


# Unpacking Privacy Labels: A Measurement and Developer Perspective on
Google’s Data Safety Section

Rishabh Khandelwal Equal Contribution  University of Wisconsin – Madison
Asmit Nayak∗ University of Wisconsin – Madison  Paul Chung  University of
Wisconsin – Madison  Kassem Fawaz  University of Wisconsin – Madison

###### Abstract

Google has mandated developers to use Data Safety Sections (DSS) to increase
transparency in data collection and sharing practices. In this paper, we
present a comprehensive analysis of Google’s Data Safety Section (DSS) using
both quantitative and qualitative methods. We conduct the first large-scale
measurement study of DSS using apps from Android Play store (n=1.1M). We find
that there are internal inconsistencies within the reported practices. We also
find trends of both over and under-reporting practices in the DSSs. Next, we
conduct a longitudinal study of DSS to explore how the reported practices
evolve over time, and find that the developers are still adjusting their
practices. To contextualize these findings, we conduct a developer study,
uncovering the process that app developers undergo when working with DSS. We
highlight the challenges faced and strategies employed by developers for DSS
submission, and the factors contributing to changes in the DSS. Our research
contributes valuable insights into the complexities of implementing and
maintaining privacy labels, underlining the need for better resources, tools,
and guidelines to aid developers. This understanding is crucial as the
accuracy and reliability of privacy labels directly impact their
effectiveness.

##  1 Introduction

Privacy policies have traditionally served as the primary method for conveying
the privacy practices of a service to users. However, studies have
demonstrated that privacy policies are often ineffective, largely because
users neglect them due to their length and vagueness [6, 13]. Introduced by
Kelly et al. [18], privacy nutrition labels summarize the privacy practices of
websites in a nutrition label format, making them easier to understand. The
concept of privacy labels has gained traction in the tech industry, with
Google introducing Data Safety Sections (DSS) and Apple introducing Apple
Privacy Labels (APL) for all new and updated apps on their app stores.

Recently, researchers showed the utility of privacy labels for users, making
privacy practices more accessible [34]. However, the utility of the privacy
labels depends on the developers properly filling out the forms. These forms
should accurately reflect the developers’ intentions regarding the privacy
practices of their apps. Underreporting these practices can lead to inaccurate
privacy labels, confusing and potentially instilling a false sense of security
and privacy among users. Conversely, overreporting privacy practices can harm
the developer by deterring users who will perceive the apps as less secure and
private than they are.

Therefore, it is essential to investigate the developer’s experience with
reporting their practices via the privacy labels. Prior research has reported
the responsiveness of developers in implementing Apple Privacy Labels and
analyzed the data collection practices of apps according to these labels [2,
23]. Through small-scale studies, researchers have discovered that inaccurate
APLs can exist due to the developer’s knowledge gaps or resource limitations
[22].

However, a considerable gap remains in comprehensively understanding the
developer’s interaction with privacy labels, particularly related to Google’s
enforcement of data safety sections. This interaction includes determining
intended privacy practices, overcoming platform challenges, accurately
submitting privacy label forms, and updating them over time. Prior research
has not accounted for this wide-ranging perspective. In this paper, we address
this gap by focusing on Google’s Data Safety Section, answering three key
research questions:

RQ1: How do developers report their app privacy practices in Google’s Data
Safety Sections?

RQ2: How have these patterns of reporting practices evolved over the year
following the implementation of DSS?

RQ3: What are the driving factors behind the changes in Google’s DSS, the
challenges, and the behaviors of app developers?

To answer these questions, we conduct a large-scale analysis of app privacy
labels on the Google Play Store. We periodically scrape the Google Play Store
to collect metadata, including permissions and DSS forms, for over 2M apps
between June 2022 and May 2023. Subsequently, we engage with 3500 developers
of these apps to understand their decision-making process concerning privacy
practices, the completion of DSS forms, and their subsequent updates. Our
qualitative analysis of developer responses helps answer the third research
question by forming an analytical framework to model the developers’
experience with DSS.

Our responses to the research questions offer new insights into Google’s DSS:

  * •

As of May 2023, privacy labels are only present for 46.8% of apps on the
Google Play Store. Among those apps featuring DSS, we observe patterns of
underreporting privacy practices, overreporting practices, and submitting
inconsistent DSS forms.

  * •

Our longitudinal analysis unveils a dynamic landscape for DSS, suggesting that
developers are still refining their comprehension and implementation of DSS
requirements. About 40% of the apps updated their DSS at least once over the
past year, adding and removing high-level privacy practices and data
categories.

  * •

Our qualitative analysis of developer responses indicates that developers
confront challenges when aligning their intended practices with DSS forms.
These challenges often lead them to prioritize successfully submitting DSS
forms over accurately populating them.

Finally, we propose novel recommendations to enhance the developer experience
with DSS based on our findings. These recommendations include action points
for platforms and regulators, such as better educational resources,
multilingual support, form simplification, consistent feedback, improved
support for third-party libraries, and mechanisms to solicit developer
feedback. Lastly, we plan to release a large-scale dataset of the metadata for
1.14M Android apps spanning June 2022 to May 2023.

##  2 Background and Related Works

Figure 1: An example of the data safety section of an Android app.

Privacy Nutrition Labels. Originally introduced by Kelley et al. [18, 19],
privacy nutrition labels summarize the privacy practices of websites in a
nutrition label format, making them easier to understand. They later designed
the “Privacy Facts” display to allow the users to consider privacy while
installing apps [20]. More recently, researchers proposed an Internet of
Things (IoT) security and privacy label [9, 10] to surface privacy and
security information about IoT devices to the users. Researchers have also
studied the design and evaluation of privacy notices and labels [4, 18, 21,
19, 20, 7, 28, 26, 11, 8].

In 2020, Apple adopted the privacy nutrition labels for the app store and
mandated that app developers provide their apps’ privacy information in the
form of the Apple Privacy Label (APL). Later in 2022, Google required
developers to add a Data Safety Section (DSS) to the Google Play Store.
Examples of Google’s privacy labels are in Figure 1.

Google Data Safety Section

Figure 2: The hierarchy of the Google Data Safety Section, showcasing the
various layers and components.

The Data Safety Section (DSS) consists of four levels (Figure 2), where the
first is high level Privacy Practices. The second and third levels consist of
Data Categories and Data Types, and the fourth level consists of Purpose.

The first level includes three practices: Data Collection, which covers the
details about the data that is collected and its intended use; Data Sharing,
where the developers disclose what data is shared with third parties; and
Security Practices that covers the data practices related to user choice and
data security. Security Practices include three tags: Encrypted in Transit,
Data Deletion Option, and Review against Global Security Standards.

In the second level, Data Categories includes 14 categories such as App Info
and Performance and App Activity. Each Data Category can also have Data Types,
which provide fine-grained information about the data used by the app. For
example, App Activity includes App Interactions and Installed App, as shown in
Figure 2. The final level of the Data Safety Section consists of Purposes that
describe the reasons for collecting or sharing the data.

Google App Submission Review. Google reviews submitted apps to the Play Store
to ensure their compliance with its guidelines about design, content, and
style [32]. As part of these guidelines, Google requires apps to comply with
its data safety policies. In particular, all developers must specify the data
collected and shared by their app, including the data handled by third-party
libraries or SDKs [16].

To provide developers with better expectations towards the privacy criteria
during the app review process, Google launched the Checks service. It gives
developers the ability to verify that their apps comply to the data safety
policies before submitting the app for review [5]. The Data Monitoring feature
of Google Checks monitors multiple channels of data collection and sharing,
including with SDKs, via in-app permissions, and to external sites. The
service provides developers evidence, such as permissions and network traffic,
that the data safety form or the privacy policy is non-compliant.

Usability of Privacy Labels. Researchers have studied the usability of APLs
from both users’ [34] and developers’ [22] perspectives. From the developers’
perspective, Li et al. [22] interviewed 12 iOS developers and reported that
developers err by under-reporting and over-reporting data collection in
privacy labels. They further concluded that the label design is confusing for
the developers either due to known factors (lack of resources, improper
documentation) or unknown factors (preconceptions, knowledge gaps). Xiao et
al. [33] characterize non-compliance of Apple privacy labels by studying data
flow to label consistency of 5K iOS apps. They provide insights for improving
label design based on their characterization. Researchers also built and
evaluated a tool [12] that helps iOS developers generate privacy labels by
identifying data flows through code analysis.

Studies on Privacy Labels. The works most similar to ours perform longitudinal
measurement of privacy labels to understand the adoption and evolution of
Apple privacy labels over time [2, 22, 29]. In particular, Scoccia et al. [29]
conducted an empirical study of 17K apps to characterize how sensitive data is
collected and shared for iOS apps. They found that free apps collect more
sensitive data for tracking purposes. Li et al. [22] and Balash et al. [2]
collected weekly snapshots of Apple privacy labels and characterized the
privacy practices mentioned in privacy labels for 573k apps. Balash et al. [2]
also perform additional correlation analysis with app meta-data like user
rating, content rating, and app size.

Our Contributions. In this paper, we investigate the data safety sections of
the apps listed on the Google Play Store, observing their evolution over the
course of a year. Based on our measurement, we conduct a large-scale study by
interacting with more than 3,500 Android app developers, using their apps as
case studies. This analysis allows us to model the developers’ engagement with
the Data Safety Section ecosystem, gaining insight into their challenges, data
practices, and the factors influencing their decision-making.

Our results complement prior works by showing that developers on the Apple and
Google platforms share common problems when completing the DSS and APL.
However, we go beyond prior work by substantiating the underreporting,
overreporting, and inconsistencies in data practices; studying the process by
which developers interact with the data safety sections; identifying
strategies they use to bypass the review process; and revealing the factors
that result in changes to the data safety sections over time.

##  3 Google Data Safety Dataset

We curate the Google Data Safety Dataset by scraping the metadata and privacy
labels of the apps from Google Play. We contact the developers of some of
these apps to understand their process of deciding on privacy practices,
filling out the DSS form, and updating the DSS. We show an overview of the
analysis pipeline in Fig. 3.

Figure 3: Our data measurement and analysis pipeline consists of scraping the
DSS of Android apps to answer the first two research questions and interacting
with app developers to answer the third research question.

##### Dataset Collection

We took 10 snapshots of the Data Safety Sections for 2.46M apps present on the
play store between June 20, 2022, and May 31, 2023. We captured eight weekly
snapshots from June 20 to Aug 1 and three more snapshots on September 2022,
November 2022, and May 2023. We chose to gather data more frequently around
July 20, 2022, the date Google required app developers to complete the Data
Safety Section. This allowed us to observe the developers’ immediate response
to this requirement. The additional three snapshots enabled us to study the
developers’ long-term reactions. We refer to this dataset with 10 snapshots as
DSS Dataset.

We initiated data collection with the apk list provided by Androzoo [1]. This
daily-updated list consists of Android app ids from various sources, including
those from the Google Play store. We capture the metadata of each app,
including its data safety sections, using the app IDs and a customized version
of publicly available google play store scraper library google-play-scraper
[17]. Across all the snapshots, we observed a total of 2.72M unique apps and
2.17M common apps. For the latest snapshot (May 2023), we retrieved metadata
for 2.46M apps, which includes apps with very low download counts. To ensure
that our statistical analysis is not skewed by these apps, we filter out apps
that have fewer than 1000 downloads resulting in a total of 1.1M apps with
539k having privacy labels.

##### Dataset Statistics

Our dataset shows that developers have been slow to add privacy labels to
their apps, even after the hard deadlines have passed. Privacy labels are
present only for 46.8% of the apps on the Google play store (as of May 31
2023). We also break down the DSS adoption rate according to app metadata,
focusing on the apps’ number of downloads, age rating, and pricing.

Number of Downloads: We examine the relationship of the DSS adoption rate and
the number of app downloads. In particular, we categorize the apps based on
their download numbers on a logarithmic scale, adhering to Google Play Store’s
binning methodology: (1000+, 5000+, …, 5B+, 10B+). We observe a perfect
correlation between the adoption rate and the downloads. Starting at 38.96%
adoption rate of apps with 1000+ downloads, the adoption rate increases
monotonically reaching 100% for apps with 10B+ downloads. This observation
suggests that developers of highly downloaded apps tend to place more emphasis
on the DSS, potentially due to access to greater resources and concern over
public perception.

Age Rating: The Play Store categorizes apps based on age ratings, including
Everyone, Teen, Mature 17+, and Everyone 10+.111Google also has Adults 18+
rating, but we found less than 200 apps in this category and decided to filter
it out for this analysis. We observed 51% of apps with the Mature 17+ rating
have a Data Safety Section (DSS), while the fraction of apps with a DSS in the
other age ratings ranges from 45% (Everyone) to 54% (Everyone 10+).

App Pricing: We study the difference in practices based on whether the app is
available for free, free with in-app purchases, or paid. We find that 65% of
the paid apps, 59% of free with in-app purchases, and, for free apps, only 44%
have DSS. We revisit this observation in the developer study where we
highlight how developers report the privacy practices of Ad libraries in free
apps.

##### Ethical Considerations

We collected data only from publicly available web pages and APIs. While our
data collection scripts might load Google’s servers, we were careful to not
abuse these resources. In particular, we added back-off strategies in case of
errors and waited for sufficient time before retrying for the failed cases.

##  4 Data Practices in Privacy Labels [RQ1]

We analyze the May 2023 snapshot of the DSS dataset to understand how
developers report high-level privacy practices, data types, and purposes.
Figure 4 depicts the percentage of apps reporting high-level privacy practices
on DSS, with data encryption practices being the most reported. Figure 5 shows
how app developers report collection and sharing for the different data
categories. Developers report collecting Location and Personal Information at
a higher rate than other data categories, primarily for App functionality and
Analytics. They also report sharing Location and Device Ids more commonly for
Advertising or Marketing purposes. The heatmap of Fig. 13 (Appendix 9.2)
provides a detailed breakdown of the declared purposes for collected and
shared data types.

We also note that apps may declare sharing data without collecting it, as
evident from Fig. 5. This discrepancy arises from the definition of Data
Collection, which covers developers retrieving user data from the device using
the app [14]. Whereas Data Sharing denotes the cases when the data is
transferred from the device to a third party. Thus, developers can share data
without collecting it if their application employs third-party libraries that
directly send data to their servers.

We further analyze this snapshot to understand how developers interact with
the DSS forms. Our analysis reveals three key patterns: overreporting privacy
practices, underreporting privacy practices, and submitting inconsistent
practices.

Figure 4: The evolution of Data Safety Section over the 10 snapshots in our
dataset Figure 5: The distribution of Top-5 data categories for high-level
practices

##### Underreporting Practices:

Analyzing Figure 4, we note that only 36% reported sharing at least one data
type, suggesting that the majority of the apps on the play store do not report
sharing data. This figure contrasts the findings from prior work [24, 31],
which found that the majority of the apps use at least one third-party library
collecting sensitive information [24]. We confirm this observation by
analyzing the 15K most popular apps from our snapshot that report not sharing
data with third parties. Specifically, we download the apk file from androzoo
and use the LibRadar library [1, 25] to determine whether third-party
libraries are used. Given an apk file, LibRadar [25] identifies the third-
party libraries and tags them into categories like Advertisement, Analytics,
etc.

Analyzing the apks, we find that 42% of the analyzed subset of apps used at
least one third-party library for advertisement or analytics. This result
clearly indicates that developers are underreporting the sharing practices.
One possible explanation is that the privacy practices of third-party
libraries are often vague and developers find it hard to understand the
collection and sharing practices of third-party libraries. We find evidence
supporting this explanation in prior work [3, 22] and our developer study
(Section 6).

##### Overreporting Practices:

Analyzing the purposes for Data Type collection, we observe that many apps
report a large number of purposes when listing datatypes. We note that out of
the 7 possible purposes for collecting data, more than 3.5K apps list 6 or
more purposes for every data type they collect, which may indicate that app
developers list all purposes out of convenience. For example, Workplace from
Meta with over 15M+ downloads, lists the same 6 purposes for all the data they
collect, like access to Installed Apps, SMS or MMS, Music Files. We also note
that while 3.5K is small compared to the dataset, it still has the potential
to impact millions of users.

A possible explanation for this observation could be that app developers lack
the knowledge required to fill the DSS and choose to select all options.
Another possible reason is that they are unaware of the policies of the third-
party applications that they use, and take a cautious approach by
overreporting. Findings in prior work [22] and from our developer study
(Section 6) align with this observation.

##### Inconsistent Practices:

We observe the developers report practices that are inconsistent with other
declared practices or with the app permissions. For example, we find that 40%
of the apps state that they do not collect or share data, but encrypt the data
in transit. We delved deeper into this observation by cross-verifying security
practices with apps’ network permission requests. 59% of apps do not request
network permissions, yet state that they encrypt data in transit. It is not
clear why would apps need to encrypt transit data if they are not collecting
or sharing data. As Encryption in transit has important implications for
privacy, we mark this trend and examine it in detail in our developer study in
Section 6.

We also cross-verified the collected and shared data types from the DSS to the
app permissions. Several apps report collecting or sharing several data types
without even asking for the corresponding permissions. For example, 11.5% of
the apps report collecting or sharing precise location data without obtaining
location permissions. Another example is 23.7% of the apps report collecting
or sharing files and documents without the “Photos/Media/Files” permissions.

In Section 6, we identify some of the reasons why developers provide
inconsistent privacy labels. One reason is developers over-reporting their
practices, as indicated above. Other reasons include developers choosing
labels by mistake, misunderstanding the label definition, and not updating the
labels after updating app features.

##  5 Evolution Of Data Safety Section [RQ2]

(a) Data Type evolution in Collection

(b) Data Type evolution in Sharing

Figure 6: Change in Data Categories between the first and the final snapshot
for (a) Data Collection and (b) Data Sharing. The figure shows that at Data
Category level, the reported practices change a lot, indicating that
developers change the DSS frequently.

Next, we conduct a longitudinal analysis of the DSS dataset to understand how
the data practices disclosed by developers evolved over time. As described in
Section 3, our data collection spanned the timeframe before and after the hard
deadline set by Google for app developers to comply with the new DSS
requirements. This design allows us to understand not only the static state of
app data safety disclosures at a given time point but also their evolution as
developers navigated this significant policy change.

Looking at Fig. 4, we observe that during the period spanning June 20, 2022 to
May 31, 2023, the number of apps with DSS increased from 312K to 539K. The
largest per-day change, of around 9K, happened between June 28, 2022, and July
1st, 2022. Interestingly, we find that from our initial snapshot, 21% of apps
removed their DSS over the course of our data collection. Specifically, 67K
updated their play store page to remove the privacy label. For example, Sport
Prediction with 1M app download had a DSS as of August 1, 2022, but did not
have it by Nov 25, 2022.

##### Updates in DSS.

We analyze the DSS dataset to understand the frequency of updates in DSS by
comparing the first snapshot of the DSS for an app with subsequent snapshots.
Figure 11 in Section 9.1 shows the CDF of updates in DSS. We find that 40%
(n=283K) of the apps updated their DSS at least once, while 4% (n=27K) updated
it at least twice. Moreover, the frequency of updates is higher in the
September and November snapshots. For example, Adobe Acrobat Reader: Edit PDF,
with over 500M downloads, updated their DSS 3 times between Jun 20, 2022, and
May 31, 2023. In Section 6, we discuss potential reasons for DSS changes.
These reasons include app feature updates or the discovery of incorrect DSS.

##### Evolution of High-Level Practices.

We also analyze the evolution of high-level practices present in the DSS.
First, we observe a shift in data collection practices. 44K applications that
initially reported collecting data have updated their DSS to state that they
no longer collect data. Conversely, some apps (n=45K) that initially claimed
not to collect data have revised their DSS, admitting to data collection.

We also observe similar trends in data-sharing practices. Several apps (n=42K)
that initially reported sharing data with third parties later updated their
DSS, indicating an end to such sharing. On the other hand, some apps (n=37K)
that did not report data sharing in their initial DSS later added such
practices. As changes in collection and sharing practices have implications on
user privacy, we highlight these trends and analyze them in detail in our
developer study in Section 6.

We observed a steady increase in the number of apps reporting Encryption of
Data in Transit, an aspect crucial for data security. The count rose from 157K
on June 20, 2022, to 353K on May 31, 2023, demonstrating a steady progression
towards improved data encryption practices (Figure 4). Similarly, we observe
an increase in the number of apps providing Data Deletion Option where the
count rose from 99K to 227K. However, despite the overall increasing trend in
Data Deletion Option and Encryption in Transit practices, a closer examination
reveals that some apps are incorporating these practices, while others are
withdrawing them. This observation implies that developers are still adjusting
their Data Safety Sections.

The ongoing changes in reported data practices, even ten months past Google’s
deadline for DSS implementation, point to a dynamic landscape, suggesting that
developers are still refining their understanding and implementation of DSS
requirements. We investigate the challenges that developers face in more depth
in our developer study (Section 6.3).

##### Evolution of Collection and Sharing Practices.

Delving deeper into the evolution at the datatype level, we investigate three
Data Categories collected most frequently: Location, Personal Identifiers, and
App Activity and Performance. Personal Identifiers include Personal Info, and
Device Id and other identifiers. We show the change in these Data Categories
in Figure 6(a). We find that a significant number of apps initially reporting
collection of user location (n=21K) and personal information (n=18K) have
revised their DSS to indicate no longer collecting these datatypes. Similarly,
around 6.2K apps that were initially collecting app info and performance data
have updated their DSS, indicating a halt in this data collection practice.

Conversely, we also identified apps initially not collecting location data or
other specific information, have now started collecting these datatypes. This
shows the bi-directional nature of these changes. We find similar trends by
analyzing evolution of these data types for Data Sharing in Figure 6(b).

These findings highlight the evolving nature of data practices which can
impact users’ trust and privacy. If an app shifts from not collecting to
collecting certain datatypes, it may expose users to new privacy risks
especially as they will not be notified to this change. The findings can
indicate developers’ difficulties or confusion in accurately understanding
their apps’ data practices as we highlight in Section 6.

##### Trends in Over-Reporting of Data Practices

In Section 4, we observe that developers over-reported the purposes for data
collection. Analyzing the DSS dataset, we find a persistent pattern in over-
reporting over time, even after ten months. This suggests that a considerable
number of developers continue to perceive an environment of low risk or
consequence in over-reporting their data practices.

We uncover the potential reasons behind this trend in our developer study. We
find that Google’s current policy enforcement does not impose penalties for
overstating data collection practices, likely due to the limitations inherent
in compliance checks, that rely on static and dynamic code analysis.
Consequently, some developers may be inclined towards a risk-averse strategy,
choosing to over-report to prevent potential policy violations. Although the
number of apps following such practices is low (n=4K) in our set, the impact
on the wider privacy label ecosystem can be substantial as it could affect
users’ trust in privacy labels.

##  6 Developer Study [RQ3]

We conduct a study with Android app developers to understand their perspective
when engaging with Data Safety Sections. We identified interesting patterns
(described in Section 4 and Section 5) in apps’ Data Safety Sections and
contacted the app developers to inquire about the factors responsible for the
patterns, and the challenges that they face.

###  6.1 Methodology

We first describe the study design and analysis methods.

Study Design. We reached out to app developers through emails, probing their
experiences and asking specific questions about the privacy labels of their
apps. We crafted these questions carefully to stimulate responses. Thus, we
contact the developers once via email and analyze their responses to answer
our research questions. As such, our study achieves a broader understanding of
developer perspectives as compared to prior work [22].

When emailing developers, we clearly identified ourselves as researchers and
stated that we were studying their application and sought information related
to their data safety section (Section 9.3). Additionally, We refrained from
collecting any personal data from the developers, using only the contact
information available publicly on the Play Store to contact them. As such, the
study has been approved by the Institutional Review Board (IRB) at our
institute.

Developer Selection. Recall that in Section 4 and Section 5, we identified the
following three trends. (A) apps stating that they encrypt data without
collecting or sharing data, (B) apps changing their practice from not
collecting/sharing data to collecting/sharing data, and (C) apps changing
their practices from collecting/sharing data to not collecting/sharing data.
Trend (A) points to inconsistency in reported security practice, whereas Trend
(B) and (C) have implications of user data privacy. We identified the apps
corresponding to these patterns, sorted them in order of real installs and
contacted the top 10,000 developers for each pattern. In response to our
emails, we received 3500 responses. To get a clean set for analysis, we filter
out automated replies and non-english responses, leaving us with 889
responses. We then qualitatively analyze these responses to explore how
developers describe their privacy practices, the challenges they face while
working with the privacy labels and the factors that prompted them to update
their data safety section.

Qualitative Analysis Method. In accordance with qualitative research
guidelines, we employed a strategy of random sampling and coding of developer
responses until data saturation was achieved [27]. In our analysis, we reached
saturation after coding 225 responses. The coding process began with two
authors evaluating an initial set of 50 responses and developing preliminary
codes. Next, the research team discussed these codes, clarified differences,
and established the initial codebook. The notable differences were in the
granularity of the codes and we opted for fine-grained codes as they can be
consolidated later. Subsequently, the two authors independently coded a
randomly selected subset of 25 responses each time, comparing their codes and
iterating on the codebook until they achieved high inter-rater reliability
($\kappa=.87)$) by the 125th response. Using the refined codebook, the two
authors continued to code independently until the 225th response, marking the
point of data saturation. After conducting a thematic analysis, we model the
developers’ engagement with the Data Safety Section ecosystem through an
analytical framework as depicted in Fig. 7. Our codebook is available
online.222https://osf.io/92sez/?view_only=0b4aa040161a4b259c0a32c7fb3ae82c

Figure 7: Our analytical framework with the high-level themes, secondary-level
themes, and developer quotes. The framework describes the lifecycle of
updating the DSS as perceived by the developers. We do not show all secondary-
level themes due to a lack of space.

Findings Overview. Figure 7 describes the lifecycle of updating the DSS as
perceived by the developers. App developers go through several stages while
working with Data Safety Section. First, they deduce their intended privacy
practices (Section 6.2) based on their app implementation. While attempting to
fill the DSS form, they face challenges (Section 6.3) mapping their intended
practices to the form. The challenges result in strategies (Section 6.4) to
initially fill out the form and get it accepted. After submitting the initial
form, there might be inaccuracies highlighted to them either via Google or
through their internal review system. Finally, they update the form (Section
6.5). In the subsequent sections, we present the findings of our qualitative
analysis, unpacking the experiences and behaviors of app developers as they go
through this process of uploading their Data Safety Sections.

###  6.2 Intended Privacy Practices of Developers

We first discuss developers’ intended data collection and sharing practices.
Our thematic analysis reveals three subthemes within the privacy practices of
the apps: third-party data collection, encryption, and first-party practices.

Third-party Data Collection: We find that developers often reported involving
third parties for various purposes. The main purposes were ads and analytics.
We also find here that developers tend to use these libraries primarily for ad
revenue. For example, one developer expressed: “… to make profit from my DJ
app, I use third party advertising, such as Google AdMob and AppLovin. They
(third party ads) are collecting users data to showing perfect ads to my DJ
users.” This is consistent with our findings from Section 4 where we find that
Ads and Analytics are the top two purposes for data sharing.

Figure 8: The distribution of high-level data practices based on price. We
note that the fraction of paid apps that collect and share data is lower than
free apps.

However, we observe developers do not tend to share data when they employ in-
app purchases. For example, when asked about changes in their DSS, one
developer noted: “In the last update, I removed all the ads of the app, so I
receive money from the few premium subscriptions. The Google AdMob SDK
collected all that data, so now it doesn’t get any type of information from
the user.” This suggests developers of paid apps tend not to share data
possibly as they do not rely on ads for revenue. In Section 3, we observed
that the fraction of apps with DSS in paid apps is higher than in free apps.
Analyzing the Data Sharing practices of paid apps (Figure 8), we indeed find
that a lower number of paid apps report sharing data than free apps.

Encryption: We find that developers often secured data during transmission.
One developer reported–“we currently utilize the IPsec protocol to ensure
secure transmission of data. IPsec is a widely adopted industry standard for
VPNs and provides robust encryption, authentication, and integrity protection
for transmitted data.” Some developers also assumed that third-party libraries
will provide encryption, although this might raise questions about their
direct control over data security. For example, one developer mentioned, “… I
use Google API like Drive, YouTube, AdMob and Firebase. They do collect user
data I guess and they state that data is encrypted”.

First-party Practices: In our analysis, developers also conveyed their own
data practices. Many developers reported not intending to collect (n=60%) or
share data (n=48%) themselves. As one developer stated, “… we do not collect
any personal data or share any personal data. Our app only collects anonymous
analytics events (button clicks and screen impressions/load) and app crashes
via Google’s Firebase framework so that we can fix user issues and crashes.”
Others asserted not integrating any ads or analytics services, while some
stressed the importance of obtaining user consent before collecting data,
underlining their commitment to respecting user privacy.

###  6.3 Challenges Faced by Developers

Next, we uncover the challenges that the developers face while mapping their
intended privacy practices to DSS forms. These challenges are categorized into
seven sub-themes with Figure 9 depicting the frequency of each sub-theme.

Figure 9: The frequencies of seven subthemes of developer’s challenges
regarding the Google Data Safety Section, as explained in Section 6.3.

Lack of Information About Third-Party Libraries: Developers indicated
uncertainty about the privacy practices of the third-party libraries used in
their apps. One developer noted: “I currently face a challenge in that I am
unsure of what specific APIs are included in the app code provided by
Appsgeyser.com that may be contributing to this issue of non-compliance, as
said by Google. Additionally, I have been unable to find a reliable source of
information on this matter.” Notably, in the play console, Google shows the
permissions required by popular libraries, as well as flags problematic SDKs,
however, there is no help regarding the collection practices of these
libraries.

Inconsistency: Some developers reported inconsistency in the acceptance of
their data safety form. A developer struggling with the review process noted:
“… we submit the same form over and over again and often times Google rejects
our answers with no or at most a vague explanation. Eventually, Google accepts
it.” Such inconsistency can lead to confusion about what is required, and
result in frequent revisions, and inaccurate DSS.

Lower English Proficiency: Non-native English speakers cited difficulties
understanding the data safety policies. For example, a developer, whose native
language is not English, was confused with an English word: “… our
administrator whose first language isn’t English did not seem to understand
the meaning of ’ephemerally’ and ticked ’No, this collected data is not
processed ephemerally’. So even though we declared the data collected/shared,
Google play did not disclose this on the app’s store listing.”

Complicated: Developers also complained that options provided in DSS form are
complicated. For instance, a developer made mistakes in the data safety form
because there were simply too many options, whereas another was confused with
the complicated description of the DSS option. This complexity can lead to
errors or misunderstandings when filling out the form.

Unclear Process: Our analysis also revealed that developers are confused
regarding Google’s review process for DSS, finding the process unclear and the
explanations vague. For instance, a developer noted “Google at one time
prompted me that my app collects data of which we knew nothing about, the data
they spoke about is the ’Devices and Other IDs Data’ and that it’s a
compulsory data been collected by most apps on Play Store even if they do not
collect user data. We don’t see or collect any user data from my users and our
app doesn’t not even use firebase, one signal or ads.”

Lack of Expertise: Developers also acknowledged difficulty arising from a lack
of expertise in both app development and data safety policies. For example,
one developer noted–“… due to my limited experience, I made this app with the
help of third party app developer-AppsGeyser. I am not sure of how I should
represent the third-party APIs in the app code and I have been unable to find
a reliable source of information on this matter..”. Another developer noted
that they do not understand half of the DSS policies and so they just keep
submitting answers in hopes of finding the right configuration. This gap in
knowledge can happens due to vague definitions in data safety policies.
Moreover, the lack of information can lead to developers having an incomplete
understanding of the privacy practices, resulting in inaccurate representation
of these practices in the DSS.

Changing Rules: Developers reported confusion stemming from changing DSS
policies within the Google Play Store. They mentioned that the constant
evolution of these rules made it difficult for them to ensure compliance. As
one developer said: ”… so we haven’t changed anything in ¡app_name¿ in almost
a decade. What has changed, and seems to keep changing every few months is
Googleś privacy policies. They are difficult to understand and they shift like
sand …”. This indicates that developers have difficulty keeping up with the
dynamic landscape of rules and regulations regarding DSS.

These challenges highlight the need for more developer support in areas such
as policy comprehension, form simplification, and the handling of third-party
libraries. Addressing these issues could significantly aid developers in
enhancing data safety and privacy in their applications.

###  6.4 Developer Behaviors and Strategies in DSS Submission

We now discuss the various strategies adopted by the developers to navigate
the challenges they face while submitting the DSS form. These behaviors and
strategies are classified into five sub-themes. Figure 10 shows the frequency
of each sub-theme in the coded responses.

Figure 10: The frequencies of five subthemes of developer’s behaviors
regarding the Google Data Safety Section, as explained in Section 6.4.

Alternative Option: When developers were unable to find the optimal data
safety card option for their app, they resorted to choosing an alternative
one. For instance, as said by one developer–“… we do not collect any data,
everything user write on our app will be stored on their device. There wasn’t
any other option to select except ’data is in encrypted in transit …” . This
highlights the potential need for more comprehensive or flexible options
within the form.

Change in Developer Behavior: Our analysis also captured developers altering
their behavior based on their experience while interacting with DSS. For
instance, one developer that stopped the development of their app noted: “…
considering all the hassle I had been going through, it has been determined
that it would not be practical to continue updating these apps on the Google
Play Store, given the minimal profit generated and the effort required to
maintain compliance.” This highlights the scenario where the complex policies
of DSS can potentially discourage developers to update their apps, hurting the
users as they might be forced to work with outdated versions of the apps.

Future Policy: Some developers acknowledged that they intentionally had an
incorrect DSS to increase flexibility, and in case of future app updates. For
example, one developer responded: “It is true that we do not share or collect
any kind of personal data. It means that if we intend to do so in the future
then we will be encrypting the data. At the moment we are not sharing or
collecting personal data.” This might convey incorrect information to the
users. Conversely, if the developer does not indicate that they collect data,
but utilize the same flexibility, it could pose a risk to users.

Overstating Collection: Some developers indicated that Google only checks for
the data that is collected but not mentioned as collected but not the other
way. As a result, they select all options for collection in the DSS form. For
example, one developer noted –“… it is safe to declare all possible data
collected in Data safety section so, in our apps, even the information (e.g.
phone numbers) that is not collected is mentioned as collected.” We also
observed similar over-reporting of practices in Section 4 and Section 5. This
observation highlights an issue with Google’s compliance analysis tools.

Form That Works: Some developers reported using a data safety form that had
previously passed Google’s review, regardless of whether it fully aligned with
the current data handling practices of the app. For example, one developer
noted –“… my app did not collect any data. but in the data safety section, I
wrote that I collect data because other than that Google would not publish my
app.” This approach suggests a pragmatic yet concerning response by the
developer as it may result in inaccurate DSS.

The findings highlight that developers attempt to circumvent the review
process either by over-reporting their privacy practices or by trying random
configurations until their DSS is accepted. Combined with the challenges
(Section 6.3) that the developers face, this indicates that the DSS submission
tool might not be usable by the developers.

###  6.5 Inaccurate Labels and Driving Factors behind Changes in Data Safety
Section

As the developers navigate the complex challenges, they may submit inaccurate
labels. We used the developers’ intended practices from Section 6.2 and
compared them with the reported practices in DSS. Additionally, our
qualitative analysis revealed several key factors that influence developers’
decisions to update their applications’ Data Safety Sections (DSSs).

####  6.5.1 Inaccurate Labels and Type of Inaccuracies

We conducted a comparative analysis between the reported practices (Data
Collection and Data Sharing) in DSS and their intended practices from the
emails. We find a notable discrepancy; 41% of developers who stated in their
emails not to engage in data collection were nonetheless reported as
collecting data in their DSS. A similar inconsistency (42%) was observed for
data sharing. For instance, the developer of an app with over 100M downloads
stated in an email, “In Google Play Console we pointed that we collect/share
data because of Ad network SDKs integrated into the game. Not the app itself,
but ad SDKs do collecting or sharing.”, but their DSS did not report any data
collection or sharing. This discrepancy highlights the fact that the DSS
sections might not be thoroughly monitored by Google and reiterates the need
for more effective mechanisms for accurate reporting of privacy practices.
Next, our qualitative analysis also reveals factors responsible for
inaccuracies.

Misconception: Some developers demonstrated misunderstandings of the data
safety policies, which resulted in inaccurate DSS. Such developers have the
misconception that they do not have to disclose practices for third-party
libraries even when they are using third-party SDKs. For example, one
developer noted –“… we do not collect any personal data or share any personal
data. Our app only collects anonymous analytics info through Google Firebase.
Hence, we chose that we do not collect data.”

Recall the interesting trend that we observed in Section 4 where we noted that
many apps state they do not collect or share data, but encrypt data in
transit. We find that this could be happening due to misconceptions about
using third-party libraries. For example, one developer noted – “I use ADMOB
for banner ads at the bottom of my app, but Google collects Device ADID. I
checked that data is encrypted for that part.” The developers report
Encryption, but due to the misconception, do not report Data Shared.

Simple Mistake: Some Developers also acknowledged that they made a mistake and
conveyed their plans to update their DSS during the communication with the
authors. For example, one developer mentioned –“previously, the process of
filling out the Data safety section was interrupted and uncompleted due to one
of the checkboxes left unchecked.”

Residual Label: Some developers failed to update their DSS after making
feature updates. The developers conveyed their plans to update their forms,
but this trend can potentially be detrimental to the DSS ecosystem, especially
if the update adds Data Collection or Data Sharing flows.

####  6.5.2 Driving Factors for Change in DSS

After submitting the initial forms, developers may need to update their DSS.
We now discuss the various factors responsible for the change in DSS. Our
analysis revealed six distinct sub-themes for these factors.

Feature Update. One significant trigger for updating DSSs was the introduction
of new features. Most commonly, developers updated their DSSs due to the
addition of advertisement or analytics SDKs into their app. For example, one
developer notes: ”… In the initial version there were no ads. But now I have
put ads in the app for which I had to change my policy as well.”

Law Update. Changes in legal and regulatory frameworks also compelled
developers to modify their DSSs. For example, one developer noted – “Our
methods and infrastructures didn’t change. This change was made in light of
new GDPR guidelines that are in effect right now.”

Policy Update. Changes imposed by the platform (like the Play Store), required
adjustments to the DSS. For instance, one developer using a third-party ad
service mentioned that: ”We don’t collect or transmit any user data. However,
some of our apps use Google’s AdMob for ads. As per new directions from
Google, we are required to reproduce Google data policies related to AdMob
even if our app does not collect any data.”

Forced Update: We also observed instances where developers were compelled to
update their DSS to avoid having their apps removed from the Play Store. For
example, one developer noted – “We detected user data transmitted off device
that you have not disclosed in your app’s Data safety form as user data
collected. If your app continues to be non-compliant after August 22, 2022,
your app updates will be rejected and your app may face additional enforcement
actions in the future.” This showcases the influence of platform policies on
developers’ decision-making processes.

Internal Review. The results of internal reviews also surfaced as a factor for
DSS updates. Specifically, a developer mentions that the updated DSS
guidelines provided by Google enabled them to more accurately portray how they
handle data. They also emphasize that: ”… the added features on the stores
rightfully illuminated some areas that were unknown to users, which made a lot
of apps appear to handle their data differently than they actually are.”

Incorrect Label. Finally, developers noted that the discovery of incorrect
labels results in DSS updates. As noted by a developer–”… Although I still do
not collect any data, I was essentially required by Google some time ago to
say that I collect data on behalf of the advertising services in order to
remain compliant with Google’s policy …”–most of the developers fixed the DSSs
after they realized that they chose the incorrect data safety card option due
to their confusion after being formally notified by Google.

This analysis underscores the complex and multifaceted nature of developers’
decision-making processes around updating DSSs, influenced by internal
practices, external regulations, and the dynamics of app development.

##  7 Discussion and Recommendations

Our research has highlighted the complex landscape of developers’ experiences
and challenges with Data Safety Sections. We follow on our findings by
presenting a set of recommendations aimed at improving the developer
experience with DSS. Then, we discuss the roles of the platforms and
regulators in addressing the deficiencies in DSS. Finally, we list the
limitations of our methods.

Recommendations
---
1\. Enhance Educational Resources: Educational resources should be enhanced to
provide developers with real-world examples, interactive tutorials, and
guidelines.
2\. Provide Multilingual Support: Resources should be available to developers
in multiple languages
3\. Simplify Data Safety Forms: Data safety forms should be simplified to
improve comprehension.
4\. Consistent Feedback from the Review Process: Developers should be given
clear and transparent feedback on the approval or rejection of their data
safety forms.
5\. Improve Support for Third-Party Library Data Practices: Platform providers
and third-party library developers should ensure transparency about their data
practices.
6\. Regular Consultations and Feedback Mechanisms: Consult with developers to
ensure continued relevance of privacy labels; allows developers to share their
feedback.
Table 1: Summary of the recommendations.

###  7.1 Recommendations

We provide a list of recommendations to improve the developer experience with
the DSS section in Google Play. These recommendations, summarized in Table 1,
go beyond those presented in prior work [22].

Enhance Educational Resources. An important need emerged for enhancing
educational resources surrounding data safety policies and the filling of data
safety forms. Such educational resources include detailed explanations about
data collection, storage, and sharing practices that are expected to be
reported in the Data Safety Section. Providing developers with real-world
examples, interactive tutorials, and guidelines to navigate data safety
requirements can go a long way in reducing misunderstandings and ensuring
proper compliance.

Provide Multilingual Support. The global landscape of app development calls
for the need to support diverse languages. Developers across the globe should
be able to access, understand, and interpret guidelines without language
acting as a barrier. Thus, providing guidelines, forms, and supporting
services in multiple languages is crucial. Platform providers could consider
deploying multilingual support teams and translation services to cater to this
diverse community.

Simplify Data Safety Forms. Several developers have reported difficulties in
understanding the complexity of data safety forms (Section 6.3, leading to
inaccuracies in reporting. Addressing this concern would involve simplifying
the forms to ensure they are easy to understand and complete. This could be
achieved by using accessible language, clear terminologies, and unambiguous
options. Additionally, redesigning the form layout to enhance readability and
ease of use might also prove beneficial.

Consistent Feedback from the Review Process. In Section 6.3, we find that
developers have little confidence in the approval process. As a result,
developers resort to filling the form to just receive the approval, without
the form actually being accurate. Developers expressed frustration as to
understanding why a particular form is accepted or rejected. Developers need
to have confidence in a consistent and transparent review process. The
approval or rejection of data safety forms should be communicated clearly with
an explanation of the reasons. This will not only help developers improve
their subsequent submissions but also reduce confusion and frustration.

Improve Support for Third-Party Library Data Practices. The increasing use of
third-party libraries poses challenges for developers to accurately represent
data practices in their apps. To mitigate this challenge, platform providers
should demand transparency from third-party providers about their data
practices. Additionally, developing tools or mechanisms that help developers
to track and represent these practices in their data safety forms would be
beneficial. An existing effort from Google is the Google Play SDK Index, which
aims to provide developers transparent information on all the third-party SDKs
that is usable in Android app development [15]. While the index informs the
users of the Android OS permissions these services are requesting, it still
fails to provide information on the specific category of data being collected.

Regular Consultations With Developers and Feedback Mechanisms. To ensure the
continued relevance and effectiveness of privacy labels, regular consultations
with developers should be conducted. This would provide a platform for
developers to share their experiences, voice their concerns, and give feedback
on existing processes. Such feedback mechanisms could provide invaluable
insights for platform providers to understand evolving challenges and adapt
their policies and support systems accordingly.

###  7.2 Discussion

Impact of Industry Intervention. Google’s recent introduction of the Checks
service [5], providesing paid compliance analysis to developers, adds to the
dynamics of the privacy labels. From the developers’ perspective, popular apps
with resources might find Checks to be an invaluable tool, simplifying the
task of policy compliance and mitigating the likelihood of errors or
misunderstandings. However, the service’s cost may create hurdles for smaller
or independent developers, possibly leading to disparity between reported
practices by developers with more substantial means and those operating under
more constrained circumstances.

As for the platform, the Checks service could fulfill several objectives for
the Play Store. It might elevate the quality of reported practices and
compliance of apps. However, Google could face criticism for monetizing a
critical aspect of the app development process, particularly if this act is
seen as establishing a ’pay-to-play’ mechanism.

The implications for users, albeit less direct, are still significant. If the
Checks service contributes to higher policy compliance and reduced errors by
developers, users could get accurate data safety practices. Conversely, if the
cost of the service results in a less diverse app marketplace due to financial
barriers for smaller developers, users may face a reduction in their choices
of apps.

Regulators Even though our analysis finds inconsistencies between privacy
labels and privacy practices, evidence [34] suggests that privacy labels have
can carry specific information about the practices. This information can be
very useful for privacy concerned users who want to ensure that they are only
using privacy respecting apps.

However, as shown in this work, the accuracy of privacy labels is not
guaranteed. While developers are required to disclose their data practices in
order to obtain a privacy label, there is no guarantee that the information
they provide is accurate or complete. Therefore, it is necessary to have
systems in place to verify the accuracy of privacy labels and to hold
developers accountable for any discrepancies. This is particularly important
because the false labels can create a false sense of security among the users.

One potential model for regulating privacy labels is a system similar to the
one used for food nutrition labels, which are regulated by the Food and Drug
Administration (FDA). A regulatory body could be established to oversee
privacy labels and ensure that they are accurate and consistent. This could
help to build trust among users and encourage developers to be more
transparent about their data practices.

Another solution could be providing the monitoring system that is used during
the app submission reviews directly to the user’s device. The system could
perform real-time dynamic analysis on the app installed on the user’s device
and show the analysis results, such as network traffic and the data or sensor
access logs. This way, the users are given a more detailed view of the privacy
practices of the app and would be able to make a better-informed decision on
whether to use the app. An example in production includes the Apple App
Privacy Report introduced in iOS and iPadOS 15.2, which provides data and
sensor access logs from apps installed on the device, and network traffic from
the apps directly to the users [30].

###  7.3 Limitations

Data Collection Our data collection process was carried out over specific
periods, and thus may not fully capture the dynamic nature of the app
ecosystem. Apps can update their DSS at any time, and changes outside of our
data collection windows would not be reflected in our analysis. The apps
included in our study are a subset of all available apps on the Google Play
Store. Although we made an effort to include popular apps and span various
categories, our sample may not be fully representative of the entire app
ecosystem. Finally, we noted discrepancies in data practices based on the
presence of third-party libraries. However, some instances of third-party
library usage might have been missed due to limitations in the tools used for
detection, leading to potential underestimation of their prevalence and
impact.

Developer Study Our developer study relies on email communication with app
developers. While this allowed us to reach a larger number of developers, it
may have biased our results toward those who were more willing to respond,
potentially neglecting the perspectives of developers who did not reply.

The nature of self-reported data could also pose challenges to the reliability
and accuracy of the information collected. There is the potential for bias in
the developers’ responses, as they may present information in a way that
portrays their apps more favorably. Another limitation of our study is the
lack of demographic data about the developers who responded to our email. As
we prioritized respecting privacy and ensuring anonymity, we did not collect
any personally identifiable information, including demographic details such as
age, gender, nationality, or years of experience in app development. Such
demographic data could potentially provide meaningful context and allow for a
more nuanced understanding of developers’ perspectives. For instance, a
developer’s experience level or geographical location might influence their
understanding of privacy issues or their familiarity with privacy regulations.
Despite the limitations outlined, our research sheds light on the challenges
that the developers face, and it fills a gap in the literature about the
factors affecting developers’ making while working with the privacy labels.

##  8 Conclusion

In conclusion, our study takes a comprehensive approach to examine the
landscape of Data Safety Sections (DSS) in Google Play Store apps. Through a
large-scale analysis, we highlighted inconsistencies in reported practices, as
well as instances of both underreporting and overreporting. Our longitudinal
study emphasized the dynamic nature of DSS implementation. We observed the
persistence of overreporting trends and revealed how developers are still
adjusting to the requirements, even ten months after Google’s imposed
deadline. We investigate the developers’ perspective by performing a
qualitative study by communicating with over 3500 developers. Our analysis
uncovers the process developers undertake when navigating the DSS landscape.
We outlined the myriad challenges developers face, the strategies they employ
to comply with DSS policies, and the factors prompting changes in their DSS.
Finally, based on the challenges and developer behaviors, we provide
recommendations aimed at improving the developer experience with Data Safety
Sections.

## References

  * [1] K. Allix, T. F. Bissyandé, J. Klein, and Y. Le Traon, “Androzoo: Collecting millions of android apps for the research community,” in _Proceedings of the 13th International Conference on Mining Software Repositories_ , ser. MSR ’16. New York, NY, USA: ACM, 2016, pp. 468–471. [Online]. Available: http://doi.acm.org/10.1145/2901739.2903508
  * [2] D. G. Balash, M. M. Ali, X. Wu, C. Kanich, and A. J. Aviv, “Longitudinal analysis of privacy labels in the apple app store,” _arXiv preprint arXiv:2206.02658_ , 2022.
  * [3] R. Balebako, A. Marsh, J. Lin, J. I. Hong, and L. F. Cranor, “The privacy and security behaviors of smartphone app developers,” _Citeseer_ , 2014.
  * [4] R. Balebako, F. Schaub, I. Adjerid, A. Acquisti, and L. Cranor, “The impact of timing on the salience of smartphone app privacy notices,” in _Proceedings of the 5th Annual ACM CCS Workshop on Security and Privacy in Smartphones and Mobile Devices_ , 2015, pp. 63–74.
  * [5] N. Castelly and F. Hurley, “Introducing Checks: simplifying privacy for app developers - Google: The Keyword,” https://blog.google/technology/area-120/checks/, Feb 2022, date accessed: 2023-06-01.
  * [6] F. H. Cate, “The limits of notice and choice,” _IEEE Security & Privacy_, vol. 8, no. 2, pp. 59–62, 2010.
  * [7] L. F. Cranor, “Necessary but not sufficient: Standardized mechanisms for privacy notice and choice,” _J. on Telecomm. & High Tech. L._, vol. 10, p. 273, 2012.
  * [8] ——, “Mobile-app privacy nutrition labels missing key ingredients for success,” _Communications of the ACM_ , vol. 65, no. 11, pp. 26–28, 2022\.
  * [9] P. Emami-Naeini, Y. Agarwal, L. F. Cranor, and H. Hibshi, “Ask the experts: What should be on an iot privacy and security label?” in _2020 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2020, pp. 447–464.
  * [10] P. Emami-Naeini, J. Dheenadhayalan, Y. Agarwal, and L. F. Cranor, “Which privacy and security attributes most impact consumers’ risk perception and willingness to purchase iot devices?” in _2021 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2021, pp. 519–536.
  * [11] G. Fox, C. Tonge, T. Lynn, and J. Mooney, “Communicating compliance: developing a GDPR privacy label,” in _24th Americas Conference on Information Systems_ , 2018.
  * [12] J. Gardner, Y. Feng, K. Reiman, Z. Lin, A. Jain, and N. Sadeh, “Helping mobile application developers create accurate privacy labels,” in _2022 IEEE European Symposium on Security and Privacy Workshops (EuroS &PW)_. IEEE, 2022, pp. 212–230.
  * [13] J. Gluck, F. Schaub, A. Friedman, H. Habib, N. Sadeh, L. F. Cranor, and Y. Agarwal, “How short is too short? implications of length and framing on the effectiveness of privacy notices,” in _Twelfth symposium on usable privacy and security (SOUPS 2016)_ , 2016, pp. 321–340.
  * [14] G. P. Help, “Understand app privacy & security practices with Google Play’s Data safety section,” https://support.google.com/googleplay/answer/11416267?sjid=17006176392115416702-NA, 2022, date accessed: 2023-06-01.
  * [15] G. P. C. Help, “Make informed choices with Google Play SDK Index,” https://support.google.com/googleplay/android-developer/answer/12034434?hl=en, 2022, date accessed: 2023-06-01.
  * [16] G. P. C. Help, “Provide information for Google Play’s data safety section,” https://support.google.com/googleplay/android-developer/answer/10787469?hl=en, Mar 2023, date accessed: 2023-06-01.
  * [17] JoMingyu, “google-play-scraper: Google play scraper for python,” https://github.com/JoMingyu/google-play-scraper, 2022, date accessed: 2023-06-01.
  * [18] P. G. Kelley, J. Bresee, L. F. Cranor, and R. W. Reeder, “A ”nutrition label” for privacy,” in _Proceedings of the 5th Symposium on Usable Privacy and Security_ , ser. SOUPS ’09. New York, NY, USA: Association for Computing Machinery, 2009. [Online]. Available: https://doi.org/10.1145/1572532.1572538
  * [19] P. G. Kelley, L. Cesca, J. Bresee, and L. F. Cranor, “Standardizing privacy notices: An online study of the nutrition label approach,” in _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , ser. CHI ’10. New York, NY, USA: Association for Computing Machinery, 2010, pp. 1573––1582. [Online]. Available: https://doi.org/10.1145/1753326.1753561
  * [20] P. G. Kelley, L. F. Cranor, and N. Sadeh, “Privacy as part of the app decision-making process,” in _Proceedings of the SIGCHI conference on human factors in computing systems_ , 2013, pp. 3393–3402.
  * [21] ——, “Privacy as part of the app decision-making process,” in _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , ser. CHI ’13. New York, NY, USA: Association for Computing Machinery, 2013, p. 3393–3402. [Online]. Available: https://doi.org/10.1145/2470654.2466466
  * [22] T. Li, K. Reiman, Y. Agarwal, L. F. Cranor, and J. I. Hong, “Understanding challenges for developers to create accurate privacy nutrition labels,” in _CHI Conference on Human Factors in Computing Systems_ , 2022, pp. 1–24.
  * [23] Y. Li, D. Chen, T. Li, Y. Agarwal, L. F. Cranor, and J. I. Hong, “Understanding ios privacy nutrition labels: An exploratory large-scale analysis of app store data,” in _CHI Conference on Human Factors in Computing Systems Extended Abstracts_ , 2022, pp. 1–7.
  * [24] J. Lin, “Understanding and capturing people’s mobile app privacy preferences,” Ph.D. dissertation, Carnegie Mellon University, 2013.
  * [25] Z. Ma, H. Wang, Y. Guo, and X. Chen, “Libradar: Fast and accurate detection of third-party libraries in android apps,” in _2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C)_ , 2016, pp. 653–656.
  * [26] A. M. McDonald, R. W. Reeder, P. G. Kelley, and L. F. Cranor, “A comparative study of online privacy policies and formats,” in _International Symposium on Privacy Enhancing Technologies Symposium_. Springer, 2009, pp. 37–55.
  * [27] B. Saunders, J. Sim, T. Kingstone, S. Baker, J. Waterfield, B. Bartlam, H. Burroughs, and C. Jinks, “Saturation in qualitative research: Exploring its conceptualization and operationalization,” _Quality & Quantity_, vol. 52, no. 4, p. 1893–1907, Sep 2017.
  * [28] F. Schaub, R. Balebako, A. L. Durity, and L. F. Cranor, “A design space for effective privacy notices,” in _Eleventh symposium on usable privacy and security (SOUPS 2015)_ , 2015, pp. 1–17.
  * [29] G. L. Scoccia, M. Autili, G. Stilo, and P. Inverardi, “An empirical study of privacy labels on the apple ios mobile app store,” in _9th IEEE/ACM International Conference on Mobile Software Engineering and Systems 2022_ , 2022\.
  * [30] A. Support, “About app privacy report,” https://support.apple.com/en-us/HT212958, Jul 2022, date accessed: 2023-06-01.
  * [31] H. Wang, Y. Guo, Z. Ma, and X. Chen, “Wukong: A scalable and accurate two-phase approach to android app clone detection,” in _Proceedings of the 2015 International Symposium on Software Testing and Analysis_ , 2015, pp. 71–82.
  * [32] G. Workspace, “About app review,” https://developers.google.com/workspace/marketplace/about-app-review#areas, May 2023, date accessed: 2023-06-01.
  * [33] Y. Xiao, Z. Li, Y. Qin, J. Guan, X. Bai, X. Liao, and L. Xing, “Lalaine: Measuring and characterizing non-compliance of apple privacy labels at scale,” _arXiv preprint arXiv:2206.06274_ , 2022.
  * [34] S. Zhang, Y. Feng, Y. Yao, L. F. Cranor, and N. Sadeh, “How usable are ios app privacy labels?” _UMBC Faculty Collection_ , 2022.

##  9 Appendix

###  9.1 Changes in DSS

Figure 11: The Cumulative Distribution Function for the frequency of DSS
updates made by apps throughout the snapshot timeline.

The above figure shows the frequency with which apps change their DSS between
the period of June 20, 2022, and May 31, 2023. While the majority of apps did
not have change labels, a significant number, 283K (40%) changed labels at
least 1 time. Moreover, there were over 1.3K apps that changed their DSS at
least 3 times.

###  9.2 Data Practices in Privacy Labels

In this section, we look into the distribution of the high-level practices of
apps based on their price, as shown in Fig. 8. Next, we look at the heatmap in
Fig. 13 to see the distribution of datatypes across purposes. The heatmap is
normalized by the total number of apps collecting or sharing a given datatype.

Figure 12: The distribution of high-level data practices based on price.
Figure 13: Heat map showing the distribution of data types across purposes.

###  9.3 Developer Study

For the Developer Study (Section 6) we sent emails to developers in 3
different categories: (A) apps stating that they encrypt data without
collecting or sharing data, (B) apps changing their practice from not
collecting/sharing data to collecting/sharing data, and (C) apps changing
their practices from collecting/sharing data to not collecting/sharing data.

For category (A) we used the following template:

  1. We hope this email finds you well. We are researchers at ¡LAB_NAME¿ and have been using your app, ¡APP_NAME¿, in our recent studies. We have noticed that in the data safety section of your app, it states that you encrypt data. However, we have also noticed that your app does not collect or share data.

We are reaching out to ask if you could clarify this for us. We are trying to
better understand the data safety section implemented in your app. We
appreciate any information you can provide.

Thank you for your time and we look forward to your response.