# Performance of an ultra-pure NaI(Tl) detector produced by an indigenously-
developed purification method and crystal growth for the COSINE-200 experiment

H. Lee 1,2,B.J. Park 1,2,J.J. Choi 2,3, O. Gileva 2, C. Ha 4, A. Iltis 5, E.J.
Jeon 2,1, D.Y. Kim 2, K.W. Kim 2, S.H. Kim 2, S.K. Kim 3, Y.D. Kim 2,1, Y.J.
Ko 2, C.H. Lee 2, H.S. Lee 2,1, I.S. Lee 2,∗, M.H. Lee 2,1, S.J. Ra 2, J.K.
Son 2, K.A. Shin 2

###### Abstract

The COSINE-100 experiment has been operating with 106 kg of low-background
NaI(Tl) detectors to test the results from the DAMA/LIBRA experiment, which
claims to have observed dark matter. However, since the background of the
NaI(Tl) crystals used in the COSINE-100 experiment is 2–3 times higher than
that in the DAMA detectors, no conclusion regarding the claimed observation
from the DAMA/LIBRA experiment could be reached. Therefore, we plan to upgrade
the current COSINE-100 experiment to the next phase, COSINE-200, by using
ultra-low background NaI(Tl) detectors. The basic principle was already proved
with the commercially available Astro-grade NaI powder from Sigma-Aldrich
company. However, we have developed a mass production process of ultra-pure
NaI powder at the Center for Underground Physics (CUP) of the Institute for
Basic Science (IBS), Korea, using the direct purification of the raw NaI
powder. We plan to produce more than 1,000 kg of ultra-pure powder for the
COSINE-200 experiment. With our crystal grower installed at CUP, we have
successfully grown a low-background crystal using our purification technique
for the NaI powder. We have assembled a low-background NaI(Tl) detector. In
this article, we report the performance of this ultra-pure NaI(Tl) crystal
detector produced at IBS, Korea.

##  1 Keywords:

NaI(Tl) crystal; Dark matter; COSINE-200; Low-background detector;
Purification

##  2 Introduction

Numerous astronomical observations support the theory that most of the matter
in universe is the invisible dark matter, although an understanding of its
nature and interactions remains elusive (1, 2, 3). Even though tremendous
efforts have been made to search for dark matter, no definitive signals have
been observed (4, 5). The only exception is the DAMA experiment, which has
observed an annual modulation of event rates using an array of NaI(Tl)
detectors (6, 7). This observation could be interpreted as dark matter-nuclei
interactions (8, 9). However, this result has been the subject of a continuing
debate because no other experimental searches have observed similar signals
(5, 10).

Several experimental efforts using the same NaI(Tl) target materials are
currently underway (11, 12, 13, 14, 15, 16, 17). The COSINE-100 experiment is
one such effort presently operating at the Yangyang underground laboratory in
Korea, which has provided several exciting physics results (9, 18, 19, 20,
21). However, due to the approximately 2–3 times higher background level, an
unambiguous conclusion regarding the observation in the DAMA experiment using
the same annual modulation signal has not been observed yet (22, 23).

As an effort to upgrade the ongoing COSINE-100 experiment for the next-phase
COSINE-200 experiment, we have conducted an R&D program aimed at producing a
low-background NaI(Tl) detector to conclude on the observed signals from
DAMA/LIBRA unambiguously. It includes the chemical purification of the raw NaI
powder (24, 25), its crystal growth (26), and detector assembly (27). We have
already proved the principle of a low-background NaI(Tl) detector using the
commercially available low-background Astro-grade NaI powder from Sigma-
Aldrich (28). As a next step, we have grown an NaI(Tl) crystal using our own
NaO power produced using the mass purification process at IBS, Korea (29).
This article reports the characteristics and performance of this indigenously-
produced NaI(Tl) crystal.

##  3 NaI purification and Crystal Growth

The COSINE-200 detector requires extremely low levels of radioactive
contamination in the materials used in the detector production. The major
contributors to the background are the decays of 40K and 210Pb in the bulk
NaI(Tl) crystal (30, 31). Because of the similarity in its chemical properties
to those of Na, which is in the same periodic table group, K is the primary
impurity contaminant, and its selective extraction from NaI powder is
challenging. We found that the fractional recrystallization method effectively
reduces the K and Pb impurities (24). In addition, using this method, the Ba
concentration was significantly reduced, indicating a reduction of Ra
impurities (24). Thus,we constructed a mass production facility at IBS,
Daejeon, Korea, for producing ultra-pure NaI powder using the fractional
recrystallization method on-site (25). The facility has been operated with a
maximum production rate of 35 kg of ultra-pure powder in a single processing
cycle of two weeks (29). Using our purification facility, we have performed
mass purification of the fractional recrystallization process using raw NaI
powder from Merck (99.99(5)$\%$ purity). In this mass purification process of
the NaI power, we have achieved a concentration of K of 6.4 ppb and that of Pb
below 0.3 ppb (29), which are consistent with contamination levels of the
Astro-grade powder.

The ultra-pure crystal was grown using a small-volume Kyropouls grower (32),
which is the same grower used for growing the proof of principle low-
background NaI(Tl) crystals using the commercial Astro-grade powder (28). In
growing the crystal, 1.7 kg of the purified NaI powder was loaded in a 12 cm
diameter, 10 cm high quartz crucible. An NaI(Tl) crystal ingot, as shown in
Figure 1()(a), of $\sim$70 mm diameter and $\sim$80 mm high and having a 1.1
kg mass, was grown in $\sim$24 h . During the crystal growth, N2 gas was
continuously flushed using a thallium trap with a flow rate of 10 L/m.

(a) NaI(Tl) Crystal ingot

(b) Cut and polished NaI(Tl) crystal

() Bare NaI(Tl) (NaI-037) crystal

##  4 Experimental setup

###  4.1 NaI(Tl) crystal

The growth of the NaI(Tl) crystal (named NaI-037) was completed on January 18,
2021, using NaI powder purified IBS (24, 32). The top and bottom sections of
the crystal ingot were cut using a diamond bandsaw, as shown in Figure 1()(b).
After cutting the top and bottom, the NaI-037 crystal is 70 mm in diameter and
51 mm in height. The flat top and bottom surfaces and a barrel-shaped side
surface were polished using aluminum oxide films ranging from 400 to 8000
grit. After polishing, the barrel was wrapped with a polytetrafluoroethylene
(PTFE) film in several layers as a diffusive reflector. A 3 mm thick copper
casing with quartz windows at each end was encapsulated the crystal
hermetically. Hamamatsu 3 inch photomultiplier tubes (PMTs), selected for high
quantum efficiency (R12669SEL), were coupled via an optical interface to each
end of the crystal. The entire assembly was performed in a glovebox, where the
humidity was maintained to be less than 10 ppm (H2O) using Ar gas and a
molecular sieve trap. Before the assembly, all parts were cleaned using
diluted Citranox liquid with sonication and baked in a vacuum oven for more
than 12 h. After assembly, the detector was delivered to the Yangyang
underground laboratory (Y2L), which has $\sim$700 m of rock overburden (33).
From the crystal growth to Y2L delivery, it took less than three weeks and
minimized the cosmogenic activation in the crystal.

###  4.2 Shielding structure

The background contamination levels of the NaI-037 crystal were evaluated
using, the same experimental apparatus as that used for the NaI(Tl) crystal
R&D at the Y2L (28, 34). It includes an array of 12 CsI(Tl) crystals
surrounded by 10 cm copper, 5 cm polyethylene, 15 cm lead, and 30 cm liquid
scintillator-loaded mineral oil (35, 36) as a radiation shield. The detector
was installed inside the CsI(Tl) array, as shown in Figure 1.

Figure 1: A schematic view of the Y2L setup. The NaI-037 crystal (red circle)
was installed inside the CsI(Tl) crystal array (blue squares).

###  4.3 Electronics

The PMTs attached to the NaI-037 crystal had two readouts each, a high-gain
signal from the anode and a low-gain signal from the fifth-stage dynode. The
anode signal was amplified by a factor of 30, whereas the dynode signal was
amplified by a factor of 100 using a custom-made preamplifier. The amplified
signals were digitized by 500 MHz, 12-bit flash analog-to-digital converters
(FADCs). Triggers from the individual channels were generated by the field-
programmable gate arrays (FPGAs) embedded in the FADC. The final trigger was
generated in the trigger and clock board (TCB) when an anode signal
corresponding to one or more photoelectrons (PEs) occurred in each PMT within
a 200 ns time window. The anode and dynode signals were recorded whenever the
anode signal produced a trigger.

Signals from the CsI(Tl) crystals were amplified by a factor of 10 and
digitized in a charge-sensitive 62.5 MHz FADC (SADC). The SADC provided the
integrated charge and the time of the signal. An integration time of 2048 ns
was used to record the CsI(Tl) signals considering their decay time. The SADC
channels did not generate triggers.

If the trigger condition was satisfied, the TCB sent trigger signals to the
FADC and SADC to store the signals from the NaI(Tl) and the CsI(Tl) crystals.
The FADC stored an 8 $\mu$s long waveform starting approximately 2.4 $\mu$s
before the time of the trigger position. The SADC stored the maximum
integrated charge within an 8 $\mu$s search window. This system is similar to
the one used in the COSINE-100 data acquisition (37).

Figure 2: Anode energy distribution obtained using a 241Am source.

###  4.4 Energy calibration and light yields

The energy calibration of the anode signal was done using a 59.54 keV X-ray
emitted from 241Am. Figure 2 shows the anode energy spectrum. A clear peak at
59.54 keV resulting from the 241Am source is shown together with the 127I
X-ray escape peak around 30 keV. The dynode signal was calibrated using the
photopeaks corresponding to 214Bi(609 keV) and 40K(1460 keV) contaminants in
the crystal.

The charge distribution of the single photoelectron (SPE) was obtained by
identifying the isolated clusters at the decay tail of the 59.54 keV X-ray
signal from the 241Am source (3-5 $\mu$s after the signal start). The light
yield was determined from the ratio of the total deposited charge and the mean
of the SPE charge for the 59.54 keV X-ray data. In this crystal, a light yield
of 17.8$\pm$0.6 number of photoelectron (NPE)/keV was obtained. It is similar
to the result for the NaI-036 crystal, which has the highest light yield among
the previously developed low-background NaI(Tl) crystals using the Astro-grade
powder (28). This light yield is also larger than those of the detectors used
in the COSINE-100 and DAMA/LIBRA experiments, as summarized in table 1.

##  5 Understanding the background in the spectrum

###  5.1 40K background

40K is one of the most problematic background sources in the search for weakly
interacting massive particles (WIMP) using NaI(Tl) crystals. The X-rays/Auger
electrons from 40K decays produce 3.2 keV energy signals, similar to the
energy signals expected for a WIMP-nuclei interaction (30, 31, 38). The 40K
decays also emit 1460 keV $\gamma$ rays, which can escape from the NaI(Tl)
crystal and hit the surrounding CsI(Tl) crystals, leading to a double
coincidence with the 3.2 keV X-rays.

Figure 3 shows the tagged low-energy spectra from the NaI(Tl) crystal by
requiring the detection of the 1460 keV $\gamma$ ray in the CsI(Tl) crystals.
The 40K background level in the NaI(Tl) crystal was determined by comparing
the measured coincidence rate from a GEANT4-based simulated data, as described
in Ref. (39). By accumulating more than six months of data, the K level was
measured to be 8.3$\pm$4.6 ppb, which was compared with the other NaI(Tl)
crystals listed in Table 1. It is well below our goal of 20 ppb, consistent
with the results from the DAMA/LIBRA crystals (34, 40) and previously
developed NaI-035 and NaI-036 crystals with the Astro-grade powder.

Figure 3: Energy deposition of the 3.2 keV 40K coincidence events in the
NaI-037 crystal. The model of the energy spectrum assumes a combination of a
Gaussian 40K signal and a constant background. Figure 4: Scatter plot of the
meantime versus the energy distribution events measured over 7.8 days for the
NaI-037 crystal. The $\alpha$ events (red dots) and the $\gamma$/$\beta$
events (black dots) are separated clearly.

###  5.2 $\alpha$ analysis

Alpha-induced events inside the NaI(Tl) crystals can be identified using the
fast decay times of their corresponding signals. The charge-weighted duration
time, called the meantime, is defined as

$\displaystyle\langle t\rangle=\frac{\Sigma_{i}A_{i}t_{i}}{\Sigma_{i}A_{i}},$
(1)

where A and t are the charge and time of the i-th digitized bin of a signal
waveform, respectively. The meantime is estimated within 1500 ns from the
pulse starting timing. Figure 4 shows a scatter plot of the energy versus the
meantime for the NaI-037 crystal. In the figure, the populations of
$\gamma$/$\beta$ and $\alpha$ events can be separated clearly due to the
faster decay times of the $\alpha$-induced events.

###  5.3 210Pb background

In the NaI(Tl) crystal experiments, the dominant background source in the low-
energy signal region is from 210Pb (31, 41, 42). The 210Pb activity can be
estimated from the alpha-decay studies, because the $\alpha$ decays of 210Po
originate from the $\beta$ decays of 210Pb. Due to the decay time of 200 days
of 210Po, the amount of 210Pb produced can be estimated using a time-dependent
fit of the alpha rate as follows:

$\displaystyle N=N_{Pb210}\left(1-e^{-(t-t_{0})/\tau_{Po210}}\right)+C,$ (2)

where $N$ is the total alpha rate, $N_{Pb210}$ is the amount of 210Pb at the
equilibrium state, $t_{0}$ is the time difference between 210Pb contamination
and the start time of data taking, $\tau_{Po210}$ is the mean lifetime of
210Po, approximately 200 days, and $C$ represents the long-lived components
from 238U and 232Th chains. Figure 5 shows the measured total alpha rates in
the NaI-037 crystal over the detector running time. The 210Pb activity in the
crystal was estimated to be 0.38$\pm$0.10 mBq/kg, which is lower than the
COSINE-100 crystals and is consistent with the activity in the NaI-036 crystal
produced using the Astro-grade powder. However, this activity is slightly
higher than the DAMA crystals and another crystal NaI-036 grown with the same
Astro-grade powder. The 0.4 mBq/kg level contamination of 210Pb is enough to
reach 1 count/kg/keV/day background level, similar to the activity in the
DAMA/LIBRA detectors, as described in Ref. (28).

Figure 5: The total alpha rate in the NaI-037 crystal as a function of time,
modeled with 210Po assuming contamination of 222Rn (and/or 210Pb). The
asymptotic line corresponds to the rate of total alpha events in the
equilibrium state.

###  5.4 232Th background

Contaminants from the 228Th subchain in the 232Th family can be estimated by
deploying the time-delayed $\alpha$–$\alpha$ coincident events of 220Rn and
216Po. The alpha decay of 216Po has a half-life of 0.145 s following its
production via alpha decay of 220Rn. Owing to the short half-life of 216Po, it
is straightforward to select two successive $\alpha$ particles with almost no
random coincident events.

The presence of the coincident events is shown in figure 6()(a) as the
distribution of the time gap between those two $\alpha$ events. The
exponential component indicates the contamination from 232Th, corresponding to
below 0.39 ppt (90$\%$ confidence level). The 232Th concentration in the
NaI-037 crystal is the lowest among the other NaI(Tl) crystals, as summarized
in table 1.

###  5.5 238U background

238U is one of the common radioisotopes because of its long half-life . The
238U content in the background can be studied using the time-delayed
$\beta$–$\alpha$ coincident events, similar to the calculation of the 232Th
background. This method exploits the $\alpha$ decay of 214Po with a half-life
of 164.3 $\mu$s, while 214Bi, the parent particle of 214Po, undergoes $\beta$
decay. Due to the 50 $\mu$s dead time of the trigger system, the coincident
events with delay times greater than 50 $\mu$s can be tagged. The results are
shown in figure 6(). The 238U activity of NaI-037 was 1.02$\pm$0.58 ppt,
similar to that observed for the other NaI(Tl) crystals, as given in Table 1.

subfigure0  (a) Time differen ofbetween two $\alpha$ decays of the 220Rn–216Po
decay chain. (b) Time difference between the 214Po $\alpha$ decay and 214Bi
$\beta$ decay.

() Time difference distributions of data (black dots) and the exponential fits to them (red-solid line). Table 1: Measured radioactive contaminants in the NaI-037 crystal, C6 of COSINE-100 (30), DAMA crystals (40, 43), and the previously grown NaI-035 and NaI-036 crystals using the Astro-grade powder (28).The upper limits are given at a 90$\%$ confidence level.  Crystal | Mass (kg) | LY (NPE/keV) |  40K (ppb) |  210Pb (mBq/kg) |  232Th (ppt) |  238U (ppt)
---|---|---|---|---|---|---
NaI-037 | 0.71 | 17.8$\pm$0.6 | 8.3$\pm$4.6 | 0.44$\pm$0.09 | 0.2$\pm$0.3 | 1.0$\pm$0.6
NaI-035 | 0.61 | 11.8$\pm$1.8 |  $<$42 | 0.01$\pm$0.02 | 1.7$\pm$0.5 | 0.9$\pm$0.3
NaI-036 | 0.78 | 17.1$\pm$0.5 |  $<$53 | 0.42$\pm$0.27 |  $<$4.9 | 36.5$\pm$3.9
COSINE-100 | 12.5 | 14.6$\pm$1.5 | 16.8$\pm$2.5 | 1.87$\pm$0.09 | 0.7$\pm$0.2 |  $<$0.02
DAMA | 9.7 | 5–10 |  $<$20 | 0.01–0.03 | 0.5–7.5 | 0.7–10

###  5.6 External Background

Because of the small size of the NaI-037 crystal and no liquid scintillator
active veto, a significantly higher background contribution is expected from
the external background compared to those found in the COSINE-100 crystals
(30, 41). The PMTs attached to the NaI(Tl) and the CsI(Tl) crystals are the
primary sources of external background. In this study, the external background
contributions were simulated using the GEANT4-based simulation toolkit used
for the COSINE-100 background modeling (30, 41).

###  5.7 Cosmogenic radionuclides

The cosmogenic production of radioactive isotopes in the NaI(Tl) crystal is
mainly due to long-lived nuclides such as 3H and 22Na (30, 44). The NaI-037
crystal was grown in Daejeon, Korea (70 m in altitude) and delivered
underground within a month. Based on the previous study, one-month exposure
time near sea level can produce 0.004 mBq/kg of 3H and 0.05 mBq/kg of 22Na
(44), respectively.

figure9  (a) Single-hit low-energy (2–60 keV) (b) Single-hit high-energy
(60–3000 keV) (c) Multiple-hit low-energy (2–60 keV) (d) Multiple-hit high-
energy (60–3000 keV)

() Measured single-hit and multiple-hit background spectra of the NaI-037
(black point) crystal fitted with the different simulated background
components using a simultaneous fit of four channels using the log-likelihood
method. The external component (purple-hatched area) is the dominant
contributor.

##  6 Background modeling

For a quantitative understanding of the background in the NaI-037 crystal,
GEANT4-based simulation, developed for the background modeling of the
COSINE-100 NaI(Tl) crystals (30, 41) and also used in the previously grown
crystals using the Astro-grade powder (28), was performed. The input values of
the contamination levels are obtained from Table 1. A simultaneous fit was
done to the single-hit low energy (3–60 keV), single-hit high energy (60 keV–3
MeV), multiple-hit low energy, and multiple-hit high energy events using the
log-likelihood method. A multiple-hit event corresponds to one or more
coincident hits in any of the surrounding CsI(Tl) crystals. The backgrounds
from the PMTs attached to the NaI(Tl) and CsI(Tl) crystals were measured using
a high-purity germanium detector (30, 31). These values were constrained to be
within 50$\%$ of the measured result because the exact locations of such
radioisotopes are uncertain. The long-lived cosmogenic radioisotopes were
constrained to be within 50$\%$ of their calculation production values whereas
the other short-lived cosmogenic components were floated. Figure 9() and Table
2 show the fitted results for the NaI-037 crystal on all simulated background
components and the summary of the fitted radioactive contaminants,
respectively. The overall energy spectra match the data for the single-hit and
multiple-hit events satisfactorily. The level of the fitted internal
components is similar to the previously grown NaI-036 crystal (28).

The expected background level in the COSINE-200 crystal can be studied from
the simulated background by assuming a 12.5 kg detector in the COSINE-100
shielding, as described in Ref. (28). If the measured backgrounds, given in
Table 2 for the simulated study, are considered, a background level of
approximately 0.5 counts/kg/keV/day in the 1–6 keV energy region is obtained,
which is similar to the result for the NaI-036 crystal in the previous study
(28). This is a slightly higher background level than observed from the
NaI-035 crystal owing to the higher 210Pb contamination. However, it is still
less than 1 count/kg/keV/day, the target background level for the COSINE-200
experiment.

Table 2: Summary of the fitted radioactive contaminants in the modeling of the NaI-037 crystal. Background source | Isotope | Activity (mBq/kg)
---|---|---
Internal | ${}^{238}{\rm U}$ | $0.025\pm 0.35$
${}^{228}{\rm Th}$ | $0.0065\pm 0.00025$
${}^{40}{\rm K}$ | $0.17\pm 0.047$
${}^{210}{\rm Pb}$ | $0.36\pm 0.11$
Cosmogenic | ${}^{125}{\rm I}$ | $0.40\pm 0.0015$
${}^{121}{\rm Te}$ | $0.80\pm 0.0029$
${}^{121m}{\rm Te}$ | $0.063\pm 0.0096$
${}^{123m}{\rm Te}$ | $0.045\pm 0.099$
${}^{125m}{\rm Te}$ | $0.14\pm 0.011$
${}^{127m}{\rm Te}$ | $0.16\pm 0.10$
${}^{109}{\rm Cd}$ | $0.0071\pm 0.0010$
${}^{113}{\rm Sn}$ | $0.020\pm 0.00094$
${}^{22}{\rm Na}$ | $0.050\pm 0.010$
${}^{3}{\rm H}$ | $0.0037\pm 0.0097$
NaI PMTs | ${}^{238}{\rm U}$ | $48.83\pm 5.90$
${}^{228}{\rm Th}$ | $23.80\pm 5.70$
${}^{40}{\rm K}$ | $58.07\pm 17.82$
CsI PMTs | ${}^{238}{\rm U}$ | $27.64\pm 6.15$
${}^{228}{\rm Th}$ | $24.18\pm 6.10$
${}^{40}{\rm K}$ | $378.28\pm 17.74$

##  7 Conclusion

In this article, we presented the performance of the first ultra-low
background NaI(Tl) crystal produced using the direct purification of the NaI
powder in our facility as a part of a program for the next-generation
COSINE-200 experiment. The results of this study show a similar quantity of
internal background contamination in the crystals grown using commercial
Astro-grade powder. It indicates that the direct powder purification and
crystal growth procedures employed at our facility can provide suitable
NaI(Tl) crystals for the COSINE-200 experiment. Based on the experience of
developing ultra-pure NaI(Tl) crystals, we are moving to full-size crystal
growth with our purified powder.

## Acknowledgments

We thank Korea Hydro and Nuclear Power Co., Ltd. (KHNP) for providing the
underground laboratory space at Yangyang. This work is supported by the
Institute for Basic Science (IBS) under the project code IBS-R016-A1.

## References

  * Clowe et al. (2006) Clowe D, et al.  A direct empirical proof of the existence of dark matter.  Astrophys. J. 648 (2006) L109.  10.1086/508162.
  * Aghanim et al. (2020) Aghanim N, et al.  Planck 2018 results. VI. Cosmological parameters.  Astron. Astrophys. 641 (2020) A6.  10.1051/0004-6361/201833910.  [Erratum: Astron.Astrophys. 652, C4 (2021)].
  * Bertone and Hooper (2018) Bertone G, Hooper D.  History of dark matter.  Rev. Mod. Phys. 90 (2018) 045002.  10.1103/RevModPhys.90.045002.
  * Undagoitia and Rauch (2016) Undagoitia TM, Rauch L.  Dark matter direct-detection experiments.  J. Phys. G 43 (2016) 013001.  10.1088/0954-3899/43/1/013001.
  * Schumann (2019) Schumann M.  Direct Detection of WIMP Dark Matter: Concepts and Status.  J. Phys. G 46 (2019) 103003.  10.1088/1361-6471/ab2ea5.
  * Bernabei et al. (2013) Bernabei R, et al.  Final model independent result of DAMA/LIBRA-phase1.  Eur. Phys. J. C 73 (2013) 2648.  10.1140/epjc/s10052-013-2648-7.
  * Bernabei et al. (2018) Bernabei R, et al.  First model independent results from DAMA/LIBRA-phase2.  Nucl. Phys. Atom. Energy 19 (2018) 307.  10.15407/jnpae2018.04.307.
  * Savage et al. (2009) Savage C, et al.  Compatibility of DAMA/LIBRA dark matter detection with other searches.  JCAP 0904 (2009) 010.  10.1088/1475-7516/2009/04/010.
  * Ko et al. (2019) Ko YJ, et al.  Comparison between DAMA/LIBRA and COSINE-100 in the light of Quenching Factors.  JCAP 1911 (2019) 008.  10.1088/1475-7516/2019/11/008.
  * Workman et al. (2022) Workman RL, et al.  Review of Particle Physics.  PTEP 2022 (2022) 083C01.  10.1093/ptep/ptac097.
  * Kim et al. (2015a) Kim KW, et al.  Tests on NaI(Tl) crystals for WIMP search at the Yangyang Underground Laboratory.  Astropart. Phys. 62 (2015a) 249.  10.1016/j.astropartphys.2014.10.004.
  * Xu et al. (2015) Xu J, et al.  SABRE – A test of DAMA with high-purity NaI(Tl) crystals.  AIP Conf. Proc. 1672 (2015) 040001.  10.1063/1.4927983.
  * Adhikari et al. (2018a) Adhikari G, et al.  Initial Performance of the COSINE-100 Experiment.  Eur. Phys. J. C 78 (2018a) 107.  10.1140/epjc/s10052-018-5590-x.
  * Fushimi (2018) Fushimi KI.  Low Background Measurement by Means of NaI(Tl) Scintillator: Improvement of Sensitivity for Cosmic Dark Matter.  RADIOISOTOPES 67 (2018) 101.  10.3769/radioisotopes.67.101.
  * Coarasa et al. (2019) Coarasa I, et al.  ANAIS-112 sensitivity in the search for dark matter annual modulation.  Eur. Phys. J. C 79 (2019) 233.  10.1140/epjc/s10052-019-6733-4.
  * Amare et al. (2019a) Amare J, et al.  Performance of ANAIS-112 experiment after the first year of data taking.  Eur. Phys. J. C 79 (2019a) 228.  10.1140/epjc/s10052-019-6697-4.
  * Suerfu et al. (2020) Suerfu B, et al.  Growth of Ultra-high Purity NaI(Tl) Crystal for Dark Matter Searches.  Phys. Rev. Research 2 (2020) 013223.  10.1103/PhysRevResearch.2.013223.
  * Adhikari et al. (2018b) Adhikari G, et al.  An experiment to search for dark-matter interactions using sodium iodide detectors.  Nature 564 (2018b) 83.  10.1038/s41586-018-0739-1.
  * Adhikari et al. (2021a) Adhikari G, et al.  Strong constraints from COSINE-100 on the DAMA dark matter results using the same sodium iodide target.  Sci. Adv. 7 (2021a) abk2699.  10.1126/sciadv.abk2699.
  * Adhikari et al. (2022a) Adhikari G, et al.  Searching for low-mass dark matter via the Migdal effect in COSINE-100.  Phys. Rev. D 105 (2022a) 042006.  10.1103/PhysRevD.105.042006.
  * Adhikari et al. (2022b) Adhikari G, et al.  An induced annual modulation signature in COSINE-100 data by DAMA/LIBRA’s analysis method (2022b).  10.48550/arXiv.2208.05158.
  * Adhikari et al. (2019) Adhikari G, et al.  Search for a Dark Matter-Induced Annual Modulation Signal in NaI(Tl) with the COSINE-100 Experiment.  Phys. Rev. Lett. 123 (2019) 031302.  10.1103/PhysRevLett.123.031302.
  * Adhikari et al. (2022c) Adhikari G, et al.  Three-year annual modulation search with COSINE-100.  Phys. Rev. D 106 (2022c) 052005.  10.1103/PhysRevD.106.052005.
  * Shin et al. (2018) Shin K, et al.  Reduction of the radioactivity in sodium iodide (NaI) powder by recrystallization method.  J. Radioanal. Nucl. Chem. 317 (2018) 1329.  10.1007/s10967-018-6006-y.
  * Shin et al. (2020) Shin K, et al.  A facility for mass production of ultra-pure NaI powder for the COSINE-200 experiment.  JINST 15 (2020) C07031.  10.1088/1748-0221/15/07/C07031.
  * Ra et al. (2019) Ra S, et al.  Scintillation crystal growth at the CUP.  PoS ICHEP2018 (2019) 668.  10.22323/1.340.0668.
  * Choi et al. (2020) Choi J, et al.  Improving the light collection using a new NaI(Tl)crystal encapsulation.  Nucl. Instrum. Meth. A 981 (2020) 164556.  10.1016/j.nima.2020.164556.
  * Park et al. (2020) Park B, et al.  Development of ultra-pure NaI(Tl) detectors for the COSINE-200 experiment.  Eur. Phys. J. C 80 (2020) 814.  10.1140/epjc/s10052-020-8386-8.
  * Shin et al. (2023) Shin K, et al.  Mass production of ultra-pure NaI powder for COSINE-200 (2023).  Paper in preparation.
  * Adhikari et al. (2018c) Adhikari P, et al.  Background model for the NaI(Tl) crystals in COSINE-100.  Eur. Phys. J. C 78 (2018c) 490.  10.1140/epjc/s10052-018-5970-2.
  * Adhikari et al. (2017) Adhikari G, et al.  Understanding NaI(Tl) crystal background for dark matter searches.  Eur. Phys. J. C 77 (2017) 437.  10.1140/epjc/s10052-017-5011-6.
  * Ra et al. (2020) Ra SJ, et al.  Status of ultra-pure scintillating crystal growth for rare process experiments by CUP.  JPCS 1468 (2020) 012144.  10.1088/1742-6596/1468/1/012144.
  * Prihtiadi et al. (2018) Prihtiadi H, et al.  Muon detector for the COSINE-100 experiment.  JINST 13 (2018) T02007.  10.1088/1748-0221/13/02/T02007.
  * Adhikari et al. (2016) Adhikari P, et al.  Understanding internal backgrounds in NaI(Tl) crystals toward a 200 kg array for the KIMS-NaI experiment.  Eur. Phys. J. C 76 (2016) 185.  10.1140/epjc/s10052-016-4030-z.
  * Lee et al. (2006) Lee HS, et al.  First limit on WIMP cross section with low background CsI(Tl) crystal detector.  Phys. Lett. B 633 (2006) 201.  10.1016/j.physletb.2005.12.035.
  * Lee et al. (2007) Lee H, et al.  Development of low-background CsI(Tl) crystals for WIMP search.  Nucl. Instrum. Meth. A 571 (2007) 644–650.  10.1016/j.nima.2006.10.325.
  * Adhikari et al. (2018d) Adhikari G, et al.  The COSINE-100 Data Acquisition System.  JINST 13 (2018d) P09006.  10.1088/1748-0221/13/09/P09006.
  * Amare et al. (2019b) Amare J, et al.  Analysis of backgrounds for the ANAIS-112 dark matter experiment.  Eur. Phys. J. C 79 (2019b) 412.  10.1140/epjc/s10052-019-6911-4.
  * Kim et al. (2015b) Kim KW, et al.  Tests on NaI(Tl) crystals for WIMP search at the Yangyang Underground Laboratory.  Astropart. Phys. 62 (2015b) 249.  10.1016/j.astropartphys.2014.10.004.
  * Bernabei et al. (2008) Bernabei R, et al.  The DAMA/LIBRA apparatus.  Nucl. Instrum. Meth. A 592 (2008) 297–315.  10.1016/j.nima.2008.04.082.
  * Adhikari et al. (2021b) Adhikari G, et al.  Background modeling for dark matter search with 1.7 years of COSINE-100 data.  Eur. Phys. J. C 81 (2021b) 837.  10.1140/epjc/s10052-021-09564-0.
  * Fushimi et al. (2021) Fushimi K, et al.  Development of highly radiopure NaI(Tl) scintillator for PICOLON dark matter search project.  PTEP 2021 (2021) 043F01.  10.1093/ptep/ptab020.
  * Bernabei et al. (2012) Bernabei R, et al.  Performances of the new high quantum efficiency pmts in dama/libra.  JINST 7 (2012) P03009.  10.1088/1748-0221/7/03/P03009.
  * de Souza et al. (2020) de Souza EB, et al.  Study of cosmogenic radionuclides in the COSINE-100 NaI(Tl) detectors.  Astropart. Phys. 115 (2020) 102390.  10.1016/j.astropartphys.2019.102390.


# Exposing Privacy Gaps: Membership Inference Attack on Preference Data for
LLM Alignment

Qizhang Feng† Siva Rajesh Kasa‡ Hyokun Yun‡ Choon Hui Teo‡ Sravan Babu
Bodapati‡
† Texas A&M University ‡ Amazon
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Large Language Models (LLMs) have seen widespread adoption due to their
remarkable natural language capabilities. However, when deploying them in
real-world settings, it is important to align LLMs to generate texts according
to acceptable human standards. Methods such as Proximal Policy Optimization
(PPO) and Direct Preference Optimization (DPO) have made significant progress
in refining LLMs using human preference data. However, the privacy concerns
inherent in utilizing such preference data have yet to be adequately studied.
In this paper, we investigate the vulnerability of LLMs aligned using human
preference datasets to membership inference attacks (MIAs), highlighting the
shortcomings of previous MIA approaches with respect to preference data. Our
study has two main contributions: first, we introduce a novel reference-based
attack framework specifically for analyzing preference data called PREMIA
(Preference data MIA); second, we provide empirical evidence that DPO models
are more vulnerable to MIA compared to PPO models. Our findings highlight gaps
in current privacy-preserving practices for LLM alignment.

Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM
Alignment

Qizhang Feng† Siva Rajesh Kasa‡ Hyokun Yun‡ Choon Hui Teo‡ Sravan Babu
Bodapati‡ † Texas A&M University ‡ Amazon<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Large language models (LLMs) have seen a surge in their adoption in the recent
past due to their remarkable capabilities on a wide range of natural language
processing (NLP) tasks such as question answering, code generation, etc Zhao
et al. (2023). When deployed in real-world scenarios, it is important to align
LLMs to human preferences. Techniques such as Proximal Policy Optimization
(PPO) and Direct Preference Optimization (DPO) play a key role in aligning
LLMs with human ethical standards by leveraging human-derived preference data
Christiano et al. (2017); Rafailov et al. (2024); Yang et al. (2024). Although
these approaches improve the alignment of models with human values, they are
fraught with privacy concerns because of their use of human-generated data. In
this work, we investigate the Membership Inference Attack (MIA), a widely-
studied vulnerability that attempts to determine whether specific data points
are used in the model’s training dataset. The study of MIA highlights
vulnerabilities in a variety of machine learning paradigms, including several
recent studies that specifically focus on LLMs (Fu et al., 2023; Shi et al.,
2024). Although existing research on MIA in the context of LLMs highlights the
need to evaluate and address the need for privacy concerns, the unique
challenges posed by alignment methods such as the PPO and DPO approaches
(where preference data directly influences model behavior) remain to be
explored. Traditional MIA frameworks fall short when applied to the complex,
context-dependent optimization procedures used in LLM alignment. In this
paper, we introduce a novel MIA framework that is specifically tailored to
address preference data vulnerabilities in LLM alignment, providing a more
precise analysis tool that can effectively mitigate these vulnerabilities. Our
contributions to this field are twofold:

  * •

Introduction of a Novel Reference-based Attack Framework: We propose a new
attack framework designed to assess the vulnerability of preference data to
MIA, providing an effective analytical tool to address the unique privacy
challenges in LLM alignment.

  * •

Comparative Vulnerability Assessment of DPO and PPO Models: Through our
framework, we find that DPO models are more vulnerable to MIA compared to PPO
models. This insight not only points to significant privacy concerns, but also
emphasizes the need for stronger privacy-preserving strategies in developing
and deploying LLMs aligned using DPO.

##  2 Preliminaries

This section introduces the notations and background concepts upon which the
rest of the paper builds. We begin by defining the frameworks of PPO and DPO,
followed by an overview of MIAs.

###  2.1 Model Alignment

Model alignment ensures LLMs adhere to human values and ethics by adjusting
their outputs to match human preferences Hendrycks et al. (2021); Ouyang et
al. (2022). Such alignment is critical for creating AI systems that act in
ways that benefit humans and reduce the risks associated with improper
alignment. Among the various model alignment techniques, PPO and DPO are some
of the widely used approaches Xu et al. (2024).

####  2.1.1 Proximal Policy Optimization (PPO)

Stiennon et al. (2020) and Bai et al. (2022) illustrate Reinforcement Learning
from Human Feedback (RLHF) that integrates human feedback into the training of
pre-trained Language Models (LMs), encompassing three phases: Supervised Fine-
Tuning (SFT), Preference Sampling with Reward Learning, and Reinforcement
Learning (RL) through PPO.

SFT begins the process by fine-tuning a pre-trained LM on task-specific data
to obtain a model $\pi^{\text{SFT}}$, enhancing the LLM’s performance on the
task at hand.

Preference Data Collection involves gathering a set of preference data pairs
$(x,y_{w},y_{l})$, where $x$ is a prompt and $y_{w}$, $y_{l}$ are two
different responses. Here, $y_{w}$ is the response preferred by human
evaluators over $y_{l}$ for the given context $x$.

Reward Modeling Phase uses the preference pairs to train the reward model
$r_{\phi}(x,y)$, where $\phi$ represents the trainable parameters. The
trainable model can be a classification header layer attached to the base
model or a separate model. The Bradley-Terry (BT) model is commonly used to
represent the probability that one response is better than another:

$\displaystyle\mathcal{L}_{R}(r_{\phi},\mathcal{D})=$ (1)
$\displaystyle\;\;\;\;\;\;-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log\sigma(r_{\phi}(x,y_{w})-r_{\phi}(x,y_{l}))\right],$

where $r_{\phi}(x,y)$ models the likelihood of preferring $y_{w}$ to $y_{l}$
given the prompt $x$, and $\mathcal{D}$ denotes the dataset of preference
pairs. This loss function measures the accuracy of the reward model in
predicting human preferences.

RL Fine-Tuning Phase then fine-tunes the LM further using the learned reward
function, striving to align model outputs with human preferences while
maintaining generative diversity:

$\displaystyle\max_{\pi_{\theta}}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(y|x)}[r_{\phi}(x,y)]$
(2)
$\displaystyle\;\;\;\;\;\;-\beta\mathbb{D}_{\textrm{KL}}[\pi_{\theta}(y|x)||\pi^{\text{SFT}}(y|x)],$

balancing fidelity to human feedback with the preservation of the model’s
original capabilities. Here, $\pi_{\theta}$ represents the policy of the
language model parameterized by $\theta$, the trainable parameters. The
optimization in equation 2 is carried out using Proximal Policy Optimization
(PPO) method (Schulman et al., 2017) and throughout the rest of the paper, we
use RLHF and PPO interchangeably to refer the same approach.

####  2.1.2 Direct Preference Optimization (DPO)

DPO offers a refined approach to fine-tuning language models by directly
leveraging preference data, bypassing the explicit reward model construction
typically associated with RLHF methodologies Rafailov et al. (2024). This
method reformulates the two-step optimization procedure in equations 1 and 2
into a single optimization problem that simultaneously optimizes the policy
and encodes an implicit reward mechanism based on the preference data.

$\begin{split}&\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=\\\
&-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\biggl{[}\log\sigma\biggl{(}\beta\log\frac{\pi_{\theta}(y_{w}\mid
x)}{\pi_{\text{ref}}(y_{w}\mid x)}\\\ &-\beta\log\frac{\pi_{\theta}(y_{l}\mid
x)}{\pi_{\text{ref}}(y_{l}\mid x)}\biggr{)}\biggr{]}.\end{split}$ (3)

Here, $\pi_{\text{ref}}$ refers to a reference model which is typically chosen
as the SFT model $\pi^{\text{SFT}}$. This optimization method is preferred
over PPO because it simplifies training by optimizing directly on the
preference data, which improves computational efficiency and is easier to
implement Rafailov et al. (2024); Xu et al. (2024). Note that in PPO (equation
2), contrary to DPO (equation 3), the final model being optimized is not
directly aligned using the data $\mathcal{D}$. This is the key intuition
behind why PPO-aligned models are less susceptible to privacy threats compared
to their DPO counterparts.

###  2.2 Membership Inference Attacks (MIA) on LLMs

MIA poses a significant privacy risk in the context of LLMs, challenging the
security of data used in training such models Shokri et al. (2017); Nasr et
al. (2018). In LLMs, MIAs seek to determine whether specific data was part of
the model’s training set, exploiting the model’s behavior or output nuances to
infer data membership. These attacks are particularly concerning for models
trained on vast datasets, where inadvertently revealing individual data points
could lead to privacy breaches.

The effectiveness of an MIA against LLMs is quantified by a score function
$\mathcal{M}$, mapping input samples and the level of access to a real-valued
score indicating the likelihood of membership. For a given threshold $\tau$,
an input $x$ is classified as a training set member if
$\mathcal{M}(x,\text{Access}(\theta))\geq\tau$.

$\mathcal{M}:\mathcal{X}\times\text{Access}(\Theta)\to\mathbb{R}.$ (4)

Research on MIAs targeting LLMs underscores the need for robust privacy-
preserving techniques to safeguard training data, with implications for the
development and deployment of secure, trustworthy AI systems Carlini et al.
(2020).

###  2.3 Problem Statement

Current research on MIAs has advanced understanding of risks in pre-trained
text models, but gaps remain in applying MIAs to preference datasets in LLM
alignment. This oversight poses substantial privacy risks, given the critical
role of preference data in shaping LLM outputs.

Let $\mathcal{D}_{\text{pref}}=\\{(x_{i},y_{wi},y_{li})\\}_{i=1}^{N}$
represent the preference dataset, where $x_{i}$ is a prompt, $y_{wi}$ is the
preferred response, and $y_{li}$ is the less preferred response. The
vulnerability of this preference data to MIAs requires a nuanced examination,
which can be categorized into three distinct attack vectors:

  * •

Attack against prompts and preferred responses: This attack determines whether
a specific pair of prompt $x$ and preferred response $y_{w}$ has been used in
training, highlighting potential privacy breaches if such data can be
identified:

$\text{MIA}_{\text{prompt, }y_{w}}:\mathcal{X}\times\mathcal{Y}\to\\{0,1\\},$
(5)

  * •

Attack against prompts and non-preferred responses: This attack focuses on
identifying if a pair consisting of a prompt $x$ and a non-preferred response
$y_{l}$ was part of the training data, potentially exposing sensitive
decision-making processes:

$\text{MIA}_{\text{prompt, }y_{l}}:\mathcal{X}\times\mathcal{Y}\to\\{0,1\\},$
(6)

  * •

Attack against the entire preference tuple: This more comprehensive attack
assesses whether the entire tuple $(x,y_{w},y_{l})$ can be traced back to the
training set, reflecting a higher risk of revealing critical training
methodologies:

$\text{MIA}_{\mathcal{D}_{\text{pref}}}:\mathcal{X}\times\mathcal{Y}\times\mathcal{Y}\to\\{0,1\\}.$
(7)

This detailed breakdown elucidates the complex vulnerabilities associated with
preference data in LLMs. By identifying these specific attack vectors, we aim
to advance privacy-preserving mechanisms that safeguard the alignment process
and ensure that models respect and protect individual privacy while adhering
to human ethical standards.

###  2.4 Hypotheses Regarding DPO vs PPO

To guide our experimental design and directly address the concerns raised by
our study, we propose the following hypotheses. These are crafted to explore
the distinct impacts of DPO and Proximal Policy Optimization (PPO) on privacy
and performance, and are structured to align with the subsequent analyses
conducted in our experiments.

Hypothesis 1: Differential Vulnerability to MIAs: We hypothesize that the DPO
model is more vulnerable to MIA than the PPO model since the DPO model uses
preference data directly, which may lead to overfitting. We empirically assess
the MIA vulnerability of DPO and PPO models.

Hypothesis 2: Influence of Model Size on MIA Risk: We postulate that larger
models, regardless of the training method (DPO or PPO), will show increased
vulnerability to MIAs due to their greater capacity to memorize training data.
This hypothesis is explored in §4.4.2, assessing how model size affects
susceptibility to data leakage.

Hypothesis 3: Trade-offs Between Performance and Privacy: We propose that
while DPO may enhance alignment with human preferences and potentially improve
task-specific performance, it also increases the risk of privacy breaches
compared to PPO. This trade-off is critically examined in §4.4.3, comparing
the performance benefits of DPO against its privacy drawbacks.

##  3 Method

Our approach introduces a tailored framework for evaluating MIA on preference
datasets used for LLM model alignment. Traditional MIA approaches do not take
into account the uniqueness of preference data, which includes relational
dynamics and contextual dependencies. Our approach addresses these nuances by
splitting the analysis into evaluating individual components and entire
preference tuples, and using conditional probability ratios to compare against
a reference model to more accurately infer membership in the data.

###  3.1 For Individual Response

Assessing the vulnerability of individual responses—either preferred ($y_{w}$)
or not preferred ($y_{l}$)—to MIAs necessitates a nuanced approach that
considers the specific characteristics of preference data. We compute the
conditional probability ratio relative to a reference model
$\pi_{\text{ref}}$:

$\rho_{y}=\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)},$ (8)

where $\pi_{\theta}$ represents the target model. This ratio measures the
likelihood that the target model will produce a specific response compared to
a baseline model, indicating potential overfitting to training data.

Employing $\rho_{y}$ enhances specificity by accounting for the subtle nuances
and context-dependent nature of response preferences, thus improving the
detection of data membership:

$\text{MIA}_{\text{single}}(x,y)=\begin{cases}1&\text{if
}\rho_{y}>\tau_{y},\\\ 0&\text{otherwise}.\end{cases}$ (9)

Although $\tau_{y}$ is mentioned, our primary metric in the experiments is the
Area Under the Receiver Operating Characteristic (AUROC), which does not
require setting a specific threshold. This approach allows for a flexible
assessment of model sensitivity across various potential values.

The choice of the reference model $\pi_{\text{ref}}$ serves as a benchmark for
comparing the behavior of the target model $\pi_{\theta}$. This model can be
the base pre-trained model from which $\pi_{\theta}$ originated or a different
base model trained on the same dataset. Our experiments, designed to test both
scenarios, consistently demonstrate robust performance of our MIA method under
various conditions.

###  3.2 For the Entire Preference Tuple

To ascertain the membership of the complete preference tuple
$(x,y_{w},y_{l})$, we compute the difference between the probability ratios of
the preferred and not preferred responses:

$\Delta\rho=\rho_{y_{w}}-\rho_{y_{l}}.$ (10)

This measure captures the comparative preference strength more effectively,
offering a nuanced insight into how preference data impacts model training:

$\text{MIA}_{\text{tuple}}(x,y_{w},y_{l})=\begin{cases}1&\text{if
}\Delta\rho>\tau_{\Delta},\\\ 0&\text{otherwise}.\end{cases}$ (11)

The specified threshold $\tau_{\Delta}$ is set based on the variance observed
within the training data, allowing a more accurate identification of the data
used during the training phase.

##  4 Experiments

###  4.1 Research Questions and Experiment Design Rationale

This subsection outlines the key research questions guiding our experimental
design, providing a rationale for our methodologies. Derived from our
hypotheses, these questions aim to evaluate the comparative effectiveness,
privacy implications, and utility of DPO and Proximal Policy Optimization
(PPO) in training LLMs.

Research Question 1: How do DPO and PPO differ in their susceptibility to
Membership Inference Attacks? This question tests Hypothesis 1 by comparing
the vulnerability of models trained using DPO and PPO to MIA to shed light on
privacy and data security issues.

Research Question 2: Does model size influence its risk of data leakage
through MIAs, and how does this vary between DPO and PPO trained models? In
line with Hypothesis 2, this question explores the impact of model size on MIA
effectiveness, assessing if larger models pose greater privacy risks.

Research Question 3: What are the performance and privacy trade-offs when
employing DPO versus PPO in LLMs? Echoing Hypothesis 3, this question examines
the trade-off between performance and data privacy in tasks that need to be
understood like humans, assessing whether greater alignment with human
preferences would compromise privacy.

###  4.2 Setup

##### Models.

Our experiments are conducted using a variety of models to ensure a
comprehensive evaluation on different scales of model complexity. We include
Mistral-7B-v0.1 Jiang et al. (2023), as well as a series of models from the
OpenAI GPT-2 family Radford et al. (2019): GPT2, GPT2-medium, GPT2-large, and
GPT2-xl. Furthermore, we incorporate Open-llama-3b and Open-llama-7b models
Geng and Liu (2023); Computer (2023); Touvron et al. (2023) to broaden our
analysis across various architectures and capacities. For the reference model
in our ratio calculations, we primarily use the SFT model trained from the
same base pre-trained version of the model being evaluated. Additionally, we
conduct experiments where the reference model differs from the base model to
evaluate the robustness of our methodology under varied conditions.

##### Datasets.

For our experiments, we utilize the Stack-Exchange-Paired
dataset111https://huggingface.co/datasets/lvwerra/stack-exchange-paired and
the IMDB-RLHF-Pair dataset222https://github.com/QiyaoWei/Reproduce-DPO. Both
datasets have a prompt $x$ accompanied by two responses: the "chosen" response
$y_{w}$ and the "rejected" response $y_{l}$. The Stack-Exchange-Paired dataset
contains questions and answers from the Stack Overflow dataset, where answers
with more votes are preferred. The IMDB-RLHF-Pair dataset is generated by
IMDB, and responses with positive sentiment are preferred. For the Stack-
Exchange-Paired dataset, the data/rl split is used for training, and
data/evaluation is used as validation data. For the IMDB-RLHF-Pair dataset,
20k entries are used for training, while the remaining is for validation.

##### Evaluation Metrics.

To comprehensively assess our models, we employ a dual-focused evaluation
framework encompassing utility performance and MIA robustness:

  * •

Utility Performance: Our evaluation includes the reward score of generated
responses given by the reward model and perplexity for assessing fluency. We
also incorporate comprehensive diversity measures: Mean Segmented Type Token
Ratio (MSSTR), Distinct-1, Distinct-2, Unique-1, and Unique-2 metrics
(Johnson, 1944; Li et al., 2015; Ramamurthy et al., 2022). Additionally, we
utilize advanced text generation quality metrics such as BERTScore Zhang et
al. (2019), ROUGE Lin (2004), BLEU Papineni et al. (2002), and METEOR Banerjee
and Lavie (2005), which collectively offer a nuanced view of the models’
performance in terms of fluency, adequacy, and diversity, closely mirroring
human judgment in text quality assessment.

  * •

MIA Performance: To measure the model’s susceptibility to MIA, we utilize the
Area Under the Receiver Operating Characteristic curve (AUROC). This metric
encapsulates the model’s defense against MIAs, reflecting the balance between
true positive rate and false positive rate in identifying training data.

##### Implementation Details.

Due to the computational efficiency of LoRA, we used LoRA for all of our model
training processes. Additionally, we hypothesized that fine-tuning LoRA at the
RL stage would help to ensure that the aligned model does not deviate
significantly from the reference model. To further improve efficiency, we also
used quantization techniques. We use
TRL333https://huggingface.co/docs/trl/index for model alignment training. More
detailed implementation information can be found in Appendix A.

###  4.3 Baselines

To accurately evaluate our approach, we position it against well-known MIA
baselines specifically tailored for language models and preference data
analysis. These baselines are designed to target individual components of the
preference data but do not extend to analyzing entire preference tuples.

##### Perplexity (PPL):

The loss attack method, based on the approach outlined in Yeom et al. (2018),
utilizes the perplexity of a sequence to gauge how well a language model
predicts the tokens within that sequence. Perplexity is defined as:

$\mathcal{P}=\exp\left(-\frac{1}{n}\sum_{i=1}^{n}\log\pi_{\theta}(x_{i}|x_{1},\ldots,x_{i-1})\right),$
(12)

where a lower perplexity indicates a higher likelihood that the sequence was
the training data.

##### Comparing to zlib Compression (Zlib):

This method measures the entropy of a sequence when compressed using zlib,
compares the perplexity of a model to its zlib compression entropy, and uses
their ratio as an inference metric Carlini et al. (2021).

##### Comparing to Lowercased Text (Lowercase):

This method evaluates the change in perplexity of a sequence before and after
it has been lowercased, to assess the model’s dependency on specific
capitalization Carlini et al. (2021):

$\text{Perplexity
Ratio}=\frac{\mathcal{P}(\text{Original})}{\mathcal{P}(\text{Lowercased})}.$
(13)

##### Comparing to Other Neural Language Models (Ref):

This approach consists of comparing the ease of error of sequences between the
target model and another small model. In our experiments, we specifically use
GPT2 as the small model. Note that our approach uses conditional
probabilities, whereas Ref does not.

##### MIN-K% PROB (MIN-K):

This method Shi et al. (2024) focuses on the minimum token probabilities
within a text. It posits that non-member examples are more likely to contain
outlier words with high negative log-likelihoods:

$\text{MIN-K}(x)=\frac{1}{E}\sum_{x_{i}\in\text{Min-K\%}(x)}\log\pi_{\theta}(x_{i}|x_{1},...,x_{i-1}).$
(14)

By analyzing these low probability tokens, MIN-K% PROB provides a distinct
method to infer membership, enhancing the diversity of our baseline
comparisons.

###  4.4 Results

This section presents the findings from our experiments, highlighting the
comparative effectiveness of our proposed MIA defense mechanism and analyzing
the trade-off between model performance and privacy protection.

####  4.4.1 Effectiveness of MIA Methodology

This subsection evaluates our MIA methodology for identifying if preference
data components were used in training language models. Our detailed
comparative analysis shows our method’s high precision in discerning sensitive
data inclusions, outperforming traditional MIA approaches not tailored for
preference data scenarios.

Table 1: AUROC scores comparing different MIA methods on Mistral-7B, Open-llama-3b, and Open-llama-7b models are presented, where higher scores indicate greater susceptibility to MIA. The best and second-best scores in each column are highlighted in orange and green, respectively. The better score between DPO and PPO trained models is underlined. |  | IMDB |  |  |  | Stack-Exchange |  |  |
---|---|---|---|---|---|---|---|---|---
|  | $\text{MIA}_{\text{Chosen}}$ |  |  $\text{MIA}_{\text{Rejected}}$ |  | $\text{MIA}_{\text{Chosen}}$ |  | $\text{MIA}_{\text{Rejected}}$ |
|  | DPO | PPO |  DPO | PPO | DPO |  PPO | DPO | PPO
Mistral-7B |  PPL | 0.569 | 0.538 |  0.588 | 0.503 | 0.572 |  0.500 | 0.561 | 0.513
|  Zlib | 0.593 | 0.568 |  0.606 | 0.535 | 0.566 |  0.528 | 0.523 | 0.528
|  Lowercase | 0.516 | 0.509 |  0.515 | 0.501 | 0.582 |  0.514 | 0.563 | 0.521
|  Ref | 0.571 | 0.533 |  0.607 | 0.510 | 0.548 |  0.504 | 0.612 | 0.512
|  MIN-K | 0.564 | 0.535 |  0.582 | 0.509 | 0.601 |  0.503 | 0.627 | 0.513
|  PREMIA-base | 0.570 | 0.524 |  0.612 | 0.517 | 0.790 |  0.507 | 0.751 | 0.556
|  PREMIA-SFT | 0.572 | 0.507 |  0.611 | 0.527 | 0.807 |  0.535 | 0.750 | 0.501
Open-llama-3b |  PPL | 0.580 | 0.508 |  0.573 | 0.526 | 0.590 |  0.506 | 0.558 | 0.507
|  Zlib | 0.602 | 0.540 |  0.595 | 0.506 | 0.530 |  0.529 | 0.522 | 0.508
|  Lowercase | 0.541 | 0.556 |  0.546 | 0.551 | 0.649 |  0.503 | 0.582 | 0.516
|  Ref | 0.587 | 0.508 |  0.590 | 0.535 | 0.610 |  0.505 | 0.603 | 0.524
|  MIN-K | 0.587 | 0.526 |  0.579 | 0.538 | 0.610 |  0.533 | 0.613 | 0.512
|  PREMIA-base | 0.564 | 0.520 |  0.562 | 0.540 | 0.748 |  0.527 | 0.717 | 0.511
|  PREMIA-SFT | 0.594 | 0.504 |  0.609 | 0.518 | 0.785 |  0.543 | 0.743 | 0.551
Open-llama-7b |  PPL | 0.577 | 0.529 |  0.572 | 0.505 | 0.594 |  0.514 | 0.543 | 0.551
|  Zlib | 0.599 | 0.559 |  0.593 | 0.525 | 0.577 |  0.501 | 0.521 | 0.500
|  Lowercase | 0.537 | 0.501 |  0.540 | 0.502 | 0.539 |  0.504 | 0.545 | 0.565
|  Ref | 0.583 | 0.515 |  0.586 | 0.503 | 0.625 |  0.507 | 0.561 | 0.582
|  MIN-K | 0.597 | 0.527 |  0.583 | 0.511 | 0.605 |  0.527 | 0.607 | 0.536
|  PREMIA-base | 0.559 | 0.511 |  0.560 | 0.504 | 0.773 |  0.541 | 0.728 | 0.510
|  PREMIA-SFT | 0.594 | 0.511 |  0.611 | 0.527 | 0.736 |  0.530 | 0.749 | 0.520
Figure 1: AUROC scores for $\text{MIA}_{\text{Pair}}$ detection for
Mistral-7B, Open-llama-3b, and Open-llama-7b models.

Table 1 presents the AUROC scores for various MIA methods across Mistral-7B,
Open-llama-3b, and Open-llama-7b models. PREMIA-base and PREMIA-SFT indicate
using the base model or SFT model as the reference model respectively. Our
method uniquely addresses the entire preference tuple and consistently
achieves the highest AUROC scores, demonstrating superior data membership
identification (see Figure 3 for paired tuple analysis). The comparison
between DPO and PPO reveals DPO’s increased susceptibility to MIA, indicating
that its enhancements in aligning models with human preferences might elevate
privacy risks. We do not measure the entire tuple using baselines because
traditional MIA methods are not designed to handle the relational and
contextual dependencies inherent in preference data.

####  4.4.2 Impact of Model Size on MIA Effectiveness

Table 2: Performance of PREMIA-SFT on various GPT2 model variants across Stack-Exchange-Paired and IMDB-RLHF-PairTwo datasets. |  | $\text{MIA}_{\text{Chosen}}$ |  |  $\text{MIA}_{\text{Rejected}}$ |  | $\text{MIA}_{\text{Pair}}$ |
---|---|---|---|---|---|---|---
|  | DPO | PPO |  DPO | PPO | DPO |  PPO
Stack Exchange |  GPT2 | 0.815 | 0.520 |  0.770 | 0.500 | 0.909 |  0.523
|  GPT2-medium | 0.809 | 0.528 |  0.698 | 0.525 | 0.889 |  0.522
|  GPT2-large | 0.838 | 0.502 |  0.694 | 0.548 | 0.891 |  0.515
|  GPT2-xl | 0.839 | 0.515 |  0.850 | 0.504 | 0.900 |  0.501
|  Open-llama-3b |  0.785 | 0.543 |  0.743 | 0.551 | 0.920 |  0.512
|  Open-llama-7b | 0.736 | 0.530 |  0.749 | 0.520 | 0.908 |  0.534
|  Mistral | 0.807 | 0.535 |  0.750 | 0.501 | 0.935 |  0.537
IMDB |  GPT2 | 0.636 | 0.550 |  0.713 | 0.511 | 0.771 |  0.549
|  GPT2-medium | 0.641 | 0.549 |  0.707 | 0.539 | 0.762 |  0.528
|  GPT2-large | 0.615 | 0.611 |  0.659 | 0.583 | 0.704 |  0.520
|  GPT2-xl | 0.623 | 0.591 |  0.643 | 0.579 | 0.692 |  0.519
|  Open-llama-3b | 0.594 | 0.504 |  0.609 | 0.518 | 0.509 |  0.518
|  Open-llama-7b | 0.594 | 0.511 |  0.611 | 0.527 | 0.500 |  0.512
|  Mistral-7B | 0.572 | 0.507 |  0.611 | 0.527 | 0.556 |  0.537
Table 3: Privacy vs Utility Trade-off analysis on the Mistral-7B model.  |  Base | SFT | PPO |  DPO
---|---|---|---|---
$\text{MIA}_{\text{Chosen}}$ |  — | 0.53 | 0.54 |  0.80
$\text{MIA}_{\text{Rejected}}$ |  — | 0.61 | 0.50 |  0.75
$\text{MIA}_{\text{Pair}}$ |  — | 0.55 | 0.50 |  0.93
Reward$\uparrow$ |  -1.922 | -1.953 | -0.771 |  -1.035
PPL$\downarrow$ |  11.148 | 7.673 | 11.671 |  14.991
msttr-100$\uparrow$ |  0.673 | 0.651 | 0.633 |  0.640
distinct 1$\uparrow$ |  0.180 | 0.127 | 0.085 |  0.123
distinct 2$\uparrow$ |  0.631 | 0.521 | 0.422 |  0.520
unique 1$\uparrow$ |  2010 | 3213 | 3530 |  3059
unique 2$\uparrow$ |  9507 | 17238 | 25205 |  18017
Bert Score$\uparrow$ |  0.876 | 0.879 | 0.883 |  0.877
ROUGE$\uparrow$ |  0.424 | 0.458 | 0.457 |  0.443
BLEU$\uparrow$ |  0.348 | 0.367 | 0.338 |  0.360
METEOR$\uparrow$ |  0.445 |  0.467 | 0.449 |  0.466

Table 2 details the PREMIA-SFT results for models of different sizes on the
Stack-Exchange and IMDB datasets. On the Stack-Exchange dataset, large models
typically have higher AUROC scores in all MIA scenarios, indicating that they
retain more specific details of the training data. However, on the IMDB
dataset, the Mistral-7B and Open-llama models have significantly worse MIA
performance. One possible reason is that the task is too simple for them. As
shown in Fig. 2, Mistral-7B achieves over 90% accuracy in distinguishing
between selected and rejected responses in only the first 0.2 epoch. Large
pre-trained models like Mistral-7B already have strong generalization
capabilities, which undermines the effectiveness of MIA. Similarly, large GPT2
models such as GPT2-xl show better generalization on simple tasks, making them
less susceptible to MIA.

Figure 2: Train/Eval Aaccuracy for Mistral-7B on IMDB.

####  4.4.3 Trade-Off between Performance and Privacy

Table 3 analyzes the trade-off between vulnerability to MIA and model utility
under various Mistral-7B model configurations on the Stack Exchange dataset.
The "Reward" row represents the average reward score given by the reward model
for each of these models, indicating how well the task was accomplished.
Clearly, DPO and PPO have better rewards compared to the rest. Further, DPO is
clearly more vulnerable to MIA. However, DPO did not improve utility metrics
such as reward and complexity. It is worth noting that PPO provides similar
utility performance to DPO, but it has a lower AUROC. These findings are in
line with existing research, which also shows that despite DPO being
relatively straightforward to train, it does not improve the model performance
compared to PPO Ivison et al. (2024); Xu et al. (2024).

####  4.4.4 Impact of Response Length on MIA Effectiveness

In this experiment, we look the effect of length of examples used in
preference alignment and their corresponding vulnerability in terms of AUC-ROC
of PREMIA-SFT. Figure 3 shows the MIA AUROC results for the GPT-2 family of
models on the IMDB dataset. As can be seen from the figure, for "Chosen"
responses, the longer the response, the more susceptible it is to MIA, while
for "Rejected" responses, the opposite is true.

Figure 3: AUROC vs Average Length for GPT-2 Models

##  5 Future Work

Our study shows that advanced privacy-preserving techniques are needed when
using preference data for LLM alignment. Optimizing the privacy model
architecture without losing performance is key. Techniques such as DP-SGD
Abadi et al. (2016), model pruning Han et al. (2015), and knowledge
distillation Hinton et al. (2015) should be evaluated. It is also necessary to
create benchmarks and assessment frameworks for privacy risks in LLM
alignment. These benchmarks and assessment frameworks should model realistic
attacks and provide metrics for comparing privacy-preserving methods to ensure
that LLMs are consistent with human values without compromising privacy.

##  6 Conclusion

This paper examines the vulnerability of preference datasets in LLM alignment
to MIAs. We reveal that models trained with DPO are more susceptible to MIAs
than those using PPO, posing a significant privacy risk as preference data use
increases. Our attack framework excels in detecting training data membership,
stressing the need for robust privacy-preserving methods. Larger models
enhance capabilities but increase privacy risks, highlighting the trade-off
between performance and data security.

##  7 Limitations

First, the analysis conducted in this study is limited to open-source LLMs and
does not include proprietary or closed-source models such as ChatGPT. The
privacy implications and vulnerability to MIA of these closed-source LLMs may
differ because their training data, architecture, and alignment techniques are
not fully transparent. Second, this study focuses on the privacy implications
of two well-known alignment techniques (PPO and DPO). However, the field of
LLM alignment is rapidly evolving, and the privacy risks associated with other
alignment methods can be more fully analyzed in future work.

## References

  * Abadi et al. (2016) Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ , pages 308–318. ACM.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _arXiv preprint arXiv:2204.05862_.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72.
  * Carlini et al. (2020) Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Úlfar Erlingsson, et al. 2020.  Extracting training data from large language models. corr abs/2012.07805 (2020).  _arXiv preprint arXiv:2012.07805_.
  * Carlini et al. (2021) Nicholas Carlini et al. 2021.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Computer (2023) Together Computer. 2023.  Redpajama-data: An open source recipe to reproduce llama training dataset.
  * Fu et al. (2023) Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. 2023.  Practical membership inference attacks against fine-tuned large language models via self-prompt calibration.  _arXiv preprint arXiv:2311.06062_.
  * Geng and Liu (2023) Xinyang Geng and Hao Liu. 2023.  Openllama: An open reproduction of llama.
  * Han et al. (2015) Song Han, Huizi Mao, and William J Dally. 2015.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  _arXiv preprint arXiv:1510.00149_.
  * Hendrycks et al. (2021) Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. 2021.  Unsolved problems in ml safety.  _arXiv preprint arXiv:2109.13916_.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  In _NIPS Deep Learning and Representation Learning Workshop_.
  * Ivison et al. (2024) Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. 2024.  Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback.  _Preprint_ , arXiv:2406.09279.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Johnson (1944) Wendell Johnson. 1944.  Studies in language behavior: A program of research.  _Psychological Monographs_ , 56(2):1–15.
  * Li et al. (2015) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015.  A diversity-promoting objective function for neural conversation models.  _arXiv preprint arXiv:1510.03055_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Nasr et al. (2018) Milad Nasr, Reza Shokri, and Amir Houmansadr. 2018.  Comprehensive privacy analysis of deep learning.  In _Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP)_ , volume 2018, pages 1–15.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36.
  * Ramamurthy et al. (2022) Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022.  Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization.  _arXiv preprint arXiv:2210.01241_.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Shi et al. (2024) Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024.  Detecting pretraining data from large language models.  In _The Twelfth International Conference on Learning Representations_.
  * Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017.  Membership inference attacks against machine learning models.  In _2017 IEEE symposium on security and privacy (SP)_ , pages 3–18. IEEE.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020.  Learning to summarize with human feedback.  _Advances in Neural Information Processing Systems_ , 33:3008–3021.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Xu et al. (2024) Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. 2024.  Is dpo superior to ppo for llm alignment? a comprehensive study.  _arXiv preprint arXiv:2404.10719_.
  * Yang et al. (2024) Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024.  Harnessing the power of llms in practice: A survey on chatgpt and beyond.  _ACM Transactions on Knowledge Discovery from Data_ , 18(6):1–32.
  * Yeom et al. (2018) Samuel Yeom et al. 2018.  Privacy risk in machine learning: Analyzing the connection to overfitting.  _IEEE 31st Computer Security Foundations Symposium (CSF)_.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.

##  Appendix A Implementation Details

We mainly refer to the TRL444https://huggingface.co/docs/trl/en/index package
for implementation.

LoRA Setting. For all experiments, we share the same LoRA setting below, using
the PEFT555https://huggingface.co/docs/peft/index package: lora_alpha 32,
lora_dropout 0.05, lora_r 16, and no bias term.

Quantization Setting. For all experiments, we use the
BitsAndBytes666https://huggingface.co/docs/bitsandbytes/index package for
4-bit quantization.

SFT Setting. The settings for SFT are detailed below. We utilized the
"train/rl" split of the stack-exchange-paired dataset, selecting 80,000 data
points for the fine-tuning process, same data is used for PPO and DPO
training. The prompt and only the preferred response are concatenated as
input. The specific training parameters are:
\- Training Epochs: 2.0
\- Learning Rate: 8e-5
\- Batch Size (Training): 4
\- Batch Size (Evaluation): 2
\- Gradient Accumulation Steps: 4
\- Learning Rate Scheduler: cosine
\- Warmup Steps: 100
\- Weight Decay: 0.05
\- Optimizer: paged_adamw_32bit
\- Mixed Precision Training: fp16

PPO Setting. The settings for PPO are detailed below. We filter out data
points with maximum length constraints. We also limit the maximum length of
the generated response. The specific training parameters are:
\- Batch Size: 16
\- Mini Batch Size: 4
\- Gradient Accumulation Steps: 4
\- PPO Epochs: 6
\- Learning Rate: 5.4e-5
\- KL Coefficient: 0.1
\- Adaptive KL Control: True
\- Target KL: 5.0
\- Horizon: 4000
\- Training Epochs: 4
\- Maximum Output Length: 128
\- Maximum Prompt Length: 256
\- Maximum Sequence Length: 1024

DPO Setting. The settings for DPO training are detailed below. The specific
training parameters are:
\- Batch Size (Training): 8
\- Batch Size (Evaluation): 2
\- Gradient Accumulation Steps: 2
\- Training Epochs: 3.0
\- Learning Rate: 5e-4
\- Warmup Steps: 100
\- Maximum Sequence Length: 1024
\- Maximum Prompt Length: 256
\- Optimizer Type: paged_adamw_32bit
\- Beta: 0.4