# Generalized Contrastive Divergence: Joint Training
of Energy-Based Model and Diffusion Model
through Inverse Reinforcement Learning

Sangwoong Yoon1, Dohyun Kwon2, Himchan Hwang3, Yung-Kyun Noh1,4, Frank C.
Park3,5
1Korea Institute for Advanced Study, 2University of Seoul,
3Seoul National University, 4Hanyang University, 5Saige Research
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

We present Generalized Contrastive Divergence (GCD), a novel objective
function for training an energy-based model (EBM) and a sampler
simultaneously. GCD generalizes Contrastive Divergence [1], a celebrated
algorithm for training EBM, by replacing Markov Chain Monte Carlo (MCMC)
distribution with a trainable sampler, such as a diffusion model. In GCD, the
joint training of EBM and a diffusion model is formulated as a minimax
problem, which reaches an equilibrium when both models converge to the data
distribution. The minimax learning with GCD bears interesting equivalence to
inverse reinforcement learning, where the energy corresponds to a negative
reward, the diffusion model is a policy, and the real data is expert
demonstrations. We present preliminary yet promising results showing that
joint training is beneficial for both EBM and a diffusion model. GCD enables
EBM training without MCMC while improving the sample quality of a diffusion
model.

##  1 Introduction

Diffusion models [2, 3] have achieved tremendous success in generating
realistic high-dimensional samples. However, several challenges are remaining.
First, a diffusion model does not directly minimize the deviation between the
model and the data distributions. While recent theories show that diffusion
models indirectly minimize a divergence between the model and data [4, 5, 6],
a direct minimization can be more effective, if possible [7]. Second,
generation is slow, requiring a large number of denoising steps. Shortening
the number of steps often incurs a trade-off in sample quality [7, 8]. Third,
the marginal likelihood of data can not be directly computed, and only its
lower bound or importance-weighted approximation is available [2, 3].

In this paper, we propose Generalized Contrastive Divergence (GCD), a novel
objective function motivated by Contrastive Divergence [1], that shows
training a diffusion model in conjunction with an energy-based model (EBM)
helps tackle these problems. First, learning with GCD is equivalent to the
minimization of integral probability metric (IPM; e.g., Wasserstein distance)
between a diffusion model and data, where the energy works as a critic in IPM.
Second, therefore, a diffusion model with fewer steps can be fine-tuned with
GCD to improve sample quality. Third, the jointly trained EBM can compute the
marginal likelihood of data (up to constant).

This paper presents preliminary experimental results showing that GCD learning
works successfully on a synthetic dataset and can be effective in improving
the sample quality of the DDPM sampler, especially when the number of steps is
small. We expect to obtain experimental results on larger real-world datasets
in the near future. Furthermore, we discuss the connection between GCD
learning and inverse reinforcement learning. This connection opens up
potential opportunities for advancing generative modeling by leveraging the
recent ideas of reinforcement learning.

The contributions of this paper can be summarized as follows:

  * •

We propose GCD learning, a novel method for jointly training EBM and a
parametric sampler. The joint training is beneficial for both models.

    * –

GCD learning enables EBM training without MCMC, which is often computationally
expensive and unstable.

    * –

GCD learning can be used to train a diffusion-style sampler with a small
number of timesteps.

  * •

We show that GCD learning has interesting equivalence to the integral
probability metric minimization problems (e.g., WGAN) and inverse
reinforcement learning.

##  2 Preliminaries

Diffusion for Sampling. In this paper, we focus on discrete-time diffusion-
based samplers, such as DDPM [2]. A sampler first draws a Gaussian noise
vector $\mathbf{x}_{0}$ and then iteratively samples from conditional
distributions $\mathbf{x}_{t+1}\sim\pi(\mathbf{x}_{t+1}|\mathbf{x}_{t},t)$ to
obtain a final sample $\mathbf{x}_{T}$. We will denote the final samples as
$\mathbf{x}=\mathbf{x}_{T}$ and their marginal distribution as
$\pi(\mathbf{x})$. The initial samples $\mathbf{x}_{0}$ are drawn from
$\mathcal{N}(0,I)$. The sampler $\pi(\mathbf{x})$ may be pre-trained through
existing diffusion model techniques.

Energy-Based Models. An energy-based model (EBM) $q(\mathbf{x})$ uses a scalar
function called an energy $E(\mathbf{x})$ to represent a probability
distribution:

$\displaystyle q(\mathbf{x})=\frac{1}{Z}\exp(-E(\mathbf{x})/\tau),\quad
E:\mathcal{X}\to\mathbb{R}\quad
Z=\int_{\mathcal{X}}\exp(-E(\mathbf{x})/\tau)d\mathbf{x},$ (1)

where $\tau>0$ is temperature, $\mathcal{X}$ is the compact domain of data,
and the normalization constant $Z$ is assumed to be finite. The integral is
replaced with a summation for discrete input data.

While multiple training algorithms for EBM have been proposed (see [9] for an
overview), the standard method is maximum likelihood training which is
equivalent to minimization of Kullback-Leibler (KL) divergence between data
and a model: $\min_{q\in\mathcal{Q}}KL(p||q)$, where $p(\mathbf{x})$ is the
data distribution, and $\mathcal{Q}$ being the set of feasible
$q(\mathbf{x})$’s. KL divergence is defined as
$KL(p||q)=\int_{\mathcal{X}}p(\mathbf{x})\log(p(\mathbf{x})/q(\mathbf{x}))d\mathbf{x}$.
Meanwhile, the exact computation of the log-likelihood gradient requires
convergent MCMC sampling.

Contrastive Divergence [1]. Hinton proposed an alternative objective function
for training EBM which does not require convergent MCMC. Let us write
$\mathcal{T}_{q}$ as a single-step MCMC operator designed to draw sample from
$q(\mathbf{x})$ and $\mathcal{T}_{q}^{k}(p)$ as the distribution of points
after $k$ steps of MCMC transition $\mathcal{T}_{q}$ starting from
$p(\mathbf{x})$. Then, Contrastive Divergence (CD) learning is defined as
follows:

$\displaystyle\min_{q\in\mathcal{Q}}KL(p||q)-KL(\mathcal{T}_{q}^{k}(p)||q).$
(2)

CD learning is known to converge when $KL(p||q)\geq
KL(\mathcal{T}_{q}^{k}(p)||q)$ is guaranteed [10]. However, CD still requires
convergent MCMC when drawing a new sample from $q(\mathbf{x})$.

##  3 Generalized Contrastive Divergence

###  3.1 Generalized Contrastive Divergence Learning

We present Generalized Contrastive Divergence (GCD), an objective function for
the joint training of EBM and a sampler. From CD (Eq. 2), we replace MCMC
distribution $\mathcal{T}_{q}^{k}(p)(\mathbf{x})$ with an arbitrary sampler
$\pi(\mathbf{x})$ which will later be set as a DDPM-like discrete-time
diffusion model. We also introduce maximization with respect to
$\pi(\mathbf{x})$, motivated by the fact that
$\mathcal{T}_{q}^{k}(p)(\mathbf{x})$ converges towards $q(\mathbf{x})$.

######  Definition (GCD).

Suppose the data distribution $p(\mathbf{x})$, the EBM $q(\mathbf{x})$, and
the sampler $\pi(\mathbf{x})$ are supported on the domain $\mathcal{X}$. Let
$\mathcal{Q}$ and $\Pi$ be the sets of feasible $q(\mathbf{x})$’s and
$\pi(\mathbf{x})$’s, respectively. GCD learning is defined as the following
minimax problem:

$\displaystyle\inf_{q\in\mathcal{Q}}{\color[rgb]{0,0,1}\sup_{\pi\in\Pi}}KL(p||q)-KL({\color[rgb]{0,0,1}\pi}||q),$
(3)

where the changes made from CD are highlighted. This minimax problem has a
desirable equilibrium.

######  Proposition 1 (Equilibrium).

Suppose $p(\mathbf{x})\in\mathcal{Q}=\Pi$. Then,
$p(\mathbf{x})=q(\mathbf{x})=\pi(\mathbf{x})$ is the equilibrium of GCD
learning.

Eq. 3 can be rewritten with respect to energy. Plugging EBM’s definition (Eq.
1), we obtain the following equivalent problem, which we solve in practice:

$\displaystyle\inf_{E\in\mathcal{E}}\sup_{\pi\in\Pi}\mathbb{E}_{p}[E(\mathbf{x})]-\mathbb{E}_{\pi}[E(\mathbf{x})]+\tau\mathcal{H}(\pi),$
(4)

where $\mathcal{H}(\pi)=-\int\pi(\mathbf{x})\log\pi(\mathbf{x})d\mathbf{x}$ is
the differential entropy of $\pi(\mathbf{x})$. The set of energy functions
$\mathcal{E}$ is derived from $\mathcal{Q}$:

$\displaystyle\mathcal{E}:=\left\\{E:\mathcal{X}\rightarrow\mathbb{R}|\exp(-E(\mathbf{x})/\tau)/Z\in\mathcal{Q}\hbox{
where }Z=\int\exp(-E(\mathbf{x})/\tau)d\mathbf{x}<\infty\right\\}.$ (5)

In other words, $E\in\mathcal{E}$ if and only if $E(\mathbf{x})=-\tau\log
q(\mathbf{x})+c$ for some $q\in\mathcal{Q}$ and a constant $c$. Similarly to
Eq. 3, GCD learning in energy (Eq. 4) also reaches the equilibrium when
$E(\mathbf{x})=-\tau\log p(\mathbf{x})+c$ and $\pi(\mathbf{x})=p(\mathbf{x})$
(see Appendix A.2). GCD learning has interesting theoretical connections to
other generative modeling problems and inverse reinforcement learning.

###  3.2 Equivalence to Entropy-Regularized Integral Probability Metric
Minimization

Generative modeling is often formulated as the minimization of Integral
Probability Metric (IPM; [11]), which measures the deviation between the data
distribution and the model. WGAN [12] is a well-known example. GCD learning
(Eq. 4) can also be viewed as the minimization of IPM between $p(\mathbf{x})$
and $\pi(\mathbf{x})$ but under entropy regularization for $\pi(\mathbf{x})$.
The energy $E(\mathbf{x})$ works as a critic in IPM, and the feasible set
$\mathcal{E}$ characterizes IPM (hence we write IPM as
$D^{\mathcal{E}}(p||\pi)$).

######  Proposition 2 (Entropy-regularized IPM minimization).

Assume that $p(\mathbf{x})\in\mathcal{Q}\cap\Pi$. In addition, suppose that
$\mathcal{E}$ is closed under negation. Consider the following problem:

$\displaystyle\inf_{\pi\in\Pi}D^{\mathcal{E}}(p||\pi)-\tau\mathcal{H}(\pi),\hbox{
where
}D^{\mathcal{E}}(p||\pi):=\sup_{E\in\mathcal{E}}\lvert\mathbb{E}_{p}[E(\mathbf{x})]-\mathbb{E}_{\pi}[E(\mathbf{x})]\rvert.$
(6)

This problem has the same optimal value and the same equilibrium point
($E(\mathbf{x})=-\tau\log p(\mathbf{x})+c$ and
$\pi(\mathbf{x})=p(\mathbf{x})$) to the GCD learning in energy (Eq. 4).

Proposition 2 holds because Eq. 4 and Eq. 6 are the primal and the dual
problems of the same objective function, where total duality holds. A detailed
proof is in Appendix A.2.

In conventional IPM minimization problems, such as WGAN, where $\tau=0$ and
$\mathcal{E}$ being the set of 1-Lipschitz functions, the critic is not
guaranteed to converge to a quantity related to $\log p(\mathbf{x})$. The
entropy regularization is critical for obtaining an accurate energy estimate
and is the key difference from a prior work [7], where DDPM is optimized for
only IPM.

###  3.3 Equivalence to Maximum Entropy Inverse Reinforcement Learning

GCD learning can also be interpreted through the lens of reinforcement
learning (RL). Eq. 4 is a special case of maximum entropy inverse
reinforcement learning (IRL; [13, 14, 15]). The connection between diffusion
modeling and Markov Diffusion Processes (MDP) is well noted in previous works
[7, 16]. The sampler $\pi(\mathbf{x})$ is an agent with a stochastic policy
$\pi(\mathbf{x}_{t+1}|\mathbf{x}_{t},t)$. The intermediate samples
$\mathbf{x}_{0},\mathbf{x}_{1},\ldots,\mathbf{x}_{T}(=\mathbf{x})$, combined
with time indices, form a trajectory of states
$\\{(\mathbf{x}_{t},t)\\}_{t=0}^{T}$. The action corresponds to the choice of
the next state.

However, as no explicit reward signal is available in generative modeling, the
reward must be inferred from the data, making the problem one of IRL. A
training data $\mathbf{x}$ serves as expert demonstrations for the terminal
state $\mathbf{x}_{T}$. There is no demonstration for intermediate states, and
thus the reward is only given for the terminal state. Then, a natural choice
of the reward for a sampler is $\log p(\mathbf{x})$, which is unknown. In GCD
learning, the energy function is responsible for learning $\log
p(\mathbf{x})$, providing the reward signal $-E(\mathbf{x})$ for training
$\pi(\mathbf{x})$.

One notable features of GCD is that it maximizes the terminal state entropy
$\mathcal{H}(\pi(\mathbf{x}_{T}))$ instead of the causal entropy
$\mathcal{H}(\pi(\mathbf{x}_{t+1}|\mathbf{x}_{t},t))$, as is done in other
max-ent IRL methods [13, 14, 15]. The connection to IRL allows us to analyze
the training from RL perspective and to employ techniques proven to be
effective in RL.

##  4 Joint Training of EBM and Diffusion Model

Now we present an algorithm for GCD learning. While GCD learning is not
restricted on the choice of $\pi(\mathbf{x})$, we shall focus on the case
where $\pi(\mathbf{x})$ is a DDPM-style sampler. We write $\theta$ and $\phi$
as the parameters of EBM $q_{\theta}(\mathbf{x})$ and a sampler
$\pi_{\phi}(\mathbf{x})$, respectively. GCD learning can be written as
follows:

$\displaystyle\min_{\theta}\max_{\phi}\mathcal{L}(\theta,\phi),\;\mathcal{L}=\mathbb{E}_{p}[E_{\theta}(\mathbf{x})]-\mathbb{E}_{\pi_{\phi}}[E_{\theta}(\mathbf{x})]+\tau\mathcal{H}(\pi_{\phi}).$
(7)

To solve this minimax problem, we alternatively update the EBM
$E_{\theta}(\mathbf{x})$ and the diffusion model $\pi_{\phi}(\mathbf{x})$ as
typically done in gradient descent-ascent algorithms, e.g., [17]. Temperature
$\tau$ is treated as a hyperparameter.

Figure 1: 2D Density Estimation on 8 Gaussians. The red shades indicate the
energy (white is low), and the dots are samples. Without entropy
regularization ($\tau=0$), the energy does not reflect the data distribution,
and the samples are collapsed to the mode.

Figure 2: Value functions at each time step ($\tau=0.1$ case). Blue indicates
a high value.

EBM Update. The energy is updated to discriminate the training data and the
samples from $\pi_{\phi}(\mathbf{x})$.

$\displaystyle\nabla_{\theta}\mathcal{L}=$
$\displaystyle\mathbb{E}_{p(\mathbf{x})}[\nabla_{\theta}E_{\theta}(\mathbf{x})]-\mathbb{E}_{\pi_{\phi}(\mathbf{x})}[\nabla_{\theta}E_{\theta}(\mathbf{x})].$
(8)

This update corresponds to the reward learning step in IRL [13, 15].

Diffusion Model Update. The sampler is updated to maximize the reward
$-E_{\theta}(\mathbf{x})$ under entropy regularization. When a sampler admits
a reparametrization trick, we may directly take the gradient of
$-E_{\theta}(\mathbf{x})$. However, the reparametrization trick may not be
applicable to a diffusion model due to the memory overhead. Instead, we can
employ REINFORCE-style policy gradient algorithm [18] (Derivation in Appendix
C):

$\displaystyle\nabla_{\phi}\mathcal{L}$
$\displaystyle=\nabla_{\phi}\left(\mathbb{E}_{\pi_{\phi}(\mathbf{x})}[-E_{\theta}(\mathbf{x})]+\tau\cdot\mathcal{H}(\pi_{\phi})\right)=\mathbb{E}_{\pi_{\phi}(\mathbf{x})}\left[\nabla_{\phi}\log\pi_{\phi}(\mathbf{x})(-E_{\theta}(\mathbf{x})-\tau\log\pi_{\phi}(\mathbf{x}))\right]$
$\displaystyle=\mathbb{E}_{\pi_{\phi}(\mathbf{x}_{0:T})}\left[\sum_{t=0}^{t={T-1}}\nabla_{\phi}\log\pi_{\phi}(\mathbf{x}_{t+1}|\mathbf{x}_{t},t)\left[-E_{\theta}(\mathbf{x}_{T})-\tau\log\pi_{\phi}(\mathbf{x}_{T})-B(\mathbf{x}_{t},t)\right]\right],$
(9)

where the term $-\tau\log\pi_{\phi}(\mathbf{x}_{T})$ can be seen as an
additional reward for exploration. This update corresponds to the forward RL
in typical IRL. For sample efficiency, we use proximal policy optimization
[19] to update the parameters multiple times for training data. The baseline
function $B(\mathbf{x}_{t},t)$ is introduced to reduce the variance [20] and
is described shortly.

Marginal Probability Estimation. The log probability
$\log\pi_{\phi}(\mathbf{x}_{T})$ in Eq. 9 can not be directly computed in
DDPM. We estimate $\log\pi_{\phi}(\mathbf{x}_{T})$ using mini-batch samples
from $\pi_{\phi}(\mathbf{x})$. Following [21], we employ the nearest-neighbor
density estimator [22], which gives $\log\pi_{\phi}(\mathbf{x})\approx D\log
d_{k}(\mathbf{x})+c$ where $D$ is the dimensionality of data,
$d_{k}(\mathbf{x})$ is the distance to $k$-nearest neighbor, and $c$ is
constant with respect to $\mathbf{x}$.

Baseline Estimation. We model the baseline of $-E_{\theta}(\mathbf{x})$ and
$-\tau\log\pi_{\phi}(\mathbf{x}_{T})$ separately. For energy, we employ the
state value function as the baseline
$V(\mathbf{x}_{t},t)=\mathbb{E}_{\pi_{\phi}}[-E_{\theta}(\mathbf{x}_{T})|\mathbf{x}_{t}]$.
We parametrize $V(\mathbf{x},t)$ as a separate model and estimate its
parameter by minimizing temporal difference loss
$\min_{V(\cdot,t)}(V(\mathbf{x}_{t+1},t+1)-V(\mathbf{x}_{t},t))^{2}$ for
$t=0,\ldots,T-1$. Meanwhile, we simply use the estimated entropy as the
baseline
$\tau\mathcal{H}(\pi_{\phi})=-\tau\mathbb{E}_{\pi_{\phi}}[\log\pi_{\phi}(\mathbf{x}_{T})]$,
where the expectation is approximated using exponential moving average. As a
result, the baseline is given as
$B(\mathbf{x}_{t},t)=V(\mathbf{x}_{t},t)+\tau\mathcal{H}(\pi_{\phi})$.

Method | T | $\tau$ |  $SW$ ($\downarrow$) |  AUC ($\uparrow$)
---|---|---|---|---
DDPM | 5 | - |  0.967$\pm$0.005 | -
DDPM | 10 | - |  0.824$\pm$0.002 | -
DDPM | 100 | - |  0.241$\pm$0.003 | -
DDPM | 1000 | - |  0.123$\pm$0.014 | -
GCD-Scratch | 5 | 0.05 |  0.114$\pm$0.025 | 0.867
GCD-FT | 5 | 0.05 |  0.086$\pm$0.008 | 0.880
GCD-FT | 5 | 0 |  0.152$\pm$0.008 | 0.513
Table 1: Fine-tuning DDPM for 8 Gaussians. $SW$ indicates sliced Wasserstein-2
distance (1,000 projections) between samples and data estimated from 10k
samples. The standard deviation is computed from 5 independent samplings. AUC
indicates how well the energy discriminates the uniform distribution over the
domain from data. AUC is computed with 10k samples. The maximum value of AUC
vs the uniform distribution is about 0.9059.

##  5 Experiments

In our experiment, we show the synergistic benefit of jointly training EBM and
a diffusion model on 2D 8 Gaussians data. We use time-conditioned MLP networks
for both a value network and a policy network (i.e., sampler). The time step
is encoded into a 128D vector using sinusoidal positional embedding and
concatenated into the hidden neurons of MLP. The last time step ($t=T$) of the
value network is treated as the negative energy. The conditional distributions
$\pi(\mathbf{x}_{t+1}|\mathbf{x}_{t},t)$ in the sampler are Gaussians where
the mean is determined by the policy network. We use the linear variance
schedule from $10^{-5}$ to $10^{-2}$. Throughout the experiment, we use $T=5$
for GCD.

We first train both EBM and a sampler from scratch (Figure 2). In the
conventional IPM minimization ($\tau=0$), the critic, i.e., the energy,
deviates significantly from the true energy of $p(\mathbf{x})$. Furthermore,
the samples tend to collapse to the mode of the data density, under-
representing the variance of data. The entropy regularization makes the energy
produce $q(\mathbf{x})$ that accurately reflects $p(\mathbf{x})$. Meanwhile,
the value functions reflect the evolution of the energy from the initial
Gaussian density (Figure 2), similar to Diffusion Recovery Likelihood [23].

Next, we show GCD can improve the sample quality of pre-trained DDPM samplers
(Table 1). DDPM samplers with varying $T$ are trained on 8 Gaussians until
convergence, and Wasserstein distances from their samples to real data are
measured. The quality of samples deteriorates significantly as $T$ becomes
small. However, if we further fine-tune DDPM using GCD learning (GCD-FT), we
obtain a Wasserstein distance even smaller than that of $T=1000$ DDPM. This
result is possible because GCD learning directly minimizes IPM between the
samples and data. Similar results were reported in [7]. If we do not fine-tune
the sampler (GCD-Scratch), the distance slightly increases, showing there is a
gain from DDPM pretraining. It is interesting to note that GCD learning with
entropy regularization ($\tau=0.05$), outperforms conventional IPM
minimization ($\tau=0$). One hypothesis is that the entropy term facilitates
the exploration and thus leads to a better policy.

##  6 Related Work

GCD is an attempt to train generative samplers using RL. When applying RL, a
key design choice is defining the reward signal. One source of reward is human
feedback [24, 25], which became popular after being applied in large language
models [26]. However, since human feedback is typically costly, a machine
learning model, such as an image aesthetic quality estimator, can be used as a
substitute for the reward function [16, 27]. Assuming the absence of human
feedback or an external reward function, GCD employs the IRL approach where
the reward function is inferred from demonstrations. In GCD, the reward is the
log-likelihood of data, learned from training data through EBM. The previous
work of Fan and Lee [7] can also be seen as IRL, but their reward function is
not interpreted as the log-likelihood.

GCD can be seen as a novel method for EBM training that does not require MCMC.
EBMs are a powerful class of generative models that shows promising
performance in compositional generation [28] and out-of-distribution detection
[29, 30]. However, EBM suffers from instability in training, often requiring
computationally expensive MCMC that is difficult to tune and not convergent in
practice [31, 21, 32, 23, 33]. GCD enables the stable training of EBM by
removing MCMC from training.

GCD bears a formal resemblance to algorithms that train EBM using minimax
formulation. CoopFlow [34] jointly trains a normalizing flow sampler and EBM,
but the flow is not directly optimized to minimize the divergence to EBM.
Divergence Triangle [35] trains a sampler, an encoder, and EBM simultaneously,
resulting in a more complex problem than a minimax problem. Due to the entropy
regularization, GCD may have a connection to entropy-constrained optimal
transport problems [36, 37] which regularizes the entropy of the joint
distribution. However, the entropy of a single marginal distribution is
regularized in GCD.

##  7 Conclusion

Bridging generative modeling and IRL, GCD opens up new opportunities to
improve generative modeling by leveraging tools from reinforcement learning.
We are planning to scale up the experiments to large-scale tasks, such as
image generation. Besides generation tasks, GCD may also be useful for other
tasks that require an accurate energy estimation, such as out-of-distribution
detection [21, 29].

## Acknowledgments and Disclosure of Funding

S. Yoon was supported by a KIAS Individual Grant (AP095701) via the Center for
AI and Natural Sciences at Korea Institute for Advanced Study. Y.-K. Noh was
partly supported by NRF/MSIT (No. 2018R1A5A7059549, 2021M3E5D2A01019545) and
IITP/MSIT (IITP-2021-0-02068, 2020-0-01373, RS-2023-00220628). This work was
supported in part by IITP-MSIT grant 2021-0-02068 (SNU AI Innovation Hub),
IITP-MSIT grant 2022-0-00480 (Training and Inference Methods for Goal-Oriented
AI Agents), KIAT grant P0020536 (HRD Program for Industrial Innovation), ATC+
MOTIE Technology Innovation Program grant 20008547, SRRC NRF grant
RS-2023-00208052, SNU-AIIS, SNU-IAMD, SNU BK21+ Program in Mechanical
Engineering, and SNU Institute for Engineering Research.

## References

  * [1] Geoffrey E Hinton.  Training products of experts by minimizing contrastive divergence.  Neural computation, 14(8):1771–1800, 2002.
  * [2] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020.
  * [3] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256–2265, Lille, France, 07–09 Jul 2015. PMLR.
  * [4] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon.  Maximum likelihood training of score-based diffusion models.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 1415–1428. Curran Associates, Inc., 2021.
  * [5] Dohyun Kwon, Ying Fan, and Kangwook Lee.  Score-based generative modeling secretly minimizes the wasserstein distance.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 20205–20217. Curran Associates, Inc., 2022.
  * [6] Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang.  Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions.  In The Eleventh International Conference on Learning Representations, 2023.
  * [7] Ying Fan and Kangwook Lee.  Optimizing ddpm sampling with shortcut fine-tuning.  arXiv preprint arXiv:2301.13362, 2023.
  * [8] Zhifeng Kong and Wei Ping.  On fast sampling of diffusion probabilistic models.  arXiv preprint arXiv:2106.00132, 2021.
  * [9] Yang Song and Diederik P Kingma.  How to train your energy-based models.  arXiv preprint arXiv:2101.03288, 2021.
  * [10] Siwei Lyu.  Unifying non-maximum likelihood learning objectives with minimum kl contraction.  In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
  * [11] Alfred Müller.  Integral probability metrics and their generating classes of functions.  Advances in Applied Probability, 29(2):429–443, 1997.
  * [12] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214–223. PMLR, 06–11 Aug 2017.
  * [13] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al.  Maximum entropy inverse reinforcement learning.  In AAAI, volume 8, pages 1433–1438. Chicago, IL, USA, 2008.
  * [14] Brian D. Ziebart, J. Andrew Bagnell, and Anind K. Dey.  Modeling interaction via the principle of maximum causal entropy.  In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML’10, page 1255–1262, Madison, WI, USA, 2010. Omnipress.
  * [15] Jonathan Ho and Stefano Ermon.  Generative adversarial imitation learning.  Advances in neural information processing systems, 29, 2016.
  * [16] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee.  Reinforcement learning for fine-tuning text-to-image diffusion models.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * [17] Tianyi Lin, Chi Jin, and Michael Jordan.  On gradient descent ascent for nonconvex-concave minimax problems.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6083–6093. PMLR, 13–18 Jul 2020.
  * [18] Ronald J Williams.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  Reinforcement learning, pages 5–32, 1992.
  * [19] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [20] Evan Greensmith, Peter L Bartlett, and Jonathan Baxter.  Variance reduction techniques for gradient estimates in reinforcement learning.  Journal of Machine Learning Research, 5(9), 2004.
  * [21] Yilun Du, Shuang Li, B. Joshua Tenenbaum, and Igor Mordatch.  Improved contrastive divergence training of energy based models.  In Proceedings of the 38th International Conference on Machine Learning (ICML-21), 2021.
  * [22] Lyudmyla F Kozachenko and Nikolai N Leonenko.  Sample estimate of the entropy of a random vector.  Problemy Peredachi Informatsii, 23(2):9–16, 1987.
  * [23] Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma.  Learning energy-based models by diffusion recovery likelihood.  In International Conference on Learning Representations, 2021.
  * [24] Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet.  Directly fine-tuning diffusion models on differentiable rewards.  arXiv preprint arXiv:2309.17400, 2023.
  * [25] Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, et al.  Hive: Harnessing human feedback for instructional visual editing.  arXiv preprint arXiv:2303.09618, 2023.
  * [26] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc., 2022.
  * [27] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine.  Training diffusion models with reinforcement learning.  arXiv preprint arXiv:2305.13301, 2023.
  * [28] Yilun Du, Shuang Li, and Igor Mordatch.  Compositional visual generation with energy based models.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6637–6647. Curran Associates, Inc., 2020.
  * [29] Sangwoong Yoon, Yung-Kyun Noh, and Frank Park.  Autoencoding under normalization constraints.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12087–12097. PMLR, 18–24 Jul 2021.
  * [30] Sangwoong Yoon, Young-Uk Jin, Yung-Kyun Noh, and Frank C. Park.  Energy-based models for anomaly detection: A manifold diffusion recovery approach.  In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  * [31] Yilun Du and Igor Mordatch.  Implicit generation and modeling with energy based models.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alche-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 3608–3618. Curran Associates, Inc., 2019.
  * [32] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat.  {VAEBM}: A symbiosis between variational autoencoders and energy-based models.  In International Conference on Learning Representations, 2021.
  * [33] Hankook Lee, Jongheon Jeong, Sejun Park, and Jinwoo Shin.  Guiding energy-based models via contrastive latent variables.  In International Conference on Learning Representations, 2023.
  * [34] Jianwen Xie, Yaxuan Zhu, Jun Li, and Ping Li.  A tale of two flows: Cooperative learning of langevin flow and normalizing flow toward energy-based model.  In International Conference on Learning Representations, 2022.
  * [35] Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu.  Divergence triangle for joint training of generator model, energy-based model, and inferential model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [36] Marco Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
  * [37] Petr Mokrov, Alexander Korotin, and Evgeny Burnaev.  Energy-guided entropic neural optimal transport.  arXiv preprint arXiv:2304.06094, 2023.
  * [38] Ernest K Ryu and Wotao Yin.  Large-scale convex optimization: algorithms & analyses via monotone operators.  Cambridge University Press, 2022.

##  Appendix A Proofs

###  A.1 Proof of Proposition 1

Solving the inner maximization yields $\pi^{*}(\mathbf{x})=q(\mathbf{x})$,
which is attainable since $\Pi=\mathcal{Q}$. Consequently, the outer
minimization simplifies to $\min_{q\in\mathcal{Q}}KL(p||q)$, which reaches its
minimum at $q^{*}(\mathbf{x})=p(\mathbf{x})$, attainable as $p\in\mathcal{Q}$.

###  A.2 Proof of Proposition 2

We will prove Proposition 2 by showing that GCD learning in energy (Eq. 4) and
the entropy-regularized IPM minimization (Eq. 6) are the primal and the dual
problems of the same objective function, where total duality holds. Total
duality implies strong duality, and thus the primal and the dual optimal
values are the same.

#### Primal and Dual Problems.

We consider a function $\Phi:\mathcal{E}\times\Pi\longrightarrow\mathbb{R}$
defined as follows:

$\displaystyle\Phi(E,\pi)=\mathbb{E}_{p}[E(\mathbf{x})]-\mathbb{E}_{\pi}[E(\mathbf{x})]+\tau\;\mathcal{H}(\pi).$
(10)

The primal problem generated by $\Phi(E,\pi)$ is given as

$\displaystyle\inf_{E\in\mathcal{E}}\sup_{\pi\in\Pi}\mathbb{E}_{p}[E(\mathbf{x})]-\mathbb{E}_{\pi}[E(\mathbf{x})]+\tau\;\mathcal{H}(\pi),$
(11)

which is equivalent to GCD learning in energy (Eq. 4). The dual problem
generated by $\Phi(E,\pi)$ is equivalent to the minimization of IPM under
entropy regularization:

$\displaystyle\inf_{\pi\in\Pi}\sup_{E\in\mathcal{E}}-\mathbb{E}_{p}[E(\mathbf{x})]+\mathbb{E}_{\pi}[E(\mathbf{x})]-\tau\;\mathcal{H}(\pi)$
$\displaystyle=\inf_{\pi\in\Pi}\sup_{E\in\mathcal{E}}\lvert\mathbb{E}_{p}[E(\mathbf{x})]-\mathbb{E}_{\pi}[E(\mathbf{x})]\rvert-\tau\;\mathcal{H}(\pi)$
(12)
$\displaystyle=\inf_{\pi\in\Pi}D^{\mathcal{E}}(p||\pi)-\tau\;\mathcal{H}(\pi),$
(13)

where we used the condition that $\mathcal{E}$ is closed under negation to
obtain Eq. 12.

#### Total Duality.

We will show that total duality holds between these primal and dual problems.
The function $\Phi$ is linear (and hence convex) for $E(\mathbf{x})$ and
concave for $\pi(\mathbf{x})$. Then, total duality holds if and only if $\Phi$
has a saddle point ([38, Ch. 1, pp. 12-14]).

Let us define a function $E^{p}(\mathbf{x})=-\tau\log{p(\mathbf{x})}+c$, where
$c$ being a constant, which can be considered as the energy of $p(\mathbf{x})$
computed under the temperature $\tau$. Let us show that
$(E^{*},\pi^{*})=(E^{p},p)$ is the saddle point of $\Phi$ for any constant $c$
in $E^{p}$. Note that $(E^{p},p)\in\mathcal{E}\times\Pi$ as
$p\in\mathcal{Q}\cap\Pi$.

$\displaystyle\Phi(E^{p},\pi)$
$\displaystyle=\mathbb{E}_{p}[E^{p}(\mathbf{x})]+\mathbb{E}_{\pi}[\tau\log
p-\tau\log\pi]$ (14) $\displaystyle=\mathbb{E}_{p}[E^{p}(\mathbf{x})]-\tau
KL(\pi||p)\leq\Phi(E^{p},p)\quad\forall\pi\in\Pi$ (15)

Also,

$\displaystyle\Phi(E,p)$
$\displaystyle=\mathbb{E}_{p}[E(\mathbf{x})]-\mathbb{E}_{p}[E(\mathbf{x})]+\tau\mathcal{H}(p)$
(16) $\displaystyle=\tau\mathcal{H}(p)=\Phi(E^{p},p)\quad\forall
E\in\mathcal{E}$ (17)

Combining Eq. 15 and Eq. 17, we obtain
$\Phi(E^{p},\pi)\leq\Phi(E^{p},p)\leq\Phi(E,p)$ for all
$(E,\pi)\in\mathcal{E}\times\Pi$, and thus $(E^{p},p)$ is a saddle point of
$\Phi$. The saddle point is the equilibrium for both the primal and the dual
problems.

##  Appendix B Algorithm

Algorithm 1 Generalized Contrastive Divergence

Initialize sample buffer $\mathcal{B}=\varnothing$.

Set $i=0$. Set sampler update period $n_{sampler}$ and the number of PPO
update $n_{PPO}$.

for $\mathbf{x}$ in dataset do

Obtain $m$ samples from $\pi(\mathbf{x})$ and append them into $\mathcal{B}$.

Update EBM $E(\mathbf{x})$ using $\mathcal{B}$ by Eq. 8.

Update value functions $V(\mathbf{x}_{t},t)$’s using $\mathcal{B}$.

if $i\;\%\;n_{sampler}=0$ then

for $j=1,...,n_{PPO}$ do

Update $\pi(\mathbf{x})$ using PPO using $\mathcal{B}_{\text{on}}$.

end for

Reset $\mathcal{B}$ as $\varnothing$.

end if

$i\leftarrow i+1$

end for

##  Appendix C Derivation of Policy Gradient

Here, we provide a detailed derivation on the policy gradient update for
sample parameters (Eq. 9).

$\displaystyle\nabla_{\phi}\mathcal{L}$
$\displaystyle=\nabla_{\phi}\int\pi_{\phi}(\mathbf{x}_{T})[-E_{\theta}(\mathbf{x}_{T})-\tau\log\pi_{\phi}(\mathbf{x}_{T})]d\mathbf{x}_{T}$
(18)
$\displaystyle=\int\nabla_{\phi}\pi_{\phi}(\mathbf{x}_{T})[-E_{\theta}(\mathbf{x}_{T})-\tau\log\pi_{\phi}(\mathbf{x}_{T})]d\mathbf{x}_{T}$
(19)
$\displaystyle=\int\nabla_{\phi}\left(\int\pi_{\phi}(\mathbf{x}_{0:T})d\mathbf{x}_{0:{T-1}}\right)[-E_{\theta}(\mathbf{x}_{T})-\tau\log\pi_{\phi}(\mathbf{x}_{T})]d\mathbf{x}_{T}$
(20)
$\displaystyle=\int\nabla_{\phi}\pi_{\phi}(\mathbf{x}_{0:T})[-E_{\theta}(\mathbf{x}_{T})-\tau\log\pi_{\phi}(\mathbf{x}_{T})]d\mathbf{x}_{0:T}$
(21)
$\displaystyle=\int\pi_{\phi}(\mathbf{x}_{0:T})\nabla_{\phi}\log\pi_{\phi}(\mathbf{x}_{0:T})[-E_{\theta}(\mathbf{x}_{T})-\tau\log\pi_{\phi}(\mathbf{x}_{T})]d\mathbf{x}_{0:T}$
(22)
$\displaystyle=\mathbb{E}_{\pi_{\phi}(\mathbf{x}_{0:T})}\left[\nabla_{\phi}\log\pi_{\phi}(\mathbf{x}_{0:T})[-E_{\theta}(\mathbf{x}_{T})-\tau\log\pi_{\phi}(\mathbf{x}_{T})]\right]$
(23)
$\displaystyle=\mathbb{E}_{\pi_{\phi}(\mathbf{x}_{0:T})}\left[\left(\sum_{t=0}^{t={T-1}}\nabla_{\phi}\log\pi_{\phi}(\mathbf{x}_{t+1}|\mathbf{x}_{t},t)\right)[-E_{\theta}(\mathbf{x}_{T})-\tau\log\pi_{\phi}(\mathbf{x}_{T})]\right].$
(24)


#  On the possibility of measuring the polarization of a ${}^{3}\rm{He}$ beam
at EIC by the HJET polarimeter

A. A. Poblaguev<EMAIL_ADDRESS>G. Atoian  N. H. Buttimore  A. Zelenski
Brookhaven National Laboratory, Upton, New York 11973, USA  School of
Mathematics, Trinity College Dublin, Dublin 2, Ireland

(December 12, 2022)

###### Abstract

The requirements for hadron polarimetry at the future Electron Ion Collider
(EIC) include measurements of the absolute helion (${}^{3}\text{He}$, $h$)
beam polarization with systematic uncertainties better than
$\sigma^{\text{syst}}_{P}/P\lesssim 1\%$. Here, we consider a possibility to
utilize the Polarized Atomic Hydrogen Gas Jet Target (HJET) for precision
measurement of polarization of the $\sim$100 GeV/n helion beam. HJET, which
serves to determine absolute proton beam polarization at the Relativistic
Heavy Ion Collider, provides the accuracy of about
$\delta^{\text{syst}}P/P\sim 0.5\%$. Potential problems for adapting the HJET
method for the EIC helion beam include (i) necessity to know the ratio of
$p^{\uparrow}{h}$ and $h^{\uparrow}{p}$ analyzing powers
$A_{\text{N}}^{ph}(t)/A_{\text{N}}^{hp}(t)$ with high precision, (ii) possible
beam ${}^{3}\text{He}$ breakup, and (iii) operation in a 10 ns bunch spacing
beam. Preliminary results of an analysis discussed here indicate that the
listed problems can be overcome and the helion beam absolute polarization can
be measured by HJET with the required accuracy.

###### keywords:

Hadron Polarimetry; He3 beam at EIC; Analyzing power; Breakup Corrections;

††journal:  The Proceedings of the 24th International Spin Symposium
(SPIN2021)

##  1 Introduction

The physics program at the future Electron Ion Collider (EIC) [1] requires
precision measurement of the proton and helion (${}^{3}\rm{He}$) beam absolute
polarization with systematic uncertainties of about [2]

$\sigma_{P}^{\text{syst}}/P\lesssim 1\%.$ (1)

For high energy, $\sim$100 GeV, polarized proton beams, such an accuracy was
already achieved at the Relativistic Heavy Ion Collider (RHIC). At RHIC, the
absolute vertical proton beam polarization is determined by the Atomic
Polarized Hydrogen Gas Jet Target (HJET) [3]. The vertically directed jet
target polarization, $P_{\text{jet}}\\!=\\!96\\!\pm\\!0.1\%$, is monitored by
a Breit-Rabi polarimeter and is flipped every 5 minutes. A critically
important feature for the HJET polarimeter is a relatively low-density gas jet
target with no walls or windows which allows one to make continuous
measurements in the Coulomb-nuclear interference (CNI) region.

During RHIC run, the stored beams with alternatively polarized bunches are
scattered on the hydrogen jet target and recoil protons are counted in the
left/right symmetric recoil spectrometer detectors at 90 degrees to the beam
direction [4]. The spectrometer is depicted in Fig. 2. Isolation of the
elastic events is explained in Fig. 2.

For elastic $\mathit{pp}$ scattering, the concurrently measured beam and jet
spin correlated asymmetries

$a_{\text{beam}}=\langle A_{\text{N}}\rangle P_{\text{beam}},\quad
a_{\text{jet}}=\langle A_{\text{N}}\rangle P_{\text{jet}},\qquad\to\qquad
P_{\text{beam}}=P_{\text{jet}}\times\left(a_{\text{beam}}/a_{\text{jet}}\right)$
(2)

allow one to determine the beam polarization $P_{\text{beam}}$ without knowing
the analyzing power $A_{\text{N}}$. At HJET, the asymmetries can be measured
as functions of the recoil proton energy
$0.6\\!<\\!T_{R}\\!<\\!10.6\,\text{MeV}$ which was used for experimental
evaluation of analyzing power dependence $A_{\text{N}}(t)$ on momentum
transfer $t\\!=\\!-2m_{p}T_{R}$. A detailed description of the HJET data
analysis, including subtraction of a background and evaluation of systematic
uncertainties, is given in Ref. [4].

In previous RHIC polarized proton Runs 15
($E_{\text{beam}}\\!=\\!100\,\text{GeV}$) and 17 (255 GeV), the beam
polarization of about $P_{\text{beam}}\\!\sim\\!55\%$ was measured with
systematic error of $\sigma^{\text{syst}}_{P}/P\\!\lesssim\\!0.5\%$ and
typical statistical uncertainties of $\sigma_{P}^{\text{stat}}\\!\sim\\!2\%$
per 8 hour RHIC store [4]. Also, the elastic ${pp}$ analyzing power was
precisely determined at both beam energies [5].

Since 2015, HJET has routinely operated in the RHIC ion beams. The recoil
spectrometer performance was found to be very similar to that of a proton beam
and is very stable for a wide range of ion species ($d$, Al, Zr, Ru, Au) and
over a wide range, 3.8 – 100 GeV/n, of the beam energies used [4].

Figure 1:  Schematic view of the HJET recoil spectrometer. Eight detectors
with vertically oriented Si strips are used to measure the recoil proton time,
amplitude, and recoil angle, concurrently for both RHIC beams.

Figure 2:  Correlation between Si strip number (in a detector) and the recoil
kinetic energy $T_{R}$ allows one to isolate elastic events. Since the
background rate is almost independent of the Si strip number (for given
$T_{R}$) [4], the background events can be subtracted from the elastic data.

Here, we investigate a possibility to adapt the HJET method for a precise
measurement of the ${}^{3}\text{He}$ beam polarization at EIC.

##  2 ${}^{3}\rm{He}$ beam polarimetry at EIC

To straightforwardly implement the HJET method (2) for absolute helion
polarimetry at EIC, one should employ a polarized ${}^{3}\rm{He}^{\uparrow}$
target. But, for such a target it may be an issue to satisfy the EIC
requirements, in particular, high polarization, which can be monitored with
$\ll\\!1\%$ accuracy, and consistency with recoil helion measurements in the
CNI region. For example, ${}^{3}\rm{He}$ gas can be polarized up to $\sim$70%
in a cell and, then, it can be injected into the collision chamber producing a
jet-like target. However, confirmed polarization uncertainty for such a
${}^{3}\rm{He}$ jet is $\sigma_{P}/P\\!=\\!3.4\%$ [6]. Some other obviously
anticipated concerns about the method adaption are: (i) the recoil helion
kinematics essentially differs from that of the proton case; (ii) possible
breakup of the beam and/or target may result in uncontrolled backgrounds;
(iii) the EIC bunch spacing (10 ns) will be much shorter compared to that (107
ns) in HJET measurements at RHIC. A possibility to overcome the mentioned
potential problems is being investigated by RHIC Polarimetry Group.

Here, we will consider HJET feasibility to determine the EIC ${}^{3}\rm{He}$
beam polarization with required accuracy (1). For such a measurement, the
$h^{\uparrow}{p}$ and $p^{\uparrow}{h}$ analyzing powers do not cancel in Eq.
(2). Thus, the ratio $A_{\text{N}}^{hp}(t)/A_{\text{N}}^{ph}(t)$ must be
predetermined. Also we will consider the effects of the helion beam breakup
and shorter bunch spacing at EIC.

##  3 The analyzing power

For high energy forward elastic $\mathit{p^{\uparrow}{p}}$ scattering, the
analyzing power structure is theoretically well understood [7, 8, 9].
Neglecting some small, well determined corrections,

$A_{N}(t)=\frac{\sqrt{-t}}{m_{p}}\,\frac{\kappa_{p}-2I_{5}-2R_{5}\,t/t_{c}}{t_{c}/t-2(\rho+\delta_{C})+t/t_{c}},$
(3)

where $\kappa_{p}\\!=\\!\mu_{p}\\!-\\!1\\!=\\!1.793$ is anomalous magnetic
moment of a proton, $\rho$ is forward real to imaginary ratio,
$\delta_{C}\\!\sim\\!0.02$ is Coulomb phase [10, 11], and
$-t_{c}=8\pi\alpha/\sigma_{\text{tot}}\\!\approx\\!0.002\,\text{GeV}^{2}$ is
expressed via total $\mathit{pp}$ cross section. The dominant term,
$\kappa_{p}$ in the numerator is defined by interference of the
electromagnetic spin flip and hadronic non-flip amplitudes. Corrections due to
the hadronic spin flip amplitude, parameterized by
$r_{5}\\!=R_{5}\\!+\\!I_{5}$ [9], are of about 2% [5]. For, elastic
$\mathit{pp}$ scattering, the denominator, which is proportional to the
differential cross section, is precisely known from numerous experimental
studies, but this is not the case for proton-helion scattering.

However, the $\mathit{hp}$ cross section uncertainties cancel in the ratio of
the $h^{\uparrow}{p}$ and $p^{\uparrow}{h}$ analyzing powers:

${\cal
R}=\frac{A_{\text{N}}^{hp}(t)}{A_{\text{N}}^{ph}(t)}=\frac{\kappa_{h}-2I_{5}^{hp}-2R_{5}^{hp}\,t/t_{c}}{\kappa_{p}-2I_{5}^{ph}-2R_{5}^{ph}\,t/t_{c}},$
(4)

where $k_{h}\\!=\\!\mu_{h}/2\\!-\\!1/3\\!=\\!-1.398$ is derived from the
magnetic moment of a helion [12]. Although small, the hadronic spin-flip
amplitude should be considered when achieving the required (1) accuracy of the
beam polarization measurements at EIC.

Using proton-proton $r_{5}^{pp}$, precisely determined at HJET [5], one can
evaluate $r_{5}^{ph}$ and $r_{5}^{hp}$ with sufficient accuracy. In Ref. [13],
it was shown that the $p^{\uparrow}\rm{A}$ value of $r_{5}$ can be well
approximated by the $p{}^{\uparrow}{}p$ one, $r_{5}^{ph}\\!=\\!r_{5}^{pp}$.
Since ${}^{3}\rm{He}$ is carried mostly by the neutron,
$r_{5}^{hp}\\!\approx\\!r_{5}^{pp}/3$ [14]. Assuming $|r_{5}^{pp}|\lesssim
0.02$, model dependent uncertainties in such extrapolation of $r_{5}^{pp}$ to
$r_{5}^{ph}$ and $r_{5}^{hp}$ can be evaluated [15] as

$r_{5}^{ph}=r_{5}^{pp}\pm 0.001\pm 0.001\>\\!i,\qquad
r_{5}^{hp}=0.27r_{5}^{pp}\pm 0.001\pm 0.001\>\\!i.$ (5)

##  4 The helion beam breakup

Since only recoil protons are detected at HJET, possible breakup of a beam
helion, $h\to d+p$ can contaminate the elastic ${hp}$ scattering data.
Generally, the total breakup cross section can be much larger than the elastic
one, e.g. in deuteron-proton scattering [16]. However, the differential
breakup cross section must vanish if $t\\!\to\\!0$, unless the beam particle
can spontaneously decay. Therefore, for forward scattering events,
$|t|\\!<\\!0.02\,\text{GeV}^{2}$ ($T_{R}\\!<\\!10\,\text{MeV}$), which can be
detected at HJET, the breakup (or quasi-elastic, qel) fraction,
$f\\!=\\!\sigma_{\text{qel}}\,/\,\sigma_{\text{el}}\big{|}_{\text{HJET}}$, is
expected to be small. In a recent evaluation of this value for a beam of Au,
it was found that $f_{\text{Au}}\\!\lesssim 1\%$.

To evaluate the analyzing power ratio $\cal{R}$ dependence on the breakup, one
should substitute the (unpolarized) hadronic elastic amplitude by a sum of
elastic and quasi-elastic amplitudes
$\phi_{+}(t)\to\phi_{+}(t)+\sum{\phi_{\text{qel}}(t,\Delta)}=\phi_{+}(t)\times[1+\tilde{\omega}(t)]$,
where $\Delta=(M_{X}^{2}-m_{h}^{2})/2m_{h}$ is the breakup mass excess. Using
Run 16 deuteron beam (10–100 GeV/n) measurements at HJET, it was calculated
that $\left|\tilde{\omega}(t)\right|\\!<\\!0.05$ (upper limit) [17] for the
helion beam. Thus,

${\cal R}_{\text{el}}\to{\cal
R}_{\text{el+qel}}=\frac{\kappa_{h}\times[1+\tilde{\omega}(t)]-2I_{5}^{hp}-2R_{5}^{hp}\,t/t_{c}}{\kappa_{p}\times[1+\tilde{\omega}(t)]-2I_{5}^{ph}-2R_{5}^{ph}\,t/t_{c}}\approx{\cal
R}_{\text{el}}.$ (6)

Here we assume that electromagnetic amplitudes are about the same for elastic
and quasi-elastic scattering (if $t\to 0$). Also, we neglected the breakup
corrections to hadronic spin flip amplitudes since $|r_{5}|\\!\sim\\!0.02$ is
small. In the approximation used, the breakup corrections cancel in the ratio
(6).

##  5 The EIC bunch spacing

Figure 3:  Left: a typical detected signal time/amplitude distribution in a Si
strip. Black lines specify the time/amplitude area corresponding to the recoil
protons. Red contours isolate the elastic $\mathit{pp}$ events. Right: the
same distribution after reconstruction of kinetic energy of the punch-trough
protons.

Figure 4:  Separation of the stopped and punch-trough recoil protons based on
the correlation of the signal waveform parameters $n$ and $A$.

For HJET measurements at RHIC, a typical time/amplitude distribution in a Si
strip is shown in Fig. 4 (left). The data rate is strongly dominated by
prompts, characterized by low deposited energy and small time of flight (TOF).
The recoil protons with energy above $7.8\,\text{MeV}$ punch through a Si
strip and, thus, only part of the kinetic energy is detected. Such events as
well as prompts might be a serious problem for beam polarization measurement
at EIC. Due to the reduced, 10 ns, bunch spacing at EIC, the elastic
$\mathrm{pp}$ signals might be mismatched with punch-trough protons and
background events (in particular, prompts) from other bunches. This is
anticipated to be a common problem for proton and helion beam polarimetry at
EIC.

Assuming that prompts are fast particles penetrating the Si strip, two
possible solutions of the problem were considered.

Since the recoil proton signal wave form shape parametrization, $w(t)\propto
At^{n}\,\exp{(-t/\tau)}$, used in the data analysis, depends on kinetic energy
$T_{R}$ of the proton [4] (see Fig. 4), $T_{R}$ for punch-through protons can
be reconstructed from the measured amplitude as shown in Fig. 4 (right). Such
a transformation significantly reduces mismatching signals from different
bunches.

Using RHIC Run 17 (255 GeV protons) data, HJET performance for 8.9 ns bunch
spacing was emulated [18]. For that, the measured time of each event was
shifted $t\to t+k\tau/12$, where $k$ is randomly chosen from
$k\in(-1.5,-0,5,0.5,1.5)$ and $\tau\\!=\\!106.6\,\text{ns}$ is the bunch
spacing in Run 17. This transformation approximates the proposed bunch
splitting into four at EIC. In addition, every event was triplicated by the
time shifts $\pm\tau/3$.

Such emulated data was processed using regular HJET data analysis software. It
was found that, for $T_{R}\\!>\\!2\,\text{MeV}$, the EIC bunch spacing will
not alter, within $|\delta{}P/P|\\!\lesssim\\!0.3\%$ uncertainty, the measured
beam polarization.

Anther approach to solve the problem is to use double-layer Si detectors to
veto prompts [18]. One of the HJET detectors was replaced with a double layer
prototype. The measurement was done using a beam of Au, which was available at
that time. The efficiency of prompt vetoing appeared to be only about
$\sim$50%. For prompts which did not trigger the second layer, it was found
that time of flight is about the same as for a photon and the signal wave form
shape is similar to that of stopped recoil protons. The result obtained
disagrees with an initial assumption about the prompts. Currently we have no
satisfactory numerical model for the prompt signals.

Nonetheless, we are positive about HJET feasibility to operate in the EIC 10
ns bunch spacing beam (if the recoil proton energy will be limited to
$T_{R}\\!>\\!2\,\text{MeV}$ in the data analysis).

##  6 Summary

Based on the analysis above, we should conclude that the absolute
${}^{3}\rm{He}$ beam polarization at EIC can be determined with the required
precision (1). For that, we suggest measuring the polarization as a function
of the recoil proton energy:

$P_{m}(T_{R})=P_{\text{jet}}\frac{a_{\text{beam}}^{(h)}(T_{R})}{a_{\text{jet}}^{(p)}(T_{R})}\times\frac{\kappa_{p}-m_{p}^{2}/m_{h}E_{\text{beam}}-2I_{5}-2R_{5}\,T_{R}/T_{c}}{\kappa_{h}-m_{h}/E_{\text{beam}}-0.54I_{5}-0.54R_{5}\,T_{R}/T_{c}}.$
(7)

Here, $\kappa_{p}=1.793$, $\kappa_{h}=-1.398$,
$T_{c}=4\pi\alpha/m_{p}\sigma_{\text{tot}}^{hp}\approx 0.7\,\text{MeV}$, and
the proton-proton $r_{5}=R_{5}+iI_{5}$ is predetermined for the same beam
energy $E_{\text{beam}}$ (in GeV/n units). Eq. (7) also includes an energy
dependent correction to the electromagnetic amplitude [19], which was not
discussed above. Extrapolating the $P_{m}(T_{R})$ dependence to $T_{R}\to 0$,
the helion beam polarization can be found:

$\\!\\!P_{\text{beam}}=P_{m}(0)\times\left[1\\!\pm\\!0.006_{\text{syst}}\\!\pm\\!0.005_{r_{5}}\\!\pm\\!0.002_{\text{mod}}\right].$
(8)

The specified uncertainties are estimates of systematic errors in HJET
measurements (syst), in the value of $r_{5}$, and in the model dependent
derivation of Eq. (7).

It should be pointed out that HJET (with possible improvements if required) is
being considered for measurement of the proton beam absolute polarization at
EIC. Here, we advocate that exactly that polarimeter also has the ability to
precisely measure helion beam polarization (if proton-proton $r_{5}$ for the
beam energy used is already known).

However, development of alternative methods of measuring the helion beam
polarization, (e.g. using polarized ${}^{3}\rm{He}$ target) is very important
to obtain a confident result. Also, it may be crucially helpful for the data
analysis to extend the measurement to lower kinetic energies
$T_{R}<2\,\text{MeV}$.

## References

  * [1] A. Accardi, L. Albacete, J., M. Anselmino et al., _Electron Ion Collider: The Next QCD Frontier_ , _Eur. Phys. J. A_ 52 (2016) 268 [1212.1701].
  * [2] R. Abdul Khalek et al., _Science Requirements and Detector Concepts for the Electron-Ion Collider: EIC Yellow Report_ , _Nucl. Phys. A_ 1026 (2022) 122447 [2103.05419].
  * [3] A. Zelenski, A. Bravar, D. Graham, W. Haeberli, S. Kokhanovski, Y. Makdisi et al., _Absolute polarized H-jet polarimeter development, for RHIC_ , _Nucl. Instrum. Meth. A_ 536 (2005) 248.
  * [4] A.A. Poblaguev, A. Zelenski, G. Atoian, Y. Makdisi and J. Ritter, _Systematic error analysis in the absolute hydrogen gas jet polarimeter at RHIC_ , _Nucl. Instrum. Meth. A_ 976 (2020) 164261 [2006.08393].
  * [5] A.A. Poblaguev, A. Zelenski, E. Aschenauer, G. Atoian, K.O. Eyser, H. Huang et al., _Precision Small Scattering Angle Measurements of Elastic Proton-Proton Single and Double Spin Analyzing Powers at the RHIC Hydrogen Jet Polarimeter_ , _Phys. Rev. Lett._ 123 (2019) 162001 [1909.11135].
  * [6] D. DeSchepper et al., _The HERMES polarized 3He internal gas target_, _Nucl. Instrum. Meth. A_ 419 (1998) 16.
  * [7] B.Z. Kopeliovich and L.I. Lapidus, _On the necessity of polarization experiments in colliding $pp$ and $\overline{p}p$ beams_, _Yad. Fiz._ 19 (1974) 218.
  * [8] N.H. Buttimore, E. Gotsman and E. Leader, _Spin-dependent phenomena induced by electromagnetic-hadronic interference at high energies_ , _Phys. Rev. D_ 18 (1978) 694.
  * [9] N.H. Buttimore, B.Z. Kopeliovich, E. Leader, J. Soffer and T.L. Trueman, _The spin dependence of high-energy proton scattering_ , _Phys. Rev. D_ 59 (1999) 114010 [hep-ph/9901339].
  * [10] R. Cahn, _Coulombic-Hadronic Interference in an Eikonal Model_ , _Z. Phys. C_ 15 (1982) 253.
  * [11] B.Z. Kopeliovich and A.V. Tarasov, _The Coulomb phase revisited_ , _Phys. Lett. B_ 497 (2001) 44 [hep-ph/0010062].
  * [12] N.H. Buttimore, _Forward helion scattering and neutron polarization_ , _AIP Conf. Proc._ 1105 (2009) 189.
  * [13] B.Z. Kopeliovich and T.L. Trueman, _Polarized proton nucleus scattering_ , _Phys. Rev. D_ 64 (2001) 034004 [hep-ph/0012091].
  * [14] N.H. Buttimore, E. Leader and T.L. Trueman, _An absolute polarimeter for high-energy protons_ , _Phys. Rev. D_ 64 (2001) 094021 [hep-ph/0107013].
  * [15] A.A. Poblaguev, _Feasibility study for precisely measuring the EIC 3He beam polarization with the Polarized Atomic Hydrogen Gas Jet Target polarimeter at RHIC_, _Phys. Rev. C_ 106 (2022) 065202 [2207.09420].
  * [16] V.V. Glagolev, G. Martinska, J. Musinsky, N.M. Piskunov and J. Urban, _The charge exchange reaction $dp\to(pp)n$_, _Cent. Eur. J. Phys._ 6 (2008) 781.
  * [17] A.A. Poblaguev, _Breakup corrections to the 3He beam polarization measurements at the future BNL Electron Ion Collider_, _Phys. Rev. C_ 106 (2022) 065203 [2207.06999].
  * [18] A.A. Poblaguev, A. Zelenski and G. Atoian, _The prospects on the absolute proton beam polarimetry at EIC_ , _PoS_ PSTP2019 (2020) 007.
  * [19] A.A. Poblaguev, _Corrections to the Elastic Proton-Proton Analyzing Power Parametrization at High Energies_ , _Phys. Rev. D_ 100 (2019) 116017 [1910.02563].