# Simulation of Beam-Beam Effects and Tevatron Experience

A. Valishev<EMAIL_ADDRESS>Yu. Alexahin  V. Lebedev  Fermilab, Batavia, IL
60510, USA  D. Shatilov  BINP SB RAS, Novosibirsk, 630090, Russia

###### Abstract

Effects of electromagnetic interactions of colliding bunches in the Tevatron
have a variety of manifestations in beam dynamics presenting vast
opportunities for development of simulation models and tools. In this paper
the computer code for simulation of weak-strong beam-beam effects in hadron
colliders is described. We report the recent operational experience, explain
major effects limiting the collider performance and compare results of
observations and measurements with simulations.

###### pacs:

29.27.Bd

##  I Introduction

Peak luminosity of the Tevatron reached 3.5$\times 10^{32}$ cm-2s-1 which
exceeds the original Run II goal of 2.7$\times 10^{32}$. This achievement
became possible due to numerous upgrades in the antiproton source, injector
chain, and in the Tevatron collider itself. The most notable rise of
luminosity came from the commissioning of electron cooling in the recycler
ring and advances in the antiproton accumulation rate. Starting from 2007, the
intensity and brightness of antiprotons delivered to the collider greatly
enhanced the importance of beam-beam effects. Several configurational and
operational improvements in the Tevatron have been planned and implemented in
order to alleviate these effects and allow stable running at high peak
luminosities.

Since the publication of paper Shiltsev et al. (2005) that gave a detailed
summary of beam dynamics issues related to beam-beam effects, the peak
luminosity of Tevatron experienced almost a tree-fold increase. In the present
article we provide an updated view based on the recent years operation
(Section II).

Development of a comprehensive computer simulation of beam-beam effects in the
Tevatron started in 1999. This simulation proved to be a useful tool for
understanding existing limitations and finding ways to mitigate them. In
Section IV the main features of the code LIFETRAC are described. In Sections
V-VI we summarize our experience with simulations of beam-beam effects in the
Tevatron, and cross-check the simulation results against various experimental
data and analytical models. We also correlate the most notable changes in the
machine performance to changes of configuration and beam conditions, and
support the explanations with simulations.

##  II  Overview of Beam-Beam Effects

A detailed description of the Tevatron collider Run II is available in other
sources Run . Here only the essential features important for understanding of
beam dynamics are provided.

Tevatron is a superconducting proton-antiproton collider ring in which beams
of the two species collide at the center of mass energy of $2\times 0.98$ TeV
at two experiments. Each beam consists of 36 bunches grouped in 3 trains of 12
with 396 ns bunch spacing and 2.6 $\mu$s abort gaps between the trains. The
beams share a common vacuum chamber with both beams moving along helical
trajectories formed by electrostatic separators. Before the high energy
physics collisions have been initiated, the proton and antiproton beams can be
moved longitudinally with respect to each other, which is referred to as
cogging. This configuration allows for 72 interactions per bunch each turn
with the total number of collision points in the ring equal to 138. The total
number of collision points is determined by the symmetry of bunch filling
pattern.

At the peak performance Tevatron operates with $N_{p}=2.8\cdot 10^{11}$
protons and $N_{a}=0.9\cdot 10^{11}$ antiprotons per bunch. The normalized
transverse 95% beam emittances are $\varepsilon_{p}=18\cdot 10^{-6}$m for
protons and $\varepsilon_{a}=7\cdot 10^{-6}$m for antiprotons. Proton and
antiproton bunch length at the beginning of a high energy physics (HEP) store
is 52 cm and 48 cm, respectively. Parameters of the beams are mostly
determined by the upstream machines.

The value of $\beta$-function at the main collision points ($\beta^{*}$) is
0.28 m. Betatron tunes are $Q_{x}=20.584$, $Q_{y}=20.587$ for protons and
$Q_{x}=20.575$, $Q_{y}=20.569$ for antiprotons.

A typical collider fill cycle is shown in Figure 1. First, proton bunches are
injected one at a time on the central orbit. After that, the helix is opened
and antiproton bunches are injected in batches of four. This process is
accompanied by longitudinal cogging after each 3 transfers. Then the beams are
accelerated to the top energy (85 s) and the machine optics is changed to
collision configuration in 25 steps over 120 seconds (low-beta squeeze). The
last two stages include initiating collisions at the two main interaction
points (IP) and removing halo by moving in the collimators.

Figure 1:  Collider fill cycle for store 5989.

It has been shown in machine studies that beam losses up the ramp and through
the low-beta squeeze are mainly caused by beam-beam effects Shiltsev et al.
(2005). In the HEP mode, the beam-beam induced emittance growth and particle
losses contribute to the faster luminosity decay. Figure 2 summarizes the
observed losses of luminosity during different stages of the collider cycle.

Figure 2:  Luminosity loss budget over a 3 year period. The labels mark: 1.
Commissioning of electron cooling. 2. Installation of extra separators and new
collision helix. 3\. Antiproton accumulation rate. 4. Correction of second-
order chromaticity. 5. Implementation of antiproton emittance blowup.

###  II.1 Beam-beam effects at injection

During injection the long range (also referred to as parasitic) beam-beam
effects cause proton losses (currently 5 to 10%). At the same time the
antiproton life time is very good and only a fraction of a per cent are lost.
Observations show that mainly off momentum particles are lost (Fig. 3) and the
betatron tune chromaticity $C=dQ/d\delta$, where $\delta=\Delta p/p$ is the
relative momentum deviation, has a remarkable effect. Early in Run II the
chromaticity had to be kept higher than 8 units in order to maintain coherent
stability of the intense proton beam, but after several improvements aimed at
reduction of the machine impedance the chromaticity is about 3 units Ivanov et
al. (2003, 2005); Ranjbar and Ivanov (2008). Figure 3 shows an interesting
feature in the behavior of two adjacent proton bunches (no. 20 and 21). Spikes
in the measured values are instrumental effects labeling the time when the
beams are cogged. Before the first cogging the bunches have approximately
equal life time. After the first cogging bunch 20 exhibits faster decay, and
bunch 21 after the second. Analysis of the collision patterns for these
bunches allowed to pinpoint a particular collision point responsible for the
life time degradation. The new injection helix has been implemented late in
2007 which improved the proton life time Moore et al. (2005); Alexahin (2007).

Figure 3:  Intensity and length of proton bunches no. 20 and 21 during
injection of antiprotons.

###  II.2 Low-beta squeeze

During the low-beta squeeze two significant changes occur - the $\beta^{*}$
value is being gradually decreased from $\sim$1.5 m to 0.28 m (hence the name
squeeze) and the helical orbits change their shape and polarity from injection
to collision configuration. The latter poses a serious limitation since the
beams separation at several long range collision points briefly decreases from
5-6$\sigma$ to $\sim$2$\sigma$. At this moment a sharp spike in losses is
observed.

Another important operational concern was the tight aperture limitation in one
of the two final focus regions (CDF). With dynamically changing orbit and
lattice parameters the local losses were often high enough to cause a quench
of the superconducting magnets even though the total amount of beam loss is
small ($\sim$ 1%). The aperture restriction has been located and fixed in
October of 2008.

Besides orbit stability two other factors were found to be important in
maintaining low losses through the squeeze: antiproton beam brightness and
betatron coupling. Figure 4 shows the dependence of proton losses on the
antiproton beam brightness. Large amount of stores lost in this stage of the
cycle caused by increase of the antiproton beam brightness after the 2007
shutdown demanded the commissioning of the antiproton emittance control system
Tan .

Figure 4:  Proton losses in low-beta squeeze vs. antiproton beam brighness
$36\cdot N_{a}/\varepsilon_{a}$.

###  II.3 High energy physics

After the beams are brought into collisions at the main IPs, there are two
head-on and 70 long range collision points per bunch. Beam-beam effects caused
by these interactions lead to emittance growth and particle losses in both
beams.

During the running prior to the 2006 shutdown the beam-beam effects at HEP
mostly affected antiprotons. The long range collision points nearest to the
main IPs were determined to be the leading cause for poor life time.
Additional electrostatic separators were installed in order to increase the
separation at these IPs from 5.4 to 6$\sigma$ Alexahin (2007). Also, the
betatron tune chromaticity was decreased from 20 to 10 units. Since then, the
antiproton life time is dominated by losses due to luminosity and no emittance
growth is observed provided that the betatron tune working point is well
controlled.

Electron cooling of antiprotons in the Recycler and increased antiproton
staching rate drastically changed the situation for protons. Figure 5 shows
the evolution of total head-on beam-beam tune shift $\xi$ for protons and
antiprotons. Note that prior to the 2006 shutdown the proton $\xi$ was well
under 0.01 and big boost occurred in 2007 when both beam-beam parameters
became essentially equal. It was then when beam-beam related losses and
emittance blowup started to be observed in protons.

Figure 5:  Head-on beam-beam tune shift vs. time.

Our analysis showed that deterioration of the proton life time was caused by a
decrease of the dynamical aperture for off-momentum particles due to head-on
collisions (see Sec. VI). It was discovered that the Tevatron optics had large
chromatic perturbations, e.g. the value of $\beta^{*}$ for off-momentum
particles could differ from that of the reference particle by as much as 20%.
Also, the high value of second order betatron tune chromaticity
$d^{2}Q/d\delta^{2}$ generated a tune spread of $\sim$0.002. A rearrangement
of sextupoles in order to correct the second order chromaticity was planned
and implemented before the 2007 shutdown Valishev et al. (2007). Figure 6
demonstrates the effect of this modification on integrated luminosity. Since
the dependence of luminosity on time is very well fitted by a
$L_{0}/(1+t/\tau)$ function, one can normalize the luminosity integral for a
given store to a fixed length $T_{0}$ by using the expression $L_{0}\tau\cdot
ln(1+T_{0}/\tau)$ Shiltsev and McCrory (2005). Here $L_{0}$ is the initial
luminosity, and $\tau$ is the luminosity life time. One can see that after the
modification the saturation at luminosities above $2.6\times 10^{32}$ was
mitigated and the average luminosity delivered to experiments increased by
$\sim$ 10%.

Figure 6:  Luminosity integral normalized by 24 h vs. initial luminosity. Blue
points and curve - before second order chromaticity correction, red - after
correction. Black line represents the ultimate integral for the given beam
parameters in the absence of beam-beam effects (see Sec. III).

Another step in the proton $\xi$ happened after the 2007 shutdown when the
transverse antiproton emittance decreased because of improvements in injection
matching. The total attained head-on beam-beam tune shift for protons exceeded
that of antiprotons and reached 0.028. This led to high sensitivity of the
proton life time to small variations of the betatron tunes, and to severe
background conditions for the experiments. The reason is believed to be the
large betatron tune spread generated by collisions of largely different size
bunches Syphers . Indeed, at times the antiproton emittance was a factor of 5
to 6 smaller than the proton emittance.

To decrease the proton to antiproton emittance ratio a system has been
commissioned which increases the antiproton emittance after the top energy is
reached by applying wide band noise to a directional strip line (line 5 in
Fig. 2) Tan . Currently, the optimal emittance ratio is $\sim$3.

Since the majority of our efforts was targeting beam-beam effects in HEP mode,
we concentrate on this topic in the remaining part of this paper. Discussion
of long range effects at injection and coherent effects Alexahin (2005) is
left out of the scope of this report.

##  III  Store analysis package

Beam-beam interaction is not the single strongest effect determining evolution
of beam parameters at collisions. There are many sources of diffusion causing
emittance growth and particle losses, including but not limited to intrabeam
scattering, noise of accelerating RF voltage, and scattering on residual gas.
Parameters of these mechanisms were measured in beam studies, and then a model
was built in which the equations of diffusion and other processes are solved
numerically Lebedev (2003). This model is able to predict evolution of the
beam parameters in the case of weak beam-beam effects. When these effects are
not small, it provides a reference for evaluation of their strength. We use
this approach on a store-by-store basis to monitor the machine performance in
real time sto  because such calculations are very fast compared to a full
numerical beam-beam simulation. Figure 7 presents an example comparison of
evolution of beam parameters in an actual high luminosity store to
calculations. Note that there is no transverse emittance blow up in both
beams, and the emittance growth is determined by processes other than beam-
beam interaction. The same is true for antiproton intensity and bunch length.
The most pronounced difference between the observation and the model is seen
in the proton intensity. Beam-beam effects cause proton life time degradation
during the initial 2-3 hours of the store until the proton beam-beam tune
shift drops from 0.02 to 0.015. The corresponding loss of luminosity integral
is about 5%.

Figure 7:  Observed beam parameters in store 6683 compared to store analysis
calculation (model). $L_{0}=3.5\cdot 10^{32}$ cm-2 s-1. a) Single bunch
Luminosity and Luminosity integral. b) Intensity of proton bunch no. 6 and of
antiproton bunch colliding with it (no. 13). c) Bunch lengths. d) Horizontal
95% normalized bunch emittances.

##  IV  Weak-strong code LIFETRAC

Initially, the beam-beam code LIFETRAC was developed for simulation of the
equilibrium distribution of the particles in circular electron-positron
colliders Shatilov (1996). In 1999 the new features have been implemented,
which allowed simulating non-equilibrium distributions, for example proton
beams. In this case the goal of simulations is not to obtain the equilibrium
distribution but to observe how the initial distribution is changing with
time. Number of simulated particles can vary in the range of $10^{3}$ to
$10^{6}$, usually it is set to $(5\div 10)\cdot 10^{3}$. The tracking time is
divided into “steps”, typically $10^{3}\div 10^{5}$ turns each. The statistics
obtained during the tracking (1D histograms, 2D density in the space of
normalized betatron amplitudes, luminosity, beam sizes and emittances) is
averaged over all particles and all turns for each step. Thus, a sequence of
frames representing evolution of the initial distribution is obtained.

Another important quantity characterizing the beam dynamics is the intensity
life time. It is calculated by placing an aperture restriction in the machine
and counting particles reaching the limit. The initial and final coordinates
of the lost particle are saved. This information is valuable for analysis of
various beam dynamics features.

The initial 6D distribution of macroparticles can be either Gaussian (by
default), or read from a separate text file. Besides, the macroparticles may
have different “weights”. This allows representing the beam tails more
reliably with limited number of particles. Usually we simulate the Gaussian
distribution with weights: particles initially located in the core region have
larger weight while the “tail” particles with smaller weight are more
numerous.

In the present bunch pattern (3 trains of 12 bunches) there are two main IPs
and 70 long range collision points. When performing transformation through a
main IP, the “strong” bunch is divided into slices longitudinally. The higher
are the orders of significant betatron resonances which are supposed to make
effect on the distribution, the greater must be the number of slices. In our
simulations 12 slices were used in the main IPs where beta-functions are
approximately equal to the bunch length and only one slice in long range
collision points where beta-functions are much greater and one can neglect the
betatron phase advance on the bunch length.

The transverse density distributions within “strong” slices are bi-Gaussian,
allowing to apply the well-known formulae Hirata et al. (1992) for 6D
symplectic beam-beam kick. However, a simple modification allowed simulating
non-Gaussian strong bunches. Namely, the strong bunch is represented as a
superposition of a few (up to three) Gaussian distributions with different
betatron emittances. The kicks from all these “harmonics” are summarized
additively. The calculation time is increased somehow (not very significantly)
but the transformation remains 6D symplectic.

###  IV.1 Tevatron optics

The parasitic collisions in Tevatron play a significant role in the beam
dynamics. In order to account their contribution correctly an accurate
knowledge of the machine lattice of the whole ring with all distortions, beta
beatings, coupling, etc. is required. This necessitated the construction of a
realistic model of the machine lattice based on beam measurements. The most
effective method proved to be the orbit responce matrix analysis Sajaev et al.
(2005); Valishev et al. (2006); Lebedev et al. (2006).

The model lattice is built in the optics code OptiM Lebedev . Both OptiM and
LIFETRAC treat betatron coupling using the same coupled beta-functions
formalism Lebedev and Bogacz . This allows the linear transport matrix between
any two points to be easily derived from the coupled lattice functions and
phase advances.

A set of scripts has been created enabling fast creation of input files for
the beam-beam simulation. These programs automate calculation of azimuthal
positions of interaction points for the chosen bunch and extraction of the
optics parameters. In the end, the machine optics is represented by a set of
6D linear maps between the interaction points.

It was estimated that resonances generated by known Tevatron nonlinearities,
such as the final focus triplets and lattice sextupoles, are much weaker than
those driven by beam-beam collisions at the present betatron tune working
point. Hence, inclusion of nonlinear lattice elements into the simulation was
deemed unnecessary.

###  IV.2 Chromaticity

Although linear optics is used for the machine lattice model, there are two
nonlinear lattice effects which are considered to be significant for beam-beam
behaviour and were included into simulations. These are the chromaticities of
beta-functions excited in the Main IPs and chromaticities of the betatron
tunes. In the Hamiltonian theory the chromaticity of beta-functions does not
come from energy-dependent focusing strength of quads (as one would
intuitively expect) but from drift spaces where the transverse momentum is
large (low-beta regions). The symplectic transformations for that are:

$\displaystyle X$ $\displaystyle=$ $\displaystyle X-L\cdot
X^{\prime}\cdot\frac{\Delta p}{p}$ $\displaystyle Y$ $\displaystyle=$
$\displaystyle Y-L\cdot Y^{\prime}\cdot\frac{\Delta p}{p}$ $\displaystyle Z$
$\displaystyle=$ $\displaystyle Z-L\cdot(X^{\prime 2}+Y^{\prime 2})/2$

where $X$, $Y$, and $Z$ are the particle coordinates, and $L$ is the
“chromatic drift” length. Then, it is necessary to adjust the betatron tune
chromaticities which are also affected by “chromatic drift”. For that, an
artificial element (insertion) is used with the following Hamiltonian:

$H=I_{x}\cdot(2\pi Q_{x}+C_{x}\frac{\Delta p}{p})+I_{y}\cdot(2\pi
Q_{y}+C_{y}\frac{\Delta p}{p}),$

where $I_{x}$ and $I_{y}$ are the action variables, $Q_{x}$ and $Q_{y}$ are
the betatron tunes, $C_{x}$ and $C_{y}$ are the [additions to the]
chromaticities of betatron tunes.

###  IV.3 Diffusion and Noise

Diffusion and noise are simulated by a single random kick applied to the
macroparticles once per turn. Strength of the kick on different coordinates is
given by a symmetrical matrix representing correlations between Gaussian
noises. In the Tevatron, the diffusion is rather slow in terms of the computer
simulation – the characteristic time for the emittance change is around an
hour, or $\sim 10^{8}$ turns. In order to match the diffusion and the computer
capabilities, the noise was artificially increased by three orders of
magnitude.

We justify this approach below. In contrast to the electron-positron colliders
there is no damping in hadron colliders. As the result, during the store time
an effect of beam-beam interaction on the emittance growth needs to be
minimized and made small relative to other diffusion mechanisms such as the
intra-beam scattering (IBS), scattering on the residual gas, and diffusion due
to RF phase noise. We will call these the extrinsic diffusion to distinguish
from the diffusion excited by beam-beam effects. For the present Tevatron
parameters the extrinsic diffusion sets the luminosity lifetime to be about 10
hours at the beginning of the store. IBS dominates both transverse and
longitudinal diffusions in the case of protons while its relative effect is
significantly smaller for antiprotons because of $\sim$5 times smaller
intensity.

Table 1 summarizes lifetimes for major beam parameters obtained with diffusion
model Lebedev and Burov (2004) for a typical Tevatron store with the
luminosity of $9\times 10^{31}cm^{-2}s^{-1}$. There are many parameters in
Tevatron which are beyond our control and therefore each store is different.
For good stores, the beam-beam effects make comparatively small contribution
to the emittance growth yielding luminosity lifetime in the range of 7-8 hours
and 10-15% loss in the luminosity integral. The planned threefold increase in
antiproton intensity will amplify the beam-beam effects. That, if not
addressed, can cause unacceptably large background in detectors and reduce
integrated luminosity. In this paper we discuss the results of numerical
simulations aimed to understand major factors contributing to the beam-beam
interaction and possible ways of their mitigation.

Parameter (lifetime, hour) | Protons | Antiprotons
---|---|---
Luminosity | 9.6 | 9.6
Transverse emittance, | -17 / -18 | -52 / -46
$(d\epsilon/dt)/\epsilon$ [hor./vert.] |  |
Longitudinal emittance | -8 | -26
Intensity | 26 | 155
Table 1: Lifetimes for major beam parameters obtained with diffusion model.

Under the real conditions at Tevatron the emittance growth rate is small and
exact simulations of beam-beam effects would require tracking for billions of
turns. That is well beyond capabilities of present computers. Fortunately, the
extrinsic diffusion is large enough in comparison with beam-beam diffusion
which resuls in the loss of phase correlation after about 50,000 turns.

The external noise plays important role in particle dynamics: it provides
particle transport in the regions of phase space which are free from resonance
islands.

To make this transport faster we can artificially increase the noise level
assuming that its effect scales as noise power multiplied by number of turns.
If we choose it so that the noise alone gives 10% emittance growth in $10^{6}$
turns (we use this level as the reference) then this number of turns of
simulation will correspond to $\sim 5$ h of time in the Tevatron.

To verify this approach we studied the effect of the noise level on luminosity
using reconstructed optics.

Fig.8 presents results of the tune scan along the main diagonal with the
reference noise level and without noise. The effect of noise on luminosity
corresponds to its level with exception for the point $Q_{y}=0.575$ where it
was larger due to some cooperation with strong 5th order resonances.

Figure 8:  Ratio of luminosity after a fixed time ($t=2\cdot 10^{6}$ turns) to
the initial luminosity vs. betatron tune. Circles - $D_{noise}=0$, diamonds -
the emittance change due to extrinsic diffusion after $t$ is 20%.

To study this cooperation in more detail we performed tracking at this working
point with different noise levels. Fig.9 shows the luminosity reduction in
$2\cdot 10^{6}$ turns (diamonds) and a fit made using just 3 points, with
relative noise level 0.5, 1 and 2.

The fit works fine for higher noise level, but predicts somewhat faster
luminosity decay in absence of noise than actually observed in tracking. This
means that there are regions in the phase space which particles cannot pass
(within the tracking time) without assistance from the external noise so that
the simple rule $D_{total}=D_{res}+D_{noise}$ does not apply. However, such
“blank spaces” may contain isolated resonance islands which would show up on a
longer time scale with the real level of external noise.

Figure 9:  Ratio of luminosities vs. noise level.

The applicablity of this rule at the reference noise level testifies that
(with the chosen number of turns) no such “blank spaces” were left so we get
more reliable predictions.

###  IV.4 Program features

Since the beam-beam code uses the “weak-strong” model, it can be very
efficiently parallelized. Each processor tracks its own set of particles and
the nodes need to communicate very rarely (at the end of each step), just to
gather the obtained statistics. Hence, the productivity grows almost linearly
with the number of nodes.

There are also two auxiliary GUI codes. The first one automates production of
the LIFETRAC input files for different bunches from the OptiM machine lattice
files. The second one is dedicated for browsing the LIFETRAC output files and
presenting the simulation results in a text and graphical (histogram) form.

We have validated the code using available experimental data. As an example,
Figs. 10 and 11 show a good reproduction of the two distinct effects in bunch
to bunch differences caused by beam-beam effects: variation of vertical bunch
centroid position due to long range dipole kicks, and variation of transverse
emittance blowup caused by difference in tunes and chromaticities.

Figure 10:  Bunch by bunch antiproton vertical orbit. Figure 11:  Bunch by
bunch antiproton emittance growth. Measured in store 3554 (red) and simulated
with lifetrac (blue).

The numerical simulation was used to justify the decrease of antiproton
betatron tune chromaticity, reduction of the $\beta^{*}$ from 0.35 m to 0.28 m
(both in 2005). Importance of separation at the long range collision points
nearest to the main IPs was also demonstrated.

Planning for the increase in amount of antiprotons available to the collider,
we identified the large chromaticity of $\beta^{*}$ as a possible source of
the proton life time deterioration. Figure 12 shows the beam-beam induced
proton life time for different values of $\xi$, and demonstrates the positive
effect of corrected chromatic $\beta^{*}$.

Figure 12:  Proton intensity evolution for different values of beam-beam
parameter per IP.

Simulations revealed an interesting feature in the behavior of the proton
bunch length at high values of $\xi$ \- the so-called bunch shaving, when the
bunch length starts to decrease after initiating head-on collisions instead of
steady growth predicted by the diffusion model (Fig. 13). This behavior was
observed multiple times during HEP stores in 2007, being especially pronounced
when the vertical proton betatron tune was set too high.

Figure 13:  Effect of corrected second order chromaticity on the proton bunch
length evolution.

The significance of the PCs is illustrated in Fig. 14, where a bunch intensity
is plotted vs. time ($2\times 10^{6}$ turns correspond to about 15 hours in
the Tevatron) with the complete set of IPs and PCs, and with PCs turned off.
It is clear that PCs dominate the particle losses.

Figure 14:  Normalized intensity of bunch #6 simulated in the presence (solid
line) and in the absence (dashed line) of long range collisions.

###  IV.5 Optics Errors

In the simulations we used 3 major Tevatron optics modifications:

  * •

“design” optics with ideal parameters of the main Interaction Points (IP),
zero coupling.

  * •

“january” optics which was in effect until March, 2004. This optics was
measured in January, 2004, and had sufficient distortions in the main IPs
(unequal beta’s, beam waists shifted from the IP), and betatron coupling.

  * •

“june” optics introduced in March, 2004, where the distortions were corrected.

Comparison of the three cases is shown in Fig. 15. This plot shows that
modifications to the optics implemented in March, 2004, made the optics close
to the design. Additional simulations revealed that the main source of
particle losses was in the long range collisions (PC) nearest to the main IPs.
Increasing the beams separation in these points and repairing the phase
advances cured high antiproton losses.

Figure 15:  Intensity of bunch #6 vs. time for different types of optics.
$\xi=0.01$, $Q_{x}=0.57$, $Q_{Y}=0.56$

###  IV.6 Scallops

Another illustration of validity of the code is simulation of scallops. Fig.
11 shows the simulated pattern of the antiproton emittance blow-up during
first minute of the store. We also demonstrated that scallops can be reduced
by moving the working point farther from 5th order resonance.

###  IV.7 Chromaticity

Reducing the betatron tune chromaticity can also be a very powerful instrument
in decreasing the particle losses. Results in Fig. 16, demonstrate that
changing the tune chromaticity from the present 15-20 units to 5-10 units may
significantly improve the beam life time. This can give about 10% in
luminosity integral.

Figure 16:  Evolution of the bunch intensity for various values of betatron
tune chromaticity. June optics, $Q_{x}=0.58$, $Q_{y}=0.575$, $\xi=0.01$.

###  IV.8 Further $\beta^{*}$ Reduction

An improvement which can be relatively easily implemented is the further
reduction of the beta-function at the main IPs. Decreasing the $\beta^{*}$
from 35 cm to 28 cm one can gain 10% both in peak luminosity and in luminosity
integral.

###  IV.9 Beams Separation, 23 Bucket Spacing

Since parasitic collisions give a strong contribution to beam-beam effects,
the nearest parasitic collision points dominate, one could increase separation
of beams in these points thus making their effect weaker. This could be done
by changing the bunch filling pattern from 3 trains of 12 bunches with 21 RF
buckets between bunches to 3 x 11 bunches with separation of 23 RF buckets,
conserving the overall intensity of antiprotons. In that case the separation
in the nearest PCs is bumped from 8.2 beam sigmas to 13.6 beam sigmas. Figure
17 shows that such modification allows to increase the attainable beam-beam
parameter for antiprotons from 0.01 per IP to 0.0125 without loss in
efficiency.

Figure 17:  Vertical emittance of bunch #6 vs. time for the 21 and 23 bucket
spacing configurations and different values of beam-beam parameter.

##  V  New collision helix

As mentioned, the strong betatron resonances affecting the collider
performance are caused by beam-beam effects. It was shown that the strength of
the 7-th order resonance is determined by the long range collisions Alexahin
(2007). Also, analytical calculations and numerical simulations predicted that
increasing the beam separation at the parasitic collision points nearest to
the main IPs would give the largest benefit. To achieve this, two extra
electrostatic separators were installed during the 2006 shutdown. As the
result of their commissioning, the separation at the IPs upstream and
downstream of CDF and D0 increased by 20% (Fig. 18).

Figure 18:  Radial beam separation at the collision helix in units of the beam size vs. azimuth starting from CDF IP. Blue - design helix, red - before installation of the new separators, green - present. Table 2: Radial separations in the first long range collision points in units of the beam size. | CDF u.s. | CDF d.s. | D0 u.s. | D0 d.s.
---|---|---|---|---
Before | 5.4 | 5.6 | 5.0 | 5.2
After | 6.4 | 5.8 | 6.2 | 5.6

The increased separation showed itself in improved proton lifetime. Figure 19
shows a comparison of the single bunch proton intensity for two HEP stores
before and after commissioning of the new helix. Initial intensities and
emittances of antiprotons in these stores were close which allows direct
comparison.

Figure 19:  Single bunch proton intensity in two HEP stores. 4581 with the old
helix, 4859 with the new helix.

A noticeable change in the bunch length behavior can be observed in Fig. 20.
Note that on the old helix protons experienced significant bunch shortening.

Figure 20:  Proton and antiproton bunch length in two HEP stores. 4581 with
the old helix, 4859 with the new helix.

Single bunch luminosity and luminosity integral for the same two stores are
shown in Fig. 21. As one can see, luminosity lifetime in the new configuration
has improved substantially. The overall gain can be quantified in terms of
luminosity integral over a fixed period of time (e.g. 24 hours) normalized by
the initial luminosity. The value of this parameter has increased by 16%.

Figure 21:  Single bunch luminosity and luminosity integral for stores 4581
and 4859.

##  VI  Second order chromaticity

Increasing the beam separation mitigated the long range beam-beam effects.
However, with advances in the antiproton production rate, the initial
antiproton intensity at collisions has been rising continuously. Head-on beam-
beam parameter for protons was pushed up to 0.008 per IP which made the head-
on beam-beam effects in the proton beam much more pronounced. One of the
possible ways for improvement is a major change of the betatron tune in order
to increase the available tune space. This, however, requires significant
investment of the machine time for optics studies and tuning. A partial
solution may be implemented by decoupling of the transverse and longitudinal
motion at the main IPs, i.e. by reducing the chromatic beta-function.

The value of chromatic beta-function $(\Delta\beta/\beta)/(\Delta p/p)$ at
both IPs is -600 which leads to the beta-function change of 10% for a particle
with 1$\sigma$ momentum deviation Valishev et al. (2007). Thus, a large
variation of focusing for particles in the bunch exists giving rise to beam-
beam driven synchrobetatron resonances.

Numerical simulations with weak-strong code predicted that elimination of the
chromatic beta-function at the main IPs would mitigate the deterioration of
proton lifetime at the present values of antiproton intensity even without
switching to the new betatron tune working point. In Fig. 22 the simulated
proton bunch intensities are plotted for the cases of corrected and
uncorrected chromatic beta-function.

Figure 22:  Normalized intensity of one proton bunch vs. time. Green line -
$\xi=0.01$, chromatic beta-function = -600. Blue line - $\xi=0.01$, chromatic
beta-function = 0. Red line - $\xi=0$. Numerical simulation.

In order to achieve the desired smaller beta-function chromaticity, a new
scheme of sextupole correctors in the Tevatron has been developed and
implemented in May 2007. The scheme uses the existing sextupole magnets split
into multiple families instead of just two original SF and SD circuits. The
effect of introducing the new circuits is illustrated in Fig. 6.

##  VII Summary and discussion

Over the past two years Tevatron routinely operated at the values of head-on
beam-beam tune shift for both proton and antiproton beams exceeding 0.02. The
transverse emittance of antiprotons is a factor of 3 to 5 smaller than the
proton emittance. This creates significantly different conditions for the two
beams.

Beam-beam effects in antiprotons are dominated by long range interactions at
four collision points with minimal separation. After the separation at these
points was increased to 6$\sigma$ no adverse effects are observed in
antiprotons at present proton intensitites.

On the contrary, protons experience life time degradation due to head-on
collisions with the beam of smaller transverse size. Correction of chromatic
$\beta$-function in the final focus and reduction of betatron tune
chromaticity increased dynamic aperture and improved proton beam life time.

Simulation of beam-beam effects developed for the Tevatron correctly describes
many observed features of the beam dynamics, has predictive power and has been
used to support changes of the machine configuration.

Further increase of the beam intensities is limited by the space available on
the tune diagram near the current working point. A change of the tune working
point from 0.58 to near the half integer resonance would allow as much as 30%
increase of intensities but requires a lengthy commissioning period which
makes it inlikely that this improvement will be realized during the time
remaining in Run II.

###### Acknowledgements.

This report is the result of hard work and dedication of many people at
Fermilab. The work was supported by the Fermi Research Alliance, under
contract DE-AC02-76CH03000 with the U.S. Dept. of Energy.

## References

  * Shiltsev et al. (2005) V. Shiltsev, Y. Alexahin, V. Lebedev, P. Lebrun, R. S. Moore, T. Sen, A. Tollestrup, A. Valishev, and X. L. Zhang, Phys. Rev. ST Accel. Beams 8, 101001 (2005).
  * (2) _Tevatron run ii handbook_ , http://www-bd.fnal.gov/runII.
  * Ivanov et al. (2003) P. Ivanov, J. Annala, A. Burov, V. Lebedev, E. Lorman, V. Ranjbar, V. Scarpine, and V. Shiltsev, in _Proceedings of the Particle Accelerator Conference, Portland, OR, USA_ (2003), pp. 3062–3064.
  * Ivanov et al. (2005) P. Ivanov, Y. Alexahin, J. Annala, V. Lebedev, and V. Shiltsev, in _Proceedings of the Particle Accelerator Conference, Knoxville, TN, USA_ (2005), pp. 2714–2716.
  * Ranjbar and Ivanov (2008) V. H. Ranjbar and P. Ivanov, Phys. Rev. ST Accel. Beams 11, 084401 (2008).
  * Moore et al. (2005) R. Moore, Y. Alexahin, J. Johnstone, and T. Sen, in _Proceedings of the Particle Accelerator Conference, Knoxville, TN, USA_ (2005), pp. 1931–1933.
  * Alexahin (2007) Y. Alexahin, in _Proceedings of the Particle Accelerator Conference, Albuquerque, NM, USA_ (2007), pp. 3874–3876.
  * (8) C.-Y. Tan, to be published.
  * Valishev et al. (2007) A. Valishev, G. Annala, V. Lebedev, and R. Moore, in _Proceedings of the Particle Accelerator Conference, Albuquerque, NM, USA_ (2007), pp. 3922–3924.
  * Shiltsev and McCrory (2005) V. Shiltsev and E. McCrory, in _Proceedings of the Particle Accelerator Conference, Knoxville, TN, USA_ (2005), pp. 2536–2537.
  * (11) M. Syphers, _Beam-beam tune distributions with differing beam sizes_ , fermilab Beams Doc. 3031.
  * Alexahin (2005) Y. Alexahin, in _Proceedings of the Particle Accelerator Conference, Knoxville, TN, USA_ (2005), pp. 544–548.
  * Lebedev (2003) V. Lebedev, in _Proceedings of the Particle Accelerator Conference, Portland, OR, USA_ (2003), pp. 29–33.
  * (14) http://www-bd.fnal.gov/SDAViewersServlets/valishev_sa_catalog2.html.
  * Shatilov (1996) D. Shatilov, Part. Accel. 52, 65 (1996).
  * Hirata et al. (1992) K. Hirata, H. Moshammer, and F. Ruggiero (1992), kEK Report 92-117.
  * Sajaev et al. (2005) V. Sajaev, V. Lebedev, V. Nagaslaev, and A. Valishev, in _Proceedings of the Particle Accelerator Conference, Knoxville, TN, USA_ (2005), pp. 3662–3664.
  * Valishev et al. (2006) A. Valishev, Y. Alexahin, J. Annala, V. Lebedev, V. Nagaslaev, and V. Sajaev, in _Proceedings of European Accelerator Coference, Edinburgh, Scotland_ (2006), pp. 2053–2055.
  * Lebedev et al. (2006) V. Lebedev, V. Nagaslaev, A. Valishev, and V. Sajaev, Nucl. Instrum. Methods Phys. Res., Sect. A 558, 299 (2006).
  * (20) V. Lebedev, http://www-bdnew.fnal.gov/pbar/organizationalchart/lebedev/OptiM/optim.htm.
  * (21) V. Lebedev and S. Bogacz, http://www.cebaf.gov/~lebedev/AccPhys/.
  * Lebedev and Burov (2004) V. Lebedev and A. Burov, in _Proceedings of the 33rd ICFA Advanced Beam Dynamics Workshop on High Intensity and High Brightness Hadron Beams, ICFA-HB2004, Bensheim, Germany_ (2004), pp. 350–354.


# Tübingen-Oslo system: Linear regression works the best at Predicting Current
and Future Psychological Health from Childhood Essays in the CLPsych 2018
Shared Task

Çağrı Çöltekin
Department of Linguistics
University of Tübingen, Germany
<EMAIL_ADDRESS>
Taraka Rama
Department of Informatics
University of Oslo, Norway
<EMAIL_ADDRESS>

###### Abstract

This paper describes our efforts in predicting current and future
psychological health from childhood essays within the scope of the
CLPsych-2018 Shared Task. We experimented with a number of different models,
including recurrent and convolutional networks, Poisson regression, support
vector regression, and L1 and L2 regularized linear regression. We obtained
the best results on the training/development data with L2 regularized linear
regression (ridge regression) which also got the best scores on main metrics
in the official testing for task A (predicting psychological health from
essays written at the age of 11 years) and task B (predicting later
psychological health from essays written at the age of 11).

##  1 Introduction

The words we use reflect who we are and how we are. There have been many
successful demonstrations of predicting personal properties and mental state
from language use of individuals from many forms of linguistic output
including social media text. Examples include predicting basic personal
features like gender or age Barbieri (2008); Peersman et al. (2011); Burger et
al. (2011); Nguyen et al. (2014), predicting personality traits Luyckx and
Daelemans (2008); Celli et al. (2013); Plank and Hovy (2015), predicting
sentiment towards the topic of a text Pang et al. (2008), and predicting
mental state or health Ramirez-Esparza et al. (2008); Coppersmith et al.
(2014). Most of these studies are based on social media text, and often obtain
the target variables (e.g., mental health) through best-effort approaches
based on reports by the subjects, or the venues where the texts appear.

The Task A of CLPsych-2018 Shared Task _Predicting Current and Future
Psychological Health from Childhood Essays_ has a similar aim. The task is to
predict the mental health of 11 year old children from the essays they have
written. The variables to be predicted are depression and anxiety levels as
well as a ‘total’ value indicating mental health of the child. The Task B,
takes a similar but a more ambitious aim, predicting future mental health from
the same essays. More precisely, task B comprises of predicting mental health
at ages 23, 33, 42, and 50 given the essays from the age of 11. The task B
also includes a surprise component where the mental health at 50 years age is
not given in the training set. In Task B, outcome variable is the
psychological distress score, based on responses to a questionnaire. Both
tasks are based on a well-known longitudinal study Power and Elliott (2005),
where the psychological health (among other variables) were assessed
approximately every ten years after the essays were written.

The problems tackled by both tasks, assessing current and/or future mental
health from linguistic output, are clearly relevant to monitoring public
health, as well as having possible applications to monitoring or diagnosing
mental health of individuals. The methods presented can be used to complement
well-established traditional methods, as well as providing an alternative
where traditional methods are not possible or difficult to employ. This all
may be possible, of course, if one can achieve reasonable accuracies in these
task.

As part of our submission to the CLPsych-2018 Shared Task, we experimented
with a number of different models, both traditional and relatively new, for
predicting the outcome of both tasks. However, we did not attempt to predict
age-50 mental health in Task B. Although, we only submitted the results from
ridge regression models, we also describe a few other promising models, such
as Poisson regression and recurrent neural networks (RNNs) that we
experimented with.

##  2 Models

In this section, we describe the models we experimented with. Essentially, we
experimented with a number of ‘traditional’ regression models, and also a few
neural network architectures, which also differ in the way the features are
presented to the systems. Although we were able to experiment with the models
discussed here extensively we discuss each model briefly and do not report
results with all of these models at this paper.

###  2.1 Regression (non-neural) models

The non-neural models are trained with bag of n-grams as features which are
obtained through concatenating the word and character n-grams of different
order and weighted through a global sub-linear TF-IDF scaling applied to all
the word and character n-grams. We experimented with combinations of different
orders of both word and character n-grams. Besides n-gram order, we also
experimented with case normalization, and feature selection based on document
frequency.

We also explored combination of the control variables (gender and social
class) with the textual bag-of-n-gram features. For the experiments where
control variables were included, gender was coded as a binary input variable,
while we used one-hot (or one-of-k) representation for the social class.

Our submitted system is based on ridge regression, where the weights were
trained to minimize the L2 regularized sum of squared error. We log
transformed the outcome variable, which was shifted linearly with a constant
(to avoid $\log 0$) before transformation. We applied the inverse
transformation at prediction time. In the case of both task A and task B, we
treat all the three target variables as outcomes in a single linear regression
model. The regularization parameter $\alpha_{j}$ was tuned separately for each
target variable ($1<=j<=3$) through cross-validation by searching for the best
combination of n-gram orders and $\alpha$ parameter.

In addition to linear regression, we also experimented with Poisson regression
for task A. The intuition behind the experiments with Poisson Regression is
that the target variable in task A is similar to count data. Therefore, we
assume that a target variable such as ‘total’ variable is drawn from an
independent Poisson random variable with mean $\lambda$. For a document
feature vector $x_{i}$, the mean $\lambda_{i}$ is equal to
$\exp(\theta^{T}\mathbf{x_{i}})$. Subsequently, the probability of observing
the target variable under a Poisson variable with mean $\lambda_{i}$ is given
by Poisson distribution. The Poisson regression model is a special case of
Generalized Linear Model (Nelder and McCullagh, 1989). The model is trained
through applying Stochastic Gradient Descent with RMSProp algorithm (Tran et
al., 2015). However, we found that the results of the Poisson regression model
were not better than the Ridge regression model at task A.

Besides being ‘count data’, another property of the outcome variables in this
task is that the outcome variables is zero for many data points. This type of
data is modeled better through so-called zero-inflated models Lambert (1992),
which in essence a two-stage model where the data points are classified as
zero and non-zero, and a regression model, e.g., Poisson regression, is
applied only to data points predicted to be non-zero. We also tried a two-
stage model along these lines. However, the initial results were rather
discouraging and we did not experiment with this model thoroughly.

We also explored a support vector regression model for task A and task B which
is trained in a similar fashion to SVM with TF-IDF features. The model’s
performance was close but lower than the L2 regularized linear regression and
therefore, we do not report the results of the model.

We also experimented with random forest regression and Bayesian regression
both of which have been shown to be useful in a number of similar tasks.
However, the implementations we have access to were not scalable due to the
dimensionality of TF-IDF vectors and the computation time required. Hence, we
do not report any results with these two methods either. The models discussed
in this section were implemented with Scikit-learn package (Pedregosa et al.,
2011) using liblinear back end (Fan et al., 2008). The Poisson regression
model was implemented using Numpy.

###  2.2 Neural networks

In this paper, we experimented with a neural model consisting of bidirectional
Gated Recurrent Units with character and word embeddings trained for the task.
The first layer of the model consists of a separate embedding layers built on
characters and words. The concatenated output vectors from character and word
embeddings are then supplied as input to a Gated Recurrent Network (Cho et
al., 2014). The length of sequence was fixed at 1500 characters for training
character embeddings and at 400 words for training word embeddings. The
documents are lowercased for training words and characters embeddings. The
number of GRU units was also fixed to reflect the sequence length in the case
of GRU units. The output of the GRU network had a dimension of 256 and is
followed by a fully connected layer with a single output that outputs a real
number. The network is trained using Adam optimizer with mean squared error as
the objective. All the neural models are implemented using Keras (Chollet et
al., 2015) with Tensorflow as the backend (Abadi et al., 2015).

##  3 Experiments and results

In this section, we describe the dataset, methods, experiments, and results.

###  3.1 Data

The data for the shared task comes from National Child Development Study
(NCDS), which is a longitudinal study following 17,416 babies born in Britain
in 1958, Power and Elliott (2005). The part of the study relevant to the
present task is the essays written by a subset of children at age 11. The
shared task training data includes 9217 essays. We used only the training data
for which none of the variables were missing. After removing the training
instances with missing target variables, we were left with 9146 instances for
Task A and 4938 instances for Task B. The training documents used for Task B
is a subset of the training documents used for Task A. The length of the
documents used for Task A have mean of 964.56 characters and 227.19 words. The
document length exhibits quite some variability with standard deviations of
503.07 and 116.52 respectively.

Besides essays, the data includes two background control variables: gender,
and the social class at age 11. We also used these variables as inputs to our
models. The outcome variables for Task A are scores (number of underlined
sentences) indicating anxiety, depression and total score (number of sentences
underlined). The outcome variables for Task B are the number of questions
indicating psychological distress. An important observation for all outcome
variables is that the distributions are heavily skewed with many zero values.

###  3.2 Evaluation metrics

The predictions of the models are evaluated using mean absolute error and
disattentuated R.111https://www.rasch.org/rmt/rmt101g.htm The systems are
ranked using disattentuated R which is a modification to Pearson’s R, to
account for the correlation between the outcome variables.

###  3.3 Experimental procedure and hyperparameter tuning

We did not do extensive parameter tuning with all the models section 2. After
some initial experiments, ridge regression and support vector regression
seemed to yield promising performance scores.222To put it another way, the
others, e.g., neural networks, we initially expected to perform better did not
yield expected results. Hence, we run a random search through the following
hyperparameter values.

c_ngmax

Maximum character n-gram order: 1–8

w_ngmax

Maximum word n-gram order: 1–5

min_df

Document frequency cutoff for feature selection: 1–5

$\alpha$

Regularization constant ($\alpha$): 0.5–20.0 (we used $1/\alpha$ for SVR
margin parameter C)

lowercase

Case normalization: character n-grams, word n-grams, both or none

ctrl_weight

Weight of control variables: 0.0–1.0

a11_weight

Weight of Age-11 predictions: 0.0–1.0 (Task B only)

We used 5-fold cross validation on the training set for determining the best
hyperparameter configuration. We trained the model with the best
hyperparameters on the complete training set before producing the final
predictions.

####  3.3.1 Task A

For Task A, we obtained best results on training set using 5-fold cross
validation using the hyperparameter configuration reported in table 1. We
obtained best disattentuated R scores of 0.5778, 0.2315 and 0.4678 for total,
anxiety and depression respectively on training set with the parameter values
in table 1. Similar performance scores were obtained using other (rather
diverse) parameter settings. From a quick inspection, we did not observe any
clear trends regarding usefulness of parameter values, except more features
(higher c_ngmax and w_ngmax values seem to help).

This model yielded disattentuated R scores of 0.5788, 0.153 and 0.4669 for
total, anxiety and depression on the training set respectively. This model
also obtained the top rank (based on total score) in Task A among other shared
task participants.

Hyperparameter | Ridge | SVR
---|---|---
c_ngmax | $5$ | $6$
w_ngmax | $3$ | $2$
min_df | $2$ | $1$
lowercase | word | word
$\alpha_{\texttt{total}}$ | $5.0$ | $5.0$
$\alpha_{\texttt{anxiety}}$ | $5.0$ | $10.0$
$\alpha_{\texttt{depression}}$ | $5.0$ | $20.0$
ctrl_weight | $0.5$ | $0.5$
Table 1: Best hyperparameter values for ridge regression (Ridge) and support
vector regression (SVR) models for Task A. The values are obtained through a
random search from approximately 400 random parameter settings.

####  3.3.2 Task B

The best parameter settings for both ridge regression and support vector
regression models for Task B are reported in table 2. The ridge regression
model with the hyperparameter settings reported in table 2 obtained
psychological distress correlations (disattentuated) of 0.4118, 0.2919 and
0.2527 for ages 23, 33 and 42, respectively. The inclusion of age-11
predictions as predictors in Task B was useful if gold-standard scores were
used. However, we did not observe any benefits if predicted age-11 outcomes
were used. As a result, we used the same model, except we did not use the
predicted age-11 scores as predictors in our final model. The model obtained
disattentuated R scores of 0.443, 0.3175 and 0.1961 for ages 23, 33 and 42,
respectively, on the official evaluation. With an average of 0.3189, it ranked
best among other participating models.

Hyperparameter | Ridge | SVR
---|---|---
c_ngmax | $4$ | $7$
w_ngmax | $5$ | $5$
min_df | $1$ | $1$
lowercase | word | word
$\alpha_{\texttt{total}}$ | $3.0$ | $8.0$
$\alpha_{\texttt{anxiety}}$ | $8.0$ | $20.0$
$\alpha_{\texttt{depression}}$ | $10.0$ | $20.0$
ctrl_weight | $1.0$ | $0.1$
a11_weight | $0.5$ | $0.1$
Table 2: Best hyperparameter values for ridge regression (Ridge) and support
vector regression (SVR) models for Task B. The values are obtained through a
random search from approximately 400 random parameter settings. a11_weight is
based on predicted age-11 outcomes.

Similar to Task A, the support vector regression model yielded similar but
again slightly lower results. We obtained disattentuated R scores of 0.4092,
0.2779 and 0.215 on the training set with the parameter values presented in
table 2.

##  4 Discussion

In this paper we described the models we experimented with in our
participation of CLPsych-2018 Shared Task, and reported our results. For our
submission, we used a ridge regression model with bag-of-n-gram features as
our final model. The model we used is simple linear regression with L2
regularization, except the outcome variable was log-transformed. The model
obtained best results on both tasks at official evaluation. We have also
experimented with a range of other models, including Poisson regression and
neural models. However, based on (somewhat) limited tuning efforts, none of
these systems achieved scores close to ridge regression and support vector
regression models (where the SVR model was close to the results of ridge
regression model).

The promise of this line of work, namely, predicting mental health from
language samples is interesting scientifically and it may also have important
applications in monitoring public and personal health. Predicting future
mental health is even more interesting as it may allow the clinicians to
identify preventive interventions. The method is even more relevant and easily
applicable due to the increase in the longitudinal collection of language
output in the last few decades., These prospects are only attainable if we can
predict present or future mental health from language samples with reasonable
accuracy. Our results show that there is in fact a rather strong signal in the
language samples for detecting mental health. As expected, the predictions are
less reliable as time gap between the language output and the prediction is
wider. However, there is small but reliable correlation between language
output at age 11 and psychological health at age 42.

The correlation results of both this paper and other participants clearly show
that the linguistic data contains cues that can predict current and future
mental health. One of the interesting questions is whether we can get more
information from this data. To this end, better modeling of the data may allow
us to get more information, i.e., detect more of the signal within the text at
hand. With models involving Poisson regression, zero-inflated models, and
neural networks, we tried to use models that are more suitable for the data at
hand. However, we obtained best results with relatively simple linear models
applied to log transformed data. The success of the simple linear models over
more complex (e.g., neural networks) is in line with our experiences in a
diverse set of text classification tasks Çöltekin and Rama (2016); Rama and
Çöltekin (2017); Çöltekin and Rama (2018). The results indicating superiority
of simple linear models, however, should not be considered as conclusive since
exploration of more complex models was not thoroughly performed due to time
limitations.

## References

  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015.  TensorFlow: Large-scale machine learning on heterogeneous systems.  Software available from tensorflow.org.
  * Barbieri (2008) Federica Barbieri. 2008.  Patterns of age-based linguistic variation in american english.  _Journal of sociolinguistics_ , 12(1):58–88.
  * Burger et al. (2011) John D Burger, John Henderson, George Kim, and Guido Zarrella. 2011.  Discriminating gender on twitter.  In _Proceedings of the conference on empirical methods in natural language processing_ , pages 1301–1309. Association for Computational Linguistics.
  * Celli et al. (2013) Fabio Celli, Fabio Pianesi, David Stillwell, Michal Kosinski, et al. 2013.  Workshop on computational personality recognition (shared task).  In _Proceedings of the Workshop on Computational Personality Recognition_.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014.  On the properties of neural machine translation: Encoder–decoder approaches.  In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_ , pages 103–111.
  * Chollet et al. (2015) François Chollet et al. 2015.  Keras.  https://github.com/keras-team/keras.
  * Coppersmith et al. (2014) Glen Coppersmith, Mark Dredze, and Craig Harman. 2014.  Quantifying mental health signals in twitter.  In _Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality_ , pages 51–60.
  * Çöltekin and Rama (2016) Çağrı Çöltekin and Taraka Rama. 2016.  Discriminating similar languages with linear SVMs and neural networks.  In _Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial3)_ , pages 15–24, Osaka, Japan.
  * Fan et al. (2008) Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008\.  LIBLINEAR: A library for large linear classification.  _Journal of Machine Learning Research_ , 9:1871–1874.
  * Lambert (1992) Diane Lambert. 1992.  Zero-inflated poisson regression, with an application to defects in manufacturing.  _Technometrics_ , 34(1):1–14.
  * Luyckx and Daelemans (2008) Kim Luyckx and Walter Daelemans. 2008.  Personae: a corpus for author and personality prediction from text.  In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08)_ , Marrakech, Morocco. European Language Resources Association (ELRA).
  * Nelder and McCullagh (1989) J. A. Nelder and Peter McCullagh. 1989.  _Generalized Linear Models_ , second edition.  CRC Press, Boca Raton, FL.
  * Nguyen et al. (2014) Dong Nguyen, Dolf Trieschnigg, A Seza Doğruöz, Rilana Gravel, Mariët Theune, Theo Meder, and Franciska De Jong. 2014.  Why gender and age prediction from tweets is hard: Lessons from a crowdsourcing experiment.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pages 1950–1961.
  * Pang et al. (2008) Bo Pang, Lillian Lee, et al. 2008.  Opinion mining and sentiment analysis.  _Foundations and Trends® in Information Retrieval_ , 2(1–2):1–135.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Peersman et al. (2011) Claudia Peersman, Walter Daelemans, and Leona Van Vaerenbergh. 2011.  Predicting age and gender in online social networks.  In _Proceedings of the 3rd international workshop on Search and mining user-generated contents_ , pages 37–44. ACM.
  * Plank and Hovy (2015) Barbara Plank and Dirk Hovy. 2015.  Personality traits on twitter—or—how to get 1,500 personality tests in a week.  In _Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , pages 92–98.
  * Power and Elliott (2005) Chris Power and Jane Elliott. 2005.  Cohort profile: 1958 british birth cohort (national child development study).  _International journal of epidemiology_ , 35(1):34–41.
  * Rama and Çöltekin (2017) Taraka Rama and Çağrı Çöltekin. 2017.  Fewer features perform well at native language identification task.  In _Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 255–260, Copenhagen, Denmark. Association for Computational Linguistics.
  * Ramirez-Esparza et al. (2008) Nairan Ramirez-Esparza, Cindy K Chung, Ewa Kacewicz, and James W Pennebaker. 2008\.  The psychology of word use in depression forums in english and in spanish: Texting two text analytic appr oaches.  In _International Conference on Weblogs and Social Media_ , pages 102–108.
  * Tran et al. (2015) Dustin Tran, Panos Toulis, and Edoardo M Airoldi. 2015.  Stochastic gradient descent methods for estimation with large data sets.  _arXiv preprint arXiv:1509.06459_.
  * Çöltekin and Rama (2018) Çağrı Çöltekin and Taraka Rama. 2018.  Tübingen-Oslo at SemEval-2018 task 2: SVMs perform better than RNNs at emoji prediction.  In _Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018)_ , New Orleans, LA, United States.


# NeuroCLIP: Neuromorphic Data Understanding by CLIP and SNN

Yufei Guo, and Yuanpei Chen
Intelligent Science & Technology Academy of CASIC, Beijing, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recently, the neuromorphic vision sensor has received more and more interest.
However, the neuromorphic data consists of asynchronous event spikes, which is
not natural and difficult to construct a benchmark, thus limiting the
neuromorphic data understanding for “unseen” objects by deep learning. Zero-
shot and few-shot learning via Contrastive Vision-Language Pre-training (CLIP)
have shown inspirational performance in 2D frame image recognition. To handle
“unseen” recognition for the neuromorphic data, in this paper, we propose
NeuroCLIP, which transfers the CLIP’s 2D pre-trained knowledge to event
spikes. To improve the few-shot performance, we also provide an inter-timestep
adapter based on a spiking neural network. Our code is open-sourced at
https://github.com/yfguo91/NeuroCLIP.git.

##  1 Introduction

Recently, the neuromorphic vision sensor, or “event camera”, such as [1, 19,
5, 4], has received growing attention. It works asynchronously on the pixel
level, that each pixel measures the incoming light intensity independently and
fires an event when the absolute intensity change exceeds a certain threshold.
Due to this unique information processing paradigm, it enjoys many significant
advantages compared to the conventional frame-based camera, like high dynamic
range (more than 120 dB), low latency (about 1 us), and sparse nature of the
events. Thus far, the neuromorphic vision sensor has been widely adopted in
various applications, such as feature detection and tracking[3, 23], optical
flow estimation[11, 16], 3D reconstruction monocular and stereo[10, 12], pose
estimation and SLAM[9, 8], image reconstruction[2, 18], motion
segmentation[22, 16], and neuromorphic control[15, 7].

However, since the neuromorphic data format is not natural and much different
from frame data, it is not easy to construct the big neuromorphic data
benchmark, thus limiting the neuromorphic data understanding by deep learning,
especially the ability to recognize “unseen” objects. While the frame data is
widely used and easy to collect, hence “unseen” object recognition problem can
be dramatically mitigated in 2D vision by Contrastive Vision-Language Pre-
training (CLIP) [20].

Then we wonder if could the 2D vision CLIP be transferred to the neuromorphic
domain and realize zero-shot and few-shot classification. Thus, in the paper,
we propose NeuroCLIP, which uses the CLIP’s 2D pre-trained knowledge to
understand the neuromorphic data. We first convert the neuromorphic data flow
to $T$ sequential images following [21] and fed these images to CLIP to
generate $T$ outputs. The final result is acquired by their weighted summation
as in [24]. For better few-shot performance, we also provide an inter-timestep
adapter based on a spiking neural network to fine-tune the model. The
framework of our method is illustrated in Fig. 1.

Figure 1: The framework of NeuroCLIP. The NeuroCLIP first projects the
neuromorphic data flow onto multi-timestep frames and conducts neuromorphic
data recognition via CLIP pre-trained in 2D. For better few-shot
classification, it also provides an inter-timestep adapter based on a spiking
neural network.

##  2 Methodology

In this section, we first simply revisit the CLIP for 2D vision understanding.
Then we introduce our NeuroCLIP for zero-shot recognition of the neuromorphic
data. Finally, we present an inter-timestep adapter for better few-shot
performance.

###  2.1 Contrastive Vision-Language Pre-training

CLIP is pre-trained on large image-text pairs collected from the internet and
has shown great potential on zero-shot. It provides a new paradigm that
matches images with their corresponding natural language descriptions, known
as the prompt. In specific, for an “unseen” picture from a dataset of $K$
classes, CLIP will generate $K$ textual features of category texts with
$C$-dimensions by the textual encoder denoted as $W_{t}\in\mathbb{R}^{K\times
C}$ and a $C$-dimensional image feature by the visual encoder denoted as
$f_{v}\in\mathbb{R}^{1\times C}$. Then the classification probability
$p\in\mathbb{R}^{1\times C}$ can be computed as,

$p={\rm SoftMax}(W_{t}f_{v}).$ (1)

###  2.2 Neuromorphic Data Understanding by CLIP

Constructing frames from the neuromorphic data. Neuromorphic data is a set of
asynchronous event spikes, which greatly differ from grid-based 2D images. To
construct the frame information from the event flow, we simply set the
background as 127, the ON event as 255, and the OFF event as 0 following [21].
Based on the length of the time windows, the raw spike data can be split into
different numbers of frames, known as timesteps.

Zero-shot Classification. Since the raw spike data will be split into $T$
frames, we can get $T$ image features based on the CLIP, denoted as $f_{vi}$,
for $i=1,2,\cdots,T$. We compute the classification probability for these
frames as

$p={\rm SoftMax}(\sum\alpha_{i}W_{t}f_{vi}),\ {\rm for}\ i=1,2,\cdots,T,$ (2)

where $\alpha_{i}$ is a hyper-parameter weighing the importance of frames.

###  2.3 Inter-timestep Adapter for NeuroCLIP

Here, we also consider using the CLIP for the few-shot classification of the
neuromorphic data. To better utilize the temporal information of the
neuromorphic data, we design an inter-timestep adapter for NeuroCLIP based on
a spiking neural network, which is shown advantageous in handling temporal
neuromorphic data [6]. Specifically, the $f_{vi}$ will be fed into the inter-
timestep adapter sequentially and transformed to new $\hat{f}_{vi}$. Then the
classification probability will update as

$p={\rm SoftMax}(\sum\alpha_{i}W_{t}\hat{f}_{vi}),\ {\rm for}\
i=1,2,\cdots,T.$ (3)

##  3 Results

We evaluated our method using ResNet50 as backbone on three well-known
datasets: N-MNIST [17], CIFAR10-DVS [13], and ES-ImageNet [14]. The results
are shown in Tab. 1.

Table 1: The results for NeuroCLIP. Dataset | Zero-shot | 16-shot
---|---|---
N-MNIST | 45.41% | 90.40%
CIFAR10-DVS | 25.31% | 60.72%
ES-ImageNet | 5.43% | 15.18%

##  4 Conclusion

In the paper, we proposed NeuroCLIP for neuromorphic data understanding.

## References

  * [1] Christian Brandli, Raphael Berner, Minhao Yang, Shih-Chii Liu, and Tobi Delbruck.  A 240$\times$ 180 130 db 3 $\mu$s latency global shutter spatiotemporal vision sensor.  IEEE Journal of Solid-State Circuits, 49(10):2333–2341, 2014.
  * [2] C. Brandli, L. Muller, and T. Delbruck.  Real-time, high-speed video decompression using a frame- and event-based davis sensor.  pages 686–689, 2014.
  * [3] J. Conradt, M. Cook, R. Berner, P. Lichtsteiner, R.J. Douglas, and T. Delbruck.  A pencil balancing robot using a pair of aer dynamic vision sensors.  In 2009 IEEE International Symposium on Circuits and Systems, pages 781–784, 2009.
  * [4] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, et al.  Event-based vision: A survey.  IEEE transactions on pattern analysis and machine intelligence, 44(1):154–180, 2020.
  * [5] Guillermo Gallego, Jon EA Lund, Elias Mueggler, Henri Rebecq, Tobi Delbruck, and Davide Scaramuzza.  Event-based, 6-dof camera tracking from photometric depth maps.  IEEE transactions on pattern analysis and machine intelligence, 40(10):2402–2412, 2017.
  * [6] Yufei Guo, Xuhui Huang, and Zhe Ma.  Direct learning-based deep spiking neural networks: A review, 2023.
  * [7] Wpmh Heemels, K. H. Johansson, and P. Tabuada.  An introduction to event-triggered and self-triggered control.  In Decision and Control (CDC), 2012 IEEE 51st Annual Conference on, 2012.
  * [8] Henri, Rebecq, Timo, Horstschaefer, Guillermo, Gallego, Davide, and Scaramuzza.  Evo: A geometric approach to event-based 6-dof parallel tracking and mapping in real time.  IEEE Robotics & Automation Letters, 2017.
  * [9] Hanme Kim, Stefan Leutenegger, and Andrew J. Davison.  Real-time 3d reconstruction and 6-dof tracking with an event camera.  In European Conference on Computer Vision, 2016.
  * [10] J. Kogler, M. Humenberger, and C. Sulzbachner.  Event-based stereo matching approaches for frameless address event stereo data.  Springer Berlin Heidelberg, 2011.
  * [11] Lagorce, X., Benosman, R., Ieng, S.-H., Clercq, and C.  Event-based visual flow.  IEEE transactions on neural networks and learning systems, 25(2):407–417, 2014.
  * [12] J. Lee, T. Delbruck, Paul K. J. Park, Michael Pfeiffer, and B. C. Kang.  Live demonstration: Gesture-based remote control using stereo pair of dynamic vision sensors.  In Circuits and Systems (ISCAS), 2012 IEEE International Symposium on, 2012.
  * [13] Hongmin Li.  Cifar10-dvs: An event-stream dataset for object classification.  Frontiers in Neuroscience, 11, 05 2017.
  * [14] Yihan Lin, Wei Ding, Shaohua Qiang, Lei Deng, and Guoqi Li.  Es-imagenet: A million event-stream classification dataset for spiking neural networks.  Frontiers in Neuroscience, 15, 2021.
  * [15] M. Miskowicz.  Event-based control and signal processing.  Chemical Papers, 52(3):581–582, 2016.
  * [16] A. Mitrokhin, C Fermüller, C. Parameshwara, and Y. Aloimonos.  Event-based moving object detection and tracking.  IEEE, 2019.
  * [17] Garrick Orchard, Ajinkya Jayawant, Gregory Cohen, and Nitish Thakor.  Converting static image datasets to spiking neuromorphic datasets using saccades, 2015.
  * [18] L. Pan, R. Hartley, C. Scheerlinck, M. Liu, X. Yu, and Y. Dai.  Bringing blurry alive at high frame-rate with an event camera.  2019\.
  * [19] Christoph Posch, Daniel Matolin, and Rainer Wohlgenannt.  A qvga 143 db dynamic range frame-free pwm image sensor with lossless pixel-level video compression and time-domain cds.  IEEE Journal of Solid-State Circuits, 46(1):259–275, 2010.
  * [20] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * [21] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad Benosman.  Hats: Histograms of averaged time surfaces for robust event-based object classification, 2018.
  * [22] T. Stoffregen and L. Kleeman.  Simultaneous optical flow and segmentation (sofas) using dynamic vision sensor.  2018\.
  * [23] Hanyu Wang, Jiangtao Xu, Zhiyuan Gao, Chengye Lu, Suying Yao, and Jianguo Ma.  An event-based neurobiological recognition system with orientation detector for objects in multiple orientations.  Frontiers in Neuroscience, 10, 2016.
  * [24] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li.  Pointclip: Point cloud understanding by clip, 2021.