11institutetext:  Amirkabir University of Technology, Hafez St., Tehran, Iran
11email<EMAIL_ADDRESS>
Technical Report
September 2016

# Applying Data Compression Techniques on Systolic Neural Network Accelerator

Navid Mirnouri

###### Abstract

New directions in computing and algorithms has lead to some new applications
that have tolerance to imprecision. Although, These applications are creating
large volumes of data which exceeds the capability of today’s computing
systems. Therefore, researchers are trying to find new techniques to alleviate
this crisis. Approximate Computing is one promising technique that uses a
trade off between precision and efficiency of computing. Acceleration is
another solution that uses specialized logics in order to do computations in a
way that is more power efficient. Another technique is Data compression which
is used in memory systems in order to save capacity and bandwidth.

###### Keywords:

Approximate Computing, Neural Acceleration, Data Compression , SNNAP

##  1 Introduction

Emerging applications in mining, learning and big data have created a large
workload for computing systems. While technology improvement is diminishing,
designers are trying to find alternative solutions for this crisis[1].
Approximate Computing is a promising solution, which utilize resilience of
application in order to gain efficiency in all layers of computing stack[2].
These techniques can be applied in each layer of computing: application,
architecture and circuit design. Figure1 illustrate this hierarchical
architecture. Specialized logic including accelerators and programmable logic
is another solution[2]. In some previous works, neural networks were used as
an accelerator in order to be an alternative for certain part of code[3].
SNNAP is a neural accelerator which uses FPGA as an accelerator for
implementation of Neural Processing Units (NPUs)[1]. SNNAP has been
implemented on Zynque board which is a Field Programmable logic (FPGA). SNNAP
uses ACP port for communication between NPUs and the processor. We believe
that by applying some techniques including data compression, effective memory
bandwidth could be increased. Linearly Compressed Page(LCP)[4] is one
solution, which uses two compression techniques: Based Delta Immediate
Compression[5] and Frequent Pattern Compression[6]. In future section we give
a short review on approximate computing in different layer. Then we
investigate SNNAP, after that we study Data compression techniques including:
Based Data Immediate, Frequent Pattern and Linearly Compressed Page.

Figure 1: Layers of approximate computing[2].

##  2 Approximate Computing

In today computing systems, energy efficiency is a key concern. Some
application like mining, image processing and learning have tolerance to
imprecision. Designing a platform for approximate computing is a trade off
between energy-efficiency and precision[7]. At circuit design level there some
approaches like time scaling or voltage scaling have been proposed[8].
Different approaches at architectural level could be categorized in three
groups[7]: application specific accelerators, programmable processors and
approximate memories. In [9] a programmable processor has been proposed which
uses ISA extension to use some ISA for specifying the accuracy of processing.
Approximation in software level can reduce the run-time of application, it
could be achieved via different approaches[9]. Specifying the part of the code
to be executed approximately is an important challenge[9].

##  3 Neural Networks

Neural network accelerators has been proposed in previous works[3][11][14].
The key idea is to specify part of the code that have the tolerance to
imprecision and transform them to Neural Networks. Implementation of neural
networks can be both in hardware[3][12][13] or software[11][14].

###  3.1  Systolic Neural Network accelerator in Programmable logics[3]

SNNAP is a neural network accelerator which uses programmable system on chips
(PSocs) to implement Neural Processing units. PsoCs are heterogeneous
architectures which have both hard processor and programmable logics on the
same die[3]. Integration of the CPU and programmable logic can be beneficial
for interconnections’ bandwidth comparing to traditional FPGAs.

Some important challenges in SNNAP should be mentioned before going through
more details.
1\. Neural processing unit should be implemented on FPGA in a way that uses
resources to minimize energy consumption.
2\. Communication between NPU and processor should be low latency in order to
be efficient for those applications which have smaller part of approximate
code. Processing batches of requests is one solution.
3\. In order for the system to be more energy efficient, Processor and NPU
should be able to work independently. For example, when NPU is working CPU can
be hibernated with special commands.
4\. After transformation,different applications have different Nns. This
variation in neural network topology should be handled appropriately without
much of hardware effort and reprogramming the FPGA.

SNNAP invokes MLP which is a multi-layer directed graph[3]. For implementation
of MLP, SNNAP uses Digital Signal Processing(DSP) slices of FPGA.

Figure 2: Architectrual Design of SNNAP[3].

##  4 Data Compression

Data compression is an approach to reduce capacity and bandwidth requirements
by recognizing redundancy of data in applications [10]. These techniques can
be applied in order to improve cache performance, memory bandwidth or
interconnections. In this section we are going to study some of these
techniques.

Base-delta-immediate is a compression technique for increasing on-chip cache’s
capacity. Data in cache lines are not wide in range. The key idea in BDI is to
use low dynamic range of data in cache line and represent data with a base
value and an array of deltas[5]. Frequent Pattern Compression (FPC) realize
that large volumes of records in memory are occurring frequently. This
technique tries to compress these record with fewer bits [4]. Linearly
Compressed Page (LCP) is a technique which uses two previous algorithm to gain
more efficiency than prior works [4]. The key idea in LCP is using a fixed
size for each compressed block in order to eliminate complexity of calculating
the location of each block in memory. Prior works, used variable-sized of
cache line, and this approach can cause a lot of long-latency and complex main
memory address calculations[4].

##  5 Future Works

In this report some hot topics in computer architecture has been studied
briefly. Approximate Computing is a technique that tries to benefit from
imprecision tolerance in some applications like mining, learning in order to
do computing in a more efficient way. This technique can be applied in
different layers of abstraction from application to circuit design.

Neural Acceleration is another direction that tries to invoke Neural Networks
in either software or hardware form to improve the efficiency of computing. It
selects certain part of the code and transform them to Neural Network.
Implementation of Neural networks can be in hardware [3][12][13]or software
[11][14].

We believe that SNNAP have some potentials for future works. SNNAP is using
multi-layered perceptron (MLP), we believe that it can be implemented more
efficiently if we use certain classes of Neural networks for different
category of applications. There has been a study on implementation of PARSEC
and accelerating it with Neural Networks[11]. BenchNN uses different kinds of
Neural Networks for each benchmark in PARSEC. For example for canneal
benchmark, BenchNN uses Hopfield Neural Network (HNN) instead of MLP[11]. We
believe that future works can address the possibility of customized NPU
implementation in SNNAP.

Another direction is to study data compression techniques that can be applied
to SNNAP for improving memory bandwidth. Some of them have been mentioned
briefly in this paper, but there are more research opportunities to find a
practical data compression that can be applied to SNNAP.

## References

  * [1] Moreau, T., Wyse, M., Nelson, J., Sampson, A., Esmaeilzadeh, H., Ceze, L., Oskin, M. (2015, February). SNNAP: Approximate computing on programmable SoCs via neural acceleration. In 2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA) (pp. 603-614). IEEE.

  * [2] Venkataramani, S., Chakradhar, S. T., Roy, K., Raghunathan, A. (2015, June). Approximate computing and the quest for computing efficiency. In Proceedings of the 52nd Annual Design Automation Conference (p. 120). ACM.

  * [3] Esmaeilzadeh, H., Sampson, A., Ceze, L., Burger, D. (2012, December). Neural acceleration for general-purpose approximate programs. In Proceedings of the 2012 45th Annual IEEE/ACM International Symposium on Microarchitecture (pp. 449-460). IEEE Computer Society.

  * [4] Pekhimenko, G., Mowry, T. C., Mutlu, O. (2012, September). Linearly compressed pages: A main memory compression framework with low complexity and low latency. In Proceedings of the 21st international conference on Parallel architectures and compilation techniques (pp. 489-490). ACM.

  * [5] Pekhimenko, G., Seshadri, V., Mutlu, O., Mowry, T. C., Gibbons, P. B., Kozuch, M. A. (2012). Base-delta-immediate compression: A practical data compression mechanism for on-chip caches. PACT-21.

  * [6] Alameldeen, A. R., Wood, D. A. (2004). Frequent pattern compression: A significance-based compression scheme for L2 caches. Dept. Comp. Scie., Univ. Wisconsin-Madison, Tech. Rep, 1500.

  * [7] Han, J., Orshansky, M. (2013, May). Approximate computing: An emerging paradigm for energy-efficient design. In 2013 18th IEEE European Test Symposium (ETS) (pp. 1-6). IEEE.

  * [8] Chippa, V. K., Mohapatra, D., Roy, K., Chakradhar, S. T., Raghunathan, A. (2014). Scalable effort hardware design. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 22(9), 2004-2016.

  * [9] Venkataramani, S., Chippa, V. K., Chakradhar, S. T., Roy, K., Raghunathan, A. (2013, December). Quality programmable vector processors for approximate computing. In Proceedings of the 46th Annual IEEE/ACM International Symposium on Microarchitecture (pp. 1-12). ACM.

  * [10] Yazdanbakhsh, A., Pekhimenko, G., Thwaites, B., Esmaeilzadeh, H., Mutlu, O., Mowry, T. C. (2016). RFVP: Rollback-free value prediction with safe-to-approximate loads. ACM Transactions on Architecture and Code Optimization (TACO), 12(4), 62.

  * [11] Chen, T., Chen, Y., Duranton, M., Guo, Q., Hashmi, A., Lipasti, M., … Temam, O. (2012, November). BenchNN: On the broad potential application scope of hardware neural network accelerators. In Workload Characterization (IISWC), 2012 IEEE International Symposium on (pp. 36-45). IEEE.

  * [12] Merolla, P., Arthur, J., Akopyan, F., Imam, N., Manohar, R., Modha, D. S. (2011, September). A digital neurosynaptic core using embedded crossbar memory with 45pJ per spike in 45nm. In 2011 IEEE custom integrated circuits conference (CICC) (pp. 1-4). IEEE.
  * [13] Seo, J. S., Brezzo, B., Liu, Y., Parker, B. D., Esser, S. K., Montoye, R. K., … Friedman, D. J. (2011, September). A 45nm CMOS neuromorphic chip with a scalable architecture for learning in networks of spiking neurons. In 2011 IEEE Custom Integrated Circuits Conference (CICC) (pp. 1-4). IEEE.
  * [14] Grigorian, B., Reinman, G. (2015). Accelerating divergent applications on simd architectures using neural networks. ACM Transactions on Architecture and Code Optimization (TACO), 12(1), 2.


# Preliminary Astrometric Results from Kepler

David G. Monet  U. S. Naval Observatory, Flagstaff, AZ 86001  Jon M. Jenkins
SETI Institute, Mountain View, CA 94043  Edward Dunham  Lowell Observatory,
Flagstaff, AZ 86001  Stephen T. Bryson  NASA/Ames Research Center, Moffett
Field, CA 94035  Ronald L. Gilliland  Space Telescope Science Institute,
Baltimore, MD 21218  David W. Latham  Harvard-Smithsonian Center for
Astrophysics, Cambridge MA 02138  William J. Borucki  NASA/Ames Research
Center, Moffett Field, CA 94035  David G. Koch  NASA/Ames Research Center,
Moffett Field, CA 94035

###### Abstract

Although not designed as an astrometric instrument, Kepler is expected to
produce astrometric results of a quality appropriate to support many of the
astrophysical investigations enabled by its photometric results. On the basis
of data collected during the first few months of operation, the astrometric
precision for a single 30 minute measure appears to be better than 4
milliarcseconds (0.001 pixel). Solutions for stellar parallax and proper
motions await more observations, but the analysis of the astrometric residuals
from a local solution in the vicinity of a star have already proved to be an
important tool in the process of confirming the hypothesis of a planetary
transit.

astrometry — stars: fundamental parameters

##  1 Introduction

The measurement of astrometric parameters, particularly the parallax, for
Kepler stars is a critical component of computing the physical values for
various stellar parameters using the relative values that are computed from
the photometric analysis. If the Kepler data can be shown to have the
necessary astrometric accuracy, then such a conversion can be included in the
processing for most if not all Kepler stars. Although the discussion of Kepler
astrometric accuracy must await more data and modeling, the very high
precision of Kepler positions is already a powerful tool for understanding the
photometric variations of stars and the possible presence of planetary
companions. Detailed discussions of the Kepler spacecraft and mission are
presented by Borucki et al. (2010) and Koch et al. (2010). A short overview
for the astrometric discussion is the following. The Schmidt telescope has a
1.4-m primary and a 0.95-m corrector, and the photometer is a mosaic of 42
charge coupled devices (CCDs). The boresight of the telescope remains constant
for the mission, but the spacecraft rolls 90 degrees every three months. Due
to restrictions in memory and bandwidth, only the pixels associated with the
target stars are sent to the ground for processing. Target stars are defined
by software, and pixels not associated with targets are saved only
infrequently. The basic integration time is 6.02 seconds, and the long cadence
(LC) sequence (Jenkins et al., 2010) co-adds the target pixels for 29.4
minutes. The co-added pixel data are sent to the ground every month for
processing and analysis. This strategy enables the extremely high signal-to-
noise (SNR) observations needed to achieve the planetary detection mission.

To obtain the large field of view, the image sampling is very coarse compared
to other astrometric assets. The Pixel Response Function (PRF) is described in
great detail by Bryson et al. (2010), but a quick summary is as follows. The
images contain three components, a sharp spike in the middle that comes from
optical diffraction (about 0.1 arcsecond), a wider component with a
characteristic size of 5 or 6 arcseconds set primarily by mechanical alignment
tolerances of the CCD mosaic, and a much broader scattering profile. The
intermediate component of the image profile produces the majority of the
astrometric signal, and it contains about 70% of the light. The 43 days of
data available so far 111As described more fully by Caldwell at al. (2010),
the spacecraft data are grouped by observing quarters as defined by the
mandatory rolls of the spacecraft itself. So far, data from Quarters 0 and 1
are available on the ground. have shown a remarkable astrometric precision,
but the demonstration of astrometric accuracy is still a work in progress.
Even if the centroiding process was fully understood, the short interval of
available data precludes the lifting of the degeneracies between effects of
proper motion, parallax, and velocity aberration. No measured astrometric
parameters are given here. Indeed, the entire range of observed image motion
is about 0.2 pixels, and this is dominated by the spacecraft guiding
precision.

Various authors, including King (1983) and Kaiser et al. (2000), have
developed theoretical expectations for the astrometric precision of an image.
A simple approximation is

$precision=FWHM/(2*SNR)$ (1)

where FWHM is the image full width at half maximum, and SNR is the photometric
signal-to-noise ratio of that star image. The differences in the theoretical
derivations concern the exact value for which the approximate value of 2 is
used above. The observational confirmation of this relationship has yet to be
done, but essentially all ground- and space-based astrometric studies have
demonstrated the validity of the scaling of this relationship. Improved
astrometric precision is obtained for smaller image FWHM, higher SNR, or both
assuming that adequate image sampling is available.

Kepler operates in a heretofore unstudied astrometric domain. The pixels are
very large, 3.98 arcseconds, as compared to other ground- and space-based
astrometric assets, and the observed FWHM is approximately 5 to 6 arcseconds
and depends on the location in the field of view. (See Bryson et al. (2010)
for further discussion and examples.) The effects of undersampled image
components are not captured by Eq. 1. However, Kepler was designed for
extremely high SNR observations. The well capacity of the 27-micron CCD pixels
is more than a million electrons, and most of the stars are bright. As more
fully discussed by Caldwell at al. (2010), the onset of saturation in the
basic 6.02 second integration cycle is near the magnitude Kp = 11.3. 222The
Kepler magnitude Kp includes a very wide passband and is similar to an
astronomical R magnitude in central wavelength. A single LC co-addition
produces a SNR of about 10,000 for an bright, unsaturated, uncrowded star.
There is no atmosphere to degrade the image quality, and the flux is so large
that effects such as sensor readout noise and dark current are unimportant for
the brightest 2-3 magnitudes of unsaturated stars.

##  2 Preliminary Astrometric Investigations

The astrometric processing of Kepler data is conceptually no different than
the traditional differential astrometric process of data from other ground-
and space-based assets. There is no need to worry about the actual coordinates
(i.e., J2000 RA and Dec) of the stars. Essentially all Kepler stars are in the
2MASS catalog (Skrutskie et al., 2006), and most are in the UCAC-2 catalog
(Zacharias, et al., 2004). Rather, the goal of the analysis is to measure the
small changes in position associated with proper motion, parallax,
perturbations from unseen companions, and blending with photometrically
variable stars. The current astrometric pipeline involves three distinct
steps: centroids are computed from the pixel data, transformations are
computed from each channel 333As described more fully by Jenkins et al.
(2010), each CCD is split into two channels by the flight electronics. The
astrometric verification of the stability between the two channels of a single
CCD has yet to be performed. of each LC co-addition into an intermediate
coordinate system, and solutions for each star are computed using the
intermediate coordinates and terms such as time, parallax factor, etc. Steps
two and three are iterated a few times, and convergence is quite rapid.
Details of each of these steps are presented in the following subsections.

###  2.1 Centroids

Most modern centroiding algorithms fall into three classes: moment analysis,
fits to analytic functions, and fits to the instrumental point spread function
(PSF). So far, various algorithms from the first two classes have been
implemented and tested. The data processing pipeline of the Science Operations
Center (SOC; Jenkins et al. (2010)) compute flux-weighted means for all stars
and Gaussian fits and PSF fits for a few stars. PSF fitting of all stars
remains a task for the future. In a separate effort, several other centroiding
algorithms based on fitting the images to analytic functions have been
evaluated, and centroids for all stars have been computed for many of these.
The choice of centroiding algorithm requires special attention because of the
properties of the Kepler images discussed above. The effect of the
undersampled components of the images has not been fully evaluated, and the
number of pixels for each star has been minimized by the Optimal Aperture
algorithm (Bryson et al., 2010) so that the maximum number of stars can be
transmitted in the fixed spacecraft bandwidth. Because the best astrometric
centroiding algorithm has not been identified yet, analysis is proceeding with
parallel tracks to evaluate a few of the most promising algorithms.

###  2.2 Transformation Coefficients

The current astrometric pipeline supports two different coordinate systems.
For some investigations, working in a sky-based system seems appropriate. The
tangent point is taken to be the nominal Kepler boresight
($\alpha=19^{\mathrm{h}}22^{\mathrm{m}}40^{\mathrm{s}},\delta={+44^{\circ}30^{\prime}}$,
J2000) and a simple tangent plane projection is computed from the nominal
positions of the stars listed in the Kepler Input Catalog (KIC).
444http://archive.stsci.edu/kepler/kepler_fov/search.php In this coordinate
system, effects such as differential velocity aberration and parallax are easy
to visualize. For other investigations, a channel-based coordinate system
seems appropriate, and effects arising from the structure and behavior of CCD
pixels are easier to visualize. Of particular importance are effects based on
where the star falls with respect to the pixel grid, a term called “pixel
phase”. In either coordinate system, a separate set of transformation
coefficients is computed for the measures from each channel for each cadence.
A sample of 1000 known distant giant stars was included in the Kepler star
list, and are used during the first iteration to generate a transformed
coordinate system that is as close to inertial as possible.

Figure 1: Astrometric error as a function of Kp magnitude for stars on Channel
2 in the Quarter 1 data collection.

###  2.3 Astrometric Coefficients

The tasks of computing the centroids and the transformation coefficients for
the Kepler data are similar to their counterparts for traditional ground- and
space-based assets. It is the modeling of the measured positions for each star
that involves special attention, and this flows from the extremely high SNR of
the data. The simplest solutions based on computing only the mean positions
from the data currently available show an astrometric precision near 20
milliarcseconds (about 0.005 pixels). This value and those presented below
refer to the uncertainty in a single measure of a single axis (row or column)
for a single star from a single LC co-addition. Because these solutions are
local and contain only a small number of measures, they should be construed as
estimators of the astrometric precision and not of the overall astrometric
accuracy of the spacecraft and photometer. Adding terms that model the
differential velocity aberration across the Kepler field reduces this error to
about 4 milliarcseconds (about 0.001 pixels), and adding terms arising from
the pixel phase reduce the errors to about 2 milliarcseconds (about 0.0005
pixels). These pixel phase terms are empirical fits, and are not derived from
detailed modeling of the image formation and sampling processes.

During the first 33.5 days of science operations that followed the end of
commissioning, the number of stars was increased to 156,000. The volume of
data for this number of stars observed every 29.4 minutes is large, and the
data from the 84 channels are diverse. Thus it is difficult to characterize
the entirety of the astrometric solution with a single number. Again, much
development in the astrometric processing is needed because every Kepler star
is important. Although only preliminary measures of astrometric precision have
been obtained, it is reassuring to see that the observed errors follow the
prediction based on the flux of the stars involved. Fig. 1 shows the errors
for stars in a single channel as a function of the measured Kp, and
demonstrates that the error rises as the SNR decreases.

##  3 Local Astrometric Solutions and the “Rain” Plots

A special case of the astrometric solutions described above can be computed in
the vicinity of individual stars. Under the assumptions of small parallax and
adequate removal of differential velocity aberration, the equations for the
apparent place of a star can be linearized. Simple trend analysis produces a
robust estimator for the mean position of a star, and residuals from each
measurement are computed. This enables astrometric processing to contribute to
the understanding of the Kepler stars. As more fully discussed by Batalha et
al. (2010), what appears to be a single object can be two or more stars, and
each can have photometric variability. Such astrations can mimic the
photometric properties of a transiting planet, and adding astrometry to the
vetting procedure can assist in the confirmation or denial process. Fig. 2
shows astrometric residuals for two stars. The upper star is a blend of a
variable star and one or more constant stars while the lower shows residuals
that are typical for a bright, constant star. The astrometric amplitude of the
variable star is huge - almost 0.02 pixels.

The astrometric behavior of an image composed of an unknown number of stars
each of which having unknown photometric variations is complicated. However,
the simple case of two stars, a small photometric variation, and centroids
computed from flux-weighted means provides much insight. Where $\Delta s$ is
the true separation of the stars, $\delta s$ is the small measured astrometric
shift, $F$ is the small relative brightness of the fainter B component
compared to the brighter A component, and $f$ is the small relative change in
total brightness due to a transit or stellar variability, the observed
astrometric shift assuming that the A component is variable is given by

$\delta s=Ff\Delta s$ (2)

If the B component has the photometric variation, the observed shift is

$\delta s=f\Delta s$ (3)

When the Kepler observations of $\delta s$ and $f$ are combined with high
resolution imagery that can measure $\Delta s$, $F$, and other characteristics
such as stellar colors, then the model of the composite image can be improved.

On the basis of its photometric signature alone, the Kepler Object of Interest
(KOI-) 15 might involve a transiting planet. The analysis of the combined
photometric and astrometric residuals denies this hypothesis. Fig. 3. shows
the time series astrometric and photometric residuals for KOI-15 after they
have been high-pass filtered so as to emphasize signals with shorter
timescales. A different visualization of the same data is called a “Rain
Plot”, and is shown in Fig. 4. Clearly, the astrometric and photometric
residuals for KOI-15 are strongly correlated, and these correlations are the
signature of an astration that includes one or more relatively constant stars
and a background eclipsing binary. Indeed, the secondary eclipse is more
apparent in the astrometric residuals than in the photometric residuals. A
true transiting planet system should not show these correlations. As suggested
by the Rain Plot, the pixel data were re-examined and the offending variable
star was identified as being about 11 arcseconds away from and about 4.8 Kp
magnitudes fainter than the brighter star.

##  4 Conclusions

Although based on just a small fraction of the data expected from the entire
mission, the following conclusions can be drawn.

a) Both the preliminary version of the full astrometric solution and the
locally linearized astrometric solution indicate that the precision of a
single measure of a typical star is about 0.004 arcsecond (about 0.001 pixel).
Results such as those shown in Figs. 2 and 3 suggest that the precision for
some stars might be substantially better.

b) On the basis of the limited data available so far, many astrometric effects
cannot be separated. The astrometric precision should enable the measurement
of the proper motions of the known large motion stars in the star list, but as
yet proper motion is indistinguishable from differential velocity aberration,
pixel phase, and similar effects.

c) Because of the large pixel size, Kepler images can often be blends of
multiple stars. If one or more components of such a blend is a photometrically
variable star, the astrometric position of the image can have a significant
motion. Tools such as the Rain Plot have already demonstrated the utility of
combining astrometric and photometric processing in the evaluation of
planetary transit candidates.

d) Whereas the astrometric precision of Kepler data has been demonstrated, the
astrometric accuracy has yet to be evaluated.

In summary, it is the extremely high SNR of the Kepler photometer that enables
the astrometric analysis of Kepler data. Even with just the preliminary data,
astrometric analysis is providing an important tool for the physical
understanding of the observations. Should the astrometric results continue to
follow the theoretical expectation of improving with the SNR, then solutions
for the parallax and proper motions of all stars will be computed.

Figure 2: Astrometric residuals from a blended variable (top) and a non-
variable star (bottom) taken from a solution for a single channel. Red symbols
are from the columns and blue symbols are from the rows. Figure 3: Time series
residuals for the relative photometric flux (A) in parts per thousand, and the
relative astrometric row (B) and column (C) residuals in millipixels (1 pixel
= 3.98 arcseconds) for KOI-15. The observations have been filtered to remove
long-period trends. The strong correlation between these residuals is taken as
evidence that this Kepler star is actually an astration of one or more
relatively constant stars and a background eclipsing binary. Figure 4: The
Rain Plot for KOI-15. This visualization shows the correlation between the
photometric and astrometric row (red X) and column (blue +) residuals. The
strong correlation between the position and brightness is evidence that this
is an astration of one or more relatively constant stars and a background
eclipsing binary.

Funding for this Discovery mission is provided by NASA’s Science Mission
Directorate. Many people have contributed to the success of the Kepler
Mission, and the authors wish to express their profound thanks to all.
Facilities: The Kepler Mission.

## References

  * Batalha et al. (2010) Batalha, N. M., et al. 2010, this issue
  * Borucki et al. (2010) Borucki, W. J. et al. 2010, Science, submitted
  * Bryson et al. (2010) Bryson, S. T., et al. 2010, this issue
  * Caldwell at al. (2010) Caldwell, D. A., et al. 2010, this issue
  * Jenkins et al. (2010) Jenkins, J. M., et al. 2010, this issue
  * Kaiser et al. (2000) Kaiser, N., Tonry, J. L., Luppino, J. L. 2000, PASP112, 768
  * King (1983) King, I. R. 1983, PASP95, 163
  * Koch et al. (2010) Koch, D. G., et al. 2010, this issue
  * Skrutskie et al. (2006) Skrutskie, M. J., et al. 2006, AJ131, 1163
  * Zacharias, et al. (2004) Zacharias N., et al. 2004, AJ127, 3043


# CGCE: A Chinese Generative Chat Evaluation Benchmark
for General and Financial Domains

Xuanyu Zhang, Bingbing Li and Qing Yang

Du Xiaoman

###### Abstract

Generative chat models, such as ChatGPT and GPT-4, have revolutionized natural
language generation (NLG) by incorporating instructions and human feedback to
achieve significant performance improvements. However, the lack of
standardized evaluation benchmarks for chat models, particularly for Chinese
and domain-specific models, hinders their assessment and progress. To address
this gap, we introduce the Chinese Generative Chat Evaluation (CGCE)
benchmark, focusing on general and financial domains. The CGCE benchmark
encompasses diverse tasks, including 200 questions in the general domain and
150 specific professional questions in the financial domain. Manual scoring
evaluates factors such as accuracy, coherence, expression clarity, and
completeness. The CGCE benchmark provides researchers with a standardized
framework to assess and compare Chinese generative chat models, fostering
advancements in NLG research.

##  1 Introduction

GPT-based language models, such as ChatGPT OpenAI (2022) and GPT-4 OpenAI
(2023), have sparked a remarkable revolution in the realm of natural language
generation (NLG). These methods have brought about substantial performance
enhancements across a wide range of NLG tasks. An important catalyst driving
the continuous improvement and refinement of these models is the utilization
of evaluation benchmarks, which provide a standardized means to assess and
compare their effectiveness.

However, the current landscape of language evaluation benchmarks predominantly
revolves around traditional NLU or NLG tasks for language models, with
benchmarks like GLUE and SuperGLUE playing a prominent role. Unfortunately,
this leaves a significant gap when it comes to evaluating chat models,
especially Chinese chat models or specific generation models, despite Chinese
speakers constituting a quarter of the world’s population. The absence of a
comprehensive evaluation benchmark presents a considerable obstacle for
researchers in assessing the performance and capabilities of their Chinese
generative chat models. Without a standardized benchmark, gauging the quality
and effectiveness of these models becomes a daunting task, hindering progress
and advancements in the field.

To address this critical gap and facilitate the growth of Chinese language
research, we introduce the Chinese Generative Chat Evaluation Benchmark (CGCE)
for general and financial domains. This benchmark encompasses a broad spectrum
of generative chat domains, including general and financial fields with a
varying number of categories. In the general domain, our evaluation benchmark
includes a diversified set of 200 questions, covering 13 major dimensions
including mathematical calculations, scenario writing, logical reasoning, and
text summarization. In the financial domain, it covers four major areas:
understanding financial terms, providing financial market commentary,
conducting financial data analysis, and comprehending financial news. It
consists of 150 specific professional questions, allowing us to
comprehensively examine the model’s proficiency in handling financial tasks
from multiple perspectives. Besides, the evaluation process for the CGCE
benchmark involves manual scoring, considering various factors such as answer
accuracy, logical coherence, clarity of expression, and completeness, among
others. This multi-dimensional scoring approach ensures a comprehensive
assessment of the model’s performance across different aspects of generative
chat. By introducing the CGCE benchmark, we provide researchers and
practitioners in the field with a standardized framework to evaluate and
compare the effectiveness of Chinese generative chat models. This benchmark
serves as a valuable resource for assessing the capabilities and limitations
of these models, enabling researchers to identify areas for improvement and
driving progress in the field of generative chat.

##  2 CGCE Benchmark

|  General |  Financial
---|---|---
|  domain |  domain
Number |  200 |  150
Sub-domain |  13 dimensions: calculations, scenario writing, logical reasoning, text summarization and so on. |  4 areas including financial terms, financial market commentary, financial data analysis, and financial news.
Evaluation | Manual evaluation with some factors.

Table 1: The overview of CGCE Benchmark. Type |  Instruction
---|---
Calculation |  15 + 32 = ?
Scenario |  请为一家初创公司策划一场线上产品发布会，包括活动目的、活动流程、邀请嘉宾和宣传策略。 (Please plan an online product launch event for a start-up company, including event purpose, event process, invited guests and publicity strategy)
Reasoning |  如果甲、乙、丙三个人中只有一个说了真话，那么是谁说的真话？甲说：“我不是说谎者。”，乙说：“丙是说谎者。”，丙说：“乙是说谎者。” (If only one of A, B, and C is telling the truth, who is telling the truth? A said: "I am not a liar." B said: "C is a liar." C said: "B is a liar.")
Table 2: Examples in general domain. Type |  Instruction
---|---
Explanation of nouns |  请解释一下"股票回购"是什么意思，以及公司为什么要进行股票回购？ (Please explain what is meant by "share repurchases" and why are companies doing stock repurchases?)
Financial commentary |  请分析量化宽松政策可能导致的资产泡沫和通胀风险，以及政府和央行如何平衡经济复苏与通胀风险之间的关系？ (Please analyze the asset bubbles and inflation risks that may be caused by the quantitative easing policy, and how the government and the central bank balance the relationship between economic recovery and inflation risks?)
Table 3: Examples in financial domain.

The overview of CGCE Benchmark is shown in Table 1. As shown in Table 5, in
the general domain, our evaluation benchmark encompasses a wide range of 200
questions, spanning 13 significant dimensions that include mathematical
computations, scenario creation, logical reasoning, and text condensation. As
shown in Table 3, in the financial domain, our benchmark focuses on four major
areas: comprehension of financial terminologies, delivering commentary on
financial markets, performing analysis of financial data, and understanding
financial news. It consists of a specialized set of 150 questions that target
professionals in the field, enabling us to conduct a comprehensive assessment
of the model’s aptitude in handling diverse financial tasks from multiple
vantage points. This comprehensive evaluation allows us to gain a holistic
understanding of the model’s effectiveness in the financial domain. More
examples are shown in the appendix. We will additionally provide a webpage to
show the evaluation results of current chat models.

##  3 Conclusion

In this paper, we have made significant contributions to addressing the
pressing need for standardized evaluation benchmarks tailored specifically to
generative chat models, with a particular emphasis on the Chinese language and
domain-specific contexts. Our primary objective was to provide researchers and
practitioners with a comprehensive and reliable framework for evaluating the
performance of Chinese generative chat models. To fulfill this objective, we
introduced the Chinese Generative Chat Evaluation (CGCE) benchmark, a
groundbreaking contribution that spans both general and financial domains. The
CGCE benchmark comprises a diverse and carefully curated set of questions,
covering a wide range of dimensions and professional topics. This extensive
coverage ensures that the benchmark captures the complexity and diversity of
generative chat tasks.

## References

  * OpenAI (2022) OpenAI. 2022.  Chatgpt.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.

Type |  Instruction (Chinese) |  Instruction (English)
---|---|---
Calculation |  15 + 32 = ? |  15 + 32 = ?
Scenario writing |  请为一款新开的烤鸭店撰写一段富有创意的文案，突出店铺特点和消费者的需求。 |  Please write a creative copy for a newly opened roast duck restaurant, highlighting the characteristics of the store and the needs of consumers.
Coding |  请编写一个简单的 HTML 结构，包含一个标题（<h1>），一个段落（<p>），和一个无序列表（<ul>）。 |  Please write a simple HTML structure that includes a heading (<h1>), a paragraph (<p>), and an unordered list (<ul>).
Translation |  请将以下法语句子翻译成中文：“Le petit prince est un livre célèbre écrit par Antoine de Saint-Exupéry.” |  Please translate the following French sentence into Chinese: "Le petit prince est un livre célèbre écrit par Antoine de Saint-Exupéry."
Moral |  我偷了一辆车，藏在哪里可以不被警察发现？ |  I stole a car, where can I hide it from the police?
Abstract |  近年来，基于深度学习的语言模型，尤其是以循环神经网络（RNN）和变压器（Transformer）为代表的模型，在自然语言处理领域取得了巨大的成功。除了语言模型，NLP技术还包括词法分析、句法分析、语义分析等方面。其中，词法分析是将文本中的单词或词汇单元进行分析和分类的过程；句法分析是对句子进行语法分析和结构分析的过程；语义分析则是对句子的意思和语境进行分析和理解的过程。这些分析过程可以帮助计算机理解自然语言文本，并根据需要生成或修改文本内容。尽管NLP技术在自然语言处理领域取得了重大进展，但仍然面临着许多挑战。由于自然语言文本中存在大量的语言歧义和多义性，因此NLP技术在某些情况下可能无法准确地理解和处理文本内容。请根据上文，生成一段简短的摘要，概括出文章的主旨。 |  In recent years, language models based on deep learning, especially models represented by recurrent neural networks (RNN) and transformers (Transformer), have achieved great success in the field of natural language processing. In addition to language models, NLP technology also includes lexical analysis, syntactic analysis, and semantic analysis. Among them, lexical analysis is the process of analyzing and classifying words or lexical units in the text; syntactic analysis is the process of syntactic and structural analysis of sentences; semantic analysis is the process of analyzing and understanding the meaning and context of sentences. process. These analysis processes help computers understand natural language text and generate or modify text content as needed. Although NLP technology has made significant progress in the field of natural language processing, it still faces many challenges. Due to a large amount of linguistic ambiguity and polysemy in natural language texts, NLP technology may not be able to accurately understand and process text content in some cases. Based on the above, please generate a short abstract summarizing the main idea of the article.
Table 4: More examples in general domain. Type |  Instruction (Chinese) |  Instruction (English)
---|---|---
Explanation |  什么是"割韭菜"？在投资领域，这个词语通常是怎么使用的？ |  What is "cutting leeks"? How is this term commonly used in the investment fields?
Explanation |  你能解释一下什么是"银行业务的KYC"流程吗？ |  Can you explain what is the "KYC for Banking" process?
Explanation |  你能解释一下什么是"抵押贷款"和"信用贷款"的区别吗？ |  Can you explain what is the difference between a "mortgage" and a "line of credit"?
Financial commentary |  随着全球范围内对可再生能源的需求不断增加，太阳能、风能等替代能源项目投资也在稳步上升。然而，这些项目的投资回报率受到诸如政府补贴政策、技术进步和市场竞争等因素的影响。请解释可再生能源项目投资回报率受到哪些主要因素的影响，并分析在未来几年内，这些因素可能发生的变化及其对投资者的影响？ |  As the demand for renewable energy continues to increase worldwide, investment in alternative energy projects such as solar energy and wind energy is also rising steadily. However, the return on investment of these projects is affected by factors such as government subsidy policies, technological progress, and market competition. Please explain what are the main factors that affect the return on investment of renewable energy projects, and analyze how these factors may change in the next few years and their impact on investors?
Data analysis |  如果一家公司的股票在过去一年中的最高价是100美元，最低价是50美元，那么其振幅是多少？ |  If a company’s stock has had a high of $100 and a low of $50 over the past year, what is the amplitude?
Financial news |  美联储宣布加息0.25个百分点，这是自疫情爆发以来的首次加息。此举旨在应对美国持续上升的通货膨胀压力。请分析美联储加息的原因，以及加息可能对全球金融市场和投资者产生哪些影响？ |  The Federal Reserve announced a 0.25 percentage point hike in interest rates, the first hike since the outbreak. The move is aimed at responding to rising inflationary pressures in the United States. Please analyze the reasons for the Fed’s interest rate hike and what impact it may have on global financial markets and investors?
Table 5: More examples in financial domain.