# Constraint-Based Inference of Heuristics for Foreign Exchange Trade Model
Optimization

Nikolay Ivanov  Dept. of Computer Science and Engineering
Michigan State University
East Lansing, MI, USA
<EMAIL_ADDRESS>Qiben Yan  Dept. of Computer Science and Engineering
Michigan State University
East Lansing, MI, USA
<EMAIL_ADDRESS>
###### Abstract

The Foreign Exchange (Forex) is a large decentralized market, on which trading
analysis and algorithmic trading are popular. Research efforts have been
focusing on proof of efficiency of certain technical indicators. We
demonstrate, however, that the values of indicator functions are not
reproducible and often reduce the number of trade opportunities, compared to
price-action trading.

In this work, we develop two dataset-agnostic Forex trading heuristic
templates with high rate of trading signals. In order to determine most
optimal parameters for the given heuristic prototypes, we perform a machine
learning simulation of 10 years of Forex price data over three low-margin
instruments and 6 different OHLC granularities. As a result, we develop a
specific and reproducible list of most optimal trade parameters found for each
instrument-granularity pair, with 118 pips of average daily profit for the
optimized configuration.

###### Index Terms:

machine learning, Forex, algorithmic trading, simulation

##  I Introduction

The Foreign Exchange, also known as Forex, is a decentralized global market
with trillions of dollars of daily turnover [1]. Some unique features of the
Forex market make it highly suitable for technical analysis and algorithmic
trading [2]. Technical indicator functions are popular heuristics used by
Forex traders to identify trading opportunities. Multiple research efforts
have been focusing on the proof of efficiency of certain existing indicator
functions for fixed trade setups [3, 4, 5, 6, 7, 8, 9]. However, as we
demonstrate in Section IV, the values of popular indicators are different for
different data sets and brokers. This compromises the accuracy of machine
learning of the historical Forex data and impedes the development of concrete
recommendation for the traders. Moreover, we also demonstrate that the
indicator heuristics reduce the overall count of trade opportunities, and thus
reduce the final profit.

In this paper, we develop two dataset-agnostic price-action heuristic
templates for trade signals on the Forex Exchange market: _candle trend
continuation_ and _candle trend reversal_. These heuristics are characterized
by a very high rate of trading opportunities, compared to popular technical
indicators. The ultimate goal of this work is to determine most optimal
parameters for the given heuristic templates for each combination of Forex
instruments and trade periods given in the input data set. In order to achieve
that goal, we develop SmartFX, a machine learning tool which analyzes 10 years
of historical Forex price data over three low-margin instruments and 6
different OHLC granularities, delivering a specific and reproducible list of
most optimal trade parameters found for each input granularity of each
instrument.

###  I-A Motivation

Although the research and analysis of the Forex market is abound with original
approaches, the practical application of these approaches is challenged by the
ambiguity of results, unrealistic assumptions, or simply the lack of proper
methodology for historical data analysis [10, 11].

Thus, the motivation of this work is to deliver a clear, specific, evidence-
supported, realistic, and reproducible practical trading model with concrete
parameters that can be applied by all tiers of Forex traders.

##  II Background

###  II-A Definitions

##### Foreign Exchange (Forex)

The Foreign Exchange is a global market in which national currencies and some
commodities are exchanged [1].

##### Forex Instrument (Forex Pair)

Forex Instrument is a pair of equities, which are either currencies (e.g. U.S.
Dollar) or commodities (e.g., gold). For example, the pair EUR/USD is a Forex
instrument for trading U.S. Dollars for Euros and vice versa. Some examples of
Forex instruments are given in Table I. The taxonomy of Forex pairs includes
several types, such as _major pair_ or _cross pair_ , but they are all traded
in the same manner.

##### Pip

Pip is a small unit of price movement (change) in the Forex market. The value
of one pip is different depending on the instrument. For EUR/USD 1 pip =
0.0001, for USD/JPY and EUR/JPY, 1 pip = 0.01.

TABLE I: Examples of Forex instruments and their types. Instrument | Description | Type
---|---|---
EUR/USD | Euro/U.S. Dollar | major pair
USD/JPY | U.S. Dollar/Japanese Yen | major pair
EUR/JPY | Euro/Japanese Yen | cross pair
GBP/AUD | Pound Sterling/Australian Dollar | cross pair
NZD/CAD | New Zealand Dollar/Canadian Dollar | cross pair
XAU/USD | Gold/U.S. Dollar | commodity
XAG/USD | Silver/U.S. Dollar | commodity

##### Long trade

The Forex market allows two trade directions: buying and selling. The buying
trade is commonly called a _long trade_.

##### Short trade

The Forex market allows to submit a selling trade even if there is no active
buying trade; for a trader it may be perceived as betting on a downward price
trend or as a reverse trade. Regardless of the trader’s perception, the
selling trade is commonly called a _short trade_.

###  II-B Spread

Many Forex brokers do not charge any direct per-trade commission. Instead,
they establish a difference between the buying price (called the _ask price_ ,
denoted $P_{a}$) and the selling price (called _bid price_ , denoted $P_{b}$).
Then the _spread_ of a broker $\beta$ for an instrument $i$, denoted
$P_{\beta,i}$, is defined as follows111In some literature, the spread is
represented as a negative value. We use the absolute value notation to
emphasize the simple fact of the difference, regardless of its sign.:

$S_{\beta,i}=|P_{b}-P_{a}|$

###  II-C Margin trading

In most days when the market is open, the Forex prices experience only minor
fluctuations, which would require traders to make very large deposits in order
to gain a feasible profit. In order to make price movement more influential,
most Forex brokers allow for _margin trading_ , in which the broker lends the
client an amount of money $V_{m}$, called the _margin_ , which is $M$ times
greater than the actual deposit $D$, thus:

$V_{m}=M\times D$

The value $M$, usually denoted $M:1$, e.g., 1:50, is called the _leverage_.
Most Forex brokers in the United States offer the 1:50 leverage.

###  II-D OHLC Candlestick Data

The raw Forex price data is usually provided by a broker in the form of a
series of _ticks_. Each instrument has its own stream of ticks, and each tick
includes a timestamp, bid price, ask price, and an optional trade volume data.
Depending on the broker and its API, the period between ticks can vary between
50 ms to several seconds. The tick data is necessary for trading, but in most
cases it is overwhelmingly excessive for a technical analysis. Thus, a more
analysis-friendly cmopressed form, called _OHLC candlestick data_ , is
commonly used in most markets, including Forex.

OHLC (Open, High, Low, Close) data is a set of points, called _OHLC candles_ ,
_OHLC candlesticks_ , or _OHLC ticks_ , summarizing the price movement during
a fixed period of time using four parameters: 1) price at the beginning of the
period (O); 2) maximum price during the period (H); 3) minimum price during
the period (L); and 4) the price at the end of the period (C). Figure 1
demonstrates the chart and CSV representations of OHLC data. The period of
time that one candle spans in a data set is called the _OHLC granularity_.
Common OHLC granularities for the Forex market are: M1 (1 minute), M2 (2
minutes), M5 (5 minutes), M10 (10 minutes), M15 (15 minutes), M30 (30
minutes), H1 (1 hour), H2 (2 hours), H4 (4 hours), H8 (8 hours), H12 (12
hours), D1 (one day), W1 (one week), and MN (one month).

Figure 1: The OHLC candlestick data in chart and comma-separated value (CSV)
representations.

###  II-E Forex Analysis

The goal of a the Forex analysis is to predict the future movement of the
price and identify trading opportunities. There are two major types of the
market analysis: a) the _fundamental analysis_ and b) the _technical
analysis_. The fundamental analysis uses economic, political, and other news
data to predict the future behavior of the market. The technical analysis uses
present and past movement of the market price to infer the future move of the
price, regardless of any outside (fundamental) events. Many traders also use
various combinations of the fundamental and technical analyses. The technical
analysis may be based on heuristic functions called _technical indicators_ ,
or it can be based solely on the past and present values of the price. The
latter case is called the _price-action_. SmartFX focuses on the pure price-
action technical analysis only.

##  III Related Work

###  III-A Research

Automated Forex exchange trading is a popular topic among peer-reviewed
studies [12, 4, 3, 13, 14, 15]. In the last two decades, many studies have
been conducted on different aspects of foreign exchange technical indicators
[16, 4, 3, 13, 14, 15]. Many patents have been filed for automatic Forex
trading systems [17, 18, 19, 20, 21]. Also, the Artificial Intelligence and
Machine Learning has been used for price prediction of the Foreign Exchange
market [22, 23, 24, 25, 3, 26, 27].

However, the number of combinations of possible settings, as well as the
abundance of pre-existing constraints for Forex automated trading is virtually
infinite. As the number of constraints is growing, the necessity to employ the
methods of Artificial Intelligence and Machine Learning becomes an urgent
necessity because the space and time complexity of a brute-force trial exceeds
storage and computational abilities of existing computers. To the best of my
knowledge, none of the existing studies delivers a complete set of trading
parameters for automated or semi-automated Forex trading with low equity and
other parameters representing a non-institutional trader.

###  III-B Trading Heuristics

Most popular heuristics for technical analysis are technical indicators [7].
For example, one of the largest Forex brokers in the United States, GAIN
Capital Group LLC [28], provides 75 classes of technical indicators as part of
its Web Trading Platform 222https://webtrader.forex.com/workspaces/default.

###  III-C Retrospective Simulation

The retrospective simulation of Forex trading is a software-based processing
of historical Forex price data, using a given trade model and a set of
heuristics. There is a number of software trade simulators available on the
market, such as MetaTrader333https://www.metatrader4.com/en,
NinjaTrader444https://ninjatrader.com/Simulate, Forex
Tester555https://forextester.com/, Soft4Fx666https://soft4fx.com/,
fxSimulator777https://fxsimulator.com/, and others. All these tools allow to
test a given strategy, automatically or manually, using a historical data.
However, to the best of my knowledge, none of these tools is capable of
_calculating the most optimal strategy_ , which is the main goal of SmartFX.

###  III-D Other Works

There are multiple books written on the topic of Forex trading optimization.
The ”forex trading” search result at the Book Department of Amazon web store
returns more than 6,000 results888https://www.amazon.com/s?k=forex+trading.
The same search request on a popular platform for online video courses, Udemy,
returns 3,934
results999https://www.udemy.com/courses/search/?src=ukw&q=forex%20trading.

##  IV Pre-Evaluation

###  IV-A Inconsistency of Technical Indicators

Moving Average Convergence/Divergence (MACD) is one of the most popular
technical indicators among Forex traders [3]. However, different data sets and
different trading tools reveal significant differences between the actual
values of popular technical indicators, and this is easily demonstrable by
comparing the charts of MACD for the same instrument with the same OHLC
granularity within the same time frame, with time zone differences properly
synchronized.

For this experiment, we observe four charts of the MACD technical indicator,
produced by different trade tools with the same parameters (fast MA: 12, slow
MA: 26, length: 9) generated for 24 OHLC H1 candlesticks representing the
Forex price during November 28, 2019 EST, accounting for the differences in
time zones101010Two out of four data sets used the GMT time zone, which was
shifted to match the EST time zone.. As we can see in Figure 2, there is a
significant differences between the MACD indicators with the same parameters
over the same time period. One reason for such a discrepancy may be the fact
that November 28, 2019 was the Thanksgiving Day in the United States, during
which the trading activity is very low, whereas in most other countries it was
just a regular business day, resulting in Forex activity difference in brokers
from different regions (which has some effect upon the price). Regardless of
the reason, the experiment demonstrated that _there was at least one day,
during which MACD showed significantly different results for different
brokers_.

(a) Forex.com/MT4 (b) Dukascopy Demo/JForex

(c) Oanda/iOS (d) Metaquotes Demo/MT5 iOS

Figure 2: MACD charts produced by different trading tools, which use different
data sets for the same period of time (November 28, 2019 00:00—November 28,
2019 23:59, EST, adjusted for differences in time zones) of the same Forex
instrument (EUR/USD) with the same OHLC granularity (H1).

###  IV-B Trade Opportunities Rate Reduction

In this work, we propose a price-action trade model, in which every OHLC
candle, upon its closure time, delivers a new trade signal. For example, for
the H1 OHLC granularity, the number of trading signals per day is 24. In other
words, in SmartFX the signal period $T_{SmartFX}$ is constant and equal to the
granularity $G$:

$T_{SmartFX}=G$

When the trade signals come from traditional technical indicators, the trade
signal period $T_{ind}$ is variable, and presumably many times larger than
$G$. In order to demonstrate this, we conducted a short experiment, in which
we counted the number of possible configurations for MACD and RSI technical
indicators that can be treated as signals over one month of historical Forex
data (November 2019, EUR/USD) with OHLC granularity H1. Table II shows the
results of this experiment.

TABLE II: Number of trade signal signatures per month produced by MACD, RSI, and SmartFX for EUR/USD pair with OHLC granularity H1, counted during November 2019. For SmartFX, se assume that there are 20 full days when the Forex market is open during the month. Heuristic | Heuristic Parameters |  | Number of Signal
---
Signatures During
November, 2019
MACD |  | Fast MA: 12
---
Slow MA: 26
Length: 9
55
RSI | Length: 14 | 64
SmartFX | N/A | 480

Let us show that on average the technical indicators produce less trade
signals than SmartFX. Suppose that there is a set of parameters $\Upsilon$ for
a traditional technical indicator $I$, which is twice as more successful than
SmartFX (with the same take-profit/stop-loss settings), and which also
produces twice as more trade signals than RSI-14. Then, if the average
proportion of successful trades for SmartFX is $N_{SmartFX}(0\leq
N_{SmartFX}\leq 1)$, then the number of successful trades per month for
SmartFX will be as follows:

$ST_{SmartFX}=480\times N_{SmartFX}$

Whereas the number of successful trades for the hypothetical indicator will be
as follows:

$ST_{I(\Upsilon)}=2\times(2\times 64\times N_{SmartFX})=256\times N_{SmartFX}$

As it is seen from the experiment, even with unrealistically optimistic
assumptions, $ST_{SmartFX}>ST_{I(\Upsilon)}$. And therefore it is fair to
conclude that even if the traditional technical indicators have higher rate of
successful signals, the decreased overall number of trading opportunities
yields a lower overall profit compared to per-candle price action.

##  V SmartFX Design

Figure 3 demonstrates the basic design of SmartFX. The market data, which is
described in detail in Section VII, is being pre-processed to facilitate the
performance of the simulation. Specifically, the market data is being saved
twice in two different data structures: hash table and serial list. This
allows to reduce the processing time because the latter is the system’s
performance bottleneck.

Figure 3: An overview of the SmartFX design. The market data, which consists
of several subsets, is pre-processed for performance optimization by
generating an auxiliary hash maps for the data subsets. The pre-processed data
is used by the generator of heuristics, along with the set of heuristic
prototypes and constant parameters (e.g., broker’s spread). The generated
heuristics are fed to the multi-threaded trade simulation learning algorithm,
along with a set of constraints (representing the trader), and the trade model
(representing the rules of the Forex market). The results of the simulation
are being analyzed to produce a set of the most optimal trading parameters.

The pre-processed market data goes to the generator of the set of heuristics,
along with the set of heuristic prototypes and constant parameters (or
assumptions). The two heuristic prototypes are presented in Algorithm 1 and
Algorithm 2. The constant parameters include: maximum total loss allowed in
pips, maximum take profit, maximum stop loss, take profit and stop loss steps,
and the broker’s spread.

The fist heuristic prototype, called the _trend continuation heuristic_ ,
denoted as $h_{1}$, simply signals a long trade at the end of a “green” candle
period, and it signals the short trade otherwise. The second heuristic
prototype, called the _trend reversal heuristic_ , denoted as $h_{2}$, signals
a short trade at the end of a “green” candle period, and it signals a long
trade otherwise. It is important to note that in order to avoid risks, only
one trade at a time can be placed in SmartFX model.

Algorithm 1 Trend Continuation Heuristic $h_{1}$

0: $O,H,L,C$

0: trade decision

1: if no current trades then

2: if $C>O$ then

3: return long trade

4: else if $C<0$ then

5: return short trade

6: else

7: return no trade

8: end if

9: end if

Algorithm 2 Trend Reversal Heuristic $h_{2}$

0: $O,H,L,C$

0: trade decision

1: if no current trades then

2: if $C<O$ then

3: return long trade

4: else if $C>0$ then

5: return short trade

6: else

7: return no trade

8: end if

9: end if

10: return no trade

When the set of heuristics is generated, they go to the trade simulation
algorithm along with the trade model (which represents the rules of the Forex
market) and the set of constraints (which represents the parameters of a Forex
trader). The trade model, constraints, and the set of heuristics are the same
for each subset of the market data. The subsets of market data are formed by
combining all the given instruments with all the given OHLC granularities.
Each subset of data can be independently processed by the trade simulation
algorithm, thus this algorithm can be run in a multi-threaded environment.

Each simulation thread maintains its own log of results, which is analyzed to
determine the set of optimal parameters. Then the sets of all optimal
parameters for each subset of the market data are processed manually in a
spreadsheet software.

##  VI SmartFX Implementation

SmartFX is a multi-threaded open-source tool written in Java, totalling 859
lines of code. The value of spread is fixed and set to 2.0 pips. The software
performs data preprocessing and generation of heuristics in a linear fashion.
However, the most computation-consuming part, the trade simulation learning,
is performed in a dedicated thread for each subset of the market data.

The range of the values of the stop loss (SL) parameter is set to [3,50]. The
same range is used for the take profit (TP) parameter, accounting for 2,209
distinct $(SL,TP)$ tuples. We stochastically determined that the largest
ranges are not likely to yield any combinations with better efficiency.
Moreover, higher values of TP reduce the number of trades, whereas larger
value of SL increases the risk of unexpected large loss of profit.

##  VII Data Set

We use 18 subsets of historical Forex data from the Dukascopy Swiss Banking
Group data
set111111https://www.dukascopy.com/swiss/english/marketwatch/historical/. The
subsets are tuples $(\eta,G)$, where $\eta$ is a Forex instrument, and $G$ is
a granularity, such that:

$\eta\in\\{EUR/USD,USD/JPY,EUR/JPY\\}$ $G\in\\{H1,H2,H4,H8,H12,D1\\}$

Each subset is stored in a comma-separated value (CSV) data file. Each CSV
entry includes the OHLC candlestick data. The entries also include the trade
volume data, but this parameter is ignored by SmartFX.

SmartFX records the following parameters for each of the 36 runs (18 subsets
of market data processed with each of the two heuristic prototypes) during the
learning simulation: a) total profit in pips; b) total loss in pips; c)
heuristic used; d) take profit (TP) policy simulated; e) stop loss (SL) policy
simulated; f) number of long trades; g) number of short trades; h) Forex
instrument; and i) granularity of OHLC data.

##  VIII Evaluation

###  VIII-A Setup and Run

We used a Dell PowerEdge T640 server with two Intel Xeon Gold 5218 processors
and 200 GB of RAM, running Ubuntu 18.04.03 LTS. The SmartFX tool was executed
in two separate processes, each running 18 threads (36 threads total). The
first process was resposible for the heuristic $h_{1}$, and the second process
was responsible for the heuristic $h_{2}$. The choice to run the tool in two
multi-threaded processes was determined by implementation convenience, and was
not intended to achieve any performance improvement. The length of the entire
cycle for all data subsets took about two hours.

###  VIII-B Results

The raw results of SmartFX execution are recorded in the Appendices, and below
are the most essential summaries and explanations.

####  VIII-B1 Worst Performers

Table III presents the four worst 10-year balances. It is important to note
that these cases are worst compared to other cases, but they are best cases
locally. For example, the first row indicates that the best combination of
take profit and stop loss values for the EUR/JPY instrument for OHLC H2
granularity and trade continuation heuristic could maximize the ending balance
only to the value of -75,380 pips. As we can see, all the four worst
performers use the trend continuation heuristic prototype. The performance of
the two heuristic prototypes is discussed later in this section.

TABLE III: Four simulations that resulted in lower 10-year ending balances. Instrument | Period | Heuristic | 10-year Balance (pips)
---|---|---|---
EUR/JPY | H2 | $h_{1}$ | -75,380
EUR/JPY | H4 | $h_{1}$ | -41,338
EUR/USD | H4 | $h_{1}$ | -39,953
EUR/USD | H2 | $h_{1}$ | -29,397

####  VIII-B2 Best Performers

The best four performers, shown in Table IV, all use the trend reversal
heuristic prototype. Also, the best four performance have finer OHLC
granularities than the four worst performing cases. As we can see from the
table, the SmartFX model can deliver on average up to 118 pips of profit. The
performance of the two heuristics is discussed in later in this section.

TABLE IV: Four simulations that resulted in best 10-year ending balances. Instrument | Period | Heuristic |  | 10-year
---
Balance (pips)
| Avg. pips
---
per day
EUR/JPY | H1 | h2 | 432,192 | 118.31
EUR/USD | H1 | h2 | 348,611 | 95.43
EUR/JPY | H2 | h2 | 302,022 | 82.68
USD/JPY | H1 | h2 | 293,121 | 80.24

####  VIII-B3 Number of Trades

Table V summarizes the number of trades statistics among the top 4 performers
discussed before in this section. We are interested in the number of trades
only for top performers because they are the ones to be used in real trades.
However, the information about the number of trades for the remaining cases is
available in Appendix -C.

TABLE V: Number of trades over the 10-year learning simulation period averaged between the four highest performing simulations, and also the calculated average number of trades per day among the top four performers. Average among the top four performers | Value | $\mathbf{\sigma}$
---|---|---
Number of trades over 10 years | 48,078 | 12,641
Average number of trades per day | 13.16 | —

####  VIII-B4 Heuristics Performance

The best possible performance of the two heuristics is summarized in Table VI.
As we can see, the performance of the trend reversal heuristic is much better
than of the trend continuation one. Also, we observe smaller average number of
trades for the trend continuation heuristic. It is also worth mentioning that,
as it is seen in Table VIII, among all the 21 cases with positive final
balances, only three are produced by the trend continuation heuristic, and all
these cases are among bottom five performers with positive balances.

TABLE VI: Summary of best possible performance of the two heuristic prototypes. | Heuristic
---
Prototype
| Average 10-year
---
Balance (pips)
| Avg. Number of
---
Trades over 10 years
$\mathbf{\sigma}$
$h_{1}$ | -10,574 | 15,101 | 13,997
$h_{2}$ | 186,869 | 19,007 | 18,399

####  VIII-B5 Optimal TP/SL Settings

Table VII is the culmination of this research, delivering the set of optimal
trading settings that allows to achieve the best profit results.

TABLE VII: Optimal TP/SL parameters for the top 7 most profitable simulation cases. Instrument | Period |  | Take Profit
---
(pips)
| Stop Loss
---
(pips)
EUR/JPY | H1 | 3 | 14
EUR/USD | H1 | 3 | 11
EUR/JPY | H2 | 3 | 20
USD/JPY | H1 | 3 | 9
EUR/USD | H2 | 3 | 16
EUR/JPY | H4 | 3 | 28
USD/JPY | H2 | 3 | 13

##  IX Discussion

SmartFX is purposefully designed to be broker-agnostic, which allows to
develop trading setups that are universally applicable to diverse trading
environments. However, this decision comes at a cost of the necessity to
specify a fixed spread value, whereas many Forex brokers use floating spread
policy. As a result, the user of SmartFX must specify the spread that is large
enough. Although the static spread value does not compromise the quality of
the result produced by SmartFX, the parameter must be chosen carefully in
order to avoid overly optimistic scenarios.

##  X Conclusion

Market trading optimization is a popular application of Artificial
Intelligence (AI), delivering endless opportunities for research and profound
insights. The Forex market is often being chosen as a target environment
because it allows unlimited spread-based day trading without additional fees,
long market hours, and easy entrance requirements. In this research, we
demonstrated that popular technical indicators may be ambiguous and less
optimal for automated trading. We developed SmartFX, an automated constraint-
based machine learning tool that determines optimal parameters for given
heuristic prototypes.

We evaluated 10 years worth of market data for 3 popular Forex instruments
using two simple price-action heuristic prototypes, along with 6 different
OHLC granularities of the input data. The results clearly demonstrated that
the trade reversal price-action heuristic is significantly more profitable
than the trade continuation, and capable of delivering up to 118 pips of
profit per day on average. We determined exact trading parameters for the
optimal trade, and thus achieved the goal of this research: to deliver
practically working, specific and evidence-based optimal trade parameters for
price-action Forex trading. We make SmartFX available under an open source
license for everyone to use, evaluate, and build upon.

## References

  * [1] D. Hudson, “Foreign exchange market,” Encyclopædia Britannica, 2017, accessed: 2019-12-07.
  * [2] G. F. Thompson, “Time, trading and algorithms in financial sector security,” _New Political Economy_ , vol. 22, no. 1, pp. 1–11, 2017.
  * [3] S. H. M. Yazdi and Z. H. Lashkari, “Technical analysis of forex by macd indicator,” _International Journal of Humanities and Management Sciences (IJHMS)_ , vol. 1, no. 2, pp. 159–165, 2013.
  * [4] Z. Liu and D. Xiao, “An automated trading system with multi-indicator fusion based on d-s evidence theory in forex market,” in _2009 Sixth International Conference on Fuzzy Systems and Knowledge Discovery_ , vol. 3, Aug 2009, pp. 239–243.
  * [5] R. Krishnan and S. S. Menon, “Impact of currency pairs, time frames and technical indicators on trading profit in forex spot market.” _International journal of Business insights & Transformation_, vol. 2, no. 2, 2009.
  * [6] Y. L. Yong, D. C. Ngo, and Y. Lee, “Technical indicators for forex forecasting: a preliminary study,” in _International Conference in Swarm Intelligence_. Springer, 2015, pp. 87–97.
  * [7] M. Ozturk, I. H. Toroslu, and G. Fidan, “Heuristic based trading system on forex data using technical indicator rules,” _Applied Soft Computing_ , vol. 43, pp. 170–186, 2016.
  * [8] S. H. M. Yazdi and Z. H. LASHKARI, “Technical analysis of forex by parabolic sar indicator,” in _International Islamic Accounting and Finance Conference_ , 2012.
  * [9] H. Ni and H. Yin, “Exchange rate prediction using hybrid neural networks and trading indicators,” _Neurocomputing_ , vol. 72, no. 13-15, pp. 2815–2823, 2009.
  * [10] P. Rosenstreich, _Forex Revolution: An Insider’s Guide to the Real World of Foreign Exchange Trading_. FT Press, 2005.
  * [11] T. S. Guides, “Personality strengths and weaknesses in forex trading,” 2017.
  * [12] R. F. de Brito and A. L. Oliveira, “Comparative study of forex trading systems built with svr+ ghsom and genetic algorithms optimization of technical indicators,” in _2012 IEEE 24th International Conference on Tools with Artificial Intelligence_ , vol. 1. IEEE, 2012, pp. 351–358.
  * [13] T. Ploysuwan and R. Chaisricharoen, “Gaussian process kernel crossover for automated forex trading system,” in _2017 14th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON)_ , June 2017, pp. 802–805.
  * [14] G. Wilson and W. Banzhaf, “Interday foreign exchange trading using linear genetic programming,” in _Proceedings of the 12th annual conference on Genetic and evolutionary computation_. ACM, 2010, pp. 1139–1146.
  * [15] R. M. C. Pinto and J. C. M. Silva, “Strategic methods for automated trading in forex,” in _2012 12th International Conference on Intelligent Systems Design and Applications (ISDA)_. IEEE, 2012, pp. 34–39.
  * [16] D. Vezeris, I. Karkanis, and T. Kyrgos, “Adturtle: An advanced turtle trading system,” _Journal of Risk and Financial Management_ , vol. 12, no. 2, p. 96, 2019.
  * [17] J. E. Schoen, “Automated trading system,” Jan. 4 2011, uS Patent 7,865,421.
  * [18] J. Chait, “Foreign exchange trading platform,” Feb. 22 2007, uS Patent App. 11/451,731.
  * [19] A. Peterson, L. Miller, I. Eaglesfield, S. Hubble, M. LeGelebart, T. Sablic, R. Sagurton, E. Theodorou, W. Wah, A. Zhou _et al._ , “Foreign exchange trading interface,” Mar. 17 2005, uS Patent App. 10/884,111.
  * [20] D. Ogg, W. Lamartin, and M. Smith, “Automated system for routing orders for foreign exchange transactions,” Nov. 3 2005, uS Patent App. 10/834,812.
  * [21] J. Owens and C. Awtry, “Automated trading system and software for financial markets,” Dec. 22 2005, uS Patent App. 11/156,428.
  * [22] A. Abraham and M. U. Chowdhury, “An intelligent forex monitoring system,” in _2001 International Conferences on Info-Tech and Info-Net. Proceedings (Cat. No.01EX479)_ , vol. 3, Oct 2001, pp. 523–528 vol.3.
  * [23] A. Abraham and M. U. Chowdhury, “An intelligent forex monitoring system,” in _2001 International Conferences on Info-Tech and Info-Net. Proceedings (Cat. No. 01EX479)_ , vol. 3. IEEE, 2001, pp. 523–528.
  * [24] P. Czekalski, M. Niezabitowski, and R. Styblinski, “Ann for forex forecasting and trading,” in _2015 20th International Conference on Control Systems and Computer Science_. IEEE, 2015, pp. 322–328.
  * [25] Y. Lee, L. C. O. Tiong, and D. C. L. Ngo, “Hidden markov models for forex trends prediction,” in _2014 International Conference on Information Science & Applications (ICISA)_. IEEE, 2014, pp. 1–4.
  * [26] R. P. Barbosa and O. Belo, “Multi-agent forex trading system,” in _Agent and multi-agent technology for internet and enterprise systems_. Springer, 2010, pp. 91–118.
  * [27] N. Shahbazi, M. Memarzadeh, and J. Gryz, “Forex market prediction using narx neural network with bagging,” in _MATEC Web of Conferences_ , vol. 68. EDP Sciences, 2016, p. 19001.
  * [28] I. Macerinskiene and A. Balciunas, “The evidence of social responsibility in foreign exchange brokers’ activities,” _Procedia-Social and Behavioral Sciences_ , vol. 213, pp. 552–556, 2015.

###  -A Resulting Balances

Table VIII shows the full list of ending balances for each of the learning
simulation cases. The column Reverse represents the heuristic: if it is
$FALSE$, $h_{1}$ is used, otherwise, $h_{2}$ is used. Depending on the
instrument, the multiplier for determining the number of pips in the third
column is either $10^{2}$ or $10^{4}$.

TABLE VIII: Resulting balances for all the simulations. If the value in the Reverse column is $FALSE$, the the heuristic $h_{1}$ is used, otherwise, the heuristic $h_{2}$ is used. The data is sorted in a descending order by the value of the third column. Instrument | Balance | Balance in Pips | Reversal
---|---|---|---
EURJPY-H1 | 4321.923 | 432192.3 | TRUE
EURUSD-H1 | 34.86112 | 348611.2 | TRUE
EURJPY-H2 | 3020.2202 | 302022.02 | TRUE
USDJPY-H1 | 2941.217 | 294121.7 | TRUE
EURUSD-H2 | 23.923656 | 239236.56 | TRUE
EURJPY-H4 | 2191.7815 | 219178.15 | TRUE
USDJPY-H2 | 1984.1091 | 198410.91 | TRUE
EURUSD-H4 | 17.544798 | 175447.98 | TRUE
EURJPY-H8 | 1635.9933 | 163599.33 | TRUE
EURJPY-H12 | 1460.575 | 146057.5 | TRUE
USDJPY-H4 | 1441.6462 | 144164.62 | TRUE
EURUSD-H8 | 12.996401 | 129964.01 | TRUE
EURUSD-H12 | 12.005339 | 120053.39 | TRUE
USDJPY-H8 | 1074.1615 | 107416.15 | TRUE
EURJPY-D1 | 963.3696 | 96336.96 | TRUE
USDJPY-H12 | 924.0237 | 92402.37 | TRUE
USDJPY-H1 | 872.2552 | 87225.52 | FALSE
EURUSD-D1 | 8.5002575 | 85002.575 | TRUE
USDJPY-D1 | 694.33014 | 69433.014 | TRUE
EURUSD-H1 | 5.1108055 | 51108.055 | FALSE
USDJPY-H2 | 183.90222 | 18390.222 | FALSE
EURJPY-H1 | -14.003174 | -1400.3174 | FALSE
EURJPY-D1 | -78.99794 | -7899.794 | FALSE
EURUSD-D1 | -0.8153805 | -8153.805 | FALSE
USDJPY-D1 | -81.747986 | -8174.7986 | FALSE
EURJPY-H12 | -148.27484 | -14827.484 | FALSE
EURUSD-H12 | -1.4840896 | -14840.896 | FALSE
USDJPY-H12 | -150.52483 | -15052.483 | FALSE
EURUSD-H8 | -2.1492684 | -21492.684 | FALSE
EURJPY-H8 | -215.3318 | -21533.18 | FALSE
USDJPY-H8 | -218.21217 | -21821.217 | FALSE
USDJPY-H4 | -257.88776 | -25788.776 | FALSE
EURUSD-H2 | -2.9397478 | -29397.478 | FALSE
EURUSD-H4 | -3.9952888 | -39952.888 | FALSE
EURJPY-H4 | -413.3832 | -41338.32 | FALSE
EURJPY-H2 | -753.8009 | -75380.09 | FALSE

###  -B Stop Loss and Take Profit Configurations

Table IX contains the most optimal take profit and stop loss parameters for
all simulation cases that ended up with positive balances. Recording the cases
with negative balances was not necessary because, by all means, these cases
cannot be defined as successful, and therefore the values of most optimal take
profit and stop loss parameters are never practically useful.

TABLE IX: Most profitable take profit (TP) and stop loss (SL) settings for all learning simulation cases that ended up with a positive balance. Simulation | SL | TP | Reverse
---|---|---|---
EURJPY-H1 | 0.1400 | 0.0300 | TRUE
EURUSD-H1 | 0.0011 | 0.0003 | TRUE
EURJPY-H2 | 0.2000 | 0.0300 | TRUE
USDJPY-H1 | 0.0900 | 0.0300 | TRUE
EURUSD-H2 | 0.0016 | 0.0003 | TRUE
EURJPY-H4 | 0.2800 | 0.0300 | TRUE
USDJPY-H2 | 0.1300 | 0.0300 | TRUE
EURUSD-H4 | 0.0024 | 0.0003 | TRUE
EURJPY-H8 | 0.4500 | 0.0300 | TRUE
EURJPY-H12 | 0.4800 | 0.0300 | TRUE
USDJPY-H4 | 0.1800 | 0.0300 | TRUE
EURUSD-H8 | 0.0038 | 0.0003 | TRUE
EURUSD-H12 | 0.0047 | 0.0003 | TRUE
USDJPY-H8 | 0.2700 | 0.0300 | TRUE
EURJPY-D1 | 0.5000 | 0.0300 | TRUE
USDJPY-H12 | 0.3500 | 0.0300 | TRUE
USDJPY-H1 | 0.0300 | 0.5000 | FALSE
EURUSD-D1 | 0.0050 | 0.0003 | TRUE
USDJPY-D1 | 0.5000 | 0.0300 | TRUE
EURUSD-H1 | 0.0004 | 0.0050 | FALSE
USDJPY-H2 | 0.0600 | 0.5000 | FALSE

###  -C Number of Trades

Table X contains total number of trades for ten years, and also a calculated
value of an average number of trades expected per one day, which is simply the
number in the second column divided by $7\cdot 365+3\cdot 366$ An important
note about the days when the market is closed: normally, during these days
there are no trades, so, depending on what the result is used for, the number
in the second column may be instead divided by $(7\cdot 365+3\cdot
366)\times{5\over 7}$.

TABLE X: Number of trades for each of the simulation learning cases, including the calculated average trades per day value. Simulation | Number of Trades | Average Trades per Day
---|---|---
EURJPY-H1 | 56078 | 15.35
EURUSD-H1 | 53299 | 14.59
EURJPY-H2 | 29205 | 7.99
USDJPY-H1 | 53729 | 14.71
EURUSD-H2 | 27823 | 7.62
EURJPY-H4 | 15318 | 4.19
USDJPY-H2 | 28013 | 7.67
EURUSD-H4 | 14670 | 4.02
EURJPY-H8 | 7977 | 2.18
EURJPY-H12 | 5594 | 1.53
USDJPY-H4 | 14838 | 4.06
EURUSD-H8 | 7821 | 2.14
EURUSD-H12 | 5515 | 1.51
USDJPY-H8 | 7786 | 2.13
EURJPY-D1 | 3062 | 0.84
USDJPY-H12 | 5425 | 1.49
USDJPY-H1 | 48709 | 13.33
EURUSD-D1 | 3007 | 0.82
USDJPY-D1 | 2967 | 0.81
EURUSD-H1 | 43709 | 11.97
USDJPY-H2 | 18504 | 5.07
EURJPY-H1 | 37402 | 10.24
EURJPY-D1 | 3123 | 0.85
EURUSD-D1 | 3126 | 0.86
USDJPY-D1 | 3118 | 0.85
EURJPY-H12 | 5714 | 1.56
EURUSD-H12 | 5723 | 1.57
USDJPY-H12 | 5711 | 1.56
EURUSD-H8 | 8314 | 2.28
EURJPY-H8 | 8314 | 2.28
USDJPY-H8 | 8300 | 2.27
USDJPY-H4 | 9440 | 2.58
EURUSD-H2 | 13397 | 3.67
EURUSD-H4 | 16087 | 4.40
EURJPY-H4 | 16096 | 4.41
EURJPY-H2 | 17044 | 4.67


0}\bra{\alpha^{\prime}_{+}}\frac{1}{\sqrt{\Tr_{B}\omega}}=\bra{\alpha^{\prime}_{+}},\qquad\lim_{\varepsilon\to
0}\sqrt{\frac{\varepsilon}{2}}\bra{\alpha^{\prime}_{-}}\frac{1}{\sqrt{\Tr_{B}\omega}}=\frac{1}{\sqrt{\absolutevalue{u_{22}}^{2}+\absolutevalue{u_{32}}^{2}}}\bra{\alpha^{\prime}_{-}}.$
(112)

With this, we have

$\lim_{\varepsilon\to
0}\sqrt{\omega}\pqty{\frac{1}{\sqrt{\Tr_{B}\omega}}\otimes\mathbb{1}}=\outerproduct{\alpha^{\prime}_{+},\beta^{\prime}_{+}}{\alpha^{\prime}_{+},\beta^{\prime}_{+}}+\frac{1}{\sqrt{1-\absolutevalue{u_{12}}^{2}}}\pqty{\outerproduct{\omega_{\perp}}{\omega_{\perp}}-\frac{1}{u_{12}^{*}}\outerproduct{\omega_{\perp}}{\alpha^{\prime}_{+},\beta^{\prime}_{-}}}.$
(113)

As such, the action of the limit of
$\bqty{\pqty{\frac{1}{\sqrt{\Tr_{B}\omega}}\bullet\frac{1}{\sqrt{\Tr_{B}\omega}}}\otimes\mathbbm{1}}\sqrt{\omega}$
can be found to be

$\displaystyle\lim_{\varepsilon\to
0}\sqrt{\omega}\bqty{\pqty{\frac{1}{\sqrt{\Tr_{B}\omega}}\bullet\frac{1}{\sqrt{\Tr_{B}\omega}}}\otimes\mathbbm{1}}\sqrt{\omega}$
$\displaystyle=\outerproduct{\alpha^{\prime}_{+},\beta^{\prime}_{+}}{\alpha^{\prime}_{+}}\bullet\outerproduct{\alpha^{\prime}_{+}}{\alpha^{\prime}_{+},\beta^{\prime}_{+}}+\outerproduct{\omega_{\perp}}{\alpha^{\prime}_{-}}\bullet\outerproduct{\alpha^{\prime}_{-}}{\omega_{\perp}}$
(114)
$\displaystyle\qquad{}+{}\frac{u_{22}}{\sqrt{\absolutevalue{u_{22}}^{2}+\absolutevalue{u_{32}}^{2}}}\outerproduct{\alpha^{\prime}_{+},\beta^{\prime}_{+}}{\alpha^{\prime}_{+}}\bullet\outerproduct{\alpha^{\prime}_{-}}{\omega_{\perp}}$
$\displaystyle\qquad{}+{}\frac{u_{22}^{*}}{\sqrt{\absolutevalue{u_{22}}^{2}+\absolutevalue{u_{32}}^{2}}}\outerproduct{\omega_{\perp}}{\alpha^{\prime}_{-}}\bullet\outerproduct{\alpha^{\prime}_{+}}{\alpha^{\prime}_{+},\beta^{\prime}_{+}}.$

Finally, by taking the partial trace, we find

$\displaystyle\hat{\mathcal{E}}_{\outerproduct{\alpha_{+}}{\alpha_{+}},2}\bqty{\outerproduct{\alpha^{\prime}_{+}}{\alpha^{\prime}_{+}}}=\lim_{\varepsilon\to
0}\hat{\mathcal{E}}_{\alpha_{\varepsilon}}\bqty{\outerproduct{\alpha^{\prime}_{+}}{\alpha^{\prime}_{+}}}$
$\displaystyle=\outerproduct{\alpha_{+}}{\alpha_{+}},$ (115)
$\displaystyle\hat{\mathcal{E}}_{\outerproduct{\alpha_{+}}{\alpha_{+}},2}\bqty{\outerproduct{\alpha^{\prime}_{-}}{\alpha^{\prime}_{-}}}=\lim_{\varepsilon\to
0}\hat{\mathcal{E}}_{\alpha_{\varepsilon}}\bqty{\outerproduct{\alpha^{\prime}_{-}}{\alpha^{\prime}_{-}}}$
$\displaystyle=\outerproduct{\alpha_{-}}{\alpha_{-}},$
$\displaystyle\hat{\mathcal{E}}_{\outerproduct{\alpha_{+}}{\alpha_{+}},3}\bqty{\outerproduct{\alpha^{\prime}_{+}}{\alpha^{\prime}_{-}}}=\lim_{\varepsilon\to
0}\hat{\mathcal{E}}_{\alpha_{\varepsilon}}\bqty{\outerproduct{\alpha^{\prime}_{+}}{\alpha^{\prime}_{-}}}$
$\displaystyle=\frac{u_{22}}{\sqrt{\absolutevalue{u_{22}}^{2}+\absolutevalue{u_{32}}^{2}}}\outerproduct{\alpha_{+}}{\alpha_{-}}.$

(3) _Naïvely define
$\hat{\mathcal{E}}_{\outerproduct{\alpha_{+}}{\alpha_{+}},3}$ with product-
preserving tuples_. In terms of $u_{jk}$, the action of
$\hat{\mathcal{E}}_{\outerproduct{\alpha_{+}}{\alpha_{+}},3}$ as defined in
Eq. (88) can be worked out directly to be

$\displaystyle\hat{\mathcal{E}}_{\outerproduct{\alpha_{+}}{\alpha_{+}},3}\bqty{\outerproduct{\alpha^{\prime}_{+}}{\alpha^{\prime}_{+}}}$
$\displaystyle=\outerproduct{\alpha_{+}}{\alpha_{+}},$ (116)
$\displaystyle\hat{\mathcal{E}}_{\outerproduct{\alpha_{+}}{\alpha_{+}},3}\bqty{\outerproduct{\alpha^{\prime}_{-}}{\alpha^{\prime}_{-}}}$
$\displaystyle=\pqty{1-\absolutevalue{u_{21}}^{2}}\outerproduct{\alpha_{-}}{\alpha_{-}}+\absolutevalue{u_{21}}^{2}\outerproduct{\alpha_{+}}{\alpha_{+}}+u_{21}^{*}u_{23}\outerproduct{\alpha_{+}}{\alpha_{-}}+u_{21}u_{23}^{*}\outerproduct{\alpha_{+}}{\alpha_{-}},$
$\displaystyle\hat{\mathcal{E}}_{\outerproduct{\alpha_{+}}{\alpha_{+}},3}\bqty{\outerproduct{\alpha^{\prime}_{+}}{\alpha^{\prime}_{-}}}$
$\displaystyle=u_{22}\outerproduct{\alpha_{+}}{\alpha_{-}}.$

Once again, we find that the three approaches result in different retrodiction
channels. Meanwhile, Eq. (116) agrees with Eq. (115) if and only if
$u_{21}=u_{12}=0$, or $u_{21}=u_{22}=0$. Therefore,
$\hat{\mathcal{E}}_{\outerproduct{\alpha_{+}}{\alpha_{+}},2}\neq\hat{\mathcal{E}}_{\outerproduct{\alpha_{+}}{\alpha_{+}},3}$
in general, so we cannot naïvely assume Eq. (31) to hold for pure output
states.

###  F.2 Product-preserved tuples are almost tabletop time-reversible with the
pseudoinverse convention

Although we have not given a preference to any particular approach for
retrodicting with rank-deficient outputs, it is worth pointing out that the
product-preserved tuples are almost tabletop time-reversible with the
pseudoinverse convention.

Given a product-preserving tuple $(U,\alpha,\beta)$ with
$U(\alpha\otimes\beta)U^{\dagger}=\alpha^{\prime}\otimes\beta^{\prime}$ where
$\alpha^{\prime}$ is not necessarily full-rank, Eq. (19) with the
pseudoinverse reads

$\displaystyle\hat{\text{T}}\text{r}_{B,\alpha^{\prime}\otimes\beta^{\prime}}[\bullet]$
$\displaystyle=\sqrt{\alpha^{\prime}}\sqrt{\alpha^{\prime}}^{\,\bm{\pmb{+}}}\bullet\sqrt{\alpha^{\prime}}^{\,\bm{\pmb{+}}}\sqrt{\alpha^{\prime}}\otimes\beta^{\prime}$
(117)
$\displaystyle=\Pi_{\alpha^{\prime}}\bullet\Pi_{\alpha^{\prime}}\otimes\beta^{\prime},$

which in turn gives a trace-non-increasing retrodiction channel

$\hat{\mathcal{E}}_{\alpha}[\bullet]=\Tr_{B}\bqty{U^{\dagger}\pqty{\Pi_{\alpha^{\prime}}\bullet\Pi_{\alpha^{\prime}}\otimes\beta^{\prime}}U}.$
(118)

We say that it is _almost_ tabletop time-reversible as Eq. (118) is the same
as Eq. (25) up to a projection onto the support of $\mathcal{E}[\alpha]$ upon
the input state.


# Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries
Through Blinded Reviewers and Text Classification Algorithms

Mayank Soni
ADAPT Centre,
Trinity College Dublin
<EMAIL_ADDRESS>
Vincent Wade
ADAPT Centre,
Trinity College Dublin
<EMAIL_ADDRESS>

###### Abstract

Large Language Models (LLMs) have gathered significant attention due to their
impressive performance on a variety of tasks. ChatGPT, developed by OpenAI, is
a recent addition to the family of language models and is being called a
disruptive technology by a few, owing to its human-like text-generation
capabilities. Although, many anecdotal examples across the internet have
evaluated ChatGPT’s strength and weakness, only a few systematic research
studies exist. To contribute to the body of literature of systematic research
on ChatGPT, we evaluate the performance of ChatGPT on Abstractive
Summarization by the means of automated metrics and blinded human reviewers.
We also build automatic text classifiers to detect ChatGPT generated
summaries. We found that while text classification algorithms can distinguish
between real and generated summaries, humans are unable to distinguish between
real summaries and those produced by ChatGPT.

##  1 Introduction

ChatGPT is a recent addition to the family of large language models.
Specifically, it is created by extending the work on InstructGPT Ouyang et al.
(2022) with a dialog based user-interface that is fine-tuned using
Reinforcement Learning with Human Feedback (RLHF) Christiano et al. (2017)
111https://beta.openai.com/docs/model-index-for-researchers. ChatGPT is
estimated to have reached about $100$ million active users Hu (2023) and is
being widely used by business’s and customers to accomplish various textual
tasks. ChatGPT is easy to use due to it’s dialog interface and large scale
training via RLHF. Part of its popularity is also abilitieswork, hitherto
unseen, such as code generation. ChatGPT has an ability to interact with users
in a conversation-style manner and can intelligently "answer follow-up
questions, admit its mistakes, challenge incorrect premises and reject
inappropriate requests" Schulman et al. (2022)

However, despite its seemingly magical abilities, anecdotal reports on ChatGPT
have suggested significantly remaining challenges - for example it fails in
elementary mathematical tasks (Frieder et al., 2023). Open AI has listed many
limitations of ChatGPT on its website expressing via a tweet, the CEO tweeted
Its a mistake to be relying on ChatGPT for anything important Altman (2022).
Despite this, there are commercial web based applications that are running on
ChatGPT as background. ChatGPT has been found to hallucinate knowledge and
frame answers confidently and incorrectly (Shen et al., 2023; Thorp, 2023).
Despite its widespread usage systematic studies of ChatGPT’s evaluation are
scarce. Towards this end, in this study, we evaluate if summaries generated by
ChatGPT can be distinguished from real summaries by humans and text-
classification algorithms and evaluate generated summaries through automated
metrics. We first prepare a dataset of $50$ summaries generated by ChatGPT and
then analyse how ChatGPT summaries are scored across various automated
metrics, then evaluate if blinded human reviewers can distinguish between real
and generated summaries and finally build a text classifier to detect the two
sources. The sections below discuss related work, data gathering, prompting
and results.

##  2 Related work

We briefly survey related work in the areas of ChatGPT evaluation and enlist
some important studies conducted in Summarization.

### ChatGPT Evaluation

Since the release of ChatGPT, there have been many anecdotal evaluation of
ChatGPT’s abilities and weaknesses posted online but ChatGPT strength and
weakness is still a small research area. Jiao et al. (2023) evaluated
ChatGPT’s ability on machine translation and found that ChatGPT performs
competitively with commercial translation products (e.g., Google Translate) on
high-resource European languages but lags behind significantly on low-
resource or distant languages.. Kung et al. (2022) evaluated ChatGPT’s
performance on United States Medical License Examination (USMLE) and found
that ChatGPT performed at or near the passing threshold for all three exams
without any specialized training or reinforcement. Bang et al. (2023) is
perhaps the most comprehensive evaluation of ChatGPT: the subsection on
studies on summarization show that Interactivity(recursively generating
summaries with modfified instruction) highlight that interactivity improves
ROUGE 1 score. Gao et al. (2022)’s work on comparing ChatGPT generated
scientific abstracts is similar to our work. In this investigation, authors
generated and compared $10$ Abstracts in specific journal format and evaluated
the generated and original abstracts through an AI output detector, a
plagiarism detector and blinded human reviewers. The study found that ChatGPT
writes believable scientific abstracts, though with completely generated data.
These are original without any plagiarism detected but are often identifiable
using an AI output detector and skeptical human reviewers.

### Text Summarization

Summarization is a task of shortening a large text to a smaller version while
retaining the main information. There are two broad approaches to
summarization: Extractive and Abstractive. Extractive summaries contain as is
parts from the original text while abstractive summaries can contain novel
words and sentences not found in original text, like human-written summaries.
Neural sequence-to-sequence models can generate _abstractive_ summaries
(meaning the summary generated is not limited to selecting and rearranging
text from the original passage) and there is substantial literature on Neural
Abstractive Summarization(See et al., 2017; Rush et al., 2015; Nallapati et
al., 2016b; Chopra et al., 2016; Lewis et al., 2019). Recent approaches have
seen Large Language Models(LLM) being utilised Devlin et al. (2018); Yang et
al. (2019); Raffel et al. (2019); Liu et al. (2019) and performance on
summarization reported as standard. In this work, we focus on generating
abstractive summaries from ChatGPT and compare it to original summaries. We
discuss the precise instructions and dataset curated in section $3$ and $4$
below.

##  3 Dataset

We utilise $50$ articles from CNN News/Daily News Dataset Nallapati et al.
(2016a); Hermann et al. (2015). The articles were selected from version
$3.0.0$ of the test-set. CNN/Daily News Dataset contains $287,226$ training
pairs, $13,368$ validation pairs and $11,490$ test pairs. This dataset
contains multi-sentence summaries ordered according to the events described in
the source article. To prepare a dataset of generated summaries real
summaries, we prompted ChatGPT Feb 13 Version (ope, ) with $50$ articles from
the CNN News/Daily News Dataset Nallapati et al. (2016a); Hermann et al.
(2015) with the following prompt: Generate a short summary of the following
paragraph in as less words as possible, the details on why we used this prompt
are in the paragraph below. We are also releasing the generates summaries at
222github.com/Mayanksoni20/ChatGPT_summaries.

### Prompting

ChatGPT is an conversation style, instruction-based large language model and
the output can depend on the prompt provided. This means two instructions with
similar goals can produce different outputs. Keeping this in mind, our goal
with selecting a prompt was to produce summaries as similar as possible to the
real summaries. Towards this goal, we experimented with a few prompts and
finally arrived at the prompt highlighted in bold in Table 1. Other prompts
rendered summaries which were longer than real summaries.

Prompt Examples
---
Generate a short summary of the following paragraph in as less words as
possible
$\Uparrow$
Generate abstractive summary of the following paragraph
$\Uparrow$
Generate highlight for the following paragraph
$\Uparrow$
Reproduce the abstractive summary in as less words as possible
$\Uparrow$
Reproduce the abstractive summary in a maximum of 5 sentences
$\Uparrow$
Generate abstractive summary of the following paragraph in 60 words or less
$\Uparrow$
Produce a short summarize of the following paragraph
$\Uparrow$
Summarize of the following paragraph in 200 words
Table 1: Prompt trials for ChatGPT to obtain responses similar on
CNN/Dailymail dataset summaries.

##  4 Results

We present the results obtained in automated metrics, human evaluation and
text detection in below paragraphs.

### Automated Metrics

To begin with evaluation, we measure ChatGPT’s performance on abstractive
summarization by comparing original and generated summaries. We utilize the
following metrics: ROUGE Lin (2004) reporting the F1 scores for ROUGE-1,
ROUGE-2 and ROUGE-L and METEOR Banerjee and Lavie (2005)), measuring the word-
overlap, bigram-overlap, longest common sequence and unigram overlap between
the real and generated summaries. We obtain the scores using HuggingFace
evaluate library 333https://github.com/huggingface/evaluate . We report the
results in Table 2. The ROUGE and METEOR scores obtained will be compared to
other baseline models in the future versions.

ROUGE-1 | ROUGE-2 | ROUGE-L | ROUGELSUM | METEOR
---|---|---|---|---
0.30 | 0.11 | 0.20 | 0.21 | 0.35

Table 2: Automated metrics results between real and generated summaries

### Human Evaluation

Next, we wanted to evaluate if blinded reviewers were able to distinguish
between generated and original summaries. The task of the reviewers was
formulated as follows: we asked reviewers to first read the news article
followed by reading the summary and then click on either of two radio buttons
guessing if the summary was generated by ChatGPT or human. Figure 1 shows a
design of the user interface based on google forms. Two volunteers (native
speakers of english) were given $50$ random summaries each. The randomisation
worked in such a way that the same volunteer did not receive both (generated
and original) summaries of the same article. We found that our reviewers were
not able to distinguish between generated and human summaries. The accuracy of
human reviewers was reported to be $0.49$.

|  | Truth
---|---|---
|  | Generated | Original
Reviewers | Generated | $33$ | $29$
Original | $20$ | $15$

Table 3: Reviewer Scores in Identifying Original and Generated Summaries

Table 3 shows the confusion matrix of the results obtained. Reviewers also
commented that they were sure that they were not able to guess between
generated and original summaries, highlighting that the generated and original
summaries were similar.

Figure 1: User Interface of Summaries Shown to Human Reviewers

### ChatGPT Summary Detection

It is very important to be able to distinguish between ChatGPT generated and
human produced text to detect ChatGPT generated text. In this next step, we
wanted to evaluate if a fine-tuned text classification model is able to
distinguish between ChatGPT and Human-generated summaries. We employ
DistillBERT from HuggingFace and fine-tune it on our dataset. After Fine-
tuning for $2$ epochs with a learning rate of $0.0000002$, we could achieve an
accuracy of $90\%$ in distinguishing between generated and original summaries.
We also utilise Sentence Embedding Reimers and Gurevych (2019) and pass them
through XGBoost Chen and Guestrin (2016) and obtain an accuracy of $0.50$.
Table 3 shows the results obtained from using the algorithms. We leave the
implementation of other algorithms for future work.

Algorithm | Accuracy | F-1 Score
---|---|---
SentTrans. +XGB | 0.50 | 0.60
Distill-BERT | 0.90 | 0.33
Table 4: Results of ChatGPT Summary Classification

##  5 Limitations

The study conducted here has few restrictions. First, there was a limitation
of $50$ on the number of summaries that could be compared. Second, we haven’t
compared the summaries produced by various prompts, and there may be other
prompts that can be used to generate summaries. Third, we have not yet
contrasted ChatGPT’s summarization performance with that of other models and
baselines. Fourth, both of our reviewers were native English speakers;
however, it may be worthwhile to examine whether non-native speakers predict
real and generated summaries differently. Fifth, the accuracy of automatic
summary detection can be increased by using more sophisticated algorithms.

##  6 Discussion and Conclusion

In this concise study, through comparing 50 real and generated summaries, we
found that while text classification algorithms can distinguish between real
and generated summaries, humans are unable to distinguish between real
summaries and those produced by ChatGPT. Our reviewers were not certain
whether a summary was produced by ChatGPT or a human. The reviewers also
commented that they were confident that they were incorrectly guessing the
source of the summaries. We attribute this to lack of any distinguishing
feature between the two sources which in-turn is attributed to our careful
selection of prompts to generate summaries as similar as possible to original
summaries. Additionally, we were able to identify ChatGPT-generated summaries
with a 90% accuracy.

## Acknowledgements

This work was conducted with the financial support of the Science Foundation
Ireland Centre for Research Training in Digitally-Enhanced Reality (d-real)
under Grant No. 18/CRT/6224 and Science Foundation Ireland ADAPT Centre under
Grant. No. 13/RC/2106.

## References

  * (1) ChatGPT — Release Notes | OpenAI Help Center — help.openai.com.  https://help.openai.com/en/articles/6825453-chatgpt-release-notes.  [Accessed 30-Mar-2023].
  * Altman (2022) Sam Altman. 2022.  Chatgpt is incredibly limited, but good enough at some things to create a misleading impression of greatness.it’s a mistake to be relying on it for anything important right now. it’s a preview of progress; we have lots of work to do on robustness and truthfulness.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  _ArXiv_ , abs/2302.04023.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system.  _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_.
  * Chopra et al. (2016) Sumit Chopra, Michael Auli, and Alexander M Rush. 2016.  Abstractive sentence summarization with attentive recurrent neural networks.  In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies_ , pages 93–98.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Frieder et al. (2023) Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023\.  Mathematical capabilities of chatgpt.  _arXiv preprint arXiv:2301.13867_.
  * Gao et al. (2022) Catherine A Gao, Frederick M Howard, Nikolay S Markov, Emma C Dyer, Siddhi Ramesh, Yuan Luo, and Alexander T Pearson. 2022.  Comparing scientific abstracts generated by chatgpt to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers.  _bioRxiv_ , pages 2022–12.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  _Advances in neural information processing systems_ , 28.
  * Hu (2023) Krystal Hu. 2023.  Chatgpt sets record for fastest-growing user base - analyst note.
  * Jiao et al. (2023) Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023.  Is chatgpt a good translator? a preliminary study.  _ArXiv_ , abs/2301.08745.
  * Kung et al. (2022) Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng. 2022.  Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models.  _PLOS Digital Health_ , 2.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Nallapati et al. (2016a) Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016a.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  In _Conference on Computational Natural Language Learning_.
  * Nallapati et al. (2016b) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016b.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  _arXiv preprint arXiv:1602.06023_.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_.
  * Raffel et al. (2019) Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _ArXiv_ , abs/1910.10683.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Rush et al. (2015) Alexander M Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  _arXiv preprint arXiv:1509.00685_.
  * Schulman et al. (2022) J Schulman, B Zoph, C Kim, J Hilton, J Menick, J Weng, JFC Uribe, L Fedus, L Metz, M Pokorny, et al. 2022.  Chatgpt: Optimizing language models for dialogue.
  * See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  _arXiv preprint arXiv:1704.04368_.
  * Shen et al. (2023) Yiqiu Shen, Laura Heacock, Jonathan Elias, Keith D Hentel, Beatriu Reig, George Shih, and Linda Moy. 2023.  Chatgpt and other large language models are double-edged swords.
  * Thorp (2023) H Holden Thorp. 2023.  Chatgpt is fun, but not an author.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Neural Information Processing Systems_.