# Detecting Pump&Dump Stock Market Manipulation from Online Forums

D. Nam D.B. Skillicorn
School of Computing
Queen’s University
Kingston. Canada
<EMAIL_ADDRESS>
###### Abstract

The intersection of social media, low-cost trading platforms, and naive
investors has created an ideal situation for information-based market
manipulations, especially pump&dumps. Manipulators accumulate small-cap
stocks, disseminate false information on social media to inflate their price,
and sell at the peak. We collect a dataset of stocks whose price and volume
profiles have the characteristic shape of a pump&dump, and social media posts
for those same stocks that match the timing of the initial price rises. From
these we build predictive models for pump&dump events based on the language
used in the social media posts.

There are multiple difficulties: not every post will cause the intended market
reaction, some pump&dump events may be triggered by posts in other forums, and
there may be accidental confluences of post timing and market movements.
Nevertheless, our best model achieves a prediction accuracy of 85% and an
F1-score of 62%. Such a tool can provide early warning to investors and
regulators that a pump&dump may be underway.

##  1 Introduction

New financial products and technologies have allowed naive investors to easily
enter financial markets. This has increased the risk of manipulation, and
detecting and investigating fraudulent activities has become much more
difficult. Many go undetected [8].

Social media has created new methods for manipulating markets. A scheme known
as _Pump and Dump_ (P&D) is one popular mechanism. Fraudsters buy quantities
of a stock, disseminate false information about it to artificially raise its
price, and then sell their purchased shares at the higher price. Social media
provides a channel for rapid dissemination and a pool of investors with little
knowledge or experience who may not detect that the information is false.

Conventional approaches to detecting manipulation look for known patterns, and
for anomalous activity such as exceeded thresholds for prices and trading
volumes. Suspicious activities can be detected using sets of rules and
triggers that cause notifications of potential manipulation. However, those
methods struggle in the presence of behaviours that deviate from historical
patterns [16]. Previous work has also focused on detecting manipulations so
that regulators can penalise those who carry them out. This does little to
help investors, either to prevent their being deceived or recovering their
investments.

Data-analytic techniques have the potential to detect false information as it
being disseminated [11, 25]. Natural language analytics can detect the posts
in social media that are intended to pump particular stocks, providing a real-
time warning to potential investors. We investigate how well P&D schemes can
be detected in posts on social media, by matching the language patterns in the
posts to the pattern of stock price corresponding to a P&D manipulation.

A penny stock is a stock that is traded by a small public company for less
than $5 per share [24]. Many of these companies are known for their volatility
due to their limited coverage by analysts and interest from institutional
buyers. Because of their low price, retail investors can buy large quantities
of these stocks without having to invest much money. This, however, makes
their prices volatile and so creates the potential for large returns on
investments; but also leaves them vulnerable to manipulation by malicious
actors. One study found that 50% of manipulated stocks are those with a small
market capitalization [1].

It might be supposed that the connection between a social media post and a P&D
event is too tenuous to be detected – after all, not every post will have the
desired effect, and a P&D might be triggered by some less visible social media
activity. We show that, at least for penny stocks, the connection is
reasonably detectable, and we achieve prediction accuracies (that a post is
intended to cause a P&D event) of 85%, with an F1 score of 67% ($\pm$ 12
percentage points) from posts alone, and 62% ($\pm$ 3 percentage points) from
posts and comments.

##  2 Tools

Stance detection is a technique to determine the attitude or viewpoint of a
text towards a target. It aims to detect whether the author of the text is in
support of or against a given entity [21]. Some applications of stance
detection have been in political debates, fake news, and social media [15, 26,
30].

Empath is a tool that was developed by Fast et al. [13] for researchers to
generate and validate new lexical categories on demand. It uses deep learning
to establish connections between words and phrases used in modern fiction.
Given a small set of seed words that represents a category, Empath can provide
new related terms using its neural embeddings. It also employs the use of
crowd-sourcing to validate the terms that it considers are related. Along with
the ability to create new categories, Empath comes with 200 built-in, pre-
validated categories for common topics (e.g., neglect, government, social
media).

SHAP (SHapley Additive exPlanation) is a tool that was developed by Lundberg
and Lee [22] to determine the impact of each attribute on the output of a
predictive model. It is based on Shapley values, a concept from game theory
that determines a fair way to distribute the payoff for players that have
worked in coalition towards an outcome [33].

Extreme Gradient Boosting is a decision-tree based ensemble algorithm that has
become known for its speed and performance [5]. Decision trees are built
sequentially so that each one reduces the errors of the previous one [35].
Random Forests is a decision-tree based ensemble algorithm with each tree
built from a subset of the rows and columns of the dataset [34]. This allows
for variation among the trees and results in lower correlation among their
predictions [37]. Support Vector Machines are a supervised learning algorithm
that finds a hyperplane that best separates the data points from two classes
[14].

Artificial Neural Networks are computational networks that are inspired by the
biological nervous system [10]. ANNs excel at prediction for data where the
amount of information in each attribute is small and there are non-linear
interactions among them. Deep learning models are a class of extensions to
ANNs that have solved long standing prediction problems in image recognition
and natural language [20]. Convolutional Neural Networks (CNNs) are a class of
deep learning networks that were designed initially to work with images but
work surprisingly well with sequence data such as texts as well. Long Short-
Term Memory (LSTM) deep learning networks are a type of recurrent neural
network designed to handle the long-term dependencies present in sequence
prediction problems [4]. Understanding text often requires looking ahead
(think of verbs in German) and so processing text in both directions, using a
biLSTM, provides better results for language [6].

##  3 Experiments

Within a typical online forum, there are two different categories of texts.
The first is a _post_ , which initiates a discussion. The second is a set of
_comments_ responding to the post. For example, an individual may post saying
that, in their opinion, a stock’s price is about to rise, with others respond
by sharing their opinions in the same thread. Responders may agree with the
original post, or disagree.

P&D is an information-based manipulation, artificially raising the price of a
stock through the dissemination of false information. As shown in Figure 1,
this manipulation strategy involves three different stages [19]. The operators
of the scheme first purchase the stock that they are planning to manipulate
(Accumulation). Once they have acquired enough shares, they will disseminate
false information to make it appear more desirable, driving up the price
(Pump). Once the price has risen to the desired level of profit, the operators
sell off their shares before anyone uncovers that the information has no basis
or the hype dies down (Dump).

Figure 1: Stages of Pump and Dump

To identify P&Ds within the market, patterns associated with the scheme must
be established. While the method of conducting a P&D may vary, two indicators
that can identify them are sharp changes in price and volume [19]. A P&D will
cause a significant price increase within a short amount of time, larger than
the fluctuations that the stock typically experiences; followed by a decrease
once the dump phase has begun. The volume also increases as the stock gains
interest among investors during and after the dissemination phase. However,
the volume will typically not immediately experience as sharp a decline as the
price when the operators begin to dump their shares because of the reluctance
of investors to believe that the price is illusory.

If the profile of a P&D manipulation can be detected in the market, then the
post that putatively caused it can be straightforwardly labelled and its
language patterns investigated. (Of course, it is possible that some of the
apparent connections are spurious, but it is relatively unlikely that a post
touting a particular stock will be disseminated exactly when the stock’s price
and volume begin a sharp rise).

Labelling comments is more complex, since the comments may agree with the
original post, or disagree. Only the language of those that agree can
contribute to predicting a P&D event.

###  3.1 Data Sources

Two different data sources were utilized. The first is the popular online
website Reddit, where users discuss the stock market. The second is Yahoo
Finance, a financial market website that provides historical data about
companies.

Reddit contains forums referred to as subreddits, each dedicated to the
discussion of a specific topic. Popular forums for the discussions of stocks
are r/pennystocks, r/wallstreetbets, r/stocks, r/RobinHoodPennyStocks,
r/TheWallStreet. We use r/pennystocks and r/RobinHoodPennyStocks,

Yahoo Finance is a website provided by Yahoo for investors to access financial
news, market data, and basic financial tools. Given a stock symbol or company
name, it provides the relevant market data.

Classification techniques such as Extreme Gradient Boosting (XGBoost), Random
Forests, Support Vector Machine (SVM), and Artificial Neural Networks (ANNs)
were used to learn predictive models, and then to identify which attributes
(i.e. words) are most predictive. Figure 2 shows the experimental workflow.

Figure 2: Experiment workflow

Data from Reddit and Yahoo Finance were collected daily for the period October
1, 2019, to June 28, 2020. A breakdown of the data is shown in Table 1. The
majority of the data is retrieved from r/pennystocks, with about a third from
r/RobinHoodPennyStocks. The number of comments is much larger than the number
of posts, with posts making up only about 5% of the texts.

Subreddit |  Number of Posts |  Number of Comments |  Total
---|---|---|---
r/pennystocks |  12,049 |  234,149 |  246,198
r/RobinHoodPennyStocks |  6,506 |  78,429 |  84,935
Total |  18,555 |  312,578 |  331,133

Table 1: Breakdown of records collected from subreddits

As shown in Figure 3, there was a sharp increase in the number of submissions
over the period of data collection:

  * •

r/pennystocks - 139,000 Members $\Rightarrow$ 257,000 Members

  * •

r/RobinHoodPennyStocks - 52,000 Members $\Rightarrow$ 133,0000 Members

This seems to reflect an increase in amateur stock market investing because of
the covid-19 pandemic, and a corresponding increase in manipulation. i.e, as
manipulators look to take advantage of new, naive investors during the
pandemic. Alerts and press releases by the SEC and the Canadian Securities
Administrators warned new investors to be vigilant about the increasing number
of P&D schemes that have occurred around that time [9, 28, 29].

Figure 3: Data Collection Volumes

The median number of words per post or comment was 22, and the total number of
distinct words was 4,862.

Figure 4: Histogram of market sectors discussed within subreddits

Replacing stock symbols by the market sector to which each business belongs
allows us to see which sectors are discussed the most, and which are the
targets of P&D. Figure 4 shows that healthcare stocks are the most mentioned,
followed by technology stocks. The pandemic clearly had an effect on both
attention to markets and manipulations. Temporal trends in the healthcare
sector, Figure 5 , show an increase in online activity at the beginning of the
pandemic, and then a further increase in the middle of 2020. Figure 6 shows
that P&D manipulations also increased in 2020.

Figure 5: Trend of posts and comments that discussed healthcare stocks Figure
6: Trend of posts that have been labelled as P&D

Table 2 shows the information collected for each post and comment.

Feature |  Description
---|---
Post Title |  Title of the post.
Post ID |  Unique identification code for post.
Post Author |  Author of the post.
Post Created |  Unix Timestamp of when post was submitted.
Post Body |  Text of the post.
Comment ID |  Unique identification code for comment.
Comment Author |  Author of the comment.
Comment Created |  Unix Timestamp of when comment was submitted.
Comment Body |  Text of the comment.

Table 2: Features of collected Reddit data

Data from Yahoo Finance was scraped using the yfinance tool [2]. Stock symbols
were extracted from Reddit posts. This step is non-trivial and required
regular expression extraction, and look ups against the publicly traded
exchanges. Posts which mentioned more than one stock were discarded, partly
because of the complexity of deciding which stock may be being touted, and
partly because P&D posts typically focus on one particular stock they are
pumping. If a stock symbol was found, yfinance was used to collect the
financial information described in Table 3.

Feature |  Description
---|---
Open |  Opening price of the stock for the given period.
High |  Highest price for the stock within the given period.
Low |  Lowest price for the stock within the given period.
Close |  Closing price of the stock for the given period.
Volume |  Total number of shares traded within the given period.
Market Sector |  Associated industry that the company is in.
Market Capitalization |  Total market value of the company’s outstanding shares.

Table 3: Features of Yahoo! Finance data

As shown in Figure 7, the daily Open, High, Low, Close, and Volume (OHLCV)
data was collected over nine business days surrounding an event. Data was
collected over five days before each post event to establish a baseline for
price and volume. Penny stocks almost always shows minor variation in price
and volume so this baseline is typically quite flat. The remaining four days
contain the pump event (sharp increase) followed by a decrease in price and a
slower decrease in volume. Sabherwal et al. [27] studied the effects of online
message boards on market manipulation and found that dumps typically occur
within four days and this is plausible because the manipulators want to sell
as soon as the price reaches a peak.

Figure 7: Time window used to collect market data.

Texts from subreddits were preprocessed using the following steps: remove
URLs, expand contractions, remove HTML Tags, remove punctuation, remove extra
whitespaces, remove numbers, lemmatization, and remove stopwords.

Stock symbols within the text were replaced by dummy stock names representing
the market sector associated with each business. This is required because the
name of the particular stock being pumped and dumped in one case has nothing
to do with the name of the stock being used in another case – but there might
be correspondences within sectors. Here is an example:

  * •

“AYTU perfect time to buy” $\Rightarrow$ “SectorHealthcare perfect time to
buy”

###  3.2 Data Labelling

To label each post, stock data surrounding the day in which the post was
submitted to Reddit were analyzed. If the market data exhibited that pattern
associated with P&D (a notable rise from the time of the post, followed by a
sharp drop) then the post was labelled accordingly. A rise was detected by
calculating the average price and volume in the five-day window before the
post. The daily average price (DAP) of the values was first calculated for
each of the five days.

$DAP(X_{t})=\frac{1}{4}(X_{t_{open}}+X_{t_{high}}+X_{t_{low}}+X_{t_{close}})$
(1)

and then the baseline average price (BAP) was calculated by

$BAP(X_{est})=\frac{1}{5}\cdot\sum_{t=T_{0}}^{T_{1}}DAP(X_{t})$ (2)

The baseline average volume (BAV) was calculated by taking the average of the
volume values over the estimation window.

$BAV(X_{est})=\frac{1}{5}\cdot\sum_{t=T_{0}}^{T_{1}}X_{t_{volume}}$ (3)

A threshold was set at two standard deviations above the average price within
the five-day estimation window. Price increases above this threshold were
considered to be pump events. A similar threshold was used to define a volume
anomaly. Events were considered to be the result of P&D if they exceeded the
threshold for both price and volume. Figure 8 shows a comparison of the stock
behaviours labelled using this approach.

Figure 8: Comparison of stock behaviours that have been labelled using anomaly
detection

A sudden price rise or volume increase might coincide with a post, but is not
necessarily caused by it. The rising region of each stock trend of a potential
P&D event was min-max normalised, and its slope calculated. Steep price
increases are more likely to arise from genuine information and less likely to
have resulted from a single manipulation post, so the median slope across the
entire dataset was calculated, and only slopes below the median were
considered as potential P&D events. Figure 9 shows the distribution of stock
price trend slopes from the entire the dataset. The median value is 0.18.

Figure 9: Distribution of stock price trend slopes

###  3.3 Agreement Model

The comments associated with the P&D post cannot all be labelled as examples
of P&D language, since not all of them will be supportive of the post they are
responding to. Manipulators, of course, will post comments in support of the
post, either from the same identity or from others.

We developed an agreement model, using ideas from stance detection. This was
done using Empath to generate a lexicon of agreement, seeding it with the
words: bought, agree, positive, increasing, good, and now. Empath returned the
words listed in Table 4.

only |  done |  better |  true |  knew
---|---|---|---|---
besides |  like |  maybe |  wanted |  liked
also |  important |  buying |  understand |  good
understood |  needed |  work |  because |  successful
knowing |  grateful |  plus |  much |  reasonable
should |  give |  happy |  course |  glad
well |  considering |  anyway |  agree |  meaning
great |  probably |  sure |  thought |  guaranteed
more |  honestly |  positive |  thankful |  actually
agreed |  special |  doubt |  guess |  though
bet |  buy |  surpass |  worth |
suppose |  although |  especially |  definitely |
certain |  figured |  given |  means |

Table 4: List of generated agreement words from Empath

Posts touting stocks also use a specialised vocabulary, shown in these
examples.

  * •

“probably go to shoot up tomorrow”

  * •

“this bad boy just rocket”

  * •

“i will see you on the moon”

An extended lexicon was determined manually by inspecting posts associated
with manipulation. Table 5 contains the list of words that were chosen using
this approach.

moon |  fast |  massive |  rich |  surprise
---|---|---|---|---
rocket |  profit |  top |  easy |  move
pump |  rally |  peak |  early |  load
soar |  climb |  worth |  shoot |  quick
jump |  rise |  sale |  money |  burst
pop |  high |  gain |  breakout |  drive
hype |  spike |  run |  cash |  nice
fly |  go |  up |  hit |  bank
awesome |  confident |  surpass |  more |  zoom
big |  great |  potential |  advantage |

Table 5: List of custom words used in the Agreement Model

Comments were labelled as associated with pumping if they contained two or
more of the agreement words, or if they were (visibly) authored by the
original poster. The following are some examples of comments that were
labelled as not P&D related based on the agreement model:

  * •

“it be the american dream to fall for snake oil salesman and then lose
everything it be a story as old as humanity”

  * •

“clearly a pump and dump scheme”

  * •

“do not touch it if the chart look like a hockey stick”

This labelling of comments is limited by the completeness of the agreement
lexicon, and also does not account for negations.

P&D posts and comments are relatively rare and so the dataset is naturally
imbalanced. Techniques such as SMOTE [3] and ADASYN [17] were tried but proved
ineffective. Instead, where predictors allowed it, class weight parameters
were set to penalise mistakes in the minority class.

###  3.4 Modelling

The following predictors were used:

  * •

Extreme Gradient Boosting (XGBoost)

  * •

Random Forest (RF)

  * •

Support Vector Machine (SVM)

  * •

Artificial Neural Networks

    * –

Multilayer Perceptron (MLP)

    * –

Convolutional Neural Network (CNN)

    * –

Bidirectional Long Short Term Memory (BiLSTM)

In each case the standard performance measures (accuracy, precision, recall,
F1-Score, confusion matrix) were calculated, as well as the Shapley values
which rank words by their importance to the predictions.

##  4 Results

Table 6 shows the class distribution for the dataset. Less than 9% of the
records are labelled as being P&D. This is typical of datasets where fraud is
present; indeed it is striking that the rate of fraud is this high.

Record Type |  P&D |  Not P&D |  Total
---|---|---|---
Posts |  3,006 |  15,549 |  18,555
Comment |  26,727 |  285,851 |  312,578
Total |  29,733 |  312,142 |  331,133

Table 6: Dataset class distribution

The results of each of the predictive model are reported in Table 7 using
5-fold cross validation and upweighting the fraud class when the model permits
it.

Model | TP | FP | TN | FN | Accuracy | Precision | Recall | F1-Score
---|---|---|---|---|---|---|---|---
XGBoost Posts | 1728 | 6615 | 8934 | 1278 | 57.46 ($\pm$3.73) | 20.71 ($\pm$0.48) | 57.49 ($\pm$0.68) | 30.45 ($\pm$2.25)
XGBoost Posts and Comments | 2007 | 7646 | 7903 | 999 | 53.41 ($\pm$1.42) | 20.79 ($\pm$0.85) | 66.77 ($\pm$1.58) | 31.71 ($\pm$0.96)
RF Posts | 271 | 646 | 14903 | 2735 | 81.78 ($\pm$0.51) | 29.55 ($\pm$1.40) | 9.01 ($\pm$0.52) | 13.81 ($\pm$0.78)
RF Posts and Comments | 414 | 211 | 15338 | 2592 | 84.89 ($\pm$0.69) | 66.24 ($\pm$1.69) | 13.77 ($\pm$0.47) | 22.80 ($\pm$0.75)
SVM Posts | 1752 | 5263 | 10286 | 1254 | 64.88 ($\pm$1.14) | 24.98 ($\pm$0.76) | 58.28 ($\pm$1.05) | 34.97 ($\pm$1.16)
SVM Posts and Comments | 2125 | 4559 | 10990 | 881 | 70.6 ($\pm$0.49) | 31.79 ($\pm$0.43) | 70.69 ($\pm$0.56) | 43.86 ($\pm$0.57)
MLP Posts | 2382 | 1718 | 13831 | 624 | 87.38 ($\pm$6.66) | 58.10 ($\pm$11.65) | 79.24 ($\pm$12.76) | 67.04 ($\pm$12.12)
MLP Posts and Comments | 2103 | 2602 | 12947 | 903 | 81.11 ($\pm$3.71) | 44.70 ($\pm$4.28) | 69.96 ($\pm$3.80) | 54.55 ($\pm$4.36)
CNN Posts | 2373 | 1709 | 13840 | 633 | 87.38 ($\pm$7.04) | 58.13 ($\pm$12.02) | 78.94 ($\pm$12.76) | 66.96 ($\pm$12.37)
CNN Posts and Comments | 2304 | 2068 | 13481 | 702 | 85.07 ($\pm$1.25) | 52.70 ($\pm$2.33) | 76.65 ($\pm$3.45) | 62.46 ($\pm$2.64)
biLSTM Posts | 2297 | 2495 | 13054 | 709 | 82.73 ($\pm$8.11) | 47.93 ($\pm$9.92) | 76.41 ($\pm$10.94) | 58.91 ($\pm$10.82)
biLSTM Posts and Comments | 2288 | 2370 | 13179 | 718 | 83.36 ($\pm$2.27) | 49.12 ($\pm$3.25) | 76.11 ($\pm$3.86) | 59.71 ($\pm$3.54)

Table 7: Summary of model performance

The neural network models perform well as expected. Models such as XGBoost,
Random Forests, and SVM had disappointing performance, and a heterogeneous
stacked classifier combining their predictions did not improve on the
performance of the individual predictors, suggesting that they make their
errors on the same records.

At first glance, the ANN models using posts perform better than those using
posts and comments. However, the standard deviations of the performance
numbers show that the inclusion of comments provides stability for correctly
identifying P&D posts. The best performing model overall is CNN, especially
with comments included. Its precision is relatively low; of all the records
that the model predicts to be P&D, only 52.7% are actually correct. If we look
at the rate at which each class is predicted to be positive, a better outlook
of the model is provided. Given a positive P&D text, the model has a 76.65%
chance of classifying it correctly, whereas, if it is given a negative text,
it has a 13.3% chance of classifying it incorrectly as positive. It is perhaps
a little surprising that biLSTM did not perform best since they are typically
strong predictors for natural language problems.

The SHAP Explainers produce diagrams that rank the attributes by their impact
on outcomes. Figure 10 shows the diagram for the CNN predictor for posts and
comments and the 30 most impactful words. Although the influence of any single
word is inevitably weak, there are visible red dots to the right for many of
these words, indicating that higher frequencies of these words are associated
with P&D events. The names of the popular sectors are indicator of P&Ds, as
are words from the agreement model such as “buy” and “go”. Across the best
performing models, the same set of words emerge as the most impactful features
(not shown).

Figure 10: CNN SHAP Summary Plot for posts and comments

Misclassifications by the model have different impacts depending on how and
where it is used. For an ordinary investor, a false positive (a post predicted
to be a P&D when it isn’t) means a missed opportunity for profit, but a false
negative means a financial loss. For a regulatory body, a false positive is
problematic, but a false negative less so. Table 8 shows some of the examples
of misclassifications by the CNN model.

Predicted Label | Actual Label | Misclassified Post
---|---|---
P&D |  Not P&D |  sectorunknown about to soar
P&D |  Not P&D |  sectorunknown fitness equipment maker owner of bow flex completely sell out of most retail store how be this look just buy in share
P&D |  Not P&D |  quick all in sectorcommunicationservices pump my first time actually do something right the lambos go to be green for gain
P&D |  Not P&D |  blast off look like gold and oil will be big player this i also suggest look at sectortechnology
P&D |  Not P&D |  sectorenergy drop time to buy it be drop below which be its day low be it a good time to buy
Not P&D |  P&D |  sectortechnology release patent news on thermal tech could be a mark sympathy play bust out over
Not P&D |  P&D |  sectorhealthcare do anyone understand why sectorhealthcare shoot up soo much i be not able to find any real catalyst
Not P&D |  P&D |  sectorhealthcare on the move this have potential reach today
Not P&D |  P&D |  sectorhealthcare to the moon
Not P&D |  P&D |  any thought on when to sell sectorenergy bought in late i be up after hour should i wait til tomorrow or sell as soon as possible in the am

Table 8: Examples of misclassified posts from CNN model

Some false positives, predicted to be P&D from the text, but without a
corresponding market movement may be instances where the post failed to
attract enough attention to cause a measurable market movement, or was so
blatant that it was not credible to typical investors. Some false negatives
may be because the posts were too short to contain the required two words,
because the pumping took place on another platform or because a market
movement happened to match the timing of the post.

##  5 Related work

The application of data analytics for detecting market manipulation is a
relatively new in the field of finance. Most research has focused on detecting
trade-based manipulation because it is most common [32]. Huang and Chang found
that of the manipulation cases prosecuted in Taiwan from 1991 to 2010, 96.61%
were trade-based, and only 3.39% were information-based [18]. Some examples
detecting trade-based manipulation are: Ogut et al. [38] in the emerging
Istanbul Stock Exchange, Wang et al. [32] for prosecuted manipulation cases
reported by the China Securities Regulatory Commission, Cao et al. [7] using
real trading data from four popular NASDAQ stocks with synthetic cases of
manipulation (spoofing and quote stuffing), Cao et al. [36] using seven
popular NASDAQ and LSE stocks data injecting ten simulated stock price
manipulations, Diaz et al. [12] using manipulation cases pursued by the U.S.
Securities and Exchange Commission (SEC) in 2003, and Golomohammadi et al.
[16] trying to detect three groups of manipulation schemes: marking the close,
wash trades, and cornering the market.

For information-based manipulation, Victor and Hagemann [31] looked at 149
confirmed P&D schemes coordinated through Telegram chats and pumped via
Twitter. Using XGBoost, they built a model that achieved a sensitivity of 85%
and specificity of 99%. They concluded that P&Ds were frequent among
cryptocurrencies that had a market capitalization of $50 million or below and
often involved trading volumes of several hundred thousand dollars within a
short time-frame.

Mirtaheri et al. [23] looked specifically at forecasting P&Ds by combining the
information from Twitter and Telegram. They manually labelled known P&D
operation messages on Telegram, and then used SVMs with a stochastic gradient
descent optimizer to label the remaining messages as P&D or not. They used
Random Forests to detect whether a manipulation event was going to take place
within the market. Their results showed that they were able to detect, with
reasonable accuracy, whether there is an unfolding manipulation scheme
occurring on Telegram. Their proposed model was able to achieve an accuracy of
87% and an F1-Score of 90%.

Some partially automated tools have also been developed. These flag suspicious
activities that can then by investigated by regulators. Delort et al. [11]
used Naive Bayes classifiers to examine collected messages from HotCopper, an
Australian stock message board. They successfully identified messages of
concern, but the number of false positives was too high to use the model in an
automated way. Owda et al. [25] compared messages to lexicon templates of
known illegal financial activities (e.g. Pump and Dump, Insider Information).
They found that, of the 3000 comments that were collected on a daily basis,
0.2% were deemed suspicious.

##  6 Conclusion

The intersection of social media with low-cost trading platforms and naive
investors has made market manipulation an attractive strategy. Pump&dump is
particularly simple to implement since it requires only the dissemination of
fictional information about the future prospects for a stock. This is
particular easy for penny stocks where validating information is difficult for
ordinary investors, and where relatively small purchase volumes can cause
large price movements.

We investigate protecting investors, and assisting regulators, by building
predictive models that label social media posts (and the responses they
elicit) as potential drivers of P&D events. We do this by collecting posts and
comments, developing a model for a P&D event based on patterns of price and
volume changes, using the match between posts and P&D events to label posts,
and extending this labelling to comments using an agreement model. Natural
language predictors then learn the language patterns associated with P&D
manipulations, so that new manipulations can be detected before they affect
the market.

Data is imbalanced, since manipulations are rare, but our best predictive
model achieves an F1-score of 62% and an accuracy of 85%. Improvements in
performance are limited by potential coincidences between a post and a price
and volume change that mimics a P&D, posts that fail to reach a sufficient
audience to cause the desired buying behaviour, and natural language issues
that arise from informal and short texts, and a specialised vocabulary used in
stock discussion forums.

## References

  * [1] R.K. Aggarwal and Guojun Wu.  Stock Market Manipulations.  The Journal of Business, 79(4):1915–1953, July 2006.
  * [2] Ran Aroussi.  yfinance.  https://aroussi.com/post/python-yahoo-finance, Accessed: 2021-09-02.
  * [3] Kevin W. Bowyer, Nitesh V. Chawla, Lawrence O. Hall, and W. Philip Kegelmeyer.  SMOTE: Synthetic Minority Over-sampling Technique.  CoRR, abs/1106.1813, 2011.
  * [4] Jason Brownlee.  A Gentle Introduction to Long Short-Term Memory Networks by the Experts, Feb 2020.  https://machinelearningmastery.com/gentle-introduction-long-short-term-memory-networks-experts/, Accessed: 2021-09-17.
  * [5] Jason Brownlee.  A Gentle Introduction to XGBoost for Applied Machine Learning, Apr 2020\.  https://machinelearningmastery.com/gentle-introduction-xgboost-applied-machine-learning/, Accessed: 2021-09-15.
  * [6] Jason Brownlee.  How to Develop a Bidirectional LSTM For Sequence Classification in Python with Keras, Aug 2020.  https://machinelearningmastery.com/develop-bidirectional-lstm-sequence-classification-python-keras/, Accessed: 2021-09-18.
  * [7] Yi Cao, Yuhua Li, Sonya Coleman, Ammar Belatreche, and T M McGinnity.  Detecting Price Manipulation in the Financial Market.  In Proceedings of the IEEE/IAFE Computational Intelligence for Financial Engineering (CIFEr), page 8, March 2014.
  * [8] Carole Comerton-Forde and Tālis J. Putniņš.  Stock Price Manipulation: Prevalence and Determinants.  Review of Finance, 18(1):23–66, 03 2013.
  * [9] Canadian securities regulators warn public of coronavirus-related investment scams, Mar 2020.  https://www.securities-administrators.ca/aboutcsa.aspx?id=1878, Accessed: 2021-09-06.
  * [10] James Dacombe.  An introduction to Artificial Neural Networks (with example), Oct 2017\.  https://medium.com/@jamesdacombe/an-introduction-to-artificial-neural-networks-with-example-ad459bb6941b, Accessed: 2021-09-16.
  * [11] Jean-Yves Delort, Bavani Arunasalam, and Cecile Paris.  Automatic moderation of online discussion sites.  International Journal of Electronic Commerce, 15(3):9–30, 2011\.
  * [12] David Diaz, Babis Theodoulidis, and Pedro Sampaio.  Analysis of stock market manipulations using knowledge discovery techniques applied to intraday trade prices.  Expert Systems with Applications, 38(10):12757–12771, September 2011.
  * [13] Ethan Fast, Binbin Chen, and Michael S. Bernstein.  Empath: Understanding Topic Signals in Large-Scale Text.  In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 4647–4657, San Jose California USA, May 2016. ACM.
  * [14] Rohith Gandhi.  Support Vector Machine - Introduction to Machine Learning Algorithms, Jul 2018.  https://towardsdatascience.com/support-vector-machine-introduction-to-machine-learning-algorithms-934a444fca47, Accessed: 2021-09-15.
  * [15] Bilal Ghanem, Paolo Rosso, and Francisco Rangel.  Stance detection in fake news a combined feature representation.  In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 66–71, Brussels, Belgium, November 2018. Association for Computational Linguistics.
  * [16] Koosha Golmohammadi, Osmar R. Zaiane, and David Diaz.  Detecting stock market manipulation using supervised learning algorithms.  In 2014 International Conference on Data Science and Advanced Analytics (DSAA), pages 435–441, Shanghai, China, October 2014\. IEEE.
  * [17] Haibo He, Yang Bai, E. A. Garcia, and Shutao Li.  ADASYN: Adaptive synthetic sampling approach for imbalanced learning.  In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), pages 1322–1328, 2008.
  * [18] Yu Chuan Huang and Yao Jen Cheng.  Stock manipulation and its effects: pump and dump versus stabilization.  Review of Quantitative Finance and Accounting, 44(4):791–815, May 2015.
  * [19] Josh Kamps and Bennett Kleinberg.  To the moon: defining and detecting cryptocurrency pump and dumps.  Crime Science, 7(1):18, December 2018.
  * [20] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  Nature, 521(7553):436–444, May 2015.
  * [21] Yingjie Li and Cornelia Caragea.  Multi-task stance detection with sentiment and stance lexicons.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6299–6305, Hong Kong, China, November 2019. Association for Computational Linguistics.
  * [22] Scott M Lundberg and Su-In Lee.  A Unified Approach to Interpreting Model Predictions.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765–4774. Curran Associates, Inc., 2017.
  * [23] Mehrnoosh Mirtaheri, Sami Abu-El-Haija, Fred Morstatter, Greg Ver Steeg, and Aram Galstyan.  Identifying and Analyzing Cryptocurrency Manipulations in Social Media.  preprint, Open Science Framework, February 2019.
  * [24] Chris B. Murphy.  Penny Stock, Aug 2020.  https://www.investopedia.com/terms/p/pennystock.asp, Accessed: 2021-09-02.
  * [25] Majdi Owda, Pei Shyuan Lee, and Keeley Crockett.  Financial Discussion Boards Irregularities Detection System (FDBs-IDS) using information extraction.  In 2017 Intelligent Systems Conference (IntelliSys), pages 1078–1082, London, September 2017. IEEE.
  * [26] Ashwin Rajadesingan and Huan Liu.  Identifying users with opposing opinions in Twitter debates.  In William G. Kennedy, Nitin Agarwal, and Shanchieh Jay Yang, editors, Social Computing, Behavioral-Cultural Modeling and Prediction, pages 153–160, Cham, 2014. Springer International Publishing.
  * [27] Sanjiv Sabherwal, Salil K. Sarkar, and Ying Zhang.  Do Internet Stock Message Boards Influence Trading? Evidence from Heavily Discussed Stocks with No Fundamental News: Do Internet Stock Message Boards Influence Trading?  Journal of Business Finance & Accounting, 38(9-10):1209–1237, November 2011.
  * [28] Investor Alerts and Bulletins: Frauds Targeting Main Street Investors – Investor Alert, Apr 2020.  https://www.sec.gov/oiea/investor-alerts-and-bulletins/ia_frauds, Accessed: 2020-09-06.
  * [29] Press Release: SEC Charges Microcap Fraud Scheme Participants Attempting to Capitalize on the COVID-19 Pandemic, Jun 2020.  https://www.sec.gov/news/press-release/2020-131, Accessed: 2020-09-06.
  * [30] Matt Thomas, Bo Pang, and Lillian Lee.  Get out the vote: Determining support or opposition from congressional floor-debate transcripts.  In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 327–335, Sydney, Australia, July 2006. Association for Computational Linguistics.
  * [31] Friedhelm Victor and Tanja Hagemann.  Cryptocurrency Pump and Dump Schemes: Quantification and Detection.  In 2019 International Conference on Data Mining Workshops (ICDMW), pages 244–251, Beijing, China, November 2019. IEEE.
  * [32] Qili Wang, Wei Xu, Xinting Huang, and Kunlin Yang.  Enhancing intraday stock price manipulation detection by leveraging recurrent neural networks with ensemble learning.  Neurocomputing, 347:46–58, June 2019.
  * [33] Cody Marie Wild.  One Feature Attribution Method to (Supposedly) Rule Them All: Shapley Values, Jan 2018.
  * [34] Thomas Wood.  Random Forests, Sep 2020.  https://deepai.org/machine-learning-glossary-and-terms/random-forest, Accessed: 2021-09-16.
  * [35] XGBoost Algorithm: XGBoost In Machine Learning, May 2020.  https://www.analyticsvidhya.com/blog/2018/09/an-end-to-end-guide-to-understand-the-math-behind-xgboost/, Accessed: 2021-09-15.
  * [36] Yi Cao, Yuhua Li, Sonya Coleman, Ammar Belatreche, and Thomas Martin McGinnity.  Adaptive Hidden Markov Model With Anomaly States for Price Manipulation Detection.  IEEE Transactions on Neural Networks and Learning Systems, 26(2):318–330, February 2015.
  * [37] Tony Yiu.  Understanding Random Forest, Aug 2019.  https://towardsdatascience.com/understanding-random-forest-58381e0602d2, Accessed: 2021-09-16.
  * [38] Hulisi Öğüt, M. Mete Doğanay, and Ramazan Aktaş.  Detecting stock-price manipulation in an emerging market: The case of Turkey.  Expert Systems with Applications, 36(9):11944–11949, November 2009\.


Further author information: (Send correspondence to J.A)
J.A.: E-mail<EMAIL_ADDRESS>
# Final Alignment and Image Quality Test for the Acquisition and Guiding
System of SOXS

José A. Araiza-Durán  INAF - Osservatorio Astronomico di Arcetri, Via E. Fermi
5 Firenze, Italy  Giuliano Pignata  Universidad de Tarapacá, Casilla 7D,
Arica, Chile  Anna Brucalassi  INAF - Osservatorio Astronomico di Arcetri, Via
E. Fermi 5 Firenze, Italy  Universidad Andrés Bello, Avda. República 252,
Santiago, Chile  Matteo Aliverti  INAF - Osservatorio Astronomico di Brera,
Via Bianchi 46, I-23807 Merate, Italy  Federico Battaini  INAF - Osservatorio
Astronomico di Padova, Vicolo dell’Osservatorio 5, I-35122 Padova, Italy
Kalyan Radhakrishnan  INAF - Osservatorio Astronomico di Padova, Vicolo
dell’Osservatorio 5, I-35122 Padova, Italy  Simone Di Filippo  INAF -
Osservatorio Astronomico di Padova, Vicolo dell’Osservatorio 5, I-35122
Padova, Italy  Luigi Lessio  INAF - Osservatorio Astronomico di Padova, Vicolo
dell’Osservatorio 5, I-35122 Padova, Italy  Riccardo Claudi  INAF -
Osservatorio Astronomico di Padova, Vicolo dell’Osservatorio 5, I-35122
Padova, Italy  Davide Ricci  INAF - Osservatorio Astronomico di Padova, Vicolo
dell’Osservatorio 5, I-35122 Padova, Italy  Mirko Colapietro  INAF -
Osservatorio Astronomico di Capodimonte, Salita Moiariello 16, I-80131 Napoli,
Italy  Rosario Cosentino  INAF - Fundación Galileo Galilei, Rambla J.A.
Fernández Pérez 7, E-38712 Breña Baja (TF), Spain  INAF - Osservatorio
Astrofisico di Catania, Via S. Sofia 78, I-95123 Catania, Italy  Sergio D’Orsi
INAF - Osservatorio Astronomico di Capodimonte, Salita Moiariello 16, I-80131
Napoli, Italy  Matteo Munari  INAF - Osservatorio Astrofisico di Catania, Via
S. Sofia 78, I-95123 Catania, Italy  Marco Dima  INAF - Osservatorio
Astronomico di Padova, Vicolo dell’Osservatorio 5, I-35122 Padova, Italy
Pietro Schipani  INAF - Osservatorio Astronomico di Capodimonte, Salita
Moiariello 16, I-80131 Napoli, Italy  Sergio Campana  INAF - Osservatorio
Astronomico di Brera, Via Bianchi 46, I-23807 Merate, Italy  Andrea Baruffolo
INAF - Osservatorio Astronomico di Padova, Vicolo dell’Osservatorio 5, I-35122
Padova, Italy  Ricardo Zanmar Sanchez  INAF - Osservatorio Astronomico di
Capodimonte, Salita Moiariello 16, I-80131 Napoli, Italy  Marco Riva  INAF -
Osservatorio Astronomico di Brera, Via Bianchi 46, I-23807 Merate, Italy
Matteo Genoni  INAF - Osservatorio Astronomico di Brera, Via Bianchi 46,
I-23807 Merate, Italy  Sagi Ben-Ami  Weizmann Institute of Science, Herzl St
234, Rehovot, 7610001, Israel  Harvard-Smithsonian Center for Astrophysics,
Cambridge, USA  Adam Rubin  European Southern Observatory, Karl Schwarzschild
Strasse 2, D-85748, Garching bei München, Germany  Rachel Bruch  Weizmann
Institute of Science, Herzl St 234, Rehovot, 7610001, Israel  Giulio Capasso
INAF - Osservatorio Astronomico di Capodimonte, Salita Moiariello 16, I-80131
Napoli, Italy  Francesco D’Alessio  INAF - Osservatorio Astronomico di Roma,
Via Frascati 33, I-00078 Monte Porzio Catone, Rome, Italy  Paolo D’Avanzo
INAF - Osservatorio Astronomico di Brera, Via Bianchi 46, I-23807 Merate,
Italy  Ofir Hershko  Weizmann Institute of Science, Herzl St 234, Rehovot,
7610001, Israel  Hanindyo Kuncarayakti  Tuorla Observatory, Department of
Physics and Astronomy, University of Turku, FI-20014 University of Turku,
Turku, Finland  FINCA - Finnish Centre for Astronomy with ESO, Turku, Finland
Marco Landoni  INAF - Osservatorio Astronomico di Brera, Via Bianchi 46,
I-23807 Merate, Italy  Salvatore Scuderi  INAF - Istituto di Astrofisica
Spaziale e Fisica Cosmica, Via Corti 12, I-20133 Milano, Italy  INAF -
Osservatorio Astrofisico di Catania, Via S. Sofia 78, I-95123 Catania, Italy
Fabrizio Vitali  INAF - Osservatorio Astronomico di Roma, Via Frascati 33,
I-00078 Monte Porzio Catone, Rome, Italy  David Young  Queen’s University
Belfast, School of Mathematics and Physics, Belfast, BT7 1NN, UK  Jani Achrén
Incident Angle Oy, Capsiankatu 4 A 29, FI-20320 Turku, Finland  Iair Arcavi
Tel Aviv University, Tel Aviv, Israel  Enrico Cappellaro  INAF - Osservatorio
Astronomico di Padova, Vicolo dell’Osservatorio 5, I-35122 Padova, Italy
Massimo Della Valle  INAF - Osservatorio Astronomico di Capodimonte, Salita
Moiariello 16, I-80131 Napoli, Italy  Rosario Di Benedetto  INAF -
Osservatorio Astrofisico di Catania, Via S. Sofia 78, I-95123 Catania, Italy
Avishay Gal-Yam  Weizmann Institute of Science, Herzl St 234, Rehovot,
7610001, Israel  Marcos Hernandez Díaz  INAF - Fundación Galileo Galilei,
Rambla J.A. Fernández Pérez 7, E-38712 Breña Baja (TF), Spain  Jari Kotilainen
FINCA - Finnish Centre for Astronomy with ESO, Turku, Finland  Tuorla
Observatory, Department of Physics and Astronomy, University of Turku,
FI-20014 University of Turku, Turku, Finland  Gianluca Li Causi  INAF -
Istituto di Astrofisica e Planetologia Spaziali, Via Fosso del Cavaliere,
I-00133 Roma, Italy  Laurent Marty  INAF - Osservatorio Astronomico di
Capodimonte, Salita Moiariello 16, I-80131 Napoli, Italy  Seppo Mattila
Tuorla Observatory, Department of Physics and Astronomy, University of Turku,
FI-20014 University of Turku, Turku, Finland  Michael Rappaport  Weizmann
Institute of Science, Herzl St 234, Rehovot, 7610001, Israel  Bernardo
Salasnich  INAF - Osservatorio Astronomico di Padova, Vicolo dell’Osservatorio
5, I-35122 Padova, Italy  Stephen Smartt  University of Oxford,Keble Road,
Oxford OX1 3RH, Oxford, UK  Queen’s University Belfast, School of Mathematics
and Physics, Belfast, BT7 1NN, UK  Maximilian Stritzinger  Aarhus
University,Ny Munkegade 120, D-8000 Aarhus, Denmark  Hector Pérez Ventura
INAF - Fundación Galileo Galilei, Rambla J.A. Fernández Pérez 7, E-38712 Breña
Baja (TF), Spain  Laura Asquini  INAF - Osservatorio Astronomico di Brera, Via
Bianchi 46, I-23807 Merate, Italy  Alex Bichkovsky  Weizmann Institute of
Science, Herzl St 234, Rehovot, 7610001, Israel  Salvatore Savarese  INAF -
Osservatorio Astronomico di Capodimonte, Salita Moiariello 16, I-80131 Napoli,
Italy  Lorenzo Cabona  INAF - Osservatorio Astronomico di Padova, Vicolo
dell’Osservatorio 5, I-35122 Padova, Italy

###### Abstract

SOXS (Son Of X-Shooter) will be the new medium-resolution (R 4500 for 1”
slit), high-efficiency, wide-band spectrograph for the ESO NTT at La Silla
Observatory, Chile. It will be dedicated to the follow-up of any kind of
transient events, ensuring fast time, high efficiency, and availability. It
consists of a central structure (common path) that supports two spectrographs
optimized for the UV-Visible and a Near-Infrared range. Attached to the common
path is the Acquisition and Guiding Camera system (AC), equipped with a filter
wheel that can provide science-grade imaging and moderate high-speed
photometry. The AC Unit was integrated and aligned during the summer months of
2022 and has since been mounted in the NTT’s telescope simulator. This work
gives an update on the Acquisition Camera Unit status, describes the Image
Quality Tests that were performed, and discusses the AC Optical Performance.

###### keywords:

Spectrograph, Transient, Medium-Resolution, NTT, AIT

##  1 INTRODUCTION

The Son Of X-Shooter (SOXS), is a spectroscopic facility for the European
Southern Observatory (ESO) New Technology Telescope (NTT) at La Silla
Observatory, Chile [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. It will be dedicated to
the follow-up of transient sources, proposed by an Italian led consortium. It
will be a two arm spectrograph able to cover the optical/NIR band
($0.35-1.8\mu m$). SOXS will be a medium resolution spectrometer ($R\sim
4500$) with light imaging capabilities in the optical with multi-band
photometry for faint transients. The spectrograph is divided into five sub-
systems: Common Path (CP) [11, 12], UV-VIS spectrograph [13, 14, 15, 16, 17],
NIR[18], and the Acquisition Camera (AC) [19, 20, 21].

The backbone of the system is the CP, which receives the light from the
telescope and distributes light to the sub-systems, see Fig. 1. The AC will
receive the light from the CP through a $45^{\circ}$ mirror, called Selector
Mirror, which has three permitted working positions: monitoring (takes the
light that surrounds a $15$ asec hole), simulate an artificial star for
calibration purposes (take the field surrounding a $0.5$ asec hole), and the
imaging mode (full field). These functions will allow the AC to do photometry,
to work in the acquisition of the target for spectrographs, for monitoring of
spectrographs co-alignment, and also can be used as a guiding system if
needed.

---
Figure 1:  Acquisition Camera and Common Path. a) CP (green) is mounted in the
Nasmyth interface of telescope and the AC is on top (purple). b) General CP
view, the light beam entrance is in the bottom center and the CAM selector
send the light to the AC Unit. c) CAM selector with circles indicating the
working position.

The SOXS project is nearing the commencement of operations, with expectations
set for the instrument to become fully operational by 2025. In 2022, the
Assembly, Integration, and Testing (AIT) for the AC was conducted. The AIT
strategy adopted involved a two-stage approach. The initial stage focused on
evaluating specific elements, comprising tests on the detector, assessment of
filter curves, evaluation of alignment and performance of the lens tube with
the detector, and examination of the light beam going through the AC Unit,
intended for use in the experimental setup. These tests were conducted prior
to the integration of the AC unit into the telescope simulator. Following this
phase, the Assembly and Readiness Review (ARR) tests were conducted, the
outcomes of which are detailed in this report. The document is divided into
four key sections: a thorough description of the instrument, a summary of the
image quality tests used to assess the system, an outline of additional tests
performed with the AC unit installed in the telescope simulator, and,
ultimately, the conclusions derived from the complete testing process.

##  2 AIT Overview

This section will describe the AC unit, including its characteristics,
evaluation criteria, tolerance analysis, and the AIT strategy employed. The
assembly and testing were conducted during the summers of 2022 and 2023 at the
facilities of the Osservatorio Astronomico di Padova.

###  2.1 Acquisition Camera

The AC Unit consist of a Collimator Lens (CO), a folding mirror (FM), a filter
wheel (FW), and a four lens camera, see Fig. 2. The focusing mechanism is the
CO, which is mounted in a linear stage. The filter wheel also uses a rotating
mechanism in order to choose between a broad-band filter set (ugrizY and
V-Johnson). The detector is a $1024x1024$ Andor iKon-M 934 camera with a 13
micron pixel, and the control software was developed and validated by SOXS
Consortium[22, 23, 24]. The system includes a CAM selector, which will be
mounted in the CP. The CAM Selector carries a pellicle beam-splitter and the
selector mirror (SM). All the system is included in a structure made of
6061-T6 Aluminum and is anodized.

---
Figure 2:  AC Unit and its components.

The AC Unit acts as a focal reducer, it reduces the incoming F/11 beam to a
F/3.6. Each pixel of the CCD will correspond to approximately $0.2$ asec,
resulting in a $3.5$ amin unvignetted field of view. The image quality
specifications is such that over $80\%$ of the geometrically encircled energy
is contained in two pixels for a central field of $6.5mm$ radius.

###  2.2 Tolerance Analysis

After preparing all components for assembly, a new tolerance analysis was
conducted to determine acceptable alignment limits, given the laser wavelength
of 632 nm. This analysis examined three primary tolerance groups: (1)
displacements along the optical axis (including separations between components
and lens thicknesses), (2) lens radius of curvature, and (3) decentration and
rotation of individual components and component groups. The sensitivity study
indicated that stricter tolerances were required for elements within the
optical tube. The Table 1 shows $10$ of the most critical areas needing
attention. Based on these findings, tolerances for components outside the tube
were set at $0.100$ mm for distances and $0.100$ degrees for rotation. For
components inside the tube and the tube as a whole, tolerances were set at
$0.050$ mm for distances and $0.050$ degrees for rotation.

Table 1: Worst Offenders. No. |  Tolerance
---|---
$1$ |  Camera Lens No.2 - Dec Y
$2$ |  Tube - Rot Y
$3$ |  Tube - Rot X
$4$ |  Camera Lens No.2 - Dec X
$5$ |  Camera Lens No.2 - Rot X
$6$ |  Camera Lens No.1 - Rot X
$7$ |  Camera Lens No.1 - Dec X
$8$ |  Camera Lens No.2 - Surf 3 - Rad
$9$ |  Camera Lens No.3 - Dec X
$10$ |  Tube Spacer No.2 - Thick

After establishing these tolerances, a Monte Carlo simulation with 1,000
trials was carried out to evaluate the feasibility of the arrangement. The
design incorporates two compensators: the primary compensator is a linear
stage with a $15$ mm stroke, on which the collimator lens is mounted. The
second compensator is used exclusively during the alignment process and is a
ring placed at the interface between the tube and the detector. These rings
have fixed dimensions and come in four sizes: $0.050$, $0.100$, $0.250$, and
$1.00$ mm. The criterion for evaluating the Monte Carlo simulation results is
the distance containing $80\%$ of the energy of the Point Spread Function
(PSF), or the $80\%$ Encircled Energy (EE).

---
Figure 3:  Tolerance Analysis: $80\%$ of Encircled Energy (Diameter). a)
Histogram of values per Monte Carlo case, and b) Cumulative Distribution
Function (CDF). In plot b), the red, green, and blue lines represent the
limits for 1, 2, and 3 pixels, respectively.

Fig. 3 presents a histogram of the Monte Carlo simulation results. The graph
shows that in $92\%$ of cases, the energy is contained within two pixels,
which meets the required specifications. The mean offset for the linear
stage’s compensator is $1.0$ mm from the center, while the average thickness
for the compensator ring is $0.18$ mm. Based on these results, the assembly
and alignment process proceeded with a maximum displacement tolerance of $100$
microns and a rotation tolerance of $0.1$ degrees for the overall arrangement.
However, more stringent tolerances were applied during the insertion and
alignment of lenses within the camera tube.

###  2.3 Assembly and Readiness Review

The concept and design of SOXS are modular, consisting of various sub-systems
like the Common Path, UV-VIS and NIR Spectrograph, Calibration Unit, and the
Acquisition Camera. Due to this modularity, each sub-system goes through an
AIT phase at different partner locations. Afterward, an internal acceptance
process is conducted, known as the ARR. The AIT is performed twice: once at
INAF-OAPD under laboratory conditions, and once at the telescope site in La
Silla.

As part of the AIT phase, a detailed procedure was defined, outlining the
steps to follow and a series of evaluations to be conducted. These evaluations
were designed to ensure that the integration and alignment process was carried
out correctly. The procedure to be followed consists of several steps,
including:

  * •

Extender Cable Test.

  * •

Camera Characterization.

  * •

Filter transmission Curves.

  * •

Camera’s Optical Tube Alignment.

  * •

Telescope Beam Simulator.

  * •

Image Quality Test.

  * •

Linear Stages Check.

The last two steps, Image Quality Test and Linear Stages Check, are considered
part of the ARR.

##  3 Image Quality Test

During the alignment process, characterized CCDs were used to trace the laser
path along the array, while a Coordinate Measuring Machine (CMM) was employed
to position components in accordance with specifications. The telescope’s
focal plane, with a light beam F/11, was evaluated by recording the generated
PSF, after the plane’s position had been previously defined using the CMM.
Fig. 4a shows the arrangement, illustrating the F/11 light beam diverging from
the telescope’s focal plane.

---
Figure 4:  Experimental Setup for: a) Optical Alignment and b) Image Quality
Test using Test Chart.

Next, the Andor camera was used to begin capturing images, allowing for minor
adjustments to achieve a PSF with rotational symmetry and a full width at half
maximum (FWHM) of less than 2 pixels. This section focuses on evaluating image
quality across the entire field using a specific filter. A test chart was
employed to examine the resolution and distortion levels of the instrument,
which was positioned in the telescope’s focal plane, as depicted in Fig. 4b.

###  3.1 Test Chart

The test chart used is a Thorlabs R3L3S5P[25], with a substrate measuring
$76.2$ x $72.6$ mm. The region containing the test patterns is $50.8$ x $50.8$
mm. The main grid comprises $17$ rows and $17$ columns of smaller grids, each
containing nine patterns. These patterns include four concentric circles and
five crosshairs of varying sizes, as shown in Figure 5b. The plate scale at
the telescope’s focal plane is $5.359$ arcseconds per millimeter (”/mm). This
scale allows us to calculate the total area occupied by the test patterns in
angular units. Given this scale, the area covered by the patterns is
approximately $4.53$ x $4.53$ arcminutes.

---
Figure 5:  Concentric Circle and Distortion Grid Target: a) Picture taken with
AC Unit, and b) Pattern Features.

In the study conducted, two specific patterns were selected. For the
resolution analysis, the largest concentric circular pattern was chosen, which
is the first element in the second row of patterns. For the distortion
analysis, the second-largest concentric circular pattern was used, which is
the first element in the third row of patterns. For the analysis, the
information from the internal rings of the circular patterns are not
considered, since the outer contour of the circle forms the basis for the
resolution analysis, and the center of mass of the pattern is calculated for
the distortion analysis. As such, only the dimensions of the outer circles
radius are required, which are $500$ and $250$ microns.

###  3.2 Distortion Test

Distortion is typically expressed as a percentage of the field height,
commonly arising from aberrations near the edges of an image[26]. Generally,
distortion within the range of $\pm 2$ to $3\%$ goes unnoticed in a vision
system[27]. One common method to measure distortion in a camera-lens
combination is the TV distortion method (ISO 9039), designed to analyze TV
camera system[28]. This method uses a test chart with a rectangular grid of
geometric patterns, such as the one shown in Fig. 5b. The TV distortion method
quantifies distortion by evaluating the curvature of a straight line from the
image center to the corners. To calculate distortion, the degree of bending at
the top edge of the image is measured. The ratio of this bending to the height
of the image, multiplied by 100, yields the percentage of picture height
distortion

$D[\%]={{\frac{\triangle H}{H}}\cdot 100}.$ (1)

To calculate the distortion associated with an optical system within a
specific spectral range, defined by one of the filters, the process begins by
applying a high-pass filter to clearly delineate the contour of the pattern
being studied. A sample is taken from the image containing the selected
pattern, and a convolution is performed on the entire image to find the
centroid of these patterns. The centroid is then stored and later compared
with the ideal geometry of the test chart. The following Fig. 6a illustrates
an analysis of an image captured with the Z filter, displaying the measurement
of every single pattern across the entire field (red dots), along with a
continuous line calculated through numerical fitting (blue line). Using this
approach, Fig. 6b presents the result of the fitted lines for each image taken
with the different filters.

---
Figure 6:  Distortion measurement: a) Actual measurements of each pattern with
the Z filter along with the fitted line, and b) numerically fitted curves for
all filters.

###  3.3 Resolution Test

The next test to discuss is the sharpness test, which evaluates the system’s
capability to resolve fine details in a scene. The method used is a variant of
the widely used slanted-edge spatial frequency response test, as outlined in
the ISO 12233 standard[29]. In this approach, samples from various horizontal
or vertical scan lines are adjusted according to their estimated edge
positions. These adjusted samples are then merged to create a new estimate of
the edge transition with enhanced resolution. The spatial frequency response
is derived by performing a Fourier transform on the first derivative of the
edge transition function[30]. The variation in this test lies in using the
circular edge instead of an extended linear edge[31]. Working with the
circle’s perimeter also allows us to increase measurement resolution. This
method lets us use the same test chart that was used for distortion
calculations, along with the same images. There are both advantages and
disadvantages to this approach, which will be discussed at the end of this
section, along with a comparison of the results against theoretical
predictions derived from the optical design software (Zemax).

---
Figure 7:  Edge Spread Function Calculation: a) selected pattern, and b) ESF
of circle’s boundary.

The study begins by following the same instructions used in the distortion
test. The selected pattern is used in a convolution procedure to obtain the
centroids of the patterns across the image. Once the centroids are obtained, a
segment of the image containing a pattern to be studied is selected, as shown
in Fig. 7a. To calculate the circle’s boundary, the coordinate space is in
radial units. The study is limited to values within a radial range of $0.075$
to $0.240$ mm, these numbers are suitable for our application. Next, the image
is analyzed pixel by pixel, and when a pixel falls within the specified
region, its position and intensity are recorded in separate arrays. Figure 7b
shows a scatter plot of the results, with the ESF given as Radius [mm] versus
Intensity [counts]. It is important to note that these recorded values are
neither ordered nor uniformly spaced. The reference position of 0.0 mm, in
Fig. 7b, represents the distance from the ideal pattern radius which is
$0.164$ mm. The next step is to create two new arrays where the position
vector has a constant separation. In this example, the sampling period
achieved was 2.36 microns, a measurement at least five times smaller than the
camera’s pixel size, allowing us to conduct the Modulation Transfer Function
(MTF) study described below.

---
Figure 8:  ESF, LSF and MTF Calculation.

After obtaining the super-resolved version of the edge transition, the line-
spread function (LSF) is estimated by differentiating the edge spread function
(ESF). This differentiation can amplify high-frequency noise, so to mitigate
this, a Gaussian curve is fitted to the LSF. The FWHM can then be derived
directly from the Gaussian fit. Lastly, the MTF is calculated by applying a
Fourier transform to the LSF curve. Fig. 8 illustrates the process, showing
the steps from the ESF to the LSF, and ultimately to the MTF.

---
Figure 9:  FWHM Results: a) Actual measurements with fitted lines derived from
Monte Carlo cases, and b) Fitted lines from experimental data taken with each
filter.

As mentioned earlier, the FWHM can be derived from the LSF. By applying this
procedure to each pattern throughout the image, it’s possible to generate a
map of FWHM values across the field of view. In Fig. 9a, the individual FWHM
measurements taken from a specific image—using Filter R—are plotted. By
fitting a line through these values, the dotted line is obtained. This fitted
line can be compared with other lines representing the estimated FWHM from the
best, mean, and worst designs according to the Monte Carlo analysis. The
observation that $90\%$ of the measurements fall within the lines representing
the best and worst cases from the Monte Carlo analysis provides confidence in
the method’s validity. Repeating this analysis with images taken for each
filter results in Fig. 9b, illustrating the overall performance of the AC
unit.

##  4 AC Unit on Telescope Simulator

When the image quality test was performed, the optimal focus positions for
each filter were recorded, along with the best position for the filter wheel.
After completing test, the AC Unit was installed in the telescope simulator
and the numbers registered were checked. The filter wheel’s position values
were consistent, while the focal positions required slight adjustments to
reach their final settings, see Table 2.

Table 2: Compensator and Filter Wheel Positions. Filter |  Linear Stage [mm] |  Filter Wheel [Deg]
---|---|---
$R$ |  $9.4$ |  $239$
$Y$ |  $5.8$ |  $14$
$I$ |  $8.2$ |  $149$
$Z$ |  $6.5$ |  $104$
$G$ |  $11.4$ |  $59$
$V$ |  $10.3$ |  $194$
$U$ |  $12.7$ |  $284$
$E$ |  $6.7$ |  $329$

In terms of optical performance, images were captured using the telescope
simulator, which provides a $F/11$ beam with a halogen lamp. The following
PSFs were obtained using the 7 available filters and the free position of the
filter wheel, as shown in Fig. 10. The graphs represent a $5x5$ matrix showing
the information around the pixel with the maximum energy, with the total sum
of the entire matrix equal to unity. When the light is concentrated in one or
two pixels, the software used to capture the images has difficulty providing a
reliable FWHM reading. This is why zooming in on a $5x5$ region of the total
image helps visualize the energy distribution in greater detail. With this
approach, the FWHM is typically contained within a radius of 2 pixels,
indicating satisfactory performance. The results suggest that the system
maintains a high level of precision and focus. Additionally, the flexure test
revealed that the PSF center of mass shifted by an average of just 1 pixel
during a full rotation of the flange. This limited displacement underscores
the stability and robustness of the system’s alignment under varying
conditions.

---
Figure 10:  PSF Study of the AC Unit mounted on the Telescope Simulator. The
study displays the PSFs captured with each filter slot: a) Free slot, b)
Filter R, c) Filter Z, d) Filter Y, e) Filter I, f) Filter G, g) Filter V, and
h) Filter U.

##  5 CONCLUSIONS

The instrument is now fully integrated and operational. Final tests are being
completed, and the required documentation is being prepared for transporting
the instrument to the telescope[32, 33, 34, 35, 36, 37, 38]. The AC Unit was
installed in the summer of 2022, with testing conducted in the summer of 2023.
Although there were setbacks and mechanical issues with the optical components
that complicated the alignment process, the problems were successfully
resolved, and the system is now functioning as expected.

##  6 ACKNOWLEDGMENT

As part of the AC Unit-SOXS team, we’d like to extend our heartfelt thanks to
the Padova work team for making us feel at home. We appreciate your support,
guidance, and patience. Special thanks to Kalyan, Federico, Simone, Gigi,
Davide, and Riccardo Claudi. We also want to acknowledge the Shark team, with
whom we shared a workspace—they were equally welcoming. We’re grateful to
Mirko and Rosario from the SOXS electronics team for their help in resolving
technical issues during system integration and testing. Thanks also to Matteo
Alliverti and Matteo Munari for their support and guidance on aspects related
to mechanical and optical design. Rafael Izazaga provided invaluable advice
and assistance during the procurement of optical components, and Andrea Tozzi
offered tremendous support throughout the integration process.

## References

  * [1] Sanchez, R. Z. and et al, “Optical design of the soxs spectrograph for eso ntt,” in [Proc. of SPIE Vol. 10702, Ground-based and Airborne Instrumentation for Astronomy VII ], (2018).
  * [2] Schipani, P. and et al, “Soxs: a wide band spectrograph to follow up transients,” in [Proc. of SPIE Vol. 10702, Ground-based and Airborne Instrumentation for Astronomy VII ], (2018).
  * [3] Aliverti, M. and et al, “The mechanical design of soxs for the ntt,” in [Proc. of SPIE Vol. 10702, Ground-based and Airborne Instrumentation for Astronomy VII ], (2018).
  * [4] Schipani, P. and et al, “Development status of the soxs spectrograph for the eso-ntt telescope,” in [Proc. SPIE 11447, Ground-based and Airborne Instrumentation for Astronomy VIII; 1144709 ], (2020).
  * [5] Genoni, M. and et al, “Soxs end-to-end simulator: development and applications for pipeline design,” in [Proceedings Volume 11450, Modeling, Systems Engineering, and Project Management for Astronomy IX ], (2020).
  * [6] Claudi, R. and et al, “Operational modes and efficiency of soxs,” in [Proceedings Volume 11447, Ground-based and Airborne Instrumentation for Astronomy VIII; 114477C ], (2020).
  * [7] Genoni, M. and et al, “Progress on the simulation tools for the soxs spectrograph: exposure time calculator and end-to-end simulator,” in [Proc. SPIE 12187, Modeling, Systems Engineering, and Project Management for Astronomy X, 121870C ], (2022).
  * [8] Schipani, P. and et al, “Progress on the soxs transients chaser for the eso-ntt,” in [Proc. SPIE 12184, Ground-based and Airborne Instrumentation for Astronomy IX; 121840O ], (2022).
  * [9] Claudi, R. and et al, “Soxs ait: a paradigm for system engineering of a medium class telescope instrument,” in [Proc. SPIE 12184, Ground-based and Airborne Instrumentation for Astronomy IX; 1218484 ], (2022).
  * [10] Scuderi, S. and et al, “The vacuum and cryogenics system of the soxs spectrograph,” in [Proc. SPIE 12188, Advances in Optical and Mechanical Technologies for Telescopes and Instrumentation V; 1218844 ], (2022).
  * [11] Claudi, R. and et al, “The common path of soxs (son of x-shooter),” in [Proc. of SPIE Vol. 10702, Ground-based and Airborne Instrumentation for Astronomy VII ], (2018).
  * [12] Biondi, F. and et al, “The aiv strategy of the common path of son of x-shooter,” in [Proc. of SPIE Vol. 11447, Ground-based and Airborne Instrumentation for Astronomy VIII ], (2020).
  * [13] Rubin, A. and et al, “Mits: the multi-imaging transient spectrograph for soxs,” in [Proc. of SPIE Vol. 10702, Ground-based and Airborne Instrumentation for Astronomy VII ], (2018).
  * [14] Cosentino, R. and et al, “The vis detector system of soxs,” in [Proc. of SPIE Vol. 10702, Ground-based and Airborne Instrumentation for Astronomy VII ], (2018).
  * [15] Rubin, A. and et al, “Progress on the uv-vis arm of soxs,” in [Proceedings Volume 11447, Ground-based and Airborne Instrumentation for Astronomy VIII; 114475L ], (2020).
  * [16] Cosentino, R. and et al, “Development status of the uv-vis detector system of soxs for the eso-ntt telescope,” in [Proc. of SPIE Vol. 11447, Ground-based and Airborne Instrumentation for Astronomy VIII ], (2020).
  * [17] Cosentino, R. and et al, “Laboratory test of the vis detector system of soxs for the eso-ntt telescope,” in [Proc. SPIE 12184, Ground-based and Airborne Instrumentation for Astronomy IX; 121845I ], (2022).
  * [18] Kuncarayakti, H. and et al, “Design and development of the soxs calibration unit,” in [Proc. of SPIE Vol. 11447, Ground-based and Airborne Instrumentation for Astronomy VIII ], (2020).
  * [19] Brucalassi, A. and et al, “The acquisition camera system for soxs at ntt,” in [Proc. of SPIE Vol. 10702, Ground-based and Airborne Instrumentation for Astronomy VII ], (2018).
  * [20] Brucalassi, A. and et al, “Final design and development status of the acquisition and guiding system for soxs,” in [Proc. of SPIE Vol. 11447, Ground-based and Airborne Instrumentation for Astronomy VIII ], (2020).
  * [21] Araiza-Durán, J. A. and et al, “The integration and alignment phase for the acquisition and guiding system of soxs,” in [Proc. SPIE 12184, Ground-based and Airborne Instrumentation for Astronomy IX; 1218483 ], (2022).
  * [22] Andor, “Andor bex2-dd chart.” https://andor.oxinst.com/assets/uploads/products/andor/documents/andor-ikon-m-934-specifications.pdf.  (Accessed: 07 May 2024).
  * [23] Ricci, D. and et al, “Architecture of the soxs instrument control software,” in [Proc. of SPIE Vol. 10707, Software and Cyberinfrastructure for Astronomy V ], (2018).
  * [24] Ricci, D. and et al, “Development status of the soxs instrument control software,” in [Proc. of SPIE Vol. 11452, Software and Cyberinfrastructure for Astronomy VI ], (2020).
  * [25] Thorlabs, “Test chart r3l3s5p.” https://www.thorlabs.com/thorproduct.cfm?partnumber=R3L3S5P.  (Accessed: 07 May 2024).
  * [26] Greivenkamp, J. E., [Field Guide to Geometrical Optics ], SPIE Press (2004).
  * [27] Edmund, “Distortion test 2.” https://www.edmundoptics.com/knowledge-center/application-notes/imaging/distortion/. (Accessed: 07 May 2024).
  * [28] Engineering, I., “Distortion test.” https://www.image-engineering.de/library/image-quality/factors/1062-distortion#:~:text=Local%20geometric%20distortion%20method%20(ISO%2017850),-The%20local%20geometric&text=When%20measuring%20the%20local%20geometric,the%20center%20of%20the%20image. (Accessed: 07 May 2024).
  * [29] K, M., “Accuracy and precision of edge-based modulation transfer function measurement for sampled imaging systems,” IEEE Access 6(22), 41079–41086 (2018).
  * [30] Xie X, Fan H, W. A. Z. N. and Y, Z., “Regularized slanted-edge method for measuring the modulation transfer function of imaging systems,” OSA: Applied Optics 57(22), 6552–6558 (2018).
  * [31] L, B. R., “Circular-edge spatial frequency response test,” in [Proc. of SPIE Vol. 5294, Image Quality and System Performance ], (2003).
  * [32] S., K. K. R. and et al, “From assembly to the complete integration and verification of the soxs common path,” in [Proc. SPIE 12184, Ground-based and Airborne Instrumentation for Astronomy IX; 1218482 ], (2022).
  * [33] Battaini, F. and et al, “The internal alignment and validation of a powered adc for soxs,” in [Proc. SPIE 12184, Ground-based and Airborne Instrumentation for Astronomy IX; 1218480 ], (2022).
  * [34] Aliverti, M. and et al, “Soxs mechanical integration and verification in italy,” in [Proc. SPIE 12184, Ground-based and Airborne Instrumentation for Astronomy IX; 1218484 ], (2022).
  * [35] Sanchez, R. Z. and et al, “Soxs: effects on optical performances due to gravity flexures, temperature variations, and subsystems alignment,” in [Proc. of SPIE Vol. 11447, Ground-based and Airborne Instrumentation for Astronomy VIII ], (2020).
  * [36] Aliverti, M. and et al, “Manufacturing, integration, and mechanical verification of soxs,” in [Proc. of SPIE Vol. 11452, Software and Cyberinfrastructure for Astronomy VI ], (2020).
  * [37] Colapietro, M. and et al, “Progress and tests on the instrument control electronics for soxs,” in [Proceedings Volume 11452, Software and Cyberinfrastructure for Astronomy VI; 1145225 (2020) ], (2020).
  * [38] Landoni, M. and et al, “The quality check system architecture for son-of-x-shooter soxs,” in [Proc. SPIE 12189, Software and Cyberinfrastructure for Astronomy VII; 121890L ], (2022).


# Architectural Support for Efficient Data Movement
in Disaggregated Systems

Christina Giannoula§$\dagger$ *Kailong Huang§ *Jonathan Tang§ s Nectarios
Koziris$\dagger$ Georgios Goumas$\dagger$ Zeshan Chishti$\ddagger$ Nandita
Vijaykumar§ _§ University of Toronto $\dagger$National Technical University of
Athens $\ddagger$Intel Corporation _

##  1 Data Movement in Disaggregated Systems

Traditional data centers include monolithic servers that tightly integrate
CPU, memory and disk (Figure 1a). Instead, _Disaggregated Systems_ (DSs) [1,
2, 3] organize multiple compute (CC), memory (MC) and storage devices as
_independent, failure-isolated_ components interconnected over a high-
bandwidth network (Figure 1b). DSs can greatly reduce data center costs by
providing improved resource utilization, resource scaling, failure-handling
and elasticity in modern data centers [4, 1, 2, 3, 5]. ††* Kailong Huang and
Jonathan Tang have equal contribution.

Figure 1: (a) Traditional systems vs (b) DSs.

The MCs provide large pools of main memory (_remote memory_), while the CCs
include the on-chip caches and a few GBs of DRAM (_local memory_) that acts as
a cache of _remote memory_. In this context, a large fraction of the
application’s data ($\sim 80$%) [1, 3, 6] is located in remote memory, and can
cause large performance penalties (see Figure 2) from remotely accessing data
over the network.

Alleviating data access overheads is challenging in DSs for the following
reasons. First, DSs are not monolithic and comprise independently managed
entities: each component has its own hardware controller, and a specialized
kernel monitor uses its own functionality to manage the component it runs on
(only communicates with other monitors via network messaging if there is a
need to access remote resources). This characteristic necessitates a
distributed and disaggregated solution that can scale to a large number of
independent components in the system. Second, there is high variability in
remote memory access latencies since they depend on the locations of the MCs,
contention from other jobs that share the same network and MCs, and data
placements that can vary during runtime or between multiple executions. This
necessitates a solution that is robust towards fluctuations in the
network/remote memory bandwidth and latencies. Third, a major factor behind
the performance slowdowns is the commonly-used approach in DSs [3, 1, 4, 6] of
moving data at page granularity. This approach effectively provides software
transparency, low metadata costs in memory management, and high spatial
locality in many applications. However, it can cause high bandwidth
consumption and network congestion, and often significantly slows down
accesses to critical path cache lines in other concurrently accessed pages.

Figure 2 presents a performance analysis of different data movement schemes
using DSs for a representative set of heterogeneous workloads. We evaluate a
MC and a CC with _local memory_ to fit $\sim$20% of the application data, and
the network bandwidth to be 1/2-1/8$\times$ of the bus bandwidth [1, 6].
Performance of all schemes is normalized to that of the monolithic approach
where _all_ data fits in the _local memory_ of the CC. We make three
observations.

Figure 2: Data movement overheads in DSs.

First, the typically-used _page_ scheme of moving data at a page granularity
incurs large slowdowns over the monolithic _Local_ configuration due to
transferring large amounts of data over the network that slows down accesses
to other pages. Instead, when pages are moved _for free_ , i.e., _page-free_ ,
performance significantly improves thanks to spatial locality benefits within
pages. Second, moving data always at a _fixed_ granularity (cache-line via LLC
or page via _local memory_) cannot provide robustness across heterogeneous
applications and network configurations: some applications can benefit from
cache line (e.g., _pr_ , _nw_) or page granularity (e.g., _pf_ , _dr_) data
movements, while the best-performing granularity also depends on network
characteristics (e.g., _bf_ , _ts_). Third, _naively_ moving data at _both_
granularities (cache-line+page) to serve data requests with the latency of the
packet that arrives earlier to the CC is still quite inefficient, while
critical cache line requests are still queued behind large concurrently
accessed pages. In contrast, Figure 2 shows that _DaeMon_ (see §3)
significantly reduces data movement overheads in DSs across various
network/application characteristics.

##  2 Prior Work

Prior works [1, 7, 8, 9, 4, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 3, 20, 21,
2, 22, 6, 23, 24, 25, 26] propose OS kernels, system-level solutions, software
management systems, architectures for DSs. These works do not tackle the data
movement challenge in DSs, and thus our work is orthogonal to them.

Prior works on hybrid systems [27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38,
39] integrate die-stacked DRAM [40] as DRAM cache of a large main memory [32,
27, 29] in a monolithic server, and tackle high page movement costs in two-
tiered physical memory via page placement/hot page selection schemes or by
moving data at smaller granularity, e.g., cache line. However, data movement
in DSs poses fundamentally different challenges. First, accesses across the
network are significantly slower than within the server, thus intelligent page
placement/movement cannot by itself address these high costs. Second, DSs
incur significant variations in access latencies based on the current network
architecture and concurrent jobs sharing the MCs/network, thus necessitating
an solution primarily designed for robustness to this variability. Finally,
DSs include independently managed MCs and networks shared by independent CCs
running unknown jobs. Thus, unlike hybrid systems, the solution cannot assume
that the memory management at the MCs can be fully controlled by the CPU side.
Our work is the first to examine the data movement problem in fully DSs, and
design an effective solution for DSs.

##  3 _DaeMon_ ’s Key Ideas

_DaeMon_ (Figure 3) is an adaptive and scalable mechanism to alleviate data
costs in DSs, consisting of three techniques.

Figure 3: High-level overview of _DaeMon_.

(I) Decoupled Multiple Granularity Data Movement. We integrate two separate
hardware queues to serve data requests from _remote memory_ at two
granularities, i.e., cache line (via the _sub-block queue_ to LLC) and page
(via the _page queue_ to _local memory_) granularity, and effectively
prioritize moving cache lines over moving pages via a _bandwidth partitioning_
approach: a queue controller serves cache line requests with a higher
_predefined fixed rate_ than page requests to ensure that any given time a
certain fraction of the bandwidth resources is always allocated to serve cache
line moves fast. This key technique enables (i) low metadata overheads by
retaining page migrations, (ii) high performance by leveraging data locality
within pages, and (iii) fewer slowdowns in cache line data movements that are
on the critical path, from expensive page moves that may have been previously
triggered.

(II) Selection Granularity Data Movement. To provide an adaptive data movement
solution, we include in each CC two separate hardware buffers to track pending
data migrations for both cache line (via the _inflight sub-block buffer_) and
page (via the _inflight page buffer_) granularity, and a selection granularity
unit to decide if a data request should be served by cache line, page or
_both_ based on the utilization of the _inflight_ buffers. Given that _DaeMon_
prioritizes cache line over page moves, the inflight buffers are utilized in
different ways, allowing us to capture the application behavior and the system
load during runtime. For example: (a) If there is _low locality_ within pages,
the page buffer has higher utilization than the sub-block buffer (cache lines
are prioritized), thus the selection unit favors moving cache lines and
throttles pages (and vice-versa). (b) Under low _bandwidth utilization_
scenarios, the page buffer utilization is low, thus the selection unit
schedules more page movements (or both granularities) to obtain data locality
benefits (and vice-versa).

(III) Link Compression on Page Movements. We employ hardware compression units
at both the CCs and MCs to highly compress pages migrated over the network.
_Link compression_ on page moves reduces the network bandwidth consumption and
alleviates network bottlenecks.

Synergy of Three Techniques. _DaeMon_ cooperatively integrates all three
techniques to significantly alleviate data movement overheads, and provide
robustness towards network, architecture and application characteristics:
(1) Prioritizing requested cache lines helps _DaeMon_ to tolerate high
(de)compression latencies in page migrations over the network, while also
leveraging benefits of page migrations (low metadata costs, spatial locality
in pages).
(2) Compressed pages consume less network bandwidth, enabling _DaeMon_ to
reserve part of the bandwidth to effectively prioritize critical path cache
line accesses.
(3) Compression on page moves helps _DaeMon_ to adapt to the data
compressibility: if the pages are highly compressible, the inflight page
buffer empties at a faster rate, and _DaeMon_ favors sending more pages (and
vice-versa).

##  4 _DaeMon_ ’s Key Results

We extend Sniper [41] to simulate DSs, and evaluate capacity intensive
workloads from graph processing, HPC, data analytics, bioinformatics, machine
learning domains. _DaeMon_ reduces data access costs and improves performance
by 3.06$\times$ and 2.39$\times$, respectively, over the widely-adopted
approach of moving data at a page granularity. _DaeMon_ leverages the synergy
of all three techniques to provide robustness, while retaining the spatial
locality, transparency and metadata management benefits of page granularity
movements. We show that _DaeMon_ achieves high system performance for various
network/architecture configurations and applications (Figure 4 top), and
multiple concurrently running jobs in the DS (Figure 4 bottom), compared to
the widely-adopted approach of moving data at page granularity.

Figure 4: _DaeMon_ ’s benefits over the _page_ scheme, (i) varying the MCs,
network bandwidth and application, and (ii) running multiple applications in a
4-CPU CC and a MC.

## References

  * [1] Yizhou Shan, Yutong Huang, Yilun Chen, and Yiying Zhang.  LegoOS: A Disseminated, Distributed OS for Hardware Resource Disaggregation.  In OSDI, 2018.

  * [2] Zhiyuan Guo, Yizhou Shan, Xuhao Luo, Yutong Huang, and Yiying Zhang.  Clio: A Hardware-Software Co-Designed Disaggregated Memory System.  In ASPLOS, 2022.
  * [3] Seung-seob Lee, Yanpeng Yu, Yupeng Tang, Anurag Khandelwal, Lin Zhong, and Abhishek Bhattacharjee.  MIND: In-Network Memory Management for Disaggregated Data Centers.  In SOSP, 2021.
  * [4] Irina Calciu, M. Talha Imran, Ivan Puddu, Sanidhya Kashyap, Hasan Al Maruf, Onur Mutlu, and Aasheesh Kolli.  Rethinking Software Runtimes for Disaggregated Memory.  In ASPLOS, 2021.
  * [5] Christina Giannoula.  Accelerating Irregular Applications via Efficient Synchronization and Data Access Techniques.  In arXiv, 2022.
  * [6] Peter X. Gao, Akshay Narayan, Sagar Karandikar, Joao Carreira, Sangjin Han, Rachit Agarwal, Sylvia Ratnasamy, and Scott Shenker.  Network Requirements for Resource Disaggregation.  In OSDI, 2016.
  * [7] Sebastian Angel, Mihir Nanavati, and Siddhartha Sen.  Disaggregation and the Application.  In HotCloud, 2020.
  * [8] Vlad Nitu, Boris Teabe, Alain Tchana, Canturk Isci, and Daniel Hagimont.  Welcome to Zombieland: Practical and Energy-Efficient Memory Disaggregation in a Datacenter.  In EuroSys, 2018.
  * [9] Sangjin Han, Norbert Egi, Aurojit Panda, Sylvia Ratnasamy, Guangyu Shi, and Scott Shenker.  Network Support for Resource Disaggregation in Next-Generation Datacenters.  In HotNets, 2013.
  * [10] Marcos K. Aguilera, Nadav Amit, Irina Calciu, Xavier Deguillard, Jayneel Gandhi, Pratap Subrahmanyam, Lalith Suresh, Kiran Tati, Rajesh Venkatasubramanian, and Michael Wei.  Remote Memory in the Age of Fast Networks.  In SoCC, 2017.
  * [11] Qizhen Zhang, Yifan Cai, Sebastian G. Angel, Vincent Liu, Ang Chen, and B. T. Loo.  Rethinking Data Management Systems for Disaggregated Data Centers.  In CIDR, 2020.
  * [12] Kevin Lim, Yoshio Turner, Jose Renato Santos, Alvin AuYoung, Jichuan Chang, Parthasarathy Ranganathan, and Thomas F. Wenisch.  System-Level Implications of Disaggregated Memory.  In HPCA, 2012.
  * [13] Chenxi Wang, Haoran Ma, Shi Liu, Yuanqi Li, Zhenyuan Ruan, Khanh Nguyen, Michael D. Bond, Ravi Netravali, Miryung Kim, and Guoqing Harry Xu.  Semeru: A Memory-Disaggregated Managed Runtime.  In OSDI, 2020.
  * [14] Pengfei Zuo, Jiazhao Sun, Liu Yang, Shuangwu Zhang, and Yu Hua.  One-sided RDMA-Conscious Extendible Hashing for Disaggregated Memory.  In ATC, 2021.
  * [15] Laurent Bindschaedler, Ashvin Goel, and Willy Zwaenepoel.  Hailstorm: Disaggregated Compute and Storage for Distributed LSM-Based Databases.  In ASPLOS, 2020.
  * [16] Ivy Peng, Roger Pearce, and Maya Gokhale.  On the Memory Underutilization: Exploring Disaggregated Memory on HPC Systems.  In SBAC-PAD, 2020.
  * [17] Andres Lagar-Cavilla, Junwhan Ahn, Suleiman Souhlal, Neha Agarwal, Radoslaw Burny, Shakeel Butt, Jichuan Chang, Ashwin Chaugule, Nan Deng, Junaid Shahid, Greg Thelen, Kamil Adam Yurtsever, Yu Zhao, and Parthasarathy Ranganathan.  Software-Defined Far Memory in Warehouse-Scale Computers.  In ASPLOS, 2019.
  * [18] Juncheng Gu, Youngmoon Lee, Yiwen Zhang, Mosharaf Chowdhury, and Kang G. Shin.  Efficient Memory Disaggregation with Infiniswap.  In NSDI, 2017.
  * [19] Marcos K. Aguilera, Nadav Amit, Irina Calciu, Xavier Deguillard, Jayneel Gandhi, Stanko Novaković, Arun Ramanathan, Pratap Subrahmanyam, Lalith Suresh, Kiran Tati, Rajesh Venkatasubramanian, and Michael Wei.  Remote Regions: A Simple Abstraction for Remote Memory.  In ATC, 2018.
  * [20] Christian Pinto, Dimitris Syrivelis, Michele Gazzetti, Panos Koutsovasilis, Andrea Reale, Kostas Katrinis, and H. Peter Hofstee.  ThymesisFlow: A Software-Defined, HW/SW co-Designed Interconnect Stack for Rack-Scale Memory Disaggregation.  In MICRO, 2020.
  * [21] K. Katrinis, D. Syrivelis, D. Pnevmatikatos, G. Zervas, D. Theodoropoulos, I. Koutsopoulos, K. Hasharoni, D. Raho, C. Pinto, F. Espina, S. Lopez-Buedo, Q. Chen, M. Nemirovsky, D. Roca, H. Klos, and T. Berends.  Rack-Scale Disaggregated Cloud Data Centers: The dReDBox Project Vision.  In DATE, 2016.
  * [22] Dhantu Buragohain, Abhishek Ghogare, Trishal Patel, Mythili Vutukuru, and Purushottam Kulkarni.  DiME: A Performance Emulator for Disaggregated Memory Architectures.  In APSys, 2017.
  * [23] Pramod Subba Rao and George Porter.  Is Memory Disaggregation Feasible? A Case Study with Spark SQL.  In ANCS, 2016.
  * [24] Georgios Zervas, Hui Yuan, Arsalan Saljoghei, Qianqiao Chen, and Vaibhawa Mishra.  Optically Disaggregated Data Centers with Minimal Remote Memory Latency: Technologies, Architectures, and Resource Allocation.  JOCN, 2018.
  * [25] Donghyun Gouk, Sangwon Lee, Miryeong Kwon, and Myoungsoo Jung.  Direct Access, High-Performance Memory Disaggregation with DirectCXL.  In ATC, 2022.
  * [26] Yang Zhou, Hassan M. G. Wassel, Sihang Liu, Jiaqi Gao, James Mickens, Minlan Yu, Chris Kennelly, Paul Turner, David E. Culler, Henry M. Levy, and Amin Vahdat.  Carbink: Fault-Tolerant Far Memory.  In OSDI, 2022.
  * [27] Xiangyu Dong, Yuan Xie, Naveen Muralimanohar, and Norman P. Jouppi.  Simple but Effective Heterogeneous Main Memory with On-Chip Memory Controller Support.  In SC, 2010.
  * [28] Haikun Liu, Yujie Chen, Xiaofei Liao, Hai Jin, Bingsheng He, Long Zheng, and Rentong Guo.  Hardware/Software Cooperative Caching for Hybrid DRAM/NVM Memory Architectures.  In ICS, 2017.
  * [29] Xiaowei Jiang, Niti Madan, Li Zhao, Mike Upton, Ravishankar Iyer, Srihari Makineni, Donald Newell, Yan Solihin, and Rajeev Balasubramonian.  CHOP: Adaptive Filter-Based DRAM Caching for CMP Server Platforms.  In HPCA, 2010.
  * [30] Jagadish B. Kotra, Haibo Zhang, Alaa R. Alameldeen, Chris Wilkerson, and Mahmut T. Kandemir.  CHAMELEON: A Dynamically Reconfigurable Heterogeneous Memory System.  In MICRO, 2018.
  * [31] Andreas Prodromou, Mitesh Meswani, Nuwan Jayasena, Gabriel Loh, and Dean M. Tullsen.  MemPod: A Clustered Architecture for Efficient and Scalable Migration in Flat Address Space Multi-level Memories.  In HPCA, 2017.
  * [32] Neha Agarwal and Thomas F. Wenisch.  Thermostat: Application-Transparent Page Management for Two-Tiered Main Memory.  In ASPLOS, 2017.
  * [33] Mitesh R. Meswani, Sergey Blagodurov, David Roberts, John Slice, Mike Ignatowski, and Gabriel H. Loh.  Heterogeneous Memory Architectures: A HW/SW Approach for Mixing Die-Stacked and Off-Package Memories.  In HPCA, 2015.
  * [34] Kai Wu, Yingchao Huang, and Dong Li.  Unimem: Runtime Data Managementon Non-Volatile Memory-Based Heterogeneous Main Memory.  In SC, 2017.
  * [35] Gabriel H. Loh and Mark D. Hill.  Efficiently Enabling Conventional Block Sizes for Very Large Die-Stacked DRAM Caches.  In MICRO, 2011.
  * [36] Gabriel Loh and Mark D. Hill.  Supporting Very Large DRAM Caches with Compound-Access Scheduling and MissMap.  IEEE Micro, 2012.
  * [37] Chia Chen Chou, Aamer Jaleel, and Moinuddin K. Qureshi.  CAMEO: A Two-Level Memory Organization with Capacity of Main Memory and Flexibility of Hardware-Managed Cache.  In MICRO, 2014.
  * [38] Jee Ho Ryoo, Mitesh R. Meswani, Andreas Prodromou, and Lizy K. John.  Silc-fm: Subblocked interleaved cache-like flat memory organization.  In HPCA, 2017.
  * [39] Djordje Jevdjic, Stavros Volos, and Babak Falsafi.  Die-Stacked DRAM Caches for Servers: Hit Ratio, Latency, or Bandwidth? Have It All with Footprint Cache.  In ISCA, 2013.
  * [40] Hongshin Jun, Jinhee Cho, Kangseol Lee, Ho-Young Son, Kwiwook Kim, Hanho Jin, and Keith Kim.  HBM DRAM Technology and Architecture.  In IMW, 2017.
  * [41] Trevor E. Carlson, Wim Heirman, and Lieven Eeckhout.  Sniper: Exploring the Level of Abstraction for Scalable and Accurate Parallel Multi-Core Simulations.  In SC, 2011.