longtable captionstableheading

# Minute-by-Minute: Financial Markets’ Reaction to the 2020 U.S. Election

Matthew DeHaven  Hannah Firestone  Chris Webster

(2024-07-03)

Authors’ note: This short article was written June 2021.

## Introduction111We thank Ricardo Correa and Juan M. Londono for their useful
comments. All errors remain our own.

The 2020 U.S. presidential election garnered notable global attention from
market participants, which was reflected, to some degree, in immediate changes
in asset prices. Since financial markets respond rapidly to new and diverse
information, it is difficult to disentangle any one event’s impact. However,
the election offers a unique opportunity to overcome this identification
obstacle: major results came in overnight, a sufficiently small window that
concurrent events were limited in scope. Using a novel data source to get
minutely estimates of probabilities for the presidential outcome, we examine
the relationship between the 2020 election and various financial markets.

We find striking correlations between the presidential election outcome
probability and major financial indicators, including USD currency pairs, bond
prices, stock index futures, and a market volatility measure. The correlations
are consistent with ‘risk-on’ behavior in markets, a term which describes
investors moving toward riskier asset classes, as the election results became
clearer. Further, we decompose the market reaction into a ‘reduction in
uncertainty’ component and a ‘probability of a Democratic party presidency’
component. This decomposition reveals how markets reacted to the increasing
certainty of the outcome as election results came in. Finally, we analyze the
differing market reactions to the presidential election and the Senate
election, including data from the unique Georgia runoffs, and demonstrate that
bond prices were particularly sensitive to the probability of a combined
Democratic Senate and Presidency.

## Background and Data

Prediction markets offer a unique source of data on political and economic
events. Betting on U.S. elections has been commonplace since at least the late
1800s (Rhode & Strumpf, 2004; Wolfers, 2009), and the advent of the internet
in the 1990s introduced centralized ‘idea futures’ markets with many users
(Hanson, 1996). These markets match two users to purchase a ‘yes’ and ‘no’
share for a contract (that is, “Will candidate X be elected?”) at a price
ranging from 0 to 1. In an efficient market, the resulting price of a ‘yes’
share can be interpreted as the probability of the event ‘X’ happening. For
example, if the price of a ‘yes’ share was $0.60 and a user predicted the true
probability of the event to be 0.9, the user would buy a ‘yes’ share, making
an expected profit of $0.30 per share. If many buyers expected the same
probability of 0.9 the price of the ‘yes’ share would rise closer to $0.90.
With enough users and enough information, the market price converges to the
expected probability of the event occurring.

Since the popularization of these betting markets in the 1990s, there has been
considerable research on their utility in predicting economic variables (Hahn
& Tetlock, 2005; Snowberg et al., 2013), forecasting elections (Arrow et al.,
2008; Reade & Vaughan Williams, 2019), and aiding corporate decision making
(O’Leary, 2015; Plott & Chen, 2002; Spears et al., 2009). The data in these
studies is often at the daily level, or even more infrequent. In this note, we
take advantage of a novel source of minutely prediction market data from
PredictIt. PredictIt is an online prediction market in the U.S., offering
markets on various U.S. and global political events. For each election (i.e.
party control of the Senate and Presidency), we get minutely data on the
market ‘yes’ and ‘no’ prices, and we take the most recent trade to arrive at
the market-implied event probability.

To isolate financial markets’ reaction to the election, we narrow the time
range to election night (November 3, 2020) and the following day. Over the
course of the night, there was a remarkable shift in the perceived probability
of either the Democratic or Republican candidate winning the presidential
election. The Democratic candidate’s probability of winning the election
(Democratic Presidency Probability) ranged from about 20% to 90% over the
course of 30 hours, providing enough variation to find statistically
significant relationships. As shown in Panel A of Figure 1, at 6:00 p.m. on
November 3, the Democratic candidate’s probability of winning the presidency
was close to 70%. As results were published, the probability fell all the way
to 20%, before steadily rebounding late into the night.

Note: Observations are by the minute. The Democratic Presidency Probability
shows the latest traded price (which can be interpreted as a probability) from
PredictIt. The AFE-USD currency index is represented as a percent change from
the value before 6:00 p.m. on November 3rd, 2020, with higher values denoting
the depreciation of the dollar.

Source: PredictIt, Bloomberg.

Figure 1: Markets during Election Night 2020.

These large shifts reflect two factors: (1) the variable times at which states
reported results, and (2) the realization of a large polling miss.222The
weighted-average error of polls within three weeks of the 2020 election was
4.2 percentage points in favor of the Democratic party candidate, compared to
a historical average of 1.3 percentage points in favor of the Democratic party
candidate (Silver, 2021). Some states report most of their results early in
the night while others do not release comprehensive counts until late into the
night or even the following day. This trickle of information provides ample
time for markets to adjust their perceptions of the likeliest outcomes of the
election. In addition, it became obvious that expectations about the election
outcome, which strongly favored Democrats based on polls, had been inaccurate.
The actual election results had a much closer margin than expected, which
forced investors to recalibrate throughout the night.

Among the important financial markets that clearly react to the events of the
election, Panel B of Figure 1 plots an AFE-USD currency index (AFE-USD Index)
as a percent change from its price prior to 6:00 p.m. on November 3.333Our
AFE-USD Index measures the exchange rate of the dollar against a basket of
major advanced foreign economy currencies (CAD, CHF, EUR, GBP, JPY, and SEK),
borrowing methodology from the Federal Reserve Dollar Indexes (von Beschwitz
et al., 2019). Likewise, our EME-USD Index does the same using emerging market
economy currencies (CNY, HKD, MXN, SGD, and THB). Currencies were selected for
inclusion in the indexes based on the availability of minutely price data on
election night. An increase in the index indicates a depreciation of the
dollar. Currency markets are global in nature and react to major real world
events, and unlike equity markets, they trade overnight, so they are well-
positioned to react to information like election results. The AFE-USD Index
clearly reacts to the events of the November presidential election, with an
extremely high correlation of 0.92 between the two series.

## Initial Market Reaction

In Panel A of Figure 2, we plot the Democratic Presidency Probability against
the change in the AFE-USD index across the same time period as above, 6pm
November 3rd through November 4th. The resulting statistically significant
coefficient—highlighted in red—of a simple linear regression, quantifies the
market’s pricing of a Democratic over a Republican victory in the race for the
presidency, in this case implying a depreciation of just over 1% in the price
of the dollar.

Using the same time window of minutely data, we can expand our scope to a host
of markets that are traded overnight: futures for the S&P 500 index, NASDAQ,
and Russell 2000 stock market indexes, the EME-USD Index, future prices for
oil (measured by WTI Crude) and the VIX stock market volatility index, and the
yield of 10-year treasury bonds. Panel B of Figure 2 plots the resulting
coefficients of a linear regression between each of these financial markets
and the Democratic Presidency Probability, with a 95% confidence interval
shown in bars around each estimate. Every relationship is statistically
significant, and only the VIX and the 10-Year Treasury yield444Throughout this
note, Treasury bond movements are quoted in yields (bond yields rise as prices
fall.) have a negative coefficient with the Democratic Presidency Probability.

Note: In Panel A, each point represents one observation of the last traded
price (probability) from PredictIt and the percent change in the AFE-USD
currency index since 6:00 p.m. November 3rd, with higher values denoting the
depreciation of the dollar. The fitted line is a simple linear regression
shown in the upper left. In Panel B, each bar shows the coefficient between a
simple linear regression of one of the financial markets and the Democratic
Presidency Probability, with a 95% confidence interval around the estimate.

Source: PredictIt, Bloomberg, authors’ calculations.

Figure 2: Financial Markets and the Presidency

There are several potential explanations for the observed co-movements between
asset prices and the changing presidency probability. One possibility is an
increasing “risk-on” sentiment, where investors are more willing to own
riskier assets in exchange for higher expected return. Evidence for such a
rotation is in the declining VIX futures, the EME currencies strengthening
more than those of AFEs, and the tech-focused NASDAQ index being the most
positively correlated of any of the assets in our sample. However, a
countervailing piece of evidence against such “risk-on” behavior is the
negative coefficient in the 10-year treasury yield. A typical “risk-on”
rotation would involve an increase in treasury yields as investors sell off
these extremely safe assets. Instead, we observe a negative correlation
between treasury yields and the Democratic presidency probability.

## A Reduction in Uncertainty

It is possible that markets were not only pricing in a Democratic or
Republican victory for the presidency, but also the uncertainty of the
election outcome. In face of a large, uncertain event, financial markets may
respond with a “risk-off” stance, putting investments into safer assets until
the uncertainty is concluded. After the event has resolved, markets then take
on more risk and lower their expectation of future volatility.

This pricing-in of uncertainty is clear in the VIX futures price, plotted
against the Democratic Presidency Probability in Panel A of Figure 3. The
simple linear regression, in blue, has a fairly strong fit to the data, as
demonstrated by its R-squared value of about 0.6, plotted in Panel B. However,
by adding an additional variable capturing the certainty of the election
outcome (calculated as the absolute value of the distance from 50-50 odds),
the model fit of a specification we label “Linear + Certainty” is even higher.
The strength of the relationship is also visually convincing in Panel A of
Figure 3, with the “Linear + Certainty” model plotted with the red line. This
suggests that to some extent, market participants were responding to the level
of certainty of the presidential outcome instead of the outcome itself.

Note: In Panel A, each point represents one observation of the last traded
price (probability) from PredictIt and the percent change in price of the VIX
futures since 6:00 p.m. November 3rd. The fitted line in blue is a simple
linear regression and the line in red has an additional term, the absolute
value of the distance from 50-50 odds. In Panel B, each bar shows the
R-Squared of the two models fitted in Panel A.

Source: PredictIt, Bloomberg, authors’ calculations.

Figure 3: Adding Certainty to the Equation

## Controlling for Other Elections

Earlier we noted that by narrowing the time window to election night, we were
reducing the probability that other events could be affecting financial
markets. Our analysis so far, however, abstracts from a host of other
elections that take place at the same time as the U.S. presidential election.
There are two other crucial outcomes: control of the Senate and of the House
of Representatives. Democrats were the favorites to win control of the House,
and the probability of that outcome did not move much over the course of the
night (ranging from 88% to 99%). Control of the Senate, on the other hand,
moved substantially, favoring Democrats heading into election night (about
70%) and completely switching by the end of the following day with a
Republican win probable (about 80%). An additional market on PredictIt asked
what the probability was of a Democratic “Trifecta” across the federal
government, with control of the Senate, House, and Presidency. The resulting
coefficients of regressing various financial markets on these four
variables—Democratic Presidency Probability, Presidential Certainty,
Democratic Senate Probability, and Democratic Trifecta Probability—are shown
in Figure 4, which paints the most complete picture of how the various
elections of 2020 were priced in by markets.555For the purposes of our
analysis, the inclusion of the Trifecta is similar to adding an interaction
term between the Democratic Presidency and Democratic Senate probabilities.

Note: Each panel shows four bars representing the coefficients on a
multivariable regression between the four prediction market variables and each
of the financial variables. The yellow error bars show the 95% confidence
interval around each estimate.

Source: PredictIt, Bloomberg, authors’ calculations.

Figure 4: A Multivariable Picture of Markets’ Reactions

Looking first at the VIX, we can see that the coefficient on Presidential
Certainty captures the largest amount of movement in the financial market
dependent variables, with the Democratic Presidency Probability still having a
large negative effect, and the Democratic Trifecta Probability associated with
an increase in the VIX. It is likely that the shock effect of one party
controlling all three institutions, which was a surprise outcome, led to
expectations of responsive volatility in equity markets in the near term. In
contrast, the 10-Year Treasury shows a decrease in yield from both of the
Presidential variables (as is consistent with the results in Figure 2), but it
has an implied increase in yield from the Democratic Senate and Trifecta
Probabilities. This is a possible sign that markets interpreted a Democratic
Senate (and even more so a Trifecta) as a precursor of a larger round of
stimulus owing to a unified government being more easily able to pass spending
legislation.

In currency markets, the AFE-USD currency index shows the relatively lowest
absolute movement in response to the election, while the EME-USD currency
index has one of the strongest absolute responses. As before, there is a much
larger implied appreciation of EME currencies compared to AFE currencies for
an increase in both Democratic Presidential Probability and Presidential
Certainty, pointing to “risk-on” behavior. Both currency indexes have a
positive and significant coefficient on Democratic Trifecta Probability, and
the AFE index has a significant negative coefficient on Democratic Senate
Probability.

The stock market index futures exhibit different behaviors, with the Russell
2000 increasing in response to each of the variables, but especially to the
probability of a Democratic Senate as well as Presidential Certainty. The
NASDAQ, on the other hand, has a large implied increase on Presidential
Certainty, but is negatively correlated with Democratic Trifecta Probability.
The S&P 500 shows similar reactions to the NASDAQ but also has significant and
positive (though small) coefficients on Democratic Presidency Probability and
Democratic Senate Probability.

Oil responded most positively to the Democratic Presidency Probability and has
one of the smallest coefficients with Presidential Certainty, suggesting that
the market is interpreting some potential policy differences between the two
presidential outcomes.

## The Georgia Runoffs and Control of the Senate

The November election left the Senate split with 50 Republican seats to 48 for
the Democrats.666This includes the two independents who caucus with the
Democrats in the Democratic total. The final two seats were both in Georgia as
no candidate won 50% of the vote, triggering automatic runoffs that would
decide control of the Senate, with Democrats only needing 50 seats to hold a
majority (using the Vice President’s vote to break ties).

We repeat the analysis above, focusing instead on January 5th from 6:00 p.m.
to midnight, as the final results of the election came in. This time there
truly was only one election outcome (two races) and the result was unexpected,
causing a steady shift in the odds as the votes were tallied. In Panel A, the
strong relationship between the Democratic Senate Probability and the 10-Year
Treasury Price is apparent. The strong increase in the VIX paints a picture of
the surprise outcome introducing volatility into the market as prices adjust.
In addition, the movements may reveal an expectation of a larger stimulus
package because of the prospect of a unified government being able to pass
bills more easily (consistent with results in Figure 4). Most notably, the
implied increase in the 10-Year Treasury is quite significant and aligns well
with this story, on the textbook macroeconomic assumption that higher
government spending leads to higher future interest rates. The opposite
reactions of the Russell 2000 compared to the S&P 500 and NASDAQ are also
revealing, again showing the importance of this political event to the
financial markets.

Note: In Panel A, each point represents one observation of the last traded
price (probability) from PredictIt and the percent change in price of the
10-Year Treasury since 6:00 p.m. January 5th. The fitted line in blue is a
simple linear regression shown in the upper left. In Panel B, each bar shows
the coefficient between a simple linear regression of one of the financial
markets and the Democratic Senate Probability, with a 95% confidence interval
around the estimate.

Source: PredictIt, Bloomberg, authors’ calculations.

Figure 5: Financial Markets and Control of the Senate

## Conclusion

We use a novel minutely data set to provide insight into the precise
relationship between financial markets and the changes in 2020 election result
probabilities. While we cannot identify the exact reasons for any of the
market movements throughout the course of the election, the striking
correlations show that prediction markets and financial markets are indeed
tightly interlinked. This result shows the importance of further studying the
utility of prediction markets to make economic and political forecasts.

## References

Arrow, K. J., Forsythe, R., Gorham, M., Hahn, R., Hanson, R., Ledyard, J. O.,
Levmore, S., Litan, R., Milgrom, P., Nelson, F. D., Neumann, G. R., Ottaviani,
M., Schelling, T. C., Shiller, R. J., Smith, V. L., Snowberg, E., Sunstein, C.
R., Tetlock, P. C., Tetlock, P. E., … Zitzewitz, E. (2008). The Promise of
Prediction Markets. Science, 320(5878), 877–878.
https://doi.org/10.1126/science.1157679

Bloomberg Finance LP. Bloomberg Terminals (Open, Anywhere, and Disaster
Recovery Licenses).

Hahn, R. W., & Tetlock, P. C. (2005). Using Information Markets to Improve
Public Decision Making. Harvard Journal of Law & Public Policy, 29, 213.

Hanson, R. (1996, June 9). Idea Futures—The Concept.
https://mason.gmu.edu/~rhanson/ideafutures.html

O’Leary, D. E. (2015). User participation in a corporate prediction market.
Decision Support Systems, 78, 28–38. https://doi.org/10.1016/j.dss.2015.07.004

Plott, C. R., & Chen, K.-Y. (2002, March). Information Aggregation Mechanisms:
Concept, Design and Implementation for a Sales Forecasting Problem (Report or
Paper No. 1131). California Institute of Technology.
https://resolver.caltech.edu/CaltechAUTHORS:20140317-135547085

PredictIt, Market Data via API, https://www.predictit.org/api/marketdata/all/

Reade, J. J., & Vaughan Williams, L. (2019). Polls to probabilities: Comparing
prediction markets and opinion polls. International Journal of Forecasting,
35(1), 336–350. https://doi.org/10.1016/j.ijforecast.2018.04.001

Rhode, P. W., & Strumpf, K. S. (2004). Historical Presidential Betting
Markets. Journal of Economic Perspectives, 18(2), 127–141.
https://doi.org/10.1257/0895330041371277

Silver, N. (2021, March 25). The Death Of Polling Is Greatly Exaggerated.
FiveThirtyEight. https://fivethirtyeight.com/features/the-death-of-polling-is-
greatly-exaggerated/

Snowberg, E., Wolfers, J., & Zitzewitz, E. (2013). Chapter 11—Prediction
Markets for Economic Forecasting. In G. Elliott & A. Timmermann (Eds.),
Handbook of Economic Forecasting (Vol. 2, pp. 657–687). Elsevier.
https://doi.org/10.1016/B978-0-444-53683-9.00011-6

Spears, B., LaComb, C., Interrante, J., Barnett, J., & Senturk-Dogonaksoy, D.
(2009). Examining Trader Behavior in Idea Markets: An Implementation of GE’s
Imagination Markets. Journal of Prediction Markets, 3(1), 17–39.

von Beschwitz, Bastian, Christopher G. Collins, and Deepa D. Datta (2019).
“Revisions to the Federal Reserve Dollar Indexes,” FEDS Notes. Washington:
Board of Governors of the Federal Reserve System, January 2019.,
https://doi.org/10.17016/2573-2129.48.

Wolfers, J. (2009). Prediction Markets: The Collective Knowledge of Market
Participants. Cfa Institute Conference Proceedings Quarterly, 26, 37–44.
https://doi.org/10.2469/cp.v26.n2.2


# Segment Anything Is Not Always Perfect:
An Investigation of SAM on Different Real-world Applications

Wei Ji
University of Alberta
<EMAIL_ADDRESS>Jingjing Li
University of Alberta
<EMAIL_ADDRESS>Qi Bi
Wuhan University
<EMAIL_ADDRESS>Tingwei Liu
Dalian University of Technology
<EMAIL_ADDRESS>Wenbo Li
Samsung Research America
<EMAIL_ADDRESS>Li Cheng
University of Alberta
<EMAIL_ADDRESS>
###### Abstract

Recently, Meta AI Research approaches a general, promptable Segment Anything
Model (SAM) pre-trained on an unprecedentedly large segmentation dataset
(SA-1B). Without a doubt, the emergence of SAM will yield significant benefits
for a wide array of practical image segmentation applications. In this study,
we conduct a series of intriguing investigations into the performance of SAM
across various applications, particularly in the fields of natural images,
agriculture, manufacturing, remote sensing, and healthcare. We analyze and
discuss the benefits and limitations of SAM, while also presenting an outlook
on its future development in segmentation tasks. By doing so, we aim to give a
comprehensive understanding of SAM’s practical applications. This work is
expected to provide insights that facilitate future research activities toward
generic segmentation. Source code is publicly available at here.

Figure 1: Results of Segment Anything Model (SAM [28]) on various real-world
applications, where we adopt Everything mode to obtain SAM segmentations
(right). The ground truth is masked with image for reference purpose (left).
Best view is obtained when zooming in.

##  1 Introduction

Growing interest has been observed in foundational models in recent years,
which can be attributed to their sufficient pre-training on web-scale datasets
and their superior ability to generalize to various downstream tasks. In
natural language processing (NLP), the foundation model Generative Pre-trained
Transformer (GPT [13]) from OpenAI has demonstrated great generalization
ability on many language generative tasks. Not long after, ChatGPT, empowered
by the GPT foundation model, has become a great commercial success, owing to
its real-time and reasonable language generation and user interaction. Going
back to the vision realm, the exploration of foundation models is still in its
infancy. The pioneering work of CLIP [42] effectively combines image-text
modalities, enabling zero-shot generalization to novel visual concepts.
However, its generalization ability for vision tasks remains unsatisfactory
[28] due to the scarcity of abundant training data, unlike in NLP.

More recently, Meta AI Research released a promptable Segment Anything Model
(SAM [28]). By incorporating a single user interface as prompt, SAM is capable
of segmenting any object in any image or any video without additional
training, which is often referred to as zero-shot transfer in the vision
community. As suggested by [28], SAM’s capabilities are driven by a vision
foundational model that has been trained on a massive SA-1B dataset containing
more than 11 million images and one billion masks. Meanwhile, the authors have
released an impressive online demo to showcase SAM’s capabilities at
https://segment-anything.com. SAM is designed to generate a valid segmentation
result for any prompt, where prompts can include foreground/background points,
a rough box or mask, freeform text, or any other information indicating what
to segment in an image. The latest project offers three prompt modes: click
mode, box mode, and everything mode. Click mode allows users to segment
objects with one or more clicks, either including and excluding them from the
object. Box mode allows for object segmentation by roughly drawing a bounding
box and using alternative click prompts. Everything mode automatically
identifies and masks all objects in an image.

The emergence of SAM has undoubtedly demonstrated strong generalization across
various images and objects, opening up new possibilities and avenues for
intelligent image analysis and understanding. Some practitioners from both
industry and academia have gone so far as to assert that ‘segmentation has
reached its endpoint’ and ‘the computer vision community is undergoing a
seismic shift’. Actually, a dedicated dataset for pre-training is hard to
encompass the vast array of unusual real-world scenarios and imaging
modalities, particularly for computer vision community with a variety of
conditions (_e.g_., low-light, bird’s-eye view, fog, rain), or employing
various input modalities (_e.g_., depth, infrared, event, point cloud, CT,
MRI), and with numerous real-world applications. Thus, it is of great
practical interest to investigate how well SAM can infer or generalize under
different scenarios and applications.

This leads us to carry out this study, examining SAM’s performance across a
diverse range of real-world segmentation applications. Specifically, we employ
SAM111Visual results are from https://segment-anything.com. Although we
provide effective human prompts in SAM1/2 to achieve the best possible
results, it should be noted that the optimality of these prompts cannot be
guaranteed, as excessive interaction effort is not desirable. in various
practical scenarios [5, 25, 17, 31, 51], including natural image, agriculture,
manufacturing, remote sensing and healthcare. Meanwhile, we discuss SAM’s
benefits and limitations in practice. Based on these studies, we have made the
following observations:

  * •

Excellent generalization on common scenes. Experiments on various images
validate SAM’s effectiveness across different prompt modes, demonstrating its
ability to generalize well to typical natural image scenarios, especially when
target regions distinct prominently from their surroundings. This emphasizes
the superiority of the promptable SAM’s model design and the strength of its
massive and diverse training data source.

  * •

Require strong prior knowledge. During the usage of SAM, we observe that for
complex scenes, _e.g_., crop segmentation and fundus image segmentation, more
manual prompts with prior knowledge are required, which could potentially
result in a suboptimal user experience. Additionally, we notice that SAM tends
to favor selecting the foreground mask. When applying the SAM model to shadow
detection task, even with a large number of click prompts, its performance
remains poor. This may be due to the strong foreground bias in its pre-
training dataset, which hinders its ability to handle certain scenarios
effectively.

  * •

Less effective in low-contrast applications. Segmenting objects with similar
surrounding elements is considered challenging scenarios, especially when
dealing with transparent or camouflaged objects that are “seamlessly” embedded
in their surroundings. Experiments reveal that there is considerable room for
exploring and enhancing SAM’s robustness in complex scenes with low-contrast
elements.

  * •

Limited understanding of professional data. We apply SAM to real-world medical
and industrial scenarios and discover that it produces unsatisfactory results
for professional data, particularly when using box mode and everything mode.
This reveals SAM’s limitations in understanding these practical scenarios.
Moreover, even with click mode, both the user and the model are required to
possess certain domain-specific knowledge and understanding of the task at
hand.

  * •

Smaller and irregular objects can pose challenges for SAM. Remote sensing and
agriculture present additional challenges, such as irregular buildings and
small-sized streets captured from the aerial imaging sensors. These
complexities make it challenging for SAM to produce complete segmentation. How
to design effective strategies for SAM in such cases is still an open issue.

In this study, we examine SAM’s performance in various scenarios, and provide
some observations and insights toward promoting the development of
foundational models in vision realm. While we have tested many tasks, not all
downstream applications have been covered. A multitude of fascinating
segmentation tasks and scenarios are encouraged to be explored in future
research.

Figure 2: Application on salient object segmentation, where SAM1/2/3 mean
using Click, Box, and Everything modes respectively. Figure 3: Application on
camouflaged object segmentation, where SAM1/2/3 mean using Click, Box, and
Everything modes respectively. Figure 4: Application on transparent object
segmentation, where SAM1/2/3 mean using Click, Box, and Everything modes
respectively. Figure 5: Application on shadow detection, where SAM1/2/3 mean
using Click, Box, and Everything modes respectively. The * indicates the SAM
results within a box prompt. Figure 6: Application on crop segmentation, where
SAM1/2/3 mean using Click, Box, and Everything modes respectively. The *
indicates the SAM results within a box prompt. Figure 7: Application on pest
and leaf disease monitoring, where SAM1/2/3 mean using Click, Box, and
Everything modes respectively. The * indicates the SAM results within a box
prompt. Figure 8: Application on anomaly detection, where SAM1/2/3 mean using
Click, Box, and Everything modes respectively. Figure 9: Application on
surface defect detection, where SAM1/2/3 mean using Click, Box, and Everything
modes respectively. Here SAM3 does not generate any results on these cases.

##  2 Qualitative Investigation

In this section, we present visual results of SAM model on diverse
segmentation tasks, and analyze its advantages and limitations from a
qualitative perspective.

###  2.1 Natural Image Scenes

####  2.1.1 Experiments on Different Subtasks

Salient Object Segmentation [5] aims at extracting the most attention-grabbing
objects from still images, differing from semantic segmentation as a class-
agnostic task. Here we apply SAM to the popular saliency dataset ReDWeb-S
[32]. As shown in Fig. 2 (a), SAM demonstrates a strong ability to identify
remarkable objects with precise location. However, when encountering rich-
detailed targets, as shown in Fig. 2 (b), SAM is less appealing in capturing
fine-grained details.

Camouflaged Object Segmentation [10, 23] involves identifying objects that are
‘seamlessly’ embedded in their surroundings, making it a highly challenging
task compared to traditional object segmentation. To verify the effectiveness
of SAM, we adopt the COD10K [10] test set. As seen in Fig. 3, SAM often fails
to detect entire objects in challenging scenarios, such as those with similar
foreground and background, or multiple objects in cluttered environments. Even
with increased human interactions (i.e., SAM1), many false positives still
appear in SAM’s results.

Transparent Object Segmentation [50] is the process of segmenting transparent
objects, such as glass, plastic, window, bottle from an image. These objects
often exhibit complex shapes and can refract or reflect light in unpredictable
ways. Here we apply SAM to the Trans10K-v2 dataset [50] to assess its
performance. As shown in Fig. 4, SAM shows a strong capability to detect the
location of target objects, but there is still room for improvement in
capturing finer details.

Shadow Detection [20] is a fundamental problem in which analyzing shadows in a
scene can help estimate light conditions and scene geometry. This task is
particularly challenging for generic segmentation methods, as the shadows of
interest are often difficult to be distinguished from surrounding backgrounds.
As shown in Fig. 5, it is not surprising that SAM fails to detect shadows in
the datasets [20, 18].

####  2.1.2 Section Recap

Through assessing on several popular natural image segmentation tasks, we
observe that SAM has powerful ability at finding the location of objects,
which is particularly impressive for everyday applications such as object
detection and online meeting background subtraction. However, in situations
involving noisy backgrounds, similar foreground and background, or scenes
requiring more detailed segmentation, SAM leaves a significant opportunity for
further exploration and improvement.

###  2.2 Agriculture

####  2.2.1 Experiments on Different Subtasks

Crop Segmentation [31], a crucial application of image segmentation, is
fundamental to agricultural information automation. It serves various
purposes, such as predicting crop growth stages, estimating density,
identifying cover crops, and monitoring crop biomass. In this study, SAM is
tested on dataset [15], with results displayed in Fig. 6. In Fig. 6 (a), SAM
achieves relatively satisfactory segmentations compared to results in Fig. 6
(b). One possible reason for this discrepancy is that, in general segmentation
tasks, crops and land are often treated as background, leading to a scarcity
of positive examples that could support SAM in achieving better generalization
in agricultural scenes.

Pest and Leaf Disease Monitoring [43] is a vital technology in agriculture,
contributing to increased food production and quality while reducing plant
diseases. Surprisingly, when SAM is applied to the agricultural domain [8], it
achieves excellent performance, exhibiting robust generalization capabilities
as demonstrated in Fig. 7 (b). However, in more challenging scenes like these
in Fig. 7 (a), detecting the entire pest body remains difficult for SAM.

####  2.2.2 Section Recap

It is surprising that SAM exhibits appealing performance in the agriculture
field, albeit not t hat perfect. We believe that in the future, with
sufficient prior knowledge of agriculture, SAM will be able to detect
challenging objects more effectively.

Figure 10: Application on building extraction, where SAM1/2/3 mean using
Click, Box, and Everything modes respectively. The * indicates the SAM results
within a box prompt. Figure 11: Application on road extraction, where SAM1/2/3
mean using Click, Box, and Everything modes respectively.

###  2.3 Manufacture

####  2.3.1 Experiments on Different Subtasks

Anomaly Detection [3] is used to detect anomalies from normal samples, which
contributes to improved quality control in the industry and increased economic
benefits. We test SAM in MVTec AD dataset [3] as shown in Fig. 8. The results
show that it has powerful recognition ability.

Surface Defect Detection [17] aims to identify substandard quality products,
such as wood, textiles, and mineral material in industrial manufacturing, to
avoid detrimental economic consequences. We further apply SAM to this
practical industrial scene. Observations in Fig. 9 indicate that, with
reasonable human prompts, SAM1 presents acceptable performance, while SAM3
under the everything mode fails to provide a meaningful response. These
results suggest that incorporating human experts’ prior knowledge is crucial
for successful AI model applications in real-world industrial settings.

####  2.3.2 Section Recap

Experiments show that SAM performs acceptably in industrial scenarios. In
particular, when examining the surface defect detection task, it becomes
evident that for practical deployment, a significant amount of expert prior
knowledge is indispensable.

###  2.4 Remote Sensing

####  2.4.1 Experiments on Different Subtasks

Building Extraction [16] from optical remote sensing imagery is one of the
fundamental tasks in remote sensing, which plays a key role in many
applications, such as urban planning and construction, natural crisis and
disaster management [4]. As seen in Fig. 10 (a), SAM is proficient at
segmenting regularly shaped objects; however, when dealing with smaller or
indiscernible targets as in Fig. 10 (b), there is still room for improvement.

Road Extraction [51] promotes the development of transportation systems, such
as automatic road navigation, unmanned vehicles, and urban planning, which is
important in both industry and daily life. We further assess the robustness of
SAM using the dataset from [37]. As presented in Fig. 11, the objects in the
scene exhibit diverse shapes, sizes, and textures, which present challenges
for achieving accurate segmentation with SAM.

####  2.4.2 Section Recap

Extracting targets from remote sensing images is challenging due to their high
variability in shape and size. In our evaluation, we test SAM’s performance in
extracting essential buildings and roads, and found that it has good
scalability for regular objects. However, for segmentation of smaller or less
distinguishable objects, it may be necessary to consider task-specific
properties and adapt the method accordingly.

###  2.5 Healthcare

####  2.5.1 Experiments on Different Subtasks

Joint Optical Disc and Cup Segmentation [14] works towards identifying and
separating the optic disc (OD) and the optic cup (OC) in retinal fundus
images. The OD is the area where ganglion cell axons exit the eye to form the
optic nerve, while the OC is usually restricted to the region inside OD. By
jointly segmenting the OD and OC, it can directly compute the cup-to-disc
ratio (CDR), an important clinical parameter for the screening of glaucoma. A
higher CDR indicates a larger OC and is often associated with a higher risk of
glaucoma. In Fig. 14, we evaluate SAM using the popular RIGA benchmark [1].
The visual results highlight severe shortcomings of the SAM model in medical
scenarios. This is understandable, as gold-standard medical image annotations
generally require senior ophthalmologists with extensive clinical experience
to perform the labeling.

Polyp Segmentation [12, 38] is crucial for the early diagnosis and treatment
of colorectal cancer. However, polyp segmentation remains a challenging task
due to the substantial variation in polyp shape and size. Here we present the
results of SAM applied to a poly segmentation benchmark [2], with results
displayed in Fig. 14. It is evident that, with adequate human prompts, SAM1
achieves impressive performance. Nevertheless, for box and automated
segmentation modes, SAM2 and SAM3 have difficulty to precisely identify lesion
regions. This observation further showcases the ongoing need for significant
attention and investigation into the development of dedicated, automated, and
effective polyp segmentation algorithms.

Skin Lesion Segmentation [7] is critical to human life, as the incidence of
skin cancer has increased considerably and is seriously threatening human
health. This task is particularly challenging due to many factors such as
indistinct boundaries, variable contrast, and color differences. Fig. 14
presents the results of SAM on [36]. Similar to polyp segmentation, SAM
delivers satisfactory outcomes when provided with sufficient human prompts.
However, there remains considerable room for improvement in SAM2 and SAM3.

####  2.5.2 Section Recap

These results indicate that SAM holds great significance in the medical field.
However, it is important to note that SAM1, despite delivering appealing
results, often requires substantial human prior knowledge. This poses a
challenge in directly attaining satisfactory results for certain tasks, such
as optic cup and disc segmentation, where effective prompts usually require a
high level of expertise. On the other hand, in box and automatic modes, SAM’s
performance falls short. This emphasizes the ongoing demand for dedicated SAM
models explicitly tailored for tackling medical scenarios.

##  3 Quantitative Investigation

In this section, we present quantitative results of SAM model on diverse
segmentation tasks, and compare it with dedicated state-of-the-art models.

Figure 12: Application on joint optical disc (OD) and optical cup (OC)
segmentation, where SAM1/2/3 mean using Click, Box, and Everything modes
respectively. Here SAM3 does not generate any results on these cases.

Figure 13: Application on polyp segmentation, where SAM1/2/3 mean using Click,
Box, and Everything modes respectively. The * indicates the SAM results within
a box prompt.

Figure 14: Application on skin lesion segmentation, where SAM1/2/3 mean using
Click, Box, and Everything modes respectively. Here SAM3 does not generate any
results on the case (b).

###  3.1 Datasets and Evaluation Metric

We thoroughly evaluate SAM on eight representative benchmarks, as follows:
DUTS [47] is currently the largest salient object segmentation dataset,
consisting of 10,553 training images and 5,019 test images.
COME15K-Diff [52] is a challenging salient object segmentation dataset,
containing many low-contrast complex scenes. Its test set has 3,000
challenging samples.
VT1000 [45] is a widely-used benchmark for RGB-Thermal salient object
segmentation. It covers diverse low-light images, and can be used to assess
the robustness of SAM in night scenes.
DIS-TE4 [41] provides 500 highly accurate pixel-level masks for images with
complex object shapes. This can evaluate SAM’s ability in perceiving boundary
details.
COD10K [10] is a substantial dataset specifically designed for camouflaged
object segmentation, featuring 10,000 images characterized by similar
foreground objects and surrounding environments.
SBU [46] provides carefully annotated shadow masks for 700 images, which is
widely used in shadow detection task.
CDS2K [11] is a comprehensive concealed defect segmentation dataset, compiled
from a variety of established industrial defect repositories. It comprises
2,492 samples, consisting of 1,330 positive and 1,162 negative instances.
ColonDB [44] is used to verify the generalization capability of SAM in medical
field, especially in the challenging polyp lesion segmentation. It consists of
380 images in total.

For quantitative evaluation, we adopt the widely-used mean absolute error
($\mathcal{M}$). A lower $\mathcal{M}$ score indicates better model
performance.

(a) Results on DUTS [47].

Model | Backbone | $\mathcal{M}$
---|---|---
VST21 [33] | T2T-ViTt-14 | 0.037
ICONet22 [59] | ResNet50 | 0.037
EDNet22 [48] | ResNet50 | 0.035
SAM23 | ViT-B | 0.121
$\Delta$diff |  $\downarrow$8.6%
ViT-L | 0.052
$\Delta$diff |  $\downarrow$1.7%
ViT-H | 0.046
$\Delta$diff |  $\downarrow$1.1%

(b) Results on COME15K-Diff [52].

Model | Backbone | $\mathcal{M}$
---|---|---
VST21 [33] | T2T-ViTt-14 | 0.054
ICONet22 [59] | ResNet50 | 0.055
EDNet22 [48] | ResNet50 | 0.050
SAM23 | ViT-B | 0.129
$\Delta$diff |  $\downarrow$7.9%
ViT-L | 0.061
$\Delta$diff |  $\downarrow$1.1%
ViT-H | 0.058
$\Delta$diff |  $\downarrow$0.8%

(c) Results on VT1000 [45].

Model | Backbone | $\mathcal{M}$
---|---|---
VST21 [33] | T2T-ViTt-14 | 0.024
ICONet22 [59] | ResNet50 | 0.032
EDNet22 [48] | ResNet50 | 0.028
SAM23 | ViT-B | 0.060
$\Delta$diff |  $\downarrow$3.6%
ViT-L | 0.035
$\Delta$diff |  $\downarrow$1.1%
ViT-H | 0.032
$\Delta$diff |  $\downarrow$0.8%

(d) Results on DIS-TE4 [41].

Model | Backbone | $\mathcal{M}$
---|---|---
Gate20 [55] | ResNet50 | 0.109
PFNet21 [34] | ResNet50 | 0.107
IS-Net22 [41] | U2Net | 0.072
SAM23 | ViT-B | 0.179
$\Delta$diff |  $\downarrow$10.7%
ViT-L | 0.166
$\Delta$diff |  $\downarrow$9.4%
ViT-H | 0.166
$\Delta$diff |  $\downarrow$9.4%

(e) Results on COD10K [10].

Model | Backbone | $\mathcal{M}$
---|---|---
SegMaR22 [26] | ResNet50 | 0.034
PFNet+23 [35] | ResNet50 | 0.037
ZoomNet22 [39] | ResNet50 | 0.029
SAM23 | ViT-B | 0.108
$\Delta$diff |  $\downarrow$7.9%
ViT-L | 0.065
$\Delta$diff |  $\downarrow$3.6%
ViT-H | 0.054
$\Delta$diff |  $\downarrow$2.5%

(f) Results on SBU [46].

Model | Backbone | $\mathcal{M}$
---|---|---
DSC18 [21] | VGG-16 | 0.032
DSDNet19 [56] | ResNext | 0.036
MTMT20 [6] | ResNext | 0.029
SAM23 | ViT-B | 0.203
$\Delta$diff |  $\downarrow$17.4%
ViT-L | 0.187
$\Delta$diff |  $\downarrow$15.8%
ViT-H | 0.183
$\Delta$diff |  $\downarrow$15.4%

(g) Results on CDS2K [11].

Model | Backbone | $\mathcal{M}$
---|---|---
SINetV222 [9] | Res2Net50 | 0.102
HitNet23 [19] | PVTv2-B2 | 0.118
DGNet23 [22] | EffiNet-B4 | 0.089
SAM23 | ViT-B | 0.372
$\Delta$diff |  $\downarrow$28.3%
ViT-L | 0.281
$\Delta$diff |  $\downarrow$19.2%
ViT-H | 0.265
$\Delta$diff |  $\downarrow$17.6%

(h) Results on ColonDB [44].

Model | Backbone | $\mathcal{M}$
---|---|---
FAPNet22 [57] | Res2Net50 | 0.038
CFA-Net23 [58] | Res2Net50 | 0.039
HSNet22 [54] | PVTv2 | 0.032
SAM23 | ViT-B | 0.111
$\Delta$diff |  $\downarrow$7.9%
ViT-L | 0.054
$\Delta$diff |  $\downarrow$2.2%
ViT-H | 0.056
$\Delta$diff |  $\downarrow$2.4%

Table 1: Quantitative results of SAM on applications of (a) salient object
segmentation in common scenes, (b) salient object segmentation in low-contrast
scenes, (c) salient object segmentation in low-light scenes, (d) salient
object segmentation with highly-accurate details (i.e., dichotomous image
segmentation), (e) camouflaged object segmentation, (f) shadow detection, (g)
concealed industrial defect detection, and (h) medical polyp lesion
segmentation. The $\mathcal{M}$ represents mean absolute error (the lower the
better). $\Delta$ shows the performance gaps between SAMs and the best
performing state-of-the-art models.

###  3.2 Numerical Results

We report the quantitative results in Table 1. To obtain SAM’s results,
following [23], we first implement
SAM222https://github.com/facebookresearch/segment-anything to infer $N$
potential object masks within an input image. Given these masks, we choose the
most suitable mask based on its alignment with the ground truth. Specifically,
for $N$ binary predictions $\mathrm{\\{P_{n}\\}_{n=1}^{N}}$, and the ground-
truth $\mathrm{G}$ for an input image, we compute intersection over union
(IoU) scores for each prediction and ground-truth pair, generating a set of
candidate scores ${\\{IoU_{n}\\}}_{n=1}^{N}$. The mask with the highest IoU is
ultimately selected. The numerical findings presented in Table 1 demonstrate a
considerable performance gap between SAM and the top-performing model across
eight benchmarks, especially for challenging scenarios and applications. For
instance, in the industrial application of Table 1 (g), SAM with ViT-H attains
a 0.265 MAE score, which is 17.6% worse than the state-of-the-art DGNet [22].
These results suggest that there is significant room for further research and
enhancement of SAM’s performance.

##  4 Discussion and Outlook

Here we discuss several potential directions for future research, as follows.
i) Application-oriented SAM & dedicated large-scale dataset: Although SAM
achieves superior performance in natural image segmentation, it still delivers
unappealing results in other applications, _e.g_., healthcare [25],
manufacture [17] and remote sensing [51]. The dedicated large-scale dataset
and foundational model can be further explored to improve scalability in
specific application scenarios.
ii) Additional prompt modes: Current SAM primarily operates with three modes,
click, box and everything. Some interesting prompt [49] can be further
advanced by exploring, e.g., voice and gesture.
iii) Pretraining strategy: SAM’s impressive success can be partially
attributed to the massive segmentation dataset, SA-1B, which is better suited
for pixel-wise prediction tasks compared to ImageNet. As such, SAM could serve
as a novel pretraining model, with SA-1B functioning as a comprehensive
pretraining dataset for industrial/medical applications. For instance,
employing effective metric learning [27] on SA-1B could bolster the
representation capabilities of features, allowing them to adapt to diverse
tasks.
iv) Multi-modal SAM: Current SAM is less powerful for challenging scenarios
such as low-contrast scenes and low-light nighttime cases. It is thus
necessary to introduce complementary sources (_e.g_., depth [24, 40] and
thermal [45]), which can effectively enhance the robustness of models.
v) Video SAM: Compared to static images, it is natural to consider dynamic
video inputs [53, 30]. Therefore, how to develop a dynamic video SAM given
initial prompt in first frame is an open issue.
vi) Semi-supervised application: The SAM has demonstrated exceptional
performance and versatility, making it a promising tool for various related
tasks. Researchers can further leverage SAM to empower semi-supervised
segmentation tasks [29], using the model in combination with suitable point
prompts, bounding box prompts, or scribble prompts to generate pseudo-labels.

##  5 Conclusion

In this paper, we conducted a preliminary investigation of SAM’s performance
across various applications, including natural image, agriculture,
manufacturing, remote sensing, and healthcare. We analyzed SAM’s benefits and
limitations, identified potential challenges, and suggested future directions.
These observations can guide the development of robust SAM algorithms and
benchmarks. Moving forward, we plan to study SAM more in-depth, and design
effective learning strategies to adapt it to a broader range of scenarios.
Acknowledgements. Thank Meta AI Research for the valuable and impressive work
on providing open-source SAM model and SA-1B dataset. This study is partially
supported by the Mitacs, CFI-JELF and NSERC Discovery grants. The views and
conclusions contained in this paper are those of the authors and should not be
interpreted as representing any funding agency.

## References

  * [1] Ahmed Almazroa, Sami Alodhayb, Essameldin Osman, Eslam Ramadan, Mohammed Hummadi, Mohammed Dlaim, Muhannad Alkatee, Kaamran Raahemifar, and Vasudevan Lakshminarayanan.  Agreement among ophthalmologists in marking the optic disc and optic cup in fundus images.  International ophthalmology, 37:701–717, 2017.
  * [2] Nguyen S An, Phan N Lan, Dao V Hang, Dao V Long, Tran Q Trung, Nguyen T Thuy, and Dinh V Sang.  Blazeneo: Blazing fast polyp segmentation and neoplasm detection.  IEEE Access, 10:43669–43684, 2022.
  * [3] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger.  Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection.  In CVPR, pages 9592–9600, 2019.
  * [4] Qi Bi, Kun Qin, Han Zhang, and Gui-Song Xia.  Local semantic enhanced convnet for aerial scene recognition.  IEEE Transactions on Image Processing, 30:6498–6511, 2021.
  * [5] Ali Borji, Ming-Ming Cheng, Qibin Hou, Huaizu Jiang, and Jia Li.  Salient object detection: A survey.  Computational Visual Media, 5:117–150, 2019.
  * [6] Zhihao Chen, Lei Zhu, Liang Wan, Song Wang, Wei Feng, and Pheng-Ann Heng.  A multi-task mean teacher for semi-supervised shadow detection.  In CVPR, pages 5611–5620, 2020.
  * [7] Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic).  In ISBI, pages 168–172, 2018.
  * [8] Dataset Contributors.  Leaf disease segmentation dataset.
  * [9] Deng-Ping Fan, Ge-Peng Ji, Ming-Ming Cheng, and Ling Shao.  Concealed object detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6024–6042, 2021.
  * [10] Deng-Ping Fan, Ge-Peng Ji, Guolei Sun, Ming-Ming Cheng, Jianbing Shen, and Ling Shao.  Camouflaged object detection.  In CVPR, pages 2777–2787, 2020.
  * [11] Deng-Ping Fan, Ge-Peng Ji, Peng Xu, Ming-Ming Cheng, Christos Sakaridis, and Luc Van Gool.  Advances in deep concealed scene understanding.  arXiv preprint arXiv:2304.11234, 2023.
  * [12] Deng-Ping Fan, Ge-Peng Ji, Tao Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, and Ling Shao.  Pranet: Parallel reverse attention network for polyp segmentation.  In MICCAI, pages 263–273. Springer, 2020.
  * [13] Luciano Floridi and Massimo Chiriatti.  Gpt-3: Its nature, scope, limits, and consequences.  Minds and Machines, 30:681–694, 2020.
  * [14] Huazhu Fu, Jun Cheng, Yanwu Xu, Damon Wing Kee Wong, Jiang Liu, and Xiaochun Cao.  Joint optic disc and cup segmentation based on multi-label deep network and polar transformation.  IEEE Transactions on Medical Imaging, 37(7):1597–1605, 2018.
  * [15] Wei Guo, Uday K Rage, and Seishi Ninomiya.  Illumination invariant segmentation of vegetation for time series wheat images based on decision tree model.  Computers and Electronics in Agriculture, 96:58–66, 2013.
  * [16] Sheng He and Wanshou Jiang.  Boundary-assisted learning for building extraction from optical remote sensing imagery.  Remote Sensing, 13(4):760, 2021.
  * [17] Ting He, Ying Liu, Chengyi Xu, Xiaolin Zhou, Zhongkang Hu, and Jianan Fan.  A fully convolutional neural network for wood defect location and identification.  IEEE Access, 7:123453–123462, 2019.
  * [18] Le Hou, Tomas F Yago Vicente, Minh Hoai, and Dimitris Samaras.  Large scale shadow annotation and detection using lazy annotation and stacked cnns.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(4):1337–1351, 2019.
  * [19] Xiaobin Hu, Deng-Ping Fan, Xuebin Qin, Hang Dai, Wenqi Ren, Ying Tai, Chengjie Wang, and Ling Shao.  High-resolution iterative feedback network for camouflaged object detection.  AAAI, 2023.
  * [20] Xiaowei Hu, Tianyu Wang, Chi-Wing Fu, Yitong Jiang, Qiong Wang, and Pheng-Ann Heng.  Revisiting shadow detection: A new benchmark dataset for complex world.  IEEE Transactions on Image Processing, 30:1925–1934, 2021.
  * [21] Xiaowei Hu, Lei Zhu, Chi-Wing Fu, Jing Qin, and Pheng-Ann Heng.  Direction-aware spatial context features for shadow detection.  In CVPR, pages 7454–7462, 2018.
  * [22] Ge-Peng Ji, Deng-Ping Fan, Yu-Cheng Chou, Dengxin Dai, Alexander Liniger, and Luc Van Gool.  Deep gradient learning for efficient camouflaged object detection.  Machine Intelligence Research, 20(1):92–108, 2023.
  * [23] Ge-Peng Ji, Deng-Ping Fan, Peng Xu, Ming-Ming Cheng, Bowen Zhou, and Luc Van Gool.  Sam struggles in concealed scenes–empirical study on” segment anything”.  arXiv preprint arXiv:2304.06022, 2023.
  * [24] Wei Ji, Jingjing Li, Qi Bi, Chuan Guo, Jie Liu, and Li Cheng.  Promoting saliency from depth: Deep unsupervised rgb-d saliency detection.  ICLR, 2022.
  * [25] Wei Ji, Shuang Yu, Junde Wu, Kai Ma, Cheng Bian, Qi Bi, Jingjing Li, Hanruo Liu, Li Cheng, and Yefeng Zheng.  Learning calibrated medical image segmentation via multi-rater agreement modeling.  In CVPR, pages 12341–12351, 2021.
  * [26] Qi Jia, Shuilian Yao, Yu Liu, Xin Fan, Risheng Liu, and Zhongxuan Luo.  Segment, magnify and reiterate: Detecting camouflaged objects the hard way.  In CVPR, pages 4713–4722, 2022.
  * [27] Mahmut Kaya and Hasan Şakir Bilge.  Deep metric learning: A survey.  Symmetry, 11(9):1066, 2019.
  * [28] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  arXiv preprint arXiv:2304.02643, 2023.
  * [29] Jingjing Li, Wei Ji, Qi Bi, Cheng Yan, Miao Zhang, Yongri Piao, Huchuan Lu, et al.  Joint semantic mining for weakly supervised rgb-d salient object detection.  NeurIPS, 34:11945–11959, 2021.
  * [30] Jingjing Li, Tianyu Yang, Wei Ji, Jue Wang, and Li Cheng.  Exploring denoised cross-video contrast for weakly-supervised temporal action localization.  In CVPR, pages 19914–19924, 2022.
  * [31] Yanan Li, Ziyun Huang, Zhiguo Cao, Hao Lu, Haihui Wang, and Shuiping Zhang.  Performance evaluation of crop segmentation algorithms.  IEEE Access, 8:36210–36225, 2020.
  * [32] Nian Liu, Ni Zhang, Ling Shao, and Junwei Han.  Learning selective mutual attention and contrast for rgb-d saliency detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):9026–9042, 2021.
  * [33] Nian Liu, Ni Zhang, Kaiyuan Wan, Ling Shao, and Junwei Han.  Visual saliency transformer.  In ICCV, pages 4722–4732, 2021.
  * [34] Haiyang Mei, Ge-Peng Ji, Ziqi Wei, Xin Yang, Xiaopeng Wei, and Deng-Ping Fan.  Camouflaged object segmentation with distraction mining.  In CVPR, pages 8772–8781, 2021.
  * [35] Haiyang Mei, Xin Yang, Yunduo Zhou, Ge-Peng Ji, Xiaopeng Wei, and Deng-Ping. Fan.  Distraction-aware camouflaged object segmentation.  SCIENTIA SINICA Informationis (SSI), 2023.
  * [36] Teresa Mendonça, Pedro M Ferreira, Jorge S Marques, André RS Marcal, and Jorge Rozeira.  Ph 2-a dermoscopic image database for research and benchmarking.  In 2013 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC), pages 5437–5440, 2013.
  * [37] Volodymyr Mnih.  Machine learning for aerial image labeling.  University of Toronto (Canada), 2013.
  * [38] Junwen Pan, Qi Bi, Yanzhan Yang, Pengfei Zhu, and Cheng Bian.  Label-efficient hybrid-supervised learning for medical image segmentation.  In AAAI, pages 2026–2034, 2022.
  * [39] Youwei Pang, Xiaoqi Zhao, Tian-Zhu Xiang, Lihe Zhang, and Huchuan Lu.  Zoom in and out: A mixed-scale triplet network for camouflaged object detection.  In CVPR, pages 2160–2170, 2022.
  * [40] Yongri Piao, Wei Ji, Jingjing Li, Miao Zhang, and Huchuan Lu.  Depth-induced multi-scale recurrent attention network for saliency detection.  In ICCV, pages 7254–7263, 2019.
  * [41] Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Luc Van Gool.  Highly accurate dichotomous image segmentation.  In ECCV, pages 38–56, 2022.
  * [42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, pages 8748–8763, 2021.
  * [43] Apurva Sriwastwa, Shikha Prakash, Swati Swarit, Khushboo Kumari, Sitanshu Sekhar Sahu, et al.  Detection of pests using color based image segmentation.  In International Conference on Inventive Communication and Computational Technologies (ICICCT), pages 1393–1396, 2018.
  * [44] Nima Tajbakhsh, Suryakanth R Gurudu, and Jianming Liang.  Automated polyp detection in colonoscopy videos using shape and context information.  IEEE Transactions on Medical Imaging, 35:630–644, 2015.
  * [45] Zhengzheng Tu, Tian Xia, Chenglong Li, Xiaoxiao Wang, Yan Ma, and Jin Tang.  Rgb-t image saliency detection via collaborative graph learning.  IEEE Transactions on Multimedia, 22:160–173, 2019.
  * [46] Tomás F Yago Vicente, Le Hou, Chen-Ping Yu, Minh Hoai, and Dimitris Samaras.  Large-scale training of shadow detectors with noisily-annotated shadow examples.  In ECCV, pages 816–832, 2016.
  * [47] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan.  Learning to detect salient objects with image-level supervision.  In CVPR, pages 136–145, 2017.
  * [48] Yu-Huan Wu, Yun Liu, Le Zhang, Ming-Ming Cheng, and Bo Ren.  Edn: Salient object detection via extremely-downsampled network.  IEEE Transactions on Image Processing, 31:3125–3136, 2022.
  * [49] Xiaojia Xiang, Qin Tan, Han Zhou, Dengqing Tang, and Jun Lai.  Multimodal fusion of voice and gesture data for uav control.  Drones, 6(8):201, 2022.
  * [50] Enze Xie, Wenjia Wang, Wenhai Wang, Peize Sun, Hang Xu, Ding Liang, and Ping Luo.  Segmenting transparent objects in the wild with transformer.  In IJCAI, 2021.
  * [51] Yongyang Xu, Zhong Xie, Yaxing Feng, and Zhanlong Chen.  Road extraction from high-resolution remote sensing imagery using deep learning.  Remote Sensing, 10(9):1461, 2018.
  * [52] Jing Zhang, Deng-Ping Fan, Yuchao Dai, Xin Yu, Yiran Zhong, Nick Barnes, and Ling Shao.  Rgb-d saliency detection via cascaded mutual information minimization.  In ICCV, pages 4338–4347, 2021.
  * [53] Miao Zhang, Jie Liu, Yifei Wang, Yongri Piao, Shunyu Yao, Wei Ji, Jingjing Li, Huchuan Lu, and Zhongxuan Luo.  Dynamic context-sensitive filtering network for video salient object detection.  In ICCV, pages 1553–1563, 2021.
  * [54] Wenchao Zhang, Chong Fu, Yu Zheng, Fangyuan Zhang, Yanli Zhao, and Chiu-Wing Sham.  Hsnet: A hybrid semantic network for polyp segmentation.  Computers in Biology and Medicine, 150:106173, 2022.
  * [55] Xiaoqi Zhao, Youwei Pang, Lihe Zhang, Huchuan Lu, and Lei Zhang.  Suppress and balance: A simple gated network for salient object detection.  In ECCV, pages 35–51, 2020.
  * [56] Quanlong Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau.  Distraction-aware shadow detection.  In CVPR, pages 5167–5176, 2019.
  * [57] Tao Zhou, Yi Zhou, Chen Gong, Jian Yang, and Yu Zhang.  Feature aggregation and propagation network for camouflaged object detection.  IEEE Transactions on Image Processing, 31:7036–7047, 2022.
  * [58] Tao Zhou, Yi Zhou, Kelei He, Chen Gong, Jian Yang, Huazhu Fu, and Dinggang Shen.  Cross-level feature aggregation network for polyp segmentation.  Pattern Recognition, 140:109555, 2023.
  * [59] Mingchen Zhuge, Deng-Ping Fan, Nian Liu, Dingwen Zhang, Dong Xu, and Ling Shao.  Salient object detection via integrity learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.