constant $c>0$ such that $\sup_{n,m\in\mathbb{N}}\left\lVert
A_{m}B_{n}\right\rVert_{V}^{p}\leq c$ a.s. and a compact set
$K_{\epsilon}\subseteq V$ satisfying $P(B_{n}\notin
K_{\epsilon})<\frac{\epsilon}{c}$ for every $n\in\mathbb{N}$. Since continuous
mapping converging to zero converges uniformly on compacts there exists
$m_{1}\in\mathbbm{N}$ such that for all $m\geq m_{1}$ we have

$\sup_{n\in\mathbb{N}}\int_{\\{B_{n}\in K_{\epsilon}\\}}\left\lVert
A_{m}B_{n}(\omega)\right\rVert_{V}^{p}\,P({\rm d}\omega)<\epsilon.$

It follows for all $n\in\mathbb{N}$ and $m\geq m_{1}$ that

$\displaystyle\mathbb{E}\,\left[\left\lVert
A_{m}B_{n}\right\rVert_{V}^{p}\right]\leq\int_{\\{B_{n}\in
K_{\epsilon}\\}}\left\lVert A_{m}B_{n}(\omega)\right\rVert_{V}^{p}\,P({\rm
d}\omega)+\int_{\\{B_{n}\notin K_{\epsilon}\\}}\left\lVert
A_{m}B_{n}(\omega)\right\rVert_{V}^{p}\,P({\rm d}\omega)\leq 2\epsilon,$

which completes the proof. ∎

## References

  * [1] S. Albeverio, L. Gawarecki, V. Mandrekar, B. Rüdiger, and B. Sarkar.  Itô formula for mild solutions of SPDEs with Gaussian and non-Gaussian noise and applications to stability properties.  Random Oper. Stoch. Equ., 25(2):79–105, 2017.
  * [2] H. Amman and J. Escher.  Analysis II.  Birkhäuer Verlag, Basel-Boston-Berlin, 2006.
  * [3] D. Applebaum and M. Riedle.  Cylindrical Lévy processes in Banach spaces.  Proc. Lond. Math. Soc. (3), 101(3):697–726, 2010.
  * [4] Jöran Bergh and Jörgen Löfström.  Interpolation spaces. An introduction, volume 223.  Springer Science & Business Media, 2012.
  * [5] G. Bodó and M. Riedle.  Stochastic integration with respect to canonical $\alpha$-stable cylindrical Lévy processes.  Electronic Journal of Probability, 27:1–23, 2022.
  * [6] Z. Brzeźniak and J. Zabczyk.  Regularity of Ornstein-Uhlenbeck processes driven by a Lévy white noise.  Potential Anal., 32(2):153–188, 2010.
  * [7] C. Chong.  Stochastic PDEs with heavy-tailed noise.  Stochastic Process. Appl., 127(7):2262–2280, 2017.
  * [8] C. Chong, R. C. Dalang, and T. Humeau.  Path properties of the solution to the stochastic heat equation with Lévy noise.  Stochastics and Partial Differential Equations: Analysis and Computations, Aug 2018.
  * [9] G. Da Prato, A. Jentzen, and M. Röckner.  A mild Itô formula for SPDEs.  Transactions of the American Mathematical Society, 372(6):3755–3807, 2019.
  * [10] G. Da Prato and J. Zabczyk.  Stochastic equations in infinite dimensions.  Encyclopedia of Mathematics and its Applications. Cambridge University Press, Cambridge, 1992.
  * [11] C. Dellacherie and P. A. Meyer.  Probabilities and potential.  North-Holland Publishing Co., Amsterdam, 1978.
  * [12] S. Diaz and F. Mayoral.  On compactness in spaces of Bochner integrable functions.  Acta Math. Hungar., 83(3):231–239, 1999.
  * [13] K. J. Engel and R. Nagel.  One-parameter semigroups for linear evolution equations.  Springer-Verlag, New York, 2000.
  * [14] I. M. Gel’fand and N. Ya. Vilenkin.  Generalized functions. Vol. 4: Applications of harmonic analysis.  New York: Academic Press, 1964.
  * [15] M. Haase.  The functional calculus for sectorial operators.  Birkhäuser Verlag, Basel, 2006.
  * [16] A. Ichikawa.  Semilinear stochastic evolution equations: boundedness, stability and invariant measures.  Stochastics, 12(1):1–39, 1984.
  * [17] J. Jacod and A. N. Shiryaev.  Limit theorems for stochastic processes.  Springer-Verlag, Berlin, second edition, 2003.
  * [18] A. Jakubowski, S. Kwapień, P. Raynaud de Fitte, and J. Rosiński.  Radonification of cylindrical semimartingales by a single Hilbert-Schmidt operator.  Infinite Dimensional Analysis Quantum Probability and Related Topics, 5:429–440, 09 2002.
  * [19] A. Jakubowski and M. Riedle.  Stochastic integration with respect to cylindrical Lévy processes.  Ann. Probab., 45(6B):4273–4306, 2017.
  * [20] O. Kallenberg.  Foundations of modern probability.  Springer, New York, 2002.
  * [21] L. J. Kelley.  General Topology.  Graduate Texts in Mathematics. Springer- Verlag, New York-Berlin, 27th edition, 1975.
  * [22] H. Komatsu.  Fractional powers of operators.  Pacific J. Math., 19:285–346, 1966.
  * [23] T. Kosmala and M. Riedle.  Stochastic evolution equations driven by cylindrical stable noise.  Stochastic Process. Appl., 149:278–307, 2022.
  * [24] W. Linde.  Probability in Banach spaces—stable and infinitely divisible distributions.  John Wiley & Sons, Ltd., Chichester, second edition, 1986.
  * [25] B. Maslowski.  Stability of semilinear equations with boundary and pointwise noise.  Annali della Scuola Normale Superiore di Pisa-Classe di Scienze, 22(1):55–93, 1995.
  * [26] M. Métivier.  Semimartingales: A course on stochastic processes.  Walter De Gruyter, 1982.
  * [27] C. Mueller.  The heat equation with Lévy noise.  Stochastic Processes and Their Applications, 74(1):67–82, 1998\.
  * [28] L. Mytnik.  Stochastic partial differential equation driven by stable noise.  Probability Theory and Related Fields, 123(2):157–201, 2002.
  * [29] K. R. Parthasarathy.  Probability measures on metric spaces.  Academic Press, New York, 1967.
  * [30] S. Peszat and J. Zabczyk.  Stochastic partial differential equations with Lévy noise, volume 113 of Encyclopedia of Mathematics and its Applications.  Cambridge University Press, Cambridge, 2007.
  * [31] G. Pólya and G. Szegő.  Problems and theorems in analysis. I.  Classics in Mathematics. Springer-Verlag, Berlin, 1998.
  * [32] P. Protter.  Stochastic integration and differential equations.  Springer-Verlag, Berlin, second edition, 2005.
  * [33] M. Riedle.  Stable cylindrical Lévy processes and the stochastic Cauchy problem.  Electronic Communications in Probability, 23, 2018.
  * [34] J Rosinski and W.A. Woyczynski.  On Itô stochastic integration with respect to p-stable motion: inner clock, integrability of sample paths, double and multiple integrals.  The Annals of Probability, pages 271–286, 1986.
  * [35] H. L. Royden and P. Fitzpatrick.  Real analysis.  China Machine Press, fourth edition, 2017.
  * [36] L. Schwartz.  Geometry and probability in Banach spaces, volume 852 of Lecture Notes in Mathematics.  Springer-Verlag, Berlin-New York, 1981.  Based on notes taken by Paul R. Chernoff.
  * [37] I. E. Segal.  Abstract probability spaces and a theorem of Kolmogoroff.  Amer. J. Math., 76:721–732, 1954.
  * [38] B. Sz.-Nagy, C. Foias, H. Bercovici, and L. Kérchy.  Harmonic Analysis of Operators on Hilbert space.  Springer, New York, second edition, 2010.
  * [39] N. N. Vakhania, V. I. Tarieladze, and S. A. Chobanyan.  Probability distributions on Banach spaces, volume 14 of Mathematics and its Applications (Soviet Series).  D. Reidel Publishing Co., Dordrecht, 1987.
  * [40] D. Willett and J.S.W. Wong.  On the discrete analogues of some generalizations of Gronwall’s inequality.  Monatshefte für Mathematik, 69:362–367, 1965.
  * [41] D. Williams.  Probability with Martingales.  Cambridge University Press, 1991.
  * [42] H. Zhao and S. Xu.  A stochastic Fubini theorem for $\alpha$-stable process.  Statist. Probab. Lett., 160, 2020.


# Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language
Models?

Ben Yaoa, Yazhou Zhangb,c, Qiuchi Lia, Jing Qinb

aUniversity of Copenhagen, bThe Hong Kong Polytechnic University, cTianjin
University

###### Abstract

Elaborating a series of intermediate reasoning steps significantly improves
the ability of large language models (LLMs) to solve complex problems, as such
steps would evoke LLMs to think sequentially. However, human sarcasm
understanding is often considered an intuitive and holistic cognitive process,
in which various linguistic, contextual, and emotional cues are integrated to
form a comprehensive understanding of the speaker’s true intention, which is
argued not be limited to a step-by-step reasoning process. To verify this
argument, we introduce a new prompting framework called SarcasmCue, which
contains four prompting strategies, $viz.$ chain of contradiction (CoC), graph
of cues (GoC), bagging of cues (BoC) and tensor of cues (ToC), which elicits
LLMs to detect human sarcasm by considering sequential and non-sequential
prompting methods. Through a comprehensive empirical comparison on four
benchmarking datasets, we show that the proposed four prompting methods
outperforms standard IO prompting, CoT and ToT with a considerable margin, and
non-sequential prompting generally outperforms sequential prompting.

##  1 Introduction

Figure 1: The comparison of the processes of mathematical reasoning and
sarcasm detection.

Sarcasm is a subtle linguistic phenomenon that uses rhetorical devices such as
hyperbole and figuration to convey true sentiments and intentions that are
opposite to the literal meanings of the words used Wen et al. (2023); Zhang et
al. (2023b). Sarcasm detection aims to combine different types of cues, such
as linguistic features, contextual information, emotional knowledge, to form a
comprehensive understanding of the author’s sarcastic attitude. Owing to its
inherent ambivalence and figurative nature, sarcasm detection has persistently
proven a formidable challenge spanning the eras from feature engineering to
prompt engineering Yue et al. (2023); Zhang et al. (2023a).

Recent large language models have demonstrated impressive performance in
downstream natural language processing (NLP) tasks, in which “System 1” - the
fast, unconscious, and intuitive tasks, e.g., sentiment classification, topic
analysis, etc., have been argued to be successfully performed Cui et al.
(2024). Instead, increasing efforts have been devoted to the other class of
tasks - “System 2”, which requires slow, deliberative and multi-step thinking,
such as logical, mathematical, and commonsense reasoning tasks Wei et al.
(2022). To improve the ability of LLMs to solve such complex problems, a
widely adopted technique is to decompose complex problems into a series of
intermediate solution steps prior to answer generation, and elicit LLMs to
think step-by-step, such as chain of thought (CoT) Wei et al. (2022), tree of
thought (ToT) Yao et al. (2024), graph of thought (GoT) Besta et al. (2024),
etc.

However, sarcasm detection, as a holistic, intuitive, and non-rational
cognitive process, is arguably in noncompliance with step-by-step logical
reasoning due to two main reasons: (1) sarcasm expression does not strictly
conform to formal logical structures, such as the law of hypothetical
syllogism (i.e., if $\mathcal{A}\Rightarrow\mathcal{B}$ and
$\mathcal{B}\Rightarrow\mathcal{C}$, then
$\mathcal{A}\Rightarrow\mathcal{C}$). For example, “Poor Alice has fallen for
that stupid Bob; and that stupid Bob is head over heels for Claire; but don’t
assume for a second that Alice would like Claire”; (2) sarcasm judgment is
typically a fluid combination of various cues, where each cue holds equal
importance to the judgment of sarcasm, and there is no rigid sequence of steps
among them. As shown in Fig. 1, linguistic, contextual and emotional factors
are all crucial for rendering the sentence as sarcastic. Hence, the main
research question can be summarized as:

RQ: Is human sarcasm detection a step-by-step reasoning process?

To answer this question, we propose a theoretical framework, called
SarcasmCue, based on the sequential and non-sequential prompting paradigm. It
consists of four prompting methods, i.e., chain of contradiction (CoC), graph
of cues (GoC), bagging of cues (BoC) and tensor of cues (ToC). A cue is
similar to a thought, which is concretely a coherent language sequence related
to linguistics, context, or emotion that serves as an intermediate indicator
toward identifying sarcasm, such as rhetorical devices, emotional words, etc.
Each of the four prompting methods has its own focus and advantages.
Specifically,

  * •

CoC. It builds upon CoT prompting and harnesses the quintessential property of
sarcasm (namely the contradiction between surface sentiment and true
intention). It aims to: (1) identify the literal meaning and surface sentiment
by extracting keywords, sentimental phrases, etc.; (2) deduce the true
intention by scrutinizing special punctuation, rhetorical devices, cultural
background, etc.; and (3) determine the inconsistency between surface
sentiment and true intention. It has a typical linear structure.

  * •

GoC. Generalizing over CoC, GoC frames the problem of sarcasm detection as a
search over a graph and treats various cues (e.g., linguistic, contextual,
emotional cues, etc.) as nodes, with the relations across cues represented as
edges. Different from CoC and ToT, It allows language models to flexibly
choose and weigh multiple cues when detecting sarcasm, rather than following a
fixed hierarchy or linear reasoning path, unconstrained by the need for unique
predecessor nodes. It represents a graphical structure. In summary, both CoC
and GoC follow a step-by-step reasoning process.

  * •

BoC. In contrast, BoC and ToC are proposed based on the assumption that
sarcasm detection is not a step-by-step reasoning process. BoC is a bagging
approach that constructs a pool of diverse cues and creates multiple cue
subsets through randomly sampling q cues at each round. LLMs are employed to
generate multiple predictions based on these subsets, and such predictions are
aggregated to produce the final result via majority voting. It has a set-based
structure.

  * •

ToC. ToC treats each type of cues (namely linguistic, contextual, and
emotional cues) as an independent, orthogonal view for sarcasm understanding
and constructs a multi-view representation through the tensor product of these
three types of cues. It allows language models to leverage higher-order
interactions among the cues. ToC can be visualized as a 3D volumetric
structure, where each coordinate axis corresponds to a distinct type of cue.
This tensorial method aims to offer a more comprehensive and expressive means
of fusing diverse cues.

We present empirical evaluations of the proposed prompting approaches across
four sarcasm detection benchmarks over 2 SOTA LLMs (i.e., GPT-4o, LLaMA 3-8B),
and compare their results against 3 SOTA prompting approaches (i.e., standard
IO prompting, CoT, ToT). We show that the proposed four prompting methods
outperforms standard IO prompting, CoT and ToT with a margin of 2%, and non-
sequential prompting generally outperforms sequential prompting. Between the
two LLMs, GPT-4o consistently beats LLaMA by a striking margin across all
tasks.

The main contributions are concluded as follows:

  * •

Our work is the first to investigate the step-wise nature of sarcasm judgment
by using both sequential and non-sequential prompting methods.

  * •

We propose a new prompting framework that consists of four sub-methods, $viz.$
chain of contradiction (CoC), graph of cues (GoC), bagging of cues (BoC) and
tensor of cues (ToC).

  * •

Comprehensive experiments over four datasets demonstrate the superiority of
the proposed prompting framework in zero-shot sarcasm detection.

##  2 Related Work

This section reviews two lines of research that form the basis of this work:
CoT prompting and sarcasm detection.

###  2.1 Chain-of-Thought Prompting

Inspired by the step-by-step thinking ability of humans, CoT prompting was
proposed to “prompt” language models to produce intermediate reasoning steps
that lead to the final answer. Wei et al. Wei et al. (2022) made a formal
definition of CoT prompting in LLMs and proved its effectiveness by presenting
empirical evaluations on arithmetic reasoning benchmarks. This work pioneered
the use of CoT prompting in NLP. However, its performance hinged on the
quality of manually crafted prompts, which was a costly and unstable process.
To fill this gap, Auto-CoT was proposed to automatically construct
demonstrations with questions and reasoning chains Zhang et al. (2022).
Different from Auto-CoT, Diao et al. Diao et al. (2023) presented an Active-
Prompt approach to determine which questions were the most important and
helpful to annotate from a pool of task-specific queries, for reducing the
human engineering workload. The impressive results of CoT prompting have
sparked a surge of exploration into designing CoT prompting strategies across
various tasks Fei et al. (2023); Li et al. (2023); Zheng et al. (2023). For
instance, Wang et al. Wang et al. (2024) used formal grammars as the
intermediate reasoning steps for domain-specific language generation.

Furthermore, Yao et al. Yao et al. (2024) introduced a non-chain prompting
framework, namely ToT, which made LLMs consider multiple different reasoning
paths and self-evaluated choices to decide the next course of action. They
proved the effectiveness of the ToT approach on the tasks requiring non-
trivial planning or search. Beyond CoT and ToT approaches, Besta et al. Besta
et al. (2024) modeled the information generated by an LLM as an arbitrary
graph (i.e., GoT), where units of information were considered as vertices and
the dependencies between these vertices were edges. Although the above-
mentioned approaches have shown exceptional performance on various arithmetic
and logical reasoning tasks, all of them adopt the sequential decoding
paradigm of “let LLMs think step by step”. Contrarily, it is argued that
sarcasm judgment does not conform to step-by-step logical reasoning, and there
is a need to develop non-sequential prompting approaches.

Figure 2: An illustration of our SarcasmCue framework that consists of four
prompting sub-methods.

###  2.2 Sarcasm Detection

Sarcasm detection is habitually treated as a text classification task, where
the target is to identify whether the given text is sarcastic or not Zhang et
al. (2024). It has evolved from early rule based and statistical learning
based approaches to traditional neural methods, such as CNN, RNN, and further
advanced to modern neural methods epitomized by Transformer models. In early
stage, the rule based approaches infer the overall sarcasm polarity based on
the refined sarcasm rules, such as the occurrence of the interjection word
Zhang et al. (2023a). Statistical learning based approaches mainly employ
statistical learning techniques, e.g., SVM, RF, NB, etc., to extract patterns
and relationships within the data Zhou et al. (2023).

As deep learning based architectures have shown the superiority over
statistical learning, numerous base neural networks, e.g., such as CNN Jain et
al. (2020), LSTM Ghosh et al. (2018), GCN Liang et al. (2022), etc., have been
predominantly utilized during the middle stage of sarcasm detection research,
aiming to learn and extract complex features in an end-to-end fashion. As the
field of deep learning continues to evolve, sarcasm detection research has
stepped into the era of pre-trained language models (PLMs). An increasing
number of researchers are designing sophisticated PLM architectures to serve
as encoders for obtaining effective text representations. For example, Liu et
al. Liu et al. (2022) proposed a dual-channel framework by modeling both
literal and implied sentiments separately. They also constructed two conflict
prompts to elicit PLMs to generate the sarcasm polarity Liu et al. (2023b).
Qiao et al. Qiao et al. (2023) presented a mutual-enhanced incongruity
learning network to take advantage of the underlying consistency between the
two modules to boost the performance. Tian et al. Tian et al. (2023) proposed
a dynamic routing Transformer network to activate different routing
transformer modules for modeling the dynamic mechanism in sarcasm detection.

However, the above-mentioned works still focus on how to utilize PLMs to
extract effective features, without leveraging the extraodinary context
learning capabilities of LLMs. In contrast, this paper makes the first attempt
to explore the potential of prompting LLMs in sarcasm detection.

##  3 The Proposed Framework: SarcasmCue

The overall schematic illustration of the proposed SarcasmCue framework is
illustrated in Fig. 2. We qualitatively compare SarcasmCue to other prompting
approaches in Tab. 1. SarcasmCue is the only one to fully support chain-based,
tree-based, graph-based, set-based and multidimensional array-based reasoning.
It is also the only one that simultaneously supports both sequential and non-
sequential prompting methods.

Table 1: Comparison of prompting methods.

Scheme | Seq? | Non-Seq?
---|---|---
Chain? | Tree? | Grap? | Set? | Tensor?
IO |  |  |  |  |
CoT |  |  |  |  |
ToT |  |  |  |  |
GoT |  |  |  |  |
SarcasmCue |  |  |  |  |

###  3.1 Task Definition

Consider a sarcasm detection task. Given the data set
$\mathcal{D}=\left\\{\left(\mathcal{X},\mathcal{Y}\right)\right\\}$, where
$\mathcal{X}=\\{x_{1},x_{2},\ldots,x_{n}\\}$ denotes the input text sequence
and $\mathcal{Y}=\\{y_{1},y_{2},\ldots,y_{n}\\}$ denotes the output label
sequence. We use $\mathcal{L}_{\theta}$ to represent a large language model
with parameter $\theta$. Our task is to leverage a collection of cues
$\mathcal{C}=\\{c_{1},c_{2},...,c_{k}\\}$ to bridge the input $\mathcal{X}$
and the output $\mathcal{Y}$, where each cue $c_{i}$ is a coherent language
sequence related to linguistics, context, or emotion that serves as an
intermediate indicator toward identifying sarcasm.

###  3.2 Chain of Contradiction

We capture the inherent paradoxical nature of sarcasm, which is the
incongruity between the surface sentiment and the true intention, and
introduce chain of contradiction, a CoT-style paradigm that allows LLMs to
decompose the problem of sarcasm detection into intermediate steps and solve
each before making decision (Fig. 2 (a)). Each cue
$c_{k}\sim\mathcal{L}_{\theta}^{CoC}\left(c_{k}|\mathcal{X},c_{1},c_{2},...,c_{k-1}\right)$
is sampled sequentially, then the output
$\mathcal{Y}\sim\mathcal{L}_{\theta}^{CoC}\left(\mathcal{Y}|\mathcal{X},c_{1},...,c_{k}\right)$.
A specific instantiation of CoC involves three steps:

Step 1. We first ask LLM to detect the surface sentiment via the following
prompt $p_{1}$:

Given the input sentence [$\mathcal{X}$], what is the SURFACE sentiment, as
indicated by clues such as keywords, sentimental phrases, emojis?

The output sequence
$y_{1}\sim\mathcal{L}_{\theta}^{CoC}\left(\mathcal{Y}|p_{1}\right)$ is
generated from the language model $\mathcal{L}_{\theta}^{CoC}$ conditioned on
input prompt $p_{1}$.

Step 2. We then ask LLM to carefully discover the true intention via the
following prompt $p_{2}$:

Deduce what the sentence really means, namely the TRUE intention, by carefully
checking any rhetorical devices, language style, unusual punctuation, common
senses.

The output sequence, denoted as $y_{2}$, is generated from the language model
conditioned on prompt $p_{2}$ as well as the previous interaction
$p_{1},y_{1}$, formulated as
$y_{2}\sim\mathcal{L}_{\theta}^{CoC}\left(\mathcal{Y}|p_{1},y_{1},p_{2}\right).$

Step 3. We finally ask LLM to examine the consistency between surface
sentiment and true intention and make the final prediction:

Based on Step 1 and Step 2, evaluate whether the surface sentiment aligns with
the true intention. If they do not match, the sentence is probably
‘Sarcastic’. Otherwise, the sentence is ‘Not Sarcastic’. Return the label
only.

$y_{3}$ is therefore generated based on a joint understanding of the preceding
context $y_{1}$, $y_{2}$ and $p_{1}$, $p_{2}$, $p_{3}$:
$y_{3}\sim\mathcal{L}_{\theta}^{CoC}\left(\mathcal{Y}|p_{1},y_{1},p_{2},y_{2},p_{3}\right)$.
The sarcasm label is identified from $y_{3}$ as the output of CoC.

Notably, CoC is built based on the presumption that all the cues are linearly
correlated, and detects human sarcasm through step-by-step reasoning.
Different from the original CoT, however, the steps are explicitly designed
for the sarcasm detection context. Further details are presented in Algorithm
1 in App. A.

###  3.3 Graph of Cues

The linear structure of CoC restricts it to a single path of reasoning. To
fill this gap, we introduce graph of cues, a GoT-style paradigm that allows
LLMs to flexibly choose and weigh multiple cues, unconstrained by the need for
unique predecessor nodes (Fig. 2 (b)). GoC frames the problem of sarcasm
detection as a search over a graph, and is formulated as a tuple
$\left(\mathcal{M},\mathcal{G},\mathcal{E}\right)$, where $\mathcal{M}$ is the
cue maker used to define what are the common cues, $\mathcal{G}$ is a graph of
“sarcasm detection process”, $\mathcal{E}$ is cue evaluator used to determine
which cues to keep selecting and in which order. Unlike ToT and GoT, GoC does
not involve the modules of “thought generator” and “thought aggregation”.

1\. Cue maker. Human sarcasm judgment often relies on the combination and
analysis of one or more cues to achieve an accurate understanding. Such cues
can be broadly categorized into three types: linguistic cues, contextual cues
and emotional cues. Linguistic cues refer to the linguistic features inherent
in the text, including keywords, rhetorical devices, punctuation and language
style. Contextual cues refer to the environment and background of the text,
including topic, cultural background, common knowledge. Emotional cues denote
the emotional stance conveyed by the text, including emotional words, special
symbols (such as emojis) and emotional contrasts. A total number of 4+3+3=10
cues are adopted.

2\. Graph construction. In $\mathcal{G}=\left(V,E\right)$, the cues are
regarded as vertices constituting the vertex set $V$, while the relations
across cues form the edge set $E$. If there is an edge between cues $c_{k}$
and $c_{j}$, it is considered that $c_{k}$ and $c_{j}$ are closely related.
Given the cue $c_{k}$, the cue evaluator $\mathcal{E}$ considers cue $c_{j}$
to provide the most complementary information to $c_{k}$, which would combine
with $c_{k}$ to facilitate a deep understanding of sarcasm.

3\. Cue evaluator. We involve $\mathcal{G}$ in the LLM detecting sarcasm
process. To advance this process, the cue evaluator $\mathcal{E}$ assesses the
current progress towards judging sarcasm by means of determining whether the
cumulative cues obtained so far are sufficient to yield an accurate judgment.
If so, the search goes to an end. Otherwise, it serves as a heuristic for the
search algorithm, determining which additional cues to select and in what
order, to further the detection process. Similar to ToT, an LLM is used as the
cue evaluator $\mathcal{E}$.

We employ a voting strategy to determine the most valuable cue for selection,
by explicitly comparing multiple potential cue candidates in a voting prompt,
such as:

Given an input text $\mathcal{X}$, the target is to accurately detect sarcasm.
Now, we have collected the keyword information as the first step: {keywords},
judge if this provides over 95% confidence for accurate detection. If so,
output the result. Otherwise, from the remaining cues {rhetorical devices,
punctuation, …}, vote the most valuable one to improve accuracy and confidence
for the next step. This step can be formulated as
$\mathcal{E}\left(\mathcal{L}_{\theta}^{GoC},c_{j+1}\right)\sim
Vote\left\\{\mathcal{L}_{\theta}^{GoC}\left(c_{j+1}|\mathcal{X},c_{1,2,...,j}\right)\right\\}_{c_{j+1}\in\\{c_{j+1},...,c_{k}\\}}$.
In a nutshell, it greedily selects the most valuable cue until the final
judgment is reached.

Although the GoC enables the exploration of many possible paths across the cue
graph, its nature remains grounded in a step-by-step reasoning paradigm (see
Algorithm 2 in App. A).

###  3.4 Bagging of Cues

We further relax the assumption that the cues for sarcasm detection are inter-
related. We introduce bagging of cues, an ensemble learning based paradigm
that allows LLMs to independently consider varied combinations of cues without
assuming a fixed order or dependency among them (Fig. 2 (c)).

BoC constructs a pool of the pre-defined $k=10$ cues $\mathcal{C}$. From this
pool, $\mathcal{T}$ subsets are random sampled, each consisting of
$q~{}\left(i.e.,1\leq q\leq k\right)$ cues. BoC thus leverages LLMs to
generate $\mathcal{T}$ independent sarcasm predictions $\hat{y}_{t}$ based on
the cues of each subset. Finally, such predictions are aggregated using a
majority voting mechanism to produce the final sarcasm detection result. This
approach embraces randomness in cue selection, enhancing the LLM’s ability to
explore numerous potential paths, thus improving the robustness and accuracy
of sarcasm detection. BoC consists of the following key steps:

Step 1. Cue subsets construction. A total of $\mathcal{T}$ cue subsets
$\mathcal{S}_{t\in[1,2,...,\mathcal{T}]}=\left\\{(c_{t_{1}},c_{t_{2}},...,c_{t_{q}}),t\in[1,2,...,\mathcal{T}]\right\\}$
are created by randomly sampling without replacement from the complete pool of
cues $\mathcal{C}$. Each sampling is independent.

Step 2. LLM prediction. For each subset $\mathcal{S}_{t}$, an LLM
$\mathcal{L}_{\theta}^{BoC}$ is used to independently make sarcasm prediction
through the comprehensive analysis of the cues in the subset and the input
text. This can be conceptually encapsulated as
$\hat{y}_{t}\sim\mathcal{L}_{\theta}^{BoC}\left(\mathcal{Y}|\mathcal{S}_{t},\mathcal{X}\right)$.

Step 3. Prediction aggregation. These individual predictions are then combined
using an aggregation function, i.e., majority voting, to yield the final
prediction: $Y\sim
Vote\left(\left\\{\hat{y}_{1},\hat{y}_{2},...,\hat{y}_{\mathcal{T}}\right\\}\right)$.

BoC treats all cues as independent and does not follow the step-by-step
reasoning paradigm for sarcasm detection (see Algorithm 3 in App. A).

###  3.5 Tensor of Cues

CoC and GoC methods mainly handle low-order interactions between cues, while
BoC assumes cues are independent. To capture high-order interactions among
cues, we introduce tensor of cues, a novel paradigm that allows LLMs to
amalgamate three types of cues ($viz.$ liguistic, contextual and emotional
cues) into a high-dimensional representation (Fig. 2 (d)).

ToC treats each type of cues as an independent, orthogonal view for sarcasm
understanding, and constructs a multi-view representation through the tensor
product of such three types of cues. We first ask the LLM to extract
linguistic, contextual, and emotional cues respectively via a simple prompt.
Taking linguistic cue extraction as an example:

Instruction: Please extract the linguistic cues from the input sentence for
sarcasm detection, such as keywords, rhetorical devices, punctuation and
language style. Input: [$\mathcal{X}$]

We take the outputs of the LLM’s final hidden layer as the embeddings of the
linguistic, contextual and emotional cues, and apply a tensor fusion mechanism
to fuse the cues as additional inputs to the sarcasm detection prompt.
Inspired by the success of tensor fusion network (TFN) for multi-modal
sentiment analysis Zadeh et al. (2017), we apply token-wise tensor fusion to
aggregate the cues. In particular, the embeddings are projected on a low-
dimensional space, i.e.,
$\vec{Lin}=\left(e_{1}^{l},e_{2}^{l},...,e_{L}^{l}\right)^{T}$,
$\vec{Con}=\left(e_{1}^{c},e_{2}^{c},...,e_{L}^{c}\right)^{T}$,
$\vec{Emo}=\left(e_{1}^{e},e_{2}^{e},...,e_{L}^{e}\right)^{T}$. Suppose the
LLM has a hidden dimensionality of $d$, fully-connected layers
$f_{lin},f_{con},f_{emo}$ are constructed to map the embeddings to
dimensionality of $\\{d_{l},d_{c},d_{e}\\}$, respectively for linguistic,
contextual and emotional cues. Then, a tensor product is computed to combine
the cues into a high-dimensional representation
$\mathcal{Z}=\left(e_{1},e_{2},...,e_{L}\right)^{T}$, where

$\displaystyle e_{i}=\begin{bmatrix}e_{i}^{l}\\\
1\end{bmatrix}\otimes\begin{bmatrix}e_{i}^{c}\\\
1\end{bmatrix}\otimes\begin{bmatrix}e_{i}^{e}\\\ 1\end{bmatrix},\forall
i\in[1,2,...,L].$ (1)

The additional value of 1 facilitates an explicit rendering of single-cue
features and bi-cue interactions, leading to a comprehensive fusion of
different cues encapsulated in each fused token
$e_{i}\in\mathcal{R}^{(d_{l}+1)\times(d_{c}+1)\times(d_{e}+1)}$. The values of
$d_{l}$, $d_{c}$ and $d_{e}$ are delicately chosen such that the
dimensionality of fused token is precisely $d$111Otherwise the fused tokens
are truncated to d-dim vectors. That enables an integration of the aggregated
cues to the main prompt via:

Consider the information provided in the current cue above. Classify whether
the input text is sarcastic or not. If you think the Input text is sarcastic,
answer: yes. If you think the Input text is not sarcastic, answer: no. Input:
[$\mathcal{X}$]

The embedded prompt above is prepended with the aggregated cue sequence
$\mathcal{Z}$ before fed to the LLM. As it is expected to output a single
token of “yes” or “no” by design, we take the logit of the first generated
token and decode the label accordingly as the output of ToC.

ToC facilitates deep interactions among these cues, providing a powerful and
flexible framework for processing complex linguistic phenomena (see Algorithm
4 in App. A). Notably, as ToC manipulates cues on the vector level via neural
structures, it requires access to the LLM structure and calls for supervised
training on a collection of labeled samples. During training, the weights of
the LLM are frozen, and the linear weights in $f_{lin},f_{con},f_{emo}$ are
updated as an adaptation of LLM to the task context.

##  4 Experiments

###  4.1 Experiment Setups

Datasets. Four benchmarking datasets are selected as the experimental beds,
$viz.$ IAC-V1 Lukin and Walker (2013), IAC-V2 Oraby et al. (2016), SemEval
2018 Task 3 Van Hee et al. (2018) and MUStARD Castro et al. (2019).

IAC-V1 and IAC-V2 are from the Internet Argument Corpus (IAC) Lukin and Walker
(2013), specifically designed for the task of identifying and analyzing
sarcastic remarks within online debates and discussions. It encompasses a
balanced mixture of sarcastic and non-sarcastic comments.

SemEval 2018 Task 3 is collected using irony-related hashtags (i.e. #irony,
#sarcasm, #not) and are subsequently manually annotated to minimise the amount
of noise in the corpuses. It emphasize the challenges inherent in identifying
sarcasm within the constraints of MUStARD’s concise format, and highlight the
importance of context and linguistic subtleties in recognizing sarcasm.

MUStARD is compiled from popular TV shows including Friends, The Golden Girls,
The Big Bang Theory, etc. It consists of 690 samples total of 3,000
utterances. Each sample is a conversation consisting of several utterances. In
this work, we only use the textual information. The statistics for each
dataset are shown in Table 2.

Table 2: Dataset statistics.

Dataset | Avg. Length | #Train | #Dev | #Test
---|---|---|---|---
IAC-V1 | 68 | 1,595 | 80 | 320
IAC-V2 | 43 | 5,216 | 262 | 1,042
SemEval 2018 | 14 | 3,634 | 200 | 784
MUStARD | 14 | 552 | - | 138

Baselines. A wide range of SOTA baselines are included for comparison. They
are:

  * •

PLMs. (1) RoBERTa Liu et al. (2019), (2) BNS-Net Zhou et al. (2023), (3) DC-
Net Liu et al. (2022), (4) QUIET Liu et al. (2023a) and (5) SarcPrompt Liu et
al. (2023b) are five SOTA PLMs based approaches for sarcasm detection via pre-
trained language modeling and refined representations.

  * •

Prompt tuning. (6) IO, (7) CoT Wei et al. (2022) and (8) ToT Yao et al. (2024)
are four SOTA prompting approaches by leveraging advanced prompt approaches to
enhance LLM’s performance.

  * •

LLMs. (9) GPT-4o222https://openai.com/index/hello-gpt-4o/ and (10) LLAMA
3-8B-Instruct333https://llama.meta.com/llama3/ are the strongest general LLMs.

Implementation. We have implemented the prompting methods for GPT-4o and
LLaMA3-8B-Instruct, and reported the performance of PLMs in their original
papers. The GPT-4o methods are implemented with the official openAI Python API
library444https://github.com/openai/openai-python, while the LLaMA methods are
implemented based on the Hugging Face Transformers
library555https://huggingface.co/docs/transformers. All prompting strategies
are implemented for GPT-4o and LLaMA3-8B-Instruct except for ToC, which can
solely be deployed on open-sourced LLMs. Following previous works in this
field, LangChain666https://github.com/langchain-ai/langchain is employed for
the implementation of ToT and GoC. For the training of ToC, cross-entropy loss
between the output logit and the true label token is computed to update the
weights of the fully-connected layers.

Table 3: Performance on four datasets. For LLMs, all strategies but ToC are
based on a zero-shot setting.

Paradigm | Method | IAC-V1 | IAC-V2 | SemEval 2018 | MUStARD | Avg. of F1
---|---|---|---|---|---|---
Acc. | Ma-F1 | Acc. | Ma-F1 | Acc. | Ma-F1 | Acc. | Ma-F1
PLMs | RoBERTa | 72.10 | 72.90 | 82.70 | 82.70 | 73.90 | 72.80 | 66.27 | 65.16 | 73.39
BNS-Net | 66.13 | 65.95 | 75.93 | 75.92 | 73.47 | 73.44 | - | - | 71.77
DC-Net | 66.50 | 66.40 | 82.10 | 82.10 | 76.70 | 76.30 | - | - | 74.93
QUIET | - | - | - | - | - | - | 72.36 | 72.13 | -
SarcPrompt | 75.20 | 75.20 | 84.90 | 84.90 | 76.90 | 76.60 | 66.58 | 66.63 | 75.78
GPT-4o | IO | 70.63 | 70.05 | 73.03 | 71.99 | 64.03 | 63.17 | 67.24 | 65.79 | 68.14
CoT | 61.56 | 58.49 | 58.83 | 56.42 | 58.92 | 51.99 | 58.11 | 55.76 | 55.67
ToT | 71.56 | 71.17 | 70.63 | 69.07 | 63.90 | 63.02 | 69.00 | 68.27 | 67.46
2-10 | CoC (Ours) | 72.19 | 71.52 | 73.36 | 72.31 | 70.79 | 70.60 | 69.42 | 68.48 | 70.73
| GoC (Ours) | 85.38 | 68.08 | 64.97 | 61.30 | 74.03 | 74.02 | 70.69 | 69.91 | 68.33
| BoC (Ours) | 68.75 | 67.36 | 71.35 | 69.39 | 62.12 | 61.85 | 69.42 | 68.45 | 66.79
LLaMA 3-8B-Instruct | IO | 55.94 | 46.40 | 54.70 | 43.74 | 49.36 | 44.46 | 54.64 | 44.99 | 44.90
CoT | 56.25 | 47.28 | 54.22 | 42.96 | 49.36 | 44.55 | 54.20 | 44.86 | 44.91
ToT | 52.50 | 48.98 | 55.95 | 53.05 | 50.64 | 48.63 | 54.35 | 50.56 | 50.31
2-10 | CoC (Ours) | 56.25 | 46.95 | 54.03 | 42.6 | 49.23 | 44.36 | 54.93 | 45.66 | 44.89
| GoC (Ours) | 57.10 | 54.96 | 42.20 | 41.61 | 57.33 | 57.24 | 52.77 | 52.67 | 51.62
| BoC (Ours) | 62.50 | 59.28 | 62.57 | 58.11 | 59.82 | 58.40 | 59.71 | 56.70 | 58.12
| ToC (Ours) | 70.31 | 70.29 | 79.08 | 79.07 | 77.93 | 76.86 | 73.33 | 72.85 | 74.77

###  4.2 Main Results

We report both Accuracy and Macro-F1 results for SarcasmCue and baselines in a
zero-shot setting in Table 3, except for ToC which requires supervised
training for context adaption.

LLMs do not possess a unique advantage on sarcasm detection. Since sarcasm
indicates the manifestation of sentiments and intentions opposite to the
literal meaning of the texts, it usually violates logical reasoning pipelines
that LLMs are known to excel at Wei et al. (2022). This is empirically
validated in the experiment where LLMs are observed to have consistently lower
performance over PLMs in terms of average F1 scores across the four datasets.
This highlights the need to investigate prompting strategies for adapting LLMs
for sarcasm detection, towards which this work has made the first attempt and
achieved preliminary success.

Human sarcasm detection does not necessarily follow a step-by-step reasoning
process. The comparison between sequential (CoT, CoC, GoC, ToT) and non-
sequential (BoC, ToC) prompting strategies fails to provide clear empirical
evidences on whether sarcasm detection follows a step-by-step reasoning
process. Nevertheless, the results on LLaMA3-8B-Instruct are more indicative
to GPT-4o, since the latter has a strong capacity on its own (IO) and does not
significantly benefit from any prompting strategies on its top. On
LLaMA3-8B-Instruct where in-context learning is necessary for sarcasm
detection due to its poor IO performance, non-sequential approaches can
apparently offer more benefits over sequential ones, with a remarkable margin
consistently present on all four datasets. This seems to support our
hypothesize that sarcasm has a non-sequential nature.

SarcasmCue successfully adapts LLMs to sarcasm detection. The proposed
prompting strategies in the SarcasmCue framework achieve an overall superior
performance to the baseline prompting methods and bring about accuracy
increase over the original LLMs in a zero-shot setting. In particular, by
explicitly designing the reasoning steps for sarcasm detection, CoC beats CoT
by a tremendous margin on GPT-4o, whilst performing in par with CoT on
LLaMA3-8B-Instruct, an interesting result that further suggests the non-
squential nature of sarcasm detection. By pre-defining the set of cues on 3
main aspects, GoC and BoC manage to guide LLMs to reason along the correct
paths, leading to more accuracy judgment of sarcasm than the freestyle
thinking in ToT. The proposed trainable neural architecture in ToC achieves an
effective tensor fusion of multi-aspect cues for sarcasm detection, pushing
the capacity to a comparable level to PLMs without tuning the LLM parameters.

##  5 Acknowledgments

This work is supported by National Science Foundation of China under grant No.
62006212, Fellowship from the China Postdoctoral Science Foundation
(2023M733907), Natural Science Foundation of Hunan Province of China
(242300421412).

##  6 Conclusion

In this work, we aim to study the step-wise reasoning nature of sarcasm
detection, and introduce a prompting framework (called SarcasmCue) containing
four sub-methods, $viz.$ chain of contradiction (CoC), graph of cues (GoC),
bagging of cues (BoC) and tensor of cues (ToC). It elicits LLMs for human
sarcasm detection by considering sequential and non-sequential prompting
methods. Our comprehensive evaluations across multiple benchmarks and state-
of-the-art LLMs demonstrate that SarcasmCue outperforms traditional methods,
with non-sequential prompting methods (GoC and ToC) showing particularly
strong performance. In the future, we plan to develop the multi-modal version
of SarcasmCue for multi-modal sarcasm detection.

##  7 Limitations

The proposed SarcasmCue model has several limitations: (1) It incorporates
only three types of cues – linguistic, contextual, and emotional – while other
potentially useful cues, such as multimodal information, have not been
integrated, potentially limiting the model’s comprehensive understanding of
sarcasm; (2) the performance of SarcasmCue is influenced by the capabilities
of the underlying large language models (LLMs), meaning it performs better
with more powerful LLMs.

## References

  * Besta et al. (2024) Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. 2024.  Graph of thoughts: Solving elaborate problems with large language models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 17682–17690.
  * Castro et al. (2019) Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019.  Towards multimodal sarcasm detection (an _obviously_ perfect paper).  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Florence, Italy. Association for Computational Linguistics.
  * Cui et al. (2024) Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. 2024.  A survey on multimodal large language models for autonomous driving.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 958–979.
  * Diao et al. (2023) Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023.  Active prompting with chain-of-thought for large language models.  _arXiv preprint arXiv:2302.12246_.
  * Fei et al. (2023) Hao Fei, Bobo Li, Qian Liu, Lidong Bing, Fei Li, and Tat-Seng Chua. 2023.  Reasoning implicit sentiment with chain-of-thought prompting.  _arXiv preprint arXiv:2305.11255_.
  * Ghosh et al. (2018) Debanjan Ghosh, Alexander R Fabbri, and Smaranda Muresan. 2018.  Sarcasm analysis using conversation context.  _Computational Linguistics_ , 44(4):755–792.
  * Jain et al. (2020) Deepak Jain, Akshi Kumar, and Geetanjali Garg. 2020.  Sarcasm detection in mash-up language using soft-attention based bi-directional lstm and feature-rich cnn.  _Applied Soft Computing_ , 91:106198.
  * Li et al. (2023) Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2023.  Structured chain-of-thought prompting for code generation.  _arXiv preprint arXiv:2305.06599_.
  * Liang et al. (2022) Bin Liang, Chenwei Lou, Xiang Li, Min Yang, Lin Gui, Yulan He, Wenjie Pei, and Ruifeng Xu. 2022.  Multi-modal sarcasm detection via cross-modal graph convolutional network.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 1767–1777. Association for Computational Linguistics.
  * Liu et al. (2023a) Yaochen Liu, Yazhou Zhang, and Dawei Song. 2023a.  A quantum probability driven framework for joint multi-modal sarcasm, sentiment and emotion analysis.  _IEEE Transactions on Affective Computing_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2022) Yiyi Liu, Yequan Wang, Aixin Sun, Xuying Meng, Jing Li, and Jiafeng Guo. 2022.  A dual-channel framework for sarcasm recognition by detecting sentiment conflict.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1670–1680, Seattle, United States. Association for Computational Linguistics.
  * Liu et al. (2023b) Yiyi Liu, Ruqing Zhang, Yixing Fan, Jiafeng Guo, and Xueqi Cheng. 2023b.  Prompt tuning with contradictory intentions for sarcasm recognition.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 328–339.
  * Lukin and Walker (2013) Stephanie Lukin and Marilyn Walker. 2013.  Really? well. apparently bootstrapping improves the performance of sarcasm and nastiness classifiers for online dialogue.  In _Proceedings of the Workshop on Language Analysis in Social Media_ , pages 30–40, Atlanta, Georgia. Association for Computational Linguistics.
  * Oraby et al. (2016) Shereen Oraby, Vrindavan Harrison, Lena Reed, Ernesto Hernandez, Ellen Riloff, and Marilyn Walker. 2016.  Creating and characterizing a diverse corpus of sarcasm in dialogue.  In _Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 31–41, Los Angeles. Association for Computational Linguistics.
  * Qiao et al. (2023) Yang Qiao, Liqiang Jing, Xuemeng Song, Xiaolin Chen, Lei Zhu, and Liqiang Nie. 2023.  Mutual-enhanced incongruity learning network for multi-modal sarcasm detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 9507–9515.
  * Tian et al. (2023) Yuan Tian, Nan Xu, Ruike Zhang, and Wenji Mao. 2023.  Dynamic routing transformer network for multimodal sarcasm detection.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2468–2480, Toronto, Canada. Association for Computational Linguistics.
  * Van Hee et al. (2018) Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018.  SemEval-2018 task 3: Irony detection in English tweets.  In _Proceedings of the 12th International Workshop on Semantic Evaluation_ , pages 39–50, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wang et al. (2024) Bailin Wang, Zi Wang, Xuezhi Wang, Yuan Cao, Rif A Saurous, and Yoon Kim. 2024.  Grammar prompting for domain-specific language generation with large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * Wen et al. (2023) Changsong Wen, Guoli Jia, and Jufeng Yang. 2023.  Dip: Dual incongruity perceiving network for sarcasm detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2540–2550.
  * Yao et al. (2024) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024.  Tree of thoughts: Deliberate problem solving with large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Yue et al. (2023) Tan Yue, Rui Mao, Heng Wang, Zonghai Hu, and Erik Cambria. 2023.  Knowlenet: Knowledge fusion network for multimodal sarcasm detection.  _Information Fusion_ , 100:101921.
  * Zadeh et al. (2017) Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017.  Tensor fusion network for multimodal sentiment analysis.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1103–1114, Copenhagen, Denmark. Association for Computational Linguistics.
  * Zhang et al. (2023a) Yazhou Zhang, Dan Ma, Prayag Tiwari, Chen Zhang, Mehedi Masud, Mohammad Shorfuzzaman, and Dawei Song. 2023a.  Stance-level sarcasm detection with bert and stance-centered graph attention networks.  _ACM Transactions on Internet Technology_ , 23(2):1–21.
  * Zhang et al. (2024) Yazhou Zhang, Yang Yu, Qing Guo, Benyou Wang, Dongming Zhao, Sagar Uprety, Dawei Song, Qiuchi Li, and Jing Qin. 2024.  Cmma: Benchmarking multi-affection detection in chinese multi-modal conversations.  _Advances in Neural Information Processing Systems_ , 36.
  * Zhang et al. (2023b) Yazhou Zhang, Yang Yu, Dongming Zhao, Zuhe Li, Bo Wang, Yuexian Hou, Prayag Tiwari, and Jing Qin. 2023b.  Learning multi-task commonness and uniqueness for multi-modal sarcasm detection and sentiment analysis in conversation.  _IEEE Transactions on Artificial Intelligence_.
  * Zhang et al. (2022) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022.  Automatic chain of thought prompting in large language models.  _arXiv preprint arXiv:2210.03493_.
  * Zheng et al. (2023) Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. 2023.  Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.  _Advances in Neural Information Processing Systems_ , 36:5168–5191.
  * Zhou et al. (2023) Liming Zhou, Xiaowei Xu, and Xiaodong Wang. 2023.  Bns-net: A dual-channel sarcasm detection method considering behavior-level and sentence-level conflicts.  _arXiv preprint arXiv:2309.03658_.

##  Appendix A Algorithms of Four Prompting Methods

1\. CoC. We present further details of CoC in Algorithm 1.

Algorithm 1 Chain of contradiction

1:

2:Input: Sentence $\mathcal{X}$, an LLM $\mathcal{L}_{\theta}$

3:

4:Output: Sarcasm Label $\mathcal{Y}$

5:Step 1: Detect surface sentiment

6:Output cue $c_{1}$:
$c_{1}\sim\mathcal{L}_{\theta}^{CoC}\left(c_{1}|\mathcal{X},p_{1}\right)$

7:Step 2: Discover true intention

8:Output cue $c_{2}$:
$c_{2}\sim\mathcal{L}_{\theta}^{CoC}\left(c_{2}|\mathcal{X},c_{1},p_{2}\right)$

9:Step 3: Evaluate consistency and make prediction

10:Output cue $c_{3}$:
$c_{3}\sim\mathcal{L}_{\theta}^{CoC}\left(c_{3}|\mathcal{X},c_{1},c_{2},p_{3}\right)$

11:$\mathcal{Y}=\begin{cases}\text{Sarcastic}&\text{if }c_{1}\neq c_{2}\\\
\text{Not Sarcastic}&\text{otherwise}\end{cases}$

12:return $\mathcal{Y}$

2\. GoC. We present further details of GoC in Algorithm 2.

Algorithm 2 Graph of Cues (GoC) for Sarcasm Detection

1:

2:Input: Sentence $\mathcal{X}$, an LLM $\mathcal{L}_{\theta}$

3:

4:Output: Sarcasm Label $\mathcal{Y}$

5:1\. Graph Construction

6:Construct graph $\mathcal{G}=(V,E)$ where 10 cues are vertices $V$ and
relationships between cues are edges $E$

7:2\. Sarcasm Detection Process

8:Initialize selected cues $C_{\text{selected}}=\emptyset$, $j=0$

9:Initialize current confidence $\mathbb{C}=0$

10:while $\mathbb{C}<0.95\cap j\leq 10$ do

11: Select the most valuable cue:

12: $c_{\text{j+1}}\sim
Vote\left\\{\mathcal{L}_{\theta}^{GoC}\left(c_{j+1}|\mathcal{X},c_{1},c_{2},...,c_{j}\right)\right\\}_{c_{j+1}\in\\{c_{j+1},...,c_{10}\\}}$

13: Add $c_{\text{j+1}}$ to $C_{\text{selected}}$

14: Update current confidence $\mathbb{C}$, $j$++

15:Make final judgment based on $C_{\text{selected}}$:
$\mathcal{Y}=\mathcal{L}_{\theta}^{GoC}\left(\mathcal{Y}|\mathcal{X},C_{\text{selected}}\right)$

16:return $\mathcal{Y}$

Algorithm 3 Bagging of cues

1:

2:Input: Sentence $\mathcal{X}$, Cue Pool $\mathcal{C}$, Number of Subsets
$\mathcal{T}$, Number of Cues per Subset $q$, an LLM $\mathcal{L}_{\theta}$

3:

4:Output: Sarcasm Label $Y$

5:Step 1: Cue Subsets Construction

6:for $t=1$ to $\mathcal{T}$ do

7: Randomly sample a subset
$\mathcal{S}_{t}=\\{c_{t1},c_{t2},\ldots,c_{tq}\\}$ from $\mathcal{C}$

8:Step 2: LLM Prediction

9:for $t=1$ to $\mathcal{T}$ do

10: Generate sarcasm prediction
$\hat{y}_{t}\sim\mathcal{L}_{\theta}^{BoC}(\hat{y}_{t}|\mathcal{S}_{t},\mathcal{X})$

11:Step 3: Prediction Aggregation

12:Aggregate predictions using majority voting:

13:$Y\sim Vote(\\{\hat{y}_{1},\hat{y}_{2},\ldots,\hat{y}_{\mathcal{T}}\\})$

14:return $Y$

3\. BoC. We present further details of BoC in Algorithm 3.

4\. ToC. We present further details of ToC in Algorithm 4.

Algorithm 4 Tensor of cues

1:

2:Input: Sentence $\mathcal{X}$, an LLM $\mathcal{L}_{\theta}$

3:

4:Output: Sarcasm Label $\mathcal{Y}$

5:Step 1: Extract Cues

6:Obtain linguistic cue embeddings
$\vec{Lin}=(e_{1}^{l},e_{2}^{l},\ldots,e_{m}^{l})^{T}$, contextual cue
embeddings $\vec{Con}=(e_{1}^{c},e_{2}^{c},\ldots,e_{p}^{c})^{T}$, emotional
cue embeddings $\vec{Emo}=(e_{1}^{e},e_{2}^{e},\ldots,e_{s}^{e})^{T}$

7:Step 2: Construct Tensor Representation

8:Compute tensor product to combine cues:
$\mathcal{Z}=\vec{Lin}\otimes\vec{Con}\otimes\vec{Emo}$

9:Step 3: Sarcasm Detection

10:Take tensor $\mathcal{Z}$ as input to a LLM for sarcasm detection:

11:$\mathcal{Y}\sim\mathcal{L}_{\theta}^{ToC}(\mathcal{Y}|\mathcal{Z},\mathcal{X})$

12:return $\mathcal{Y}$


This work presents HiLMa-Res, a hierarchical framework leveraging reinforcement learning to tackle manipulation tasks while performing continuous locomotion using quadrupedal robots. Unlike most previous efforts that focus on solving a specific task, HiLMa-Res is designed to be general for various loco-manipulation tasks that require quadrupedal robots to maintain sustained mobility. The novel design of this framework tackles the challenges of integrating continuous locomotion control and manipulation using legs. It develops an operational space locomotion controller that can track arbitrary robot end-effector (toe) trajectories while walking at different velocities. This controller is designed to be general to different downstream tasks, and therefore, can be utilized in high-level manipulation planning policy to address specific tasks. To demonstrate the versatility of this framework, we utilize HiLMa-Res to tackle several challenging loco-manipulation tasks using a quadrupedal robot in the real world. These tasks span from leveraging state-based policy to vision-based policy, from training purely from the simulation data to learning from real-world data. In these tasks, HiLMa-Res shows better performance than other methods.

§ INTRODUCTION

Using legs as manipulators to perform non-prehensile manipulation tasks is a natural behavior commonly observed among humans.
Beyond just locomotion, people are capable of using their legs for a variety of actions, such as dribbling a soccer ball while walking or running, or moving an object by kicking it rather than lifting.
Enabling legged robots, like quadrupeds, to perform such loco-manipulation tasks with their legs while moving could significantly enhance their versatility and applicability.
However, this is a hard problem. Using the quadrupedal robot as an example, the robot needs to consider its stability while walking and also has to adjust its legs and use whole-body maneuvers to accomplish fine-grained manipulation tasks.
The robot has to update its leg movement strategy not only for locomotion control but also for manipulation planning in real-time.
Developing a general solution for different loco-manipulation tasks further complicates this due to the distinct objectives and environments associated with different manipulation tasks.
Many attempts have been made to address the challenge of loco-manipulation, which involves the coupled problem of locomotion control and manipulation planning. Due to its difficulty, many efforts have opted to simplify the scenario.
This simplification includes focusing on a single task, such as just ball dribbling <cit.>, or developing separate controllers like <cit.>: one for manipulation using the robot's legs and another for continuous locomotion skills.
It is still an open question to develop a general framework that can solve different manipulation tasks using the robot's legs while performing continuous locomotion.
In this work, we propose a hierarchical framework, named HiLMa-Res, to divide the complex loco-manipulation task and conquer the control and planning sub-problems individually by reinforcement learning (RL). It can be applied to different loco-manipulation tasks in the real world, as shown in Fig. <ref>.

§.§ Contributions

The main contribution of this work is the introduction of HiLMa-Res. (1) It is a general framework enabling quadrupedal robots to perform various manipulation tasks while maintaining sustained mobility through hierarchical reinforcement learning (RL). (2) The novel design of HiLMa-Res addresses the challenges of integrating manipulation skills with continuous locomotion. This includes implementing a single task-independent operational space locomotion controller, such as a walking controller, to track arbitrary end-effector (toe) trajectories, and a high-level task-specific manipulator planner for determining a residual trajectory for the end-effector.
(3) We demonstrate that this design offers a versatile solution for different loco-manipulation tasks, requiring minimal effort for retraining on individual tasks.
This framework supports different observation spaces and RL algorithms.
(4) Our experiments showcase the ability of HiLMa-Res by realizing various loco-manipulation tasks in the real world, including ball dribbling (training from simulation data), stepping over stones (using visuomotor skills), and load navigation (training from real-world data). Additionally, we show that HiLMa-Res outperforms other state-of-the-art RL methods in the challenging loco-manipulation task.

§ RELATED WORK

Different from the efforts on mobile manipulation with wheeled bases which does not need to consider the robot's stability <cit.>, loco-manipulation using legged robots requires different solutions. Previous work mainly focuses on two approaches: arm-equipped legged robots, and using the robot's legs as manipulators.

Mobile Manipulation with Legged Robots:
Equipping a legged robot, like a quadruped, with an additional arm can expand its manipulation capabilities by utilizing its mobility and whole-body maneuvers.
Previous research has explored various methods to improve the coordination between locomotion and manipulation.
Some have proposed a residual learning framework to enhance skill coordination <cit.>, requiring a pre-existing locomotion controller.
Others have focused on developing a unified whole-body policy for both manipulation and locomotion <cit.>, though without incorporating high-level planning for task space, and focusing solely on joint-level control.
Model-based approaches, using predictive and whole-body control, facilitate coordinated movements by accounting for multi-rigid body dynamics but depend on predefined contact sequences and task-specific constraints <cit.>. Techniques combining trajectory optimization and contact planning have achieved versatile multi-contact loco-manipulation, albeit with the necessity for offline planning <cit.>.
However, integrating an arm increases the system's complexity and necessitates additional hardware, more degrees of freedom, and the added load.

Using Legs for Both Locomotion and Manipulation:
Attempts leveraging legs as end effectors for manipulation using legged robots could have simplicity in hardware but limit tasks to non-prehensile manipulation <cit.>.
Legs are critical for stabilizing the robot, and using them for extra manipulation tasks poses challenges in maintaining stable gaits.
Some work avoids the need for precise manipulation by only focusing on manipulating large objects <cit.>.
To directly use legs for manipulation tasks, some prior work used two cascaded RL-based policies for control and planning in the robot's operational space, respectively.
This method enables a quadrupedal robot to shoot <cit.> or intercept <cit.>.
Others opt to develop a unified policy
However, these simplify the loco-manipulation problem by the constrained mobility, such as just standing <cit.> or only jumping once <cit.>.
In other work like <cit.>, a single policy is developed for individual loco-manipulation tasks that are still limited to constrained mobility, such as standing against a wall to push a button.
If there is a need for continuous locomotion, such as walking, to enlarge the robot's working space, a separate locomotion controller and rule-based policy selector are necessary in this work <cit.>.
Specialized tasks, like dribbling soccer balls while walking, have seen some development but lack scalability <cit.>.
In the humanoid robotics field, besides heuristic-based approaches like <cit.>, hierarchical learning <cit.> framework has been explored for using legs in manipulation tasks, yet this work focuses on task-specific training without a generalizable low-level controller for downstream manipulation tasks.
This work introduces HiLMa-Res, a hierarchical framework aiming for generality across loco-manipulation tasks, addressing these limitations.

The HiLMa-Res framework. This hierarchical framework consists of a controller training stage where we train a task-independent locomotion controller that tracks desired end-effector trajectories. This is an addition from sampled CPG trajectories and sampled Bézier residual trajectories. A planner training stage is designed to reuse the locomotion controller to train a task-specific manipulation planner for downstream loco-manipulation tasks. We highlight the importance of reusing a pre-trained locomotion controller that has been evaluated in the real-world environment, which enables fast and efficient learning of the planner and prevents learning dynamically infeasible actions.

§ THE HILMA-RES FRAMEWORK AND LOCO-MANIPULATION TASKS

In this section, we provide an overview of the HiLMa-Res framework, as visualized in Fig. <ref>.
We frame the loco-manipulation tasks with continuous locomotion as a combination of two problems with different levels of difficulty of generalization using RL: versatile locomotion control that can be realized by a single policy and object manipulation that can be hard to generalize to different tasks.

The HiLMa-Res contains two parts: a task-independent operational space locomotion controller and a task-specific manipulation planner, as shown in Fig. <ref>.
The locomotion controller is responsible for joint-level control of the whole body of the quadrupedal robot, leveraging the robot's proprioceptive feedback as input.
We adopt a motion tracking method where a parameterized reference motion provided by a Central Pattern Generator (CPG) <cit.> is utilized to provide gait priors for tracking various commanded velocities.
Note that the CPG is designed to generate end-effector (toe) trajectories for the quadrupedal robot with different contact sequences, different walking velocities, and turning rates.
This facilitates planning and control in the operational space.
Residual trajectories, represented by Bézier curves, are added to the nominal end-effector trajectories of the swing legs from CPG, while leaving the stance leg unchanged.
The details for training this operational space locomotion controller are described in Sec. <ref>.
After the locomotion controller is available and evaluated in the real world, it can be kept and re-used by different task-specific planners.
The planner is designed to accomplish specified loco-manipulation tasks by specifying the residual trajectories of the end-effector and the commands for the robot moving base for the locomotion controller.
The input of the planner includes not only the robot's proprioceptive feedback but also task-specific information, such as the object's location, or directly using depth vision as input.
Moreover, the planner can be constructed through different methods, as described in Sec. <ref>.

To demonstrate the versatility of the HiLMa-Res framework, we evaluate our method on three distinct loco-manipulation tasks, with increase in difficulty:

Ball Dribbling (Dribble): In this task, a quadrupedal robot dribbles a soccer ball and controls the ball's velocity and direction while dribbling, using a vision-based ball detection with onboard cameras.

Stepping over Stones (StepOStone): This task requires a quadrupedal robot to walk through a path littered with small stones or ground obstacles without collisions, inspired by the agile movements of a cat (as seen in <cit.>, and as tackled by a quadrupedal robot in simulation only in <cit.>).
Since measuring the positions of the small stone could introduce significant errors, we directly leverage the robot's onboard vision in the manipulation planner to prevent the robot's end-effectors from hitting the stones. This could allow the policy developed in simulations to be applied in the real world.

Navigating Load (NavLoad): the robot uses its front legs to push a small yet relatively heavy box towards a specified target location, requiring strategic long-term planning. The difference between simulated environments and the real world (like ground friction, size and stiffness of the load) can lead to accumulated errors over the long horizon. Therefore, this task necessitates real-world training to successfully and efficiently push the load to the target.

These tasks showcase a wide range of loco-manipulation capabilities: (1) the use of visual observation (StepOStone) versus state observation (Dribble, NavLoad), (2) the necessity for short-term planning (StepOStone, Dribble) versus long-term planning (NavLoad), and (3) tasks that demand training with real-world data (NavLoad) as opposed to those that can be transferred directly from simulation to the real world (StepOStone, Dribble).

§ TASK-INDEPENDENT QUADRUPEDAL LOCOMOTION CONTROL IN OPERATIONAL SPACE

The first component of HiLMa-Res is an RL-trained quadrupedal locomotion controller designed to be broadly applicable to a wide range of downstream loco-manipulation tasks.
The locomotion controller is trained to enable the robot to follow a diverse set of end-effector trajectories for the swing legs while walking, i.e., operating in the operational space.
For brevity, we use “trajectories" to refer to “end-effector trajectories of the robot's swing legs".

§.§ Parameterized Reference Trajectory in Operational Space

We first provide a parameterized reference trajectory.
Such a parameterized reference trajectory is the addition of two parts: (1) nominal trajectories for a trotting gait from a Central Pattern Generator (CPG), and (2) residual trajectories represented by Bézier trajectories that are added to the nominal trajectories, as illustrated in Fig. <ref>.
Given a trotting gait, there are two nominal and residual trajectories for the two swing legs, respectively.
All the trajectories are defined in the base footprint frame of the robot.
Having this design for the trajectories is the key to developing a versatile framework to enable the robot to learn to walk while adjusting the swing leg trajectories for manipulation tasks.

In this work, the quadrupedal robot employs a trotting gait, characterized by two diagonal legs swinging simultaneously. For each swing leg, there is a desired end-effector trajectory, which is a summation of a nominal CPG trajectory $\bm{\xi}_n$ and a residual Bézier trajectory $\bm{\xi}_r$. By learning to change the control points of Bézier trajectories and the base movement, the policy could adjust the swing trajectories to perform various loco-manipulation tasks.

§.§.§ Nominal Trajectories from CPG

The Central Pattern Generator (CPG) is a bio-inspired way to obtain swing foot trajectories, which is commonly used in quadrupedal locomotion control.
CPG represents the trajectories of all four feet in 3D Cartesian space by high-order polynomials that are parameterized by desired base velocity $\dot{q}^d_{x,y}$ and turning yaw rate $\dot{q}^d_\psi$, with a given step height at the apex for swing legs.
By using a periodic oscillator, the contact sequence of each leg can be obtained by choosing different oscillator parameters, which results in different rhythmic gaits, such as trotting, pacing, and etc.
For more details of CPG, we refer readers to <cit.>.
For simplicity, we only use CPG to develop a trotting gait by fixing the parameters of the oscillator in this work.
In this way, with a fixed gait period $T_g$, by varying the given command, the periodic trajectories $\bm{\xi}_n(t)$ can be well defined with respect to the time $t$.
These serve as nominal trajectories for the operational space locomotion controller.
We further denote $\dot{q}^d_{x,y,\psi}$ as CPG parameters which could be varied in this work.

§.§.§ Residual Trajectories Represented by Bézier Curves

Bézier curves are utilized to specify the change of the nominal trajectories from CPG, representing the residual trajectories $\bm{\xi}_r(t)$.
In this work, the Bézier curves are parameterized by three control points $\mathbf{P}_i$ ($i=0,1,2$) in 3D Cartesian space.
To ensure the swing legs' trajectories start and end on the ground (with vertical displacement $z=0$), we set the vertical placement of the first and last control points to zero, leveraging the characteristic of Bézier curves to always pass through these points.
In this way, the Bézier parameters that can be chosen are $\mathbf{P}_{0,2}\in \mathbb{R}^2$ and $\mathbf{P}_1\in \mathbb{R}^3$.
Then the change of position in the trajectory at time $t$ can be well defined by the Bézier function $\bm{\xi}_r(t)$.
The time $t$ needs to be first normalized to be within $[0,1]$ w.r.t. the timespan of the Bézier trajectory (i.e., the robot swing period $T_{sw} = T_g / 2$).

§.§.§ Control Objective

The objective of this locomotion controller at each time $t$ is to track the desired end-effector trajectories of the swinging legs, $\bm{\xi}_n(t) + \bm{\xi}_r(t)$, while tracking the commands for robot base, such as walking velocity and turning yaw rate (the CPG parameters $\dot{q}^d_{x,y,\psi}$).

§.§ Control Environment

The control policy is trained via RL in a simulated environment to track target end-effector trajectories and robot base commands while trotting.

§.§.§ Action

The control policy's output actions specify desired motor positions $\mathbf{a}^c_k$ at each timestep $k$. The policy is queried at 50 Hz, and the target motor positions are then utilized in a joint-level PD controller running at 1 kHz to calculate motor torques.

§.§.§ Observation

At each timestep $k$, the control policy receives an observation from the environment and determines an appropriate action for that scenario. The observation $\mathbf{o}^c_k$ is the robot's proprioceptive feedback, which includes the robot's current angular position $q_{\phi,\theta,\psi}$, angular velocity $\dot{q}_{\phi,\theta,\psi}$, motor position $\mathbf{q}_m$, and motor velocity $\dot{\mathbf{q}}_m$.
Alongside the current observation, we also keep track of a short 4-timestep history of the robot's past input (action) and output (observation), i.e., $(\mathbf{o}^c_{k:k-4},\mathbf{a}^c_{k-1:k-4})$, which are provided as input to the control policy.
This short I/O history helps to facilitate state estimation and system identification.
Additionally, we include a phase variable $\varphi=2\pi t/T_g$ for the control policy to indicate the time $t$ of one period of a trotting gait $T_g$. The phase variable is encoded using a smooth sinusoidal representation $(\sin{\varphi},\cos{\varphi})$.

§.§.§ Goal

In addition to the observations, the policy is also conditioned on a goal, which consists of two components. The first component is the desired end-effector trajectories to track, which are altered by Bézier parameters $\mathbf{P}_{0,1,2}$ for each swing leg.
The second part includes the command for the robot's moving base, including desired sagittal velocity $\dot{q}^d_x$, lateral velocity $\dot{q}^d_y$, and turning yaw rate $\dot{q}^d_{\psi}$, which are also parameters for the CPG.

§.§.§ Reward

At each timestep $k$, the robot executes an action $\mathbf{a}_k$ from the control policy. The environment then transitions to a new state and produces a reward $r^c_k$.
The reward is a weighted composition of different terms.
The most dominating term (with the largest weight) is the end-effector tracking reward, which encourages the robot to track the desired trajectories of the four feet.
For swing legs, the reference trajectories are obtained in Sec. <ref>.
For stance legs, the reference trajectories become a single point to keep the foot unmoved.
This reward is formulated as $r^c_\text{tracking} = \exp(-\sigma_e||\mathbf{x}_{e,t}-(\bm{\xi}_n(t)+\bm{\xi}_r(t))||_2)$ where $\mathbf{x}_{e,t}$ is the robot end-effector position $\mathbf{x}_e$ at time $t$ and $\sigma_e>0$ is a hyperparameter.
Furthermore, penalties are also introduced for roll, pitch angular velocities and vertical linear velocity to stabilize the robot base and yaw position tracking.

§.§.§ Task Randomization

To diversify training, we randomize the controller policy's tasks by varying the goal, by sampling Bézier parameters and CPG parameters uniformly from ranges in Table <ref>.
Diverse trajectories are generated by altering these parameters every 10 seconds.
Training policies to perform a diverse range of tasks can also improve the robustness of the learned models <cit.>.

4cBézier Parameters Randomization Range

Control points     $x$ Range [m]    $y$ Range [m]    $z$ Range [m]

$\mathbf{P}_0$     [-0.07, 0.03]     [-0.1, 0.1]     -

$\mathbf{P}_1$     [-0.035, 0.035]     [-0.1, 0.1]     [-0.05, 0.05]

$\mathbf{P}_2$     [-0.03, 0.07]     [-0.1, 0.1]     -

4cCPG Parameters Randomization Range

Command     Range     Unit     -

$\dot{q}^d_x$     [-1., 1.]     m/s     -

$\dot{q}^d_y$     [-0.3, 0.3]     m/s     -

$\dot{q}^d_\psi$     [-1., 1.]     rad/s     -

Range of Bézier parameters (control points) and CPG parameters used during training.

§.§.§ Dynamics Randomization

Since the training environment is built in simulation,  we also include dynamics randomization <cit.> to facilitate transfer from simulation to the real world. The randomized dynamics parameters include joint PD gains, ground friction, base mass, and random perturbations, whose ranges are detailed in  <cit.>.

§.§ Training and Deployment Details

The episode is designed to last 4000 timesteps or 80 seconds, trained with GPU-accelerated Isaac Gym physics simulator. The policy uses a three-layer MLP with hidden sizes of $[128, 64, 32]$ and ELU activation, optimized by PPO <cit.>.
After convergence, the obtained locomotion controller can be zero-shot transferred to the robot hardware.

§ TASK-SPECIFIC QUADRUPEDAL MANIPULATION PLANNING FOR RESIDUAL TRAJECTORIES

After obtaining the operational space locomotion controller that tracks arbitrary end-effector trajectories on the quadruped hardware, we can reuse this controller for different downsteam manipulation tasks.
Given the diverse nature of the manipulation tasks, as described in Sec. <ref>, we choose to develop task-specific high-level planning policies to specify the goal for the task-independent control policy.

§.§ Planning Environment

In this work, we use RL to develop the planning policy.
As a note, HiLMa-Res allows us to leverage alternative approaches, like learning from demonstrations, for solving manipulation planning.

§.§.§ Task-independent Action

The manipulation planner determines the end-effector trajectories and robot base movement for the controller based on the given task.
Specifically, the planner action $\mathbf{a}^p_k$ at each timestep includes Bézier parameters for swing legs' trajectories (control points $\mathbf{P}_{0,1,2}$) and CPG parameters (desired base velocity and turning rate $\dot{q}^d_{x,y,\psi}$).
The selection of Bézier parameters (for residual trajectories) solely influences end-effector trajectories, while CPG parameters (for nominal trajectories) alter desired base movement.
This action space ensures compatibility with a task-independent locomotion controller across various tasks.

§.§.§ Task-independent Observation

The planner's observation consists of task-independent elements consistent across tasks, and task-specific elements unique to individual tasks.
Task-independent observations include the robot base's measurable feedback, including its angular position and velocity, and the phase pair $(\sin{\varphi},\cos{\varphi})$ for synchronizing with the low-level locomotion controller's periodic gait.

§.§.§ Additional Task-specific Observation

Each task-specific planner requires unique observations from the environment, either state or sensory feedback.
HiLMa-Res allows us to flexibly manage both observation types.

State Representation
In Dribble task, the robot uses the relative position of the ball and the global heading of the robot to dribble the ball, conditioned on the given desired ball velocity (both speed and angle).
For Dribble, where controlling the ball's velocity is crucial, we input a 4-timestep history of the robot and ball positions into the policy to infer the ball velocity without relying on noisy velocity estimates.
In NavLoad task, the robot is provided with robot pose and load position, conditioned on the navigation target, all in the global frame, to navigate the load to the target.
We do not provide history as input in this position-based task.

Vision Representation
In the task of StepOStone, obtaining an accurate state estimator and acquiring a precise height map that correctly labels the scattered stones could be difficult on a quadrupedal robot with onboard sensors.
Therefore, the robot directly leverages its raw and downsampled depth vision from the ego view to learn to extract features related to the task.

§.§.§ Reward

The planning agent's reward $r^p_k$ at each timestep $k$ is task-specific within the hierarchical framework.
Thanks to the HiLMa-Res, we are able to provide simple task objectives without worrying the complexity of motion control, streamlining task learning and minimizing complex reward-tuning efforts present in end-to-end methods.

Dribble The reward is to encourage minimizing the tracking error of the desired linear velocity and moving direction of the ball. We also include a proximity term to encourage the robot to stay close to the ball.
Specifically, $r^p_\text{close} = \exp(-\sigma_{\text{ball}}d_{\text{ball}})$ where $d_\text{ball}$ is the distance between the robot and the ball and $\sigma_{\text{ball}}>0$ is a hyperparameter.

StepOStone The rewards are given for not stepping on stones or contacting with its calf. Specifically, $r^p_\text{contact} = 0$ if any leg is in contact, $1$ otherwise. Similarly, $r_\text{stepping} = 0$ if any leg steps on an object, $1$ otherwise. The weights are $0.7, 0.3$ respectively.

NavLoad We encourage the change of the distance between the load and the goal location.
$r^p_\text{move\_load} = \Delta d_\text{load} = d_{k-1,\text{load}} - d_{k,\text{load}}$, where $d$ is the Euclidian distance between the goal and the load.
We also include a proximity term formulated the same as the dribbling task, i.e., $r^p_\text{close} = \exp(-\sigma_{\text{load}}d_{\text{load}})$.

§.§ Training for Different Tasks

Using the HIMLA-Res, we explore different setups of the RL training, ranging from more capable but data-hungry algorithms that are mainly limited to simulation to data-efficient methods that can leverage real-world data.

§.§.§ Tasks for State-based Policy

Since the Dribble planner incorporates only a short history of the ball's position, we use an MLP with three hidden layers of dimensions $[128, 64, 32]$ and ELU activation. Trained via PPO in Isaac Gym, we simulate a dragging force on the ball introduced in <cit.> to facilitate zero-shot transfer from simulation to the real world.

§.§.§ Tasks for Vision-based Policy

Since the StepOStone policy directly uses depth vision, we firstly process the visual input via a CNN encoder with the first layer being convolution of size ([kernel, filter]) $[5, 32]$, second layer being Max Pooling of size ([kernel, stride]) $[2, 2]$, third layer being convolution of $[3, 64]$, ReLU activation, and a linear layer that maps the visual data into a latent space of hidden size of 64. Temporal features are later captured by a GRU with the same hidden size. It is then combined with state-based observation, and further processed by two MLP layers with hidden sizes of $[64, 32]$ and ELU activation.
Unlike prior works that have to first train in state space and then transition to vision via teacher-student methods due to their low sample efficiency <cit.>, our efficient controller allows for direct end-to-end training with depth input in simulation by PPO in just five hours wall time.
After training, the vision-based planner can be directly transferred to the real world.

§.§.§ Tasks that Requires Real-World Data

In the NavLoad task, we use a data-efficient DroQ <cit.> algorithm to learn from real-world data.
The actor network is an MLP whose first hidden layer has 128 ReLU units and the second has 64 Tanh units.
The critic network is a larger MLP with $[256, 128]$ hidden neurons, ReLU activation, and LayerNorm before the activation function.
we leverage RLPD <cit.> to fasten the real-world training.
First, we train the policy in simulation with DroQ (denoted as the base policy) and use this base policy to perform rollouts in the real world.
We collect a replay buffer of 10,000 transitions.
Then, we train a new policy from scratch using DroQ whose replay buffer is combined with both data collected by the base policy and the newly collected data from the real world, with a ratio of $70\%:30\%$.
This could facilitate efficient learning in the real-world environment as described in <cit.>.

Till now, details of the development of the HiLMa-Res have been introduced.

§ EXPERIMENT RESULTS

In this section, we evaluate the HiLMa-Res policies for different loco-manipulation tasks extensively in both simulation and the real world.
Due to the space limitation, we use NavLoad task as an example for extensive benchmark, as it is more challenging, and use Dribble and StepOStone as extension examples in the real world.

§.§ Navigating a Load

We first benchmark the performance of different methods for the NavLoad task in high-fidelity Gazebo simulation which provides a controlled environment.
Then, we report real-world training results, followed by a discussion on the benefits of HIMLA-Res compared to the baseline methods.

§.§.§ Baseline

We compare the HIMLA-Res method against three popular end-to-end baselines on the NavLoad task. The baselines include:

* Reward Shaping: This baseline adopts a reward scheme from <cit.> without reference motions and adds task reward directly to the original reward terms.

* Adversarial Motion Prior (AMP): This baseline is adopted from <cit.> and leverages adversarial imitation learning to learn from reference motions.

* Motion Tracking (MT): This baseline learns locomotion by tracking a given end-effector trajectory while improving the task reward <cit.>. In implementation, we adopt the locomotion controller of HIMLA-Res without Bezier residuals and CPG commands, and add task-specific observation and reward for this specific task.

These baselines reflect the state-of-the-art end-to-end locomotion controller, which has the potential to be extended to individual loco-manipulation tasks using task-specific policy.
We train each policy until it converges and achieves comparable success rates in training environments (Isaac Gym).

§.§.§ Simulation Benchmarking

First, we benchmark the performance of HIMLA-Res against other methods in a high-fidelity Gazebo simulation. Each method is tested for four different goals (1-meter in front, behind, left, or right of the load) with two attempts per goal, while the robot starts 1-meter behind the load at each reset.
As shown in Table <ref>, HIMLA-Res outperformed all end-to-end comparison methods in success rates. Specifically, the Reward Shaping method struggles to perform effective movement patterns during the sim-to-sim transfer, quickly losing balance and only kicking the load by chance. AMP manages to move when the load is distant but faces difficulties in stabilizing the body (being confused about which motion the robot should exert) as it approached the load, likely due to mode collapsing in adversarial learning. The MT baseline can move the loads to some extent but has trouble balancing motion imitation and task rewards, leading to unstable learning and evaluation performance.
In contrast, HIMLA-Res, with its stable locomotion controller and focused task planner, completes the task without issues, highlighting the advantage of a hierarchical approach for complex tasks.

Method     Success Rate (%)     Average Time (s)

3cSim-to-Sim Transfer

Reward Shaping
25     1.9

0     /

Motion Tracking
62.5     40.94

100     23.2

3cReal World Experiments

Ours (Base Policy)
80     57.75

Ours (Trained in Real)
100     47.6

Benchmark of the NavLoad task in sim-to-sim transfer and real experiments. We find that all end-to-end methods cannot achieve a high success rate in simulation other than the one it was trained on, highlighting the difficulty of achieving both robustness and high task performance in end-to-end training. In comparison, our method achieves a 100% success rate in simulation and can be transferred at an 80% success rate zero-shot in real environments. Furthermore, we are able to achieve a 100% success rate in the real environment after training with real data from scratch directly with RLPD algorithm.

Trial #     Stones Passed     Stones Fell       % Fell

22     2     9.09

20     3     15

19     2     10.5

19     3     15.8

20     2.5     12.5

Results of four trials on the StepOStone task in real-world experiments. We report the number of stones the robot passed, and the number of stones which the robot steps on or hits. The robot avoids on average $87.5\%$ of the stones.

§.§.§ Real World Training and Evaluation

We further test HiLMa-Res for NavLoad in the real world.
In the experiments, we utilize a localization system using Ultra-wideband (UWB) to measure the robot and load's positions and the robot's heading in the global framework.
When we apply the simulation-trained base policy to real hardware, a notable drop in both success rates and efficiency is witnessed, due to the load's shape not being a perfect cube as in the simulation and the high noise in localization when the robot gets close to the load, leading to inaccurate pushes and frequent misses, as shown in Fig. <ref>(a).
To address these challenges, we utilize the RLPD algorithm to train with real-world data.
The training from scratch with the RLPD algorithm takes just 9 minutes and 1,000 steps, leading to notable improvements in success rates and efficiency for real-world load navigation tasks.
Compared to the base policy, which frequently misses the load and makes wide and inefficient turns, the policy trained with real-world data navigates more carefully.
Shown in Fig. <ref>(b), this strategy increases effective kicks and successful load interactions.
On missing the load, it also adjusts with tighter turns for faster retries, showing better robustness to sensor noise.
These result in a shorter path to reach the target.
This efficiency underscores the benefits of a hierarchical approach, enabling efficient and reliable real-world task learning.

§.§ Ball Dribbling

In this Dribble task, we use the robot onboard stereo fisheye cameras to detect the ball and calculate its relative position.
With a proprioceptive state estimator, we can determine the robot's heading in the world frame for this short-term task. Our results as shown in Fig. <ref> demonstrate the dribbling planner's effectiveness, capable of executing a sharp U-Turn with the ball in spaces as narrow as 3.3 meters. The heading profile indicates a smooth approach to the target direction.
Despite a significant ball detection error at 16 seconds, the planner is still robust and goes directly to the ball's position, with the help of its history observations that help to filter noisy measurements.
The robot can dribble the ball at varying speeds, though the error in velocity tracking is large due to the gap in the physical properties of simulated and real soccer balls. Without a more accurate measurement system, we cannot learn the task directly in the real world due to noisy reward functions.

§.§ Step Over Stone

In the StepOStone task, we utilize a depth camera, downsampled to $43 \times 29$ to match our training setup, and apply a simulation-trained policy directly in real-world tests.
These tests involve randomly placed wood blocks ranging from $3$cm to $5$cm wide and $5cm$ to $10cm$ tall on a padded surface.
In Fig. <ref>, on the center left snippets, the robot exhibits deliberate toe trajectories to bypass a tall wood block by adjusting the CPG trajectories (nominally 10cm high, risking collision) using Bézier residuals, enabling it to step over and land safely beyond the obstacle.
On the center right snippets, the planner adeptly retracts the left front leg, preventing collision with another tall block, unlike the expected path of the nominal curve.
Quantitatively, the robot can $87.5\%$ of the wood blocks on average across four trials, as recorded in Table <ref>.
These results demonstrate the proposed framework's capability in mastering precise foot placements, notably in the StepOStone task, and underscore the locomotion controller's accurate end-effector tracking.

§ CONCLUSION AND FUTURE WORK

In conclusion, we present HIMLA-Res, a general hierarchical RL framework to tackle the loco-manipulation tasks.
It combines a task-independent operational space locomotion controller that enables tracking control for both robot moving base and end-effector trajectories, and task-specific manipulation planners for downstream tasks.
The HIMLA-Res is general and solves different loco-manipulation tasks.
Through these tasks, we show the flexibility of the HIMLA-Res framework in dealing with real-world training, agile loco-manipulation, and precise foot placements with the help of the policy hierarchy and the Bézier residual learning.
Compared to end-to-end methods, HIMLA-Res refrains the planner from overcoming sim-to-real challenges and benefits it with significantly higher sample efficiency.

Not limited to the presented tasks in this work, HiLMa-Res has the potential for other tasks, such as button pushing.
We also note that the framework can also be extended to different quadrupedal gaits other than trotting by training with more CPG parameters such as different contact patterns or stepping heights.
In the future, we think HiLMa-Res could also be useful for solving loco-manipulation tasks using humanoid robots.

§ ACKNOWLEDGEMENTS

This work was in part supported by The AI Institute and InnoHK of the Government of the Hong Kong Special Administrative Region via the Hong Kong Centre for Logistics Robotics. The authors thank the DAMODA Co., Ltd provides the UWB system.