# Surface Enhanced Infrared Absorption mechanism and modification of the
plasmonic response

Tanguy Colleu1, Adam Fekete1111Present address : Institut für Physik,
Humboldt-Universität zu Berlin, Newtonstraße 15, 12489 Berlin, Germany, Xavier
Gonze2, Alexandre Cloots2, Vincent Liégeois3, Gian-Marco Rignanese2, Luc
Henrard1 1Solid State Laboratory (LPS), Namur Institute of Structured Matter
(NISM), University of Namur, Rue de Bruxelles 61, 5000 Namur, Belgium
<EMAIL_ADDRESS><EMAIL_ADDRESS>,  2Institute of Condensed
Matter and Nanosciences, Université catholique de Louvain, Rue de
l’observatoire 8, bte L07.03.01, B-1348 Louvain-la-Neuve, Belgium
3Theoretical Chemistry Laboratory (LCT), Namur Institute of Structured Matter
(NISM), University of Namur, Rue de Bruxelles 61, 5000 Namur, Belgium

###### Abstract

Surface Enhanced Infrared Absorption (SEIRA) is an experimental method where
trace amount of a compound can be detected with high sensibility. This high
detection sensibility is the result of the interaction of the molecules with a
localized plasmon, usually from a metallic nano-particle. In this study we
numerically investigate by discrete dipole approximation the origin of the
Fano-like response of the system, including the induced transparency when the
plasmon resonance and the molecular vibrational mode coincide. The detailed
analysis of the localization of the absorption show that the modification of
the absorption cross-section when the molecule is present comes from a change
of the plasmonic resonance, not from the direct molecular response which is
negligible. This sheds a new light on the SEIRA mechanism. In particular, it
demonstrates that the sensibility is associated with the influence of the
molecule on the plasmon resonance rather than with the local field enhancement
itself.

Keywords : SEIRA, plasmonics , LSPR , nanorods, DDA, infrared absorption,
Vibrational properties

##  1 Introduction

Vibrational spectroscopies are powerful optical characterization techniques
that offer the advantages of being label free and non-destructive. In addition
to Raman spectroscopy, infrared (IR) absorption is widely used for direct
analysis of molecular functional groups and in situ analysis of surface
reactions [1, 2, 3, 4, 5, 6]. However, its small absorption cross-section,
although much larger than that of Raman spectroscopy, limits its utility for
large quantities of materials. In contrast, surface-enhanced infrared
absorption (SEIRA), based on the coupling with a plasmonic system, increases
the sensibility of the detection by a factor of 101 to 107, enabling optical
sensing of trace amounts of molecules [2, 3, 4, 5, 7, 8, 6].

SEIRA necessitates that the plasmonic system resonates at the same frequency
as that of the specific molecular vibrational mode under consideration. In
practice, gold nanoparticles are frequently utilized due to their high
chemical stability and their localized surface plasmon resonance (LSPR) in the
IR range. Gold nanorods are particularly popular due to the good control of
the synthesis or fabrication processes and the high tunability of their
optical resonance, through adjusting their length and radius [9, 10, 11, 6].
Alternative nanoparticle shapes with sharper edges such as gold bowtie [8] or
trimers [12] have been considered. Besides gold nanoparticles, graphene is
also a substrate of choice due to its tunable IR plasmon via doping [2, 13,
14, 15].

The interpretation of SEIRA relies on the comparison of the absorbance or
transmittance associated with the LSPR with or without the presence of the
probed molecules. When both resonances coincide, the optical response
diminishes with the presence of the molecules, resulting in a negative
differential cross-section and an induced transparency phenomenon. If the
resonances are slightly mismatched, a Fano-like behavior is observed [9, 10,
8, 14, 15, 16, 17, 18, 19, 20, 7, 21, 22]. The remarkable sensibility of SEIRA
is linked to the fact that the magnitude of the differential cross-section is
orders of magnitude larger than that of isolated molecules [2, 3, 4, 5, 7, 8].
This observation is often associated with the enhancement of the local
electromagnetic field at the molecule’s position [18, 16, 14, 6]. However,
this interpretation does not account for the induced transparency, or the
Fano-like shape [23, 10]. The relationship of the SEIRA with the square of the
local electric field which follows this first analysis has also never been
clearly evidenced.

In an early investigation [5] and in a more recent review [23], it was
suggested that the influence of the molecules on the polarizability of the
metal nanoparticles at the LSPR is the underlying mechanism for SEIRA. This
argument which was based on simulations of the optical response of ellipsoidal
metallic nanoparticles coated with a dielectric layer, posits that the
plasmonic particles act as an antenna, relaying the signal of the molecule,
rather than just exalting the electromagnetic fields.

In this study, we present a comprehensive examination of the SEIRA mechanism
through numerical simulations. The manuscript is organized as follows. Our
numerical simulations show that the SEIRA effect is associated with a
surprisingly high influence of the molecules on the LSPR resonance. They
demonstrate that the molecule differential cross-section does not originate
from changes of the absorption cross-section of the molecule. Furthermore, our
findings corroborate that the scattering cross-section exhibits a similar
behavior as the absorption cross-section in SEIRA, despite the fact that
molecules themselves do not scatter light [10].

In Sec.II, we introduce discrete dipole approximation (DDA) [24, 25] which is
used throughout this study. While DDA has proven to be effective and accurate
in describing the optical properties of nanomaterials, it has not been
previously applied in the context of SEIRA, to the best of our knowledge.
Previous numerical investigations of SEIRA, mainly utilizing the Finite
Difference Time Domain approach, have successfully replicated the Fano-like
spectra but have not provided definitive conclusions regarding the origin of
this phenomenon.

In Sec.III, we delve into the SEIRA response of a model molecular system,
featuring a vibrational mode corresponding to the CO stretching mode,
positioned within the hot spot created by two gold nanorods.

##  2 Methods

In DDA, the materials are modeled by a set of coupled polarizable dipoles
excited by an external field, a monochromatic plane wave in our case. The
formal derivation from Maxwell’s equations and boundary conditions [26]
reveals that the discretization of the bulk materials and the definition of
the local polarizability play a crucial role in determining the simulation
accuracy. For the gold nanorod, we derived the local polarizability from
lattice dispersion relation, which is a correction of Clausius-Mossotti’s
formula [24], from the experimental bulk refractive index [27]. The
simulations have been performed with the DDSCAT code [24]

In our simulations, we consider nano-rods with a $30$ nm diameter and a length
of $900$ nm, modeled by $199\,680$ dipoles placed on a cubic grid with a
periodicity of $1.875$ nm, ensuring the convergence (see also Ref [28] for an
investigation of the response of similar nano-object by DDA).

After the coupling with the external fields, the polarization of each dipole
is computed and the local properties such as the electromagnetic near-field is
determined. Far-field properties such as the absorption and scattering cross-
sections are calculated by summing the contribution of each individual dipole
and can be attributed to specific regions of the system.

To describe the response of the probed molecules in SEIRA, we take advantage
of the versatility of the DDA by representing them as point dipoles with a
polarizability associated with their vibrational modes described by a Drude-
Lorentz model. The vibrational mode wavenumber is varied from $1400$ cm-1 to
$2000$ cm-1, corresponding to the CO stretching modes in different chemical
environments, to investigate the detuning between the LSPR and one of the
molecular vibrational modes, with a damping parameter set at 16 cm-1. Of
course, in experimental systems, the vibrational frequency is fixed by the
molecular composition and cannot be tuned. In the same way, if the plasmon
resonance can be tuned by a modification of the geometry of the system, it
cannot be modified dynamically. Our main goal in this study is to investigate
the SEIRA effect with both the metallic nanoparticle and the molecules
presenting an optical resonance at the same frequency or at nearby
frequencies. For practical reasons, the metallic plasmonic system will be kept
fixed in the following, while the molecular mode will be tuned. Note that the
optical resonance of a nanorod can be modified thanks to LASER engraving [28].
The molecules are modeled by 88 dipoles dispersed in the gap with a distance
of at least $3.25$ nm with their closest neighbor which diminishes self-
interacting phenomenon.

Figure 1:  a) Extinction, absorption and scattering cross-section of the
single nano-rod (right, purple) and the double nano-rod (left, green). b)
Electric field in a plane at the center of the gold nanorods, zoomed on the
gap, at 1667 cm-1. The enhanced electric field is normalized to the incident
electric field amplitude. Figure 2:  a) Studied system with two gold nanorods
separated by $9.375$ nm with molecules in between. The incident field is along
the nanorods. b) Extinction, absorption and scattering cross-section of (a)
with molecules (solid line) and without molecules (dotted line). c) Extinction
cross-section difference between the situation with and without molecules
(green line) compared with that of the molecules without gold nanorods (blue
line). The latter has been multiplied by 106 to ease the comparison.

##  3 Results and Discussion

The electromagnetic response of the gold nanorods in the infrared is due to
the plasmon resonance that can be described, for an infinite nanorod, by the
azimuthal number m and the wavevector along the nanorod axis [28]. For finite
nanorod of length $L$, the wavector is quantified ($k=n\pi/L$) and the m=1
mode is excited by light polarized perpendicular to the nanorod axis and the
m=0 mode for parallel polarization [29]. The optical response of our gold
nanorod is shown in Fig. 1a for a longitudinal excitation in the model
wavenumber range of interest for this study. The resonance at $2150$ cm-1 to
$2000$ corresponds to the plasmon with the longest wavelength that can take
place in the nanorod ($n=1$).

The coupling with a second nanorod modifies the optical response and creates a
hot spot of intense local electromagnetic field, depending strongly on the
separation distance. In Fig. 1a, a redshift to $1667$ cm-1 is obtained for
nanorods separated by $9.375$ nm, a distance small enough to get a strong hot
spot but that can be described by our discretization grid. The corresponding
near-field amplitude in the hot spot region at the resonance wavenumber is, in
our case, enhanced by more than three orders of magnitude compared to the
incident field (Fig. 1b).

We investigate the SEIRA response of the molecule lying in the hot spot, as
illustrated on Fig. 2a. The wavenumber of the vibrational mode is chosen to be
$1667$ cm-1 to $2000$ corresponding to the maximum of the plasmon resonance of
the coupled nanorods.

The absorption, scattering and extinction cross-sections are shown with and
without molecules in Fig. 2b. We recover here the well-described decrease of
the total cross-section in the presence of the molecule [23, 16, 30, 9, 13,
14, 15, 1, 18, 4, 19, 10, 20, 8, 7, 31, 21, 2, 22, 12, 32, 17, 11] that is
observed for both absorption and scattering of light [10]. The difference of
the extinction cross-sections (Fig. 2c) evidences this transparency induced by
the molecules more clearly but also displays the Fano-like profile with an
increase of the extinction on the two sides of the main negative peak.

It is hardly justified to speak about enhancement for SEIRA due to the very
different shape of the response of the isolated molecule and of the SEIRA
signal. This is even clearer for the Fano line shape when the resonance
frequencies of the plasmonic system and of the molecules do not coincide (see
later). We then prefer the term sensibility that we define by the difference
between the extrema of the differential cross-section, divided by the maximum
absorption cross-section of the isolated molecule, following Ref [3]. In the
example above, the sensibility is $6.54\text{\times}{10}^{5}$. Experimentally,
the sensibility is difficult to evaluate as the optical cross-section of the
exact same plasmonic system without the molecules is not available and the
number of molecules in the hot spot is not known.

Figure 3:  Extinction, absorption and scattering cross-section of Fig. 2a with
molecules. Each set of curves corresponds to a different model molecule whose
vibrational mode wavenumber is indicated on the right, in the same color as
the associated curves. Figure 4:  Extinction cross-section of Fig. 2a with
molecules subtracted by the extinction cross-section without molecules. Each
curve is a model molecule with a different wavenumber for its vibrational
mode.

The case where both resonances coincide is very specific and does not
correspond to most of the experimental results. Fig. 3 shows the extinction,
absorption and scattering for five different vibrational wavenumbers below and
above the plasmonic resonance. Fig. 4 presents the Fano-like difference of
cross-section, with a curve shape in agreement with previous studies [9, 10,
8, 14, 15, 16, 17, 18, 19, 20, 7, 21, 22]. The overall shape of the SEIRA
signal depends strongly on the difference of the resonances between the
particles and the molecules [33], ranging from an induced transparency
described above to, mainly, an increase of the extinction when the resonances
do not coincide.

Figure 5:  Extinction cross-section of Fig. 4 for $1667$ cm-1 to $2000$ (green
line) with the contribution of the molecules (purple dash) and the
contribution of the nano-rods (green dash and point) separated. Figure 6:
Polarization average along the nano-rods without molecules (purple top line)
and with molecules (green bottom line).

In a DDA calculation, we can discriminate the spatial origin of the optical
response as it is expressed as a sum over the local dipoles [24]. It turns out
that the change of extinction originates almost entirely from the nanorods, as
it is illustrated for the case of the strongest SEIRA response in Fig. 5 (the
same conclusion can be drawn for the other computed spectra). This conclusion
is also supported by the analysis of the polarization profile of the nanorod
along its axis with and without the molecules (Fig. 6). The polarization
profile associated with the LSPR mode is very similar in both cases but with a
global drop of the response. This behavior is similar to the one expected if a
dielectric medium were added between the interacting nanorods which would make
the LSPR resonance to be less pronounced, tying in with previous qualitative
explanations [5, 23].

Finally, it is interesting to note that the SEIRA response is observed in both
the absorption and the scattering spectra (Fig. 3 and Ref [10]). As there is
no scattering from a single molecule (or from a point dipole in our model),
the change in scattering comes entirely from the nanorods, going in the same
direction as our previous conclusions. SEIRA is then associated with a
modification of the LSPR resonance due to the presence of the molecules rather
than to a direct contribution of the molecule to the spectra.

##  4 Conclusion

SEIRA is a very sensitive vibrational spectroscopy. Its modeling is not an
easy task at it requires to describe at the same time, the plasmonic response
of a metallic system and the vibrational response of a molecule. In this paper
we have proposed to investigate SEIRA within a DDA approach, reproducing its
main features such as the induced transparency and the Fano-like shape of the
resonance for different mismatch of the resonance frequencies.

We attribute the sensibility of SEIRA signature to a modification of the
plasmon excitation due to the presence of the molecules rather than to a
response of the molecules interacting with the local field. This is of prime
importance for the design of new SEIRA samples as the more adapted plasmonic
systems will be the ones that are more sensitive to a change of environment
(the molecules) rather than the ones with the highest local-field enhancement.

The calculations were performed on computers of the Consortium des Equipements
de Calcul Intensif, including those of the Technological Platform of High-
Performance Computing, supported by the FNRS-FRFC (Conventions No. 2.4.61707.F
and 2.5020.11) and the University of Namur. The authors The authors
acknowledge the funding by the Federation Wallonia-Brussels through the ARC
SURFASCOPE (UNamur-UCLouvain, Convention 19/24-102). V. Liégeois thanks the
F.R.S.–FNRS for his Research Associate position.

## References

  * [1] Hwang I, Kim M, Yu J, Lee J, Choi J H, Park S A, Chang W S, Lee J and Jung J Y 2021 Small Methods 5 2100277 ISSN 2366-9608
  * [2] Hu H, Yang X, Guo X, Khaliji K, Biswas S R, García de Abajo F J, Low T, Sun Z and Dai Q 2019 Nature Communications 10 1131 ISSN 2041-1723 number: 1 Publisher: Nature Publishing Group URL https://www.nature.com/articles/s41467-019-09008-0
  * [3] Baillieul M, Rinnert E, Lemaitre J, Michel K, Colas F, Bodiou L, Demésy G, Kakuta S, Rumyantseva A, Lerondel G, Boukerma K, Renversez G, Toury T, Charrier J and Nazabal V 2022 ACS Omega 7 47840–47850 publisher: American Chemical Society URL https://doi.org/10.1021/acsomega.2c05502
  * [4] Ishikawa A, Hara S, Tanaka T, Hayashi Y and Tsuruta K 2017 Scientific Reports 7 3205 ISSN 2045-2322 number: 1 Publisher: Nature Publishing Group URL https://www.nature.com/articles/s41598-017-03545-8
  * [5] Osawa M, Ataka K I, Yoshii K and Nishikawa Y 1993 Applied Spectroscopy 47 1497–1502 ISSN 0003-7028 publisher: SAGE Publications Ltd STM URL https://doi.org/10.1366/0003702934067478
  * [6] Kozuch J, Ataka K and Herbele J 2023 Nat Rev Methods Primers 3 ISSN 71 URL https://www.nature.com/articles/s43586-023-00261-8
  * [7] Neubrech F, Huck C, Weber K, Pucci A and Giessen H 2017 Chemical Reviews 117 5110–5145 ISSN 0009-2665 URL https://doi.org/10.1021/acs.chemrev.6b00743
  * [8] Dong L, Yang X, Zhang C, Cerjan B, Zhou L, Tseng M L, Zhang Y, Alabastri A, Nordlander P and Halas N J 2017 Nano Letters 17 5768–5774 ISSN 1530-6984 URL https://doi.org/10.1021/acs.nanolett.7b02736
  * [9] Giordano M C, Tzschoppe M, Barelli M, Vogt J, Huck C, Canepa F, Pucci A and Buatier de Mongeot F 2020 ACS Applied Materials & Interfaces 12 11155–11162 ISSN 1944-8244 publisher: American Chemical Society URL https://doi.org/10.1021/acsami.9b19719
  * [10] Neuman T, Huck C, Vogt J, Neubrech F, Hillenbrand R, Aizpurua J and Pucci A 2015 The Journal of Physical Chemistry C 119 26652–26662 ISSN 1932-7447 URL https://doi.org/10.1021/acs.jpcc.5b08344
  * [11] Abb M, Wang Y, Papasimakis N, de Groot C H and Muskens O L 2014 Nano Letters 14 346–352 ISSN 1530-6984 publisher: American Chemical Society URL https://doi.org/10.1021/nl404115g
  * [12] Mackin R T, Cohn B, Engelman B, Goldner A, Rubtsov I V and Chuntonov L 2019 The Journal of Physical Chemistry C 123 24731–24739 ISSN 1932-7447 publisher: American Chemical Society URL https://doi.org/10.1021/acs.jpcc.9b07045
  * [13] Farmer D B, Avouris P, Li Y, Heinz T F and Han S J 2016 ACS Photonics 3 553–557 publisher: American Chemical Society URL https://doi.org/10.1021/acsphotonics.6b00143
  * [14] Li Y, Yan H, Farmer D B, Meng X, Zhu W, Osgood R M, Heinz T F and Avouris P 2014 Nano Letters 14 1573–1577 ISSN 1530-6984 publisher: American Chemical Society URL https://doi.org/10.1021/nl404824w
  * [15] Yan H, Low T, Guinea F, Xia F and Avouris P 2014 Nano Letters 14 4581–4586 ISSN 1530-6984 publisher: American Chemical Society URL https://doi.org/10.1021/nl501628x
  * [16] Zvagelsky R, Chubich D, Pisarenko A, Bedran Z and Zhukova E 2021 The Journal of Physical Chemistry C 125 4694–4703 ISSN 1932-7447 publisher: American Chemical Society URL https://doi.org/10.1021/acs.jpcc.0c10433
  * [17] Cerjan B, Yang X, Nordlander P and Halas N J 2016 ACS Photonics 3 354–360 publisher: American Chemical Society URL https://doi.org/10.1021/acsphotonics.6b00024
  * [18] Langer J, Jimenez de Aberasturi D, Aizpurua J, Alvarez-Puebla R A, Auguié B, Baumberg J J, Bazan G C, Bell S E J, Boisen A, Brolo A G, Choo J, Cialla-May D, Deckert V, Fabris L, Faulds K, García de Abajo F J, Goodacre R, Graham D, Haes A J, Haynes C L, Huck C, Itoh T, Käll M, Kneipp J, Kotov N A, Kuang H, Le Ru E C, Lee H K, Li J F, Ling X Y, Maier S A, Mayerhöfer T, Moskovits M, Murakoshi K, Nam J M, Nie S, Ozaki Y, Pastoriza-Santos I, Perez-Juste J, Popp J, Pucci A, Reich S, Ren B, Schatz G C, Shegai T, Schlücker S, Tay L L, Thomas K G, Tian Z Q, Van Duyne R P, Vo-Dinh T, Wang Y, Willets K A, Xu C, Xu H, Xu Y, Yamamoto Y S, Zhao B and Liz-Marzán L M 2020 ACS Nano 14 28–117 ISSN 1936-0851 publisher: American Chemical Society URL https://doi.org/10.1021/acsnano.9b04224
  * [19] Neubrech F and Pucci A 2013 IEEE Journal of Selected Topics in Quantum Electronics 19 4600809–4600809 ISSN 1558-4542 conference Name: IEEE Journal of Selected Topics in Quantum Electronics
  * [20] Neubrech F, Beck S, Glaser T, Hentschel M, Giessen H and Pucci A 2014 ACS Nano 8 6250–6258 ISSN 1936-0851 URL https://doi.org/10.1021/nn5017204
  * [21] Wei J, Li Y, Chang Y, Hasan D M N, Dong B, Ma Y, Qiu C W and Lee C 2019 ACS Applied Materials & Interfaces 11 47270–47278 ISSN 1944-8244 publisher: American Chemical Society URL https://doi.org/10.1021/acsami.9b18002
  * [22] Tanaka T, Yano T a and Kato R 2022 Nanophotonics 11 2541–2561 ISSN 2192-8614 publisher: De Gruyter URL https://www.degruyter.com/document/doi/10.1515/nanoph-2021-0661/html
  * [23] Yang X, Sun Z, Low T, Hu H, Guo X, García de Abajo F J, Avouris P and Dai Q 2018 Advanced Materials 30 1704896 ISSN 1521-4095
  * [24] Draine B T and Flatau P J 1994 J. Opt. Soc. Am. A 11 1491–1499 URL http://josaa.osa.org/abstract.cfm?URI=josaa-11-4-1491
  * [25] Flatau P J and Draine B T 2012 Opt. Express 20 1247–1252 URL http://www.opticsexpress.org/abstract.cfm?URI=oe-20-2-1247
  * [26] Yurkin M A and Hoekstra A G 2007 Journal of Quantitative Spectroscopy and Radiative Transfer 106 558–589 ISSN 0022-4073 URL https://www.sciencedirect.com/science/article/pii/S0022407307000556
  * [27] Olmon R L, Slovick B, Johnson T W, Shelton D, Oh S H, Boreman G D and Raschke M B 2012 Physical Review B 86 235147 URL https://link.aps.org/doi/10.1103/PhysRevB.86.235147
  * [28] Pelaez-Fernandez M, Majerus B, Dufour R, Funes-Hernando D, Duvail J L, Henrard L and Arenal R 2023 Microscopy and Microanalysis 29 378–379 ISSN 1431-9276 URL https://doi.org/10.1093/micmic/ozad067.178
  * [29] Gómez-Medina R, Yamamoto N, Nakano M and Abajo F J G d 2008 New Journal of Physics 10 105009 ISSN 1367-2630 URL https://dx.doi.org/10.1088/1367-2630/10/10/105009
  * [30] Durmaz H, Li Y and Cetin A E 2018 Sensors and Actuators B: Chemical 275 174–179 ISSN 0925-4005 URL https://www.sciencedirect.com/science/article/pii/S0925400518314564
  * [31] Wang T, Dong Z, Koay E H H and Yang J K W 2019 ACS Photonics 6 1272–1278 publisher: American Chemical Society URL https://doi.org/10.1021/acsphotonics.9b00229
  * [32] Barbillon G 2022 International Journal of Molecular Sciences 23 10592 ISSN 1422-0067 number: 18 Publisher: Multidisciplinary Digital Publishing Institute URL https://www.mdpi.com/1422-0067/23/18/10592
  * [33] Tu X, Mario L Y and Mei T 2010 Optics Express 18 18820–18831 ISSN 1094-4087 URL https://opg.optica.org/oe/abstract.cfm?uri=oe-18-18-18820


# Complexity-Based Prompting for Multi-step Reasoning

Yao Fu♠, Hao Peng♣, Ashish Sabharwal♣, Peter Clark♣, Tushar Khot♣
♠University of Edinburgh ♣Allen Institute for AI
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Work done during internship at Allen Institute for AI

###### Abstract

We study the task of prompting large-scale language models to perform multi-
step reasoning. Existing work shows that when prompted with a chain of
thoughts (CoT), sequences of short sentences describing intermediate reasoning
steps towards a final answer, large language models can generate new reasoning
chains and predict answers for new inputs. A central question is which
reasoning examples make the most effective prompts. In this work, we propose
complexity-based prompting, a simple and effective example selection scheme
for multi-step reasoning. We show that prompts with higher reasoning
complexity, i.e., chains with more reasoning steps, achieve substantially
better performance on math word reasoning tasks over strong baselines. We
further extend our complexity-based criteria from prompting (selecting inputs)
to decoding (selecting outputs), where we sample multiple reasoning chains
from the model, then choose the majority of generated answers from complex
reasoning chains (over simple chains). When used to prompt GPT-3, our approach
substantially improves multi-step reasoning accuracy, with an 8.6% absolute
improvement on GSM8K, and 6.4% on MathQA. Compared with existing example
selection schemes like manual tuning or retrieval-based selection, selection
based on reasoning complexity is intuitive, easy to implement, and annotation-
efficient. Further results demonstrate the robustness of our methods under
format perturbation and distribution shift.

##  1 Introduction

We consider the problem of prompting large language models for multi-step
reasoning. Recent breakthroughs (Wei et al., 2022b; Wang et al., 2022b) show
that language models, when large enough ($>$100B parameters), exhibit the
emergent ability (Wei et al., 2022a) of performing complex multi-step
reasoning when provided with only a few reasoning examples. In the regime of
large models, prompting achieves comparable or even better performance than
full training set finetuning while being substantially more sample-efficient
(Wei et al., 2022b; Kojima et al., 2022; Lewkowycz et al., 2022). In
particular, Wei et al. (2022b) show that chain-of-thoughts (CoT) prompts,
sequences of short sentences describing intermediate reasoning steps towards
final answers (Fig. 1A), can elicit strong reasoning capabilities from large-
enough language models for complex tasks such as math word problems.

Figure 1: A: Chain of thoughts (in blue) are intermediate reasoning steps
towards a final answer. The input of CoT prompting is a stack of few (often 8)
CoT cases before a test question. Then the language model will continue
generating an output CoT for the test question. B: Chains of harder reasoning
complexity are chains with more reasoning steps (9 steps in this case, v.s.
only 2 steps in subfigure A). C: During decoding, we sample $N$ reasoning
chains from the language model ($N=5$ here), and take the majority answer over
the $K$ ($K=3$ here) most complex generated chains.

This work studies _example selection_ in chain-of-thoughts multi-step
reasoning. Example selection is a central problem in the prompting literature
(Liu et al., 2022; Rubin et al., 2022; Su et al., 2022; Lazaridou et al.,
2022). It asks what instances make the best prompts for solving the tasks of
interest. For CoT prompting, example selection is further related to
annotation efficiency, as CoT requires manually-annotated reasoning chains.
For datasets where reasoning annotations are easy to obtain, one may want to
know which annotated chains make the best prompt; if annotations are hard to
obtain, one may identify the best cases to annotate, rather than annotating
the entire dataset.

We propose complexity-based prompting, a new example selection scheme for
chain-of-thoughts multi-step reasoning. Existing sample selection methods are
usually based on manual tries (Wei et al., 2022b), heuristic rules (Wallace et
al., 2019), optimization and search (Shin et al., 2020), or retrieval from a
large training set (Rubin et al., 2022). Different from these schemes,
complexity-based prompting chooses examples with complex reasoning chains,
i.e., chains with more reasoning steps, as the prompt. Fig. 1A shows a simple
example with 2 reasoning steps, versus the example in subfigure B is a complex
case with 9 reasoning steps. As we will show in the experiments (section 4.2),
the reasoning performance of GPT-3 175B (Brown et al., 2020) clearly improves
with the increased input prompt complexity, where complex prompts achieve
better performance than simple prompts.

We further extend the complexity-based selection criteria from the input space
(the prompts) to the output space (reasoning chains generated by the language
model). Our extension is based on the idea of self-consistency (Wang et al.,
2022b; a), where they sample multiple reasoning chains (instead of using
greedy decoding) from the model that lead to possibly different answers, then
choose the majority of the generated answers. Here we propose complexity-based
consistency, where instead of taking a majority vote over all generated
chains, we vote over the top $K$ complex chains, as is shown in Fig. 1C. In
section 4.2, we will show that complexity-based consistency leads to further
performance gain, on top of the existing gain from complexity-based prompting.

Putting everything together, our methods achieve strong performance on GSM8K,
MultiArith, and MathQA benchmarks (all requiring strong multi-step reasoning
ability), with substantial performance gains over Wei et al. (2022b). Compared
with existing sample selection schemes, we show that complexity-based
prompting achieves better performance in most cases (see section 4.2).
Furthermore, performance gains from complex samples are consistent in
different prompt distributions (in-distribution, transfer, and noisily-
labeled, see section 4.2) and are also consistent with regard to alternative
proxies for complexity (e.g., question or formula lengths, see section 4.3)
when the dataset does not contain annotated reasoning chains. A careful
analysis shows that the number of reasoning steps is the most prominent
factor, over confounders like prompt lengths or the number of input cases
(section 4.3). We hope this work will open new research possibilities in in-
context learning, large language models, and multi-step reasoning.

##  2 Related Work

Emergent Abilities and Multi-Step Reasoning With the recent trend in scaling
language models (Brown et al., 2020; Chowdhery et al., 2022), a central
question is what unique abilities emerge as models become large (Kaplan et
al., 2020; Wei et al., 2022a). Generally, the ability to follow the format of
given prompts (typically few-shot) thus solving the corresponding tasks (also
referred as in-context learning), is something that large language models are
particularly skilled at (Shin et al., 2020; Liu et al., 2021). Among the wide
language understanding task spectrum, we are particularly interested in multi-
step reasoning because of its two uniqueness: (1). multi-step reasoning is a
task where large models substantially outperform smaller models (Wei et al.,
2022b), versus performance gains on tasks like sentiment classification can be
very limited with large models (Shin et al., 2020); (2). multi-step reasoning
is where few-shot prompting starts to outperform full training set fine-
tuning, even when fine-tuning is conducted on the same large model (Lewkowycz
et al., 2022). This work takes an important step forward in multi-step
reasoning by showing the critical role of prompt complexity.

Chain-of-Thoughts Reasoning A prominent work demonstrating the multi-step
reasoning of language models is chain-of-thoughts prompting (Fig. 1A),
proposed by Wei et al. (2022b). They show that the reasoning ability can only
be elicited by chain of thoughts, but not standard prompting where an answer
directly follows a question without intermediate reasoning steps. Further
works show that CoT can be improved by self-consistency (Wang et al., 2022b),
pretraining the model with latex-formated data (Lewkowycz et al., 2022),
context selection (Creswell et al., 2022), or even adding certain magic
phrases like “Let’s think step by step” (Kojima et al., 2022). The original
CoT paper (Wei et al., 2022b) uses 8 manually written examples as the prompt,
which are reused by most follow-up works. Our work sits in the context of CoT
reasoning, and propose a new complexity-based prompt selection that
substantially outperforms the original CoT.

Example Selection for Prompting Designing prompts can be challenging due to
the instability, as multiple works have shown the performance is sensitive to
prompt, task, dataset, and model changes (Zhao et al., 2021; Lu et al., 2022;
Su et al., 2022). Despite works on automatic prompt searching (which is more
suitable for smaller models, e.g., Shin et al., 2020; Li & Liang, 2021),
currently, prompt engineering for large models is (still) a community-wide
collective trial and error effort (there is even a prompt marketplace named
PromptBase, ). The difficulty is that it is extremely hard to extract
generalizable regularity from empirical observations that can form effective
selection criteria. One notable exception is similarity-based prompt
selection, which retrieves the most similar training instances as the prompt
for a given test case (Rubin et al., 2022). Yet for CoT prompting, retrieving
different prompts for different test cases requires reasoning chain
annotations for the whole training set, which compromises the advantage of
being few-shot. Given this background, our core contribution is identifying
complexity as an effective and robust selection criterion and in many cases,
it outperforms existing prompt selection schemes while being annotation-
efficient.

Relation to Classical Semantic Parsing The procedure of chain of thoughts
prompting is conceptually similar to classical semantic parsing where one
generates a logical form then executes it upon a knowledge base to reach a
final answer (Liang, 2016; Cheng et al., 2019). The practice of sampling then
voting is also similar to marginalizing out semantic parses (Yin et al.,
2018). There are further works linking the relationship between in-context
learning and classical Bayesian inference (Wei et al., 2021; Xie et al.,
2022). From our perspective, we tend to view chain-of-thoughts as flexible,
language model styled “logical forms” which are “executed” by the language
model itself. We leave further study on connecting classical parsing and CoT
to future work.

##  3 Complexity-based Prompting

We study multi-step reasoning tasks, and use math word problems, mathematical
problems expressed in natural language, as our testbed. This task, as is
measured by solve rate (accuracy), is to predict the answer (typically a
number) of a given math word problem via intermediate steps. We follow the
chain-of-thoughts prompting framework and compare all prompting schemes using
GPT-3 text-davinci-002. An example problem, as well as the chain-of-thoughts
workflow, is shown in Fig. 1A. The input is a stack of a few (often 8) CoT
cases followed by a test question, then the language model continues
generating an output CoT for the test question. Our goal is to improve the
reasoning accuracy by identifying and exploiting more effective input and
output reasoning chains.

###  3.1 Selecting Complex Samples as Prompts

Our method is simply selecting complex prompts over simple prompts. We
hypothesize that language models’ reasoning performance will increase if we
use complex instances as in-context “training example”, as they intuitively
subsume simple instances (Richardson & Sabharwal, 2022). We define complex
instances as instances with more reasoning steps (Fig. 1B), as the name
“multi-step reasoning” indicates. Note that using reasoning steps as the
notion of complexity is also the practice of previous works like (Sugawara et
al., 2018; Lai et al., 2021). We further define a step as a line, separated by
the linebreak “\n”.

There are two aspects that need more discussion: (1) The notion of complexity.
There are other complexity indicators than number of steps, such as questions
lengths or the length of the underlying formula for solving a given problem.
We will show that the trend that better performance comes with more complex
prompts is consistent across various complexity indicators, such as question
lengths and formula length. Consequently, for datasets that do not have
annotated reasoning chains, we can use questions lengths to identify complex
instances, then only annotate the identified few-shot instances, thus reducing
the annotation cost. (2) Confounders of number of steps. The increase in
performance with more complex examples in the prompt could be explained by
correlated factors like the increase in the total number of reasoning steps in
the prompts or just the increased length of the prompt. To account for this,
we evaluate prompts with simpler examples but the same number of reasoning
steps (e.g. 24 cases with 3 steps vs. 8 cases with 9 steps, both of 72 steps
in total). We also consider prompts of the longest lengths (but not most
steps). We show that the number of steps per example is the most prominent
source of performance gains over confounders.

###  3.2 Complexity-Based Consistency

Complexity-based prompting can be further enhanced with a new output selection
method following the same intuition, which we present in this section.
Existing evidence shows that the expressive neural models can take _shortcuts_
during reasoning, relying on spurious correlations that inevitably exist in
the training data (Mudrakarta et al., 2018; Sugawara et al., 2018; Lai et al.,
2021). This often leads to suboptimal generalization to unseen data. To
alleviate this issue, we explicitly promote outputs with more complex
reasoning chains at inference time. Specifically, our method follows the self-
consistency practice in Wang et al. (2022b), which samples $N$ reasoning
chains for a test question. Different reasoning chains may lead to different
answers, and Wang et al. (2022b) takes the majority answer as the prediction.
In our case, instead of voting among all $N$ chains, we only vote among top
$K$ ($K\leq N$) complex (more steps) reasoning chains, as shown in Fig. 1C. We
name our method Complexity-based Consistency. Note that when $K=N$ we recover
the original self-consistency method. In our experiments, we set $N$ to 50,
and observe that the optimal $K$ is always smaller than $N$ (typically 30-40).
This provides clear evidence that voting among more complex reasoning chains
generalizes better than voting among all. We also show that if we do the
opposite and vote among answers produced by $K$ simplest reasoning chains, the
accuracy is always worse than voting among all. This further validates that
complex chains, not simple chains, should be considered more during decoding.

##  4 Experiments

We first discuss our experimental settings in section 4.1. In section 4.2, we
present three main results: (1) on GSM8K (Cobbe et al., 2021), MultiArith (Roy
& Roth, 2015), and MathQA (Amini et al., 2019) benchmarks, our complexity-
based prompting substantially outperforms the original CoT (Wei et al.,
2022b); (2) complex prompts consistently outperforms simple prompts in
different prompt distributions, either the prompts are in-distribution,
transferred from another distribution, or from a highly-noisy distribution;
(3) compared with other example selection schemes (random, heuristic and
retrieval), complexity-based example selection often achieves the best or
competitive results with minimal annotation budget. In section 4.3, we
strengthen our results with addition analysis about notions of complexity,
format perturbation, confounders, and decoding methods.

###  4.1 Experimental Settings

Datasets We use three math word problems datasets (GSM8K, MultiArith, and
MathQA) as our testbed. We choose GSM8K and MultiArith also because they are
the datasets used by prior work on CoTs (Wei et al., 2022b; Wang et al.,
2022b; Kojima et al., 2022), allowing fair comparison to existing methods.
MathQA’s annotation are much noisier than others, and we use it to evaluate
the robustness of our approach. There are 1.3K test instances in GSM8K, 600 in
MultiArith, and 600 in MathQA. For each dataset, we randomly draw 200
instances from the training data to create a validation split. The cost of
prompting GPT-3 is proportional to the size of test set. Evaluating on a
200-instances validation set costs about 6-8 US dolars for greedy decoding (1
output chain) and $12-$24 for sampling 50 output chains.

Language Models We consider two paradigms: fine-tuning and prompting. For
fine-tuning, we report the existing SOTA performance: a fine-tuned GPT3 with a
verifier (Cobbe et al., 2021) on GSM8K, a relevance and LCA operation
classifier (Roy & Roth, 2015) on MultiArith and a customized sequence to
sequence model (Amini et al., 2019) on MathQA. For prompting, we consider the
following four language models: (1). LaMDA (Thoppilan et al., 2022), a 137B
model used as the baseline in Wei et al. (2022b); (2). PaLM (Chowdhery et al.,
2022), the primary 540B model used in the CoT papers; (3). Minerva (Lewkowycz
et al., 2022), a 540B large model that trains on LaTeXdata; it achieves SOTA
performance in math reasoning on GSM8K; (4). GPT-3 175B (davinci-002 from
Brown et al., 2020), the language model used to evaluate our method. LaMDA,
PaLM and Minerva are not accessible to the public, and their numbers are from
their corresponding papers.

Prompts and Hyperparameters The training sets of GSM8k and MathQA contain
human annotated reasoning chains, within which we search for complex prompts.
MultiArith does not have annotated reasoning chains, so we consider two
strategies. (1). in-distribution annotation, which uses question lengths as an
alternative proxy for complexity, then manually annotates reasoning chains for
complex questions; (2). prompts transfer from GSM8K training data. Unless
otherwise noted, all prompts contain 8 cases (a case = a question + a chain of
thoughts + an answer). Following Kojima et al. (2022), we add “Let’s think
step by step” before the reasoning chains for all prompting schemes to improve
the performance.

###  4.2 Main Results

Table 1:  Test set accuracy on GSM8K, MultiArith, and MathQA. All GPT-3 models use the davinci-002 engine (as of August 2022). $\dagger$ models are not publicly accessible, and the numbers are from their papers. Our performance gain (+blue) is computed over the handcrafted CoT used in Wei et al. (2022b), which is our primary baseline. Our methods substantially increase the performance over Wei et al. (2022b) and random CoT baselines.  |  | #Params | GSM8K | MultiArith | MathQA
---|---|---|---|---|---
Previous finetuning SOTA |  $\leq$175B | 57.0 | 60.5 | 37.4
Greedy decoding (Wei et al., 2022b)
LaMDA† (Thoppilan et al., 2022) | 137B | 17.1 | 51.8 | -
PaLM† (Chowdhery et al., 2022) | 540B | 58.1 | 94.7 | -
Minerva† (Lewkowycz et al., 2022) | 540B | 58.8 | - | -
GPT-3 | Handcrafted CoT | 175B | 48.1 | 90.8 | 30.1
| Random CoT | 175B | 49.7 | 89.5 | 34.8
| Complex CoT | 175B | 55.4 (+7.3) | 94.2 (+3.4) | 36.0 (+5.9)
Voting among multiple outputs (Wang et al., 2022b)
LaMDA† (Thoppilan et al., 2022) | 137B | 27.7 | 75.7 | -
PaLM† (Chowdhery et al., 2022) | 540B | 74.4 | 99.3 | -
Minerva† (Lewkowycz et al., 2022) | 540B | 78.5 | - | -
GPT-3 | Handcrafted CoT | 175B | 64.0 | 98.2 | 43.8
| Random CoT | 175B | 62.0 | 95.2 | 48.5
| Complex CoT | 175B | 71.5 | 97.3 | 49.5
| \+ Complexity-based Consistency | 175B | 72.6 (+8.6) | 98.7 (+0.5) |  50.2 (+6.4)
Table 2:  GSM8K validation set performance improvements broken down on to various design choices. More than half of the accuracy improvements can be attributed to our complexity-based prompting and output selection (indicated by $\dagger$).  Greedy Decoding | Acc. | Majority Vote | Acc.
---|---|---|---
CoT Original | 43.5 | CoT Original | 55.5
Add “Let’s think step by step” | 48.5 (+5.0) | Add “Let’s think step by step” | 61.0 (+5.5)
(Kojima et al., 2022) | and change “Q: ” to “Question:”
Use complex prompt† | 54.0 (+5.5) | Use complex prompt† | 67.0 (+6.0)
Change “Q: ” to “Question: ” | 58.0 (+4.0) | Voting within complex sample† | 71.0 (+4.0)
Figure 2:  Validation set performance. X-axis means reasoning steps and y-axis
means accuracy. More reasoning steps in prompts overall achieve higher
accuracy when prompts are in-distribution (left), noisily labeled (middle),
and out of distribution (right).

Overall Test Performance Table 1 shows the overall performance of models. We
consider two decoding strategies: (1) greedy decoding (the first block of
Table 1) and (2) majority vote (section 3.2; the second block of Table 1).
Note that PaLM and Minerva are more than three times larger than GPT-3, the
model we use to evaluate our method, and Minerva is additionally pretrained on
latex data. Therefore, they are by no means comparable to the methods based on
GPT-3. We nevertheless compare to them and show that our method can achieve
comparable performance.

Our main focus is to compare performance of differet prompts using the same
sized model model (GPT-3 175B). We consider three prompting schemes: (1).
Handcrafted CoT constructed originally by Wei et al. (2022b) then reused in
following-up works (Wang et al., 2022b; Kojima et al., 2022; Wang et al.,
2022a). (2). Random CoT: randomly drawing samples from the training set. GSM8K
and MathQA training data have reasoning chain annotations, so we directly use
them. MultiArith does not have reasoning annotations, so we randomly sample
eight training cases then annotate the chains manually. (3). Complex CoT. For
GSM8K and MathQA, we choose eight training cases with the most numbers of
reasoning steps; For MultiArith, we use the question length as the proxy for
complexity, and manually annotate reasoning chains for the eight training
cases with the longest questions. Complex prompt selection results in
substantially more reasoning steps: it averages 9.0 steps on GSM8K, while the
handcrafted and random schemes yield 3.4 and 2.8 steps respectively. The
trends are similar on the other two datasets. The handcrafted prompts uses the
same fixed prompt for all three datasets but the cases within the prompt does
not come from any of the datasets (so they are in a sense, out of
distribution). Complex prompts and random prompts all come from their
corresponding training sets (so these two are in a sense, in-distribution).

As Table 1 shows, our method achieves substantially better performance than
the baselines. Besides, our proposal of voting among complex chains
outperforms voting among all (last two lines in Table 1. Furthermore, our
performance using GPT-3 is close to PaLM and Minerva, two language models that
are more than three times larger than GPT-3 and are not publicly accessible.
These results directly demonstrate the effectiveness of our methods.

Performance Improvements Breakdown Improving prompting performance commonly
requires prompt engineering. A common scepticism or criticism towards
performance improvements on leaderboards like Table 1 is whether the accuracy
gains are from the proposed method or other independent engineering efforts.
Here we list all the efforts we made for leaderboard climbing on Table 2.
While techniques like adding “Let’s think step by step” (Kojima et al., 2022)
improves the accuracy, the performance gains can be primarily attributed to
complexity-based prompting, validating the effectiveness of our methods.

Consistent Performance Improvements in Different Prompt Distributions We
investigate the performance of our complexity-based prompting when the prompts
are: (1) from clean in-distribution training set (GSM8K); (2) from noisy
annotation (MathQA); (3) are transferred from another dataset (MultiArith).
Here as MultiArith does not have annotated reasoning chains, and their
questions are similar to the ones in GSM8K; we use (transfer) prompts from
GSM8K for MultiArith. Figure 2 shows that in general, more complex prompts
achieve better performance, and this trend is consistent in all the three
settings, except for one particular case on MultiArith.

Table 3:  Comparison to other prompt example selection schemes (validation accuracy). On GSM8K and MultiArith, complexity-based selection outperforms all the baselines. On MathQA, although retrieval performs better than complexity, it requires substantially more annotation.  | #Annotations | GSM8K | MultiArith | MathQA
---|---|---|---|---
Random | Few-shot (8) | 52.5 | 86.5 | 33.0
Centroid | Few-shot (8) | 52.0 | 92.0 | 32.0
Retrieval | Full training set ($\geq$10000) | 56.0 | 88.0 | 69.5
Complexity (ours) | Few-shot (8) | 58.5 | 93.0 | 42.5

Comparison to other Example Selection Schemes As we view the reasoning
complexity as the basis of a new example selection scheme, we compare it with
existing selection schemes. We consider: (1) random selection; (2) Centroid,
where we select examples whose question embeddings (produced by a pretrained
sentence encoder Reimers & Gurevych, 2019) are the closest to the embeddings
of all other questions, i.e., questions at the center part of the dataset. The
intuition is that centroid examples may be the most typical or representative
cases of a dataset; (3) Retrieval, where we retrieve questions from a training
set whose embeddings are closest test question measured in Euclidean distance.
Notably, there are important differences between retrieval and other methods:
retrieval uses different prompts for different test cases, while other methods
use fixed prompts for all. Therefore, the annotation cost of retrieval scales
with the size of the test set, and is usually about the full-training-set-
sized annotation (more than 10K cases), while others only require few-shot
annotation (in our cases, only 8 examples).

As shown in Table 3, complexity-based selection outperforms all other methods
on GSM8K and MultiArith. On MathQA, although retrieval-based selection
outperforms complexity-based selection, it has two importance restrictions
that we do not have: (1) as mentioned, retrieval requires substantially more
CoT annotation, while we only requires few-shot; (2) the performance of
retrieval is critically determined by how similar the test cases and the
training questions are to each other, and the similarity may not always hold.
We further note that on MathQA, many dev. questions are quite similar to their
retrieved training questions (some of them only have minor changes like “8
apples plus 9 bananas” to “10 apples plus 5 bananas” while the underlying
computations are the same). So in general, complexity-based prompting has the
advantage of good performance while being annotation efficient.

Direction of Generalization

Figure 3:  X-axis means reasoning steps of dev set cases and y-axis means
accuracy. We observe a very interesting generalization direction: complex
prompts improve the performance on simple cases while simple prompts cannot
help complex cases.

Intuitively, one may attribute the improvements of complexity-based prompting
to accuracy gains on complex test cases. Yet interestingly, our analysis
suggests the opposite. Fig. 3 compares the validation set accuracy of complex
and simple prompts, varying the number of reasoning steps in the _gold
annotation_. We observe a clear trend on both GSM8K and MathQA: complex
prompts perform on par with simple prompts on _hard_ cases, while achieving
more clear gains on cases with fewer number of reasoning steps. This finding
suggests that complexity-based prompting generalizes to simpler test cases. We
conjecture that this is because the reasoning capabilities elicited by complex
prompts may cover simple questions better. Further investigation into the
underlying mechanism is definitely interesting, and is left to future work.

###  4.3 Analysis

Table 4:  Alternative complexity measure: Q Len. = question length, F Len. =
formula length. More complex prompts consistently outperform simpler ones.

| Q Len. | GSM8K | F Len. | MathQA
---|---|---|---|---
Simple | 70 | 49.0 | 7.5 | 37.5
Mid | 226 | 51.0 | 55 | 33.5
Complex | 815 | 52.5 | 165 | 43.5

Figure 4: Relationship between confounders.

In this section, we develop in-depth analysis supporting our claims. All
experiments in this section are performed on validation sets. We first show
that the performance improvements with more reasoning complexity is consistent
in terms of: (1). different proxies for complexity and (2). different step
formatting. Then we show that the number of reasoning step is the most
prominent factor for performance improvements over its confounders. Finally,
we strengthen our conclusion of complexity-based consistency, and show that
the optimal performance is always achieved by majority voting over complex
chains, not simple chains.

Alternative Proxies for Complexity Complexity-based prompting is equally
applicable when the data does not come with reasoning chain annotations, as we
have already shown that selecting cases with longest questions also improves
performance (§4.2). In Table 4, we confirm that in addition to number of
steps, either using questions length or formula length as the measure of
complexity, the optimal performance is achieved with complex prompts. These
results mean that the effectiveness of complex prompts are consistent with
regard to the notion of complexity.

Table 5:  Sensitivity analysis on step formatting. Complex prompts consistently lead to better performance with regard to different step formatting.  | Linebreak “\n” | Period “.” | Explicit “step i” | Semicolon “;”
---|---|---|---|---
GSM8K-Complex | 58.5 | 54.5 | 52.0 | 54.0
GSM8K-Simple | 43.0 | 40.5 | 42.0 | 41.0
MathQA-Complex | 42.5 | 39.0 | 36.0 | 39.5
MathQA-Simple | 34.0 | 34.5 | 33.5 | 37.0

Sensitivity Analysis on Step Format A common concern with prompting is that
the performance can be sensitive to the format of the input (Shin et al.,
2020; Liu et al., 2022) and may change with input perturbations. Here we study
one important perturbation: the splitter of steps, which is an existing
concern of CoT-styled prompting in Rong ; Akyurek & Akyurek (2022). As
alternatives to the linebreak “\n” we use, we consider two more types of
splitters: (1). explicit phrases “step i” (2). two punctuation marks, period
“.” and semicolon “;” The performance is shown in Table 5. Although these
perturbations do have an influence on the performance, complex prompts
consistently lead to better performance with regard to different step
formatting.

Confounder Analysis

Table 6:  Confounder analysis. Although there exist confounders like number of cases or total prompt length, the number of reasoning step is the most prominent factor for performance gain.  More number of simple cases v.s. less but complex cases | GSM8K | MathQA
---|---|---
Total reasoning step | 72 | 72 | 45 | 45
Number of cases in prompt | 24 | 8 | 19 | 8
Per-case reasoning step | 3 | 9 | 2.25 | 5.625
Accuracy | 51 | 58.5 | 37.5 | 42.5
Most number of reasoning steps v.s. longest prompt | GSM8K | MathQA
Number of cases in prompt | 8 | 8 | 8 | 8
Length of prompt | 12.6k | 8.4k | 7.6k | 4.9k
Number of total reasoning step | 59 | 72 | 32 | 45
Accuracy | 57 | 58.5 | 31 | 42.5

All experiments so far keeps the number of instance to be 8 in all prompts.
Yet when choosing complex examples with more reasoning steps, we observe that
the following factors are correlated (also illustrated in Fig. 4): (1). when
per-case reasoning step increases (for example, in GSM8K we choose cases with
9 reasoning steps), the total number of step in the whole prompt also increase
(in GSM8K, we have 8 cases in the prompt, so there are 8 × 9 = 72 steps in
total). This might be compensated by using more number of simple cases (e.g.,
24 simple cases, each with 3 steps, can also make 72 steps in total). These
factors are shown in the upper part of Table 6. (2). when per-case step
increases, the full length of the prompt (= number of characters) also
increases, which may be compensated by longer (more characters) but less step
examples. These factors are shown in the lower part of Table 6. From the
accuracy results we can see that: (1). keeping full number of reasoning steps
the same, using more number of simple cases does not outperform less number of
complex cases; (2). longest prompts does not outperform complex prompts. This
means that despite the existence of confounders, the number of reasoning steps
per example is the most prominent factor for performance gain.

Voting among Complex Chains Outperforms Voting among All

Figure 5: Majority voting over top K complex / simple generated samples. The
optimal performance is achieved on selecting complex samples over simple
samples.

Now we analyze the properties of complexity-based consistency, which
generalizes the reasoning complexity selection criteria from the input space
(prompts) to the output space (sampled solutions from the language model).
Complexity-based consistency first sample $N$ reasoning chains from the model,
then take the majority answer voted from the top $K$ complex chains. Here we
set $N=50$, and control $K=10,20,30,40,50$. Note that when $K=50$ we recover
the original self-consistency (no complexity-based selection). As a further
comparison, we consider the other way around: instead of voting over top $K$
complex samples, we vote over top $K$ simple samples. As is shown in Fig. 5,
we see: (1). voting over simple samples always underperform full sample; ,
indicating this is not a correct direction for performance; (2). both datasets
achieve the best performance on some $K^{\star}<N$ with complex voting. These
results again, validate the choice of complex samples.

##  5 Conclusion

This paper proposes a new complexity-based instance selection scheme for
prompting language models to perform multi-step reasoning. In addition to
substantial performance improvements on math word reasoning tasks, our methods
exhibit multiple advantages such as being intuitive, annotation-efficient, and
robustly effective in different in-context learning settings. We hope this
work will open new research possibilities in prompting, language models, and
multi-step reasoning.

## References

  * Akyurek & Akyurek (2022) Ekin Akyurek and Afra Feyza Akyurek.  Notes on teaching gpt-3 adding numbers, 2022.  URL https://lingo.csail.mit.edu//blog/arithmetic_gpt3.
  * Amini et al. (2019) Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi.  MathQA: Towards interpretable math word problem solving with operation-based formalisms.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 2357–2367, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1245.  URL https://aclanthology.org/N19-1245.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Cheng et al. (2019) Jianpeng Cheng, Siva Reddy, Vijay Saraswat, and Mirella Lapata.  Learning an Executable Neural Semantic Parser.  _Computational Linguistics_ , 45(1):59–94, 03 2019.  ISSN 0891-2017.  doi: 10.1162/coli˙a˙00342.  URL https://doi.org/10.1162/coli_a_00342.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_ , 2021.
  * Creswell et al. (2022) Antonia Creswell, Murray Shanahan, and Irina Higgins.  Selection-inference: Exploiting large language models for interpretable logical reasoning.  _arXiv preprint arXiv:2205.09712_ , 2022.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  _arXiv preprint arXiv:2205.11916_ , 2022.
  * Lai et al. (2021) Yuxuan Lai, Chen Zhang, Yansong Feng, Quzhe Huang, and Dongyan Zhao.  Why machine reading comprehension models learn shortcuts?  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pp. 989–1002, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-acl.85.  URL https://aclanthology.org/2021.findings-acl.85.
  * Lazaridou et al. (2022) Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev.  Internet-augmented language models through few-shot prompting for open-domain question answering.  _arXiv preprint arXiv:2203.05115_ , 2022.
  * Lewkowycz et al. (2022) Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al.  Solving quantitative reasoning problems with language models.  _arXiv preprint arXiv:2206.14858_ , 2022.
  * Li & Liang (2021) Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 4582–4597, 2021.
  * Liang (2016) Percy Liang.  Learning executable semantic parsers for natural language understanding.  _Communications of the ACM_ , 59(9):68–76, 2016\.
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen.  What makes good in-context examples for GPT-3?  In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pp. 100–114, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.deelio-1.10.  URL https://aclanthology.org/2022.deelio-1.10.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _arXiv preprint arXiv:2107.13586_ , 2021.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 8086–8098, 2022\.
  * Mudrakarta et al. (2018) Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere.  Did the model understand the question?  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1896–1906, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-1176.  URL https://aclanthology.org/P18-1176.
  * (19) PromptBase.  Promptbase.  https://promptbase.com.
  * Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3982–3992, 2019.
  * Richardson & Sabharwal (2022) Kyle Richardson and Ashish Sabharwal.  Pushing the limits of rule reasoning in transformers through natural language satisfiability.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pp. 11209–11219, 2022.
  * (22) Frieda Rong.  Extrapolating to unnatural language processing with gpt-3’s in-context learning: The good, the bad, and the mysterious.  http://ai.stanford.edu/blog/in-context-learning/.
  * Roy & Roth (2015) Subhro Roy and Dan Roth.  Solving general arithmetic word problems.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pp. 1743–1752, Lisbon, Portugal, September 2015\. Association for Computational Linguistics.  doi: 10.18653/v1/D15-1202.  URL https://aclanthology.org/D15-1202.
  * Rubin et al. (2022) Ohad Rubin, Jonathan Herzig, and Jonathan Berant.  Learning to retrieve prompts for in-context learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2655–2671, Seattle, United States, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.naacl-main.191.  URL https://aclanthology.org/2022.naacl-main.191.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 4222–4235, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.346.  URL https://aclanthology.org/2020.emnlp-main.346.
  * Su et al. (2022) Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al.  Selective annotation makes language models better few-shot learners.  _arXiv preprint arXiv:2209.01975_ , 2022.
  * Sugawara et al. (2018) Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa.  What makes reading comprehension questions easier?  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 4208–4219, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1453.  URL https://aclanthology.org/D18-1453.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ , 2022.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 2153–2162, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1221.  URL https://aclanthology.org/D19-1221.
  * Wang et al. (2022a) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou.  Rationale-augmented ensembles in language models.  _arXiv preprint arXiv:2207.00747_ , 2022a.
  * Wang et al. (2022b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_ , 2022b.
  * Wei et al. (2021) Colin Wei, Sang Michael Xie, and Tengyu Ma.  Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning.  _Advances in Neural Information Processing Systems_ , 34:16158–16170, 2021.
  * Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_ , 2022a.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ , 2022b.
  * Xie et al. (2022) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma.  An explanation of in-context learning as implicit bayesian inference.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=RdJVFCHjUMI.
  * Yin et al. (2018) Pengcheng Yin, Chunting Zhou, Junxian He, and Graham Neubig.  StructVAE: Tree-structured latent variable models for semi-supervised semantic parsing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 754–765, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-1070.  URL https://aclanthology.org/P18-1070.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh.  Calibrate before use: Improving few-shot performance of language models.  In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 12697–12706. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/zhao21c.html.