# Asymmetric Faraday Effect in a Magnetophotonic Crystal

D.O. Ignatyeva<EMAIL_ADDRESS>Photonic and Quantum technologies
school, Faculty of Physics, Lomonosov Moscow State University, Leninskie gori,
119991 Moscow, Russia  Russian Quantum Center, 121205 Moscow, Russia
Institute of Physics and Technology, V.I. Vernadsky Crimean Federal
University, 295007 Simferopol, Crimea  T.V. Mikhailova  Institute of Physics
and Technology, V.I. Vernadsky Crimean Federal University, 295007 Simferopol,
Crimea  P.O. Kapralov  Russian Quantum Center, 121205 Moscow, Russia  S.D.
Lyashko  Institute of Physics and Technology, V.I. Vernadsky Crimean Federal
University, 295007 Simferopol, Crimea  V.N. Berzhansky  Institute of Physics
and Technology, V.I. Vernadsky Crimean Federal University, 295007 Simferopol,
Crimea  V.I. Belotelov  Institute of Physics and Technology, V.I. Vernadsky
Crimean Federal University, 295007 Simferopol, Crimea  Photonic and Quantum
technologies school, Faculty of Physics, Lomonosov Moscow State University,
Leninskie gori, 119991 Moscow, Russia  Russian Quantum Center, 121205 Moscow,
Russia

###### Abstract

It is widely known that the magneto-optical Faraday effect is linear in
magnetization and therefore the Faraday angles for the states with opposite
magnetizations are of opposite sign but equal in modulus. Here we
experimentally study propagation of light through a one-dimensional all-garnet
magnetophotonic crystal to demonstrate an asymmetric Faraday effect (AFE) for
which Faraday angles for opposite magnetic states differ not only in sign but
in the absolute value as well. AFE appears in the vicinity of the cavity
resonance for an oblique incidence of light which plane of polarization is
inclined to the incidence plane. Under proper incidence and polarization
angles the magnitude of AFE could be very large reaching 30$\%$ of the
absolute value of the Faraday effect. The effect originates from the
difference in Q-factors for p- and s- polarized cavity modes that breaks the
symmetry between the two opposite directions of polarization rotation. The
discovered AFE is of prime importance for nanoscale magnonics and
optomagnetism.

††preprint: APS/123-QED

The first phenomenon which demonstrated interaction between magnetism and
optics and established a basis for modern magnetophotonics is the Faraday
effect - transformation of polarization of light passing through a magnetized
material. It was discovered by Michael Faraday in 1846 [1]. The Faraday effect
is characterized by the Faraday rotation angle of the polarization plane of
light transmitted through a material along its magnetization. The internal
mechanism of the effect is in magnetic circular birefringence, i.e. the
magnetization-induced difference in the phase velocities of two circular
polarizations with opposite helicities [2]. The Faraday effect is known to be
odd in magnetization and thus has a nonreciprocal character [3, 4, 5, 6].
Initially, the Faraday’s discovery had a purely fundamental significance, but
a lot of applications of the magneto-optics for data storage, fiber-optic
communication lines (isolators, modulators, deflectors) [7, 8, 9], sensors
[10, 11, 12] and magnetometry [13, 14] have been suggested in the following
decades [15]. For further progress in this direction, it is necessary to find
some ways for enhancement and advanced control of the Faraday effect.

Fabrication of nanostructures is the most common solution that has been widely
used over some decades to increase the magneto-optical (MO) effects. It is
possible to design a system which exhibits an enhanced MO response by
concentrating the electromagnetic field of light inside magneto-optically
active components in the magnetophotonic crystals (MPCs) with localized states
[16, 17, 18, 19, 20, 21, 22], guided modes [23, 24, 25, 26], in
magnetoplasmonic structures [27, 28, 29, 30, 14, 31, 32, 33] and metasurfaces
[34, 35, 36, 37, 38]. Enhancement of the Faraday rotation most often occurs
due to an increase in purely magneto-optical contribution (multi-pass mode and
localization of light). Additionally, such structures may include the pure
optical effects of anisotropy or reducing the reflectivity to enhance the
conversion of polarization state (magnetoplasmonic structures [39, 40, 41] or
multilayer structures at oblique incidence [42, 43]). The combination of pure
magneto-optical and optical mechanisms is responsible for various types of the
Faraday effect enhancement [19, 31, 43, 44, 45, 46, 47].

Oblique incidence of light onto magnetic films and structures brings some
peculiarities of the MO effects [48] which are related to a lower reflectance
of p-polarized light with respect to s-polarization. Thus, the odd MO
intensity effect appears in the Faraday configuration [49]. In MPCs, the
Fresnel coefficients at oblique incidence set different interference
conditions for s- and p-polarized light. Thus the same MPC has a higher
optical Q-factor for s-polarized light and lower Q-factor for p-polarized
light [43, 45, 46, 47, 50]. An s-polarized wave demonstrates a higher
localization of electromagnetic field inside the MO active cavity and layers,
and, therefore, higher values of the Faraday rotation [43]. Furthermore, for
MPC with negligible absorption, a wave converted into s-polarized state due to
the Faraday effect can be trapped inside the MO active cavity at the
Brewster’s angle [46].

In all of the aforementioned studies, the Faraday rotation was odd in
magnetization, i.e., reversing the direction of the external magnetic field
changed its sign but not its magnitude (see Supplementary S1). In this work,
we discover that an additional spatial symmetry break might lead to an even
magnetization contribution to the magneto-optical Faraday rotation and,
therefore, the emergence of the asymmetric Faraday effect (AFE), which changes
not only its sign but also its value if the magnetic field is reversed. We
observe such peculiar Faraday effect by illuminating a microcavity MPC with
obliquely incident light of some intermediate polarization between pure s- and
p-ones. Breaking the oddness of the Faraday effect provides a unique
possibility to observe magneto-optical polarization rotation even in the
material with non-uniform magnetization direction and zero net magnetization.

Figure 1: Schematic representation of spatial symmetry origin of the oddness
of the Faraday effect and its violation in the case of symmetry breaking by a
director $L$.

(a) (b) (c)

Figure 2: (a) Schematic diagram of the investigated MPC on the GGG substrate.
Optical ellipticity (b, top), transmittance (b, bottom), and the asymmetric
Faraday effect (c) spectra of the MPC for the oblique illumination at
$\theta=63^{\circ}$, $\Psi=70^{\circ}$. Experimental data is presented by
color spheres, while simulations - by solid curves.

Let’s discuss how the spatial symmetry causes the oddness of the Faraday
rotation in the usual case and how it is modified if the spatial symmetry is
broken. Figure 1 illustrates the discussed phenomenon from the point of view
of what an observer might see in the real world and looking into a mirror. The
mirror is placed in the plane of the magnetization $\mathbf{M}$, see Fig. 1.
One might see in the mirror an oppositely magnetized medium with
$-\mathbf{M}$, as $\mathbf{M}$ is an axial vector. At the same time, the
director (a director is a quasi-vector with the two equivalent opposite
directions) $\mathrm{E}$ describing the light polarization is transformed by
usual reflection rules in the mirror, so that the angle between the reflected
$\mathrm{E}$ and the mirror is opposite to the angle between the real
$\mathrm{E}$ and the mirror. Thus, the magneto-optical polarization rotation
caused by passing through a magnetized medium that is observed in the mirror
world with $-\mathbf{M}$ magnetization is equal in magnitude and opposite in
sign to the one observed in the real world (Fig. 1):
$-\Phi(\mathbf{M})=\Phi(-\mathbf{M})$ and, therefore, the magneto-optical
rotation is odd in magnetization.

The situation changes if the spatial symmetry of the considered configuration
(including both the light and the structure) is broken with respect to the
reflection in the mirror placed along $\mathbf{M}$ due to the presence of the
preferred direction described by a director $\mathrm{L}$ (see Fig. 1, bottom
panel). Actually, such a preferred direction could arise due to the presence
of some vector describing the light or the structure in the considered
configuration, too. It might be considered a partial case of the symmetry
break described by a director; therefore, in the following we will consider a
more general case of a director. Since a director is transformed by usual
reflection rules, the configuration in the mirror characterized by the
reflected director $\mathrm{L}^{\mathrm{ref}}$ and polarization $\mathbf{E}$
differs from one observed in the real world, except for the case when
$\mathrm{L}$ is perpendicular or parallel to $\mathbf{E}$. By repeating the
considerations similar to the ones made above, one concludes that
$\Phi(-\mathbf{M},\mathrm{L}^{\mathrm{ref}})=-\Phi(\mathbf{M},\mathrm{L})$, so
the absolute values of $\Phi$ that are equal to each other refer to different
structures:
$|\Phi(\mathbf{M},\mathrm{L}^{\mathrm{ref}})|=|\Phi(\mathbf{M},\mathrm{L})|$.
At the same time, there are no limitations on the values of $\Phi$ for the
structures with the same $\mathrm{L}$; $|\Phi(-\mathbf{M},\mathrm{L})|$ and
$|\Phi(\mathbf{M},\mathrm{L})|$ may differ from each other. Consequently, we
can conclude that if $\mathrm{L}$ is oblique with respect to the polarization
$\mathbf{E}$, then the asymmetric Faraday effect (AFE) is allowed:
$|\Phi(\mathbf{M},L)|\neq|\Phi(-\mathbf{M}),L|$. However, if $\mathrm{L}$ is
parallel or perpendicular to $\mathbf{E}$, then the AFE is prohibited:
$|\Phi(\mathbf{M})|=|\Phi(-\mathbf{M})|$.

Such an impact of the spatial symmetry violation described by the director
$\mathrm{L}$ might also be understood as the arisen non-equivalence between
the clock- and counterclockwise rotation directions. In particular, such non-
equivalence reveals itself as the difference between the two opposite
directions of the magneto-optical rotations of linear polarization
corresponding to the opposite magnetizations (see also Supplementary S4 and
Fig. S4.2). Obviously, if $\mathrm{E}$ is not parallel or perpendicular to
$\mathrm{L}$, then its rotation in counter- and clockwise directions results
in different configurations described by a mutual orientation of $\mathrm{E}$
and $\mathrm{L}$. Actually, the non-equivalence of counter- and clockwise
rotations, as well as the non-equivalence of the ’real-world’ and ’mirror-
world’ configurations, are known as manifestations of the chirality
phenomenon. The considerations made above show that such a chiral-type impact
on the Faraday effect caused by the presence of the director $\mathrm{L}$
tilted with respect to the polarization $\mathbf{E}$ describing the structure
symmetry could arise in non-chiral nanostructures and films.

It is important that such a symmetry break can be introduced by the design of
a nanostructure (for example, by depositing a 1D grating on top of the
magnetic film, Supplementary S4) or by incident light itself: an oblique
incidence gives birth to the in-plane component of the light wavevector
$k_{\tau}$. In this case, $k_{\tau}$ determines the direction of $\mathrm{L}$
in the general consideration above. Such a configuration can be implemented in
almost any type of magneto-optical structure, including smooth films and
crystals (see Supplementary S4). In the present work, we chose this option to
introduce a symmetry break by the oblique incidence and demonstrate an
asymmetric behavior of the Faraday rotation in MPC.

The MPC consists of two magnetic Bragg mirrors containing pairs of magnetic
and nonmagnetic quarter-wave layers and a magnetic half-wave cavity layer
sandwiched in between the mirrors (Fig. 2a). It is illuminated by a linearly
polarized light incident on the sample at the incidence angle $\theta$.
Orientation of the light polarization is determined by the angle $\Psi$ formed
by $\mathrm{E}$ and the plane of light incidence: $\Psi=0^{\circ}$ for the
p-polarized light, and $\Psi=90^{\circ}$ for the s-polarized light (Fig. 2a).
On the other hand, $\Psi$ is also an angle between $k_{\tau}$ and
$\mathrm{E}$, and, consequently, it defines the spatial symmetry violation
described above. We will focus our attention on the intermediate configuration
$0<\Psi<90^{\circ}$, for which a spatial symmetry break takes place and AFE is
expected.

The asymmetric Faraday effect can be characterized by $\Delta\Phi$:

$\Delta\Phi=\Phi(+M_{z})-(-\Phi(-M_{z}))$ (1)

where $\Phi(+M_{z})$ and $\Phi(-M_{z})$ are the Faraday rotation angles at
opposite orientations of the magnetization vector $\mathbf{M}$ aligned along
$z$ axis. Here we determine the Faraday angles as the difference between the
total polarization rotation angles for the sample magnetized $\pm M_{z}$ and
not magnetized $M_{z}=0$ along $z$-axis, therefore, purely optical
polarization modification is omitted.

Experimental observation of the asymmetric Faraday effect was performed on the
all-garnet MPC (Fig. 2a). The MPC Bragg mirrors consist of six pairs of
diamagnetic $\mathrm{Sm_{3}Ga_{5}O_{12}}$ (SGG) and the MO active
ferrimagnetic $\mathrm{Bi_{2.97}Er_{0.03}Al_{0.5}Ga_{0.5}O_{12}}$ (BIG) garnet
layers ($h_{\mathrm{BIG}}=75$ nm, $h_{\mathrm{SGG}}=100$ nm) synthesized by
RF-magnetron sputtering on (111) $\mathrm{Gd_{3}Ga_{5}O_{12}}$ (GGG) substrate
[51, 52]. There is a cavity layer of BIG in between the Bragg mirrors: GGG /
[BIG / SGG]6 /2 BIG / [SGG / BIG]6 responsible for a cavity mode (see
Supplementary S2 and S3 for the details).

The Faraday rotation was experimentally found using a conventional scheme
where the transmitted light intensity $I_{\mathrm{out}}$ is measured in the
presence of polarizer and analyzer crossed by angle of $45^{\circ}$, so that
$I_{\mathrm{out}}=\frac{1}{2}I_{0}\cdot(1-\sin(2\mathrm{OR}+2\Phi)$, where
$\mathrm{OR}$ is a purely optical non-magnetic polarization rotation, and
$I_{0}$ is the intensity of the light transmitted through the MPC without a
polarizer (see Supplementary S3.3 for the details). Strictly speaking, this
formula is not valid in the presence of ellipticity in the sample. But still
it can be applied in the vicinity of the cavity resonance since the
ellipticity vanishes at the resonant wavelength (Fig. 2b).

We chose at first the polarization angle $\Psi=70^{\circ}$. Both experimental
measurements (color spheres in Fig. 2b) and numerical simulations performed
using the transfer matrix method 4×4 [53] (solid curves in Fig. 2b, see
Supplementary S2 for simulation details) show that at the oblique incidence
the cavity mode is observed at $\lambda=716$ nm for $\theta=63^{\circ}$. The
cavity resonance is accompanied with a pronounced peak of the Faraday rotation
$\Phi\sim 20^{\circ}$ (Fig. 2c), which gives a high magneto-optical quality
factor $Q_{\mathrm{MO}}=-2|\Phi|/\ln T=42.6^{\circ}$ and MO figure of merit
$\mathrm{MOFOM}=T\sin 2\Phi=20\%$ [21].

Figure 3: The dependence of the asymmetric Faraday effect on the initial light
polarization $\Psi$ for $\theta$ = 63∘. The inset shows the AFE spectra for
$\Psi=20^{\circ},30^{\circ},70^{\circ}$. Experimental data is presented by
color spheres, while simulations – by solid curve.

(a) (b)

Figure 4: Spectra of transmittance and Faraday rotation at the oblique
incidence of $\theta=63$ deg for (a) p- and (b) s-polarized light
(experiments).

At $\Psi=70^{\circ}$ a notable AFE characterized by $\Delta\Phi=5.5^{\circ}$
appears in the vicinity of the cavity resonance (Fig. 2c). Since at this
wavelength $\Phi\sim 20^{\circ}$, it means that AFE accounts about 30$\%$ of
the Faraday effect. Similar AFE is observed for oblique incidence of any
intermediate polarization of light and vanishes only for pure p- or
s-polarizations for which there is no spatial symmetry violation since in
those cases $k_{\tau}$ is either parallel or perpendicular to $\mathbf{E}$,
respectively (Fig. 3). The largest value of AFE appears at $\Psi=80^{\circ}$
and equals to $\Delta\Phi=7.5^{\circ}$, which is $38\%$ of the $\Phi$ value.

As we discussed above, in our experiments, the difference between
$|\Phi(+M_{z})|$ and $|\Phi(-M_{z})|$ for intermediate $\Psi$ values appears
due to the break of the spatial symmetry defined by $k_{\tau}$. This symmetry
break gives birth to the even in magnetization terms of the Faraday rotation:
$\Phi(M_{z})=\phi_{1}\frac{M_{z}}{M_{z0}}+\phi_{2}\left(\frac{M_{z}}{M_{z0}}\right)^{2}+...$
(where $M_{z0}$ is the saturation magnetization and $\phi_{j}$ are the
expansion coefficients). In the ’usual’ cases of normal incidence, or pure p-
or s-polarized light at the oblique incidence, which are used in most of the
magneto-optical experiments, $\phi_{2}=\phi_{2n}=0$, where $n$ is an integer
(see Supplementary S1.2).

As AFE is a nonlinear part of the Faraday rotation, one might expect to
observe high AFE in cases where the linear Faraday effect is large. From this
point of view, MPCs at oblique incidence are the ideal candidates to observe
AFE. On the one hand, they possess a high Faraday rotation of tens of degrees
(see blue lines in Figs. 4a,b). On the other hand, they exhibit a prominent
difference between the p- and s- polarized states. The origin of this
difference is the variation of the reflectivity of the Bragg mirrors
surrounding the cavity layer, leading to a difference in Q-factors. Figure 4
demonstrates a more than 5-fold difference in the transmittance (green lines)
and Faraday rotation (blue lines) for p-polarized (Fig. 4a) and s-polarized
(Figure 4b) light (see also Supplementary S3.2). Thus, MPC properties are very
sensitive to the mutual orientations of $\mathbf{E}$ and $\mathbf{k}_{\tau}$
vectors, which clearly differentiates $\Psi+\Phi$ and $\Psi-\Phi$ states
corresponding to the opposite rotation directions. This provides large values
of AFE. For the considered MPC, the second-order term contribution grows
significantly up to $\phi_{2}/\phi_{1}=0.19$. For a comparison, in a smooth
film of the same thickness, the quadratic term is very small:
$\phi_{2}/\phi_{1}=0.009$ (see Supplementary S4). Therefore, the role of the
MPC here is to enhance the AFE and make it easily measurable.

The Faraday effect is commonly used for the magneto-optical studies of the
magnetic materials and magnetization distributions. However, the obvious
difficulties arise if the magnetization distribution is non-uniform so that
the net magnetization in the area illuminated by the probe beam is zero:
$\int_{S_{\mathrm{beam}}}M_{z}(x,y)dxdy=0$. This situation takes place, for
example, for macroscopic static subwavelength domain structures [54] and
dynamic $M_{z}$ oscillations in ultrashort spin waves [55, 56] and spin-wave
resonances [24]. Although there is magnetization in each spatial point of the
material, the Faraday rotation is absent due to the zero average
$\left<M_{z}\right>$ in the illuminated area. The situation changes if the
spatial symmetry break is used to produce AFE, for example, as demonstrated in
the present work. Even if the net magnetization is zero, AFE makes it possible
to measure the polarization rotation arisen due to the non-zero
$\left<M_{z}^{2}\right>$. This opens a unique possibility to study various
types of materials with inhomogeneous static or dynamic spatial magnetization
distributions.

Therefore, here we demonstrate the asymmetric Faraday effect that arises if an
additional symmetry break with respect to the reflection in a mirror parallel
to magnetization is introduced to the magneto-optical system. We confirm this
prediction by studying the magneto-photonic crystal at oblique incidence,
which provides a unique modification of the Faraday effect’s symmetry and
magnetization dependence. Experimentally, a pronounced asymmetry of the
Faraday rotation spectra for oppositely directed magnetic fields reaching
$|\Phi(+M_{z})|-|\Phi(-M_{z})|\sim 30\%$ is obtained. Such behavior results
from a strong non-equivalence of the counter- and clockwise magneto-optical
rotations of linear polarization and brings the quadratic in the magnetization
term to the Faraday rotation spectra. For the magnetophotonic crystal, this
quadratic term has the same order of magnitude as the linear one, providing a
unique possibility to observe magneto-optical polarization rotation even in
the material with a non-uniform magnetization direction and zero net
magnetization. At the same time, we show that AFE is a quite general effect
that might be observed in various structures with a particular type of spatial
symmetry breaking. This is a key feature for optical studies of the static and
dynamic magnetic patterns with zero net magnetization and ultrashort spin
waves [55, 56].

###### Acknowledgements.

This work was financially supported by the Ministry of Science and Higher
Education of the Russian Federation, Megagrant project No. 075-15-2022-1108.
The authors thank A.N. Kalish and A.A.Voronov for the fruitful discussion.

## References

  * Faraday [1846] M. Faraday, On the magnetization of light and the illumination of magnetic lines of force, Philosophical Transactions of the Royal Society of London 136, 1 (1846).
  * Zvezdin and Kotov [1997] A. K. Zvezdin and V. A. Kotov,  _Modern magnetooptics and magnetooptical materials_ (CRC Press, 1997).
  * Krumme  _et al._ [1984] J.-P. Krumme, V. Doormann, and C.-P. Klages, Measurement of the magnetooptic properties of bismuth-substituted iron garnet films using piezobirefringent modulation, Applied optics 23, 1184 (1984).
  * Dionne and Allen [1994] G. F. Dionne and G. A. Allen, Molecular-orbital analysis of magneto-optical bi-o-fe hybrid excited states, Journal of Applied Physics 75, 6372 (1994).
  * Deb  _et al._ [2012] M. Deb, E. Popova, A. Fouchet, and N. Keller, Magneto-optical faraday spectroscopy of completely bismuth-substituted bi3fe5o12 garnet thin films, Journal of Physics D: Applied Physics 45, 455001 (2012).
  * Caloz  _et al._ [2018] C. Caloz, A. Alu, S. Tretyakov, D. Sounas, K. Achouri, and Z.-L. Deck-Léger, Electromagnetic nonreciprocity, Physical Review Applied 10, 047001 (2018).
  * Karki  _et al._ [2019] D. Karki, R. El-Ganainy, and M. Levy, Toward high-performing topological edge-state optical isolators, Physical Review Applied 11, 034045 (2019).
  * Ho  _et al._ [2018] K.-S. Ho, S.-J. Im, J.-S. Pae, C.-S. Ri, Y.-H. Han, and J. Herrmann, Switchable plasmonic routers controlled by external magnetic fields by using magneto-plasmonic waveguides, Scientific reports 8, 1 (2018).
  * Śmigaj  _et al._ [2010] W. Śmigaj, J. Romero-Vivas, B. Gralak, L. Magdenko, B. Dagens, and M. Vanwolleghem, Magneto-optical circulator designed for operation in a uniform external magnetic field, Optics letters 35, 568 (2010).
  * Maccaferri  _et al._ [2015a] N. Maccaferri, K. E. Gregorczyk, T. V. De Oliveira, M. Kataja, S. Van Dijken, Z. Pirzadeh, A. Dmitriev, J. Åkerman, M. Knez, and P. Vavassori, Ultrasensitive and label-free molecular-level detection enabled by light phase control in magnetoplasmonic nanoantennas, Nature communications 6, 1 (2015a).
  * Ignatyeva  _et al._ [2016] D. Ignatyeva, P. Kapralov, G. Knyazev, S. Sekatskii, G. Dietler, M. Nur-E-Alam, M. Vasiliev, K. Alameh, and V. Belotelov, High-q surface modes in photonic crystal/iron garnet film heterostructures for sensor applications, JETP letters 104, 679 (2016).
  * Borovkova  _et al._ [2020] O. Borovkova, D. Ignatyeva, S. Sekatskii, A. Karabchevsky, and V. Belotelov, High-q surface electromagnetic wave resonance excitation in magnetophotonic crystals for supersensitive detection of weak light absorption in the near-infrared, Photonics Research 8, 57 (2020).
  * Ignatyeva  _et al._ [2021] D. O. Ignatyeva, G. A. Knyazev, A. N. Kalish, A. I. Chernov, and V. I. Belotelov, Vector magneto-optical magnetometer based on resonant all-dielectric gratings with highly anisotropic iron garnet films, Journal of Physics D: Applied Physics 54, 295001 (2021).
  * Knyazev  _et al._ [2018] G. A. Knyazev, P. O. Kapralov, N. A. Gusev, A. N. Kalish, P. M. Vetoshko, S. A. Dagesyan, A. N. Shaposhnikov, A. R. Prokopov, V. N. Berzhansky, A. K. Zvezdin, _et al._ , Magnetoplasmonic crystals for highly sensitive magnetometry, ACS Photonics 5, 4951 (2018).
  * Inoue  _et al._ [2013] M. Inoue, M. Levy, and A. V. Baryshev,  _Magnetophotonics: From theory to applications_ (Springer Science & Business Media, 2013).
  * Inoue  _et al._ [2006] M. Inoue, R. Fujikawa, A. Baryshev, A. Khanikaev, P. Lim, H. Uchida, O. Aktsipetrov, A. Fedyanin, T. Murzina, and A. Granovsky, Magnetophotonic crystals, Journal of Physics D: Applied Physics 39, R151 (2006).
  * Zhdanov  _et al._ [2006] A. Zhdanov, A. Fedyanin, O. Aktsipetrov, D. Kobayashi, H. Uchida, and M. Inoue, Enhancement of faraday rotation at photonic-band-gap edge in garnet-based magnetophotonic crystals, Journal of Magnetism and Magnetic Materials 300, e253 (2006).
  * Lyubchanskii  _et al._ [2003] I. Lyubchanskii, N. Dadoenkova, M. Lyubchanskii, E. Shapovalov, and T. Rasing, Magnetic photonic crystals, Journal of Physics D: Applied Physics 36, R277 (2003).
  * Maccaferri  _et al._ [2020] N. Maccaferri, I. Zubritskaya, I. Razdolski, I.-A. Chioar, V. Belotelov, V. Kapaklis, P. M. Oppeneer, and A. Dmitriev, Nanoscale magnetophotonics, Journal of Applied Physics 127, 080903 (2020).
  * Merzlikin  _et al._ [2007] A. Merzlikin, A. Vinogradov, A. Dorofeenko, M. Inoue, M. Levy, and A. Granovsky, Controllable tamm states in magnetophotonic crystal, Physica B: Condensed Matter 394, 277 (2007).
  * Mikhailova  _et al._ [2018] T. Mikhailova, V. Berzhansky, A. Shaposhnikov, A. Karavainikov, A. Prokopov, Y. M. Kharchenko, I. Lukienko, O. Miloslavskaya, and M. Kharchenko, Optimization of one-dimensional photonic crystals with double layer magneto-active defect, Optical Materials 78, 521 (2018).
  * Yu  _et al._ [2020] G. Yu, H. Yang, J. Fu, X. Zhang, and R. Cao, Nonreciprocal transmission using a multilayer magneto-optical dispersive material with defect, Journal of Electromagnetic Waves and Applications 34, 1400 (2020).
  * Voronov  _et al._ [2020] A. A. Voronov, D. Karki, D. O. Ignatyeva, M. A. Kozhaev, M. Levy, and V. I. Belotelov, Magneto-optics of subwavelength all-dielectric gratings, Optics Express 28, 17988 (2020).
  * Chernov  _et al._ [2020] A. I. Chernov, M. A. Kozhaev, D. O. Ignatyeva, E. N. Beginin, A. V. Sadovnikov, A. A. Voronov, D. Karki, M. Levy, and V. I. Belotelov, All-dielectric nanophotonics enables tunable excitation of the exchange spin waves, Nano letters 20, 5259 (2020).
  * Royer  _et al._ [2020] F. Royer, B. Varghese, E. Gamet, S. Neveu, Y. Jourlin, and D. Jamon, Enhancement of both faraday and kerr effects with an all-dielectric grating based on a magneto-optical nanocomposite material, ACS omega 5, 2886 (2020).
  * Bsawmaii  _et al._ [2020] L. Bsawmaii, E. Gamet, F. Royer, S. Neveu, and D. Jamon, Longitudinal magneto-optical effect enhancement with high transmission through a 1d all-dielectric resonant guided mode grating, Optics express 28, 8436 (2020).
  * Belotelov and Zvezdin [2006] V. Belotelov and A. Zvezdin, Magnetooptics and extraordinary transmission of the perforated metallic films magnetized in polar geometry, Journal of Magnetism and Magnetic Materials 300, e260 (2006).
  * López-Ortega  _et al._ [2020] A. López-Ortega, M. Zapata-Herrera, N. Maccaferri, M. Pancaldi, M. Garcia, A. Chuvilin, and P. Vavassori, Enhanced magnetic modulation of light polarization exploiting hybridization with multipolar dark plasmons in magnetoplasmonic nanocavities, Light: Science & Applications 9, 1 (2020).
  * Kalish  _et al._ [2018] A. N. Kalish, R. S. Komarov, M. A. Kozhaev, V. G. Achanta, S. A. Dagesyan, A. N. Shaposhnikov, A. R. Prokopov, V. N. Berzhansky, A. K. Zvezdin, and V. I. Belotelov, Magnetoplasmonic quasicrystals: an approach for multiband magneto-optical response, Optica 5, 617 (2018).
  * Maccaferri  _et al._ [2015b] N. Maccaferri, X. Inchausti, A. García-Martín, J. C. Cuevas, D. Tripathy, A. O. Adeyeye, and P. Vavassori, Resonant enhancement of magneto-optical activity induced by surface plasmon polariton modes coupling in 2d magnetoplasmonic crystals, ACS Photonics 2, 1769 (2015b).
  * Armelles  _et al._ [2013] G. Armelles, A. Cebollada, A. García-Martín, and M. U. González, Magnetoplasmonics: magnetoplasmonics: combining magnetic and plasmonic functionalities, Advanced Optical Materials 1, 2 (2013).
  * Lodewijks  _et al._ [2014] K. Lodewijks, N. Maccaferri, T. Pakizeh, R. K. Dumas, I. Zubritskaya, J. Åkerman, P. Vavassori, and A. Dmitriev, Magnetoplasmonic design rules for active magneto-optics, Nano letters 14, 7207 (2014).
  * Khramova  _et al._ [2019] A. E. Khramova, D. O. Ignatyeva, M. A. Kozhaev, S. A. Dagesyan, V. N. Berzhansky, A. N. Shaposhnikov, S. V. Tomilin, and V. I. Belotelov, Resonances of the magneto-optical intensity effect mediated by interaction of different modes in a hybrid magnetoplasmonic heterostructure with gold nanoparticles, Optics express 27, 33170 (2019).
  * Ignatyeva  _et al._ [2020] D. O. Ignatyeva, D. Karki, A. A. Voronov, M. A. Kozhaev, D. M. Krichevsky, A. I. Chernov, M. Levy, and V. I. Belotelov, All-dielectric magnetic metasurface for advanced light control in dual polarizations combined with high-q resonances, Nature communications 11, 1 (2020).
  * Qin  _et al._ [2020] J. Qin, L. Deng, T. Kang, L. Nie, H. Feng, H. Wang, R. Yang, X. Liang, T. Tang, J. Shen, _et al._ , Switching the optical chirality in magnetoplasmonic metasurfaces using applied magnetic fields, ACS nano 14, 2808 (2020).
  * Bi  _et al._ [2021] Y. Bi, L. Huang, X. Li, and Y. Wang, Magnetically controllable metasurface and its application, Frontiers of Optoelectronics , 1 (2021).
  * Yang  _et al._ [2022] W. Yang, Q. Liu, H. Wang, Y. Chen, R. Yang, S. Xia, Y. Luo, L. Deng, J. Qin, H. Duan, _et al._ , Observation of optical gyromagnetic properties in a magneto-plasmonic metamaterial, Nature communications 13, 1 (2022).
  * Xia  _et al._ [2022] S. Xia, D. O. Ignatyeva, Q. Liu, H. Wang, W. Yang, J. Qin, Y. Chen, H. Duan, Y. Luo, O. Novák, _et al._ , Circular displacement current induced anomalous magneto-optical effects in high index mie resonators, Laser & Photonics Reviews 16, 2200067 (2022).
  * Fujikawa  _et al._ [2008] R. Fujikawa, A. Baryshev, J. Kim, H. Uchida, and M. Inoue, Contribution of the surface plasmon resonance to optical and magneto-optical properties of a bi: Yig-au nanostructure, Journal of Applied Physics 103, 07D301 (2008).
  * Baryshev  _et al._ [2013] A. Baryshev, H. Uchida, and M. Inoue, Peculiarities of plasmon-modified magneto-optical response of gold–garnet structures, JOSA B 30, 2371 (2013).
  * Fan  _et al._ [2019] B. Fan, M. E. Nasir, L. H. Nicholls, A. V. Zayats, and V. A. Podolskiy, Magneto-optical metamaterials: Nonreciprocal transmission and faraday effect enhancement, Advanced Optical Materials 7, 1801420 (2019).
  * Baek  _et al._ [2011] S. Baek, A. V. Baryshev, and M. Inoue, Multiple bragg diffraction in magnetophotonic crystals, Applied Physics Letters 98, 101111 (2011).
  * Grishin and Khartsev [2019] A. M. Grishin and S. Khartsev, Waveguiding in all-garnet heteroepitaxial magneto-optical photonic crystals, JETP Letters 109, 83 (2019).
  * Tomilin  _et al._ [2020] S. Tomilin, V. Berzhansky, A. Shaposhnikov, A. Prokopov, A. Karavaynikov, E. Milyukova, T. Mikhailova, and O. Tomilina, Vertical displacement of the magnetooptical hysteresis loop in the magnetoplasmonic nanocomposite, Physics of the Solid State 62, 144 (2020).
  * Vasiliev  _et al._ [2006] M. Vasiliev, V. I. Belotelov, A. N. Kalish, V. A. Kotov, A. K. Zvezdin, and K. Alameh, Effect of oblique light incidence on magnetooptical properties of one-dimensional photonic crystals, IEEE transactions on magnetics 42, 382 (2006).
  * Ignatyeva and Belotelov [2020] D. Ignatyeva and V. Belotelov, Bound states in the continuum enable modulation of light intensity in the faraday configuration, Optics Letters 45, 6422 (2020).
  * Lyubchanskii  _et al._ [2006] I. L. Lyubchanskii, N. N. Dadoenkova, M. I. Lyubchanskii, E. A. Shapovalov, A. E. Zabolotin, Y. P. Lee, and T. Rasing, Response of two-defect magnetic photonic crystals to oblique incidence of light: Effect of defect layer variation, Journal of Applied Physics 100, 096110 (2006).
  * Schäfer  _et al._ [2021] R. Schäfer, P. M. Oppeneer, A. V. Ognev, A. S. Samardak, and I. V. Soldatov, Analyzer-free, intensity-based, wide-field magneto-optical microscopy, Applied Physics Reviews 8 (2021).
  * Krinchik  _et al._ [1978] G. Krinchik, E. Chepurova, and S. V. Ehgamov, Magneto-optical intensity effects in ferromagnetic metals and dielectrics.[bicavfe/sub 5/o/sub 12/], Zh. Eksp. Teor. Fiz.;(USSR) 74 (1978).
  * Wu  _et al._ [2021] B.-R. Wu, J.-H. Yang, P. S. Pankin, C.-H. Huang, W. Lee, D. N. Maksimov, I. V. Timofeev, and K.-P. Chen, Quasi-bound states in the continuum with temperature-tunable q factors and critical coupling point at brewster’s angle, Laser & Photonics Reviews 15, 2000290 (2021).
  * Ansari  _et al._ [2012] N. Ansari, S. Khartsev, and A. Grishin, Multicolor filter all-garnet magneto-optical photonic crystals, Optics letters 37, 3552 (2012).
  * Passler and Paarmann [2017] N. C. Passler and A. Paarmann, Generalized 4$\times$ 4 matrix formalism for light propagation in anisotropic stratified media: study of surface phonon polaritons in polar dielectric heterostructures, JOSA B 34, 2128 (2017).
  * Li [2003] L. Li, Fourier modal method for crossed anisotropic gratings with arbitrary permittivity and permeability tensors, Journal of Optics A: Pure and Applied Optics 5, 345 (2003).
  * Danneau  _et al._ [2002] R. Danneau, P. Warin, J. Attané, I. Petej, C. Beigné, C. Fermon, O. Klein, A. Marty, F. Ott, Y. Samson, and M. Viret, Individual domain wall resistance in submicron ferromagnetic structures, Physical review letters 88, 157201 (2002).
  * Dieterle  _et al._ [2019] G. Dieterle, J. Förster, H. Stoll, A. Semisalova, S. Finizio, A. Gangwar, M. Weigand, M. Noske, M. Fähnle, I. Bykova, J. Grafe, D. Bozhko, H. Musiienko-Shmarova, V. Tiberkevich, A. Slavin, C. Back, J. Raabe, G. Schutz, and S. Wintz, Coherent excitation of heterosymmetric spin waves with ultrashort wavelengths, Physical review letters 122, 117202 (2019).
  * Sluka  _et al._ [2019] V. Sluka, T. Schneider, R. A. Gallardo, A. Kákay, M. Weigand, T. Warnatz, R. Mattheis, A. Roldán-Molina, P. Landeros, V. Tiberkevich, _et al._ , Emission and propagation of 1d and 2d spin waves with nanoscale wavelengths in anisotropic spin textures, Nature nanotechnology 14, 328 (2019).


#  PoE: Process of Elimination for Multiple Choice Reasoning

Chenkai Ma1 Xinya Du2
1School of Computer Science and Engineering,
University of Electronic Science and Technology of China
2Deparment of Computer Science, University of Texas at Dallas
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Language models (LMs) are capable of conducting in-context learning for
multiple choice reasoning tasks, but the options in these tasks are treated
equally. As humans often first eliminate wrong options before picking the
final correct answer, we argue a similar two-step strategy can make LMs better
at these tasks. To this end, we present the Process of Elimination (PoE), a
two-step scoring method. In the first step, PoE scores each option, and
eliminates seemingly wrong options. In the second step, PoE masks these wrong
options, and makes the final prediction from the remaining options. Zero-shot
experiments on 8 reasoning tasks illustrate the effectiveness of PoE, and a
following analysis finds our method to be especially performant on logical
reasoning tasks. We further analyze the effect of masks, and show that PoE
applies to few-shot settings and large language models (LLMs) like
ChatGPT.111Code is available at https://github.com/KasMasVan/PoE.

##  1 Introduction

> How often have I said to you that when you have eliminated the impossible
> whatever remains, however improbable, must be the truth? (Doyle, 1890)

In natural language processing, many reasoning tasks are multiple choice-
based, in which a model chooses the best option from several options, given a
question. Current LMs exhibit remarkable performance on diverse reasoning
tasks, like commonsense reasoning (Wang et al., 2023a; Holtzman et al., 2021),
logical reasoning (Ye et al., 2023), and arithmetic reasoning (Shum et al.,
2023).

There are two types of in-context learning methods for multiple choice
reasoning: scoring and prompting. Given the question, scoring methods score
each option, and select the highest-scored option. One common score is
language modeling likelihood ($P(y_{i}|x)$, Brown et al., 2020), and there are
also many alternatives (Zhao et al., 2021; Holtzman et al., 2021; Fei et al.,
2023; Min et al., 2022; Malkin et al., 2022; Robinson and Wingate, 2023; Lyu
et al., 2023). Tailored for powerful LLMs (Chowdhery et al., 2022; Chen et
al., 2021), prompting methods (Wei et al., 2022; Wang et al., 2023b; Kojima et
al., 2022; Besta et al., 2023; Yao et al., 2023) append all options to the
question, and prompt the model to generate a sequence, then extract the option
from the sequence.

Figure 1: An illustration of the Process of Elimination (PoE) for a multiple
choice question. In the first step, PoE eliminates some wrong options. In the
second step, it enforces the masks, and makes the final prediction.

One limitation of these two types of approaches is that they both treat each
option equally, i.e., they either consider each option independently, or
consider all options jointly. On the contrary, when humans solve a multiple
choice reasoning task, they often eliminate some wrong options, then choose
from the remaining ones. This so-called process of elimination is widely used
in college exams and can be stronger than usual Gold style learning (Freivalds
et al., 2002). Therefore, we assume that language models can similarly benefit
from this elimination process, i.e., eliminating wrong options and choosing
the best option are two types of reasoning that can be disentangled into two
steps.

To this end, we present the Process of Elimination (PoE), a two-step scoring
method for multiple choice reasoning, as shown in Figure 1. In the first step,
PoE scores each option, then eliminates some wrong options based on their
scores. In the second step, it masks these wrong options, then chooses the
best one from the remaining options. We conduct experiments on 8 reasoning
tasks that cover diverse domains. PoE achieves the best zero-shot performance
on most tasks. A following analysis shows that it favors logical reasoning
tasks. We also measure the effect of masks, and find our method applicable to
few-shot settings and compatible with LLMs like ChatGPT (Ouyang et al., 2022).

Our contributions are twofold: (1) We present PoE, a two-step scoring method
for multiple choice reasoning; (2) Through comprehensive experiments and
analysis, we demonstrate the effectiveness and generalizability of PoE.

##  2 Method

Problem Setting. A multiple choice reasoning task instance includes a question
$x$, several options $Y=\\{y_{1},...,y_{n}\\}$, and the correct option $y$.
There are two kinds of in-context learning approaches to this problem: scoring
and prompting.

Scoring uses an LM to compute a score for each option $y_{i}$, and chooses the
highest-scored option:

$s_{i}=\text{score}(x,y_{i}),$ (1) $\hat{y}=\arg\max_{i}s_{i}.$ (2)

A commonly-used score is language modeling likelihood ($P(y_{i}|x)$, Brown et
al., 2020), More recent scores include average log probability (Brown et al.,
2020), calibrated log probability (Holtzman et al., 2021), and channel (Min et
al., 2022). As shown in Equation 1, most scoring methods consider each option
in isolation, except multiple choice prompting (Robinson and Wingate, 2023).

In contrast, prompting methods provide all the options in the input (Wei et
al., 2022; Wang et al., 2023b; Kojima et al., 2022). The model then generates
raw output from the input. Finally, these methods extract the option from the
raw output:

$\text{raw output}=\text{generate}(x,Y),$ (3)
$\hat{y}=\text{extract}(\text{raw output}).$ (4)

PoE. Our method is a two-step scoring method that considers all options but
treats them differently. We also implement a prompting-based variation of PoE
for LLM (Section 5).

The first step of PoE is elimination, in which it eliminates some wrong
options. PoE starts by scoring each option. Then, unlike common scoring
methods that choose the highest-scored option, it uses the scores to eliminate
some options with low scores. In particular, PoE computes the average score of
all options, and eliminates options whose score is below average, i.e.,
$Y_{\text{wrong}}$:

$Y_{\text{wrong}}=\\{y_{i}|s_{i}<\text{avg}(s_{1},...,s_{i})\\}.$ (5)

The intuition behind this elimination strategy ("Below Average") is that the
scores of wrong options deviate from others, and are thus easy to identify,
which we verify in Appendix B.2. We compare other elimination strategies in
Section 5.2.

The second step of PoE is prediction, which chooses the best answer that is
not in $Y_{\text{wrong}}$. Specifically, PoE computes binary masks $m_{i}$ for
options:

$m_{i}=\begin{cases}0,&\text{if}\ y_{i}\in Y_{\text{wrong}}\\\
1,&\text{otherwise}\\\ \end{cases}$ (6) $\text{mask}=[m_{1},...,m_{n}].$ (7)

For these options, PoE enforces the masks.222We don’t remove wrong options
because that requires extra work like relabeling remaining options (e.g.,
”ADE” to ”ABC”), but brings negligible difference to performance. In
particular (shown in Figure 1), it uses a template $T$ to first wrap the
question with all options and their symbols like "A", then append an
instruction to the question which asks the model to neglect masked options,
and finally replace eliminated options with a special text sequence "[MASK]":

$x_{\text{mask}}=T(x,Y,\text{mask}).$ (8)

Then, PoE scores each option, and chooses the highest-scored option, during
which the scores of eliminated options are set to negative infinity:

$s_{\text{mask},i}=\begin{cases}\text{score}(x_{\text{mask}},y_{i}),&\text{if}\
y_{i}\notin Y_{\text{wrong}}\\\ -\text{inf},&\text{otherwise}\\\ \end{cases}$
(9) $\hat{y}=\arg\max_{i}s_{\text{mask},i}.$ (10)

Task | Method
---|---
LM | AVG | Calibration | Channel | MCP | PoE
ANLI | 38.62.9 | 38.03.4 | 37.23.5 | 36.02.7 | 57.83.0 |  55.02.4
CQA | 64.45.9 | 54.21.3 | 69.64.8 | 44.65.6 |  87.22.6 | 89.22.2
SIQA | 56.04.5 | 62.61.8 | 57.66.9 | 39.85.9 |  79.05.6 | 82.06.7
LD |  48.82.7 | 45.84.5 | 39.24.0 | 20.83.6 | 39.83.7 | 53.67.0
DQA | 45.86.6 | 51.82.0 | 48.02.9 | 39.66.6 | 67.81.9 |  67.42.5
CC | 44.80.8 | 51.80.8 | 54.40.5 | 44.00.7 |  60.21.5 | 72.20.8
SS | 39.02.5 | 40.61.8 | 43.41.7 | 26.61.7 |  74.00.7 | 76.60.9
SIT | 24.81.9 | 20.65.3 | 21.03.9 | 17.04.6 | 25.42.9 |  25.24.0
Table 1: Accuracy scores (with standard deviation) on 8 tasks. The best scores
are boldfaced, and the second-best scores are underlined. LM refers to
language modeling, AVG refers to average language modeling, MCP refers to
multiple choice prompting. Our method (PoE) achieves the best or comparable
performance on all tasks.

##  3 Experiment Setup

Data. We consider 8 multiple choice reasoning tasks to cover diverse domains.
We include three traditional reasoning tasks: ANLI (Nie et al., 2020),
CommonsenseQA (CQA, Talmor et al., 2019), and Social IQa (SIQA, Sap et al.,
2019). We also include five BIG-bench tasks (Srivastava et al., 2023), with
the first two from BIG-Bench Hard (Suzgun et al., 2023): Logical Deduction
(LD), Disambiguation_QA (DQA), Conceptual Combinations (CC), Strange Stories
(SS), and Symbol Interpretation Task (SIT). For all tasks, we use their test
sets if available, otherwise their development sets. To reduce cost, we sample
100 instances from each task. We present more task information and pre-
processing in Appendix A.

Model. We use FLAN-T5-XL (3B) (Chung et al., 2022) in both steps of PoE,
because it is an economical and performant instruction-tuned model. In Section
5, we also apply PoE to LLMs like ChatGPT (Ouyang et al., 2022).333We do not
use recent larger instruction-following models based on LLaMA (Touvron et al.,
2023), because 1) they are not optimized for traditional NLP tasks, and 2)
they evolve quickly, which makes it hard for a fair comparison.

Methods. We consider 5 scoring baselines: language modeling (LM, baseline in
Zhao et al., 2021), average language modeling (AVG, Brown et al., 2020),
calibration (Holtzman et al., 2021), channel (Min et al., 2022), and multiple
choice prompting (MCP, Robinson and Wingate, 2023). For PoE, we use MCP in
both steps. We discuss other possible scoring methods and elimination
strategies for PoE in Section 5.2, and present input and output samples for
all methods in Appendix C.444We do not include prompting baselines like Chain-
of-Thought Prompting (Wei et al., 2022), because they are tailored for very
large (e.g., 175B) models.

Settings. We evaluate PoE in the zero-shot setting. We use accuracy as the
metric, and average results over 5 random seeds. We consider other settings in
Section 5.3 and 5.4.

##  4 Results

In Table 1, we compare PoE with other baselines on 8 reasoning tasks, and
present the accuracy and standard deviation. Our method achieves the best or
second-best performance on all 8 tasks. We notice that multiple choice
prompting (MCP) is a strong baseline that consistently outperforms other
baselines. Nevertheless, PoE beats MCP on 5 tasks, and is comparable on the
remaining three. Since PoE uses MCP in both steps, the result demonstrates the
effectiveness of the former, especially elimination. In addition, PoE is
exceptionally performant on CC, beating the second-best method by 12%
absolute. This corroborates our hypothesis on two types of reasoning for
multiple choice tasks, because while MCP sometimes fails to simultaneously
eliminate wrong options and choose the right option, PoE disentangles the two
jobs in two steps, and thus achieves better performance. We present a case
study in Appendix B.2.

##  5 Analysis

###  5.1 When Is PoE Better than MCP?

In Table 1, we find MCP to be the best baseline, and sometimes even beats PoE.
This finding leads us to wonder: when (on which tasks) can we expect PoE to
dramatically outperform MCP?

We start by calculating their performance gaps (PoE \- MCP) on all 8 tasks. We
find the gaps on LD (13.8%) and CC (12%) are much larger than those of other
tasks (less than 3%). Furthermore, we find both LD and CC are logical
reasoning tasks, while other tasks rely more on commonsense or social
reasoning.555SIT is another logical reasoning task, but it also requires
visual reasoning, which is not ideal for language models. We thus hypothesize
that PoE is better than MCP on tasks that mainly require logical reasoning,
while not being consistently performant on social and commonsense reasoning.
One explanation is that logical reasoning is a general skill, whereas social
or commonsense reasoning relies on specific knowledge, some of which may be
absent in certain language models.

To verify this hypothesis, we compare PoE and MCP in the zero-shot setting on
8 new tasks from BIG-Bench, including 4 logical reasoning tasks: Logical
Arguments (LA), Identify Math Theorems (IMT), Code Description (CLD),
Reasoning about Colored Objects (RACO); and 4 control tasks that are based on
commonsense or social reasoning: Counterfactual Conditionals (CAI), The
Essential, the Excessive, and the Extraneous (EIE), RiddleSense (RS), Identify
Odd Metaphor (IOM). We present task information in Appendix A.

Task | Method
---|---
MCP | PoE |  PoE \- MCP
LA | 50.0 | 68.8 | +18.8
IMT | 34.0 | 47.2 | +13.2
CLD | 67.2 | 75.9 | +8.7
RACO | 53.8 | 60.6 | +6.8
CAI | 84.1 | 81.8 | -2.3
EIE | 25.0 | 19.1 | -5.9
RS | 55.1 | 49.0 | -6.1
IOM | 56.2 | 50.0 | -6.2
Table 2: Comparison of MCP and PoE accuracy scores on 8 new tasks. The top 4
tasks are logical reasoning tasks. PoE largely outperforms MCP on 4 logical
reasoning tasks, and underperforms MCP on other 4 tasks.

As shown in Table 2, we find PoE consistently and dramatically outperforms MCP
on all 4 logical reasoning tasks (top 4 tasks). We find the largest
performance gap on LA (18.8%), a task of GRE-style logical questions. Human
test-takers commonly use elimination-based methods to solve such GRE
questions, which further supports the motivation of PoE. We also find it
underperforms MCP on 4 control tasks (bottom 4 tasks), which means our method
is not consistent on commonsense or social reasoning tasks. These findings
support our hypothesis that compared to MCP, PoE is most performant on logical
reasoning tasks.

###  5.2 What Makes Good Masks?

As shown in Section 2, scoring methods and elimination strategies jointly
determine which options to eliminate ($Y_{\text{wrong}}$), and create masks
accordingly. In this section, we analyze the effect of using different
configurations of these two factors.666We also tried mask tokens other than
”[MASK]”, but did not find one consistently better than others. So we don’t
tune this token to avoid unnecessary costs. We define masking accuracy
($\text{Acc}_{\text{mask}}$) as the ratio of instances that have their correct
option kept after elimination. $\text{Acc}_{\text{mask}}$ represents the upper
bound of PoE, which we aim to maximize to reduce error propagation (Du and
Cardie, 2020). We measure two quantities: $\text{Acc}_{\text{mask}}$ and final
accuracy (Acc). We consider four scoring methods (Calibration, Channel, LM,
MCP) and two elimination strategies ("Below Average" and "Lowest", the latter
of which means eliminating the option with the lowest score). We average zero-
shot results over all 8 tasks in Table 1.

Figure 2: Effect of different scoring methods and elimination strategies on
masking accuracy ($\text{Acc}_{\text{mask}}$) and final accuracy (Acc). The
best configuration is (MCP, Lowest), with a large gap between
$\text{Acc}_{\text{mask}}$ and Acc.

As shown in Figure 2, the best scoring method is MCP, and the two elimination
strategies are comparable to each other. We find that higher
$\text{Acc}_{\text{mask}}$ leads to higher Acc. The best configuration of
scoring method and elimination strategy (MCP, Lowest) leads to 91.9%
$\text{Acc}_{\text{mask}}$.777We use (MCP, Below Average) in the main
experiment, because this configuration makes PoE more separable on tasks that
it performs well, i.e., the gap between PoE and the second-best baseline is
larger. This means a moderate-sized LM (FLAN-T5-XL) can eliminate wrong
options while keeping the correct ones most of the time. The corresponding
68.7% Acc means a large performance gap, which suggests a more powerful LM
like FLAN-T5-XXL for prediction. We also find $Y_{\text{wrong}}$ and masks to
make PoE more interpretable, because they provide intermediate reasoning
outputs. Enforcing masks in prediction also makes our method faithful and
factual.

###  5.3 Does PoE Work with LLMs?

To measure whether PoE is compatible with LLMs, we implement a prompting-based
variation of PoE and apply it to ChatGPT (gpt-3.5-turbo-0613). We then measure
its zero-shot performance on 8 original tasks in Table 1 and 4 logical
reasoning tasks in Table 2. For PoE, we only use ChatGPT in the second step
(prediction), as the base LM suffices for masking (Section 5.2). Concretely,
we prompt ChatGPT to complete $x_{\text{mask}}$, and extract the last
occurrence of any option symbol as the prediction. We compare PoE to MCP,
which is similarly modified to work with ChatGPT.

Task | Method
---|---
MCP | PoE |  PoE \- MCP
SIT | 2.0 | 19.4 | +17.4
ANLI | 40.2 | 44.6 | +4.4
SS | 79.4 | 82.8 | +3.4
CQA | 75.0 | 77.0 | +2.0
LD | 37.6 | 38.0 | +0.4
SIQA | 72.6 | 71.2 | -1.4
DQA | 48.8 | 42.6 | -6.2
CC | 79.6 | 66.8 | -12.8
RACO | 4.8 | 50.4 | +45.6
IMT | 34.0 | 49.8 | +15.8
LA | 50.0 | 55.0 | +5.0
CLD | 92.4 | 91.4 | -1.0
Table 3: PoE’s zero-shot performance on ChatGPT (gpt-3.5-turbo-0613). The
bottom 4 tasks are logical reasoning tasks from Section 5.1. PoE consistently
outperforms MCP, especially on logical reasoning tasks.

We present the result in Table 3. The result from ChatGPT is consistent with
those from FLAN-T5-XL (Table 1): PoE beats MCP on 5 out of the 8 original
tasks, and performs well on 4 logical reasoning tasks. These findings suggest
our method also works with LLMs. Nevertheless, we find ChatGPT underperforms
FLAN-T5-XL on some tasks, which requires further experiments and is beyond the
scope of this work.

###  5.4 Does PoE Work in Few-Shot Settings?

To measure PoE in the few-shot settings, we compare its zero-shot and three-
short performance with MCP on ANLI, CC, and LD. We build three-shot
demonstrations by randomly sampling from the training sets of ANLI and test
sets of CC and LD.888For CC and LD, we run experiments on the remaining data
after sampling.

We present the result in Figure 3. For ANLI, PoE underperforms MCP in both
settings, but their performance gap drops from 5.3% (zero-shot) to 1.5%
(three-shot). For LD and CC, PoE outperforms MCP in both settings, but their
performance gap similarly diminishes in the three-shot setting. Comparing PoE
with MCP, We find three-shot PoE to be less performant than its zero-shot
counterpart, and possible reasons include: 1) we did not optimize prompts or
demonstrations for three-shot PoE; 2) the instructions we use in zero-shot PoE
are already powerful, and three-shot demonstrations may introduce some noise.
Still, our findings suggest PoE is applicable to few-shot settings, which we
will continue to explore in future work.

Figure 3: Three-shot and zero-shot accuracy scores on ANLI, CC, and LD.
Although PoE shows mixed performance, it is applicable to few-shot settings.

##  6 Conclusion

We present PoE, a two-step scoring method for multiple choice reasoning. PoE
eliminates some wrong options in the first step, and chooses from the
remaining options in the second step. PoE performs well on reasoning
(especially logical reasoning) tasks in the zero-shot setting, and also works
with LLMs or in the few-shot setting. In the future, we will improve the
generalizability of our method, and use fine-tuning to better enforce the
masks.

## Limitations

The first limitation of our work is the enforcement of the masks. Although we
replace eliminated options with "[MASK]", and also replace their scores
afterward, the model still considers these options in prediction. It would be
better if the model does not consider these options at all. A second
limitation is that we do not fully explore the components of PoE: We fix the
prompts, which may not be optimal. A third limitation of our work is the scope
of the LLM experiment: We only test in the zero-shot setting. It would be
better if we also run few-shot LLM experiments, and incorporate complex
prompting methods like Chain-of-Thought Prompting (Wei et al., 2022). A fourth
limitation is that this work considers only one modality, i.e., language. It
would be interesting to extend PoE to tasks involving other modalities, like
vision (Seo et al., 2022).

## Ethics Statement

The tasks and models in this work are publicly available. They could contain
bias, and should be used with discretion.

## References

  * Besta et al. (2023) Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. 2023.  Graph of thoughts: Solving elaborate problems with large language models.  _arXiv preprint arXiv:2308.09687_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Doyle (1890) Arthur Conan Doyle. 1890.  _The Sign of Four_.  Spencer Blackett.
  * Du and Cardie (2020) Xinya Du and Claire Cardie. 2020.  Event extraction by answering (almost) natural questions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 671–683, Online. Association for Computational Linguistics.
  * Fei et al. (2023) Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023.  Mitigating label biases for in-context learning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14014–14031, Toronto, Canada. Association for Computational Linguistics.
  * Freivalds et al. (2002) Rusins Freivalds, Marek Karpinski, Carl H. Smith, and Rolf Wiehagen. 2002.  Learning by the process of elimination.  _Inf. Comput._ , 176:37–50.
  * Holtzman et al. (2021) Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021.  Surface form competition: Why the highest probability answer isn’t always right.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7038–7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  In _Advances in Neural Information Processing Systems_.
  * Lyu et al. (2023) Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  Z-ICL: Zero-shot in-context learning with pseudo-demonstrations.  In _ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models_.
  * Malkin et al. (2022) Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. 2022.  Coherence boosting: When your pretrained language model is not paying enough attention.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8214–8236, Dublin, Ireland. Association for Computational Linguistics.
  * Min et al. (2022) Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Noisy channel language model prompting for few-shot text classification.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5316–5330, Dublin, Ireland. Association for Computational Linguistics.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online. Association for Computational Linguistics.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_.
  * Robinson and Wingate (2023) Joshua Robinson and David Wingate. 2023.  Leveraging large language models for multiple choice question answering.  In _The Eleventh International Conference on Learning Representations_.
  * Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019.  Social IQa: Commonsense reasoning about social interactions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.
  * Seo et al. (2022) Minji Seo, YeonJoon Jung, Seungtaek Choi, Seung-won Hwang, and Bei Liu. 2022.  Debiasing event understanding for visual commonsense tasks.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 782–787, Dublin, Ireland. Association for Computational Linguistics.
  * Shum et al. (2023) KaShun Shum, Shizhe Diao, and Tong Zhang. 2023.  Automatic prompt augmentation and selection with chain-of-thought from labeled data.  _arXiv preprint arXiv:2302.12822_.
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2023.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_.
  * Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023.  Challenging BIG-bench tasks and whether chain-of-thought can solve them.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 13003–13051, Toronto, Canada. Association for Computational Linguistics.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wang et al. (2023a) PeiFeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. 2023a.  PINTO: Faithful language reasoning using prompt-generated rationales.  In _The Eleventh International Conference on Learning Representations_.
  * Wang et al. (2023b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_.
  * Ye et al. (2023) Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2023.  Satisfiability-aided language models using declarative prompting.  _arXiv preprint arXiv:2305.09656_.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _International Conference on Machine Learning_ , pages 12697–12706. PMLR.

##  Appendix A Task Information and Pre-Processing

All tasks we use are expressed in English. ANLI is originally a classification
task, and we convert it to multiple choice by mapping labels to options, i.e.,
"0" to "entailment", "1" to "neutral", and "2" to "contradiction". For BIG-
bench tasks, we remove instances whose number of options deviate from others.
For ANLI and CC, we aggregate instances from all subtasks; For LD, we use the
subtask with 5 options; For SS, we use the multiple choice subtask, and treat
options with scores of 0.5 as wrong. We provide other details in Table 4. The
first three tasks can be accessed on Hugging Face.999https://huggingface.co/
The last thirteen tasks can be accessed on BIG-
bench.101010https://github.com/google/BIG-bench

Task | # Options | Train | Dev | Test
---|---|---|---|---
ANLI | 3 | 162865 | 3200 | 3200
CQA | 5 | 9741 | 1221 | N/A
SIQA | 3 | 33410 | 1954 | N/A
CAI | 4 | N/A | N/A | 44
CC | 4 | N/A | N/A | 103
CLD | 4 | N/A | N/A | 60
DQA | 3 | N/A | N/A | 255
EIE | 5 | N/A | N/A | 68
IMT | 4 | N/A | N/A | 53
IOM | 4 | N/A | N/A | 47
LA | 5 | N/A | N/A | 32
LD | 5 | N/A | N/A | 500
RACO | 18 | N/A | N/A | 2000
RS | 5 | N/A | N/A | 49
SIT | 5 | N/A | N/A | 990
SS | 4 | N/A | N/A | 121
Table 4: Task Information. # Options means the number of options for the task.

##  Appendix B Additional Analysis

###  B.1 Different Number of Options

Since our method eliminates wrong options in the first step, we assume the
number of options affects performance. Consequently, we run experiments on
three subtasks of logical deduction (LD (3), LD (5), LD (7)), which have 3, 5,
and 7 options respectively. The other experiment settings are consistent with
the main experiment (Section 3).

Method | Task
---|---
LD (3) | LD (5) | LD (7)
LM |  57.03.9 |  48.82.7 | 47.22.7
AVG | 55.82.6 | 45.84.5 |  47.64.6
Calibration | 45.03.1 | 39.24.0 | 38.85.1
Channel | 40.03.8 | 20.83.6 | 18.22.2
MCP | 53.45.3 | 39.83.7 | 45.27.3
PoE | 70.24.2 | 56.03.8 | 53.05.5
Table 5: Accuracy scores on three subtasks of logical deduction, which have
different numbers of options. Best scores are bold, and second-best scores are
underlined. PoE applies to tasks of varying difficulties.

We present the result in Table 5. We find most methods (including PoE) perform
worse as the number of options increases. This conforms to intuition, because
more options require more reasoning steps, and the questions get harder.
Nevertheless, PoE is the best on all subtasks. This means although PoE does
not counterintuitively perform better on harder questions, it works well on
questions of different difficulties, and thus applies to a variety of multiple
choice reasoning tasks.

###  B.2 Case Study

In Figure 4, we present a case study to show why PoE works. We compare PoE
with MCP on one instance from CC. The question invents the word "wajey" with a
definition, and forms a surprising and uncommon conceptual combination between
"wajey" and "grape". The correct option is A.

Figure 4: A case study comparing PoE and MCP. The scores are negative log-
likelihood (lower means better). MCP eliminates options B and D, but gets
distracted by option C. In contrast, PoE eliminates options B and D in the
first step, and chooses the correct option A in the second step when B and D
are masked.

We find MCP unable to solve this task, and chooses C. Taking a closer look at
MCP scores (lower is better) for each option, we find that MCP assigns high
scores to options B and D, two seemingly wrong options. This verifies our
hypothesis that some wrong options’ scores deviate from others, and can thus
be eliminated. Nevertheless, MCP is distracted by option C, and we assume this
is caused by the correlation between "grape" and "Muscat", the latter of which
is a kind of grape. This instance shows although MCP can eliminate some wrong
options, it struggles to simultaneously choose the right option.

For PoE, since it uses MCP in the first step, it masks options B and D. We
examine the final PoE scores for each option, and find that it correctly
chooses option A. In addition, we find the PoE score for option A is lower
than its MCP score, and conversely for option C. This means our method is not
distracted by option C. This case study verifies our assumption that
eliminating wrong options and choosing the best option are two types of
reasoning that can be disentangled into two steps, and that such
disentanglement is beneficial.

##  Appendix C Input and Output Samples for All Methods

We do not optimize prompts, because our goal is to evaluate PoE with a fixed
prompt format. Therefore, we follow Holtzman et al. (2021) and Robinson and
Wingate (2023) to construct task-agnostic prompts, and use them to wrap
questions. We use one instance from SIQA. The question $x$ is "Kendall was
searching for ring with their eyes closed. They hit something. Why did Kendall
do this?". The options $Y$ are "kendall who has searching his ring" ($y_{1}$),
"kendall who has wanted to close their eyes" ($y_{2}$), and "find the rings"
($y_{3}$).

We present input and output samples for all methods for this instance in Table
6. Calibration computes two scores ($\log P(y_{i}|x)$) and ($\log
P(y_{i}|\text{text})$), so we present samples for each of them. PoE uses
different prompts for each step, so we present them separately.

Method | Score |  Input | Output
---|---|---|---
LM | $\log P(y_{i}|x)$ |  $x$ the answer is: | $y_{1}$
AVG | $1/\text{len}(y_{i})*\log P(y_{i}|x)$ |  $x$ the answer is: | $y_{1}$
Calibration | $\log P(y_{i}|x)$ |  $x$ the answer is: | $y_{1}$
$\log P(y_{i}|\text{text})$ |  the answer is: | $y_{1}$
Channel | $\log P(x|y_{i})$ |  $y_{1}$ |  $x$ the answer is:
MCP | $\log P(y_{i}|x,Y)$ |  Question: $x$ | A
A. $y_{1}$
B. $y_{2}$
C. $y_{3}$
Answer:
PoE: Elimination | $\log P(y_{i}|x,Y)$ |  Question: $x$ | A
A. $y_{1}$
B. $y_{2}$
C. $y_{3}$
Answer:
PoE: Prediction | $\log P(y_{i}|x_{\text{mask}},Y\setminus Y_{\text{wrong}})$ |  Select the most suitable option to answer the question. Ignore [MASK] options. | B
Question: $x$
A. [MASK]
B. $y_{2}$
C. $y_{3}$
Answer:
Table 6: Input and output samples for all methods, using one instance from
SIQA.