# Numerical modeling of active thermo-plasmonics experiments

Giuseppe Emanuele Lio  E-mail<EMAIL_ADDRESS>Giovanna Palermo
Antonio De Luca  CNR-Nanotec, 87036 Arcavacata di Rende (CS), Italy  Physics
Department, University of Calabria, 87036 Arcavacata di Rende (CS), Italy
Roberto Caputo  CNR-Nanotec, 87036 Arcavacata di Rende (CS), Italy  Physics
Department, University of Calabria, 87036 Arcavacata di Rende (CS), Italy
Institute of Fundamental and Frontier Sciences, University of Electronic
Science and Technology of China, Chengdu 610054

###### Abstract

In this paper, we present a simple and robust numerical method able to
predict, with high accuracy, the photo-thermal effects occurring for a gold
nanoparticles arrangement under externally applied strain. The physical system
is numerically implemented in the COMSOL Multiphysics simulation platform. The
gold nanoparticles distributions are excited by linearly polarized light. By
considering the system at rest and under the action of a mechanical stress, we
analyze the extinction cross section, and we observe the production of heat at
the nanoscale. The purpose of this work is to describe how sensitive the local
temperature of the gold nanoparticles arrangement is to the formation of
localized photo-thermal hot spots.

##  I Introduction

In the last decade, the multi-disciplinary character of research in materials
science is gradually increasing the degree of complexity of the systems under
study. An analytical approach to the modeling of such systems is de facto
leaving the field to a numerical counterpart. This trend in research is thus
promoting the evolution of comprehensive numerical packages providing an
extremely accurate description of physical processes. As a consequence, the
availability of such powerful tools are modifying the way research is
performed, eventually suggesting new possibilities to optimize the scientific
procedure itself. An emerging direction in plasmonics is the development of
plamonic heaters intended as systems where the heat is generated at the
nanoscale, by light excitation of plasmonic sub-entitites, that can be
dynamically controlled in intensity and direction of its flow. Here, we
present the performance of a specific design of plasmonic heaters through a
numerical characterization performed by the COMSOL Multiphysics FEM commercial
platform. As reported in previous works, it is possible to evaluate the photo-
thermal effects related to the interaction of an electromagnetic wave with an
arrangement of gold nanoparticles (GNPs) arrangements. The behavior of the
considered system can be accurately predicted when the GNPs is modified
because of the application of a mechanical tensile strain to the whole
structure. Palermo  _et al._ (2018); Lio  _et al._ (2019a) By consecutively
exploiting Structural Mechanics, Electromagnetic Waves and Heat Transfer
modules of the COMSOL software allows calculating: i) the new GNPs
displacements as a function of a specific percentage of strain; ii) the
plasmonic response of the GNPs distribution for “at rest” and “stretched”
conditions, in terms of extinction cross section ($\sigma_{ext}$); iii) the
temperature variation $\Delta T=T-T_{0}$, with respect to the environment
temperature ($T_{0}$), measured when the GNPs geometry is resonantly excited
by a linearly polarized plane wave. Considered the fine control of the local
temperature allowed by plasmonic heaters, their exploitation can widen the
actual scenario of thermo-plasmonic application in cellular biology, nano-
medical diagnostics and therapy. In fact, most fascinating challenges could
include but are not limited to the design of photo-thermal devices with a high
localized photo-thermal efficiency, controllable through external macroscopic
stimuli. Although several theoretical and experimental studiesBaffou, Quidant,
and Girard (2010); Govorov  _et al._ (2014); Lio  _et al._ (2019b) are all
aimed to go in this direction, the actual possibility to realize mechanically
tunable plasmonic nano-heaters by exploiting plasmo-mechanicsMaurer  _et al._
(2015); Cataldi  _et al._ (2014); Gontier  _et al._ (2017); Marae-Djouda  _et
al._ (2018) and active plasmonics MacDonald and Zheludev (2010); Schuller  _et
al._ (2010); Caputo  _et al._ (2013) is still an open challenge.

##  II Results and Discussion

In this section, we show how it is possible to reproduce an entire
experimental procedure in a numerical way. We will start to approach the
experiment by applying an external strain to modify the initial arrangement of
the considered GNPs supported by an elastic substrate. The geometry at rest is
a square of 25 NPs, characterized by a radius ($R$) of 20 nm and center-to-
center interdistance $r$ equal to 3$R$. The first step consists in the
application of the mechanical strain, followed by the electromagnetic
modeling. Finally, it is shown how the opto-mechanical effects are exploited
to enhance the photo-thermal response of the structure for different amount of
stretching.

###  II.1 Mechanical strain

In a real experiment, it is possible to apply a strain to a sample in
different ways, the more common is to use a tensile-strain machine. In our
numerical experiments the external strain is applied as a boundary load at the
edge of the elastic substrate. As in the reality, the GNPs arrangement is
placed onto a transparent and elastic polymer matrix made of
Polydimethylsiloxane (PDMS). The stretching appears when we apply a normal (N)
tensile strains with positive (F) and negative forces (-F) along the $x$
direction. It produces the effect shown in Figure 1a, where it is possible to
evaluate the stretching of the substrate. In our case we are interested on the
detailed displacement of each GNP. Figure 1b shows the GNPs arrangement at
rest, while in Figure 1c-e are shown the displacements of the GNPs for
stretching percentages of 8%, 19% and 27%, respectively.

Figure 1: (a) Sketch of the entire system (GNPs/substrate) under stretching.
The edges where the loads assume a red coloration indicate the maximum
displacement and are deformed in accordance with the strain direction. On the
other hand, the other edges of the substrate appear green because of the
compression undergone in the direction perpendicular to the applied strain.
(b) The GNPs arrangement at the rest (b) and for the stretching percentage of
(c) 8%, (d) 19% and (e) 27%. The blurred blue spheres in c-e represent the
GNPs position at rest.

###  II.2 Electromagnetic analysis

When the new positions of the GNPs in the stretched geometries are calculated,
we can use them to study what happens when a polarized electromagnetic wave
impinges on the resulting different particles arrangements. To study the
plasmonic response of the GNPs arrangements (at rest and stretched) we collect
the absorption cross section $\sigma_{abs}$ and the scattering cross section
$\sigma_{sca}$ to obtain the extinction cross section:
$\sigma_{ext}=\sigma_{abs}+\sigma_{sca}$. In the methods section we report all
details related to the strategy that we adopted to analyze $\sigma_{ext}$ and
the materials involved in our model. We stress out that by modifying the GNPs
arrangements through an externally applied strain, it is possible to change
the wavelength corresponding to the plasmonic resonance $\lambda_{p0}$ of the
initial system.

In Figure 2a, we report the $\sigma_{ext}$ for different stretching
percentages, from the rest condition to the maximum applicable stretching yet
avoiding the GNPs overlapping. It is possible to observe that by stretching
the substrate from 0% to 27% a redshift of the $\lambda_{p0}$ from 510 nm to
$\lambda_{p27\%}$=515 nm occurs.

The electric field (E) maps, for the GNPs arrangement at rest (Figure 2b) and
for the different percentages of stretching (Figure 2c-d), show the different
enhancement and distribution of the electric field around the NPs as a
function of the particles displacement. These maps are collected at
$\lambda$=532 nm, that corresponds to the typical excitation wavelength
source, used in laboratory, to excite photo-thermal effects in spherical NPs.
As we can see, by increasing the stretching percentage we obtain an
enhancement of the E field due to the interaction between the nanoparticles
that are closer than in the rest condition. It is clear that the
electromagnetic coupling between the GNPs follows the spectral behavior of the
extinction curves. In fact, if we plot the E map for the max stretching
percentage considered (27%) at the new $\lambda_{p27\%}$= 515 nm we obtain an
intensity of Emax equal to 3.2 $\cdot$ 106 V/m, that is even higher with
respect to the value obtained at 532 nm (Emax=2.2 $\cdot$106 V/m).

Figure 2: (a) Extinction cross section curves for the different stretching
percentages: the rest condition is the black solid line with the plasmonic
peak around $\lambda_{p0}$= 510 nm. By stretching the structure the amplitude
of $\sigma_{ext}$ increases and the peak shifts until $\lambda_{p27\%}$=
515nm. The green dashed line indicates the wavelength of the used laser in the
thermal ”numerical experiments” ($\lambda_{therm}=532nm$). (b-d) Electric
field maps for the rest condition and two different stretching values under
the $\lambda_{therm}$ excitation. (e) E maps for the 27% of stretching at the
new $\lambda_{p27\%}$.

###  II.3 Thermal analysis

After stretching the structure and collecting the modification of its
plasmonic response due to the corresponding change in $\sigma_{ext}$, the last
part of the work is devoted to characterize the GNPs arrangement in terms of
photo-induced heating. The square structure allows the formation of chains
that interact as electromagnetic and thermal hot spots resulting in a
temperature enhancement. In Figure 3a-c, we reported the thermal heating maps
($\Delta$T) of the GNPs distribution for different stretching percentages
calculated at $\lambda_{therm}$. The temperature variation $\Delta T$
gradually increases following the cross section amplitude: in fact, a $\Delta
T_{MAX}$=26.5∘ for the substrate at rest (0%) is modified in a much higher
$\Delta T_{MAX}$=41.3∘ in case of a 27% strain percentage. Noticebably, by
calculating the $\Delta T$ increase for the wavelength $\lambda_{p27\%}$
corresponding to the highest $\sigma_{ext}$ value, $\Delta T_{MAX}$ is the
highest with (44.8∘).

Figure 3: Thermal maps of the (a) rest, (b) 19%, (c) 27% stretched structure
at $\lambda_{therm}$. (d) Thermal map for the max stretching percentage of 27%
at $\lambda_{p_{27\%}}$.

##  III Conclusions

In this contribution, we presented a comprehensive electromagnetic and thermal
analysis of 2D square arrangement of plasmonic nano-structures. The proposed
numerical simulations are obtained by using COMSOL Multiphysics for the entire
experimental process from the stretching to the photo-thermal evaluation. By
using these coupled multi-physics models we pave the route to study and
prototype, in a simple and fast way, tunable/active plasmonic systems that can
work like nano-heaters. This kind of studies evidence specific regions of
thermal hot-spots and thus envision the possibility to realize devices in a
wide scenario of applications including medical, environmental and biological
field to name a few.

## Methods

By the use of the mechanical tool, we study how an applied external strain can
deform the initial structure. To use it, we fix the force (F) that represents
the boundary load, and the relative constrains. Generally, the constrains
involved for a stretching case are the following: the edges along the $y$
direction are left free to move, and they are fixed as zero in $x$ and $z$
directions to avoid bend and shear actions. For the same reasons the edges in
the $x$ direction are free to move along $x$ and are constrained in $y$ and
$z$ directions. To reproduce arrangements characterized by different
displacements, it is necessary to change step by step F and save the
corresponding geometry as .stl file. The resulting new geometry, that
corresponds to the stretched structure, should be imported in the
electromagnetic-thermic model to follow the process to study the
$\sigma_{ext}$ and $\Delta$T. The extinction cross sections and temperature
variations, evaluated for different stretching percentages, have been obtained
by implementing a 3D simulation in COMSOL Multiphysics. The simulation
involves a controlled spherical volume, also called perfect matched layer
(PML), whose size is much larger than the radius of the NPs, $r_{PML}$ = 20R
or 30R, depending on the specific geometry.

The absorption and scattering cross sections are calculated, in the Electro-
Magnetic Wave, Frequency Domain (emw) module, by means of the following
relations: $\sigma_{abs}$=$W_{abs}/P_{in}$, $\sigma_{sca}$=$W_{sca}/P_{in}$,
where $P_{in}$ is the incident irradiance, defined as energy flux of the
incident wave; $W_{abs}$ is the energy rate absorbed by particle that is
derived by integrating the energy loss $Q_{loss}$ over the volume of the
particle, while $W_{sca}$ is the energy rate absorbed by particle that is
derived by integrating is derived by integration the Poynting vector over an
imaginary sphere around the particle. All the other parameters as scattered
field method, proper polarization and propagation direction as opportunely set
to numerically investigate the near field distribution and the plasmonic
response of the GNPs. The light beam intensity, has been evaluated as
$I=E_{0}^{2}/(2Z_{0const})$ = 1.33 E7 W/m2, where E0 is the initial electric
field (in our case 1E5 V/m) and $Z_{0const}$= 376.73 $\Omega$ is the impedance
of the system that takes into account also the incident area. This tool
package offers the possibility to study the energy flow that passes through
the NPs when they are shone by a light beam and the energy flow scattered from
NPs and collected on the external layer used as an integrating sphere.

Finally, the Heat Transfer in Solids module is used to study the interaction
between light and NPs in terms of localized heating. Here it is necessary to
set the area that interacts with the electromagnetic wave (NP surfaces and
surrounding medium) and the room temperature $T_{0}$.

The materials involved in these simulations are gold, air and PDMS. For each
one, in the Materials section of the software the following data have been
considered:

Gold: real and imaginary part of permittivity as a function of wavelength from
the Materials library of the software, permeability $\mu$ = 1, the electrical
conductivity $\sigma=1$, Young module $E=7\cdot 10^{10}Pa$, Poisson ratio
$\nu=0.44$, thermal conductivity $K=314W/m\cdot K$, the density
$\rho=19300kg/m^{3}$ and the heat capacity $C_{p}=126J/kg\cdot K$.

Air: real and imaginary part of permittivity as a function of wavelength,
permeability $\mu=1$, the electrical conductivity $\sigma=1$, thermal
conductivity $K=1W/m\cdot K$, the density $\rho=1kg/m^{3}$ and the heat
capacity $C_{p}=1J/kg\cdot K$.

PDMS: real and imaginary part of permittivity as a function of wavelength,
permeability $\mu=1$, the electrical conductivity $\sigma=1$, Young module
$E=7.5\cdot 10^{5}Pa$, Poisson ratio $\nu=0.49$, thermal conductivity
$K=0.16W/m\cdot K$, the density $\rho=970kg/m^{3}$ and the heat capacity
$C_{p}=1460J/kg\cdot K$.

## Acknowledgements

The authors thank the “Area della Ricerca di Roma 2”, Tor Vergata, for the
access to the ICT Services (ARToV-CNR) for the use of the COMSOL Multiphysics
Platform and Origin Lab, and the Infrastructure “BeyondNano” (PONa3-00362) of
CNR-Nanotec for the access to research instruments.

## References

## References

  * Palermo  _et al._ (2018) G. Palermo, U. Cataldi, A. Condello, R. Caputo, T. Bürgi, C. Umeton, and A. De Luca, “Flexible thermo-plasmonics: an opto-mechanical control of the heat generated at the nanoscale,” Nanoscale , 16556–16561 (2018).
  * Lio  _et al._ (2019a) G. E. Lio, G. Palermo, R. Caputo, and A. De Luca, “Opto-mechanical control of flexible plasmonic materials,” Journal of Applied Physics 125, 082533 (2019a).
  * Baffou, Quidant, and Girard (2010) G. Baffou, R. Quidant, and C. Girard, “Thermoplasmonics modeling: A green?s function approach,” Physical Review B 82, 165424 (2010).
  * Govorov  _et al._ (2014) A. O. Govorov, H. Zhang, H. V. Demir, and Y. K. Gun?ko, “Photogeneration of hot plasmonic electrons with metal nanocrystals: Quantum description and potential applications,” Nano Today 9, 85–101 (2014).
  * Lio  _et al._ (2019b) G. E. Lio, G. Palermo, A. De Luca, and R. Caputo, “Tensile control of the thermal flow in plasmonic heaters realized on flexible substrates,” The Journal of Chemical Physics 151, 244707 (2019b).
  * Maurer  _et al._ (2015) T. Maurer, J. Marae-Djouda, U. Cataldi, A. Gontier, G. Montay, Y. Madi, B. Panicaud, D. Macias, P.-M. Adam, G. Lévêque, _et al._ , “The beginnings of plasmomechanics: towards plasmonic strain sensors,” Frontiers of Materials Science 9, 170–177 (2015).
  * Cataldi  _et al._ (2014) U. Cataldi, R. Caputo, Y. Kurylyak, G. Klein, M. Chekini, C. Umeton, and T. Bürgi, “Growing gold nanoparticles on a flexible substrate to enable simple mechanical control of their plasmonic coupling,” Journal of Materials Chemistry C 2, 7927–7933 (2014).
  * Gontier  _et al._ (2017) A. Gontier, J. Marae-Djouda, R. Caputo, Y. Madi, M. Molinari, G. Léveque, P.-M. Adam, and T. Maurer, “Optical properties of gold nanorods macro-structure: a numerical study,” Photonics Letters of Poland 9, 23–25 (2017).
  * Marae-Djouda  _et al._ (2018) J. Marae-Djouda, A. Gontier, R. Caputo, G. Lévêque, B. Bercu, Y. Madi, G. Montay, P.-M. Adam, M. Molinari, S. Stagon, _et al._ , “Dense brushes of tilted metallic nanorods grown onto stretchable substrates for optical strain sensing,” ACS Applied Nano Materials 1, 2347–2355 (2018).
  * MacDonald and Zheludev (2010) K. F. MacDonald and N. I. Zheludev, “Active plasmonics: current status,” Laser & Photonics Reviews 4, 562–567 (2010).
  * Schuller  _et al._ (2010) J. A. Schuller, E. S. Barnard, W. Cai, Y. C. Jun, J. S. White, and M. L. Brongersma, “Plasmonics for extreme light concentration and manipulation,” Nature materials 9, 193 (2010).
  * Caputo  _et al._ (2013) R. Caputo, L. De Sio, U. Cataldi, and C. Umeton, “Active plasmonics in self-organized soft materials,” in  _Amorphous Nanophotonics_ (Springer, 2013) pp. 307–326.


* Brunner et al. (2018) Gino Brunner, Yuyi Wang, Roger Wattenhofer, and Michael Weigelt. 2018.  Natural language multitasking analyzing and improving syntactic saliency of latent representations.  _arXiv_ (1 2018).  http://arxiv.org/abs/1801.06024
  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018.  e-SNLI: Natural Language Inference with Natural Language Explanations. In _Advances in Neural Information Processing Systems_ , Vol. 2018-Decem. 9539–9549.  http://arxiv.org/abs/1812.01193
  * Carvalho et al. (2019) Diogo V. Carvalho, Eduardo M. Pereira, and Jaime S. Cardoso. 2019.  Machine Learning Interpretability: A Survey on Methods and Metrics.  _Electronics_ 8, 8 (7 2019), 832\.  https://doi.org/10.3390/electronics8080832
  * Chakraborty et al. (2017) Supriyo Chakraborty, Richard Tomsett, Ramya Raghavendra, Daniel Harborne, Moustafa Alzantot, Federico Cerutti, Mani Srivastava, Alun Preece, Simon Julier, Raghuveer M. Rao, Troy D. Kelley, Dave Braines, Murat Sensoy, Christopher J. Willis, and Prudhvi Gurram. 2017.  Interpretability of deep learning models: A survey of results. In _2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI)_. IEEE, 1–6.  https://doi.org/10.1109/UIC-ATC.2017.8397411
  * Chang et al. (2009) Jonathan Chang, Jordan Boyd-graber, Sean Gerrish, Chong Wang, David M. Blei, Jordan Boyd-graber, and David M. Blei. 2009.  Reading Tea Leaves: How Humans Interpret Topic Models. In _Advances in Neural Information Processing Systems_ , Y Bengio, D Schuurmans, J Lafferty, C Williams, and A Culotta (Eds.), Vol. 22. Curran Associates, Inc., 288–296.  https://proceedings.neurips.cc/paper/2009/file/f92586a25bb3145facd64ab20fd554ff-Paper.pdf
  * Chatzimparmpas et al. (2020) A. Chatzimparmpas, R. M. Martins, I. Jusufi, K. Kucher, F. Rossi, and A. Kerren. 2020\.  The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations.  _Computer Graphics Forum_ 39, 3 (6 2020), 713–756.  https://doi.org/10.1111/cgf.14034
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019.  What Does BERT Look at? An Analysis of BERT’s Attention. In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. Association for Computational Linguistics, Stroudsburg, PA, USA, 276–286.  https://doi.org/10.18653/v1/W19-4828
  * Clouatre et al. (2021) Louis Clouatre, Prasanna Parthasarathi, Amal Zouaq, and Sarath Chandar. 2021.  Local Structure Matters Most: Perturbation Study in NLU.  (7 2021).  http://arxiv.org/abs/2107.13955
  * Coenen et al. (2019) Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, Martin Wattenberg, Ann Yuan Been Kim, Adam Pearce, Fernanda Viégas, Martin Wattenberg, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019\.  Visualizing and Measuring the Geometry of BERT. In _Advances in Neural Information Processing Systems_ , H Wallach, H Larochelle, A Beygelzimer, F d\textquotesingle Alché-Buc, E Fox, and R Garnett (Eds.), Vol. 32. Curran Associates, Inc., 8594–8603.  https://proceedings.neurips.cc/paper/2019/file/159c1ffe5b61b41b3c4d8f4c2150f6c4-Paper.pdf
  * Conneau et al. (2018) Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018\.  What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Stroudsburg, PA, USA, 2126–2136.  https://doi.org/10.18653/v1/P18-1198
  * Cook and Weisberg (1980) R. Dennis Cook and Sanford Weisberg. 1980.  Characterizations of an Empirical Influence Function for Detecting Influential Cases in Regression.  _Technometrics_ 22, 4 (11 1980), 495–508.  https://doi.org/10.1080/00401706.1980.10486199
  * Costello and Osborne (2005) Anna B. Costello and Jason W. Osborne. 2005.  Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis.  _Practical Assessment, Research and Evaluation_ 10, 7 (2005), 1–9.  https://doi.org/10.7275/jyj1-4868
  * Crawford and Ferguson (1970) Charles B. Crawford and George A. Ferguson. 1970.  A general rotation criterion and its use in orthogonal rotation.  _Psychometrika_ 35, 3 (9 1970), 321–332.  https://doi.org/10.1007/BF02310792
  * Danilevsky et al. (2020) Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. 2020\.  A Survey of the State of Explainable AI for Natural Language Processing.  Section 5 (2020).  http://arxiv.org/abs/2010.00711
  * Devlin et al. (2019) Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding. In _NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference_ , Vol. 1. Association for Computational Linguistics (ACL), 4171–4186.  http://arxiv.org/abs/1810.04805
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards A Rigorous Science of Interpretable Machine Learning.  _arXiv_ (2 2017).  http://arxiv.org/abs/1702.08608
  * Doshi-Velez et al. (2017) Finale Doshi-Velez, Mason Kortz, Ryan Budish, Christopher Bavitz, Sam Gershman, David O’Brien, Kate Scott, Stuart Schieber, James Waldo, David Weinberger, Adrian Weller, and Alexandra Wood. 2017\.  Accountability of AI Under the Law: The Role of Explanation.  _SSRN Electronic Journal_ Online (11 2017).  https://doi.org/10.2139/ssrn.3064761
  * Du et al. (2019) Mengnan Du, Ninghao Liu, and Xia Hu. 2019.  Techniques for interpretable machine learning.  _Commun. ACM_ 63, 1 (12 2019), 68–77.  https://doi.org/10.1145/3359786
  * Ebrahimi et al. (2018) Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018\.  HotFlip: White-Box Adversarial Examples for Text Classification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ , Vol. 2. Association for Computational Linguistics, Stroudsburg, PA, USA, 31–36.  https://doi.org/10.18653/v1/P18-2006
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020\.  Evaluating Models’ Local Decision Boundaries via Contrast Sets. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Stroudsburg, PA, USA, 1307–1323.  https://doi.org/10.18653/v1/2020.findings-emnlp.117
  * Garrido-Muñoz et al. (2021) Ismael Garrido-Muñoz, Arturo Montejo-Ráez, Fernando Martínez-Santiago, and L. Alfonso Ureña-López. 2021.  A Survey on Bias in Deep NLP.  _Applied Sciences_ 11, 7 (4 2021), 3184\.  https://doi.org/10.3390/app11073184
  * Ghorbani et al. (2019) Amirata Ghorbani, James Wexler, James Zou, and Been Kim. 2019\.  Towards automatic concept-based explanations.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Goyal et al. (2019) Yash Goyal, Uri Shalit, and Been Kim. 2019.  Explaining classifiers with causal concept effect (CaCE).  _arXiv_ (7 2019).  http://arxiv.org/abs/1907.07165
  * Guo et al. (2020) Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. 2020.  FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging.  _arXiv_ (12 2020).  http://arxiv.org/abs/2012.15781
  * Gupta et al. (2020) Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2020.  Neural Module Networks for Reasoning over Text. In _International Conference on Learning Representations (ICLR)_.  https://openreview.net/forum?id=SygWvAVFPr
  * Han et al. (2020) Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. 2020\.  Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, 5553–5563.  https://doi.org/10.18653/v1/2020.acl-main.492
  * Hase et al. (2020) Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. 2020\.  Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Stroudsburg, PA, USA, 4351–4367.  https://doi.org/10.18653/v1/2020.findings-emnlp.390
  * Hewitt and Liang (2019) John Hewitt and Percy Liang. 2019.  Designing and Interpreting Probes with Control Tasks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Stroudsburg, PA, USA, 2733–2743.  https://doi.org/10.18653/v1/D19-1275
  * Ho and Xiang (2020) Daniel E Ho and Alice Xiang. 2020.  Affirmative Algorithms: The Legal Grounds for Fairness as Awareness.  _arXiv_ (12 2020).  http://arxiv.org/abs/2012.14285
  * Hooker et al. (2019) Sara Hooker, Dumitru Erhan, Pieter-Jan Jan Kindermans, and Been Kim. 2019.  A benchmark for interpretability methods in deep neural networks. In _Advances in Neural Information Processing Systems_ , Vol. 32.  http://arxiv.org/abs/1806.10758
  * Ibrahim et al. (2019) Mark Ibrahim, Melissa Louie, Ceena Modarres, and John Paisley. 2019.  Global Explanations of Neural Networks. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_. ACM, New York, NY, USA, 279–287.  https://doi.org/10.1145/3306618.3314230
  * Jacovi and Goldberg (2020) Alon Jacovi and Yoav Goldberg. 2020.  Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, 4198–4205.  https://doi.org/10.18653/v1/2020.acl-main.386
  * Jain and Wallace (2019) Sarthak Jain and Byron C. Wallace. 2019.  Attention is not Explanation. In _Proceedings of the 2019 Conference of the North_ , Vol. 1. Association for Computational Linguistics, Stroudsburg, PA, USA, 3543–3556.  https://doi.org/10.18653/v1/N19-1357
  * Jang and Lukasiewicz (2021) Myeongjun Jang and Thomas Lukasiewicz. 2021.  Are Training Resources Insufficient? Predict First Then Explain!  _arXiv_ (8 2021).  http://arxiv.org/abs/2110.02056
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020\.  Scaling Laws for Neural Language Models.  _arXiv_ (1 2020).  http://arxiv.org/abs/2001.08361
  * Kaufmann and Kalyanakrishnan (2013) Emilie Kaufmann and Shivaram Kalyanakrishnan. 2013.  Information complexity in bandit subset selection. In _Journal of Machine Learning Research_.  http://proceedings.mlr.press/v30/Kaufmann13.pdf
  * Kaushik et al. (2020) Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. 2020\.  Learning The Difference That Makes A Difference With Counterfactually-Augmented Data. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=Sklgs0NFvr
  * Kim et al. (2018) Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018.  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV).  _35th International Conference on Machine Learning, ICML 2018_ 6 (11 2018), 4186–4195.  http://arxiv.org/abs/1711.11279
  * Kindermans et al. (2017) Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. 2017.  The (Un)reliability of saliency methods.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 11700 LNCS (11 2017), 267–280.  https://doi.org/10.1007/978-3-030-28954-6{_}14
  * Koh and Liang (2017) Pang Wei Koh and Percy Liang. 2017.  Understanding Black-box Predictions via Influence Functions.  _34th International Conference on Machine Learning, ICML 2017_ 4 (3 2017), 2976–2987.  http://arxiv.org/abs/1703.04730
  * Köhn (2015) Arne Köhn. 2015\.  What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Stroudsburg, PA, USA, 2067–2073.  https://doi.org/10.18653/v1/D15-1246
  * Kumar and Talukdar (2020) Sawan Kumar and Partha Talukdar. 2020.  NILE : Natural Language Inference with Faithful Natural Language Explanations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, 8730–8742.  https://doi.org/10.18653/v1/2020.acl-main.771
  * Latcinnik and Berant (2020) Veronica Latcinnik and Jonathan Berant. 2020.  Explaining Question Answering Models through Text Generation.  _arXiv_ (4 2020).  http://arxiv.org/abs/2004.05569
  * Lei et al. (2016) Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016.  Rationalizing Neural Predictions. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Stroudsburg, PA, USA, 107–117.  https://doi.org/10.18653/v1/D16-1011
  * Li et al. (2016) Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016\.  Visualizing and Understanding Neural Models in NLP. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Stroudsburg, PA, USA, 681–691.  https://doi.org/10.18653/v1/N16-1082
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016\.  Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies.  _Transactions of the Association for Computational Linguistics_ 4, 1990 (12 2016), 521–535.  https://doi.org/10.1162/tacl{_}a{_}00115
  * Lipton (2018) Zachary C Lipton. 2018\.  The mythos of model interpretability.  _Commun. ACM_ 61, 10 (9 2018), 36–43.  https://doi.org/10.1145/3233231
  * Liu et al. (2019b) Hui Liu, Qingyu Yin, and William Yang Wang. 2019b.  Towards Explainable NLP: A Generative Explanation Framework for Text Classification. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, 5570–5581.  https://doi.org/10.18653/v1/P19-1560
  * Liu et al. (2019a) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a.  RoBERTa: A robustly optimized BERT pretraining approach.  https://github.com/pytorch/fairseq
  * Lundberg and Lee (2017) Scott Lundberg and Su-In Lee. 2017.  A Unified Approach to Interpreting Model Predictions.  _Advances in Neural Information Processing Systems_ 2017-Decem (5 2017), 4766–4775.  http://arxiv.org/abs/1705.07874
  * Madsen (2019) Andreas Madsen. 2019\.  Visualizing memorization in RNNs.  _Distill_ 4, 3 (3 2019).  https://doi.org/10.23915/distill.00016
  * Madsen et al. (2021) Andreas Madsen, Nicholas Meade, Vaibhav Adlakha, and Siva Reddy. 2021.  Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining.  _arXiv_ (2021).  http://arxiv.org/abs/2110.08412
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, 3428–3448.  https://doi.org/10.18653/v1/P19-1334
  * Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021\.  A Survey on Bias and Fairness in Machine Learning.  _Comput. Surveys_ 54, 6 (2021), 1–35.  https://doi.org/10.1145/3457607
  * Meister et al. (2021) Clara Meister, Stefan Lazov, Isabelle Augenstein, and Ryan Cotterell. 2021.  Is Sparse Attention more Interpretable?. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_. Association for Computational Linguistics, Stroudsburg, PA, USA, 122–129.  https://doi.org/10.18653/v1/2021.acl-short.17
  * Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. 2019.  Are Sixteen Heads Really Better than One?  _Advances in Neural Information Processing Systems_ 32 (5 2019), 1–13.  http://arxiv.org/abs/1905.10650
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013\.  Efficient estimation of word representations in vector space. In _1st International Conference on Learning Representations, ICLR 2013 - Workshop Track Proceedings_.  http://ronan.collobert.com/senna/
  * Miller (2019) Tim Miller. 2019\.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial Intelligence_ 267 (2 2019), 1–38.  https://doi.org/10.1016/j.artint.2018.07.007
  * Molnar (2019) Christoph Molnar. 2019\.  _Interpretable Machine Learning_.  Independent. 318 pages.  https://christophm.github.io/interpretable-ml-book/
  * Mu and Andreas (2020) Jesse Mu and Jacob Andreas. 2020.  Compositional Explanations of Neurons. In _Advances in Neural Information Processing Systems_.  http://arxiv.org/abs/2006.14032
  * Mudrakarta et al. (2018) Pramod K. Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. 2018.  Did the model understand the question?. In _ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)_ , Vol. 1. 1896–1906.  https://doi.org/10.18653/v1/p18-1176
  * Obermeyer et al. (2019) Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019.  Dissecting racial bias in an algorithm used to manage the health of populations.  _Science_ 366, 6464 (10 2019), 447–453.  https://doi.org/10.1126/science.aax2342
  * Papineni et al. (2001) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001\.  BLEU: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02_ , Vol. 371. Association for Computational Linguistics, Morristown, NJ, USA, 311.  https://doi.org/10.3115/1073083.1073135
  * Park et al. (2017) Sungjoon Park, JinYeong Bak, and Alice Oh. 2017.  Rotated Word Vector Representations and their Interpretability. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Stroudsburg, PA, USA, 401–411.  https://doi.org/10.18653/v1/D17-1041
  * Pearl (2001) Judea Pearl. 2001\.  Direct and Indirect Effects. In _Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence_ _(UAI’01)_. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 411–420.  https://doi.org/10.5555/2074022.2074073
  * Pearson (1901) Karl Pearson. 1901\.  LIII. On lines and planes of closest fit to systems of points in space.  _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ 2, 11 (11 1901), 559–572.  https://doi.org/10.1080/14786440109462720
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global Vectors for Word Representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Stroudsburg, PA, USA, 1532–1543.  https://doi.org/10.3115/v1/D14-1162
  * Pruthi et al. (2020a) Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C. Lipton. 2020a.  Learning to Deceive with Attention-Based Explanations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, 4782–4793.  https://doi.org/10.18653/v1/2020.acl-main.432
  * Pruthi et al. (2020b) Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. 2020b.  Estimating Training Data Influence by Tracing Gradient Descent. In _Advances in Neural Information Processing Systems_.  http://arxiv.org/abs/2002.08484
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving Language Understanding by Generative Pre-Training.  _OpenAI_ (2018).  https://openai.com/blog/language-unsupervised/
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.  https://openai.com/blog/better-language-models/
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ 21 (2020), 1–67.  https://jmlr.org/papers/v21/20-074.html
  * Rajani et al. (2019) Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Explain Yourself! Leveraging Language Models for Commonsense Reasoning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, 4932–4942.  https://doi.org/10.18653/v1/P19-1487
  * Ramamurthy et al. (2020) Karthikeyan Natesan Ramamurthy, Bhanukiran Vinzamuri, Yunfeng Zhang, and Amit Dhurandhar. 2020.  Model Agnostic Multilevel Explanations.  _arXiv_ (3 2020).  http://arxiv.org/abs/2003.06005
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  "Why should i trust you?" Explaining the predictions of any classifier. In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , Vol. 13-17-Augu. ACM, New York, NY, USA, 1135–1144.  https://doi.org/10.1145/2939672.2939778
  * Ribeiro et al. (2018a) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018a.  Anchors: High-precision model-agnostic explanations. In _32nd AAAI Conference on Artificial Intelligence, AAAI 2018_. 1527–1535.  https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16982
  * Ribeiro et al. (2018b) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018b.  Semantically Equivalent Adversarial Rules for Debugging NLP models. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Vol. 1. Association for Computational Linguistics, Stroudsburg, PA, USA, 856–865.  https://doi.org/10.18653/v1/P18-1079
  * Robnik-Šikonja and Bohanec (2018) Marko Robnik-Šikonja and Marko Bohanec. 2018.  Perturbation-Based Explanations of Prediction Models.  Springer, Cham, 159–175.  https://doi.org/10.1007/978-3-319-90403-0{_}9
  * Rogers et al. (2020) Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020\.  A Primer in BERTology: What We Know About How BERT Works.  _Transactions of the Association for Computational Linguistics_ 8 (12 2020), 842–866.  https://doi.org/10.1162/tacl{_}a{_}00349
  * Ross et al. (2020) Alexis Ross, Ana Marasović, and Matthew E. Peters. 2020\.  Explaining NLP Models via Minimal Contrastive Editing (MiCE).  _arXiv_ (12 2020).  http://arxiv.org/abs/2012.13985
  * Rudin (2019) Cynthia Rudin. 2019\.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ 1, 5 (2019), 206–215.  https://doi.org/10.1038/s42256-019-0048-x
  * Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020.  WinoGrande: An Adversarial Winograd Schema Challenge at Scale.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 34, 05 (4 2020), 8732–8740.  https://doi.org/10.1609/aaai.v34i05.6399
  * Sangroya et al. (2020) Amit Sangroya, Mouli Rastogi, C Anantaram, and Lovekesh Vig. 2020.  Guided-LIME: Structured sampling based hybrid approach towards explaining blackbox machine learning models. In _CEUR Workshop Proceedings_ , Vol. 2699.
  * Schölkopf et al. (2001) Bernhard Schölkopf, Ralf Herbrich, and Alex J. Smola. 2001\.  A Generalized Representer Theorem.  In _International Conference on Computational Learning Theory_. Springer, 416–426.  https://doi.org/10.1007/3-540-44581-1{_}27
  * Serrano and Smith (2019) Sofia Serrano and Noah A Smith. 2019.  Is Attention Interpretable?. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, 2931–2951.  https://doi.org/10.18653/v1/P19-1282
  * Shapley (1953) Shapley. 1953\.  A value for N-Person Games.  _Contributions to the Theory of Games (AM-28), Volume II_ (1953), 307–317.  https://apps.dtic.mil/dtic/tr/fulltext/u2/604084.pdf
  * Shi et al. (2016) Xing Shi, Inkit Padhi, and Kevin Knight. 2016.  Does string-based neural MT learn source syntax?. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Stroudsburg, PA, USA, 1526–1534.  https://doi.org/10.18653/v1/d16-1159
  * Sinha et al. (2021) Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2021.  UnNatural Language Inference. In _ACL 2021 - 59th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_.  http://arxiv.org/abs/2101.00010
  * Slack et al. (2020) Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020.  Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. ACM, New York, NY, USA, 180–186.  https://doi.org/10.1145/3375627.3375830
  * Socher et al. (2013) Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013.  Parsing with compositional vector grammars.  In _ACL 2013 - 51st Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_. Vol. 1. Association for Computational Linguistics, 455–465.  https://aclanthology.org/P13-1045/
  * Strobelt et al. (2019) Hendrik Strobelt, Sebastian Gehrmann, Michael Behrisch, Adam Perer, Hanspeter Pfister, and Alexander M. Rush. 2019.  Seq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models.  _IEEE Transactions on Visualization and Computer Graphics_ 25, 1 (2019), 353–363.  https://doi.org/10.1109/TVCG.2018.2865044
  * Subramanian et al. (2020) Sanjay Subramanian, Ben Bogin, Nitish Gupta, Tomer Wolfson, Sameer Singh, Jonathan Berant, and Matt Gardner. 2020.  Obtaining Faithful Interpretations from Compositional Neural Networks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, 5594–5608.  https://doi.org/10.18653/v1/2020.acl-main.495
  * Sun et al. (2021) Xiaofei Sun, Diyi Yang, Xiaoya Li, Tianwei Zhang, Yuxian Meng, Han Qiu, Guoyin Wang, Eduard Hovy, and Jiwei Li. 2021.  Interpreting Deep Learning Models in Natural Language Processing: A Review.  (2021), 1–36.  http://arxiv.org/abs/2110.10470
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks. In _34th International Conference on Machine Learning, ICML 2017_ , Vol. 7. 5109–5118.  http://arxiv.org/abs/1703.01365
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In _Proceedings of the 2019 Conference of the North_. Association for Computational Linguistics, Stroudsburg, PA, USA, 4149–4158.  https://doi.org/10.18653/v1/N19-1421
  * Tenney et al. (2019a) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a.  BERT Rediscovers the Classical NLP Pipeline. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, 4593–4601.  https://doi.org/10.18653/v1/P19-1452
  * Tenney et al. (2020) Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, and Ann Yuan. 2020.  The language interpretability tool: Extensible, interactive visualizations and analysis for NLP models.  _arXiv_ (8 2020).  https://doi.org/10.18653/v1/2020.emnlp-demos.15
  * Tenney et al. (2019b) Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b.  What do you learn from context? Probing for sentence structure in contextualized word representations. In _7th International Conference on Learning Representations, ICLR 2019_. 1–17.  https://openreview.net/forum?id=SJzSgnRcKX
  * Tjoa and Guan (2020) Erico Tjoa and Cuntai Guan. 2020.  A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI.  _IEEE Transactions on Neural Networks and Learning Systems_ 14, 8 (2020), 1–21.  https://doi.org/10.1109/TNNLS.2020.3027314
  * Van Der Maaten and Hinton (2008) Laurens Van Der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ 9 (2008).  https://www.jmlr.org/papers/v9/vandermaaten08a.html
  * Vashishth et al. (2019) Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019.  Attention Interpretability Across NLP Tasks.  _arXiv_ (9 2019).  http://arxiv.org/abs/1909.11218
  * Vig et al. (2020) Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020.  Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In _Advances in Neural Information Processing Systems_ , H Larochelle, M Ranzato, R Hadsell, M F Balcan, and H Lin (Eds.), Vol. 33. Curran Associates, Inc., 12388–12401.  https://proceedings.neurips.cc/paper/2020/file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf
  * Voita and Titov (2020) Elena Voita and Ivan Titov. 2020.  Information-Theoretic Probing with Minimum Description Length. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Stroudsburg, PA, USA, 183–196.  https://doi.org/10.18653/v1/2020.emnlp-main.14
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal Adversarial Triggers for Attacking and Analyzing NLP. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Stroudsburg, PA, USA, 2153–2162.  https://doi.org/10.18653/v1/D19-1221
  * Wallace et al. (2020) Eric Wallace, Matt Gardner, and Sameer Singh. 2020\.  Interpreting Predictions of NLP Models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts_. Association for Computational Linguistics, Stroudsburg, PA, USA, 20–23.  https://doi.org/10.18653/v1/2020.emnlp-tutorials.3
  * Wang et al. (2019a) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=rJ4km2R5t7
  * Wang et al. (2019b) Wenqi Wang, Run Wang, Lina Wang, Zhibo Wang, and Aoshuang Ye. 2019b.  Towards a Robust Deep Neural Network in Texts: A Survey.  (2019), 1–22.  http://arxiv.org/abs/1902.07285
  * Wiegreffe and Marasović (2021) Sarah Wiegreffe and Ana Marasović. 2021.  Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing.  _Arxiv_ (2 2021).  http://arxiv.org/abs/2102.12060
  * Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019.  Attention is not not Explanation.  _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ (8 2019), 11–20.  https://doi.org/10.18653/v1/D19-1002
  * Wieting and Gimpel (2018) John Wieting and Kevin Gimpel. 2018.  ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Stroudsburg, PA, USA, 451–462.  https://doi.org/10.18653/v1/P18-1042
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018\.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , Vol. 1. Association for Computational Linguistics, Stroudsburg, PA, USA, 1112–1122.  https://doi.org/10.18653/v1/N18-1101
  * Williams et al. (2016) Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z. Gajos, Walter S. Lasecki, and Neil Heffernan. 2016.  AXIS. In _Proceedings of the Third (2016) ACM Conference on Learning @ Scale_. ACM, New York, NY, USA, 379–388.  https://doi.org/10.1145/2876034.2876042
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020\.  Transformers: State-of-the-Art Natural Language Processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Association for Computational Linguistics, Stroudsburg, PA, USA, 38–45.  https://doi.org/10.18653/v1/2020.emnlp-demos.6
  * Wu et al. (2021) Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S. Weld. 2021.  Polyjuice: Automated, General-purpose Counterfactual Generation.  _arXiv_ (1 2021).  http://arxiv.org/abs/2101.00288
  * Yeh et al. (2019) Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Suggala, David I Inouye, Pradeep K Ravikumar, Arun Sai Suggala, David I Inouye, and Pradeep K Ravikumar. 2019.  On the (In)fidelity and Sensitivity of Explanations.  In _Advances in Neural Information Processing Systems 32_ , H Wallach, H Larochelle, A Beygelzimer, F d\textquotesingle Alché-Buc, E Fox, and R Garnett (Eds.). Curran Associates, Inc., Vancouver, Canada, 10967–10978.  http://papers.nips.cc/paper/9278-on-the-infidelity-and-sensitivity-of-explanations.pdf
  * Yeh et al. (2018) Chih-Kuan Yeh, Joon Sik Kim, Ian E. H. Yen, and Pradeep Ravikumar. 2018.  Representer Point Selection for Explaining Deep Neural Networks. In _Advances in Neural Information Processing Systems_. 9291–9301.  http://arxiv.org/abs/1811.09720
  * Zhang and Bowman (2018) Kelly Zhang and Samuel Bowman. 2018.  Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. Association for Computational Linguistics, Stroudsburg, PA, USA, 359–361.  https://doi.org/10.18653/v1/W18-5448
  * Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019\.  PAWS: Paraphrase adversaries from word scrambling. In _Proceedings of the 2019 Conference of the North_. Association for Computational Linguistics, Stroudsburg, PA, USA, 1298–1308.  https://doi.org/10.18653/v1/N19-1131