# Invited: Neuromorphic architectures based on augmented silicon photonics
platforms

Matěj Hejda  0000-0003-4493-9426 Hewlett Packard Labs, Hewlett Packard
EnterpriseHermeslaan 1ADiegemBelgium<EMAIL_ADDRESS>,  Federico Marchesin
Ghent University - imecGhentBelgium<EMAIL_ADDRESS>,  George
Papadimitriou  Department of Informatics and Telecomm., University of Athens30
Shuangqing RdAthensGreece<EMAIL_ADDRESS>,  Dimitris Gizopoulos
Department of Informatics and Telecomm., University of Athens30 Shuangqing
RdAthensGreece<EMAIL_ADDRESS>,  Benoit Charbonnier  Univ. Grenoble Alpes,
CEA, LETI8600 Datapoint DriveGrenobleFrance<EMAIL_ADDRESS>,  Régis
Orobtchouk  Univ. Lyon, ECL, INSA-Lyon, UCBL, CPE Lyon, CNRS, INLLyonFrance
<EMAIL_ADDRESS>,  Peter Bienstman  Ghent University - imec1
Thørväld CircleGhentBelgium<EMAIL_ADDRESS>,  Thomas Van Vaerenbergh
Hewlett Packard Labs, Hewlett Packard EnterpriseHermeslaan 1ADiegemBelgium
<EMAIL_ADDRESS>and  Fabio Pavanello  0000-0002-6889-2012 Univ.
Grenoble Alpes, Univ. Savoie Mont Blanc, CNRS, INPG, CROMA3 parvis Louis Néel,
CS 50257GrenobleFrance38016<EMAIL_ADDRESS>
(2024)

###### Abstract.

In this work, we discuss our vision for neuromorphic accelerators based on
integrated photonics within the framework of the Horizon Europe NEUROPULS
project. Augmented integrated photonic architectures that leverage phase-
change and III-V materials for optical computing will be presented. A CMOS-
compatible platform will be discussed that integrates these materials to
fabricate photonic neuromorphic architectures, along with a gem5-based
simulation platform to model accelerator operation once it is interfaced with
a RISC-V processor. This simulation platform enables accurate system-level
accelerator modeling and benchmarking in terms of key metrics such as speed,
energy consumption, and footprint.

Artificial Intelligence, Photonics, Computing, Neuromorphic, AI accelerators,
gem5, Accelerators modeling, Simulation

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
61st ACM/IEEE Design Automation Conference; June 23–27, 2024; San Francisco,
CA††isbn: 978-1-4503-XXXX-X/18/06††ccs: Hardware Neural systems††ccs: Applied
computing Physics††ccs: Computing methodologies Modeling and simulation

##  1\. Introduction

Recent advances in deep learning (and more recently genAI) empower machines
with remarkable information processing and synthesis capabilities. At the same
time, the exponential growth in data volumes generated by an extensive range
of consumer devices and industrial sensors calls for novel approaches to
advanced and efficient data processing. Although advanced AI models are
typically deployed in purpose-built high-performance cloud clusters, some
applications require advanced data processing locally at the edge, that is,
closer to where data were originally generated (Cao et al., 2020). More
specifically, edge computing requires the development of lightweight
accelerators capable of providing AI-tailored data processing locally with low
latency and high energy efficiency (Satyanarayanan, 2017).

Figure 1. CMOS-compatible augmented platform in the NEUROPULS project.
Reprinted with permissions from (Pavanello et al., 2023). IEEE©

Current state-of-the-art AI is enabled by graphical processing units (GPUs),
tensor processing cores and similar application-specific hardware that allows
for high computing parallelization and optimized computation of linear
algebraic operations that underpin modern deep learning workloads (Samek et
al., 2021). In line with other conventional computing approaches, these
digital architectures utilize Von Neumann processor architectures with
distinct memory and computing blocks. This poses some inherent limitations in
terms of high demands on data movement between these functional blocks in
terms of bandwidth, energy efficiency, and power consumption.

The goal of alleviating these bottlenecks is one of the main objectives in
fields such as in-memory computing, where (a part of) the data needed for the
computation is co-located with the computing core architecture, thus requiring
less data movement (Zhou et al., 2023). In neuromorphic (brain-inspired)
engineering, the principle of memory-compute co-location is further enhanced
with signaling and information processing principles analogous to those
observed in the brain, which promises further improvements in computation
sparsity and energy efficiency (Frenkel et al., 2023). Although significant
efforts have been devoted to leveraging existing CMOS technology to develop
digital and mixed-signal neuromorphic computing cores (Höppner et al., 2022;
Davies et al., 2021), alternative technologies such as spintronics or
photonics are nowadays gaining considerable momentum for the implementation of
next-generation computing hardware (Grollier et al., 2020; Shastri et al.,
2021). However, there are still certain practical limitations towards
achieving accelerators based on integrated photonics. Current CMOS-compatible
silicon photonic platforms do not provide a full range of required functional
modalities. In addition, photonic accelerators are often investigated
standalone, rather than being interfaced with a system-level architecture and
used in real-world scenarios. Finally, full-scale, system-level simulation
platforms of a photonic accelerator interfaced with a processor core remain
mostly unexplored. In the next sections, we will discuss how we are aiming to
tackle all these aspects within the framework of the NEUROPULS project.

##  2\. CMOS-compatible SiPh platform

Unlike in the electronic neuromorphic approaches, photonics allow one to
leverage desirable properties of lightwaves such as wavelength multiplexing,
low-loss signal propagation without Joule heating as well as access to very
high bandwidth devices (above tens of GHz) to e.g. encode or read-out data
(Tait et al., 2019). In particular, integrated photonics has emerged as a
promising size, weight and power (SWaP)-optimized platform. Silicon photonics
(SiPh) currently represents arguably the most promising approach to photonic
integration due to its compatibility with existing CMOS approaches for cost
and volume. However, certain functionalities are not available in pure
Silicon-On-Insulator (SOI) platforms. In particular, the ability to realize
non-volatile optical memory elements is one of such missing functionalities
that is key to achieving energy-efficient optical computing architectures
(Feldmann et al., 2019, 2021). In addition, active devices (such as lasers)
cannot be realized in silicon because of its indirect bandgap. Therefore,
additional materials such as III-V compound semiconductors need to be co-
integrated into SOI platforms, typically using heterogeneous or hybrid
integration methods. A monolithic fabrication approach is the most desirable
way to achieve these missing functionalities in a compact manner with
excellent alignment tolerances between patterned layers and consequent ease of
coupling, as well as packaging costs lower than those of hybrid approaches.
Such platform has not yet been developed or presented as an open access
service, as is the case for pure SOI platforms (Rahim et al., 2018).

One of the goals of the Horizon Europe NEUROPULS project is to develop a
platform that can accommodate these additional functionalities in a monolithic
manner. In Figure 1, the integration strategies for PCMs (e.g., GSST or other
types) and III-V materials are shown. This integration does not affect other
building blocks, such as high-speed modulators and detectors (above 50 GHz
bandwidth) that have already been developed for the platform and are already
provided to the end users. Therefore, novel building blocks taking advantage
of these additional functionalities will be developed to further enhance the
existing selection of building blocks on the SOI platform.

##  3\. PCM/III-V augmented SOI building blocks

One of the key building blocks for broadband neuromorphic photonic
architectures is the Mach-Zenhder interferometer (MZI, shown in Figure 2(a)).
An individual MZI consists of couplers and phase-shifters (PS) and represents
a $SU(2)$ transformation. Typically in SOI, a specific phase-shift is induced
through the thermo-optic effect via an adjacent heater, and continuously
consumes electrical power. Given that this phase-shift remains constant for a
set weight matrix (that is, during inference), a non-volatile approach would
be ideal to remove this constant energy consumption (Miscuglio et al., [n.
d.]). Furthermore, besides the non-volatile nature of optical phase shift or
attenuation effects, the devices should be compact with minimized optical loss
to enable deep arrangements of MZIs. One of the goals of NEUROPULS is the
development of low-loss, compact, and reconfigurable multilevel PCM-based MZIs
with heaters above PCM patches and waveguides (see Figure 2(a)). Various
approaches are currently being investigated to benefit from the properties of
PCMs such as GeSe and GSST that present a large figure of merit (FOM) given by
$\delta n/\delta k$, where $\delta n$ and $\delta k$ are the refractive index
contrasts for the real and imaginary part, respectively, around the standard
telecom wavelength of 1550 nm (Soref et al., 2015; Dory et al., 2020).

Figure 2. (a) MZI with the PCM-augmented (in green) non-volatile optical
phase-shifters with heaters on top (in yellow) for programmability. (b) An
example of an MZI mesh architecture (here implementing $8\times 8$ matrix)
dedicated to accelerating matrix-vector multiplication operations via in-
memory optical computing.

Furthermore, Q-switched III-V on-chip lasers are explored as chipscale
excitable spiking sources. Pioneering works in this direction have already
been carried out; however, spikes were generated off-chip, unlike the approach
that we will be pursuing in NEUROPULS (Feldmann et al., 2019). By leveraging
the ultrafast response (sub-ns) and accumulation behavior of PCM-based devices
to optical pulses, the viability of photonic spiking neural networks (SNN) and
bio-inspired learning rules such as spike-timing dependent plasticity (STDP)
will be investigated.

##  4\. Computing architectures

The main focus of the photonic neuromorphic architecture is to realize optical
in-memory acceleration of linear algebra operations that underpin a majority
of current deep learning models. In particular, we focus on realizing a
photonic matrix-vector multiplication (MVM) engine to enable a generalized
matrix-matrix (GeMM) core. These architectures are based on meshes of
programmable integrated MZIs that operate as multiport interferometers with a
degree of matrix expressivity (universality) determined by component
arrangement. Within the NEUROPULS project, various mesh architectures of MZIs
(or, more generally, couplers and phase-shifters) are investigated and
evaluated. These include previously proposed mesh architectures such as the
Clements (Clements et al., 2016) architecture with compacted interferometers
(Bell and Walmsley, 2021) (shown in Figure 2(b)) or the Fldzhyan (Fldzhyan et
al., 2020) architecture with parallel PS blocks (Bell and Walmsley, 2021), as
well as newly proposed multiport interferometer architectures. In these, input
vectors are encoded into amplitude/phase of individual inputs (typically using
high-speed Mach Zehnder modulators), and the multiplication (weighting) matrix
is encoded in the state of the programmable PS blocks. Generalization to GeMM
operations can be realized through separating of the input matrix into rows,
and processing those either via time-division multiplexing or through encoding
into multiple dense wavelength division multiplexed (DWDM) channels that can
be processed in parallel in a single multiport interferometer without
incurring additional resource costs.

##  5\. Simulation platform

In the ever-evolving landscape of computing systems, the integration of
neuromorphic accelerators and hardware security primitives on a consolidated
simulation platform is crucial. Both electronic and photonic domain-specific
accelerators (DSAs) continue to be active areas of research and development as
the demand for specialized and efficient computing solutions grows across
various industries

In the NEUROPULS project, we will: (a) create efficient full system simulation
tools on top of the gem5 simulator (Binkert et al., 2011) to model and
evaluate complete computing systems with neuromorphic accelerators and
security primitives, as shown in Figure 3; (b) explore the diverse design
space of heterogeneous computing systems employing photonic neuromorphic
accelerators and hardware primitives; and (c) facilitate detailed system-level
evaluation of both software and hardware, with a specific emphasis on the
security properties of the computing platform.

Figure 3. gem5-based system architecture modeling and simulation
infrastructure overview.

NEUROPULS simulator is based on gem5, an open-source, system-level computer
architecture simulator that provides a flexible and modular framework for
modeling and simulating various aspects of computer systems. It supports
cycle-level simulation of a wide range of computer architectures, including
x86, Arm, RISC-V, and others, making it a versatile platform at the system
level (including the microarchitecture, architecture, operating systems, and
application layers of the computing stack). Given the tremendous growth of the
RISC-V ecosystem in the past few years, we have taken gem5-SALAM, which uses
an advanced dynamic graph execution engine based on LLVM (Rogers et al., 2020)
and only supports Arm ISA processor cores (the CPU part in Figure 3), and
ported it to support the RISC-V ISA and system configuration. We introduce
gem5-MARVEL (Chatzopoulos et al., 2024), which is based on LLVM IR
(Intermediate Representation) to model DSAs using C descriptions of their
functionality. The gem5-based simulation infrastructure comprises two core
components: the Compute Unit and the Communications Interface. The Compute
Unit represents the custom accelerator’s datapath, while the Communications
Interface facilitates memory access, control, and synchronization through
memory access ports, Memory-Mapped Registers (MMRs), and interrupt lines. The
memory access ports allow parallel access to different memory types, such as
scratchpad memories (SPMs) and register banks (these two types of memories
occupy the largest part of the area of many accelerators). MMRs consist of
configurable status, control, and data registers, allowing low-level device
configuration and facilitating communication between the accelerator and the
host, as well as between multiple accelerators (i.e., processing elements -
PEs) in a cluster (as shown in the right-most side of Figure 3). By treating
the accelerator as a memory-mapped device, the host can utilize the provided
interrupt signals for synchronization without the need for constant polling.
Additionally, the gem5-based infrastructure includes Direct Memory Access
(DMA) devices and custom memories that can be seamlessly integrated into
accelerator designs, enhancing its versatility. gem5-MARVEL is also a fault
injection framework that operates at the microarchitecture level and supports
transient and permanent fault injections to all hardware structures of the CPU
and for the three prevailing ISAs (Arm, x86, RISC-V). The fault injection
feature was implemented in the simulation framework to support the reliability
aspect of the NEUROPULS project.

##  6\. Conclusions

Integrated photonics represent one of the technological platforms with
potential to empower modern heterogeneous computing systems by enabling high
bandwidth and improved energy efficiency during data movement and computing.
While silicon photonics benefits from compatibility with CMOS technology,
further augmentation with additional material platforms is required to unlock
the full scale of needed chipscale optical building blocks. In the NEUROPULS
project, we have augmented a CMOS-compatible SOI photonic platform with III-V
semiconductor technology (enabling on-chip active optical devices), and
chalcogenide-based PCMs (to realize non-volatile optical modulators). Using
this augmented platform, we propose and evaluate a system-level implementation
of a neuromorphic accelerator based on an in-memory photonic MVM accelerator
core. Various MZI mesh architectures are evaluated for the MVM core, including
their performance, matrix expressivity and robustness. Furthermore, we have
developed a novel gem5-based simulation framework with RISC-V ISA support to
allow for extensive performance evaluation and benchmarking of the complete
photonic-enabled accelerator interfaced with controllers and processors. We
believe this comprehensive system-level implementation is key to realize
practical, photonic neuromorphic accelerator.

###### Acknowledgements.

This work has received funding from the European Union’s Horizon Europe
research and innovation program under grant agreement No. 101070238. Views and
opinions expressed are however those of the author(s) only and do not
necessarily reflect those of the European Union. Neither the European Union
nor the granting authority can be held responsible for them.

## References

  * (1)
  * Bell and Walmsley (2021) B. A. Bell and I. A. Walmsley. 2021.  Further Compactifying Linear Optical Unitaries.  _APL Photonics_ 6, 7 (July 2021), 070804.  https://doi.org/10.1063/5.0053421
  * Binkert et al. (2011) Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K. Reinhardt, Ali Saidi, et al. 2011\.  The Gem5 Simulator.  _SIGARCH Comput. Archit. News_ 39, 2 (aug 2011), 1–7.  https://doi.org/10.1145/2024716.2024718
  * Cao et al. (2020) Keyan Cao, Yefan Liu, Gongjie Meng, and Qimeng Sun. 2020.  An Overview on Edge Computing Research.  _IEEE Access_ 8 (2020), 85714–85728.  https://doi.org/10.1109/ACCESS.2020.2991734
  * Chatzopoulos et al. (2024) Odysseas Chatzopoulos, George Papadimitriou, Vasileios Karakostas, and Dimitris Gizopoulos. 2024.  gem5-MARVEL: Microarchitecture-Level Resilience Analysis of Heterogeneous SoC Architectures. In _IEEE International Symposium on High-Performance Computer Architecture (HPCA 2024)_. 543–559.  https://doi.org/10.1109/HPCA57654.2024.00047
  * Clements et al. (2016) William R. Clements, Peter C. Humphreys, Benjamin J. Metcalf, W. Steven Kolthammer, and Ian A. Walsmley. 2016.  Optimal design for universal multiport interferometers.  _Optica_ 3, 12 (Dec. 2016), 1460.  https://doi.org/10.1364/OPTICA.3.001460
  * Davies et al. (2021) Mike Davies, Andreas Wild, Garrick Orchard, Yulia Sandamirskaya, Gabriel A. Fonseca Guerra, et al. 2021\.  Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook.  _Proc. IEEE_ 109, 5 (May 2021), 911–934.  https://doi.org/10.1109/JPROC.2021.3067593 Conference Name: Proceedings of the IEEE.
  * Dory et al. (2020) J.-B. Dory, C. Castro-Chavarria, A. Verdy, J.-B. Jager, M. Bernard, et al. 2020\.  Ge–Sb–S–Se–Te amorphous chalcogenide thin films towards on-chip nonlinear photonic devices.  _Sci Rep_ 10, 1 (July 2020), 11894.  https://doi.org/10.1038/s41598-020-67377-9
  * Feldmann et al. (2021) J. Feldmann, N. Youngblood, M. Karpov, H. Gehring, X. Li, et al. 2021\.  Parallel convolutional processing using an integrated photonic tensor core.  _Nature_ 589, 7840 (Jan. 2021), 52–58.  https://doi.org/10.1038/s41586-020-03070-1
  * Feldmann et al. (2019) J. Feldmann, N. Youngblood, C. D. Wright, H. Bhaskaran, and W. H. P. Pernice. 2019.  All-optical spiking neurosynaptic networks with self-learning capabilities.  _Nature_ 569, 7755 (May 2019), 208–214.  https://doi.org/10.1038/s41586-019-1157-8
  * Fldzhyan et al. (2020) S. A. Fldzhyan, M. Yu Saygin, and S. P. Kulik. 2020.  Optimal Design of Error-Tolerant Reprogrammable Multiport Interferometers.  _Optics Letters_ 45, 9 (May 2020), 2632–2635.  https://doi.org/10.1364/OL.385433
  * Frenkel et al. (2023) Charlotte Frenkel, David Bol, and Giacomo Indiveri. 2023.  Bottom-Up and Top-Down Approaches for the Design of Neuromorphic Processing Systems: Tradeoffs and Synergies Between Natural and Artificial Intelligence.  _Proc. IEEE_ 111, 6 (June 2023), 623–652.  https://doi.org/10.1109/JPROC.2023.3273520
  * Grollier et al. (2020) J. Grollier, D. Querlioz, K. Y. Camsari, K. Everschor-Sitte, S. Fukami, et al. 2020\.  Neuromorphic spintronics.  _Nat Electron_ 3, 7 (July 2020), 360–370.  https://doi.org/10.1038/s41928-019-0360-9 Publisher: Nature Publishing Group.
  * Höppner et al. (2022) Sebastian Höppner, Yexin Yan, Andreas Dixius, Stefan Scholze, Johannes Partzsch, et al. 2022\.  The SpiNNaker 2 Processing Element Architecture for Hybrid Digital Neuromorphic Computing.  http://arxiv.org/abs/2103.08392
  * Miscuglio et al. ([n. d.]) Mario Miscuglio, Jiawei Meng, Omer Yesiliurt, Yifei Zhang, Ludmila J Prokopeva, et al. [n. d.].  Artificial Synapse with Mnemonic Functionality using GSST-based Photonic Integrated Memory.  ([n. d.]), 8.
  * Pavanello et al. (2023) Fabio Pavanello, Cedric Marchand, Ian O’Connor, Regis Orobtchouk, Fabien Mandorlo, et al. 2023\.  NEUROPULS: NEUROmorphic energy-efficient secure accelerators based on Phase change materials aUgmented siLicon photonicS.  _2023 IEEE European Test Symposium (ETS)_.  https://doi.org/10.1109/ETS56758.2023.10173974
  * Rahim et al. (2018) Abdul Rahim, Thijs Spuesens, Roel Baets, and Wim Bogaerts. 2018.  Open-Access Silicon Photonics: Current Status and Emerging Initiatives.  _Proc. IEEE_ 106, 12 (Dec. 2018), 2313–2330.  https://doi.org/10.1109/JPROC.2018.2878686 Conference Name: Proceedings of the IEEE.
  * Rogers et al. (2020) Samuel Rogers, Joshua Slycord, Mohammadreza Baharani, and Hamed Tabkhi. 2020.  gem5-SALAM: A System Architecture for LLVM-based Accelerator Modeling. In _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_. 471–482.  https://doi.org/10.1109/MICRO50266.2020.00047
  * Samek et al. (2021) Wojciech Samek, Gregoire Montavon, Sebastian Lapuschkin, Christopher J. Anders, and Klaus-Robert Muller. 2021.  Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications.  _Proc. IEEE_ 109, 3 (March 2021), 247–278.  https://doi.org/10.1109/JPROC.2021.3060483
  * Satyanarayanan (2017) Mahadev Satyanarayanan. 2017.  The Emergence of Edge Computing.  _Computer_ 50, 1 (Jan. 2017), 30–39.  https://doi.org/10.1109/MC.2017.9
  * Shastri et al. (2021) Bhavin J. Shastri, Alexander N. Tait, Thomas Ferreira de Lima, Wolfram H. P. Pernice, Harish Bhaskaran, et al. 2021\.  Photonics for Artificial Intelligence and Neuromorphic Computing.  _Nature Photonics_ 15, 2 (Feb. 2021), 102–114.  https://doi.org/10.1038/s41566-020-00754-y
  * Soref et al. (2015) Richard Soref, Joshua Hendrickson, Haibo Liang, Arka Majumdar, Jianwei Mu, et al. 2015\.  Electro-optical switching at 1550 nm using a two-state GeSe phase-change layer.  _Opt. Express, OE_ 23, 2 (Jan. 2015), 1536–1546.  https://doi.org/10.1364/OE.23.001536 Publisher: Optica Publishing Group.
  * Tait et al. (2019) Alexander N. Tait, Thomas Ferreira de Lima, Mitchell A. Nahmias, Heidi B. Miller, Hsuan-Tung Peng, et al. 2019\.  Silicon Photonic Modulator Neuron.  _Phys. Rev. Appl._ 11, 6 (June 2019), 064043.  https://doi.org/10.1103/PhysRevApplied.11.064043 Publisher: American Physical Society.
  * Zhou et al. (2023) Wen Zhou, Bowei Dong, Nikolaos Farmakidis, Xuan Li, Nathan Youngblood, et al. 2023\.  In-memory photonic dot-product engine with electrically programmable weight banks.  _Nat Commun_ 14, 1 (May 2023), 2887.  https://doi.org/10.1038/s41467-023-38473-x


* Kim et al. [2016] Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo.  Examples are not enough, learn to criticize! criticism for interpretability.  _Advances in neural information processing systems_ , 29, 2016.
  * Koh and Liang [2017] Pang Wei Koh and Percy Liang.  Understanding black-box predictions via influence functions.  In _International conference on machine learning_ , pages 1885–1894. PMLR, 2017.
  * Ustun and Rudin [2016] Berk Ustun and Cynthia Rudin.  Supersparse linear integer models for optimized medical scoring systems.  _Machine Learning_ , 102(3):349–391, 2016.
  * Wang et al. [2017] Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille.  A bayesian framework for learning rule sets for interpretable classification.  _The Journal of Machine Learning Research_ , 18(1):2357–2393, 2017.
  * Rudin [2019] Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  _Nature Machine Intelligence_ , 1(5):206–215, 2019.
  * Teso et al. [2023] Stefano Teso, Öznur Alkan, Wolfang Stammer, and Elizabeth Daly.  Leveraging explanations in interactive machine learning: An overview.  _Frontiers in Artificial Intelligence_ , 2023.
  * Kambhampati et al. [2022] Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and Lin Guan.  Symbols as a lingua franca for bridging human-ai chasm for explainable and advisable ai systems.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 12262–12267, 2022.
  * Kim et al. [2018] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.  Interpretability beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV).  In _International conference on machine learning_ , pages 2668–2677. PMLR, 2018.
  * Fong and Vedaldi [2018] Ruth Fong and Andrea Vedaldi.  Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8730–8738, 2018.
  * Ghorbani et al. [2019] Amirata Ghorbani, Abubakar Abid, and James Zou.  Interpretation of neural networks is fragile.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 3681–3688, 2019.
  * Zhang et al. [2021] Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein.  Invertible concept-based explanations for cnn models with non-negative concept activation vectors.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 11682–11690, 2021.
  * Fel et al. [2023a] Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre.  Craft: Concept recursive activation factorization for explainability.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2711–2721, 2023a.
  * Alvarez-Melis and Jaakkola [2018] David Alvarez-Melis and Tommi S Jaakkola.  Towards robust interpretability with self-explaining neural networks.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pages 7786–7795, 2018.
  * Chen et al. [2019] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su.  This looks like that: Deep learning for interpretable image recognition.  _Advances in Neural Information Processing Systems_ , 32:8930–8941, 2019.
  * Koh et al. [2020] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang.  Concept bottleneck models.  In _International Conference on Machine Learning_ , pages 5338–5348. PMLR, 2020.
  * Marconato et al. [2022] Emanuele Marconato, Andrea Passerini, and Stefano Teso.  Glancenets: Interpretabile, leak-proof concept-based models.  In _NeurIPS_ , 2022.
  * Espinosa Zarlenga et al. [2022] Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Liò, and Mateja Jamnik.  Concept embedding models.  2022\.
  * Lipton [2018] Zachary C Lipton.  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.  _Queue_ , 16(3):31–57, 2018.
  * Schwalbe [2022] Gesina Schwalbe.  Concept embedding analysis: A review.  _arXiv preprint arXiv:2203.13909_ , 2022.
  * Stammer et al. [2021] Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting.  Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3619–3629, 2021.
  * Bontempelli et al. [2023] Andrea Bontempelli, Stefano Teso, Fausto Giunchiglia, and Andrea Passerini.  Concept-level debugging of part-prototype networks.  In _International Conference on Learning Representations_ , 2023.
  * Hoffmann et al. [2021] Adrian Hoffmann, Claudio Fanconi, Rahul Rade, and Jonas Kohler.  This looks like that… does it? shortcomings of latent space prototype interpretability in deep networks.  _arXiv preprint arXiv:2105.02968_ , 2021.
  * Xu-Darme et al. [2023] Romain Xu-Darme, Georges Quénot, Zakaria Chihani, and Marie-Christine Rousset.  Sanity checks for patch visualisation in prototype-based image classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3690–3695, 2023.
  * Chen et al. [2020] Zhi Chen, Yijie Bei, and Cynthia Rudin.  Concept whitening for interpretable image recognition.  _Nature Machine Intelligence_ , 2(12):772–782, 2020.
  * Margeloiu et al. [2021] Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller.  Do concept bottleneck models learn as intended?  _arXiv preprint arXiv:2105.04289_ , 2021.
  * Mahinpei et al. [2021] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan.  Promises and pitfalls of black-box concept learning models.  In _International Conference on Machine Learning: Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI_ , volume 1, pages 1–13, 2021.
  * Silver and Mitchell [2023] Daniel L Silver and Tom M Mitchell.  The roles of symbols in neural-based ai: They are not what you think!  _arXiv preprint arXiv:2304.13626_ , 2023.
  * Schölkopf et al. [2021] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio.  Toward causal representation learning.  _Proceedings of the IEEE_ , 109(5):612–634, 2021\.
  * Bengio et al. [2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(8):1798–1828, 2013.
  * Higgins et al. [2018] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner.  Towards a definition of disentangled representations.  _arXiv preprint arXiv:1812.02230_ , 2018.
  * Beckers and Halpern [2019] Sander Beckers and Joseph Y Halpern.  Abstracting causal models.  In _Proceedings of the aaai conference on artificial intelligence_ , volume 33, pages 2678–2685, 2019.
  * Beckers et al. [2020] Sander Beckers, Frederick Eberhardt, and Joseph Y Halpern.  Approximate causal abstractions.  In _Uncertainty in Artificial Intelligence_ , pages 606–615. PMLR, 2020.
  * Geiger et al. [2023a] Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D Goodman.  Finding alignments between interpretable causal variables and distributed neural representations.  _arXiv preprint arXiv:2303.02536_ , 2023a.
  * Lockhart et al. [2022] Joshua Lockhart, Nicolas Marchesotti, Daniele Magazzeni, and Manuela Veloso.  Towards learning to explain with concept bottleneck models: mitigating information leakage.  _arXiv preprint arXiv:2211.03656_ , 2022.
  * Pearl [2009] Judea Pearl.  _Causality_.  Cambridge university press, 2009.
  * Peters et al. [2017] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf.  _Elements of causal inference: foundations and learning algorithms_.  2017\.
  * Eastwood and Williams [2018] Cian Eastwood and Christopher KI Williams.  A framework for the quantitative evaluation of disentangled representations.  In _International Conference on Learning Representations_ , 2018.
  * Suter et al. [2019] Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer.  Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness.  In _International Conference on Machine Learning_ , pages 6056–6065. PMLR, 2019.
  * Reddy et al. [2022] Abbavaram Gowtham Reddy, Vineeth N Balasubramanian, et al.  On causally disentangled representations.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 8089–8097, 2022.
  * von Kügelgen et al. [2021] Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello.  Self-supervised learning with data augmentations provably isolates content from style.  In _Proceedings of the 35nd International Conference on Neural Information Processing Systems_ , 2021.
  * Koller and Friedman [2009] Daphne Koller and Nir Friedman.  _Probabilistic graphical models: principles and techniques_.  MIT press, 2009.
  * Bontempelli et al. [2021] Andrea Bontempelli, Fausto Giunchiglia, Andrea Passerini, and Stefano Teso.  Toward a Unified Framework for Debugging Gray-box Models.  In _The AAAI-22 Workshop on Interactive Machine Learning_ , 2021.
  * Zarlenga et al. [2022] Mateo Espinosa Zarlenga, Barbiero Pietro, Ciravegna Gabriele, Marra Giuseppe, Francesco Giannini, Michelangelo Diligenti, Shams Zohreh, Precioso Frederic, Stefano Melacci, Weller Adrian, et al.  Concept embedding models: Beyond the accuracy-explainability trade-off.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 21400–21413. Curran Associates, Inc., 2022.
  * Fel et al. [2023b] Thomas Fel, Victor Boutin, Mazda Moayeri, Rémi Cadène, Louis Bethune, Léo andéol, Mathieu Chalvidal, and Thomas Serre.  A holistic approach to unifying automatic concept extraction and concept importance estimation, 2023b.
  * Teso [2019] Stefano Teso.  Toward faithful explanatory active learning with self-explainable neural nets.  In _Proceedings of the Workshop on Interactive Adaptive Learning (IAL 2019)_ , pages 4–16, 2019.
  * Pfau et al. [2021] Jacob Pfau, Albert T Young, Jerome Wei, Maria L Wei, and Michael J Keiser.  Robust semantic interpretability: Revisiting concept activation vectors.  _arXiv preprint arXiv:2104.02768_ , 2021.
  * Gabbay et al. [2021] Aviv Gabbay, Niv Cohen, and Yedid Hoshen.  An image is worth more than a thousand words: Towards disentanglement in the wild.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Matthey et al. [2017] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner.  dsprites: Disentanglement testing sprites dataset.  https://github.com/deepmind/dsprites-dataset/, 2017.
  * Havasi et al. [2022] Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez.  Addressing leakage in concept bottleneck models.  2022\.
  * Cover [1999] Thomas M Cover.  _Elements of information theory_.  John Wiley & Sons, 1999.
  * Montero et al. [2020] Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers.  The role of disentanglement in generalisation.  In _International Conference on Learning Representations_ , 2020.
  * Montero et al. [2022] Milton Montero, Jeffrey Bowers, Rui Ponte Costa, Casimir Ludwig, and Gaurav Malhotra.  Lost in latent space: Examining failures of disentangled models at combinatorial generalisation.  _Advances in Neural Information Processing Systems_ , 35:10136–10149, 2022.
  * Sun et al. [2020] Xin Sun, Zhenning Yang, Chi Zhang, Keck-Voon Ling, and Guohao Peng.  Conditional gaussian distribution learning for open set recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13480–13489, 2020.
  * Hyvarinen and Morioka [2017] Aapo Hyvarinen and Hiroshi Morioka.  Nonlinear ica of temporally dependent stationary sources.  In _Artificial Intelligence and Statistics_ , pages 460–469. PMLR, 2017.
  * Khemakhem et al. [2020a] Ilyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma, and Aapo Hyvärinen.  Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica, 2020a.
  * Rubenstein et al. [2017] Paul K Rubenstein, Sebastian Weichwald, Stephan Bongers, Joris M Mooij, Dominik Janzing, Moritz Grosse-Wentrup, and Bernhard Schölkopf.  Causal consistency of structural equation models.  _arXiv preprint arXiv:1707.00819_ , 2017.
  * Zennaro [2022] Fabio Massimo Zennaro.  Abstraction between structural causal models: A review of definitions and properties.  _arXiv preprint arXiv:2207.08603_ , 2022.
  * Geiger et al. [2023b] Atticus Geiger, Chris Potts, and Thomas Icard.  Causal abstraction for faithful model interpretation.  _arXiv preprint arXiv:2301.04709_ , 2023b.
  * Marti et al. [2023] Louis Marti, Shengyi Wu, Steven T Piantadosi, and Celeste Kidd.  Latent diversity in human concepts.  _Open Mind_ , 7:79–92, 2023.
  * Zaidi et al. [2020] Julian Zaidi, Jonathan Boilard, Ghyslain Gagnon, and Marc-André Carbonneau.  Measuring disentanglement: A review of metrics.  _arXiv preprint arXiv:2012.09276_ , 2020.
  * Eastwood et al. [2022] Cian Eastwood, Andrei Liviu Nicolicioiu, Julius Von Kügelgen, Armin Kekić, Frederik Träuble, Andrea Dittadi, and Bernhard Schölkopf.  Dci-es: An extended disentanglement framework with connections to identifiability.  _arXiv preprint arXiv:2210.00364_ , 2022.
  * Chen et al. [2018] Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud.  Isolating sources of disentanglement in vaes.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pages 2615–2625, 2018.
  * Locatello et al. [2019] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem.  Challenging common assumptions in the unsupervised learning of disentangled representations.  In _International Conference on Machine Learning_ , pages 4114–4124, 2019.
  * Oikarinen et al. [2022] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng.  Label-free concept bottleneck models.  In _ICLR_ , 2022.
  * Lage and Doshi-Velez [2020] Isaac Lage and Finale Doshi-Velez.  Learning interpretable concept-based models with human feedback.  _arXiv preprint arXiv:2012.02898_ , 2020.
  * Chauhan et al. [2023] Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham.  Interactive concept bottleneck models.  In _AAAI_ , 2023.
  * Muggleton and De Raedt [1994] Stephen Muggleton and Luc De Raedt.  Inductive logic programming: Theory and methods.  _The Journal of Logic Programming_ , 19:629–679, 1994.
  * De Raedt et al. [2020] Luc De Raedt, Sebastijan Dumancic, Robin Manhaeve, and Giuseppe Marra.  From statistical relational to neuro-symbolic artificial intelligence.  In _IJCAI_ , 2020.
  * Khemakhem et al. [2020b] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen.  Variational autoencoders and nonlinear ica: A unifying framework.  In _International Conference on Artificial Intelligence and Statistics_ , pages 2207–2217. PMLR, 2020b.
  * Graziani et al. [2023] Mara Graziani, An-phi Nguyen, Laura O’Mahony, Henning Müller, and Vincent Andrearczyk.  Concept discovery and dataset exploration with singular value decomposition.  In _ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML_ , 2023.
  * Li et al. [2018] Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin.  Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * Rymarczyk et al. [2021] Dawid Rymarczyk, Łukasz Struski, Jacek Tabor, and Bartosz Zieliński.  ProtoPShare: Prototypical Parts Sharing for Similarity Discovery in Interpretable Image Classification.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, page 1420–1430, 2021.
  * Nauta et al. [2021] Meike Nauta, Ron van Bree, and Christin Seifert.  Neural prototype trees for interpretable fine-grained image recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14933–14943, 2021.
  * Singh and Yow [2021] Gurmail Singh and Kin-Choong Yow.  These do not look like those: An interpretable deep learning model for image recognition.  _IEEE Access_ , 9:41482–41493, 2021.
  * Davoudi and Komeili [2021] Seyed Omid Davoudi and Majid Komeili.  Toward faithful case-based reasoning through learning prototypes in a nearest neighbor-friendly space.  In _International Conference on Learning Representations_ , 2021.
  * Zhou et al. [2018] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba.  Interpretable basis decomposition for visual explanation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 119–134, 2018.
  * Kazhdan et al. [2020] Dmitry Kazhdan, Botty Dimanov, Mateja Jamnik, Pietro Liò, and Adrian Weller.  Now you see me (cme): concept-based model extraction.  _arXiv preprint arXiv:2010.13233_ , 2020.
  * Gu and Tresp [2019] Jindong Gu and Volker Tresp.  Semantics for global and local interpretation of deep neural networks.  _arXiv preprint arXiv:1910.09085_ , 2019.
  * Esser et al. [2020] Patrick Esser, Robin Rombach, and Bjorn Ommer.  A disentangling invertible interpretation network for explaining latent representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9223–9232, 2020.
  * Yuksekgonul et al. [2022] Mert Yuksekgonul, Maggie Wang, and James Zou.  Post-hoc concept bottleneck models.  _arXiv preprint arXiv:2205.15480_ , 2022.
  * Sawada and Nakamura [2022] Yoshihide Sawada and Keigo Nakamura.  Concept bottleneck model with additional unsupervised concepts.  _IEEE Access_ , 2022.
  * Erculiani et al. [2023] Luca Erculiani, Andrea Bontempelli, Andrea Passerini, and Fausto Giunchiglia.  Egocentric hierarchical visual semantics.  _arXiv preprint arXiv:2305.05422_ , 2023.
  * Higgins et al. [2016] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  $\beta$-vae: Learning basic visual concepts with a constrained variational framework.  In _International Conference on Learning Representations_ , 2016.
  * Kingma and Welling [2014] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In _International conference on machine learning_. PMLR, 2014.
  * Kim and Mnih [2018] Hyunjik Kim and Andriy Mnih.  Disentangling by factorising.  In _International Conference on Machine Learning_ , pages 2649–2658. PMLR, 2018.
  * Esmaeili et al. [2019] Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, and Jan-Willem Meent.  Structured disentangled representations.  In _The 22nd International Conference on Artificial Intelligence and Statistics_ , pages 2525–2534. PMLR, 2019.
  * Rhodes and Lee [2021] Travers Rhodes and Daniel Lee.  Local disentanglement in variational auto-encoders using jacobian $l\\_1$ regularization.  _Advances in Neural Information Processing Systems_ , 34:22708–22719, 2021.
  * Locatello et al. [2020a] Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem.  Disentangling factors of variations using few labels.  In _International Conference on Learning Representations_ , 2020a.
  * Shu et al. [2020] Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Weakly supervised disentanglement with guarantees.  In _International Conference on Learning Representations_ , 2020.
  * Locatello et al. [2020b] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen.  Weakly-supervised disentanglement without compromises.  In _International Conference on Machine Learning_ , pages 6348–6359. PMLR, 2020b.
  * Lachapelle et al. [2022] Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien.  Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica.  In _Conference on Causal Learning and Reasoning_ , pages 428–484. PMLR, 2022.
  * Horan et al. [2021] Daniella Horan, Eitan Richardson, and Yair Weiss.  When is unsupervised disentanglement possible?  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Stammer et al. [2022] Wolfgang Stammer, Marius Memmel, Patrick Schramowski, and Kristian Kersting.  Interactive disentanglement: Learning concepts by interacting with their prototype representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10317–10328, 2022.
  * Comon [1994] Pierre Comon.  Independent component analysis, a new concept?  _Signal processing_ , 36(3):287–314, 1994.
  * Hyvärinen et al. [2001] Aapo Hyvärinen, Juha Karhunen, and Erkki Oja.  Independent component analysis, adaptive and learning systems for signal processing, communications, and control.  _John Wiley & Sons, Inc_, 1:11–14, 2001.
  * Naik and Kumar [2011] Ganesh R Naik and Dinesh K Kumar.  An overview of independent component analysis and its applications.  _Informatica_ , 35(1), 2011.
  * Hyvärinen and Pajunen [1999] Aapo Hyvärinen and Petteri Pajunen.  Nonlinear independent component analysis: Existence and uniqueness results.  _Neural networks_ , 12(3):429–439, 1999.
  * Buchholz et al. [2022] Simon Buchholz, Michel Besserve, and Bernhard Schölkopf.  Function classes for identifiable nonlinear independent component analysis.  _Advances in Neural Information Processing Systems_ , 35:16946–16961, 2022.
  * Yeh et al. [2020] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar.  On completeness-aware concept-based explanations in deep neural networks.  _Advances in neural information processing systems_ , 33:20554–20565, 2020.
  * Zarlenga et al. [2023] Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik.  Towards robust metrics for concept representation evaluation, 2023.
  * Manhaeve et al. [2018] Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt.  DeepProbLog: Neural Probabilistic Logic Programming.  _NeurIPS_ , 2018.
  * Donadello et al. [2017] Ivan Donadello, Luciano Serafini, and Artur D’Avila Garcez.  Logic tensor networks for semantic image interpretation.  _arXiv preprint arXiv:1705.08968_ , 2017.
  * Diligenti et al. [2017] Michelangelo Diligenti, Marco Gori, and Claudio Sacca.  Semantic-based regularization for learning and inference.  _Artificial Intelligence_ , 244:143–165, 2017.
  * Fischer et al. [2019] Marc Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, and Martin Vechev.  Dl2: Training and querying neural networks with logic.  In _International Conference on Machine Learning_ , pages 1931–1941. PMLR, 2019.
  * Giunchiglia and Lukasiewicz [2020] Eleonora Giunchiglia and Thomas Lukasiewicz.  Coherent hierarchical multi-label classification networks.  _NeurIPS_ , 2020.
  * Yang et al. [2020] Zhun Yang, Adam Ishay, and Joohyung Lee.  NeurASP: Embracing neural networks into answer set programming.  In _IJCAI_ , 2020.
  * Huang et al. [2021] Jiani Huang, Ziyang Li, Binghong Chen, Karan Samel, Mayur Naik, Le Song, and Xujie Si.  Scallop: From probabilistic deductive databases to scalable differentiable reasoning.  _NeurIPS_ , 2021.
  * Marra and Kuželka [2021] Giuseppe Marra and Ondřej Kuželka.  Neural markov logic networks.  In _Uncertainty in Artificial Intelligence_ , 2021.
  * Ahmed et al. [2022] Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck, and Antonio Vergari.  Semantic Probabilistic Layers for Neuro-Symbolic Learning.  In _NeurIPS_ , 2022.
  * Misino et al. [2022] Eleonora Misino, Giuseppe Marra, and Emanuele Sansone.  VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programming.  _NeurIPS_ , 2022.
  * Winters et al. [2022] Thomas Winters, Giuseppe Marra, Robin Manhaeve, and Luc De Raedt.  DeepStochLog: Neural Stochastic Logic Programming.  In _AAAI_ , 2022.
  * van Krieken et al. [2022] Emile van Krieken, Thiviyan Thanapalasingam, Jakub M Tomczak, Frank van Harmelen, and Annette ten Teije.  A-nesi: A scalable approximate method for probabilistic neurosymbolic inference.  _arXiv preprint arXiv:2212.12393_ , 2022.
  * Ciravegna et al. [2023] Gabriele Ciravegna, Pietro Barbiero, Francesco Giannini, Marco Gori, Pietro Lió, Marco Maggini, and Stefano Melacci.  Logic explained networks.  _Artificial Intelligence_ , 314:103822, 2023.
  * Marconato et al. [2023a] Emanuele Marconato, Gianpaolo Bontempo, Elisa Ficarra, Simone Calderara, Andrea Passerini, and Stefano Teso.  Neuro symbolic continual learning: Knowledge, reasoning shortcuts and concept rehearsal.  In _ICML_ , 2023a.
  * Marconato et al. [2023b] Emanuele Marconato, Stefano Teso, Antonio Vergari, and Andrea Passerini.  Not all neuro-symbolic concepts are created equal: Analysis and mitigation of reasoning shortcuts.  _arXiv preprint arXiv:2305.19951_ , 2023b.