indices. As both nested integrals in (371) are related by exchanging
$(\beta_{1},\beta_{2},-\beta_{x})\leftrightarrow(\beta_{3},\beta_{4},\beta_{x})$,
the singularity condition we obtain from the second nested integral is again
(374), and no cancellation can occur as both terms contribute with the same
sign.

The singularities of AdS exchange diagrams are thus predicted by the
combination of the ‘vertex’ and ‘total’ singularity conditions (372) and
(374). These conditions are sufficient to account for the degrees of
divergence (obtained by direct integration) in table 2. In particular, the
quadratic divergences arise when multiple singularity conditions are satisfied
simultaneously (see also Bzowski:2015pba ).

###  C.3 dS exchanges

The singularity conditions for dS exchanges can be obtained using the
holographic formula (3.5.2). Once again, taking the imaginary parts in this
formula acts to project out all singularities of AdS correlators that have a
purely ultralocal momentum dependence, namely those arising from any all-minus
condition. This projection can equivalently be seen from the zeros of the sine
factors in (3.5.2). The holographic formula relates the dS exchange diagram to
both the AdS exchange diagram, whose singularities are given by (372) and
(374), and a product of AdS 3-point functions, whose singularities are given
by (368) with $n=3$. From the AdS exchange contribution, the dS exchange
inherits the ‘total’ singularity condition

$d+\sum_{i=1}^{4}\sigma_{i}\beta_{i}=-2k_{T},\qquad k_{T}\in\mathbb{Z}^{+}$
(375)

where any independent choice of signs $\sigma_{i}\in\pm 1$ is permitted except
for the all-minus case. The ‘vertex’ singularity conditions arise from both
the AdS exchange contribution and the product of AdS 3-point functions, and
read

$\displaystyle\frac{d}{2}+\sigma_{1}\beta_{1}+\sigma_{2}\beta_{2}+\sigma_{x}^{L}\beta_{x}=-2k_{L},\quad\frac{d}{2}+\sigma_{3}\beta_{3}+\sigma_{4}\beta_{4}+\sigma_{x}^{R}\beta_{x}=-2k_{R},\quad
k_{L},\,k_{R}\in\mathbb{Z}^{+}.$ (376)

Note that both signs $\sigma_{x}^{L},\sigma_{x}^{R}\in\pm 1$ can appear here,
since while only the plus sign appears in the vertex condition for AdS
exchanges (372), both signs appear in the 3-point conditions (368). (This
explains why the dS amplitude $ds_{[22,22x3]}$ is divergent while the
corresponding AdS amplitude $i_{[22,22x3]}$ is finite, see table 2.) However,
the all-minus cases $(\sigma_{1},\sigma_{2},\sigma_{x}^{L})=(-,-,-)$ and
$(\sigma_{3},\sigma_{4},\sigma_{x}^{R})=(-,-,-)$ are excluded since the
corresponding AdS singularities are ultralocal and hence projected out when we
pass to dS. Moreover, an analysis using the shadow holographic formulae in the
following section shows that the $(\sigma_{1},\sigma_{2},\sigma_{x})=(-,-,+)$
and $(\sigma_{3},\sigma_{4},\sigma_{x})=(-,-,+)$ cases of (376) also excluded,
i.e, do not give rise to singularities of the dS exchange diagram. This is due
to a cancellation between the singularities of the AdS exchange and 3-point
contributions in the holographic formula (3.5.2) as we will see shortly.

The conditions above, (375) and (376) (where all cases for which
$(\sigma_{1},\sigma_{2},\sigma_{3},\sigma_{4})=(-,-,-,-)$ are excluded), are
sufficient to account for the degrees of divergence for dS exchange diagrams
in table 2. As above, quadratic divergences arise when more than one condition
is satisfied.

###  C.4 Compatibility with the shadow formula

The singularity conditions for dS exchanges can also be analysed via the
shadow formula (5.2), or equivalently (321) and (322). These formulae relate
the dS exchange to a linear combination of the AdS exchange and shadow
exchange diagrams, where the latter is obtained by replacing
$\beta_{x}\rightarrow-\beta_{x}$. Starting from the exchange singularity
conditions in AdS, (372) and (374), we then recover the dS singularity
conditions (376) and (375) after resolving the following two subtleties.

Firstly, we need to establish that the all-minus case of the ‘total’
singularity condition (375) is absent. This occurs due to a cancellation of
singularities between the AdS exchange and shadow exchange diagrams. The
‘total’ singularity for both these diagrams is the same, as can be seen from
the analysis in section C.2, with the cancellation then following from the
identity

$\displaystyle\sin\Big{[}\frac{\pi}{2}(\frac{d}{2}-\beta_{1}-\beta_{2}+\beta_{x})\Big{]}\sin\Big{[}\frac{\pi}{2}(\frac{d}{2}-\beta_{3}-\beta_{4}+\beta_{x})\Big{]}$
$\displaystyle\qquad-\sin\Big{[}\frac{\pi}{2}(\frac{d}{2}-\beta_{1}-\beta_{2}-\beta_{x})\Big{]}\sin\Big{[}\frac{\pi}{2}(\frac{d}{2}-\beta_{3}-\beta_{4}-\beta_{x})\Big{]}$
$\displaystyle=\sin(\pi\beta_{x})\sin\Big{[}\frac{\pi}{2}(d-\beta_{T})\Big{]}.$
(377)

The relative sign between the two terms on the left-hand side arises since the
sign of $\beta_{x}$ is flipped between the exchange and the shadow exchange,
and the denominator in (322) contains a factor of $\sin(\pi\beta_{x})$.

The second point to be understood is the absence of ‘vertex’-type dS
singularities corresponding to (376) for the cases
$(\sigma_{1},\sigma_{2},\sigma_{x})=(-,-,+)$ and
$(\sigma_{3},\sigma_{4},\sigma_{x})=(-,-,+)$. For these cases, the sine
factors appearing in the shadow formula (322) have zeros cancelling the
corresponding singularities, however an equivalent cancellation is not
immediately apparent in our earlier approach based on the holographic formula
(3.5.2).

Closer inspection shows, however, that a cancellation of singularities between
the AdS exchange and 3-point contributions to the holographic formula (3.5.2)
indeed occurs. When either $d/2-\beta_{1}-\beta_{2}+\beta_{x}=-2k_{L}$ and/or
$d/2-\beta_{3}-\beta_{4}+\beta_{x}=-2k_{R}$, the first term in the
trigonometric identity (C.4) vanishes. The numerator of the holographic
formula (3.5.2) then becomes

$\displaystyle\sin\left(\frac{\pi}{2}(\beta_{T}-d)\right)i^{\text{reg}}_{[\Delta_{1}\Delta_{2};\Delta_{3}\Delta_{4}x\Delta_{x}]}$
(378)
$\displaystyle\qquad+2a_{\beta_{x}}s^{-2\beta_{x}}i^{\text{reg}}_{[\Delta_{1}\Delta_{2}\Delta_{x}]}(q_{1},q_{2},s)i^{\text{reg}}_{[\Delta_{x}\Delta_{3}\Delta_{4}]}(s,q_{3},q_{4})$
$\displaystyle\qquad\qquad\times\sin\left(\frac{\pi}{2}(\frac{d}{2}-\beta_{1}-\beta_{2}-\beta_{x})\right)\sin\left(\frac{\pi}{2}(\frac{d}{2}-\beta_{3}-\beta_{4}-\beta_{x})\right)$
$\displaystyle=\sin\left(\frac{\pi}{2}(\beta_{T}-d)\right)\Big{[}i^{\text{reg}}_{[\Delta_{1}\Delta_{2};\Delta_{3}\Delta_{4}x\Delta_{x}]}$
$\displaystyle\qquad\qquad+2\sin(\pi\beta_{x})a_{\beta_{x}}s^{-2\beta_{x}}i^{\text{reg}}_{[\Delta_{1}\Delta_{2}\Delta_{x}]}(q_{1},q_{2},s)i^{\text{reg}}_{[\Delta_{x}\Delta_{3}\Delta_{4}]}(s,q_{3},q_{4})\Big{]}.$

The singularities of the integrals on the right-hand side now cancel. This can
most readily be seen by using the relation (315) between the AdS bulk-bulk and
shadow bulk-bulk propagator, which yields

$\displaystyle
i^{\text{reg}}_{[\Delta_{1}\Delta_{2};\Delta_{3}\Delta_{4}x\Delta_{x}]}+2\sin(\pi\beta_{x})a_{\beta_{x}}s^{-2\beta_{x}}i^{\text{reg}}_{[\Delta_{1}\Delta_{2}\Delta_{x}]}(q_{1},q_{2},s)i^{\text{reg}}_{[\Delta_{x}\Delta_{3}\Delta_{4}]}(s,q_{3},q_{4})$
$\displaystyle\qquad=i^{\text{reg}}_{[\Delta_{1}\Delta_{2};\Delta_{3}\Delta_{4}x\,d-\Delta_{x}]}.$
(379)

This shadow AdS exchange diagram is manifestly finite as the conditions
$d/2-\beta_{1}-\beta_{2}+\beta_{x}=-2k_{L}$ and
$d/2-\beta_{3}-\beta_{4}+\beta_{x}=-2k_{R}$ correspond to
$d/2-\beta_{1}-\beta_{2}-\bar{\beta}_{x}=-2k_{L}$ and
$d/2-\beta_{3}-\beta_{4}-\bar{\beta}_{x}=-2k_{R}$ where
$\bar{\beta}_{x}=\bar{\Delta}_{x}-d/2=-\beta_{x}$. From the analysis of the
AdS exchange diagram in section C.2, however, there are no all-minus ‘vertex’
singularities of this type, see (372).

In summary, dS ‘vertex’-type singularities corresponding to (376) for
$(\sigma_{1},\sigma_{2},\sigma_{x})=(-,-,+)$ and/or
$(\sigma_{3},\sigma_{4},\sigma_{x})=(-,-,+)$ are indeed absent. This is
manifest in the shadow holographic formula due to the vanishing of the sine
factors, but arises through a non-trivial cancellation in the conventional
holographic formula. Conversely, the absence of the all-minus ‘total’
singularity in dS is readily visible in the conventional holographic formula,
but arises from a non-trivial cancellation in the shadow holographic formula.

##  Appendix D Shadow relations in AdS

In the absence of divergences, amplitudes in AdS satisfy the shadow relations

$\displaystyle i^{\text{fin}}_{[\bar{\Delta}\bar{\Delta}]}(q)$
$\displaystyle=-\frac{4\beta^{2}}{i^{\text{fin}}_{[\Delta\Delta]}(q)},$ (380)
$\displaystyle
i^{\text{fin}}_{[\bar{\Delta}_{1},\bar{\Delta}_{2},\bar{\Delta}_{3}]}(q_{i})$
$\displaystyle=\left(\prod_{j=1}^{3}\frac{2\beta_{j}}{i^{\text{fin}}_{[\Delta_{j}\Delta_{j}]}(q_{j})}\right)i^{\text{fin}}_{[\Delta_{1},\Delta_{2},\Delta_{3}]}(q_{i}),$
(381) $\displaystyle
i^{\text{fin}}_{[\bar{\Delta}_{1},\bar{\Delta}_{2},\bar{\Delta}_{3},\bar{\Delta}_{4}]}(q_{i})$
$\displaystyle=\left(\prod_{j=1}^{4}\frac{2\beta_{j}}{i^{\text{fin}}_{[\Delta_{j}\Delta_{j}]}(q_{j})}\right)i^{\text{fin}}_{[\Delta_{1}\Delta_{2},\Delta_{3}\Delta_{4}]}(q_{i}),$
(382) $\displaystyle
i^{\text{fin}}_{[\bar{\Delta}_{1},\bar{\Delta}_{2},\bar{\Delta}_{3},\bar{\Delta}_{4}x\bar{\Delta}_{x}]}(q_{i},s)$
$\displaystyle=\left(\prod_{j=1}^{4}\frac{2\beta_{j}}{i^{\text{fin}}_{[\Delta_{j}\Delta_{j}]}(q_{j})}\right)\Bigg{[}i^{\text{fin}}_{[\Delta_{1}\Delta_{2},\Delta_{3}\Delta_{4}x\Delta_{x}]}(q_{i},s)$
$\displaystyle\qquad-\frac{i^{\text{fin}}_{[\Delta_{1}\Delta_{2}\Delta_{x}]}(q_{1},q_{2},s)i^{\text{fin}}_{[\Delta_{x}\Delta_{3}\Delta_{4}]}(s,q_{3},q_{4})}{i^{\text{fin}}_{[\Delta_{x}\Delta_{x}]}(s)}\Bigg{]}$
(383)

where $\bar{\Delta}_{j}=d-\Delta_{j}$ are the shadow dimensions.

In deriving these formulae, we used the standard holographic normalisations
(see (143) for the 2-point function, with higher-point functions as defined in
Bzowski:2022rlz ), and applied the same normalisations to shadow fields
replacing
$\beta_{j}\rightarrow\bar{\beta}_{j}=\bar{\Delta}_{j}-d/2=-\beta_{j}$. In
addition, we used the identities (314) and (315) connecting the AdS
propagators to their shadows.

Alternatively, these formulae can be understood as arising from the Legendre
transform: for each field, we add to the partition function,
$Z_{AdS}[\varphi_{(0)}]$ in (32), the product of the source $\varphi_{(0)}$
with its conjugate variable $\Pi_{(\Delta)}$ (see appendix B.1), namely $\int
d^{d}x\sqrt{\gamma}\Pi_{(\Delta)}\varphi_{(0)}$, and integrate over
$\varphi_{(0)}$. Completing squares, one may then integrate out
$\varphi_{(0)}$ giving

$\varphi_{(0)}=\frac{2\beta}{i^{\text{fin}}_{[\Delta\Delta]}}\varphi_{(\Delta)}.$
(384)

Now $\varphi_{(\Delta)}$ acts as a source for the shadow operator of dimension
$\bar{\Delta}=d-\Delta$, and functionally differentiating with respect to it,
one may obtain (380)-(383).

## References

  * (1) A. Strominger, _The dS/CFT correspondence_ , _JHEP_ 10 (2001) 034 [hep-th/0106113].
  * (2) A. Strominger, _Inflation and the dS/CFT correspondence_ , _JHEP_ 11 (2001) 049 [hep-th/0110087].
  * (3) C.M. Hull, _Timelike T-duality, de Sitter space, large N gauge theories and topological field theory_ , _JHEP_ 07 (1998) 021 [hep-th/9806146].
  * (4) E. Witten, _Quantum gravity in de Sitter space_ , in _Strings 2001: International Conference_ , 6, 2001 [hep-th/0106109].
  * (5) J.M. Maldacena, _The Large N limit of superconformal field theories and supergravity_ , _Adv. Theor. Math. Phys._ 2 (1998) 231 [hep-th/9711200].
  * (6) D. Anninos, T. Hartman and A. Strominger, _Higher Spin Realization of the dS/CFT Correspondence_ , _Class. Quant. Grav._ 34 (2017) 015009 [1108.5735].
  * (7) S. de Haro, S.N. Solodukhin and K. Skenderis, _Holographic reconstruction of space-time and renormalization in the AdS/CFT correspondence_ , _Commun. Math. Phys._ 217 (2001) 595 [hep-th/0002230].
  * (8) I. Papadimitriou and K. Skenderis, _AdS/CFT correspondence and geometry_ , _IRMA Lect. Math. Theor. Phys._ 8 (2005) 73 [hep-th/0404176].
  * (9) S. de Haro, K. Skenderis and S.N. Solodukhin, _Gravity in warped compactifications and the holographic stress tensor_ , _Class. Quant. Grav._ 18 (2001) 3171 [hep-th/0011230].
  * (10) J.M. Maldacena, _Non-Gaussian features of primordial fluctuations in single field inflationary models_ , _JHEP_ 05 (2003) 013 [astro-ph/0210603].
  * (11) F. Larsen, J.P. van der Schaar and R.G. Leigh, _De Sitter holography and the cosmic microwave background_ , _JHEP_ 04 (2002) 047 [hep-th/0202127].
  * (12) F. Larsen and R. McNees, _Inflation and de Sitter holography_ , _JHEP_ 07 (2003) 051 [hep-th/0307026].
  * (13) D. Seery and J.E. Lidsey, _Non-Gaussian Inflationary Perturbations from the dS/CFT Correspondence_ , _JCAP_ 06 (2006) 001 [astro-ph/0604209].
  * (14) D. Harlow and D. Stanford, _Operator Dictionaries and Wave Functions in AdS/CFT and dS/CFT_ , 1104.2621.
  * (15) J.M. Maldacena and G.L. Pimentel, _On graviton non-Gaussianities during inflation_ , _JHEP_ 09 (2011) 045 [1104.2846].
  * (16) I. Mata, S. Raju and S. Trivedi, _CMB from CFT_ , _JHEP_ 07 (2013) 015 [1211.5482].
  * (17) A. Ghosh, N. Kundu, S. Raju and S.P. Trivedi, _Conformal Invariance and the Four Point Scalar Correlator in Slow-Roll Inflation_ , _JHEP_ 07 (2014) 011 [1401.1426].
  * (18) D. Anninos, T. Anous, D.Z. Freedman and G. Konstantinidis, _Late-time Structure of the Bunch-Davies De Sitter Wavefunction_ , _JCAP_ 11 (2015) 048 [1406.5490].
  * (19) N. Kundu, A. Shukla and S.P. Trivedi, _Constraints from Conformal Symmetry on the Three Point Scalar Correlator in Inflation_ , _JHEP_ 04 (2015) 061 [1410.2606].
  * (20) P. McFadden and K. Skenderis, _Holography for Cosmology_ , _Phys. Rev. D_ 81 (2010) 021301 [0907.5542].
  * (21) K. Skenderis and P.K. Townsend, _Hidden supersymmetry of domain walls and cosmologies_ , _Phys. Rev. Lett._ 96 (2006) 191301 [hep-th/0602260].
  * (22) K. Skenderis and P.K. Townsend, _Pseudo-Supersymmetry and the Domain-Wall/Cosmology Correspondence_ , _J. Phys. A_ 40 (2007) 6733 [hep-th/0610253].
  * (23) P. McFadden and K. Skenderis, _The Holographic Universe_ , _J. Phys. Conf. Ser._ 222 (2010) 012007 [1001.2007].
  * (24) P. McFadden and K. Skenderis, _Holographic Non-Gaussianity_ , _JCAP_ 05 (2011) 013 [1011.0452].
  * (25) P. McFadden and K. Skenderis, _Cosmological 3-point correlators from holography_ , _JCAP_ 06 (2011) 030 [1104.3894].
  * (26) A. Bzowski, P. McFadden and K. Skenderis, _Holographic predictions for cosmological 3-point functions_ , _JHEP_ 03 (2012) 091 [1112.1967].
  * (27) J. Garriga, K. Skenderis and Y. Urakawa, _Multi-field inflation from holography_ , _JCAP_ 01 (2015) 028 [1410.3290].
  * (28) T. Hertog and J. Hartle, _Holographic No-Boundary Measure_ , _JHEP_ 05 (2012) 095 [1111.6090].
  * (29) J.B. Hartle, S.W. Hawking and T. Hertog, _Accelerated Expansion from Negative $\Lambda$_, 1205.3807.
  * (30) G. Conti, T. Hertog and E. van der Woerd, _Holographic Tunneling Wave Function_ , _JHEP_ 12 (2015) 025 [1506.07374].
  * (31) T. Hertog and E. van der Woerd, _Primordial fluctuations from complex AdS saddle points_ , _JCAP_ 02 (2016) 010 [1509.03291].
  * (32) G. Araujo-Regado, R. Khan and A.C. Wall, _Cauchy slice holography: a new AdS/CFT dictionary_ , _JHEP_ 03 (2023) 026 [2204.00591].
  * (33) S. Antonini, P. Simidzija, B. Swingle and M. Van Raamsdonk, _Accelerating Cosmology from a Holographic Wormhole_ , _Phys. Rev. Lett._ 130 (2023) 221601 [2206.14821].
  * (34) E.A. Bergshoeff, J. Hartong, A. Ploegh, J. Rosseel and D. Van den Bleeken, _Pseudo-supersymmetry and a tale of alternate realities_ , _JHEP_ 07 (2007) 067 [0704.3559].
  * (35) K. Skenderis, P.K. Townsend and A. Van Proeyen, _Domain-wall/cosmology correspondence in adS/dS supergravity_ , _JHEP_ 08 (2007) 036 [0704.3918].
  * (36) R. Dijkgraaf, B. Heidenreich, P. Jefferson and C. Vafa, _Negative Branes, Supergroups and the Signature of Spacetime_ , _JHEP_ 02 (2018) 050 [1603.05665].
  * (37) R. Blumenhagen, M. Brinkmann, A. Makridou, L. Schlechter and M. Traube, _dS Spaces and Brane Worlds in Exotic String Theories_ , _JHEP_ 06 (2020) 077 [2002.11746].
  * (38) C. Coriano, L. Delle Rose and M. Serino, _Three and Four Point Functions of Stress Energy Tensors in D=3 for the Analysis of Cosmological Non-Gaussianities_ , _JHEP_ 12 (2012) 090 [1210.0136].
  * (39) S. Kawai and Y. Nakayama, _Improvement of energy-momentum tensor and non-Gaussianities in holographic cosmology_ , _JHEP_ 06 (2014) 052 [1403.6220].
  * (40) M. Dias, _Cosmology at the boundary of de Sitter using the dS/QFT correspondence_ , _Phys. Rev. D_ 84 (2011) 023512 [1104.0625].
  * (41) R. Easther, R. Flauger, P. McFadden and K. Skenderis, _Constraining holographic inflation with WMAP_ , _JCAP_ 09 (2011) 030 [1104.2040].
  * (42) N. Afshordi, C. Coriano, L. Delle Rose, E. Gould and K. Skenderis, _From Planck data to Planck era: Observational tests of Holographic Cosmology_ , _Phys. Rev. Lett._ 118 (2017) 041301 [1607.04878].
  * (43) N. Afshordi, E. Gould and K. Skenderis, _Constraining holographic cosmology using Planck data_ , _Phys. Rev. D_ 95 (2017) 123505 [1703.05385].
  * (44) H. Nastase and K. Skenderis, _Holography for the very early Universe and the classic puzzles of Hot Big Bang cosmology_ , _Phys. Rev. D_ 101 (2020) 021901 [1904.05821].
  * (45) H. Nastase, _Holographic cosmology solutions of problems with pre-inflationary cosmology_ , _JHEP_ 12 (2020) 026 [2008.05630].
  * (46) R. Jackiw and S. Templeton, _How Superrenormalizable Interactions Cure their Infrared Divergences_ , _Phys. Rev. D_ 23 (1981) 2291.
  * (47) T. Appelquist and R.D. Pisarski, _High-Temperature Yang-Mills Theories and Three-Dimensional Quantum Chromodynamics_ , _Phys. Rev. D_ 23 (1981) 2305.
  * (48) G. Cossu, L. Del Debbio, A. Juttner, B. Kitching-Morley, J.K.L. Lee, A. Portelli et al., _Nonperturbative Infrared Finiteness in a Superrenormalizable Scalar Quantum Field Theory_ , _Phys. Rev. Lett._ 126 (2021) 221601 [2009.14768].
  * (49) A. Bzowski, P. McFadden and K. Skenderis, _Holography for inflation using conformal perturbation theory_ , _JHEP_ 04 (2013) 047 [1211.4550].
  * (50) P. McFadden, _On the power spectrum of inflationary cosmologies dual to a deformed CFT_ , _JHEP_ 10 (2013) 071 [1308.0331].
  * (51) I. Antoniadis, P.O. Mazur and E. Mottola, _Conformal invariance, dark energy, and CMB non-Gaussianity_ , _JCAP_ 09 (2012) 024 [1103.4164].
  * (52) P. Creminelli, _Conformal invariance of scalar perturbations in inflation_ , _Phys. Rev. D_ 85 (2012) 041302 [1108.0874].
  * (53) A. Kehagias and A. Riotto, _Operator product expansion of inflationary correlators and conformal symmetry of de Sitter_ , _Nucl.Phys._ B864 (2012) 492 [1205.1523].
  * (54) A. Kehagias and A. Riotto, _The Four-point Correlator in Multifield Inflation, the Operator Product Expansion and the Symmetries of de Sitter_ , _Nucl. Phys._ B868 (2013) 577 [1210.1918].
  * (55) K. Schalm, G. Shiu and T. van der Aalst, _Consistency condition for inflation from (broken) conformal symmetry_ , _JCAP_ 03 (2013) 005 [1211.2157].
  * (56) J. Garriga and Y. Urakawa, _Inflation and deformation of conformal field theory_ , _JCAP_ 07 (2013) 033 [1303.5997].
  * (57) N. Arkani-Hamed and J. Maldacena, _Cosmological Collider Physics_ , 1503.08043.
  * (58) N. Arkani-Hamed, D. Baumann, H. Lee and G.L. Pimentel, _The Cosmological Bootstrap: Inflationary Correlators from Symmetries and Singularities_ , _JHEP_ 04 (2020) 105 [1811.00024].
  * (59) N. Arkani-Hamed, P. Benincasa and A. Postnikov, _Cosmological Polytopes and the Wavefunction of the Universe_ , 1709.02813.
  * (60) P. Benincasa, _From the flat-space S-matrix to the Wavefunction of the Universe_ , 1811.02515.
  * (61) C. Sleight, _A Mellin Space Approach to Cosmological Correlators_ , _JHEP_ 01 (2020) 090 [1906.12302].
  * (62) C. Sleight and M. Taronna, _Bootstrapping Inflationary Correlators in Mellin Space_ , _JHEP_ 02 (2020) 098 [1907.01143].
  * (63) D. Baumann, C. Duaso Pueyo, A. Joyce, H. Lee and G.L. Pimentel, _The cosmological bootstrap: weight-shifting operators and scalar seeds_ , _JHEP_ 12 (2020) 204 [1910.14051].
  * (64) D. Baumann, C. Duaso Pueyo, A. Joyce, H. Lee and G.L. Pimentel, _The Cosmological Bootstrap: Spinning Correlators from Symmetries and Factorization_ , _SciPost Phys._ 11 (2021) 071 [2005.04234].
  * (65) E. Pajer, D. Stefanyszyn and J. Supeł, _The Boostless Bootstrap: Amplitudes without Lorentz boosts_ , _JHEP_ 12 (2020) 198 [2007.00027].
  * (66) C. Sleight and M. Taronna, _From AdS to dS exchanges: Spectral representation, Mellin amplitudes, and crossing_ , _Phys. Rev. D_ 104 (2021) L081902 [2007.09993].
  * (67) H. Goodhew, S. Jazayeri and E. Pajer, _The Cosmological Optical Theorem_ , _JCAP_ 04 (2021) 021 [2009.02898].
  * (68) S. Céspedes, A.-C. Davis and S. Melville, _On the time evolution of cosmological correlators_ , _JHEP_ 02 (2021) 012 [2009.07874].
  * (69) E. Pajer, _Building a Boostless Bootstrap for the Bispectrum_ , _JCAP_ 01 (2021) 023 [2010.12818].
  * (70) S. Jazayeri, E. Pajer and D. Stefanyszyn, _From locality and unitarity to cosmological correlators_ , _JHEP_ 10 (2021) 065 [2103.08649].
  * (71) S. Melville and E. Pajer, _Cosmological Cutting Rules_ , _JHEP_ 05 (2021) 249 [2103.09832].
  * (72) D. Baumann, W.-M. Chen, C. Duaso Pueyo, A. Joyce, H. Lee and G.L. Pimentel, _Linking the singularities of cosmological correlators_ , _JHEP_ 09 (2022) 010 [2106.05294].
  * (73) D. Meltzer, _The inflationary wavefunction from analyticity and factorization_ , _JCAP_ 12 (2021) 018 [2107.10266].
  * (74) M. Hogervorst, J.a. Penedones and K.S. Vaziri, _Towards the non-perturbative cosmological bootstrap_ , _JHEP_ 02 (2023) 162 [2107.13871].
  * (75) L. Di Pietro, V. Gorbenko and S. Komatsu, _Analyticity and unitarity for cosmological correlators_ , _JHEP_ 03 (2022) 023 [2108.01695].
  * (76) C. Sleight and M. Taronna, _From dS to AdS and back_ , _JHEP_ 12 (2021) 074 [2109.02725].
  * (77) G.L. Pimentel and D.-G. Wang, _Boostless cosmological collider bootstrap_ , _JHEP_ 10 (2022) 177 [2205.00013].
  * (78) J. Bonifacio, H. Goodhew, A. Joyce, E. Pajer and D. Stefanyszyn, _The graviton four-point function in de Sitter space_ , _JHEP_ 06 (2023) 212 [2212.07370].
  * (79) S.A. Salcedo, M.H.G. Lee, S. Melville and E. Pajer, _The Analytic Wavefunction_ , _JHEP_ 06 (2023) 020 [2212.08009].
  * (80) D.-G. Wang, G.L. Pimentel and A. Achúcarro, _Bootstrapping multi-field inflation: non-Gaussianities from light scalars revisited_ , _JCAP_ 05 (2023) 043 [2212.14035].
  * (81) S. Albayrak and S. Kharel, _All plus four point (A)dS graviton function using generalized on-shell recursion relation_ , _JHEP_ 05 (2023) 151 [2302.09089].
  * (82) C. Armstrong, H. Goodhew, A. Lipstein and J. Mei, _Graviton trispectrum from gluons_ , _JHEP_ 08 (2023) 206 [2304.07206].
  * (83) S. Albayrak, P. Benincasa and C. Duaso Pueyo, _Perturbative Unitarity and the Wavefunction of the Universe_ , 2305.19686.
  * (84) N. Arkani-Hamed, D. Baumann, A. Hillman, A. Joyce, H. Lee and G.L. Pimentel, _Kinematic Flow and the Emergence of Time_ , 2312.05300.
  * (85) N. Arkani-Hamed, D. Baumann, A. Hillman, A. Joyce, H. Lee and G.L. Pimentel, _Differential Equations for Cosmological Correlators_ , 2312.05303.
  * (86) A. Bzowski, P. McFadden and K. Skenderis, _A handbook of holographic 4-point functions_ , _JHEP_ 12 (2022) 039 [2207.02872].
  * (87) T. Falk, R. Rangarajan and M. Srednicki, _The Angular dependence of the three point correlation function of the cosmic microwave background radiation as predicted by inflationary cosmologies_ , _Astrophys. J. Lett._ 403 (1993) L1 [astro-ph/9208001].
  * (88) M. Zaldarriaga, _Non-Gaussianities in models with a varying inflaton decay rate_ , _Phys. Rev. D_ 69 (2004) 043508 [astro-ph/0306006].
  * (89) D. Seery, K.A. Malik and D.H. Lyth, _Non-gaussianity of inflationary field perturbations from the field equation_ , _JCAP_ 03 (2008) 014 [0802.0588].
  * (90) K. Skenderis, _Lecture notes on holographic renormalization_ , _Class. Quant. Grav._ 19 (2002) 5849 [hep-th/0209067].
  * (91) T. Chakraborty, J. Chakravarty, V. Godet, P. Paul and S. Raju, _Holography of information in de Sitter space_ , 2303.16316.
  * (92) T. Chakraborty, J. Chakravarty, V. Godet, P. Paul and S. Raju, _The Hilbert space of de Sitter quantum gravity_ , 2303.16315.
  * (93) A. Bzowski, P. McFadden and K. Skenderis, _Scalar 3-point functions in CFT: renormalisation, beta functions and anomalies_ , _JHEP_ 03 (2016) 066 [1510.08442].
  * (94) A. Petkou and K. Skenderis, _A Nonrenormalization theorem for conformal anomalies_ , _Nucl. Phys._ B561 (1999) 100 [hep-th/9906030].
  * (95) A. Bzowski, P. McFadden and K. Skenderis, _Implications of conformal invariance in momentum space_ , _JHEP_ 03 (2014) 111 [1304.7760].
  * (96) A. Bzowski, P. McFadden and K. Skenderis, _Conformal $n$-point functions in momentum space_, _Phys. Rev. Lett._ 124 (2020) 131602 [1910.10162].
  * (97) X. Chen, Y. Wang and Z.-Z. Xianyu, _Schwinger-Keldysh Diagrammatics for Primordial Perturbations_ , _JCAP_ 12 (2017) 006 [1703.10166].
  * (98) N.D. Birrell and P.C.W. Davies, _Quantum Fields in Curved Space_ , Cambridge Monographs on Mathematical Physics, Cambridge Univ. Press, Cambridge, UK (2, 1984), 10.1017/CBO9780511622632.
  * (99) P. Breitenlohner and D.Z. Freedman, _Positive Energy in anti-De Sitter Backgrounds and Gauged Extended Supergravity_ , _Phys. Lett. B_ 115 (1982) 197.
  * (100) P. Breitenlohner and D.Z. Freedman, _Stability in Gauged Extended Supergravity_ , _Annals Phys._ 144 (1982) 249.
  * (101) I.R. Klebanov and E. Witten, _AdS/CFT correspondence and symmetry breaking_ , _Nucl. Phys. B_ 556 (1999) 89 [hep-th/9905104].
  * (102) H. Isono, H.M. Liu and T. Noumi, _Wavefunctions in dS/CFT revisited: principal series and double-trace deformations_ , _JHEP_ 04 (2021) 166 [2011.09479].
  * (103) M. Bianchi, D.Z. Freedman and K. Skenderis, _How to go with an RG flow_ , _JHEP_ 08 (2001) 041 [hep-th/0105276].
  * (104) M. Bianchi, D.Z. Freedman and K. Skenderis, _Holographic renormalization_ , _Nucl. Phys. B_ 631 (2002) 159 [hep-th/0112119].
  * (105) M. Henningson and K. Skenderis, _The Holographic Weyl anomaly_ , _JHEP_ 07 (1998) 023 [hep-th/9806087].
  * (106) M. Henningson and K. Skenderis, _Holography and the Weyl anomaly_ , _Fortsch. Phys._ 48 (2000) 125 [hep-th/9812032].
  * (107) F. Caloro and P. McFadden, _$\mathcal{A}$ -hypergeometric functions and creation operators for Feynman and Witten diagrams_, 2309.15895.
  * (108) A. Bzowski, _Handbook of derivative AdS amplitudes_ , 2312.11625.
  * (109) A.A. Starobinsky, _Fundamental Interactions_ , MGPI Press, Moscow (1984).
  * (110) L.H. Ford, _Quantum Instability of De Sitter Space-time_ , _Phys. Rev. D_ 31 (1985) 710.
  * (111) I. Antoniadis, J. Iliopoulos and T.N. Tomaras, _Quantum Instability of De Sitter Space_ , _Phys. Rev. Lett._ 56 (1986) 1319.
  * (112) A.A. Starobinsky, _Stochastic de Sitter (inflationary) stage in the early universe_ , _Lect. Notes Phys._ 246 (1986) 107.
  * (113) D.S. Salopek and J.R. Bond, _Nonlinear evolution of long-wavelength metric fluctuations in inflationary models_ , _Phys. Rev. D_ 42 (1990) 3936.
  * (114) A.A. Starobinsky and J. Yokoyama, _Equilibrium state of a selfinteracting scalar field in the De Sitter background_ , _Phys. Rev. D_ 50 (1994) 6357 [astro-ph/9407016].
  * (115) N.C. Tsamis and R.P. Woodard, _Stochastic quantum gravitational inflation_ , _Nucl. Phys. B_ 724 (2005) 295 [gr-qc/0505115].
  * (116) S. Weinberg, _Quantum contributions to cosmological correlations_ , _Phys. Rev. D_ 72 (2005) 043514 [hep-th/0506236].
  * (117) A. Riotto and M.S. Sloth, _On Resumming Inflationary Perturbations beyond One-loop_ , _JCAP_ 04 (2008) 030 [0801.1845].
  * (118) C.P. Burgess, L. Leblond, R. Holman and S. Shandera, _Super-Hubble de Sitter Fluctuations and the Dynamical RG_ , _JCAP_ 03 (2010) 033 [0912.1608].
  * (119) L. Senatore and M. Zaldarriaga, _On Loops in Inflation_ , _JHEP_ 12 (2010) 008 [0912.2734].
  * (120) S.B. Giddings and M.S. Sloth, _Semiclassical relations and IR effects in de Sitter and slow-roll space-times_ , _JCAP_ 01 (2011) 023 [1005.1056].
  * (121) C.P. Burgess, R. Holman, L. Leblond and S. Shandera, _Breakdown of Semiclassical Methods in de Sitter Space_ , _JCAP_ 10 (2010) 017 [1005.3551].
  * (122) D. Marolf and I.A. Morrison, _The IR stability of de Sitter: Loop corrections to scalar propagators_ , _Phys. Rev. D_ 82 (2010) 105032 [1006.0035].
  * (123) A. Rajaraman, _On the proper treatment of massless fields in Euclidean de Sitter space_ , _Phys. Rev. D_ 82 (2010) 123522 [1008.1271].
  * (124) L. Senatore and M. Zaldarriaga, _On Loops in Inflation II: IR Effects in Single Clock Inflation_ , _JHEP_ 01 (2013) 109 [1203.6354].
  * (125) G.L. Pimentel, L. Senatore and M. Zaldarriaga, _On Loops in Inflation III: Time Independence of zeta in Single Clock Inflation_ , _JHEP_ 07 (2012) 166 [1203.6651].
  * (126) A.M. Polyakov, _Infrared instability of the de Sitter space_ , 1209.4135.
  * (127) L. Senatore and M. Zaldarriaga, _The constancy of $\zeta$ in single-clock Inflation at all loops_, _JHEP_ 09 (2013) 148 [1210.6048].
  * (128) J. Serreau and R. Parentani, _Nonperturbative resummation of de Sitter infrared logarithms in the large-N limit_ , _Phys. Rev. D_ 87 (2013) 085012 [1302.3262].
  * (129) E.T. Akhmedov, U. Moschella, K.E. Pavlenko and F.K. Popov, _Infrared dynamics of massive scalars from the complementary series in de Sitter space_ , _Phys. Rev. D_ 96 (2017) 025002 [1701.07226].
  * (130) V. Gorbenko and L. Senatore, _$\lambda\phi^{4}$ in dS_, 1911.00022.
  * (131) M. Mirbabayi, _Infrared dynamics of a light scalar field in de Sitter_ , _JCAP_ 12 (2020) 006 [1911.00564].
  * (132) M. Baumgart and R. Sundrum, _De Sitter Diagrammar and the Resummation of Time_ , _JHEP_ 07 (2020) 119 [1912.09502].
  * (133) D. Green and A. Premkumar, _Dynamical RG and Critical Phenomena in de Sitter Space_ , _JHEP_ 04 (2020) 064 [2001.05974].
  * (134) T. Cohen and D. Green, _Soft de Sitter Effective Theory_ , _JHEP_ 12 (2020) 041 [2007.03693].
  * (135) M. Baumgart and R. Sundrum, _Manifestly Causal In-In Perturbation Theory about the Interacting Vacuum_ , _JHEP_ 03 (2021) 080 [2010.10785].
  * (136) T. Heckelbacher and I. Sachs, _Loops in dS/CFT_ , _JHEP_ 02 (2021) 151 [2009.06511].
  * (137) S. Céspedes, A.-C. Davis and D.-G. Wang, _On the IR Divergences in de Sitter Space: loops, resummation and the semi-classical wavefunction_ , 2311.17990.
  * (138) M. Beneke, P. Hager and A.F. Sanfilippo, _Cosmological Correlators in massless ${\phi}^{4}$-theory and the Method of Regions_, 2312.06766.
  * (139) C. Chowdhury, A. Lipstein, J. Mei, I. Sachs and P. Vanhove, _The Subtle Simplicity of Cosmological Correlators_ , 2312.13803.
  * (140) D. Seery, _Infrared effects in inflationary correlation functions_ , _Class. Quant. Grav._ 27 (2010) 124005 [1005.1649].
  * (141) B.-L. Hu, _Infrared Behavior of Quantum Fields in Inflationary Cosmology – Issues and Approaches: an overview_ , 1812.11851.
  * (142) M. Bañados, E. Bianchi, I. Muñoz and K. Skenderis, _Bulk renormalization and the AdS/CFT correspondence_ , _Phys. Rev. D_ 107 (2023) L021901 [2208.11539].
  * (143) I. Papadimitriou and K. Skenderis, _Correlation functions in holographic RG flows_ , _JHEP_ 10 (2004) 075 [hep-th/0407071].
  * (144) A. Bzowski, _Dimensional renormalization in AdS/CFT_ , 1612.03915.
  * (145) A. Bzowski and M. Schillo, _Dimensional regularization for holographic RG flows_ , _JHEP_ 10 (2019) 025 [1906.02234].


# RACE-IT: A Reconfigurable Analog CAM-Crossbar Engine for In-Memory
Transformer Acceleration

Lei Zhao1, Luca Buonanno1, Ron M. Roth1, Sergey Serebryakov1, Archit Gajjar1
John Moon1, Jim Ignowski1, Giacomo Pedretti1  1Artificial Intelligence
Research Lab (AIRL), Hewlett Packard Labs, USA
Email: {lei.zhao<EMAIL_ADDRESS>
###### Abstract

Transformer models represent the cutting edge of Deep Neural Networks (DNNs)
and excel in a wide range of machine learning tasks. However, processing these
models demands significant computational resources and results in a
substantial memory footprint. While In-memory Computing (IMC) offers promise
for accelerating Matrix-Vector Multiplications (MVMs) with high computational
parallelism and minimal data movement, employing it for implementing other
crucial operators within DNNs remains a formidable task. This challenge is
exacerbated by the extensive use of Softmax and data-dependent matrix
multiplications within the attention mechanism. Furthermore, existing IMC
designs encounter difficulties in fully harnessing the benefits of analog MVM
acceleration due to the area and energy-intensive nature of Analog-to-Digital
Converters (ADCs).

To tackle these challenges, we introduce a novel Compute Analog Content
Addressable Memory (Compute-ACAM) structure capable of performing various non-
MVM operations within Transformers. Together with the crossbar structure, our
proposed RACE-IT accelerator enables efficient execution of all operations
within Transformer models in the analog domain. Given the flexibility of our
proposed Compute-ACAMs to perform arbitrary operations, RACE-IT exhibits
adaptability to diverse non-traditional and future DNN architectures without
necessitating hardware modifications. Leveraging the capability of Compute-
ACAMs to process analog input and produce digital output, we also replace
ADCs, thereby reducing the overall area and energy costs.

By evaluating various Transformer models against state-of-the-art GPUs and
existing IMC accelerators, RACE-IT increases performance by 10.7$\times$ and
5.9$\times$, and reduces energy by 1193$\times$, and 3.9$\times$,
respectively.

##  I Introduction

Transformer models [49] have gained immense popularity in the field of natural
language processing [26, 20], computer vision [8, 24], recommendation systems
[48, 33], etc., thanks to their remarkable ability to capture complex patterns
in sequential and image data. However, the performance of Transformers comes
at a cost: their appetite for computational resources and memory. These models
often feature millions, if not billions, of parameters, demanding substantial
computation and data movement. The sheer scale of Transformer-based
architectures, exemplified by models like GPT-3 [3] and BERT [7], necessitates
specialized hardware and high-capacity memory systems, presenting a
significant challenge to deploy them at scale. As a result, efficiently
processing Transformer models remains a crucial concern in the quest to unlock
their full potential for a wider range of applications.

In-memory Computing (IMC) stands out as a promising solution to alleviate the
computational and memory challenges posed by Transformer models [53, 15, 56,
23]. IMC involves performing computations directly within the memory,
bypassing the need for frequent data movement between memory and the
processing units, which is a significant bottleneck in traditional computing
architectures. Emerging memories such as Resistive Random Access Memories
(ReRAM) and Phase Change Memories (PCMs) offer a compelling avenue for IMC due
to their non-volatile nature and high integration potential. Crossbar arrays
of such emerging memories can efficiently map neural network weights in their
conductance and accelerate a layer evaluation by parallel Matrix-Vector
Multiplications (MVMs) in the analog domain [13]. High-performance silicon
demonstrations of multi-core IMC accelerators with ReRAM [50] and PCM [21]
have been recently presented, finally grounding years of research, in
experimental measurements. However, one notable drawback is that these designs
may not be optimally suited for critical operations like Softmax and data-
dependent matrix multiplications, which are fundamental to the Transformer
model’s architecture and performance. Addressing these limitations and
optimizing IMC designs for these specific operations is a critical area of
ongoing research to enhance the applicability of IMC in Transformer-based
processing.

To address this challenge, existing IMC designs fall into four main
approaches. The first employs a direct ad-hoc solution, integrating
specialized CMOS operators customized for specific workloads, as seen in ISAAC
[43]. However, this approach lacks flexibility, making the adoption of new
operators challenging. The second approach, exemplified by the PUMA
architecture [1], utilizes programmable Vector Functional Units (VFUs) to
support a wide variety of operations but incurs a performance cost,
particularly in data-dependent matrix multiplications within Transformer
models. The third approach explores ReRAM’s potential for computing the NOR
operation [2, 19, 45, 9], offering a convenient solution for diverse
operations but with drawbacks such as slow write speed and poor endurance. The
fourth approach employs content addressable memories (CAMs) [37], which have
gained attention in hyper-dimensional computing and genomics. In CAMs, a given
input is searched and its location, i.e. address, is returned. The address can
then be used to access a conventional RAM. CAM-based IMC has recently gained
attention, accelerating, for example, hyper-dimensional computing [14] and
genomics [10]. CAMs were also used as look-up tables [37] where the match
output indexes a memory array that stores pre-computed results, such as the
activations of DNNs. However, every computation requires two steps, a CAM
search and a subsequent memory read, both contributing to latency and energy
overheads. In addition, it is challenging to scale up for high parallelism due
to the substantial area overhead of the memory array. Recently, ReRAM-based
Analog CAMs (ACAMs) have also been proposed [22]. Each ACAM cell encodes a
range and checks whether an analog input falls within the range. ACAMs are
naturally suitable to accelerate decision trees and their ensemble, by mapping
tree branches directly within their ranges [31, 32]. ACAMs have also been used
to implement various single-operand operations [46, 58], while still adhering
to the look-up table approach, resulting in marginal improvements.

In this paper, we present a new ACAM cell structure, called Compute-ACAM, to
support computing a wide range of arbitrary operations. It directly outputs
each bit of the result, eliminating the need for a memory array and surpassing
the look-up table approach. By reprogramming the ranges stored in the cells,
the same Compute-ACAM array can be reconfigured for different operations
without requiring any hardware modifications. This characteristic positions it
as an ideal solution to construct a reconfigurable architecture capable of
supporting all operators in Transformer models and even future operators that
have yet to be conceived. We also leverage ACAM’s capability of accepting
analog input and producing digital output, to replace the Analog-to-Digital
Converters (ADCs), which account for more than 30% of the area and 50% of the
power in ReRAM-based accelerators. Notably, all operations implemented using
Compute-ACAM occupy less area than their CMOS-based counterparts.

Leveraging Compute-ACAM, we present RACE-IT, a Reconfigurable Analog CAM-
crossbar Engine for In-memory Transformer acceleration, an accelerator
architecture capable of efficiently handling operations across the entirety of
Transformer models. We employ a pipelined execution fashion to optimize for
throughput. Beyond Transformer models, RACE-IT possesses the versatility to
support a broad spectrum of DNN architectures, including those integrating
future operators yet to be introduced. To validate the effectiveness of our
approach, we compare RACE-IT against state-of-the-art GPUs and ReRAM-based
accelerators. The experimental results demonstrate significant performance
enhancements of 10.7$\times$ and 5.9$\times$, along with substantial energy
savings of 1193$\times$ and 3.9$\times$, respectively.

In summary, this paper offers the following contributions:

  * •

A novel ACAM-based computing structure (Compute-ACAMs) that supports arbitrary
operations. To the best of our knowledge, this is the first CAM-based design
that does not need a memory array for arbitrary computation. Its
reconfigurability, small area, and fast execution speed open up a new paradigm
for reconfigurable computation in DNN accelerators.

  * •

An encoding-based optimization that reduces the number of Compute-ACAM arrays
needed to perform computation. This is an essential part of the design to make
the proposed Compute-ACAM structure smaller enough for massive replication to
achieve high parallelism.

  * •

A full-fledged reconfigurable accelerator with crossbars and Compute-ACAMs
that can efficiently execute full Transformer models and even future operators
without adding new specialized components.

  * •

We benchmark RACE-IT against state-of-the-art GPUs and ReRAM-based
accelerators, achieving substantial performance improvement and energy saving.

##  II Background

###  II-A Crossbar-based MVM Computing Engine

Figure 1: ReRAM crossbar computing MVM with spatial bit slicing for weight and
temporal bit slicing for input.

ReRAMs are emerging non-volatile memory with significant potential for
accelerating MVMs when arranged in a crossbar structure. Figure 1 demonstrates
computing a $4\times 4$ input feature map with two $4\times 4$ weight matrices
mapped in ReRAM crossbar arrays. The input elements are transformed into a
vector of voltages applied to the word lines. In contrast, weights are
programmed as ReRAM conductance within the crossbar. Kirchhoff’s law ensures
that the current flowing out on the bit lines represents the dot product
between the input vector and the weight matrices. However, converting the
analog current values back into the digital domain necessitates the use of
ADCs that are area and power-expensive.

There is a disparity between the hardware resolution and algorithmic
precision. DNN commonly needs 8 bits for inference [18, 25, 28, 40, 55]. While
ReRAM cells have demonstrated programmability of up to 8 bits [44], it is
often not possible to reach this precision on a large array due to stochastic
variation of the device, such as drift and noise [30]. Furthermore, the
resolution of word line voltages is limited due to hardware constraints
related to Digital-to-Analog Converters (DACs). Therefore, a bit slicing [43]
scheme is needed for both weights and input. In Figure 1, we assume a
configuration of 2-bit input, 4-bit weights, 2-bit ReRAM cells, and 1-bit
DACs. Spatial bit slicing is applied to weights by partitioning each weight
into a 2-bit Most Significant Bit (MSB) part and a 2-bit Least Significant Bit
(LSB) part. The MSBs and LSBs are stored in two cells located in adjacent
columns. Consequently, the output from these columns is a partial sum that
necessitates shifting and subsequent addition for consolidation. Temporal bit
slicing is applied to input by sending each input bit to the crossbar over
separate cycles. In practice, 1-transistor-1-resistor (1T1R) structures are
employed to build the array, with the ReRAM biased at a fixed voltage and
input bits applied on the access transistor gate. This also improves the
reliability of matrix multiply to avoid errors due to the non-linear
conductance of ReRAMs. Therefore, shift and add are also needed to aggregate
the results from each cycle.

###  II-B Analog Content Addressable Memory

Figure 2: (a) An ACAM cell structure. (b) An ACAM cell’s operational
principle. (c) Increased-precision ACAM cell structure

A conventional digital CAM cell is limited to comparing a single binary input
bit with a stored binary value. Ternary CAMs (TCAMs) have also been proposed,
in which a wildcard ’X’ can be used both as input or stored value, to match
either 1s or 0s. The recently proposed ACAM cell structure [22] allows
comparing an analog input value within a defined range stored in the cell.
Figure 2(a) presents the schematic of an ACAM cell. Two memristors, M1 and M2,
are utilized to store the lower and upper thresholds, respectively. The blue-
shaded area compares the input with the lower threshold of the range, while
the red-shaded part compares the input with the upper threshold of the range.
The combined operation of the series transistors (S1 and S2) and memristors
act as voltage dividers, controlling the voltage supplied to the pull-down
transistors (T1 and T2). In the blue-shaded section, when the input on the
Data Line (DL) is greater than or equal to the data stored in M1, T1 is off,
thus keeping the Matching Line (ML) at a high voltage. The red-shaded portion
operates similarly, with an inverter added before the pull-down transistor T2.
Figure 2(b) illustrates the operational principle of an ACAM cell. It is
possible to represent a ”Don’t Care” value on either side of the range
threshold by programming M1 (M2) to the lowest (highest) possible resistance,
as depicted by the lower two rows in Figure 2(b). Studies have demonstrated up
to 4-bit analog search in ACAM [22, 31].

Figure 2(c) shows an extended cell that facilitates 8-bit comparisons [32], to
accommodate the typical 8-bit precision requirement of many machine learning
algorithms. This advanced structure comprises two original ACAM cells
interconnected. Inputs, lower and upper thresholds are divided into a 4-bit
Most Significant Bit (MSB) part and a 4-bit Least Significant Bit (LSB) part
and are compared separately. By exploiting pull-down logic and re-use of pre-
charged match lines, the overall match operation is reconstructed with two
cycles.

###  II-C Transformers and Multi-Head Attention

Figure 3: (a) Transformer structure. (b) Multi-head attention.

####  II-C1 Transformer Model Structure

Transformer models consist of both an encoder stack and a decoder stack [49],
sharing a fundamental architecture showcased in Figure 3(a). Typically, the
encoder processes the input sequence, and the decoder generates the output
sequence. However, depending on the specific machine learning task, models
might solely include either the encoder stack [20, 7, 41, 5, 26] or the
decoder stack [16, 34, 35, 6]. Each encoder/decoder block comprises two key
modules: the Multi-Head Attention (MHA) and the Feed-Forward Network (FFN).
Subsequent to each module, a residual block aggregates the input and output
from its previous MHA or FFN, transmitting the sum to a normalization layer.
The contrast between an encoder block and a decoder block lies in an
additional MHA, accompanied by its respective residual block and normalization
layer within the decoder block. This additional MHA in the decoder block,
apart from utilizing the output from its preceding MHA or FFN as input, also
incorporates features from the last encoder block of the encoder stack.

####  II-C2 Multi-Head Attention

The most fundamental component of Transformer models is the MHA, depicted in
Figure 3(b). MHA operates on a sequence of tokens as its input, comprising
three types of token sequences: the query token sequence $\mathbf{X_{Q}}$, the
key token sequence $\mathbf{X_{K}}$, and the value token sequence
$\mathbf{X_{V}}$. Generally, these three token sequences are identical, except
for the decoder block’s additional MHA, which employs the output of the
encoder stack as the query and key token sequences. For each of these token
sequences, a distinct linear layer (denoted by $\mathbf{W_{Q}}$,
$\mathbf{W_{K}}$, and $\mathbf{W_{V}}$ in Figure 3(b)) is applied to generate
respective matrices: a query matrix $\mathbf{Q}$, a key matrix $\mathbf{K}$,
and a value matrix $\mathbf{V}$. This operation is defined as:

$\mathbf{Q},\mathbf{K},\mathbf{V}=\mathbf{X_{Q}}\cdot\mathbf{W_{Q}},\mathbf{X_{K}}\cdot\mathbf{W_{K}},\mathbf{X_{V}}\cdot\mathbf{W_{V}}$
(1)

These resulting matrices are then partitioned into multiple segments, often
referred to as heads, which undergo identical sets of computations,
specifically the attention mechanism, expressed as:

$Attention(\mathbf{K_{i}},\mathbf{Q_{i}},\mathbf{V_{i}})=\text{softmax}\left(\frac{\mathbf{Q_{i}}\cdot\mathbf{K_{i}}^{T}}{\sqrt{d_{k}}}\right)\cdot\mathbf{V_{i}}$
(2)

where $i\in num\\_heads$, and $d_{k}$ represents the dimension of a query and
key vector. Ultimately, the output from each head is concatenated and goes
through another linear layer to form the overall output of this MHA.

##  III Compute-ACAM Cell Structure

###  III-A General Computing with ACAM

Figure 4: (a) Computing $y=GeLU(x)$ in ACAM as an example for one-operand
computation. Assuming $x$ and $y$ are 4-bit fixed point data in 1-0-3 format.
(b) Computing $z=x*y$ in ACAM as an example for two-operand computation.
Assuming $x$ and $y$ are 2-bit fixed point data in 1-0-1 format, and $z$ is a
4-bit fixed-point numbers in 1-1-2 format.

We provide two examples to show the insights of performing one-variable and
two-variable operations within ACAM. We utilize the S-I-F notation to
represent the bit allocation for the sign, integer, and fraction parts of the
fixed-point format. For instance, 1-1-2 signifies a 4-bit fixed-point format,
where the MSB is designated as the sign bit, followed by 1 bit for the integer
part, and the remaining 2 LSBs for the fraction part.

Figure 4(a)-(c) depicts an illustrative example of computing the GeLU
function, which is the state-of-the-art activation used in Transformer models.
We use the fixed point format of 1-0-3 for both input and output. Figure 4(a)
showcases the first two columns presenting all possible binary ($x_{B}$) and
decimal ($x_{D}$) values of the input, followed by columns listing the decimal
output of the GeLU function ($y_{D}$) and their values after quantization to
the 1-0-3 format ($Q(y_{D})$). The final column displays the binary bits of
$Q(y_{D})$. Rather than storing all computed results ($Q(y_{D})$) in a memory
array, we seek the ACAM to directly output the binary bits of the results,
employing the MLs to output each bit of $Q(y_{D})$. The core concept involves
using one row of ACAM cells to determine if the input falls into any of the
ranges that set the specific output bit to 1. For example, grouping the 1s in
the last column in Figure 4(a) (highlighted in red) reveals that the MSB of
the output equals 1 when the input falls within the range of [-1, -0.125),
indicated by the blue border. Figure 4(b) displays all combinations of input
ranges, resulting in each output bit being 1. Figure 4(c) demonstrates the
mapping of the ranges in Figure 4(b) onto our desired ACAM array structure to
execute this operation. This elucidates the first new requirement for our
desired ACAM array structure, namely, an OR operation is necessary for the
cells along the same ML. However, the original ACAM array could only execute
an AND operation along the ML.

Figure 4(d)-(f) showcases a two-variable multiplication operation. For the
sake of brevity, we adopt a 2-bit multiplication in this example to ensure the
table fits into the figure conveniently. Both input variables, denoted as $x$
and $y$, utilize the 1-0-1 format, while the resulting output $z$ utilizes the
1-1-2 format. In contrast to the one-variable example, a thorough examination
of all range combinations of $x$ and $y$ is necessary. For instance, one
feasible input range that triggers the MSB of the output to assume 1 entails
$x$ within [-1, -0.5) and $y$ within [0.5, 1), shown by the blue border in
Figure 4(d). Consequently, this underscores the second imperative for our
envisioned ACAM structure—specifically, the need for an AND operation
encompassing two ranges from distinct inputs within the same cell.

###  III-B Cell Structure

Figure 5: (a) Modified conventional 4-bit ACAM cell to support OR operation
among ranges. (b) Proposed 8-bit Compute-ACAM cell structure. (c) Mapping
8-bit comparison onto a Compute-ACAM cell. (d) Mapping 4-bit comparison onto a
Compute-ACAM cell. (e) Mapping two-variable 4-bit comparison onto a Compute-
ACAM cell.

To simplify the array and cell structure, and satisfy the first requirement,
we use a conventional AND-type pull-down logic but with a custom OR-type ACAM
cell. Following De-Morgan’s Law, by inverting the output of the ML the overall
operation is equivalent to implementing an OR-type pull-down logic with an
AND-type ACAM cell. The schematic of the modified cell is presented in Figure
5(a). The blue-shaded subpart calculates the upper threshold, while the red-
shaded subpart computes the lower threshold.

To meet the second requirement and perform two-variable computations, a direct
approach involves serially stacking two cells. However, considering that most
operations in DNNs necessitate a minimum of 8-bit precision, we designed a
cell capable of performing both two-variable computation and double the
precision of the cell, with a similar technique as the one in Figure 2(b). The
finalized Compute-ACAM cell structure is illustrated in Figure 5(c).

This cell structure supports three distinct computing modes: 8-bit one-
variable computing (Figure 5(c)), 4-bit one-variable computing (Figure 5(d)),
and 4-bit two-variable computing (Figure 5(e)). Having two additional subparts
compared to the original 8-bit precision ACAM cell structure in Figure 2(c),
it possesses the advantage of executing 8-bit computations within a single
cycle by storing $U_{MSB}+1$ and $L_{MSB}+1$ in memristors, as depicted in
Figure 5(c). For 4-bit computing on a single variable, only the middle two
subparts are required. Consequently, the memristors in the top pair are set to
the maximum resistance, while those in the bottom pair are set to the minimum
resistance, rendering these two pairs as ”Don’t Care” during computation, as
illustrated in Figure 5(d). Figure 5(e) elucidates how this cell can be
utilized to carry out computations on two 4-bit inputs. One subpart in each of
the two pairs is designated as a ”Don’t Care”, thus the remaining subparts are
serially connected. Two of these subparts are allocated for the first input
($X$), and the remaining two are assigned to the second input ($Y$).

##  IV Computing with Compute-ACAM

Figure 6: Principle scheme of the analog S&A operation, enabling a 2-fold
conversion of the 8-bit precision value from the crossbar.

In this section, we elaborate on utilizing Compute-ACAM’s three computing
modes to implement a diverse range of non-MVM operations within Transformer
models.

###  IV-A 4-bit One-Variable Computing (ADC)

Given that Compute-ACAM processes analog input and yields a digital output, it
is well-suited to replace ADCs. Because an ADC transforms an analog
representation into a digital one without altering the inherent value, we can
construct a Compute-ACAM-based ADC in a similar approach as in Figure 4(a) by
substituting the GeLU function with the Identity function ($y=x$). Utilizing
the same crossbar configuration as in ISAAC [43] and PUMA [1], our Compute-
ACAM-based ADC should maintain a precision equivalent to their ADCs, which is
8 bits. However, despite Compute-ACAM having an 8-bit one-variable computing
mode, we could not use it for converting 8-bit analog data. This is because
the 8-bit input needs to be partitioned into a 4-bit MSB part ($X_{MSB}$) and
a 4-bit LSB part ($X_{LSB}$) before sending to the Compute-ACAM, as
illustrated in Figure 5(c), whereas the output from the crossbar is a current
representing a single 8-bit value.

To split the crossbar output to two 4-bit parts for Compute-ACAM, we propose a
scheme similar to the one found in conventional folded ADCs. Figure 6 shows
the conversion process. The conversion is divided into two successive steps.
Fist, once the 8-bit analog data is ready from the crossbar for the
conversion, the Compute-ACAM-based ADC is used to convert the analog data as-
is, resulting in the 4 MSB of converted digital result. In the second step,
the DACs subtract an amount of signal equivalent to the 4 MSBs from the 8-bit
analog value, and the subtraction result is rescaled in order to match
Compute-ACAM’s input dynamics; the Compute-ACAM-based ADC is then activated
again, which results in the 4 LSB of the converted digital result.

###  IV-B 4-bit Two-Variable Computing (Multiplication)

Each MHA requires two data-dependent matrix multiplications, denoted by the
$\cdot$ operator in Equation 2. Different from MVM, both operands of data-
dependent matrix multiplications are not fixed during inference. If we use the
crossbar structure for data-dependent matrix multiplications, one of the
operands needs to be written into the crossbar before each computation.
However, this approach is susceptible to the long write latency and low
endurance issues associated with ReRAM.

Figure 7: Merging ranges for multiplication, assuming $x$ and $y$ is in 1-1-2
format, $z$ is in 1-2-1 format.

We facilitate 4-bit multiplications by utilizing Compute-ACAM’s 4-bit two-
variable computing mode. An 8-bit multiplication can be decomposed into four
4-bit multiplications and three adds. Figure 4(b) has already demonstrated an
example of 2-bit multiplication. Extending it to 4-bit multiplication is
analogous, albeit necessitating a larger Compute-ACAM array to accommodate all
the range combinations. Figure 7 illustrates the combinations of the two
operands ($x$ and $y$) that result in each bit of the product ($z$) being
equal to 1 (denoted by dots in the figure). For instance, the blue-circled dot
in the top-left subfigure of Figure 7 demonstrates that the output’s MSB
($z_{3}$) is 1 when $x=12$ and $y=1$. Since a cell stores a range of values,
there is no need to individually program every dot in Figure 7 in a Compute-
ACAM cell. Instead, we consolidate multiple dots into a single range if they
can form a rectangle, and store this range in a single Compute-ACAM cell. For
instance, the rectangle that contains the blue-circled dot could be programmed
in one Compute-ACAM cell storing the range of $(9\leq x<16)\land(0\leq y<4)$.
Therefore, the number of rectangles in each subfigure of Figure 7 determines
the number of Compute-ACAM cells required to implement the 4-bit
multiplication, i.e., computing $z_{3}$, $z_{2}$, $z_{1}$ and $z_{0}$ requires
a row of 8, 21, 36 and 58 Compute-ACAM cells, respectively.

###  IV-C 8-bit One-Variable Computing (Softmax)

Softmax stands as a critical component within Transformer models, consuming
approximately 20% to 40% of execution time on GPUs [4]. Even in specialized
accelerators designed for Transformer models, Softmax remains a challenging
design aspect due to its entailed exponentiation and division operations, both
of which incur high hardware costs. The Softmax function is defined as
follows:

$\text{softmax}(x_{i})=\frac{e^{x_{i}}}{\sum_{{}_{j}=1}^{L}e^{x_{j}}}$ (3)

Here, $x_{i}$ represents an element in the input vector
$\mathbf{X}=[x_{1},...,x_{L}]$, and $L$ denotes the input vector length.

Figure 8: Dataflow of computing Softmax using ACAM.

The expensive exponentiation and division operations can be easily mitigated
with Compute-ACAM-based computing. Figure 8 portrays the dataflow of Compute-
ACAM-based Softmax. We first leverage Compute-ACAM’s 8-bit one-variable
computing mode for exponentiation functions. Subsequently, these exponents are
aggregated to yield the denominator in Equation 3. To circumvent the division
operation and its associated hardware costs, we follow a similar approach as
presented in [53], replacing division with logarithm, subtraction, and
exponentiation based on the equivalence:

$\frac{a}{b}=e^{\left(\log(a)-\log(b)\right)}$ (4)

The logarithm can be implemented like exponentiation within the 8-bit one-
variable computing mode. Note that we do not need memory arrays to store pre-
computed results for exponentiation and logarithm, resulting in a more compact
and faster implementation compared to [53]. Given the strict order of
operations in Figure 8 due to data dependencies, most resources can be
effectively reused. Notably, steps 1 and 5 can utilize the same set of
Compute-ACAM arrays for computing the exponentiation, and since subtraction
can be implemented with adders, steps 2 and 4 can share the same set of
adders.

It is worth noting that the logarithm is undefined for input 0. Compute-ACAM
does not need any specific hardware to handle this special case. Instead, we
only need to hard set $\log(0)=m$ for the truth table of the logarithm, where
$m$ is the minimum value that can be represented by the data format of the
output.

##  V Optimizing Compute-ACAM Array Size

To implement an arbitrary function using Compute-ACAM, the number of cells
needed is determined by two key factors. The height (number of rows) is
determined by the output width since each ML generates a binary bit for the
output. The width (number of columns) is determined by the number of non-
contiguous input ranges for each output binary bit, as contiguous ranges can
be merged into a single range and stored in a single cell. Although the height
is predetermined once the output bit width is established, there are
opportunities to reduce the effective number of columns by merging more
ranges. In this section, we propose two methods aimed at diminishing the
number of Compute-ACAM cells needed to execute a function.

###  V-A Encoding

Revisiting the last column of the table in Figure 4(a) reveals the potential
for merging contiguous ranges, where the number of ranges for an output bit is
determined by the count of runs of 1s. For an $n$-bit one-variable function,
the total number of ranges is $2^{n}$, reaching a worst-case scenario with
interleaved 1s and 0s, resulting in $2^{n-1}$ runs of 1s. In Figure 7, the
worst-case scenario for an output bit of a two-variable function (with
operands of width $m$ and $n$ bits) is $2^{n+m-1}$, with interleaved dots and
blanks in the two-dimensional space. LSBs of the output exhibit a larger
number of non-contiguous ranges due to increased toggle frequency as input
values rise. Reducing the toggle rate of output bits enhances the potential
for merging multiple ranges.

TABLE I: 4-bit Gray Code Encoding. Decimal | Binary | Gray | Decimal | Binary | Gray
---|---|---|---|---|---
0 | 0000 | 0000 | 8 | 1000 | 1100
1 | 0001 | 0001 | 9 | 1001 | 1101
2 | 0010 | 0011 | 10 | 1010 | 1111
3 | 0011 | 0010 | 11 | 1011 | 1110
4 | 0100 | 0110 | 12 | 1100 | 1010
5 | 0101 | 0111 | 13 | 1101 | 1011
6 | 0110 | 0101 | 14 | 1110 | 1001
7 | 0111 | 0100 | 15 | 1111 | 1000

Figure 9: (a) Merging ranges for 4-bit GeLU after applying Gray Code on
output. (b) Merging ranges for 4-bit multiplication after applying Gray Code
on output.

To minimize toggling in the LSBs, we adopt Gray Code for the output, as
demonstrated in Table I for a 4-bit example. Originally designed to reduce
error probabilities in analog-to-digital signal translation, Gray Code ensures
only one bit changes at a time between consecutive numbers. This aligns with
our goal of reducing bit toggle frequency between consecutive numbers. Instead
of producing the original binary output bits, Compute-ACAM outputs Gray Code
encoded results. Figure 9 (a) and (b) display merged ranges after applying
Gray Code encoding to 4-bit GeLU and multiplications in Section III-A. A
comparison with the last column of the table in Figure 4(a) and Figure 7
reveals that encoding the output with Gray Code reduces the runs of 1s by
roughly a factor of 2 for all output bits, except the MSB, which already
exhibits the fewest ranges.

Because Compute-ACAM still takes original binary bits as input, we need to
convert the Gray Code encoded output back to the original binary format before
sending it for subsequent computations. The conversion process is very simple:
each bit in the binary representation can be obtained by performing an XOR
operation with all the higher-order bits in the Gray representation, except
the MSB, which is the same between these two formats, i.e.,

$b_{i}=\begin{cases}g_{i}&i=n-1\\\
XOR(g_{n-1},g_{n-2},...,g_{i+1})&i<n-1\end{cases}$

where $b_{i}$ and $g_{i}$ are the $i$th bit in the binary and Gray Code
format, respectively. $n$ is the bit width of the output. Therefore, the
conversion only needs some simple XOR gates.

###  V-B Finer-Grained Compute-ACAM Array Size

Figure 10: (a) Implementing 4-bit multiplication with a single large array and
(b) multiple small $4\times 8$ arrays whose MLs can be individually connected.
(c) A Compute-ACAM array group that supports configurable connection between
rows from different arrays.

Utilizing a single large array of Compute-ACAM cells to compute each output
bit results in significant cell underutilization, especially for higher-order
bits. Figure 10(a) shows cell utilization for Compute-ACAM-based 4-bit
multiplication with encoding applied, revealing that 51% of cells (denoted in
grey) remain unused, essentially functioning as a ’Don’t Care’ for input.
These unused cells cannot be repurposed for other functions, as they cannot be
excluded from participating in the computation of 4-bit multiplication.

To address this issue, we employ smaller-sized Compute-ACAM arrays and
introduce reconfigurability for constructing larger arrays with non-uniform
shapes. Multiple smaller-sized arrays are grouped together, allowing
individual row configurations within each array to connect to the global ML
with pull-down logic. Within the group, all arrays share the same set of sense
amplifiers and inverters (Figure 10(c)). For example, with a 4$\times$8 array
size in Figure 10(b), 1 row is allocated for the MSB and 5 rows for the LSB,
reducing unused cells to 12%. While small array size improves utilization, it
increases hardware overhead due to complex control circuits. Compute-ACAM,
supporting 4-bit and 8-bit computing modes, necessitates a minimum output bit
width of 4 bits; reducing the row size below 4 does not enhance utilization.
For a 4-bit input, the maximum non-contiguous range is 16, reducing to 8 after
Gray Code encoding, leading us to choose a 4$\times$8 array size in our
design. The number of Compute-ACAM arrays in a group is determined by the
worst-case number of cells needed to compute an output bit for an 8-bit one-
variable function. After applying Gray Code encoding, an 8-bit one-variable
function requires a maximum of 128 cells for one output bit, resulting in each
Compute-ACAM group containing 16 arrays.

##  VI Accelerator Architecture and Pipeline

Figure 11: Overall architecture of RACE-IT. TABLE II: RACE-IT parameters. Component |  Params |  Spec |  Power ($mW$) |  Area ($mm^{2}$)
---|---|---|---|---
Core (12 cores per tile)
DAC |  | resolution
---
number
| 1bit
---
$8\times 128$
0.95532 |  0.00006
S&A |  number |  $128$ |  0.95 |  0.02064
Memristor Array |  | number
---
size
bits per cell
| 8
---
$128\times 128$
2
2.4 |  0.0002
Adder |  number |  1024 |  12.2281 |  0.01032
Register File |  size |  4KB |  0.01573 |  0.00122
Control |  - |  - |  0.0597 |  0.00135
XOR |  number |  6144 |  0.1536 |  0.00098
Compute-ACAM Array |  | number
---
size
bits per cell
| 1536
---
$4\times 8$
4
19.16928 |  0.10899
Core Total |  - |  - |  35.93175 |  0.14378
Tile (121 tiles per chip)
eDRAM Buffer |  | size
---
num_banks
bus_width
| 256KB
---
2
256bits
0.17308 |  0.08001
eDRAM-to-IMA bus |  num_wire |  384 |  1.67181 |  0.0369
Router |  | flit_size
---
num_ports
| 32
---
8
10.03087 |  | 0.06191
---
(shared by
4 tiles)
Inst Mem |  size |  8KB |  0.02721 |  0.0024
Control |  - |  - |  0.11941 |  0.00059
Core |  number |  12 |  431.18 |  1.72547
Tile Total |  - |  - |  435.68 |  1.86087
Chip
Hyper Tr |  | links/freq
---
link bw
| 4/1.6GHz
---
6.4GB/s
2.483K |  9.3808
Tile |  number |  121 |  52.717K |  225.16573
Chip Total |  - |  - |  53.602K |  203.17369

Figure 11 illustrates the architecture of RACE-IT, with detailed parameters,
area, and power specifications for each component listed in Table II. RACE-IT
employs a two-level tile-core hierarchy, the same as in PUMA [1]. Each core
has a multi-issue pipeline similar to that found in a superscalar CPU. The
pipeline comprises three lanes, each capable of independently accepting new
instructions. Data dependencies between these lanes are resolved during the
compilation phase.

The Dot Product Engine (DPE) lane, similar to the MVM pipeline in PUMA and
ISAAC, differs in its use of Compute-ACAM-based ADCs in RACE-IT. Because the
amount of ADCs is determined by the number and size of crossbars, we reserve a
fixed amount of Compute-ACAM arrays to be configured as ADCs, specifically,
256 arrays are configured into 256 4-bit ADCs, 32 for each crossbar. The
configuration for crossbars and DACs follows ISAAC and PUMA, employing
$128\times 128$ crossbars with 1-bit DACs and 2-bit ReRAM cells. ISAAC’s
weight encoding scheme is also adopted to reduce the precision for analog-to-
digital conversion by 1 bit. The second lane consists of CMOS adders for
handling additions in residual layers, Softmax, and data-dependent
multiplications. We use CMOS adders because they offer excellent area and
power efficiency after decades of optimization. The final lane is a
reconfigurable General Computing Engine (GCE) composed of 1280 4$\times$8
Compute-ACAM arrays.

We extend the instruction set of PUMA with a new gce-compute instruction:

gce-compute unit-id, src, dest, vec-width

The unit-id field specifies which computing unit configured from Compute-ACAM
performs this computation. The source and destination register addresses are
indicated by src and dest fields. vec-width field indicates the length of the
vector operation.

Figure 12: Pipeline of RACE-IT for Transformer.

To harness the computation parallelism provided by the multi-issue core, we
carefully generate the instruction sequences for Transformer workloads to
increase the utilization of all the lanes. The computation of MHA is divided
into five stages, as illustrated in Figure 12. mvm stage computes one row of
Q, denoted by $\mathbf{q}=\mathbf{X_{Q}}[q,:]\cdot\mathbf{W_{Q}}$. matmul-1
stage performs data-dependent multiplications between $\mathbf{q}$ and
$\mathbf{K}$, i.e., $\mathbf{r}=\mathbf{q}\cdot\mathbf{K^{T}}$. div-add stage
involves dividing and applying masks on $\mathbf{r}$. softmax stage computes
the softmax on $\mathbf{r}$, resulting in
$\mathbf{s}=\text{Softmax}(\mathbf{r})$. matmul-2 stage computes another data-
dependent multiplication between $\mathbf{s}$ and $\mathbf{V}$, yielding the
final output $\mathbf{out}=\mathbf{s}\cdot\mathbf{V}$.

A sequence of these five stages computing for the same $\mathbf{out}$ is
referred to as a computing sequence. As we process the rows of $\mathbf{Q}$
sequentially, the computation of $\mathbf{K}$ and $\mathbf{V}$ for the next
$\mathbf{Q}$ can occur in parallel with the processing of the current
$\mathbf{Q}$. The order of these five stages within a computing sequence must
be preserved, but stages from different computing sequences can overlap since
there are no data dependencies between them. In cases where two stages from
different computing sequences request the same computing units, the later
issued one will be stalled until the requested resource is available. The
bottom part of Figure 12 shows the computing resources needed by each stage in
MHA. Because a core has a large number of adders (1024), matmul-1, div-add,
and softmax could use a different set of them.

The FFN only contains computation in conventional DNNs, such as linear layer,
activation, residual, etc., which have been extensively explored to be
implemented in ReRAM-based accelerators in prior works. The FFN and MHA can
form another pipeline at an upper level. Compared to MHA, FNN involves
considerably less computation, making this upper-level pipeline stage latency
primarily determined by MHA.

##  VII Methodology

We evaluated RACE-IT’s performance and energy efficiency by comparing it with
two popular server-grade GPUs (NVIDIA P100 and H100 GPUs) and previous ReRAM-
based accelerators that support Transformers. P100 GPU has 3,584 CUDA cores
with 16 GB memory capacity, operating at a base frequency of 1.3 GHz, while
the more recent H100 GPU has 6,144 CUDA cores with 32 GB memory capacity,
operating at a base frequency of 1.4 GHz. PUMA [1] is a programmable ReRAM-
based accelerator equipped with general-purpose VFUs to facilitate the
acceleration of diverse machine learning inference workloads, including
Transformer models. ReTransformer [53] is a recent ReRAM-based accelerator
designed explicitly for Transformers, utilizing crossbar functionality to
compute data-dependent matrix multiplications. We scale up the number of VFUs
in PUMA to match its chip area with that of RACE-IT and implement
ReTransformer on top of PUMA without utilizing the VFUs during computation.
Table III shows the significant differences between RACE-IT and the baseline
ReRAM-based accelerators

TABLE III: Comparison to prior works. Architecture |  | High-Cost
---
ADC
| Arbitrary
---
ML
| Program
---
Crossbar
| In-Memory
---
Attention
PUMA [1] |  Yes |  Yes |  No |  No
ReTransformer [53] |  Yes |  No |  Yes |  Yes
RACE-IT |  No |  Yes |  No |  Yes

We expanded the capabilities of PUMAsim [1] to simulate RACE-IT and
ReTransformer in a cycle-detailed manner. We use FinCACTI [42] to model the
power and area for all SRAM buffers and eDRAM at 16nm. ACAM parameters are
adopted from [22] and [31]. Other parameters are adopted from ISAAC and PUMA
and scaled to 16nm according to [47]. All the parameters are listed in Table
II. We use the CUDA event API in Pytorch to measure GPU performance and use
NVIDIA-SMI [29, 12] to measure GPU power. The performance and power
measurements are averaged across 30 runs, and we ignore the first run to
mitigate the influence of library initialization and cache warmup.

We conducted tests on diverse Transformer models, including Bert-Base [7],
Bert-Large [7], and GPT-2-Large [35]. Bert-Base is a moderate-size encoder-
only model comprising 110 million parameters, while Bert-Large is a more
extensive and computationally intensive model with 340 million parameters.
GPT-2-Large, a decoder-only model, boasts a substantial parameter count of 1.5
billion. For models exceeding single-chip capacity, we distribute their
weights across multiple RACE-IT chips. As RACE-IT processes the rows of
$\mathbf{Q}$ sequentially, a few hundred to one thousand bytes are generated
at a time, which can be transferred in parallel with the computation of next
row. The inter-chip connection bandwidth is 1.6GB/s, ensuring minimal
throughput and energy overheads. Besides Transformers, ResNet50 [11] was
incorporated for benchmarking conventional CNN models. Pre-trained Transformer
models on GLUE [51] and SQuAD [36] datasets were obtained from the Hugging
Face Model Hub [52], and ResNet50 weights from TorchVision [39]. For accuracy
metrics, we use Matthews correlation for CoLA, Pearson correlation for STS-B,
F1 for SQuAD, and accuracy for all remaining tasks. Because the same
Transformer model on different tasks only differs in the last linear layer
(called classification head), we only report the performance and energy of the
backbone of the Transformer models without the classification heads.

##  VIII Experimental Results

###  VIII-A Evaluation of Compute-ACAM-base Computing Units

TABLE IV: Area and power Comparison between Compute-ACAM-based (w/ and w/o encoding) and CMOS-based operators. Operator | Compute-ACAM | CMOS
---|---|---
Encode |  Power (mW) |  Area ($um^{2}$) |  Power (mW) |  Area ($um^{2}$)
4-bit ADC | No | 0.012 | 70.9 | 0.113 | 116
Yes | 0.012 | 70.9
4-bit Mult | No | 0.053 | 301 | 0.00225 | 1104
Yes | 0.034 | 195
8-bit GeLU | No | 0.078 | 443 | 0.334 | 1054
Yes | 0.059 | 337
8-bit Softmax | No | 0.124 | 648 | 0.077 | 1131
Yes | 0.099 | 506

Table IV compares the area and power consumption for multiple operators in
Transformer models using Comput-ACAM-based computing in RACE-IT and
conventional digital CMOS circuitry.

Parameters for the 4-bit ADC are adopted from [54], the GeLU unit is
implemented using a non-uniform piecewise approximation [38], and parameters
for the Softmax unit are adopted from [17]. All parameters are scaled down to
16nm. Across all operators, Compute-ACAM-based computing units are 39% to 82%
smaller than their CMOS counterparts. Additionally, Compute-ACAM-based
computing units exhibit lower power consumption, except for multipliers and
Softmax units. The utilization of encoding yields a notable reduction in the
area and power of Compute-ACAM-based operators, ranging from 22% to 35%.
However, the 4-bit ADC does not exhibit a reduction in area and energy because
the unencoded implementation already fits within one 4$\times$8 Compute-ACAM
array. As ADCs constitute the majority of power in ReRAM-based accelerators,
RACE-IT maintains significantly higher power efficiency compared to other
ReRAM-based accelerators, as detailed in Section VIII-B.

###  VIII-B Performance and Energy

TABLE V: Comparison of Computation and Energy Efficiency. Architecture | Bert-Base | Bert-Large | GPT-2-Large | ResNet50 | Peak/Core
---|---|---|---|---|---
TOPS | TOPS/W | TOPS | TOPS/W | TOPS | TOPS/W | TOPS | TOPS/W | TOPS | TOPS/W
PUMA | 19.27 | 27.48 | 33.59 | 34.87 | 42.16 | 18.59 | 3.33 | 128.88 | 6.31 | 130.21
ReTransformer | 64.63 | 28 | 89.04 | 36.14 | 182.76 | 69.03 | 3.33 | 128.88 | 6.31 | 130.21
RACE-IT | 110.11 | 109 | 191.90 | 129.1 | 268.2 | 80 | 3.8 | 194.14 | 7.07 | 196.94

Figure 13: (a) Inference speedup and (b) energy savings normalized to NVidia
P100 GPU. The highlighted area shows the Transformer models performance.

Figure 13(a) shows the speedup of all tested architectures normalized to P100
GPU. All three ReRAM-based designs surpass GPU baselines thanks to the
enhanced MVM parallelism facilitated by the crossbar structure. RACE-IT
significantly outperforms PUMA and ReTransformer for Transformer models. PUMA
encounters computational bottlenecks as it performs data-dependent matrix
multiplications within a limited number of VFUs. Despite we scaled up the
number of VFUs in PUMA to match its chip area with that of RACE-IT for a fair
comparison, each PUMA core still can only execute 64 multiplications at a
time. Consequently, PUMA achieves just 6.4$\times$ speedup over the P100 GPU
on Transformer models. On the other hand, ReTransformer, which computes data-
dependent matrix multiplications within the crossbar, offers higher
computation parallelism but is constrained by the time-consuming crossbar
write operation. Therefore, ReTransformer achieves 9.3$\times$ speedup over
the P100 GPU on Transformer models. RACE-IT optimizes the majority of its
ACAMs as multipliers for data-dependent multiplications, achieving high
computation parallelism (details on the balanced configuration can be found in
Section VIII-D). In comparison to the P100 GPU, H100 GPU, PUMA, and
ReTransformer, RACE-IT achieves speedups of 38$\times$, 10.7$\times$,
5.9$\times$, and 4$\times$, respectively. Note that larger models result in a
larger speedup for RACE-IT, which can exploit at best the high parallelism.
RACE-IT is still capable of accelerating conventional CNNs, with improved
performance than PUMA and ReTransformer, because of much larger amount of
activation units configured from ACAMs. As an example, we compared the
inference of ResNet50 with the other architectures, showing 1.14$\times$
speedup.

Figure 14: Accuracy comparison for (a) BERT-Base and (b) BERT-Large for full
precision, uniform quantization and Power of Two quantization

Figure 13(b) shows the energy saving of all tested architectures normalized to
P100 GPU. Once again, all three ReRAM-based architectures outperform GPU
baselines in terms of energy saving. Notably, ReTransfomer exhibits the
highest energy consumption among those three due to frequent crossbar writes.
Although ReTransfomer employs a decomposed data-dependent multiplication
strategy to mitigate the latency impact of excessive crossbar writes, it also
reduces data reuse, thus increasing computational demands. RACE-IT achieves
improved energy savings over PUMA and ReTransfomer, primarily due to the
elimination of conventional energy-intensive ADCs. Consequently, RACE-IT
demonstrates the greatest energy savings for both Transformer models and CNNs.
In comparison to PUMA and ReTransfomer, RACE-IT achieves 3.9$\times$ and
5.8$\times$ more energy savings on Transformer models and 1.5$\times$ more
energy savings on ResNet50.

Table V shows the computation efficiency (TOPS) and energy efficiency (TOPS/W)
of the three ReRAM-based architectures. RACE-IT achieves both the best
computation efficiency and energy efficiency for all benchmarks. RACE-IT has
1.12$\times$ higher peak computation efficiency and 1.56$\times$ higher peak
power efficiency over PUMA and ReTransformer.

###  VIII-C Accuracy

Given that ACAMs can only be configured to accept at most 8-bit operands,
Transformer models need to be quantized. While the topic of quantization needs
a separate paper to explore best practices for Transformers, this section
focuses on delineating the unique quantization requirements of RACE-IT and
presents an evaluation of one simple solution. Despite the widespread
exploration of quantization techniques, the majority has primarily
concentrated on CNNs, particularly in the quantization of layers with
trainable weights. However, Transformer models involve numerous non-
parameterized computations, such as data-dependent multiplication and Softmax,
which also require quantization in RACE-IT. Figure 14 illustrates the accuracy
of employing full-precision data-dependent multiplication and Softmax, along
with the application of two different quantization schemes. Since RACE-IT
computes Softmax in five steps (Figure 8), separate quantization is essential.
However, straightforward application of uniform quantization yields poor
accuracy, i.e., 47% accuracy loss. This is because uniform quantization
assumes a uniform distribution on the underlying data, while the output of
exponent functions (steps 1 and 5 in Figure 8) exhibit an exponential
distribution. To address it, we utilized the Power of Two (PoT) quantization
[27, 57], effectively resolving this issue. PoT quantizes values to the powers
of two, particularly suited for value distributions with a sparse and power-
law nature. We apply PoT on exponent functions and uniform quantization on all
other operations, and the accuracy loss reduces to 0.2%. Importantly, since
our ACAM can be configured for arbitrary functions, it is easy to support
arbitrary non-uniform quantization schemes.

###  VIII-D GCE Configuration

Figure 15: Speedup over NVidia P100 GPU as a function of $k$, namely the ratio
between the allocation of ACAMs as multipliers and exponential units. The
dashed line shows our design choice.

Transformer models necessitate four types of computing units configured from
GCE: multipliers for data-dependent multiplications, exponentiation units and
logarithm units for Softmax, as well as activation units for computing the
GeLU activation. From Figure 8, we know that only one logarithm operation is
required for Softmax, while the number of exponentiation operations depends on
the input size. Activation is only computed in FFN. As described in Section
VI, FFN has much fewer computations compared to MHA and, therefore does not
pose a critical impact on pipeline stage latency. Even with only one
activation unit per core, FFN remains faster than MHA. Consequently, our
exploration focuses on finding the optimal tradeoff between multipliers and
exponentiation units, with logarithm units and activation units fixed at 1. We
represent the ratio between multipliers and exponentiation units configured
from GCE with $k$, ensuring it does not exceed the available Compute-ACAMs in
the core. Figure 15 indicates that the optimal $k$ falls between 3.7 to 38 for
Bert-Base and Bert-Large and 13.4 to 38 for GPT-2-Large. For all benchmarks,
we adopted $k=28.3$, corresponding to 454 multipliers and 16 exponent units,
denoted by the vertical line in the figure.

##  IX Conclusion

In this paper, we present RACE-IT, a reconfigurable ReRAM-based accelerator
for Transformer architectures. Our multi-lane ReRAM-based accelerator
leverages a novel Compute-ACAM structure capable of computing arbitrary 4-bit
and 8-bit one-variable, as well as 4-bit two-variable functions. It
demonstrates superior performance, outpacing both GPU and existing ReRAM-based
baselines by a significant margin. We demonstrate 10.7× and 5.9× speedup,
1193× and 3.9× increased energy efficiency compared with state-of-the-art GPU
and IMC accelerators, respectively.

## References

  * [1] A. Ankit, I. E. Hajj, S. R. Chalamalasetti, G. Ndu, M. Foltin, R. S. Williams, P. Faraboschi, W.-m. W. Hwu, J. P. Strachan, K. Roy _et al._ , “Puma: A programmable ultra-efficient memristor-based accelerator for machine learning inference,” in _Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems_ , 2019, pp. 715–731.
  * [2] J. Borghetti, G. S. Snider, P. J. Kuekes, J. J. Yang, D. R. Stewart, and R. S. Williams, “‘memristive’switches enable ‘stateful’logic operations via material implication,” _Nature_ , vol. 464, no. 7290, pp. 873–876, 2010.
  * [3] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , vol. 33, pp. 1877–1901, 2020.
  * [4] J. Choi, H. Li, B. Kim, S. Hwang, and J. H. Ahn, “Accelerating transformer networks through recomposing softmax layers,” in _2022 IEEE International Symposium on Workload Characterization (IISWC)_. IEEE, 2022, pp. 92–103.
  * [5] K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, “Electra: Pre-training text encoders as discriminators rather than generators,” in _International Conference on Learning Representations_ , 2020.
  * [6] Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,” in _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_. Association for Computational Linguistics, 2019, pp. 2978–2988.
  * [7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, 2019, pp. 4171–4186.
  * [8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021.
  * [9] S. Gupta, M. Imani, and T. Rosing, “Felix: Fast and energy-efficient logic in memory,” in _2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_. IEEE, 2018, pp. 1–7.
  * [10] R. Hanhan, E. Garzón, Z. Jahshan, A. Teman, M. Lanuzza, and L. Yavits, “Edam: Edit distance tolerant approximate matching content addressable memory,” in _Proceedings of the 49th Annual International Symposium on Computer Architecture_ , ser. ISCA ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 495–507. [Online]. Available: https://doi.org/10.1145/3470496.3527424
  * [11] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [12] H. H. Holm, A. R. Brodtkorb, and M. L. Sætra, “Gpu computing with python: Performance, energy efficiency and usability,” _Computation_ , vol. 8, no. 1, p. 4, 2020.
  * [13] M. Hu, J. P. Strachan, Z. Li, E. M. Grafals, N. Davila, C. Graves, S. Lam, N. Ge, J. J. Yang, and R. S. Williams, “Dot-product engine for neuromorphic computing: Programming 1t1m crossbar to accelerate matrix-vector multiplication,” in _Proceedings of the 53rd Annual Design Automation Conference_ , ser. DAC ’16. New York, NY, USA: Association for Computing Machinery, 2016. [Online]. Available: https://doi.org/10.1145/2897937.2898010
  * [14] M. Imani, A. Rahimi, D. Kong, T. Rosing, and J. M. Rabaey, “Exploring hyperdimensional associative memory,” in _2017 IEEE International Symposium on High Performance Computer Architecture (HPCA)_ , 2017, pp. 445–456.
  * [15] M. Kang, H. Shin, J. Shin, and L.-S. Kim, “A framework for area-efficient multi-task bert execution on reram-based accelerators,” in _2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD)_. IEEE, 2021, pp. 1–9.
  * [16] N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher, “CTRL: A conditional transformer language model for controllable generation,” _CoRR_ , vol. abs/1909.05858, 2019.
  * [17] I. Kouretas and V. Paliouras, “Hardware implementation of a softmax-like function for deep learning,” _Technologies_ , vol. 8, no. 3, p. 46, 2020.
  * [18] R. Krishnamoorthi, “Quantizing deep convolutional networks for efficient inference: A whitepaper,” _arXiv preprint arXiv:1806.08342_ , 2018.
  * [19] S. Kvatinsky, G. Satat, N. Wald, E. G. Friedman, A. Kolodny, and U. C. Weiser, “Memristor-based material implication (imply) logic: Design principles and methodologies,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , vol. 22, no. 10, pp. 2054–2066, 2013.
  * [20] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” in _International Conference on Learning Representations_ , 2020.
  * [21] M. Le Gallo, R. Khaddam-Aljameh, M. Stanisavljevic, A. Vasilopoulos, B. Kersting, M. Dazzi, G. Karunaratne, M. Brändli, A. Singh, S. M. Müller, J. Büchel, X. Timoneda, V. Joshi, M. J. Rasch, U. Egger, A. Garofalo, A. Petropoulos, T. Antonakopoulos, K. Brew, S. Choi, I. Ok, T. Philip, V. Chan, C. Silvestre, I. Ahsan, N. Saulnier, V. Narayanan, P. A. Francese, E. Eleftheriou, and A. Sebastian, “A 64-core mixed-signal in-memory compute chip based on phase-change memory for deep neural network inference,” _Nature Electronics_ , vol. 6, no. 9, pp. 680–693, Sep. 2023. [Online]. Available: https://doi.org/10.1038/s41928-023-01010-1
  * [22] C. Li, C. E. Graves, X. Sheng, D. Miller, M. Foltin, G. Pedretti, and J. P. Strachan, “Analog content-addressable memories with memristors,” _Nature communications_ , vol. 11, no. 1, p. 1638, 2020.
  * [23] W. Li, M. Manley, J. Read, A. Kaul, M. S. Bakir, and S. Yu, “H3datten: Heterogeneous 3-d integrated hybrid analog and digital compute-in-memory accelerator for vision transformer self-attention,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , 2023.
  * [24] W. Li, H. Liu, R. Ding, M. Liu, P. Wang, and W. Yang, “Exploiting temporal contexts with strided transformer for 3d human pose estimation,” _IEEE Transactions on Multimedia_ , vol. 25, pp. 1282–1293, 2022.
  * [25] T. Liang, J. Glossner, L. Wang, S. Shi, and X. Zhang, “Pruning and quantization for deep neural network acceleration: A survey,” _Neurocomputing_ , vol. 461, pp. 370–403, 2021.
  * [26] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.
  * [27] D. Miyashita, E. H. Lee, and B. Murmann, “Convolutional neural networks using logarithmic data representation,” _arXiv preprint arXiv:1603.01025_ , 2016.
  * [28] M. Nagel, M. v. Baalen, T. Blankevoort, and M. Welling, “Data-free quantization through weight equalization and bias correction,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1325–1334.
  * [29] Nvidia, “Nvidia system management interface,” https://developer.nvidia.com/nvidia-system-management-interface, 2019.
  * [30] G. Pedretti, E. Ambrosi, and D. Ielmini, “Conductance variations and their impact on the precision of in-memory computing with resistive switching memory (rram),” in _2021 IEEE International Reliability Physics Symposium (IRPS)_ , 2021, pp. 1–8.
  * [31] G. Pedretti, C. E. Graves, S. Serebryakov, R. Mao, X. Sheng, M. Foltin, C. Li, and J. P. Strachan, “Tree-based machine learning performed in-memory with memristive analog cam,” _Nature communications_ , vol. 12, no. 1, p. 5806, 2021.
  * [32] G. Pedretti, J. Moon, P. Bruel, S. Serebryakov, R. M. Roth, L. Buonanno, T. Ziegler, C. Xu, M. Foltin, J. Ignowski _et al._ , “X-time: An in-memory engine for accelerating machine learning on tabular data with cams,” _arXiv preprint arXiv:2304.01285_ , 2023.
  * [33] C. Pei, Y. Zhang, Y. Zhang, F. Sun, X. Lin, H. Sun, J. Wu, P. Jiang, J. Ge, W. Ou _et al._ , “Personalized re-ranking for recommendation,” in _Proceedings of the 13th ACM conference on recommender systems_ , 2019, pp. 3–11.
  * [34] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever _et al._ , “Improving language understanding by generative pre-training,” 2018.
  * [35] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , vol. 1, no. 8, p. 9, 2019.
  * [36] P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for squad,” _arXiv preprint arXiv:1806.03822_ , 2018.
  * [37] M. S. Razlighi, M. Imani, F. Koushanfar, and T. Rosing, “Looknn: Neural network with no multiplication,” in _2017 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 2017.
  * [38] E. Reggiani, R. Andri, and L. Cavigelli, “Flex-sfu: Accelerating dnn activation functions by non-uniform piecewise approximation,” _arXiv preprint arXiv:2305.04546_ , 2023.
  * [39] F. A. Research, “Torchvision - pytorch vision library,” https://github.com/pytorch/vision, 2022.
  * [40] B. Rokh, A. Azarpeyvand, and A. Khanteymoori, “A comprehensive survey on model quantization for deep neural networks in image classification,” _ACM Transactions on Intelligent Systems and Technology_ , 2023.
  * [41] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” _arXiv preprint arXiv:1910.01108_ , 2019.
  * [42] A. Shafaei, Y. Wang, X. Lin, and M. Pedram, “Fincacti: Architectural analysis and modeling of caches with deeply-scaled finfet devices,” in _2014 IEEE Computer Society Annual Symposium on VLSI_. IEEE, 2014, pp. 290–295.
  * [43] A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubramonian, J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar, “Isaac: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars,” _ACM SIGARCH Computer Architecture News_ , vol. 44, no. 3, pp. 14–26, 2016.
  * [44] X. Sheng, C. E. Graves, S. Kumar, X. Li, B. Buchanan, L. Zheng, S. Lam, C. Li, and J. P. Strachan, “Low-conductance and multilevel cmos-integrated nanoscale oxide memristors,” _Advanced Electronic Materials_ , vol. 5, no. 9, p. 1800876, 2019. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/aelm.201800876
  * [45] A. Siemon, S. Menzel, R. Waser, and E. Linn, “A complementary resistive switch-based crossbar array adder,” _IEEE journal on emerging and selected topics in circuits and systems_ , vol. 5, no. 1, pp. 64–74, 2015.
  * [46] T. Song, X. Chen, X. Zhang, and Y. Han, “Brahms: Beyond conventional rram-based neural network accelerators using hybrid analog memory system,” in _2021 58th ACM/IEEE Design Automation Conference (DAC)_. IEEE, 2021, pp. 1033–1038.
  * [47] A. Stillmaker and B. Baas, “Scaling equations for the accurate prediction of cmos device performance from 180 nm to 7 nm,” _Integration_ , vol. 58, pp. 74–81, 2017.
  * [48] F. Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang, “Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer,” in _Proceedings of the 28th ACM international conference on information and knowledge management_ , 2019, pp. 1441–1450.
  * [49] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [50] W. Wan, R. Kubendran, C. Schaefer, S. B. Eryilmaz, W. Zhang, D. Wu, S. Deiss, P. Raina, H. Qian, B. Gao, S. Joshi, H. Wu, H.-S. P. Wong, and G. Cauwenberghs, “A compute-in-memory chip based on resistive random-access memory,” _Nature_ , vol. 608, no. 7923, pp. 504–512, Aug. 2022. [Online]. Available: https://doi.org/10.1038/s41586-022-04992-8
  * [51] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural language understanding,” _arXiv preprint arXiv:1804.07461_ , 2018.
  * [52] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz _et al._ , “Huggingface’s transformers: State-of-the-art natural language processing,” _arXiv preprint arXiv:1910.03771_ , 2019.
  * [53] X. Yang, B. Yan, H. Li, and Y. Chen, “Retransformer: Reram-based processing-in-memory architecture for transformer acceleration,” in _Proceedings of the 39th International Conference on Computer-Aided Design_ , 2020, pp. 1–9.
  * [54] G. Yuan, P. Behnam, Z. Li, A. Shafiee, S. Lin, X. Ma, H. Liu, X. Qian, M. N. Bojnordi, Y. Wang _et al._ , “Forms: Fine-grained polarized reram-based in-situ computation for mixed-signal dnn accelerator,” in _2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA)_. IEEE, 2021, pp. 265–278.
  * [55] X. Zhao, Y. Wang, X. Cai, C. Liu, and L. Zhang, “Linear symmetric quantization of neural networks for low-precision integer hardware,” in _International Conference on Learning Representations_ , 2020.
  * [56] Q. Zheng, S. Li, Y. Wang, Z. Li, Y. Chen, and H. H. Li, “Accelerating sparse attention with a reconfigurable non-volatile processing-in-memory architecture,” in _2023 60th ACM/IEEE Design Automation Conference (DAC)_. IEEE, 2023, pp. 1–6.
  * [57] A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen, “Incremental network quantization: Towards lossless cnns with low-precision weights,” _arXiv preprint arXiv:1702.03044_ , 2017.
  * [58] H. Zhu, K. Zhu, J. Gu, H. Jin, R. T. Chen, J. A. Incorvia, and D. Z. Pan, “Fuse and mix: Macam-enabled analog activation for energy-efficient neural acceleration,” in _Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design_ , 2022, pp. 1–9.