Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

PhysioCHI: Towards Best Practices for Integrating Physiological Signals in
HCI, May 11, 2024, Honolulu, HI, USA

<EMAIL_ADDRESS>]

[email=<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

<EMAIL_ADDRESS>]

# Implicit gaze research for XR systems

Naveen Sendhilnathan  Reality Labs Research, Meta Platforms, WA, USA  Ajoy S.
Fernandes  Michael J. Proulx  Tanya R. Jonker

(2024)

###### Abstract

Although eye-tracking technology is being integrated into more VR and MR
headsets, the true potential of eye tracking in enhancing user interactions
within XR settings remains relatively untapped. Presently, one of the most
prevalent gaze applications in XR is input control; for example, using gaze to
control a cursor for pointing. However, our eyes evolved primarily for sensory
input and understanding of the world around us, and yet few XR applications
have leveraged natural gaze behavior to infer and support users’ intent and
cognitive states. Systems that can represent a user’s context and interaction
intent can better support the user by generating contextually relevant
content, by making the user interface easier to use, by highlighting potential
errors, and more. This mode of application is not fully taken advantage of in
current commercially available XR systems and yet it is likely where we’ll
find paradigm-shifting use cases for eye tracking. In this paper, we elucidate
the state-of-the-art applications for eye tracking and propose new research
directions to harness its potential fully.

###### keywords:

Eye tracking extended reality explicit gaze implicit gaze

##  1 Introduction

The integration of eye-tracking technology into virtual reality and mixed
reality (XR) headsets is becoming increasingly common. However, the extent of
the role of eye tracking in these systems has yet to be discovered, and,
likely, its greatest potential has yet to be realized.

Currently, a popular use of gaze applications in XR systems revolves around
using it as an explicit input control mechanism [1]. For example, eye tracking
might be used to control a cursor for object targeting, which may be visible
or invisible to the user. Although eye-tracking-based input has been used in
many interactions and interfaces, this approach presents various limitations
which could result in fatigue, both physically from moving their eyes
unnaturally, and mentally, as users exert willpower, going against how they
would naturally attend to objects to maintain focus. Although improvements in
explicit input control will continue to unlock opportunities in interactions,
the number of applications might be limited relative to the opportunities
presented when using eyes consistently with how they naturally move and how
they have evolved.

Perhaps the most promising domain for the transformative value of eye tracking
for XR lies in inferring user interaction intent and state. Users’ natural
gaze behavior provides a uniquely promising "implicit" signal to a user’s
internal state. Indeed, spatiotemporal gaze dynamics have been shown to
correlate strongly with various cognitive and motor processes, including
visual attention [2], motor preparation and execution [3, 4], user confidence
[5, 6], decision-making [7], frustration [8], and performance monitoring [9].
Therefore, while utilizing gaze as a controller might be a necessary starting
point to catalyze explorations, a fruitful and vast research opportunity
involves exploring how to add value to end users with help from their natural
gaze to infer their intended future actions and cognitive states. This
information can significantly enhance user experience, for example, by
enabling personalization, content recommendations, adaptive guidance, and
more. For example, the system might automatically bring up weather information
when the user looks outside their window before they head out to work, or it
might simply bring up the definition of a word if the user fixates steadily on
that word within a blurb of text. However, the largely untapped potential of
understanding and utilizing natural gaze behavior in current commercially
available XR systems creates a gap in realizing the full capabilities of this
technology.

In this workshop paper, we first provide a detailed definition of the two
modes of gaze interactions in XR (explicit and implicit). We then provide a
comprehensive discussion about the current state-of-the-art applications,
highlighting their advantages and limitations. Finally, we propose suggestions
and recommendations for advancing eye-tracking research and outline the
necessary future work to establish an XR interaction platform where eye
tracking serves as a significant modality, enhancing user experience and
facilitating effortless XR interactions.

##  2 Modes of Gaze Interaction in XR Systems

Gaze can be used in two different modes of interaction [10]. First, gaze can
be used in the “explicit mode”, where one must control their eye movements to
achieve a particular outcome. For example, a user might direct their eyes to
particular interactable elements of a UI, so that they can navigate and manage
the UI (e.g., using gaze position for cursor control or using dwell time and
gaze gestures for selection [1]). Alternatively, gaze can be used in the
"implicit mode", where natural eye movements are preserved and can reflect a
user’s internal cognitive and emotional state. “Implicit” behaviors capture
intentions, surprise, cognitive load, strain, learnability, and skill level.

The main differences between the explicit and implicit modes of interactions
are summarized in Table 1 and will be expanded in detail in the following
sections.

Table 1: Explicit and Implicit Modes of Interaction |  Explicit gaze interactions |  Implicit gaze interactions
---|---|---
Mode of UI control |  Eye gaze |  Other modalities such as hand
Primary use of gaze |  Pointing, selecting |  Visual search, passive gaze grazing etc
Distinct signatures of interaction |  Specific gaze features such as saccade vigor, velocity might be higher due to enhanced goal-directed movements. Only overt attention is used. |  Both overt and covert attention are utilized
Advantages |  Can be used to make high bandwidth interaction based selections quickly. Can be used when hands are occupied / unable to provide input. |  User’s natural gaze behavior is used to infer intentions, surprise, cognitive load, strain, learnability, skill level etc.
Limitations |  Physical load of using eyes explicitly is high and might cause eye fatigue with prolonged use. Often suffers from Midas Touch problem and needs high-quality design to bypass it. Inability to use eyes to survey environment, spot dangers, etc. |  Degeneracy of gaze behavior: Same / similar implicit gaze dynamics could be picked up by different machine learning models to decode different states.

##  3 Applications of Explicit Mode of Gaze Interactions

Gaze interactions allows users to interact with elements in their field of
view by explicitly looking at them and selecting them. Some examples of this
type of interaction include typing on keyboards by looking at the keys and
pressing a button [11] or using gaze as a cursor to navigate a UI and select
elements using fixations or other gaze gestures [12] and redirect stylus input
to gaze location [13]. The complexity of these interactions primarily depends
on the density of UI elements, the coverage required for the user population
and the eye tracker hardware capabilities such as accuracy, precision, depth
estimate, frame rate, latency and also the types of interactions that the
system is designed to achieve.

Design choices, such as refining gaze targeting, determining the
characteristics of interactable objects like item dimensions, padding, and
spacing, as well as defining interaction techniques and providing visual
feedback, can effectively tackle specific user behaviors and offset
inaccuracies in eye tracking [14]. For instance, instead of strictly requiring
the gaze cursor to be directly on a button for selection, one can utilize a
margin between items. This can further be made dynamic based on context. For
example, dynamically increase the margin size of certain objects that are most
likely the target objects based on user history, time, etc. Additionally,
providing visual feedback, such as highlighting buttons, has the potential to
enhance the user experience and increase efficiency.

##  4 Applications of Implicit Mode of Gaze Interactions

Gathering real-time gaze information throughout different temporal stages,
including before, during, and after an interaction, enables the inference of
users’ cognitive and behavioral states. Combined with the temporal axis, we
explore two areas of research: 1) using implicit gaze for inferring
interaction intent, and 2) leveraging implicit gaze for inferring the user’s
cognitive state.

###  4.1 Using Implicit Gaze to Predict the Time and the Target for Point and
Select Interaction

Monitoring user gaze is a potential tool for evaluating design usability and
enhancing workflow efficiency. In interactions involving multiple modalities,
the combination of implicit gaze behavior and preceding hand movements, as
demonstrated by David-John et al. [15], reliably predicts user’s intended
interactions. Such machine learning models that leverage gaze information to
forecast user intent, assisting in tasks such as highlighting objects for
selection and reducing false positive activations, thus mitigating physical
strain and fatigue. While such approaches, including the utilization of
multimodal biosignals to predict target locations and interaction intent, are
feasible, their accuracy depends on contextual features. For example, in the
context of grocery shopping, knowledge of aisle locations informs potential
targets, and item features aid in the selection process. A comprehensive model
would take into consideration multimodal biosignals, context, and semantics,
enabling the precise prediction of upcoming targets with high precision few
tens of milliseconds or even seconds before user indication.

###  4.2 Using Implicit Gaze to Enhance Interaction Primitives

XR interactions like swipe typing, touch typing, scrolling, pointing, and
selecting often demand significant physical and mental effort to be accurate.
Leveraging contextual signals and implicit gaze can enhance user experiences,
particularly in text entry and UI navigation. Using implicit gaze for
wristband typing improves signal quality and reduces the need for precise
input. In gesture typing for example swipe typing [16], aligning cursor speed
with gaze fixation reduces hand movement by 30% without compromising speed or
accuracy. Dynamic acceleration adjustment near gaze fixation reduces hand
movement by 22%. Low effort wrist input, enabled by Flexible Pointing,
Guidelines, and Magnets, facilitates quick interactions in a 2D UI during
concurrent activities. Enhancements like gaze-based Magnets and scanpath
history anticipate future targets, improving pointing precision and reducing
interaction friction.

###  4.3 Using Implicit Gaze to Detect Errors After User Interaction

Despite the rapid advancements, the interaction with XR systems is still in
its early stages of development due to ‘interaction errors’ that occur when
there is a discrepancy between the system’s response to a user’s action and
the user’s anticipated outcome of that action. These errors usually arise from
limitations in sensing, gesture recognition, and system decision-making. For
instance, when a user attempts to pick up a pen using their index finger and
thumb, the system may erroneously interpret it as a pinch gesture, resulting
in unexpected system behaviors. These errors have a detrimental impact on user
experiences as they cause frustration, necessitate manual corrections, and
decrease trust in the system. In severe cases, they can even lead to users
abandoning the system altogether. Recent research has shown that natural gaze
can be used to detect user and system errors as early as 500-600 ms after
their occurrence [17, 9] in a point-and-select interaction task. Further
research is necessary to build models that can generalize across different
interaction methods and recognize different types of user and system errors.
Such rapid detection of interaction errors can decrease the cognitive and
motor loads of users and improve the usability of systems.

###  4.4 Gaze Metrics To Estimate Users’ Cognitive State

In addition to using implicit gaze signals for user interactions, they can
also serve to estimate a user’s cognitive processes, which are not otherwise
easily accessible. Gaze metrics derived from spatial and temporal gaze
patterns are strong correlates of cognitive and motor processes such as visual
attention [18], motor preparation and execution [4], decision making [19],
performance monitoring [9], and more. Therefore, implicit gaze behavior is a
valuable modality for investigating and measuring the cognitive state of the
user while performing a task.

Presently, manual adjustments are necessary in MR to determine the visibility
of applications and the extent of information they present. This choice must
be modified each time users transition between contexts, such as changing
tasks or environments. Considering the frequent occurrences of context
switches throughout the day, introducing automation to MR interfaces becomes
essential to address this challenge. A real-time optimization-based approach
could be developed by inferring the user’s current cognitive load using gaze
metrics (for example, the Index of Pupillary Activity [20]) and knowledge of
their task and surroundings [21]. This approach would enable automatic control
over the timing, location, and amount of information displayed by
applications.

Implicit gaze effectively measures user confidence while manually mid-air
typing on a virtual keyboard in VR [6]. For instance, postural reinforcement
haptic feedback reduced expert QWERTY typists’ visual attention to the mid-air
keyboard by 44%, as shown by increased comfort, higher angular distance
between gaze and key, and reduced gaze switching. Gaze metrics offer an
objective way to quantify subjective user ratings, providing an alternative
perspective on user confidence during typing.

Gaze metrics may also be employed to estimate the user’s current skill level
and assess performance [22]. Essentially, gaze data can be utilized to create
metrics related to expertise [23], enabling the tracking of learning
progression [24]. Establishing gaze-based metrics that identify trends in user
behavior during XR experiences can contribute to an expertise engine. This
engine interacts with the user and system, providing feedback to enhance skill
levels and overall experience. For example, increased expertise might require
adjusting feedback levels or switching modalities. By integrating expertise-
based gaze metrics, the expertise engine facilitates these adaptations,
improving the user’s skill level and technology on-boarding experience.

###  4.5 Application: Using Implicit Gaze And Context to Optimize User
Interface Prior To Interaction

Contextual and Adaptive UIs provide needle-in-the-haystack interfaces, most
aligned with a user’s context, in a timely manner for users to interact with
without exerting willpower to bring up that interface. A user’s context
relates to the user’s current environment. It could include the time,
location, task being performed, objects in the environment and the state of
these objects, and many more. Context and gaze analysis enhance interaction
efficiency by preloading content and displaying pertinent UI elements before
user engagement. By assessing the user’s context, and considering factors like
task, location, and environment, the system can anticipate needs and load
relevant content. For example, given the location of the aisle the user is in
and given other contextual factors such as user selection history, time of the
day etc, the system can preload and adjust the shopping list to reflect the
most relevent items for the user at that moment. Gaze behavior, tracking eye
movements and fixations, offers insights into attention and intention [25,
26]. Recognizing areas of interest allows a prioritized display of relevant UI
elements. For example, if the user fixates on an object in the world, the
system can dynamically load and make it interactable, anticipating it as the
intended target [27, 28]. This proactive approach reduces cognitive load since
the the user is given the right amount of information at the right time
without being overloaded, streamlining access to information and actions,
ensuring a smoother and more efficient user experience without delays or
additional searching.

##  5 Operating gaze in explicit and implicit modes

Head mounted devices today tend to constrain users to use their eyes primarily
as a targeting mechanism, which, as described above, is an explicit form of
input. While the industry is still early in leveraging gaze for spatial
interactions, there are multiple advantages of having systems utilize the
implicit mode of eye gaze and we discuss them here.

###  5.1 Recommendations for Pushing XR Gaze Research From Explicit to
Implicit Mode

emoving as now mostly covered in intro Targeting with eyes is a good first
step in taking advantage of what our eyes offer in XR interactions. Yet, while
this might work well relative to other current input modalities [1], our eyes
have not evolved to perform explicit targeting tasks, where we intentionally
look at and maintain our attention on the object we want to target and
manipulate. In other words, our eyes are not evolved to work like a mouse.
Having people use their eyes in a way they typically will not, naturally
requires them to be conscious of their movements, which requires what is known
as kinesthetic awareness [fitton2023dancing]; yet, attention to the motor
effort to move the eyes has not been assessed in the literature. Over time,
this could result in fatigue, both physically from straining their eyes
unnaturally, and mentally, as users exert willpower, going against how they
would naturally attend to objects to maintain focus. Furthermore, using gaze
for targeting is significantly limited by the eye-tracking hardware
capabilities. Hardware error such as accuracy and precision compounded with
user error can lead to inconsistent inaccuracies in pointing. This is not to
say that our eyes are a poor form of input and can’t be used to perform such
tasks; surely, people are adapting to use this current paradigm or seem
willing to go with it for now. However, if we were to iterate on the current
state of gaze targeting, we can design systems to be even better aligned with
how people use their eyes to interact with technology.

If we were to iterate on the current explicit state of gaze targeting, and
employ the implicit mode of gaze, we can design systems to be even better
aligned with how people use their eyes to interact with technology. We use our
eyes naturally for scanning, visual search, navigation, and joint attention,
among other tasks[29, 30, 31, 32]. In the existing platforms of human-computer
interaction such as personal and mobile computing, the eyes play an implicit
role while manual selections are executed through button presses, like on a
keyboard or through touch gestures. Analogously, in the newly emerging spatial
computing platform, given the potential limitations with using gaze
explicitly, we strongly advocate future gaze interactions to leverage existing
technologies to understand better how we can leverage natural gaze movements.
As shown earlier, not all eye movements are conscious. Our eyes tend to
explore things and are mapped to many other activities (thinking, attraction)
that heavily depend on context. Using natural gaze behavior to estimate a
user’s behavioral state and using this information to enhance user experience
will allow users to use their gaze naturally thus decreasing physical and
mental exertion.

We receive a lot of information from what people naturally do using their
eyes, which is powerful when providing them with what they want. At the same
time, these are also closely coupled with events happening contextually. As
we’ve described before, this gives insights into users’ state of mind, and
doing so enables technology to serve their needs best. While using gaze as an
explicit targeting modality is a good first step, it is essential to realize
that there is so much more that the eyes have to offer in naturalistic HCI if
we were to capitalize on the implicit tendencies of our eyes.

In the past, research was constrained to devices with eye trackers but lacked
world-facing sensors. In limited scenarios, groups collected video-based world
information combined with eye tracking, but such resources were scarce. Now,
with the advent of wearables featuring eye trackers and context-sensing
capabilities, researchers can gain deeper insights into natural gaze behavior.
Utilizing these devices for large-scale research, such as Aria program glasses
[33], allows capturing naturalistic gaze data during users’ daily activities.
These could include daily tasks such as going on a typical walk, taking a bike
ride, cooking or gardening. These open up a plethora of research opportunities
for the larger research community to act on. Research explorations could look
deeper into cognitive load, scanpaths across objects [34, 35], and much more
as users perform various tasks and do so in scale. Before applying gaze
algorithms, it’s essential to grasp human behavior in captured scenes. With
this understanding, AI advancements enable understanding expected behaviors,
considering hand and eye coordination. Segmentation of objects, environments,
and scenes provides crucial contextual information. While these are
naturalistic gaze interactions in the physical world, virtual interfaces
introduce new possibilities, such as studying gaze patterns during tasks like
cooking with a virtual UI, or riding a bike with virtual directional
instructions. This is a potential area of future research – how do people now
perform tasks and how how does gaze data and behavior change with virtual UI
placed in physical or virtual environments? Furthermore how does gaze behavior
change with more automation in people’s day to day life? As contextual apps
with gaze integration proliferate, iterative development of implicit gaze
research becomes feasible over time.

###  5.2 Challenges for Accommodating Different Modes of Interactions

Although eye tracking can be applied in both explicit and implicit modes,
integrating interactions that utilize eye gaze for these two modes presents
challenges to be addressed. For instance, implicit gaze-based models, which
rely on capturing reflexive reactionary behavior [9], might mis-classify
intentional overt movements made by users during explicit UI control. This can
lead to misinterpretations and inaccurate predictions of user intent. As eye
gaze becomes increasingly vital in the XR future, serving as a valuable data
source for AI or as an interaction modality, it becomes crucial to distinguish
between epochs when users are employing gaze implicitly or explicitly. This
classification allows for the appropriate triggering of models tailored to the
specific interaction mode.

To address this and other challenges, advancements in eye tracking technology
and machine learning algorithms are necessary. The development of
sophisticated algorithms capable of accurately discerning between explicit and
implicit gaze usage can greatly enhance the effectiveness of XR interactions.
By precisely identifying when users are engaging in explicit targeting versus
when their gaze behavior reflects implicit behavior, tailored models and
interaction approaches can be implemented accordingly. One such approach is to
learn the aligned embeddings between gaze and hand movements so that we can
use this knowledge of eye-hand coordination to classify implicit vs explicit
gaze behavior. Moreover, creating intuitive UIs that seamlessly transition
between explicit and implicit gaze-based interactions can further improve the
user experience. This includes considering an explainable AI (XAI) approach,
such as providing visual cues or feedback to inform users about the mode of
gaze interaction they are currently employing [36]. Clear indications help
users understand how their gaze behavior is being interpreted and ensure that
their intentions are accurately captured. Of course, it is also important to
investigate privacy protection opportunities, and to assess whether any
aspects of the user’s state raise concerns and require additional privacy-
protection measures.

##  6 Conclusion

In this paper, we introduced how eye tracking can be used in different ways in
XR to improve user experience. Through a comprehensive discussion on the
state-of-the-art technology for explicit and implicit modes of gaze
interaction, we encourage researchers to accelerate research leveraging
implicit gaze as a means to understand user intent and state. While explicit
gaze is a first step to interactions using eye tracking, implicit gaze allows
systems to grasp context intuitively from the user’s perspective. This
approach aligns with the natural eye usage patterns of users, empowering
systems to deliver precise responses and interfaces tailored to their
immediate context. Systems that are capable of discriminating implicit gaze
tasks gain a deeper understanding of users, marking a crucial step towards
contextual interfaces. As research systems evolve to capture gaze data with
users in various environments and contexts, they provide many unique
opportunities for research to evaluate how users naturally use gaze in their
day to day, and this will enable an iterative approach to build future
implicit interfaces.

## References

  * Fernandes et al. [2023] A. S. Fernandes, T. S. Murdison, M. J. Proulx,  Leveling the playing field: A comparative reevaluation of unmodified eye tracking as an input and interaction modality for vr,  IEEE Transactions on Visualization and Computer Graphics 29 (2023) 2269–2279.
  * Pastukhov and Braun [2010] A. Pastukhov, J. Braun,  Rare but precious: microsaccades are highly informative about attentional allocation,  Vision research 50 (2010) 1173–1184.
  * De Vries et al. [2018] S. De Vries, R. Huys, P. Zanone,  Keeping your eye on the target: eye–hand coordination in a repetitive fitts’ task,  Experimental Brain Research 236 (2018) 3181–3190.
  * Sendhilnathan et al. [2021] N. Sendhilnathan, D. Basu, M. E. Goldberg, J. D. Schall, A. Murthy,  Neural correlates of goal-directed and non–goal-directed movements,  Proceedings of the National Academy of Sciences 118 (2021) e2006372118.
  * Vine et al. [2016] S. J. Vine, L. J. Moore, M. R. Wilson,  An integrative framework of stress, attention, and visuomotor performance,  Frontiers in Psychology 7 (2016) 1671.
  * Gupta et al. [2023] A. Gupta, N. Sendhilnathan, J. Hartcher-O’Brien, E. Pezent, H. Benko, T. R. Jonker,  Investigating eyes-away mid-air typing in virtual reality using squeeze haptics-based postural reinforcement,  in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023, pp. 1–11.
  * Gidlöf et al. [2013] K. Gidlöf, A. Wallin, R. Dewhurst, K. Holmqvist,  Using eye tracking to trace a cognitive process: Gaze behaviour during decision making in a natural environment,  Journal of eye movement research 6 (2013).
  * Stone and Chapman [2023] S. A. Stone, C. S. Chapman,  Unconscious frustration: Dynamically assessing user experience using eye and mouse tracking,  Proceedings of the ACM on Human-Computer Interaction 7 (2023) 1–17.
  * Sendhilnathan et al. [2022] N. Sendhilnathan, T. Zhang, B. Lafreniere, T. Grossman, T. R. Jonker,  Detecting input recognition errors and user errors using gaze dynamics in virtual reality,  in: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, 2022, pp. 1–19.
  * Plopski et al. [2022] A. Plopski, T. Hirzle, N. Norouzi, L. Qian, G. Bruder, T. Langlotz,  The eye in extended reality: A survey on gaze interaction and eye tracking in head-worn extended reality,  ACM Computing Surveys (CSUR) 55 (2022) 1–39.
  * Rajanna and Hansen [2018] V. Rajanna, J. P. Hansen,  Gaze typing in virtual reality: impact of keyboard design, selection method, and motion,  in: Proceedings of the 2018 ACM symposium on eye tracking research & applications, 2018, pp. 1–10.
  * Khamis et al. [2018] M. Khamis, C. Oechsner, F. Alt, A. Bulling,  Vrpursuits: Interaction in virtual reality using smooth pursuit eye movements,  in: Proceedings of the 2018 international conference on advanced visual interfaces, 2018, pp. 1–8.
  * Pfeuffer et al. [2015] K. Pfeuffer, J. Alexander, M. K. Chong, Y. Zhang, H. Gellersen,  Gaze-shifting: Direct-indirect input with pen and touch modulated by gaze,  in: Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, 2015, pp. 373–383.
  * Feit et al. [2017] A. M. Feit, S. Williams, A. Toledo, A. Paradiso, H. Kulkarni, S. Kane, M. R. Morris,  Toward everyday gaze input: Accuracy and precision of eye tracking and implications for design,  in: Proceedings of the 2017 Chi conference on human factors in computing systems, 2017, pp. 1118–1130.
  * David-John et al. [2021] B. David-John, C. Peacock, T. Zhang, T. S. Murdison, H. Benko, T. R. Jonker,  Towards gaze-based prediction of the intent to interact in virtual reality,  in: ACM Symposium on Eye Tracking Research and Applications, 2021, pp. 1–7.
  * Zhao et al. [2023] M. Zhao, A. M. Pierce, R. Tan, T. Zhang, T. Wang, T. R. Jonker, H. Benko, A. Gupta,  Gaze speedup: Eye gaze assisted gesture typing in virtual reality,  in: Proceedings of the 28th International Conference on Intelligent User Interfaces, 2023, pp. 595–606.
  * Peacock et al. [2022] C. E. Peacock, B. Lafreniere, T. Zhang, S. Santosa, H. Benko, T. R. Jonker,  Gaze as an indicator of input recognition errors,  Proceedings of the ACM on Human-Computer Interaction 6 (2022) 1–18.
  * Frischen et al. [2007] A. Frischen, A. P. Bayliss, S. P. Tipper,  Gaze cueing of attention: visual attention, social cognition, and individual differences.,  Psychological bulletin 133 (2007) 694.
  * Reppert et al. [2015] T. R. Reppert, K. M. Lempert, P. W. Glimcher, R. Shadmehr,  Modulation of saccade vigor during value-based decision making,  Journal of Neuroscience 35 (2015) 15369–15378.
  * Duchowski et al. [2018] A. T. Duchowski, K. Krejtz, I. Krejtz, C. Biele, A. Niedzielska, P. Kiefer, M. Raubal, I. Giannopoulos,  The index of pupillary activity: Measuring cognitive load vis-à-vis task difficulty with pupil oscillation,  in: Proceedings of the 2018 CHI conference on human factors in computing systems, 2018, pp. 1–13.
  * Lindlbauer et al. [2019] D. Lindlbauer, A. M. Feit, O. Hilliges,  Context-aware online adaptation of mixed reality interfaces,  in: Proceedings of the 32nd annual ACM symposium on user interface software and technology, 2019, pp. 147–160.
  * Chew et al. [2017] J. Y. Chew, K. Ohtomi, H. Suzuki,  Skill metrics for mobile crane operators based on gaze fixation pattern,  in: Advances in Human Aspects of Transportation: Proceedings of the AHFE 2016 International Conference on Human Factors in Transportation, July 27-31, 2016, Walt Disney World®, Florida, USA, Springer, 2017, pp. 1139–1149.
  * Wang et al. [2022] F. S. Wang, C. Gianduzzo, M. Meboldt, Q. Lohmeyer,  An algorithmic approach to determine expertise development using object-related gaze pattern sequences,  Behavior Research Methods 54 (2022) 493–507.
  * Diaz-Piedra et al. [2017] C. Diaz-Piedra, J. M. Sanchez-Carrion, H. Rieiro, L. L. Di Stasi,  Gaze-based technology as a tool for surgical skills assessment and training in urology,  Urology 107 (2017) 26–30.
  * Mahanama et al. [2022] B. Mahanama, Y. Jayawardana, S. Rengarajan, G. Jayawardena, L. Chukoskie, J. Snider, S. Jayarathna,  Eye movement and pupil measures: A review,  frontiers in Computer Science 3 (2022) 733531.
  * Hadnett-Hunter et al. [2019] J. Hadnett-Hunter, G. Nicolaou, E. O’Neill, M. Proulx,  The effect of task on visual attention in interactive virtual environments,  ACM Transactions on Applied Perception (TAP) 16 (2019) 1–17.
  * Tönnis and Klinker [2014a] M. Tönnis, G. Klinker,  Boundary conditions for information visualization with respect to the user’s gaze,  in: Proceedings of the 5th Augmented Human International Conference, 2014a, pp. 1–8.
  * Tönnis and Klinker [2014b] M. Tönnis, G. Klinker,  Placing information near to the gaze of the user,  in: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, 2014b, pp. 377–378.
  * Trepkowski et al. [2019] C. Trepkowski, D. Eibich, J. Maiero, A. Marquardt, E. Kruijff, S. Feiner,  The effect of narrow field of view and information density on visual search performance in augmented reality,  in: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), IEEE, 2019, pp. 575–584.
  * Foulsham et al. [2011] T. Foulsham, E. Walker, A. Kingstone,  The where, what and when of gaze allocation in the lab and the natural environment,  Vision research 51 (2011) 1920–1931.
  * Olk et al. [2018] B. Olk, A. Dinu, D. J. Zielinski, R. Kopper,  Measuring visual search and distraction in immersive virtual reality,  Royal Society open science 5 (2018) 172331.
  * Kredel et al. [2017] R. Kredel, C. Vater, A. Klostermann, E.-J. Hossner,  Eye-tracking technology and the dynamics of natural gaze behavior in sports: A systematic review of 40 years of research,  Frontiers in psychology 8 (2017) 1845.
  * Somasundaram et al. [2023] K. Somasundaram, J. Dong, H. Tang, J. Straub, M. Yan, M. Goesele, J. J. Engel, R. De Nardi, R. Newcombe,  Project aria: A new tool for egocentric multi-modal ai research,  arXiv preprint arXiv:2308.13561 (2023).
  * Burlingham et al. [2024] C. S. Burlingham, N. Sendhilnathan, O. Komogortsev, T. S. Murdison, M. J. Proulx,  Motor “laziness” constrains fixation selection in real-world tasks,  Proceedings of the National Academy of Sciences 121 (2024) e2302239121.
  * Basu et al. [2021] D. Basu, N. Sendhilnathan, A. Murthy,  Neural mechanisms underlying the temporal control of sequential saccade planning in the frontal eye field,  Proceedings of the National Academy of Sciences 118 (2021) e2108922118.
  * Xu et al. [2023] X. Xu, A. Yu, T. R. Jonker, K. Todi, F. Lu, X. Qian, J. M. Evangelista Belo, T. Wang, M. Li, A. Mun, et al.,  Xair: A framework of explainable ai in augmented reality,  in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023, pp. 1–30.


Sh2}(y_{t+1}|x_{t+1};\mathbb{P}^{(t)}_{\Theta_{t+1}|y,x_{t+1}}\big{)}\\\
&-\mathcal{S}_{\rm Sh1}(y_{t+1}|x_{t+1};\pi^{(t)}),\end{aligned}$ (85)

which, using Equation 80, makes the proof complete.

To prove the 2nd equality, we note that (c.f. Equation 35)

$\ \begin{aligned} \mathcal{S}_{\rm Po2}&(y_{t+1}|x_{t+1};\pi^{(t)})\\\
&=D_{\rm
KL}\Big{[}P\big{(}.|x_{t+1};\pi^{(t)}\big{)}||P\big{(}.|x_{t+1};\pi^{(t+1)}\big{)}\Big{]}\\\
&=\mathbb{E}_{P\big{(}.|x_{t+1};\pi^{(t)}\big{)}}\Big{[}\log\frac{P\big{(}Y|x_{t+1};\pi^{(t)}\big{)}}{P\big{(}Y|x_{t+1};\pi^{(t+1)}\big{)}}\Big{]},\end{aligned}$
(86)

and

$\ \begin{aligned}
&\log\frac{P\big{(}y|x_{t+1};\pi^{(t)}\big{)}}{P\big{(}y|x_{t+1};\pi^{(t+1)}\big{)}}=\\\
&\log\frac{P\big{(}y|x_{t+1};\mathbb{P}^{(t)}_{\Theta_{t+1}}\big{)}}{P\big{(}y|x_{t+1};\pi^{(t+1)}\big{)}}+\log\frac{P\big{(}y|x_{t+1};\pi^{(t)}\big{)}}{P\big{(}y|x_{t+1};\mathbb{P}^{(t)}_{\Theta_{t+1}}\big{)}}.\end{aligned}$
(87)

Therefore, using Equation 85 and the definition of $D_{\rm KL}$, the proof is
complete. $\hfill\blacksquare$


# DCT-Based Decorrelated Attention
for Vision Transformers

Hongyi Pan∗†, Emadeldeen Hamdan‡, Xin Zhu‡, Koushik Biswas†, Ahmet Enis
Cetin‡, Ulas Bagci†
†Machine and Hybrid Intelligence Lab, Northwestern University, Chicago, IL
60611
‡Department of Electrical and Computer Engineering, University of Illinois
Chicago Chicago, IL 60607
<EMAIL_ADDRESS>

###### Abstract

Central to the Transformer architectures’ effectiveness is the self-attention
mechanism, a function that maps queries, keys, and values into a high-
dimensional vector space. However, training the attention weights of queries,
keys, and values is non-trivial from a state of random initialization. In this
paper, we propose two methods. (i) We first address the initialization problem
of Vision Transformers by introducing a simple, yet highly innovative,
initialization approach utilizing Discrete Cosine Transform (DCT)
coefficients. Our proposed DCT-based attention initialization marks a
significant gain compared to traditional initialization strategies; offering a
robust foundation for the attention mechanism. Our experiments reveal that the
DCT-based initialization enhances the accuracy of Vision Transformers in
classification tasks. (ii) We also recognize that since DCT effectively
decorrelates image information in the frequency domain, this decorrelation is
useful for compression because it allows the quantization step to discard many
of the higher-frequency components. Based on this observation, we propose a
novel DCT-based compression technique for the attention function of Vision
Transformers. Since high-frequency DCT coefficients usually correspond to
noise, we truncate the high-frequency DCT components of the input patches. Our
DCT-based compression reduces the size of weight matrices for queries, keys,
and values. While maintaining the same level of accuracy, our DCT compressed
Swin Transformers obtain a considerable decrease in the computational
overhead.

##  1 Introduction

In the last decade, convolutional neural networks (CNNs) have dominated image-
processing tasks. However, inspired by the success of transformer
architectures in natural language processing, Vision Transformers (ViTs) in
recent years have redefined the norms of visual data interpretation. Self-
attention mechanism, a key component of transformers, is pivotal in reshaping
how models perceive and analyze visual information [49]. Unlike CNNs, ViTs
view an image as a sequence of informative patches, adopting a more holistic
approach. By treating image patches like “visual words”, ViTs can leverage
powerful transformer architectures, originally designed for natural language
processing, to learn global context and long-range dependencies. This novel
approach has led to remarkable performance gains on a wide range of computer
vision tasks, demonstrating the potential of ViTs to revolutionize the field.

The initialization of self-attention weights is a critical step significantly
impacting the training dynamics and performance of the Transformers. The core
of Transformers is the self-attention mechanism, which includes three weight
matrices: queries ($\mathbf{W}_{Q}$), keys ($\mathbf{W}_{K}$), and values
($\mathbf{W}_{V}$). Proper initialization of these matrices ensures that the
model starts from a stable state, facilitating effective learning. The most
common approach is to initialize the weights randomly, often using
distributions like the normal or uniform distribution. For example, Xavier
initialization [16] and He initialization [18] are popular choices. They are
designed to keep the scale of the gradients roughly the same in all layers.
Although less common, in some cases, biases or certain parts of the weight
matrices are initialized to zeros [32, 60]. This is less common for queries,
keys, and values matrices, as it can lead to symmetry-breaking issues during
training [20].

Efficiency is an ongoing concern in Transformers. While conventional
initialization strategies are often suitable for generic applications, new
algorithms are increasingly tailored to the specific characteristics and needs
of Transformers. These advancements in initialization techniques contribute to
the ongoing improvements in the performance and efficiency of Transformer-
based models. Orthogonal initialization [40, 21], sparse initialization [31],
variance scaling, pre-trained initialization [51, 4], layer-wise, domain-
specific and attention-specific initialization [48] are some of the new trends
in initialization approaches. In this study, we propose a new family of
algorithms for initialization strategy in the intersection of attention and
domain-specific approaches: discrete cosine attention.

Can DCT in Vision Transformer be a game changer? Discrete cosine transform
(DCT) has been widely used in signal processing and data compression
applications. It reduces the dimensionality of the input data while preserving
its essential features. Transforming the input data into the frequency domain
helps capture the most important features of the input data. In this paper, we
propose two methods that apply DCT to the attention for vision transformers.
First, we initialize the weight matrices for the queries, keys, and values
computation as the DCT matrix. Our DCT-based initialization covers the entire
spectrum utilizing the DCT basis vectors, and each weight vector has a
different initial bandwidth. As a comparison, to cover the entire spectrum in
the classical initialization, all the weight matrices are initialized as
random numbers due to their independent generation. These random numbers can
be treated as white noise. All the weight initialization starts with the same
spectra. Second, we introduce a DCT-based compressed attention for vision
transformers. With this compression methodology, compared to the vanilla Swin
transformers, the number of parameters and computational overhead of the Swin
Transformers are reduced. Meanwhile, the accuracy of the models is comparable.

##  2 Related works

Vision Transformers Transformer architectures have merged as a dominant
standard for natural language processing (NLP) tasks [49, 53]. Vision
Transformer (ViT) [14] built a bridge between language and vision. It shows
that pure transformers can also be applied directly to sequences of image
patches for classification tasks. In other words, it partitions images into
patches and treats them as word sequences. Then, a shifted windowing scheme
was proposed in Swin Transformer [29] to obtain greater efficiency. This
limits self-attention computation to non-overlapping local windows while
allowing for cross-window connection. Later, to overcome the limitations of
the heavyweights in ViTs, MobileViT [33] combined the strengths of MobileNets
[39] and ViTs. It is a lightweight and low-latency network for mobile vision
tasks. Moreover, gSwin [17] combined MLP-based architecture with Swin
Transformer and attains superior accuracy with a smaller model size. Vision
transformers are now supreme in various computer vision tasks such as super
resolution [27, 57], compressive sensing [43, 54], image segmentation [59, 23,
47], and human mesh reconstruction [62].

Frequency-based Neural Networks Orthogonal transforms such as Fourier
transform [10, 5], Hadamard transform [34, 36], and discrete cosine Stockwell
transform [64] have been used to improve the performance of the neural
networks. In the orthogonal transform family, DCT is a mathematical technique
commonly used in signal processing and image and video compression [35].
Compared to the Fourier transform, which is the standard frequency
representation, DCT does not generate complex-valued numbers. Therefore, it is
computationally more efficient. DCT-Former [41], for instance, applied DCT
compression on the input sequence along the length channel before feeding into
the self-attention layers to reduce the computational overhead in the NLP
problems. Furthermore, inspired by JPEG compression, DCFormer [26] highlighted
the visually significant signals by leveraging the input information
compression on its frequency domain representation. These two DCT-based works
and our DCT-based compression all aim to reduce the computational overhead.
Therefore, we mainly compare our DCT-based compression method with them. DCT-
Former [41] is designed for the NLP transformers and takes DCT along the width
and height of the input tensor patches, while our method is designed for the
vision transformers and it compresses the tensor along the channel axis.
DCFormer [26] averages DCT coefficients via average pooling, which does not
take advantage of the property that DCT components in low-frequency contain
more important information. On the contrary, our DCT-based compression method
retains the low-frequency components very accurately and discards the
unnecessary extremely high-frequency components corresponding to noise. Under
the same input image size, compared to the vanilla transformer models,
DCFormer reduces the FLOPs and retains the number of parameters while
sacrificing accuracy. Our DCT-based compression method, on the other hand,
saves both trainable parameters and FLOPs and obtains on-par or better
performance in addition to other benefits that our DCT-based compression
method has. Furthermore, we introduce a DCT-based initialization and we have
not found any related initialization in the literature.

Initialization Methods for Self-Attention Mimetic initialization [48] is a
compute-free method to initialize the weights of self-attention layers. It
simulated the patterns present in the pre-trained weights to achieve
performance improvement of the transformers.

Efficient Multi-head Self-Attention To reduce the memory and computational
costs of dot-product attention, an efficient mechanism [42] in linear
complexity was proposed. Besides, Hydra Attention [3] increased the number of
heads to address the issue of the high computational complexity of the self-
attention mechanism on large-sized images. Further, MetaFormer [55] verified
that the overall architecture of Transformers, rather than the specific token
mixer module, plays a more crucial role in the performance.

##  3 Methodology

###  3.1 Background: Multi-Head Self-Attention

The overall structure of the vanilla MSA is drawn in Figure 1a. To handle a 2D
image tensor $\mathbf{I}\in\mathbb{R}^{H\times\ W\times C}$, where $H\times W$
are the resolution of the image and $C$ is the number of channels, we first
split it into patches $\mathbf{X}\in\mathbb{R}^{N\times M^{2}\times C}$, where
$M\times M$ is the resolution of each image patch and $N=\frac{WH}{M^{2}}$ is
the resulting number of patches. After that, three linear layers
($\mathcal{L}_{Q}$, $\mathcal{L}_{K}$, $\mathcal{L}_{V}$) are applied on
$\mathbf{X}$ to compute the queries, keys, and values
$\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{R}^{N\times M^{2}\times C}$:

$\mathbf{Q}=\mathbf{XW}_{Q}^{T}+\mathbf{b}_{Q},\ \
\mathbf{K}=\mathbf{XW}_{K}^{T}+\mathbf{b}_{K},\ \
\mathbf{V}=\mathbf{XW}_{V}^{T}+\mathbf{b}_{V},$ (1)

where $\mathbf{W}_{Q},\mathbf{W}_{K},\mathbf{W}_{V}\in\mathbb{R}^{C\times C}$
and $\mathbf{b}_{Q},\mathbf{b}_{K},\mathbf{b}_{V}\in\mathbb{R}^{1\times C}$
are trainable weights and biases. Next, to perform multi-head attention with
$P$ heads, $\mathbf{Q,K,V}$ are split into $P$ slices as
$\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i}\in\mathbb{R}^{N\times
M^{2}\times d}$ for $i=0,\ldots,P-1$, where $d=\frac{C}{P}$. The scaled dot-
product attention $\mathbf{A}_{i}\in\mathbb{R}^{N\times M^{2}\times d}$ is
computed as:

$\mathbf{A}_{i}=\text{Attention}(\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i})=\sigma\left(\frac{\mathbf{Q}_{i}\mathbf{K}_{i}^{T}}{\sqrt{d}}+\mathbf{B}_{i}\right)\mathbf{V}_{i},$
(2)

where $\sigma(\cdot)$ is the Softmax function.
$\mathbf{B}_{i}\in\mathbb{R}^{M^{2}\times M^{2}}$ is the relative position
bias. Since the relative position along each axis lies in $[-M+1,M-1]$, values
in $B_{i}$ are taken from a parameterized smaller-sized bias matrix
$\hat{\mathbf{B}}_{i}\in\mathbb{R}^{(2M-1)\times(2M-1)}$. After that,
$\\{\mathbf{A}_{i}\\}$ are concatenated to $\mathbf{A}\in\mathbb{R}^{N\times
M^{2}\times C}$ to feed into a linear layer $\mathcal{L}_{O}$ to obtain the
attention output $\mathbf{Y}\in\mathbb{R}^{N\times M^{2}\times C}$:

$\mathbf{Y}=\mathbf{AW}_{O}^{T}+\mathbf{b}_{O}.$ (3)

Finally, $\mathbf{Y}$ is reversed into the image shape
$\mathbf{J}\in\mathbb{R}^{H\times W\times C}$.

(a) Vanilla / DCT-initialized MSA. (b) DCT-Compressed MSA.

Figure 1: Multi-head Self-Attention. In (b), to reduce the computational
overhead, IDCT and the final linear layer $\mathcal{L}_{O}$ are implemented as
a single linear layer as no non-linearity exists in between.

###  3.2 DCT-based Initialization for Attention

In the attention layer, the first three linear layers ($\mathcal{L}_{Q}$,
$\mathcal{L}_{K}$, and $\mathcal{L}_{V}$) extract different features of the
input patches $\mathbf{X}$. However, their weights $\mathbf{W}_{Q}$,
$\mathbf{W}_{K}$, and $\mathbf{W}_{V}$ are trained simultaneously. Such
training from scratch is difficult, especially when the dataset is
insufficient and small-scale. On the other hand, DCT is a well-designed
feature extractor. In our proposed DCT initialization strategy, one of
$\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, $\mathbf{W}_{V}\in\mathbb{R}^{C\times C}$
is initialized using the orthogonal type-II DCT matrix $\mathcal{D}$ [1]:

$\\{\mathbf{X}\mathcal{D}^{T}\\}[k]=\begin{cases}\frac{\sqrt{2}}{C}\sum_{n=0}^{C-1}\mathbf{X}[n],&k=0,\\\
\frac{2}{C}\sum_{n=0}^{C-1}\mathbf{X}[n]\cos\frac{(2n+1)k\pi}{2nC},&k=1,\ldots,C-1.\end{cases}$
(4)

$\mathcal{D}$ can be obtained from applying DCT on an identity matrix
$\mathbf{I}_{C\times C}$ because DCT is a linear transform. The remaining
weight matrices and other trainable parameters are still initialized
regularly. We don’t initialize multiple weight matrices to avoid the weight
symmetry problem [20] caused by initializing the multiple weights as the same
values, and our ablation study has verified this point.

The DCT-based initialization can be treated as applying DCT instead of linear
projection on X to obtain $\mathbf{Q}$, $\mathbf{K}$, or $\mathbf{V}$. Some
DCT coefficients capture low-frequency information, while others capture high-
frequency details. Therefore, the DCT matrix is a good starting point for the
weight matrices.

###  3.3 DCT-Based Compressed Attention

Data compression using DCT is common in signal, image, and video processing
for energy concentration, compression efficiency, and noise reduction. DCT not
only decorrelates the input image data similar to the Karhunen-Loeve Transform
(KLT) or principal component analysis (PCA) but also has a frequency
interpretation because it can be computed using the discrete Fourier transform
(DFT) [1]:

$\\{\mathbf{X}\mathcal{D}^{T}\\}[k]=\begin{cases}\frac{\sqrt{2}}{C}\sum_{n=0}^{C-1}\mathbf{X}[n],&k=0,\\\
\frac{2}{C}\text{Re}\left(e^{-\frac{ik\pi}{2C}}\sum_{n=0}^{2C-1}\mathbf{X}[n]W^{kn}\right),&k=1,\ldots,C-1,\end{cases}$
(5)

where $W=e^{-\frac{i2\pi}{2C}}$, $i^{2}=-1$, and $\text{Re}(\cdot)$ implies
the real part of the term enclosed. Usually, higher-frequency DCT coefficients
contribute less to perceptual quality and contain more noise. Therefore,
preserving essential information in lower frequencies while discarding less
critical details in higher frequencies results in efficient signal
representation and storage. The specific truncation strategy depends on the
application’s requirements and the acceptable level of information loss.

We observed that the attention layer matrices are correlated with each other
so that we can compress them using DCT. Figure 1b shows the method of DCT-
based compression for the attention layer. First, the input patches
$\mathbf{X}\in\mathbb{R}^{N\times M^{2}\times C}$ are encoded by DCT along the
channel of $C$ with the high-frequency components truncated. If we keep the
first $\tau C$ coefficients ($\tau<1$), the essential information of
$\mathbf{X}$ is still preserved, while the input and output dimensions of the
linear layers ($\mathcal{L}_{Q},\mathcal{L}_{K},$ and $\mathcal{L}_{V}$) are
reduced from $\mathbb{R}^{N\times M^{2}\times C}$ to $\mathbb{R}^{N\times
M^{2}\times\tau C}$. In this way, the computational cost from
$\mathcal{L}_{Q},\mathcal{L}_{K}$, and $\mathcal{L}_{V}$ is reduced
significantly. In our experiments, $\tau$ is chosen from 25%, 50%, and 75%.
After obtaining the attention $\mathbf{A}\in\mathbb{R}^{N\times
M^{2}\times\tau C}$ from the queries $\mathbf{Q}$, keys $\mathbf{K}$, and
values $\mathbf{V}$, we apply zero-padding along the last channel to make the
dimension back to $\mathbb{R}^{N\times M^{2}\times C}$. Before being fed into
the linear layer $\mathcal{L}_{O}$, these padded patches are decoded by the
inverse DCT (IDCT):

$\\{\mathbf{A}(\mathcal{D}^{-1})^{T}\\}[n]=\frac{\sqrt{2}}{2}\mathbf{A}[0]+\sum_{k=1}^{C-1}\mathbf{A}[k]\cos\frac{(2n+1)k\pi}{2C},n=0,1,\ldots,C-1.$
(6)

DCT and IDCT can be implemented in a fast manner with the complexity of
$O(C\log_{2}C)$ using butterfly algorithm [9]. However, such an implementation
is not officially supported in PyTorch currently. Therefore, in this work, we
implement the DCT and IDCT via matrix multiplication between the input tensor
and the truncated DCT and IDCT matrices $\bar{\mathcal{D}}\in\mathbb{R}^{\tau
C\times C}$ and $\bar{\mathcal{D}}^{-1}\in\mathbb{R}^{C\times\tau C}$. These
transform matrices are obtained by truncating the DCT and IDCT matrices along
rows or columns to keep $\tau$ coefficients or zero padding, respectively.
Then, DCT with truncation or IDCT with zero padding is implemented as one
matrix multiplication to reduce memory pressure. Furthermore, as no non-
linearity exists between IDCT and the final linear layer $\mathcal{L}_{O}$,
these two steps can be implemented as a single linear layer whose weight
matrix is $\mathbf{W}_{O}\bar{\mathcal{D}}^{-1}$ and the bias is still
$\mathbf{b}_{O}$. However, such a combination is not applied on DCT and
$\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, or $\mathbf{W}_{V}$. This is because it
may not always reduce the computational cost for them: To compute
$\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$, with such a combination,
$3kNM^{2}C^{3}$ multiplications are needed. In contrast, if they are computed
separately, there are $\tau NM^{2}C^{3}$ multiplications from DCT and
$\tau^{2}NM^{2}C^{3}$ multiplications from each linear layer. As a result,
there are a total of $(\tau+3\tau^{2})NM^{2}C^{3}$ multiplications. Hence,
combing DCT with $\mathbf{W}_{Q}$, $\mathbf{W}_{K}$ and $\mathbf{W}_{V}$
brings computational cost saving only when $\tau>\frac{2}{3}$. To maintain the
consistency of the framework, we do not combine DCT with $\mathbf{W}_{Q}$,
$\mathbf{W}_{K}$, or $\mathbf{W}_{V}$.

Computational cost comparison of the vanilla versus the proposed DCT-
compressed attentions is presented in Table 1. We evaluate both naive and
simplified implementations of the DCT-compressed attention with the vanilla
attention. The simplification is implemented by fusing IDCT into the linear
layer $\mathcal{L}_{O}$. Computational overhead from the Softmax function is
omitted because the Softmax function is based on exponential terms. Despite
this, the Softmax function in the DCT-compressed attention requires less
computational cost than in the vanilla attention as the number of entries in
$\left(\frac{\mathbf{QK}^{T}}{\sqrt{d}}+\mathbf{B}\right)$ in the DCT-
compressed attention is $\tau^{2}$ times ($\tau<1$) as in the vanilla
attention.

Table 1: Multiplications in MSA on $\mathbf{X}\in\mathbb{R}^{N\times M^{2}\times C}$. “$-$” means this step is not required. $\tau<1$.  Operation | Vanilla MSA | DCT-Compressed MSA
---|---|---
Naive | Simplified
DCT | - | $\tau NM^{2}C^{3}$ | $\tau NM^{2}C^{3}$
$\mathcal{L}_{Q}$, $\mathcal{L}_{K}$, $\mathcal{L}_{V}$ | $3NM^{2}C^{3}$ | $3\tau^{2}NM^{2}C^{3}$ | $3\tau^{2}NM^{2}C^{3}$
$\mathbf{QK}^{T}/\sqrt{d}+\mathbf{B}$ | $NM^{4}C^{2}$ | $\tau^{2}NM^{4}C^{2}$ | $\tau^{2}NM^{4}C^{2}$
Multiply with $\mathbf{V}$ | $NM^{4}C^{2}$ | $\tau^{2}NM^{4}C^{2}$ | $\tau^{2}NM^{4}C^{2}$
IDCT | - | $\tau NM^{2}C^{3}$ | -
$\mathcal{L}_{O}$ | $NM^{2}C^{3}$ | $NM^{2}C^{3}$ | $\tau NM^{2}C^{3}$
Total | $4NM^{2}C^{3}$ | $(1+2\tau+3\tau^{2})NM^{2}C^{3}$ | $(2\tau+3\tau^{2})NM^{2}C^{3}$
$+2NM^{4}C^{2}$ | $+2\tau^{2}NM^{4}C^{2}$ | $+2\tau^{2}NM^{4}C^{2}$

###  3.4 Frequency Domain Motivation of the DCT-based Initialization

In traditional random initialization, all the weight matrices are initialized
as random numbers that are independently generated. As a result, the initial
matrices can be considered as a realization of white noise covering the entire
spectrum. A good initialization should take advantage of the full bandwidth to
cover the entire spectrum of the input.

The DCT-based initialization also covers the entire frequency domain. The set
of DCT basis vectors $\\{\frac{\sqrt{2}}{2},\cos\frac{(2n+1)k\pi}{2C}\\}$ is
the class of discrete Chebyshev polynomials:

$T_{k}[n]=\begin{cases}\frac{\sqrt{2}}{2},&k=0,\\\
\cos\frac{(2n+1)k\pi}{2C},&k=1,\ldots,C-1,\end{cases}$ (7)

for $n=0,1,\ldots,C-1$. The basis member $\cos\frac{(2n+1)k\pi}{2nC}$ is the
$k$-th Chebyshev polynomial $T_{k}[\xi]$ evaluated at the $n$-th zero of
$T_{n}[\xi]$ [1]. Figure 2 draws the basis vectors in Eq. (7) for an $8\times
8$ DCT matrix and the magnitude of their discrete Fourier transform
($\mathcal{F}\\{\cdot\\}$):

$\hat{T_{l}}[k]=\mathcal{F}\\{T_{l}[n]\\},l=0,1,\ldots,C-1.$ (8)

Therefore, the entire spectrum is covered as
$\sqrt{\sum_{l=0}^{C-1}|\hat{T_{l}}[k]|^{2}}$ has perfect full-band coverage,
and each weight vector in the DCT-based initialization covers a different
bandwidth. On the other hand, in the traditional random initialization, all of
the weights start with the same spectra. This is why our DCT-based
initialization is superior to the traditional random initialization for the
attention layer.

(a) $D_{0}$. (b) $D_{1}$.

(c) $D_{2}$. (d) $D_{3}$.

(e) $D_{4}$. (f) $D_{5}$.

(g) $D_{6}$. (h) $D_{7}$.

(i) Frequency response magnitudes. Frequencies are normalized into $[0,\pi]$.

Figure 2: Basis vectors of an $8\times 8$ DCT matrix and their frequency
response magnitudes. $D_{i}$ refers to the $i$-th column. The basis vectors
provide a good approximation to the eigenvectors of the Teoplitz matrix with
$\rho=0.9$ (KLT). Each frequency response is shown in a solid curve in
different colors, and their squared sum is denoted in the black dashed line,
showing the entire spectrum is covered.

###  3.5 Motivation of the DCT-based Compression

The mechanism behind the DCT-based compression mimics the Karhunen-Loeve
transform (KLT), which is also known as principal component analysis (PCA), a
mathematical technique used for dimensionality reduction in signal processing
[24, 61]. Neighboring image patches are highly correlated with each other.
During training, the attention matrices are inter-correlated because they are
obtained using similar gradient vectors. Therefore, they can be decorrelated
and compressed using DCT. When the input data is highly correlated, the
covariance matrix can be approximated by a Teoplitz matrix $\phi$ of
$[1,\rho,\rho^{2},\cdots,\rho^{C-1}]$ with $\rho<1$ close to 1 [2]. This is
the covariance matrix of an autoregressive model random process AR(1) with the
correlation coefficient $\rho$ [22, 46]. Figure 2 shows that DCT basis vectors
approximate the KLT of a first-order Markov process with high correlation
among the neighboring samples [13, 38]. We observe that $\mathbf{Q}$,
$\mathbf{K}$, and $\mathbf{V}$ have parameters with high correlations among
themselves. We can directly use DCT in data compression since its matrix
approximates the corresponding KLT matrix [1],

Although KLT can also be utilized in data compression, it requires massive
computation for the covariance matrices and the corresponding eigenvectors of
$\mathbf{X}$. On the other hand, DCT is very successful in decorrelating the
data without computing the covariance matrix and eigenvectors. As a
consequence, it is also used in JPEG image [50, 52] and MPEG video [15]
compression methods.

##  4 Experimental Results

We conduct experiments on the CIFAR-10 [25] and ImageNet-1K [12]
classification tasks and COCO 2017 detection and segmentation tasks [28]. Swin
Transformers [29] are employed as the backbone network. Implementation details
are presented in the technical appendices.

###  4.1 CIFAR-10 Classification

Training vision transformers from scratch on the CIFAR-10 dataset is
notoriously difficult. We demonstrate the benefits of using our DCT
initialization and DCT compression for the Swin Transformer Tiny (Swin-T) on
from-scratch CIFAR-10 training to show the effectiveness of our methods on the
small-scale dataset. Table 2 presents the performance comparison on the
CIFAR-10 test dataset, where the number of parameters only counts trainable
parameters.

Table 2: CIFAR-10 test dataset evaluation. In column “DCT", Q/K/V denotes the weights initialized as the DCT matrix, and numbers mean the proportion of coefficients kept in DCT-compressed attention. Models with higher or lower accuracy are highlighted in pink or cyan, respectively. Model | DCT | Params (M) | Top-1 Acc.
---|---|---|---
Swin-T | - | 27.53 | 90.51%
Swin-T | Q | 27.53 | 90.78%
Swin-T | K | 27.53 | 90.83%
Swin-T | V | 27.53 | 90.81%
Swin-T | 0.25 | 21.45 | 90.46%
Swin-T | 0.5 | 22.67 | 90.67%
Swin-T | 0.75 | 24.69 | 90.90%

We start by training Swin-T with the vanilla attention function for the
baseline and obtain 90.51% accuracy on the CIFAR-10 test dataset. Then, we
investigate the DCT-initialized attention in Swin-T on the CIFAR datasets. The
accuracy of Swin-T on the CIFAR-10 dataset is always improved regardless of
which attention weight is initialized as the DCT matrix. When the weights to
compute keys ($\mathbf{K}$) are initialized as the DCT matrices, the accuracy
is improved to 90.83%.

Next, we evaluate the DCT-compressed attention with different truncating rates
in Swin-T on the CIFAR-10 dataset. Table 2 presents when we only keep 25% of
DCT coefficients, the number of trainable parameters is reduced from 27.53M to
21.45M (22.1% off), while the accuracy only drops 0.05%. If we keep more DCT
coefficients, the accuracy even improves. If we keep 75% of DCT coefficients,
the accuracy increases from 90.51% to 90.90%. This is because although the
number of trainable parameters is reduced, transforming the input data into
the frequency domain using DCT can boost the extraction of important features.
As a result, fewer trainable parameters are required.

###  4.2 ImageNet-1K Classification

Table 3 presents the performance comparison on the ImageNet-1K dataset, where
the number of parameters only counts trainable parameters. First, we
investigate the DCT-initialized attention in the Swin-T and the Swin-S on the
ImageNet-1K dataset. The accuracy on the ImageNet-1K validation dataset is
always improved nevertheless which weight matrix is initialized as the DCT
matrix. For example, when we initialize the weights to compute keys
($\mathbf{K}$) as the DCT weights, the accuracy of Swin-T from 81.474% to
81.728% and Swin-S are improved from 83.196% to 83.378%, respectively.

Table 3: ImageNet-1K validation dataset evaluation. After DCT, Q/K/V denotes the weights initialized as the DCT matrix, and numbers mean the proportion of coefficients kept in DCT-compressed attention. Our models with higher or lower top-1 accuracy are highlighted in pink or cyan, respectively.  Method | Params (M) | Flops (G) | Top-1 Acc. | Top-5 Acc.
---|---|---|---|---
Swin-T (Torchvision) [29] | 28.29 | 4.49 | 81.474% | 95.776%
PoolFormer-S24 [55] | 21 | 3.5 | 80.3% | -
DCFormer-SW-T ($\tau=1)$ [26] | 28.29 | 4.5 | 81.2% | -
DCFormer-SW-T ($\tau=2)$ [26] | 28.29 | 1.3 | 79.2% | -
gSwin-T [17] | 22 | 3.6 | 81.715% | -
Swin-T-DCT-Q | 28.29 | 4.49 | 81.568% | 95.862%
Swin-T-DCT-K | 28.29 | 4.49 | 81.728% | 95.862%
Swin-T-DCT-V | 28.29 | 4.49 | 81.528% | 95.850%
Swin-T-DCT-0.25 | 22.21 | 3.24 | 79.580% | 94.936%
Swin-T-DCT-0.5 | 23.43 | 3.64 | 80.762% | 95.520%
Swin-T-DCT-0.75 | 25.45 | 4.18 | 81.474% | 95.728%
Swin-S (Torchvision) [29] | 49.61 | 8.74 | 83.196% | 96.360%
PoolFormer-M36 [55] | 56 | 8.8 | 82.1% | -
DCFormer-SW-S ($\tau=1)$ [26] | 49.61 | 8.7 | 82.8% | -
DCFormer-SW-S ($\tau=2)$ [26] | 49.61 | 2.7 | 80.9% | -
gSwin-S [17] | 40 | 7.0 | 83.014% | -
Swin-S-DCT-Q | 49.61 | 8.74 | 83.298% | 96.298%
Swin-S-DCT-K | 49.61 | 8.74 | 83.378% | 96.398%
Swin-S-DCT-V | 49.61 | 8.74 | 83.200% | 96.364%
Swin-S-DCT-0.25 | 38.54 | 6.28 | 81.840% | 95.704%
Swin-S-DCT-0.5 | 40.76 | 7.07 | 82.622% | 96.036%
Swin-S-DCT-0.75 | 44.45 | 8.13 | 83.202% | 96.388%
ViT-B-32 [14] | 88.22 | 4.41 | 73.120% | 90.306%
ViT-B-32-DCT-Q | 88.22 | 4.41 | 73.278% | 90.574%
ViT-B-32-DCT-K | 88.22 | 4.41 | 73.614% | 90.870%
ViT-B-32-DCT-V | 88.22 | 4.41 | 73.306% | 90.502%
ViT-B-32-DCT-0.5 | 66.97 | 3.51 | 72.948% | 90.390%
ViT-B-32-DCT-0.75 | 75.83 | 4.05 | 73.226% | 90.642%

Then, we evaluate the DCT-compressed attention with different truncating rates
and compare the results with DCFormer [26], as it is also a DCT-based method
to reduce the computational cost. Table 3 presents that when we keep 75% of
DCT coefficients, we can obtain a comparable accuracy result on Swin-T and a
slightly better accuracy result on Swin-S. Contrarily, the trainable
parameters of the Swin-T and the Swin-S are reduced from 28.29M to 25.45M
(13.1% off) and from 49.61M to 44.45M (10.4% off), respectively. Flops are
reduced from 4.49G to 4.18G (6.9% off) and 8.13G to 8.74G (7.0% off),
respectively. If fewer DCT coefficients are kept, the swim transformers can
get further savings on parameters and computational overhead. Meanwhile, the
accuracy will drop correspondingly. As a comparison, although DCFormer-SW-T/S
[26] with $\tau=2$ reduce the FLOPs significantly ($\tau=2$ means $4\times$
frequency compression), the performance drops seriously because the DCFormer
takes both low-pass and high-pass DCT components equivalent important.

We further evaluate the two proposed DCT-based attentions in the ViT-B-32
backbone [14] to verify their performance on other vision transformers. When
we initialize the weights to compute keys ($\mathbf{K}$) as the DCT weights,
the accuracy of ViT-B-32 is improved from 73.120% to 73.614%. When we apply
DCT-compressed attention with 75% coefficients kept, the accuracy increases
from 73.120% to 73.226% while the number of trainable parameters and FLOPs
decline from 88.22M to 75.83M (14.0% off) and reduced from 4.41G to 4.05G
(8.2% off), respectively.

###  4.3 Ablation Study

Multiple DCT Initialization Table 4 presents the ablation study about
initializing multiple matrices of $\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, and
$\mathbf{W}_{V}$ as the DCT matrix. Although such an initialization still
improves the accuracy for Swin-T and Swin-S, the improvement is smaller than
with the single DCT initialization. For example, in the Swin-T experiments, if
$\mathbf{W}_{Q}$ and $\mathbf{W}_{K}$ are both initialized as the DCT matrix,
the top-1 accuracy is 81.540%, lower than 81.568% from initializing
$\mathbf{W}_{Q}$ only and 81.728% from initializing $\mathbf{W}_{K}$ only.
This deterioration is caused by the weight symmetry problem [20] from
initializing the multiple weights as the same values. In other words, the
attention mechanism is designed to attend to relevant information in the input
sequence. If $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ are the same, the
model may not effectively capture the relationships between different
positions in the input sequence. Moreover, one of the advantages of self-
attention is its ability to model complex dependencies between different
positions in the sequence. If $\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, and
$\mathbf{W}_{V}$ are the same, the capacity of the model to capture diverse
patterns will be limited, and the performance of the model will drop
correspondingly.

Table 4: ImageNet-1K ablation study on multiple weights initialization as the DCT matrix. Model | DCT | Top-1 Acc. | Top-5 Acc.
---|---|---|---
Swin-T | - | 81.474% | 95.776%
Swin-T-DCT | Q | 81.568% | 95.862%
Swin-T-DCT | K | 81.728% | 95.862%
Swin-T-DCT | V | 81.528% | 95.850%
Swin-T-DCT | QK | 81.540% | 95.754%
Swin-T-DCT | QV | 81.548% | 95.748%
Swin-T-DCT | KV | 81.536% | 95.770%
Swin-S | - | 83.196% | 96.360%
Swin-S-DCT | Q | 83.298% | 96.298%
Swin-S-DCT | K | 83.378% | 96.398%
Swin-S-DCT | QK | 83.208% | 96.378%

Non-Trainable DCT Weights Initialization In this case, computing queries,
keys, and values is equivalent to shifting the DCT representation of the input
patterns. Table 5 shows we obtain comparable accuracy results as the baseline
but use fewer trainable parameters on the CIFAR-10 dataset. It also shows on
the ImageNet-1K dataset if we take non-trainable DCT weights to compute
$\mathbf{Q}$ or $\mathbf{K}$, the top-1 accuracy drops insignificantly (less
than 0.3%). It is worthwhile noting that in this work, the DCT is implemented
by matrix multiplication. Hence, the computational overhead of using non-
trainable DCT weights is the same as the vanilla attention. Still, the DCT
with butterfly operation can further reduce this cost as we only need to apply
DCT in each attention layer.

Table 5: Ablation study on non-trainable DCT weights. Model | DCT | CIFAR-10 | ImageNet-1K
---|---|---|---
Params (M) | Top-1 Acc. | Params (M) | Top-1 Acc. | Top-5 Acc.
Swin-T | - | 27.53 | 90.51% | 28.29 | 81.474% | 95.776%
Swin-T | Q | 25.37 | 90.47% | 26.13 | 81.214% | 95.674%
Swin-T | K | 25.37 | 90.64% | 26.13 | 81.302% | 95.788%
Swin-T | V | 25.37 | 90.75% | 26.13 | 80.486% | 95.236%
Swin-T | QK | 23.21 | 90.70% | 23.98 | 80.608% | 95.446%
Swin-T | QV | 23.21 | 90.50% | 23.98 | 80.234% | 95.208%
Swin-T | KV | 23.21 | 90.62% | 23.98 | 80.228% | 95.276%

###  4.4 COCO 2017 Object Detection and Instance Segmentation

We further evaluate the proposed methods on COCO 2017 object detection and
instance segmentation tasks [28]. We use the Swin Transformers with
ImageNet-1K pre-trained weights as the backbone and the Cascade Mask R-CNN [6,
7] as the pipeline. Since initializing $\mathbf{W}_{K}$ as the DCT matrix
obtains the best accuracy results in the ImageNet-1K experiments, we use Swin-
T-DCT-K and Swin-S-DCT-K to evaluate the DCT-based initialization. As Table 6
presents, both Swin-T-DCT-K and Swin-S-DCT-K obtain slightly better average
precision (AP) results than the vanilla models in the two tasks. Swin-T-
DCT-0.75 obtains comparable AP results as Swin-T with about 3M fewer
parameters, and Swin-S-DCT-0.75 gets better AP results than Swin-S with about
5M fewer parameters.

Table 6: COCO results. Our methods with higher or lower $\text{AP}^{\text{Mask}}$ are highlighted in pink or cyan. Backbone | Params (M) | Object Detection | Instance Segmentation
---|---|---|---
$\text{AP}^{\text{Mask}}$ | $\text{AP}^{\text{Mask}}_{0.5}$ | $\text{AP}^{\text{Mask}}_{0.75}$ | $\text{AP}^{\text{Mask}}$ | $\text{AP}^{\text{Mask}}_{0.5}$ | $\text{AP}^{\text{Mask}}_{0.75}$
Swin-T [29] | 86 | 50.3 | 69.1 | 54.3 | 43.7 | 66.6 | 47.3
DCFormer-T [26] | 86 | 43.5 | 62.6 | 47.4 | 38.0 | 59.3 | 40.7
Swin-T-DCT-K | 86 | 50.4 | 69.2 | 54.3 | 43.9 | 66.6 | 47.6
Swin-T-DCT-0.75 | 83 | 50.0 | 69.1 | 53.9 | 43.7 | 66.8 | 47.6
Swin-S [29] | 107 | 51.8 | 70.4 | 56.3 | 44.7 | 67.9 | 48.5
DCFormer-S [26] | 107 | 46.6 | 64.9 | 50.4 | 40.5 | 62.2 | 43.7
Swin-S-DCT-K | 107 | 52.4 | 70.8 | 57.3 | 45.1 | 68.3 | 49.1
Swin-S-DCT-0.75 | 102 | 52.1 | 70.6 | 57.0 | 45.0 | 68.0 | 48.8

##  5 Limitation

DCT in this work is implemented by matrix multiplication because the fast DCT
is not officially supported in PyTorch. In the future, we will implement fast
DCT to further simplify the computation.

##  6 Conclusion

In this work, we introduced two new methods using Discrete Cosine Transform
(DCT) to decorrelate the input in the attention for vision transformers.
First, we presented a novel DCT-based initialization approach to address the
challenges associated with training attention weights. We demonstrated a
significant improvement over traditional initialization, providing a robust
foundation for the attention mechanism. Second, we formulated a DCT-based
compression method that can effectively reduce the computational overhead by
truncating high-frequency components. It resulted in smaller weight matrices
with compromising considerable accuracy. We validated the efficacy of these
breakthroughs, showcasing their potential to enhance the performance and
efficiency of Transformer models in the classification tasks. These
contributions underscore the importance of thoughtful initialization
strategies and compression techniques in advancing the capabilities of the
Swin Transformer architectures.

## References

  * [1] Nasir Ahmed, T_ Natarajan, and Kamisetty R Rao.  Discrete cosine transform.  IEEE transactions on Computers, 100(1):90–93, 1974.
  * [2] Ali N Akansu and Mustafa U Torun.  Toeplitz approximation to empirical correlation matrix of asset returns: A signal processing perspective.  IEEE Journal of Selected Topics in Signal Processing, 6(4):319–326, 2012.
  * [3] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman.  Hydra attention: Efficient attention with many heads.  In European Conference on Computer Vision, pages 35–49. Springer, 2022.
  * [4] Stevo Bozinovski.  Reminder of the first paper on transfer learning in neural networks, 1976\.  Informatica, 44(3), 2020.
  * [5] Tim-Oliver Buchholz and Florian Jug.  Fourier image transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1846–1854, 2022.
  * [6] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: Delving into high quality object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154–6162, 2018.
  * [7] Zhaowei Cai and Nuno Vasconcelos.  Cascade r-cnn: High quality object detection and instance segmentation.  IEEE transactions on pattern analysis and machine intelligence, 43(5):1483–1498, 2019.
  * [8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [9] Wen-Hsiung Chen, CH Smith, and Sam Fralick.  A fast computational algorithm for the discrete cosine transform.  IEEE Transactions on communications, 25(9):1004–1009, 1977.
  * [10] Lu Chi, Borui Jiang, and Yadong Mu.  Fast fourier convolution.  Advances in Neural Information Processing Systems, 33:4479–4488, 2020.
  * [11] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation policies from data.  arXiv preprint arXiv:1805.09501, 2018.
  * [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [13] R Dony et al.  Karhunen-loeve transform.  The transform and data compression handbook, 1(1-34):29, 2001.
  * [14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [15] Touradj Ebrahimi and Caspar Horne.  Mpeg-4 natural video coding–an overview.  Signal Processing: Image Communication, 15(4-5):365–385, 2000.
  * [16] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * [17] Mocho Go and Hideyuki Tachibana.  gswin: Gated mlp vision model with hierarchical structure of shifted window.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [20] Shell Xu Hu, Sergey Zagoruyko, and Nikos Komodakis.  Exploring weight symmetry in deep neural networks.  Computer Vision and Image Understanding, 187:102786, 2019.
  * [21] Wei Hu, Lechao Xiao, and Jeffrey Pennington.  Provable benefit of orthogonal initialization in optimizing deep linear networks.  In International Conference on Learning Representations, 2019.
  * [22] Michael Jachan, Gerald Matz, and Franz Hlawatsch.  Time-frequency-autoregressive random processes: Modeling and fast parameter estimation.  In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP’03)., volume 6, pages VI–125. IEEE, 2003.
  * [23] Debesh Jha, Nikhil Kumar Tomar, Vanshali Sharma, and Ulas Bagci.  Transnetr: Transformer-based residual network for polyp segmentation with multi-center out-of-distribution testing.  In Medical Imaging with Deep Learning, 2023.
  * [24] Xiaozhong Ji, Boyuan Jiang, Donghao Luo, Guangpin Tao, Wenqing Chu, Zhifeng Xie, Chengjie Wang, and Ying Tai.  Colorformer: Image colorization via color memory assisted hybrid-attention transformer.  In European Conference on Computer Vision, pages 20–36. Springer, 2022.
  * [25] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [26] Xinyu Li, Yanyi Zhang, Jianbo Yuan, Hanlin Lu, and Yibo Zhu.  Discrete cosin transformer: Image modeling from frequency domain.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5468–5478, 2023.
  * [27] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  Swinir: Image restoration using swin transformer.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1833–1844, 2021.
  * [28] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [30] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2018.
  * [31] James Martens et al.  Deep learning via hessian-free optimization.  In ICML, volume 27, pages 735–742, 2010.
  * [32] Sarfaraz Masood and Pravin Chandra.  Training neural network with zero weight initialization.  In Proceedings of the CUBE International Information Technology Conference, pages 235–239, 2012.
  * [33] Sachin Mehta and Mohammad Rastegari.  Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer.  In International Conference on Learning Representations, 2021.
  * [34] Hongyi Pan, Diaa Badawi, and Ahmet Enis Cetin.  Block walsh–hadamard transform-based binary layers in deep neural networks.  ACM Transactions on Embedded Computing Systems, 21(6):1–25, 2022\.
  * [35] Hongyi Pan, Emadeldeen Hamdan, Xin Zhu, Salih Atici, and Ahmet Enis Cetin.  Multichannel orthogonal transform-based perceptron layers for efficient resnets.  IEEE Transactions on Neural Networks and Learning Systems, 2024\.
  * [36] Hongyi Pan, Xin Zhu, Salih Furkan Atici, and Ahmet Cetin.  A hybrid quantum-classical approach based on the hadamard transform for the convolutional layer.  In International Conference on Machine Learning, pages 26891–26903. PMLR, 2023.
  * [37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [38] Anabeth P Radünz, Fábio M Bayer, and Renato J Cintra.  Low-complexity rounded klt approximation for image compression.  Journal of Real-Time Image Processing, pages 1–11, 2022.
  * [39] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [40] Andrew M Saxe, James L McClelland, and Surya Ganguli.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  arXiv preprint arXiv:1312.6120, 2013.
  * [41] Carmelo Scribano, Giorgia Franchini, Marco Prato, and Marko Bertogna.  Dct-former: Efficient self-attention with discrete cosine transform.  Journal of Scientific Computing, 94(3):67, 2023.
  * [42] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li.  Efficient attention: Attention with linear complexities.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531–3539, 2021.
  * [43] Jiechong Song, Chong Mou, Shiqi Wang, Siwei Ma, and Jian Zhang.  Optimization-inspired cross-attention transformer for compressive sensing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6174–6184, 2023.
  * [44] Vladislav Sovrasov.  ptflops: a flops counting tool for neural networks in pytorch framework, 2018–2024.
  * [45] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al.  Sparse r-cnn: End-to-end object detection with learnable proposals.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14454–14463, 2021.
  * [46] Matteo Testa and Enrico Magli.  Compressive estimation and imaging based on autoregressive models.  IEEE Transactions on Image Processing, 25(11):5077–5087, 2016.
  * [47] Nikhil Kumar Tomar, Annie Shergill, Brandon Rieders, Ulas Bagci, and Debesh Jha.  Transresu-net: A transformer based resu-net for real-time colon polyp segmentation.  In 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 1–4. IEEE, 2023.
  * [48] Asher Trockman and J Zico Kolter.  Mimetic initialization of self-attention layers.  arXiv preprint arXiv:2305.09828, 2023.
  * [49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [50] Gregory K Wallace.  The jpeg still picture compression standard.  Communications of the ACM, 34(4):30–44, 1991.
  * [51] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang.  A survey of transfer learning.  Journal of Big data, 3(1):1–40, 2016.
  * [52] Weidong Xiao, Nianbin Wan, Alan Hong, and Xiaoyan Chen.  A fast jpeg image compression algorithm based on dct.  In 2020 IEEE International Conference on Smart Cloud (SmartCloud), pages 106–110. IEEE, 2020.
  * [53] Haoran Xu, Benjamin Van Durme, and Kenton Murray.  Bert, mbert, or bibert? a study on contextualized embeddings for neural machine translation.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6663–6675, 2021.
  * [54] Dongjie Ye, Zhangkai Ni, Hanli Wang, Jian Zhang, Shiqi Wang, and Sam Kwong.  Csformer: Bridging convolution and transformer for compressive sensing.  IEEE Transactions on Image Processing, 2023.
  * [55] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan.  Metaformer is actually what you need for vision.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819–10829, 2022.
  * [56] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032, 2019.
  * [57] Bowen Zhang, Shuyang Gu, Bo Zhang, Jianmin Bao, Dong Chen, Fang Wen, Yong Wang, and Baining Guo.  Styleswin: Transformer-based gan for high-resolution image generation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11304–11314, 2022.
  * [58] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In International Conference on Learning Representations, 2018.
  * [59] Zheyuan Zhang and Ulas Bagci.  Dynamic linear transformer for 3d biomedical image segmentation.  In International Workshop on Machine Learning in Medical Imaging, pages 171–180. Springer, 2022.
  * [60] Jiawei Zhao, Florian Tobias Schaefer, and Anima Anandkumar.  Zero initialization: Initializing neural networks with only zeros and ones.  Transactions on Machine Learning Research, 2022.
  * [61] Lin Zhao, Zihao Wu, Haixing Dai, Zhengliang Liu, Tuo Zhang, Dajiang Zhu, and Tianming Liu.  Embedding human brain function via transformer.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 366–375. Springer, 2022.
  * [62] Ce Zheng, Xianpeng Liu, Guo-Jun Qi, and Chen Chen.  Potter: Pooling attention transformer for efficient human mesh recovery.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1611–1620, 2023.
  * [63] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001–13008, 2020.
  * [64] Xin Zhu, Daoguang Yang, Hongyi Pan, Hamid Reza Karimi, Didem Ozevin, and Ahmet Enis Cetin.  A novel asymmetrical autoencoder with a sparsifying discrete cosine stockwell transform layer for gearbox sensor data compression.  Engineering Applications of Artificial Intelligence, 127:107322, 2024.

##  Appendix A Technical Appendices

###  A.1 Implementation Details

Models are trained on a server with 8 NVIDIA A100 80GB GPUs using Python with
PyTorch. The code of this work will be released upon the paper’s acceptance.

####  A.1.1 Classification Tasks Implementation Details

Training models for the image classification tasks from scratch mostly follow
the settings in [29]. We employ AdamW optimizer [30] for 300 epochs using a
cosine decay learning rate scheduler and 20 epochs of linear warm-up. We set
the batch size as 1024, the initial learning rate as 0.001, and the weight
decay as 0.05. Data augmentation is performed as the following: The CIFAR-10
augmentation follows [19]. First, the training images are padded with 4
pixels. Then, they are randomly cropped to get 32 by 32. Finally, the images
are randomly flipped horizontally. The images are normalized with the means of
[0.4914, 0.4822, 0.4465] and the standard variations of [0.2023, 0.1994,
02010]. The ImageNet-1K augmentation follows PyTorch Torchvision’s GitHub
repository
[37]111https://github.com/pytorch/vision/tree/release/2.0/references/classification.
As a start, images are resized to make the smaller edge of the image 232 for
Swin-T and 246 for Swin-S. Next, they are cropped to 224 by 224 images and
randomly flipped horizontally. Techniques such as Autoaugment [11], mixup
[58], cutmix [56], and random erasing [63] are applied. Images are normalized
with the means of [0.485, 0.456, 0.406] and the standard variations of [0.229,
0.224, 0.225]. During the training, the best models are stored based on the
top-1 accuracy on the CIFAR-10 test dataset and the ImageNet-1K validation
dataset.

Flops computation for the Swin Transformers and our revisions uses the
function “flops” in the official implementation of
[29]222https://github.com/microsoft/Swin-
Transformer/blob/main/models/swin_transformer.py. To follow their work, the
flops from the Softmax function are omitted.

Flops computation for the ViT-B-32 and our revisions uses the function
“ptflops” in the official implementation of [44], with manually adding the
flops from DCT.

####  A.1.2 COCO Detection and Segmentation Tasks Implementation Details

Training models for the COCO tasks follow the settings in [29]. Models start
from ImageNet-1K pre-trained weights. We use AdamW optimizer [30] for 36
epochs with an initial learning rate of 0.0001, weight decay of 0.05, batch
size of 16, 3$\times$ schedule, and multi-scale training [8, 45].