# A Framework for Computational Design and Adaptation of Extended Reality User
Interfaces

Kashyap Todi<EMAIL_ADDRESS>Meta Reality LabsUSA and  Tanya R. Jonker
<EMAIL_ADDRESS>Meta Reality LabsUSA

(2023)

###### Abstract.

To facilitate high quality interaction during the regular use of computing
systems, it is essential that the user interface (UI) deliver content and
components in an appropriate manner. Although extended reality (XR) is
emerging as a new computing platform, we still have a limited understanding of
how best to design and present interactive content to users in such immersive
environments. Adaptive UIs offer a promising approach for optimal presentation
in XR as the user’s environment, tasks, capabilities, and preferences vary
under changing context. In this position paper, we present a design framework
for adapting various characteristics of content presented in XR. We frame
these as five considerations that need to be taken into account for adaptive
XR UIs: _What?_ , _How Much?_ , _Where?_ , _How?_ , and _When?_. With this
framework, we review literature on UI design and adaptation to reflect on
approaches that have been adopted or developed in the past towards identifying
current gaps and challenges, and opportunities for applying such approaches in
XR. Using our framework, future work could identify and develop novel
computational approaches for achieving successful adaptive user interfaces in
such immersive environments.

††journalyear: 2023††copyright: rightsretained††conference: CHI Conference on
Human Factors in Computing Systems; April 23-28, 2023; Hamburg,
Germany††booktitle: CHI Conference on Human Factors in Computing Systems (CHI
’23), April 23-28, 2022, Hamburg, Germany††conference: ACM CHI Conference on
Human Factors in Computing Systems; April 23–April 28; Hamburg, Germany.

Figure 1. We propose a framework for design and adaptation of XR UIs. The
framework addresses key questions related to content selection, presentation,
and placement. By addressing these aspects, future XR systems can
systematically determine content and layout of UIs that are optimal for
varying contexts of usage.

A diagram illustrating our framework for design and adaptation of XR UIs. The
framework addresses key questions related to content selection, presentation,
and placement. By addressing these aspects, future XR systems can
systematically determine content and layout of UIs that are optimal for
varying contexts of usage.

##  1\. Introduction

Extended reality (XR) is a growing area of post-desktop computing, spanning
virtual reality (VR), augmented reality (AR), and mixed reality (MR). It
offers the promise of a new computing environment for the future, where users
are able to seamlessly interact with both the real and the virtual world. To
realize this promise, XR requires a user interface (UI) that overlays virtual
content onto or embeds it within the real, physical environment without costs
to users’ attention, effort, and safety, among other things. Prototypes of
possible UIs include approaches that either take advantage of semantics shared
between real and virtual content to associate UIs to the environment (Cheng et
al., 2021) or operate independently by assuming virtual content is anchored to
the user (Damian et al., 2015). Further, representation and modality are
additional key aspects for rendering UIs in XR: information can be presented
via non-visual modalities, such as audio and haptics; the size or level of
detail can be modulated; and the same information can be presented visually
using various two- or three-dimensional representations. As such, there is an
immense design space for XR UIs.

User’s environment, task, and capabilities, often referred to collectively as
context (Lieberman and Selker, 2000), have large variance during continual use
of XR systems and influence what a ‘good UI’ might be. Intermixing of real and
virtual introduces substantial risk of distraction, additional attention
costs, and even safety risks. As such, visual ‘real estate’ for virtual
content is (1) reduced and (2) constantly changing. Furthermore, as users’
tasks and contexts continually change, their interest in and need for virtual
interaction also constantly vary. In contrast to typical computing platforms,
where adaptive behavior often plays a small (and neglectable) role, it is
believed that adaptive UIs might be the default for future XR environments
(Lindlbauer, 2022; Jonker et al., 2020).

Prior work has looked into various aspects related to design and adaptation of
UIs, as applied to various domains ranging from traditional 2D menus and
graphical UIs (e.g. (Gobert et al., 2019; Lok and Feiner, 2001; Li et al.,
2019)) to cross-device (Park et al., 2018) and immersive 3D UIs (Lindlbauer et
al., 2019; Grubert et al., 2017). Correspondingly, there have been several
computational approaches, such as constraint solving (Zeidler et al., 2012),
optimization (O’Donovan et al., 2014), and machine learning (Li et al., 2019),
developed and applied for achieving promising design solutions. While existing
literature can provide valuable insights on approaches that can be adopted
towards achieving certain adaptive behavior for future XR systems, current
understanding of what aspects should be addressed when adapting such UIs is
limited and fragmented.

This position paper is motivated by the need to holistically understand
adaptive behavior of UIs that will be required for achieving highly usable and
performant interactions in future XR environments. To this end, we introduce a
novel framework that characterizes UI design and adaptation along five
dimensions (or questions): what?, how much?, how?, where?, and when?. By
answering these questions, adaptive XR systems could dynamically select
appropriate content, presentation, and placement of UIs. We review prior
literature on computational design and adaptation approaches across a range of
application domains to understand what properties of the UI they address and
features that drive adaptations in the UI. We also summarize some of the
technical approaches that have been developed and applied towards achieving
optimal design and adaptation. Our review highlights the need for further
studying various UI adaptations directly in the context of extended reality,
with varying tasks, environments, and capabilities. Our findings can inform
future research directions in the area of adaptive XR UIs, driven by
computational methods, to improve usability and user experience.

### Overview: Adaptive XR UI Framework

Context, including user’s task, situational capabilities, and the environment,
plays a key role in determining the right UI in XR environments (Lieberman and
Selker, 2000; Lindlbauer et al., 2019; Jonker et al., 2020) . We develop a
framework for designing and adapting XR UIs based on varying context. In our
framework, we consider different aspects that influence the final UI available
to the user (Figure 1). This includes: (1) _content selection_ (section 2),
which addresses _what_ content should be made available and _how much_ of it;
(2) _presentation_ (section 3), which refers to _how_ and _when_ content and
components are presented; and (3) _placement_ (section 4), which determines
_where_ UIs are positioned in the 3D environment.

##  2\. Content Selection: What? and How Much?

Selecting the right content (‘what?’) to present to the user is necessary so
as to ensure that users have important task-relevant elements readily
available, while minimizing task-irrelevant distractions. Further, the right
amount, or level of detail, of content (‘how much?’) should be modulated to
ensure that users can interact at the appropriate granularity.

What: Prior works have explored several factors that can influence what?
should be presented to users in adaptive UIs, including: interaction history,
user preferences, user capabilities, task, environment, aesthetics, and device
capabilities. Interaction history, containing prior usage of a system, has
been used in traditional desktop-based environments to identify and select
important items for adaptation based on features such as frequency, recency,
and user interests (Gobert et al., 2019; Bylinskii et al., 2017; Todi et al.,
2018, 2021). Other works have investigated the role of user preferences (Park
et al., 2018; Schrier et al., 2008) and capabilities (Sarcar et al., 2018) to
determine and adapt content selection to individual users. Further, task
(Julier et al., 2000) and environment (Evangelista Belo et al., 2022) are also
deemed to be key factors that can inform what content should be presented to
the user: for example, some applications may be used more consistently for
certain tasks or some components have higher priority in certain environments.
Aesthetics can also influence content selection (Balinsky et al., 2009). In
traditional 2D interfaces, aesthetics relates exclusively to how a selection
of UI components can compose a harmonic and aesthetically pleasing UI; for XR,
however, the harmonization of the real world and virtual elements can also
influence content selection. Finally, device capabilities can inform suitable
content; this has been used, for example, to select content for presentation
(Schrier et al., 2008; Nichols et al., 2006a) or appropriate distribution
across devices (Park et al., 2018; Grubert et al., 2015).

How Much: In addition to identifying what content should be selected, prior
works have also investigated how much content is appropriate. ‘How much’
content can be defined by properties such as number of items, level of detail,
and information density. For example, cognitive load (Sweller, 1988) is a key
factor that influences usability and is closely related to how much content is
presented to the user. As such, prior work has used this as a measure to
determine the appropriate granularity (Lindlbauer et al., 2019). Task and
environment are also key factors that have been used to determine how much
content or UI should be presented (Schrier et al., 2008; DiVerdi et al., 2004;
Rao and Card, 1994; Grubert et al., 2017; Julier et al., 2000; Oulasvirta,
2017). As the context changes, the utility of various content changes too,
which can be used to determine the amount of information presented. Similarly,
some prior works have considered user abilities and capabilities to drive
adaptations or design variations (Gajos et al., 2007; Julier et al., 2000;
Sarcar et al., 2018), and others have looked at adapting information density,
level of detail, or granularity based on varying device capabilities (Gajos
and Weld, 2004; Nichols et al., 2006a; DiVerdi et al., 2004; Hallaway et al.,
2004).

In general, content selection is key to ensuring that desired content is
readily available to users, while distracting and irrelevant content is
minimized. Inappropriate selection can lead to increased cognitive load (e.g.
due to excessive content or inappropriate granularity), additional interaction
steps to retrieve unavailable content, and decreased performance when
interacting with the UI. As a first step of the adaptation process, we believe
that future XR systems should first apply computational methods to optimally
select content at the right level of detail.

##  3\. Presentation: How? and When?

Content and UIs can often be presented to users via multiple representations.
For example, information can be presented textually or graphically, incoming
notifications can be pushed using audio or haptics, or a continuous value
selection component can be presented using a slider or a dial. Selecting the
appropriate representation is crucial to ensure usability. Further, suitable
timing for presenting content or adapting the presented content is important
to ensure users are not surprised or confused by changes and continuity is
maintained.

How: Selecting the appropriate modality and representation for content or
components can improve how users perceive and interact with the UI. As such,
this aspect has been studied by prior works in the context of 2D and 3D UIs.
One approach for selecting a representation is to consider other UIs – from
different applications with similar features or different devices with similar
applications – that are also being used: ensuring consistency or compatibility
across UIs can make it easier for users to learn and understand the system
(Nichols et al., 2006a; Gajos and Weld, 2004). To improve aesthetics, color
harmony across components has been studied in the context of UIs (Todi et al.,
2016; Cohen-Or et al., 2006). For XR systems, in addition to harmony across
virtual components, an additional consideration of harmonizing with the
surrounding environment would be beneficial as the level of integration
between virtual elements and the real world influences decision making and
reaction time (Flittner* et al., 2020). Task and environment have also been
used by prior works for determining appropriate representation of UIs (Bell et
al., 2001; Nichols et al., 2006b). Finally, varying device and user
capabilities have also influenced how content should be represented (Beshers
and Feiner, 1989; Gajos and Weld, 2004; Gajos et al., 2007; Nichols et al.,
2006a, b).

When: Timing of content presentation or adaptation is another important factor
that can determine how distracting, confusing, and usable the system might be;
an inappropriately timed adaptation can at best surprise the user and at worst
prevent them from completing their tasks. Unlike other factors, to date, there
is only a limited understanding of when changes should be triggered in
adaptive systems. Some prior works have used heuristics such as changes in the
environment, task, or perceived cognitive load to determine appropriate timing
(Lindlbauer et al., 2019; Madsen et al., 2016; Grubert et al., 2017). A more
promising approach is to model and predict performance improvements that would
be achieved if the system triggered an adaptation. This has been studied in
the context of 2D menu-based UIs (Todi et al., 2021) and 3D XR UIs
(Evangelista Belo et al., 2022; Yu et al., 2022) as a principled approach to
adapt the system.

We hypothesize that the representation and timing aspects when presenting UIs
to users in XR environments will largely influence how acceptable and usable
these systems will be during all-day usage under varying contexts.

##  4\. Placement: Where?

The last question or consideration for designing and adapting UIs in XR
environments is their placement: where should content and UI components be
placed such that users can interact with them with minimal effort? As such,
placement affects key usability aspects such as discoverability, reachability,
exertion, and performance. Determining placement is an especially hard problem
as special attention needs to be paid to relative placement of components to
other components, in addition to each element’s absolute position. This
includes aspects such as sequential or logical ordering, reading order, and
semantic relationships between components.

A wide range of factors have been used by prior research to determine where
UIs and content should be placed, in the context of 2D, 3D, and cross-device
UIs. Constraint-based layouts, where relationship between components are
defined using constraints, have been one common approach to determine final
placement of elements (Hosobe, 2001; Schrier et al., 2008; Badros et al.,
2001; Gal et al., 2014). Similarly, abstract UI specifications have been used
to generate final UI placement for varying context (Gajos and Weld, 2004;
Beshers and Feiner, 1989; Nichols et al., 2006a). This enables systems to
position components while maintaining consistency when features such as screen
size and aspect ratio vary. Grid and geometric based approaches have also been
widely used (Jacobs et al., 2003; Dayama et al., 2020; Hosobe, 2001; Balinsky
et al., 2009): here, structural properties are captured using mathematical
formulation to ensure properties such as alignment and rectangularity.
Occlusion avoidance and ensuring viewability also influence placement
decisions and have been used to generate or adapt UIs (Cheng et al., 2021;
Hosobe, 2001; Azuma and Furmanski, 2003; Bell et al., 2001; Tanaka et al.,
2008; Evangelista Belo et al., 2022). Some prior works have made placement
decision based on other relative placement in other UIs to ensure consistency
and similarity (Todi et al., 2018; Kumar et al., 2011; Nichols et al., 2006a;
Li et al., 2019). Appropriate placement in UIs also highly correlates to task
performance and perceived aesthetics. As such, interactive systems have
studied UI placement to optimize these qualities (Todi et al., 2016; Bailly et
al., 2013; Lok et al., 2004; Balinsky et al., 2009; O’Donovan et al., 2014;
Evangelista Belo et al., 2021). Finally, user task and environment has also
been utilized as a factor that determines where components are placed
(Lindlbauer et al., 2019; Cheng et al., 2021; Madsen et al., 2016; Fender et
al., 2018, 2017; Nuernberger et al., 2016; Bell et al., 2001; Evangelista Belo
et al., 2022; Nichols et al., 2006b; Bordes et al., 2011).

In XR, content and components do not occupy a dedicated screen or canvas;
instead, they are overlaid on the real world and can span the entire 360°
environment around the user. Further, virtual components can not only have
semantic relationships with each other, but with real world objects too. As
such, typical approaches such as grid- and constraint- based layouts can not
be directly applied. Recently, some research has explored how aspects such as
task (Lindlbauer et al., 2019) and semantic relationships (Cheng et al., 2021)
can be used to adaptive determine where elements should be placed in XR
environments. We suggest that future XR systems will need to carefully address
placement issues by considering various aspects pertaining to both the user
and the environment, such as cognitive load, ergonomics, task performance,
aesthetics, semantic consistency, user task and capabilities, and device
constraints.

##  5\. Discussion

In this position paper, we have studied design and adaptation of UIs in the
context of extended reality (XR) systems. We introduce a framework consisting
of five dimensions – what?, how much?, how?, where?, and when? – to
characterize various factors that can influence the final quality of a UI. By
addressing each of these, future XR systems could systematically select
appropriate content, manipulate their presentation, and place them to
optimally support interactions. We discuss existing literature on
computational approaches for UI design and adaptations that touch upon these
questions to understand the state-of-the-art and identity opportunities for
future research. By discussing prior work, we can identify gaps, challenges,
and opportunities for novel contributions towards making adaptive-first XR a
reality. Firstly, it is evident that we need to carefully consider the
blending of the real environment and virtual UIs when making decisions related
to content selection and presentation; this includes aspects related to both
performance and aesthetics. Second, as situational capabilities vary with
changing contexts, this is another key aspect that should be studied when
determine adaptive system behavior. Finally, inherently adaptive systems will
need to pay more attention to timing of changes and content representations
that are suitable to the user’s context.

As the computing paradigm shifts from strict separation of the real world from
virtual interactive interfaces to a blending of these environments in extended
reality, we face new and amplified challenges related to developing design and
development of UIs. While there is extensive prior research on computational
design and adaptation of UIs that can be applied to XR scenarios, our
understanding so far is fragmented and targeted towards specific challenges.
We hope that our design framework can provide new insights that lead to a
structured approach for adaptive XR systems of the future that tackle key
considerations holistically.

## References

  * (1)
  * Azuma and Furmanski (2003) R. Azuma and C. Furmanski. 2003.  Evaluating label placement for augmented reality view management. In _The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings._ 66–75.  https://doi.org/10.1109/ISMAR.2003.1240689
  * Badros et al. (2001) Greg J. Badros, Alan Borning, and Peter J. Stuckey. 2001\.  The Cassowary Linear Arithmetic Constraint Solving Algorithm.  _ACM Trans. Comput.-Hum. Interact._ 8, 4 (dec 2001), 267–306.  https://doi.org/10.1145/504704.504705
  * Bailly et al. (2013) Gilles Bailly, Antti Oulasvirta, Timo Kötzing, and Sabrina Hoppe. 2013.  MenuOptimizer: Interactive Optimization of Menu Systems. In _Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology_ (St. Andrews, Scotland, United Kingdom) _(UIST ’13)_. Association for Computing Machinery, New York, NY, USA, 331–342.  https://doi.org/10.1145/2501988.2502024
  * Balinsky et al. (2009) Helen Y. Balinsky, Anthony J. Wiley, and Matthew C. Roberts. 2009.  Aesthetic Measure of Alignment and Regularity. In _Proceedings of the 9th ACM Symposium on Document Engineering_ (Munich, Germany) _(DocEng ’09)_. Association for Computing Machinery, New York, NY, USA, 56–65.  https://doi.org/10.1145/1600193.1600207
  * Bell et al. (2001) Blaine Bell, Steven Feiner, and Tobias Höllerer. 2001\.  View Management for Virtual and Augmented Reality. In _Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology_ (Orlando, Florida) _(UIST ’01)_. Association for Computing Machinery, New York, NY, USA, 101–110.  https://doi.org/10.1145/502348.502363
  * Beshers and Feiner (1989) C. M. Beshers and S. Feiner. 1989.  Scope: Automated Generation of Graphical Interfaces. In _Proceedings of the 2nd Annual ACM SIGGRAPH Symposium on User Interface Software and Technology_ (Williamsburg, Virginia, USA) _(UIST ’89)_. Association for Computing Machinery, New York, NY, USA, 76–85.  https://doi.org/10.1145/73660.73670
  * Bordes et al. (2011) Lucie Bordes, Toby P Breckon, Ioannis Katramados, and Alireza Kheyrollahi. 2011.  Adaptive object placement for augmented reality use in driver assistance systems.  (2011).
  * Bylinskii et al. (2017) Zoya Bylinskii, Nam Wook Kim, Peter O’Donovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister, Fredo Durand, Bryan Russell, and Aaron Hertzmann. 2017.  Learning Visual Importance for Graphic Designs and Data Visualizations. In _Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology_ (Québec City, QC, Canada) _(UIST ’17)_. Association for Computing Machinery, New York, NY, USA, 57–69.  https://doi.org/10.1145/3126594.3126653
  * Cheng et al. (2021) Yifei Cheng, Yukang Yan, Xin Yi, Yuanchun Shi, and David Lindlbauer. 2021.  SemanticAdapt: Optimization-Based Adaptation of Mixed Reality Layouts Leveraging Virtual-Physical Semantic Connections. In _The 34th Annual ACM Symposium on User Interface Software and Technology_ (Virtual Event, USA) _(UIST ’21)_. Association for Computing Machinery, New York, NY, USA, 282–297.  https://doi.org/10.1145/3472749.3474750
  * Cohen-Or et al. (2006) Daniel Cohen-Or, Olga Sorkine, Ran Gal, Tommer Leyvand, and Ying-Qing Xu. 2006.  Color Harmonization.  _ACM Trans. Graph._ 25, 3 (jul 2006), 624–630.  https://doi.org/10.1145/1141911.1141933
  * Damian et al. (2015) Ionut Damian, Chiew Seng (Sean) Tan, Tobias Baur, Johannes Schöning, Kris Luyten, and Elisabeth André. 2015.  Augmenting Social Interactions: Realtime Behavioural Feedback Using Social Signal Processing Techniques. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ (Seoul, Republic of Korea) _(CHI ’15)_. Association for Computing Machinery, New York, NY, USA, 565–574.  https://doi.org/10.1145/2702123.2702314
  * Dayama et al. (2020) Niraj Ramesh Dayama, Kashyap Todi, Taru Saarelainen, and Antti Oulasvirta. 2020.  GRIDS: Interactive Layout Design with Integer Programming. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ (Honolulu, HI, USA) _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376553
  * DiVerdi et al. (2004) S. DiVerdi, T. Hollerer, and R. Schreyer. 2004.  Level of detail interfaces. In _Third IEEE and ACM International Symposium on Mixed and Augmented Reality_. 300–301.  https://doi.org/10.1109/ISMAR.2004.38
  * Evangelista Belo et al. (2021) João Marcelo Evangelista Belo, Anna Maria Feit, Tiare Feuchtner, and Kaj Grønbæk. 2021.  XRgonomics: Facilitating the Creation of Ergonomic 3D Interfaces. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 290, 11 pages.  https://doi.org/10.1145/3411764.3445349
  * Evangelista Belo et al. (2022) João Marcelo Evangelista Belo, Mathias N. Lystbæk, Anna Maria Feit, Ken Pfeuffer, Peter Kán, Antti Oulasvirta, and Kaj Grønbæk. 2022.  AUIT – the Adaptive User Interfaces Toolkit for Designing XR Applications. In _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_ (Bend, OR, USA) _(UIST ’22)_. Association for Computing Machinery, New York, NY, USA, Article 48, 16 pages.  https://doi.org/10.1145/3526113.3545651
  * Fender et al. (2018) Andreas Fender, Philipp Herholz, Marc Alexa, and Jörg Müller. 2018.  OptiSpace: Automated Placement of Interactive 3D Projection Mapping Content. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (Montreal QC, Canada) _(CHI ’18)_. Association for Computing Machinery, New York, NY, USA, 1–11.  https://doi.org/10.1145/3173574.3173843
  * Fender et al. (2017) Andreas Fender, David Lindlbauer, Philipp Herholz, Marc Alexa, and Jörg Müller. 2017\.  HeatSpace: Automatic Placement of Displays by Empirical Analysis of User Behavior. In _Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology_ (Québec City, QC, Canada) _(UIST ’17)_. Association for Computing Machinery, New York, NY, USA, 611–621.  https://doi.org/10.1145/3126594.3126621
  * Flittner* et al. (2020) Jonathan Flittner*, John Luksas, and Dr. Joseph L. Gabbard. 2020\.  Predicting User Performance in Augmented Reality User Interfaces with Image Analysis Algorithms.  _Proceedings of the Human Factors and Ergonomics Society Annual Meeting_ 64, 1 (2020), 2108–2112.  https://doi.org/10.1177/1071181320641511 arXiv:https://doi.org/10.1177/1071181320641511
  * Gajos and Weld (2004) Krzysztof Gajos and Daniel S. Weld. 2004.  SUPPLE: Automatically Generating User Interfaces. In _Proceedings of the 9th International Conference on Intelligent User Interfaces_ (Funchal, Madeira, Portugal) _(IUI ’04)_. Association for Computing Machinery, New York, NY, USA, 93–100.  https://doi.org/10.1145/964442.964461
  * Gajos et al. (2007) Krzysztof Z. Gajos, Jacob O. Wobbrock, and Daniel S. Weld. 2007\.  Automatically Generating User Interfaces Adapted to Users’ Motor and Vision Capabilities. In _Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology_ (Newport, Rhode Island, USA) _(UIST ’07)_. Association for Computing Machinery, New York, NY, USA, 231–240.  https://doi.org/10.1145/1294211.1294253
  * Gal et al. (2014) Ran Gal, Lior Shapira, Eyal Ofek, and Pushmeet Kohli. 2014\.  FLARE: Fast layout for augmented reality applications. In _2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)_. 207–212.  https://doi.org/10.1109/ISMAR.2014.6948429
  * Gobert et al. (2019) Camille Gobert, Kashyap Todi, Gilles Bailly, and Antti Oulasvirta. 2019.  SAM: A Modular Framework for Self-Adapting Web Menus. In _Proceedings of the 24th International Conference on Intelligent User Interfaces_ (Marina del Ray, California) _(IUI ’19)_. Association for Computing Machinery, New York, NY, USA, 481–484.  https://doi.org/10.1145/3301275.3302314
  * Grubert et al. (2015) Jens Grubert, Matthias Heinisch, Aaron Quigley, and Dieter Schmalstieg. 2015.  MultiFi: Multi Fidelity Interaction with Displays On and Around the Body. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ (Seoul, Republic of Korea) _(CHI ’15)_. Association for Computing Machinery, New York, NY, USA, 3933–3942.  https://doi.org/10.1145/2702123.2702331
  * Grubert et al. (2017) Jens Grubert, Tobias Langlotz, Stefanie Zollmann, and Holger Regenbrecht. 2017.  Towards Pervasive Augmented Reality: Context-Awareness in Augmented Reality.  _IEEE Transactions on Visualization and Computer Graphics_ 23, 6 (2017), 1706–1724.  https://doi.org/10.1109/TVCG.2016.2543720
  * Hallaway et al. (2004) Drexel Hallaway, Steven Feiner, and Tobias Höllerer. 2004\.  BRIDGING THE GAPS: HYBRID TRACKING FOR ADAPTIVE MOBILE AUGMENTED REALITY.  _Applied Artificial Intelligence_ 18, 6 (2004), 477–500.  https://doi.org/10.1080/08839510490462768 arXiv:https://doi.org/10.1080/08839510490462768
  * Hosobe (2001) Hiroshi Hosobe. 2001\.  A Modular Geometric Constraint Solver for User Interface Applications. In _Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology_ (Orlando, Florida) _(UIST ’01)_. Association for Computing Machinery, New York, NY, USA, 91–100.  https://doi.org/10.1145/502348.502362
  * Jacobs et al. (2003) Charles Jacobs, Wilmot Li, Evan Schrier, David Bargeron, and David Salesin. 2003.  Adaptive Grid-Based Document Layout. In _ACM SIGGRAPH 2003 Papers_ (San Diego, California) _(SIGGRAPH ’03)_. Association for Computing Machinery, New York, NY, USA, 838–847.  https://doi.org/10.1145/1201775.882353
  * Jonker et al. (2020) Tanya R Jonker, Ruta Desai, Kevin Carlberg, James Hillis, Sean Keller, and Hrvoje Benko. 2020\.  The Role of AI in Mixed and Augmented Reality Interactions. In _CHI2020 ai4hci Workshop Proceedings. ACM_.
  * Julier et al. (2000) S. Julier, M. Lanzagorta, Y. Baillot, L. Rosenblum, S. Feiner, T. Hollerer, and S. Sestito. 2000.  Information filtering for mobile augmented reality. In _Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000)_. 3–11.  https://doi.org/10.1109/ISAR.2000.880917
  * Kumar et al. (2011) Ranjitha Kumar, Jerry O. Talton, Salman Ahmad, and Scott R. Klemmer. 2011.  Bricolage: Example-Based Retargeting for Web Design. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Vancouver, BC, Canada) _(CHI ’11)_. Association for Computing Machinery, New York, NY, USA, 2197–2206.  https://doi.org/10.1145/1978942.1979262
  * Li et al. (2019) Jianan Li, Tingfa Xu, Jianming Zhang, Aaron Hertzmann, and Jimei Yang. 2019.  LayoutGAN: Generating Graphic Layouts with Wireframe Discriminator. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=HJxB5sRcFQ
  * Lieberman and Selker (2000) H. Lieberman and T. Selker. 2000.  Out of context: Computer systems that adapt to, and learn from, context.  _IBM Systems Journal_ 39, 3.4 (2000), 617–632.  https://doi.org/10.1147/sj.393.0617
  * Lindlbauer (2022) David Lindlbauer. 2022\.  The Future of Mixed Reality is Adaptive.  _XRDS_ 29, 1 (oct 2022), 26–31.  https://doi.org/10.1145/3558191
  * Lindlbauer et al. (2019) David Lindlbauer, Anna Maria Feit, and Otmar Hilliges. 2019\.  Context-Aware Online Adaptation of Mixed Reality Interfaces. In _Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology_ (New Orleans, LA, USA) _(UIST ’19)_. Association for Computing Machinery, New York, NY, USA, 147–160.  https://doi.org/10.1145/3332165.3347945
  * Lok and Feiner (2001) Simon Lok and Steven Feiner. 2001.  A survey of automated layout techniques for information presentations.  _Proceedings of SmartGraphics_ 2001 (2001), 61–68.
  * Lok et al. (2004) Simon Lok, Steven Feiner, and Gary Ngai. 2004.  Evaluation of Visual Balance for Automated Layout. In _Proceedings of the 9th International Conference on Intelligent User Interfaces_ (Funchal, Madeira, Portugal) _(IUI ’04)_. Association for Computing Machinery, New York, NY, USA, 101–108.  https://doi.org/10.1145/964442.964462
  * Madsen et al. (2016) Jacob Boesen Madsen, Markus Tatzgern, Claus B. Madsen, Dieter Schmalstieg, and Denis Kalkofen. 2016.  Temporal Coherence Strategies for Augmented Reality Labeling.  _IEEE Transactions on Visualization and Computer Graphics_ 22, 4 (2016), 1415–1423.  https://doi.org/10.1109/TVCG.2016.2518318
  * Nichols et al. (2006a) Jeffrey Nichols, Brad A. Myers, and Brandon Rothrock. 2006a.  UNIFORM: Automatically Generating Consistent Remote Control User Interfaces. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Montréal, Québec, Canada) _(CHI ’06)_. Association for Computing Machinery, New York, NY, USA, 611–620.  https://doi.org/10.1145/1124772.1124865
  * Nichols et al. (2006b) Jeffrey Nichols, Brandon Rothrock, Duen Horng Chau, and Brad A. Myers. 2006b.  Huddle: Automatically Generating Interfaces for Systems of Multiple Connected Appliances. In _Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology_ (Montreux, Switzerland) _(UIST ’06)_. Association for Computing Machinery, New York, NY, USA, 279–288.  https://doi.org/10.1145/1166253.1166298
  * Nuernberger et al. (2016) Benjamin Nuernberger, Eyal Ofek, Hrvoje Benko, and Andrew D. Wilson. 2016.  SnapToReality: Aligning Augmented Reality to the Real World. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ (San Jose, California, USA) _(CHI ’16)_. Association for Computing Machinery, New York, NY, USA, 1233–1244.  https://doi.org/10.1145/2858036.2858250
  * Oulasvirta (2017) Antti Oulasvirta. 2017\.  User Interface Design with Combinatorial Optimization.  _Computer_ 50, 1 (2017), 40–47.  https://doi.org/10.1109/MC.2017.6
  * O’Donovan et al. (2014) Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann. 2014\.  Learning layouts for single-page graphic designs.  _IEEE transactions on visualization and computer graphics_ 20, 8 (2014), 1200–1213.
  * Park et al. (2018) Seonwook Park, Christoph Gebhardt, Roman Rädle, Anna Maria Feit, Hana Vrzakova, Niraj Ramesh Dayama, Hui-Shyong Yeo, Clemens N. Klokmose, Aaron Quigley, Antti Oulasvirta, and Otmar Hilliges. 2018\.  AdaM: Adapting Multi-User Interfaces for Collaborative Environments in Real-Time. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (Montreal QC, Canada) _(CHI ’18)_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi.org/10.1145/3173574.3173758
  * Rao and Card (1994) Ramana Rao and Stuart K. Card. 1994.  The Table Lens: Merging Graphical and Symbolic Representations in an Interactive Focus + Context Visualization for Tabular Information. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Boston, Massachusetts, USA) _(CHI ’94)_. Association for Computing Machinery, New York, NY, USA, 318–322.  https://doi.org/10.1145/191666.191776
  * Sarcar et al. (2018) Sayan Sarcar, Jussi P.P. Jokinen, Antti Oulasvirta, Zhenxin Wang, Chaklam Silpasuwanchai, and Xiangshi Ren. 2018.  Ability-Based Optimization of Touchscreen Interactions.  _IEEE Pervasive Computing_ 17, 1 (2018), 15–26.  https://doi.org/10.1109/MPRV.2018.011591058
  * Schrier et al. (2008) Evan Schrier, Mira Dontcheva, Charles Jacobs, Geraldine Wade, and David Salesin. 2008\.  Adaptive Layout for Dynamically Aggregated Documents. In _Proceedings of the 13th International Conference on Intelligent User Interfaces_ (Gran Canaria, Spain) _(IUI ’08)_. Association for Computing Machinery, New York, NY, USA, 99–108.  https://doi.org/10.1145/1378773.1378787
  * Sweller (1988) John Sweller. 1988\.  Cognitive load during problem solving: Effects on learning.  _Cognitive science_ 12, 2 (1988), 257–285.
  * Tanaka et al. (2008) Kohei Tanaka, Yasue Kishino, Masakazu Miyamae, Tsutomu Terada, and Shojiro Nishio. 2008\.  An information layout method for an optical see-through head mounted display focusing on the viewability. In _2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality_. 139–142.  https://doi.org/10.1109/ISMAR.2008.4637340
  * Todi et al. (2021) Kashyap Todi, Gilles Bailly, Luis Leiva, and Antti Oulasvirta. 2021.  Adapting User Interfaces with Model-Based Reinforcement Learning. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 573, 13 pages.  https://doi.org/10.1145/3411764.3445497
  * Todi et al. (2018) Kashyap Todi, Jussi Jokinen, Kris Luyten, and Antti Oulasvirta. 2018.  Familiarisation: Restructuring Layouts with Visual Learning Models. In _23rd International Conference on Intelligent User Interfaces_ (Tokyo, Japan) _(IUI ’18)_. Association for Computing Machinery, New York, NY, USA, 547–558.  https://doi.org/10.1145/3172944.3172949
  * Todi et al. (2016) Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016.  Sketchplore: Sketch and Explore with a Layout Optimiser. In _Proceedings of the 2016 ACM Conference on Designing Interactive Systems_ (Brisbane, QLD, Australia) _(DIS ’16)_. Association for Computing Machinery, New York, NY, USA, 543–555.  https://doi.org/10.1145/2901790.2901817
  * Yu et al. (2022) Difeng Yu, Ruta Desai, Ting Zhang, Hrvoje Benko, Tanya R. Jonker, and Aakar Gupta. 2022\.  Optimizing the Timing of Intelligent Suggestion in Virtual Reality. In _Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology_ (Bend, OR, USA) _(UIST ’22)_. Association for Computing Machinery, New York, NY, USA, Article 6, 20 pages.  https://doi.org/10.1145/3526113.3545632
  * Zeidler et al. (2012) Clemens Zeidler, Christof Lutteroth, and Gerald Weber. 2012\.  Constraint Solving for Beautiful User Interfaces: How Solving Strategies Support Layout Aesthetics. In _Proceedings of the 13th International Conference of the NZ Chapter of the ACM’s Special Interest Group on Human-Computer Interaction_ (Dunedin, New Zealand) _(CHINZ ’12)_. Association for Computing Machinery, New York, NY, USA, 72–79.  https://doi.org/10.1145/2379256.2379268