# Aligning Robot Representations with Humans

Andreea Bobu  University of California, Berkeley
Berkeley, CA, USA
<EMAIL_ADDRESS>Andi Peng  Massachusetts Institute of Technology
Cambridge, MA, USA
<EMAIL_ADDRESS>
###### Abstract

As robots are increasingly deployed in real-world scenarios, a key question is
how to best transfer knowledge learned in one environment to another, where
shifting constraints and human preferences render adaptation challenging. A
central challenge remains that often, it is difficult (perhaps even
impossible) to capture the full complexity of the deployment environment, and
therefore the desired tasks, at training time. Consequently, the
representation, or abstraction, of the tasks the human hopes for the robot to
perform in one environment may be misaligned with the representation of the
tasks that the robot has learned in another. We postulate that because humans
will be the ultimate evaluator of system success in the world, they are best
suited to communicating the aspects of the tasks that matter to the robot. Our
key insight is that effective learning from human input requires first
explicitly learning good intermediate representations and then using those
representations for solving downstream tasks. We highlight three areas where
we can use this approach to build interactive systems and offer future
directions of work to better create advanced collaborative robots.

###### Index Terms:

Human-robot interaction, robot learning, representation learning.

##  I Introduction

Figure 1: Under our framework, the robot first learns human-guided
representations by asking the human for representation-specific input to
capture specific aspects of the task that they care about (e.g. distance to
laptop, cup orientation, cup near table). The robot then uses the
representation to learn how to perform the task from task-specific input  like
demonstrations, corrections, etc.

Imagine a world where you wake up in the morning, arise from bed, and your
home robot assistant makes your bed. After getting ready, you head downstairs
where your robot has placed a steaming mug of fresh coffee on the table
exactly where it knows you will sit. After drinking the coffee, your robot
picks up the empty mug and places it in the dishwasher as you leave the house
and set off for work. The entire morning, your robot is incorporated
seamlessly into your daily life and home. This scene of domestic bliss
captures the essence of what we hope for from our advanced collaborative
assistants – the ability to effectively complete desired tasks while
integrating into our environments and adapting to our individual preferences,
akin to human-like collaboration.

Today, autonomous systems are increasingly able to learn advanced behaviors
like those mentioned above [1, 2, 3]. However, designing learning algorithms
that match the adaptability and generalizability of human reasoning remains
challenging: while these systems may perform their tasks successfully in the
environment(s) and under the conditions they were trained on, their learned
behaviors may not necessarily work well in novel deployment environments. This
problem can rear its head in a variety of instances: when physical constraints
change (while it’s okay for the robot to break mugs when trying out new grip
poses in the lab, we may wish for them to be more careful in a home), when
environment conditions, layouts, or compositions change (we may wish for the
robot to grasp an octopus-shaped mug that it’s never before seen), or when the
task preferences of the human that the robot interacts with change (one human
may prefer that the coffee is prepared as quickly as possible irrespective of
mess, while another may prefer that the robot prioritizes not spilling the
coffee while navigating the kitchen).

The key issue in all these cases is that, while the designer can anticipate
some of the possible task specifications when training the robot, these
specifications do not necessarily reflect the desires of the other humans the
robot will interact with in its lifetime [4, 5]. In other words, the
representation, or abstraction, of the tasks the human hopes for the robot to
perform in one environment may be misaligned with the representation of the
tasks that the robot has learned in another. Our observation is that because
humans have adapted their environments to capture the full idiosyncrasies of
completing tasks that they desire, they are best equipped to help insert
knowledge specifically describing aspects of the environment that are useful
to the robot in the learning process. Specifically, human input can best help
solve the representation alignment problem of understanding what task aspects
matter to the human when adapting to a new environment.

Traditional methods of robot learning from human input instantiate
representations as a set of hand-engineered features—specific aspects of the
task that a human may care about [6, 7, 8, 9, 10]. These features are pre-
specified by a system designer and function as state-space abstractions that
insert structure for learning the task efficiently. However, they can be
difficult to construct and impossible to exhaustively specify. Meanwhile,
state-of-the-art deep learning methods [3, 11, 12, 13, 14, 15, 8, 16] bypass
feature specification by operating directly on high-dimensional state spaces,
thereby automatically constructing an implicit representation from the
person’s task-specific input (e.g. demonstrations). Unfortunately, because
these methods are optimized to learn the task while bypassing the explicit
need to learn the representation, there is difficulty in disentangling the
high-level representation from the specific task provided [13, 17, 18].
Consequently, effective task learning requires massive amounts of training
data and renders generalization to new tasks difficult. In summary, one
paradigm inserts useful structure to solve the robot learning problem
efficiently but that structure is difficult to define; the other avoids
explicitly specifying the structure but requires too much human data to
extract it implicitly and thus struggles to generalize across different
domains.

We postulate that effective learning from human input requires methodologies
that combine the best of both traditional feature engineering and highly-
expressive deep learning worlds. Our core idea is to divide and conquer the
learning problem: explicitly focus human input on teaching robots good
intermediate representations before using those representations for downstream
tasks. We call these human-guided representations: abstractions that, if
learned well, can enable robots to better solve tasks when deployed into the
real world. We discuss several directions for learning human-guided
representations as well as strategies for identifying misalignment and
improving effective downstream task learning.

##  II Learning Human-Guided Representations

The representation learning literature has accrued a vast body of work on
learning disentangled latent spaces in an unsupervised manner [19, 20, 21].
However, because these methods are purposefully designed to bypass direct
human supervision, the disentangled factors in the learned embedding do not
necessarily correspond to concepts in the human’s representation. In other
words, the robot’s learned representation does not necessarily align with the
human’s, therefore adapting to how they want the task to be done is difficult.
Self-supervised learning inserts some human guidance by allowing for the
designer to specify proxy tasks useful for feature learning [22, 23, 24, 15,
25] (for example, predicting forward dynamics to capture what constrains
movement). In this process, the human designer hopes to instill good
representations into the robot by using their intuition to construct tasks
which illustrate specific features. However, devising proxy tasks is an
exercise that requires nontrivial effort and expertise: human effort to
manually specify features is instead traded for human effort to specify
objective functions for extracting those features.

A more direct way to guide representation alignment is to learn directly from
human input. In standard imitation learning, the robot learns a policy that
copies—or clones—human demonstrations [10, 8]. However, it cannot learn to
imitate what it has not seen before, thus rendering human input non-
generalizable to new tasks [5, 16]. Moreover, BC suffers from the problem of
covariate shift, where once a learned policy drifts away from the
demonstrations, errors compound more and more over time. Inverse reinforcement
learning (IRL) attempts to extract a reward function from demonstrations that
is intended to capture why a specific behaviour is desirable [8], but
unfortunately requires massive amounts of data to truly learn a fully-
specified reward [13, 17]. IRL also requires expert or close to expert
demonstrations [6]. Meta-learning reduces this sample complexity by reusing
demonstrations from an array of different tasks in the training distribution
[26, 27], but ultimately still requires the human to know the test time task
distribution a priori, which brings us back to the manual specification
problem: we now trade hand-crafting features for hand-crafting tasks.

Because demonstrations are intended for teaching the robot how to do the
tasks, not what matters for doing the tasks, they can only contribute to
aligning representations implicitly. This might not result in learning
algorithms extracting salient features that matter to the human for performing
the desired tasks [18]. As shown in Fig. 1, we propose that the robot should
explicitly ask for representation-specific human input to teach it the
intermediate representation before using it to learn more generalizable
downstream tasks from task-specific input. Importantly, because of this
separation, these representations are not specific to any one particular task
the human may want the robot to carry out; instead, they capture aspects
causal for the potential task distribution in the environment.

Designing human input for representation learning. One option for learning
intermediate human-guided representations is to instantiate them as feature
sets like those in traditional methods, and let the human teach individual,
novel features themselves [28, 18]. A natural way to represent any specific
new feature is via a neural network which is trained by asking the human for
supervision labels representing the feature values at different states.
Unfortunately, querying the human for labels to train this neural network
requires a burdensome amount of human interaction. Even worse, humans are
notoriously imprecise at giving these types of numerical inputs, rendering
learned representations likely erroneous [29]. We propose that a key direction
for future work is considering new types of representation-specific input that
are highly informative about the feature without requiring too much effort
from the human. For example, a new type of structured human input called a
feature trace [28], where a human guides the robot from states where the
feature is highly expressed to states where it is not, has been found to
recover more robust and generalizable rewards with far less human effort.
Moving forward, we can study additional forms of human input such as language
or gaze and pose, that can also be targeted for feature learning. Moreover, we
can also consider types of human input that recover the feature representation
as a whole (rather than one by one) via representation-specific proxy tasks –
calibration tasks where the robot’s goal is to specifically align itself with
the demonstrating human.

Transforming the representation for human input. Instead of designing the type
of input the person can give to teach the representation, we can directly
design the type of representation itself. Previously, when we instantiated the
representation as a set of learnable features, we gave the human freedom to
decide what feature each dimension of the representation was and provide
feedback for teaching it to the robot. This enabled the human to add desirable
task aspects to the representation even if the system designer did not
originally think of them. In some cases, though, it may be possible for the
system designer to specify the necessary dimensions of the representation,
just not the mapping to the representation itself. This could happen, for
example, if the designer has prior knowledge that the class of features the
robot needs to express for its tasks has a well-studied representation. For
instance, recent work defines a model to relate emotions expressed in natural
language, such as ‘happy’ or ‘sad’, into the Valence-Arousal-Dominance
spectrum inspired by social psychology [30]. The human can teach the
representation efficiently with natural language by having the robot map their
utterances to their emotive latent VAD equivalent. This way, all user feedback
for this representation contributes to learning about all emotions, and the
robot can model new emotions that interpolate those seen during training.
Moving forward, we should consider leveraging existing methods that define
transformations of natural human-comprehensible concepts, such as language or
images, into robot-comprehensible representations for downstream task learning
[31, 32].

Designing the human-robot interface for learning. In order to truly deploy
collaborative robots in the world, we must eventually develop usable
interactive interfaces that allow for effective information exchange of
representations understood by both the human and robot. Existing work has
highlighted the importance of the interface when a human and robot
collectively share the same workspace, with key considerations being ease of
use, specificity of communication, and reliability of feedback [33, 34].
Current methods suggest using visual displays, hand or face gestures, physical
interaction and haptics, and verbal language can all be viable solutions
towards effective human communication [35]. However, less work has been done
in interfaces for how the robot can effectively communicate the representation
of what it has learned with the human. For example, it would be desirable to
have an interface by which the robot can effectively demonstrate or show the
human what it thinks is the correct desired task prior to actually deploying
it in the real-world. This could be done in the form of mapping the proposed
robot policy to simulated demonstrations or even natural language to
communicate the intended behaviour. We propose that effective human-robot
interaction which leads to learning human-guided representations will require
the development of both streams of information flow in order to fully achieve
its potential.

##  III Identifying Misalignment

Along with learning transferable human-guided representations, it is also
important to detect when misalignment exists in the first place. Misaligned
representations may cause the robot to misinterpret the human’s guidance for
how to complete the task, execute unexpected or undesired behaviors, or
degrade in overall performance [36]. Ergo, we wish for the robot to know when
it does not know the aspects that matter to the human before it starts
incorrectly learning how to perform the task. If misalignment is correctly
detected, then a process which begins with expanding or re-learning the
representation will better help ultimately learn the downstream task. The key
question is: how can the robot autonomously identify representation
misalignment and know when to ask for help?

Several methods suggest an introspective approach where the robot can maintain
uncertainty in its representation’s ability to explain the human’s input. By
modeling humans as noisily rational agents choosing inputs in proportion to
their exponentiated rewards [37, 38, 39], Bayesian approaches can jointly
infer both the reward parameter and a confidence in whether the desired reward
function can be captured by the current representation [40, 41, 42, 36, 43].
When the human input refers to a reward that the robot’s representation cannot
support, the inferred confidence is low, signaling misalignment. Meanwhile,
deep learning methods often study this uncertainty through an ensemble of
neural networks [44, 45]. The intuition here is that if multiple (identically
trained) networks disagree on their predictions, this suggests that the input
is out of distribution and therefore the learned representation is misaligned.

In both cases, once the robot detects misalignment there are a few options for
how to proceed: discard the human input entirely, continue learning in
proportion to its assessed confidence, or halt execution and ask the human to
undergo the process of representation alignment from the previous section
[36]. Assuming the robot identified misalignment correctly, any of these
options are viable alternatives to re-learning from the original human
feedback. Unfortunately, robustly detecting misalignment remains difficult in
many real-world scenarios. We highlight three key areas where identifying
misalignment is particularly challenging and offer brief suggestions for
future work.

Disambiguating between misalignment and noise. When a robot’s representation
cannot explain the human input, it may be difficult to disambiguate whether
this is due to representation misalignment or human noise [36]. This issue
often arises from inexperienced users and is inherent to the types of data
designers must work with in human-robot interaction scenarios. A proposed,
albeit expensive, method of addressing this challenge is to collect more data
to balance out noise, but this solution would not fare well in online learning
scenarios where the robot must detect misalignment in real time, from just a
few observations. We suggest that a more sustainable alternative is to
investigate better human modeling for separating out these two sources of
error [46].

Poor feature learning. Misalignment can additionally occur due to two reasons:
either the robot’s representation does not fully capture an aspect that the
human cares about or it does, but poorly. The latter can occur if some of the
features the robot learned were not learned well enough; for example, a
feature might have required more data from the human in order to cover the
state space and generalize to new areas. We propose that it is crucial for the
robot to distinguish between misalignment due to an incomplete representation
or due to incorrectly learned dimensions of the representation so that instead
of attempting to re-learn a new feature, the robot knows to query for more
data on the existing one. Future work is needed for understanding whether the
robot needs to repair an existing learned feature, detecting which feature
that might be, and developing interactive methods to elicit informative data
to improve existing features.

Feature confusion. An even more fundamental issue exists when the human’s
input refers to something not captured by the robot’s learned representation,
but the representation nonetheless can explain their input. In this case, we
have confused misalignment for human noise [45, 36]. This problem will
especially occur if the representation is highly expressive and can only be
solved by intaking additional human input: each input might be explainable by
some hypothesis, but eventually no hypothesis can explain all input. More work
is needed to study how to query for a broad and diverse set of human input,
how the robot would best demonstrate the features it has learned to the human,
and how to best balance between querying for data vs. learning with existing
data.

##  IV Learning the Downstream Task

Once we have learned a human-guided representation, it is easy to then apply
that representation towards learning a downstream task by using standard
policy [4, 5, 26, 47] or reward learning techniques [48, 9, 12, 49, 14, 7,
50]. However, human-guided representations have important implications for how
they impact the downstream learning pipeline. We subsequently discuss three
considerations that future work should consider to fully close the learning
loop.

Using the right features at the right time. In this proposal, we have
advocated for learning a human-guided representation that is sufficiently
decoupled from any specific task the human may have provided feedback for and
focuses instead on capturing causal aspects for the potential task
distribution in the environment. When the robot specializes on a task, the
representation by construction will contain features that are irrelevant for
that task. If all feature dimensions in the representation were orthogonal to
one another, this would not cause any issue. However, in the real world, many
relevant features may be related and, thus, spurious correlation between
features could affect task learning [51]. Future directions of work should
enable the robot to focus on the right features at the right time. One idea
for accomplishing this is to employ feature selection strategies to activate
the subset of the representation that matters for the specific task at hand.
This strategy could be heuristic-based, like choosing the minimum set that
maximizes coverage [52]. Alternatively, since we would hope for our learned
representations to be more human interpretable in nature, we could also
consider building interfaces where the person themselves can quickly indicate
to the robot which features are important for the specific task they want
[53].

Using representations to better understand humans. Human-guided
representations also enable us to learn something about how the person
generates the task input in the first place. In particular, the previously
mentioned human decision-making models [37, 39, 54] assumed that, out of a set
of choices, the person selects their input in proportion to these choices’
exponentiated rewards. However, we suggest that human-guided representations
inform the robot how it should interpret the person’s task input, thus we
should reinterpret the available choices from the perspective of the learned
representation [55]. We suggest future research must revisit how popular robot
learning methods are affected by reinterpreting human input through the lens
of their representation.

Grounding representations to real-world tasks. Much of HRI has historically
assumed that the robot already has access to all the aspects in the
environment that the interacting human might care about. This assumption has
enabled researchers to make progress on human-robot collaborative algorithms
without needing to worry about how to formally ground the robot’s behaviour to
complex environments and tasks that we would see in the deployment scenarios.
Human-guided representations can help bridge the gap towards learning from
high-dimensional state spaces as we know the real-world to be, opening the
door to HRI applications more challenging and tractable than ever before.

##  V Conclusion

Ultimately, the true evaluators of any system deployed in the real world will
be the humans that it interacts with, and thus soliciting input from them to
effectively learn downstream tasks appears critical. Learning effective
methods to learn from human input holds the promise of enabling more advanced,
collaborative human-aligned robotic systems. In this paper, we proposed
several methods for learning more generalizable intermediate representations
from humans and suggested directions for moving towards a more continual and
interactive learning framework. It is through understanding and utilizing this
bi-directional communication flow that truly effective human-robot
collaboration can exist.

## References

  * [1] P. Abbeel, A. Coates, and A. Y. Ng, “Autonomous helicopter aerobatics through apprenticeship learning,” _The International Journal of Robotics Research_ , vol. 29, no. 13, pp. 1608–1639, 2010.
  * [2] J. Z. Kolter, C. Plagemann, D. T. Jackson, A. Y. Ng, and S. Thrun, “A probabilistic approach to mixed open-loop and closed-loop control, with application to extreme autonomous driving,” in _2010 IEEE International Conference on Robotics and Automation_. IEEE, 2010, pp. 839–845.
  * [3] M. Wulfmeier, D. Z. Wang, and I. Posner, “Watch this: Scalable cost-function learning for path planning in urban environments,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2016, pp. 2089–2095.
  * [4] A. Y. Ng, D. Harada, and S. Russell, “Policy invariance under reward transformations: Theory and application to reward shaping,” in _Icml_ , vol. 99, 1999, pp. 278–287.
  * [5] S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” _arXiv preprint arXiv:2005.01643_ , 2020.
  * [6] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, “Maximum entropy inverse reinforcement learning,” in _Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 3_ , ser. AAAI’08. AAAI Press, 2008, pp. 1433–1438. [Online]. Available: http://dl.acm.org/citation.cfm?id=1620270.1620297
  * [7] D. Hadfield-Menell, S. Milli, P. Abbeel, S. J. Russell, and A. Dragan, “Inverse reward design,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017.
  * [8] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in _Machine Learning (ICML), International Conference on_. ACM, 2004.
  * [9] A. Bajcsy, D. P. Losey, M. K. O’Malley, and A. D. Dragan, “Learning robot objectives from physical human interaction,” in _Proceedings of the 1st Annual Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, S. Levine, V. Vanhoucke, and K. Goldberg, Eds., vol. 78. PMLR, 13–15 Nov 2017, pp. 217–226. [Online]. Available: http://proceedings.mlr.press/v78/bajcsy17a.html
  * [10] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters _et al._ , “An algorithmic perspective on imitation learning,” _Foundations and Trends in Robotics_ , vol. 7, no. 1-2, pp. 1–179, 2018.
  * [11] C. Finn, S. Levine, and P. Abbeel, “Guided cost learning: Deep inverse optimal control via policy optimization,” in _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ser. ICML’16. JMLR.org, 2016, p. 49–58.
  * [12] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017.
  * [13] J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adverserial inverse reinforcement learning,” in _International Conference on Learning Representations_ , 2018. [Online]. Available: https://openreview.net/forum?id=rkHywl-A-
  * [14] J. Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine, “Variational inverse control with events: A general framework for data-driven reward definition,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , ser. NIPS’18. Red Hook, NY, USA: Curran Associates Inc., 2018, p. 8547–8556.
  * [15] D. Brown, R. Coleman, R. Srinivasan, and S. Niekum, “Safe imitation learning via fast Bayesian reward inference from preferences,” in _Proceedings of the 37th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 13–18 Jul 2020, pp. 1165–1177. [Online]. Available: http://proceedings.mlr.press/v119/brown20a.html
  * [16] F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observation,” in _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , ser. IJCAI’18. AAAI Press, 2018, p. 4950–4957.
  * [17] S. Reddy, A. D. Dragan, and S. Levine, “SQIL: imitation learning via reinforcement learning with sparse rewards,” in _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020\. [Online]. Available: https://openreview.net/forum?id=S1xKd24twB
  * [18] A. Bobu, M. Wiggert, C. Tomlin, and A. D. Dragan, “Inducing structure in reward learning by learning features,” _The International Journal of Robotics Research_ , vol. 0, no. 0, p. 02783649221078031, 0. [Online]. Available: https://doi.org/10.1177/02783649221078031
  * [19] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” in _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , ser. NIPS’16. Red Hook, NY, USA: Curran Associates Inc., 2016, p. 2180–2188.
  * [20] I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed, and A. Lerchner, “beta-vae: Learning basic visual concepts with a constrained variational framework,” in _ICLR_ , 2017.
  * [21] R. T. Q. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud, “Isolating sources of disentanglement in variational autoencoders,” in _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018.
  * [22] C. Doersch, A. K. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” _2015 IEEE International Conference on Computer Vision (ICCV)_ , pp. 1422–1430, 2015.
  * [23] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, F. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell, “Zero-shot visual imitation,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2018, pp. 2131–21 313.
  * [24] Y. Aytar, T. Pfaff, D. Budden, T. L. Paine, Z. Wang, and N. d. Freitas, “Playing hard exploration games by watching youtube,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , ser. NIPS’18. Red Hook, NY, USA: Curran Associates Inc., 2018, p. 2935–2945.
  * [25] M. Laskin, A. Srinivas, and P. Abbeel, “CURL: Contrastive unsupervised representations for reinforcement learning,” in _Proceedings of the 37th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 13–18 Jul 2020, pp. 5639–5650. [Online]. Available: https://proceedings.mlr.press/v119/laskin20a.html
  * [26] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ser. ICML’17. JMLR.org, 2017, p. 1126–1135.
  * [27] K. Xu, E. Ratner, A. Dragan, S. Levine, and C. Finn, “Learning a prior over intent via meta-inverse reinforcement learning,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 6952–6962. [Online]. Available: https://proceedings.mlr.press/v97/xu19d.html
  * [28] A. Bobu, M. Wiggert, C. Tomlin, and A. D. Dragan, “Feature expansive reward learning: Rethinking human input,” in _Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction_ , ser. HRI ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 216–224. [Online]. Available: https://doi.org/10.1145/3434073.3444667
  * [29] D. Braziunas and C. Boutilier, “Elicitation of factored utilities,” _AI Magazine_ , vol. 29, no. 4, p. 79, Dec. 2008. [Online]. Available: https://ojs.aaai.org/index.php/aimagazine/article/view/2203
  * [30] A. Sripathy, A. Bobu, Z. Li, K. Sreenath, D. S. Brown, and A. D. Dragan, “Teaching robots to span the space of functional expressive motion,” 2022. [Online]. Available: https://arxiv.org/abs/2203.02091
  * [31] M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in _Conference on Robot Learning_. PMLR, 2022, pp. 894–906.
  * [32] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8748–8763.
  * [33] J. L. Wright, J. Y. Chen, and S. G. Lakhmani, “Agent transparency and reliability in human–robot interaction: the influence on user confidence and perceived reliability,” _IEEE Transactions on Human-Machine Systems_ , vol. 50, no. 3, pp. 254–263, 2019.
  * [34] G. Bansal, B. Nushi, E. Kamar, D. S. Weld, W. S. Lasecki, and E. Horvitz, “Updates in human-ai teams: Understanding and addressing the performance/compatibility tradeoff,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 2429–2437.
  * [35] J. Berg and S. Lu, “Review of interfaces for industrial human-robot interaction,” _Current Robotics Reports_ , vol. 1, no. 2, pp. 27–34, 2020\.
  * [36] A. Bobu, A. Bajcsy, J. F. Fisac, S. Deglurkar, and A. D. Dragan, “Quantifying hypothesis space misspecification in learning from human–robot demonstrations and physical corrections,” _IEEE Transactions on Robotics_ , pp. 1–20, 2020.
  * [37] C. Baker, J. B Tenenbaum, and R. R Saxe, “Goal inference as inverse planning,” in _Proceedings of the 29th Annual Conference of the Cognitive Science Society_ , 01 2007.
  * [38] E. T. Jaynes, “Information theory and statistical mechanics,” vol. 106. American Physical Society, May 1957, pp. 620–630. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRev.106.620
  * [39] J. Von Neumann and O. Morgenstern, _Theory of games and economic behavior_. Princeton University Press Princeton, NJ, 1945.
  * [40] D. Fridovich-Keil, A. Bajcsy, J. F. Fisac, S. L. Herbert, S. Wang, A. D. Dragan, and C. J. Tomlin, “Confidence-aware motion prediction for real-time collision avoidance,” _International Journal of Robotics Research_ , 2019\.
  * [41] A. Bobu, A. Bajcsy, J. F. Fisac, and A. D. Dragan, “Learning under misspecified objective spaces,” in _Proceedings of The 2nd Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, A. Billard, A. Dragan, J. Peters, and J. Morimoto, Eds., vol. 87. PMLR, 29–31 Oct 2018, pp. 796–805. [Online]. Available: http://proceedings.mlr.press/v87/bobu18a.html
  * [42] D. P. Losey and M. K. O’Malley, “Including uncertainty when learning from human corrections,” in _CoRL_ , 2018.
  * [43] M. Zurek, A. Bobu, D. S. Brown, and A. D. Dragan, “Situational confidence assistance for lifelong shared autonomy,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 2783–2789.
  * [44] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6405–6416.
  * [45] L. Sun, X. Jia, and A. D. Dragan, “On complementing end-to-end human behavior predictors with planning,” _Robotics: Science and Systems XVII_ , 2021.
  * [46] R. Ramakrishnan, V. Unhelkar, E. Kamar, and J. Shah, “A bayesian approach to identifying representational errors,” 2021. [Online]. Available: https://arxiv.org/abs/2103.15171
  * [47] S. Levine, Z. Popovic, and V. Koltun, “Feature construction for inverse reinforcement learning,” in _Advances in Neural Information Processing Systems_ , 2010, pp. 1342–1350.
  * [48] A. Jain, S. Sharma, T. Joachims, and A. Saxena, “Learning preferences for manipulation tasks from online coactive feedback,” _The International Journal of Robotics Research_ , vol. 34, no. 10, pp. 1296–1313, 2015.
  * [49] D. Brown, W. Goo, P. Nagarajan, and S. Niekum, “Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 783–792.
  * [50] R. Shah, D. Krasheninnikov, J. Alexander, P. Abbeel, and A. Dragan, “The implicit preference information in an initial state,” in _International Conference on Learning Representations_ , 2019. [Online]. Available: https://openreview.net/forum?id=rkevMnRqYQ
  * [51] P. de Haan, D. Jayaraman, and S. Levine, “Causal confusion in imitation learning,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019.
  * [52] A. Sax, B. Emi, A. R. Zamir, L. J. Guibas, S. Savarese, and J. Malik, “Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies.” in _Conference on Robot Learning_ , 2018.
  * [53] M. Cakmak and A. L. Thomaz, “Designing robot learners that ask good questions,” in _2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , 2012, pp. 17–24.
  * [54] R. D. Luce, _Individual choice behavior._ Oxford, England: John Wiley, 1959.
  * [55] A. Bobu, D. R. R. Scobee, J. F. Fisac, S. S. Sastry, and A. D. Dragan, _LESS is More: Rethinking Probabilistic Models of Human Behavior_. New York, NY, USA: Association for Computing Machinery, 2020, p. 429–437. [Online]. Available: https://doi.org/10.1145/3319502.3374811


# SCORE-IT: A Machine Learning-based Tool for Automatic Standardization of EEG
Reports

Sam Rawal1 and Yogatheesan Varatharajah2,3 1\. Carle Illinois College of
Medicine, University of Illinois at Urbana Champaign.
2\. Department of Bioengineering, University of Illinois at Urbana Champaign.
3\. Department of Neurology, Mayo Clinic.
{scrawal2<EMAIL_ADDRESS>
###### Abstract

Machine learning (ML)-based analysis of electroencephalograms (EEGs) is
playing an important role in advancing neurological care. However, the
difficulties in automatically extracting useful metadata from clinical records
hinder the development of large-scale EEG-based ML models. EEG reports, which
are the primary sources of metadata for EEG studies, suffer from lack of
standardization. Here we propose a machine learning-based system that
automatically extracts components from the SCORE specification from
unstructured, natural-language EEG reports. Specifically, our system
identifies (1) the type of seizure that was observed in the recording, per
physician impression; (2) whether the session recording was normal or abnormal
according to physician impression; (3) whether the patient was diagnosed with
epilepsy or not. We performed an evaluation of our system using the publicly
available TUH EEG corpus and report F1 scores of 0.92, 0.82, and 0.97 for the
respective tasks.

## I. Introduction

Analysis and interpretation of EEG data is important in the diagnosis of
neurological conditions such as epilepsy. Recently, machine learning-based
analyses are becoming the mainstay of quantitative EEG-based decision making
[1]. EEG reports are the primary source of metadata for quantitative EEG
studies because they contain rich information regarding the patients’
condition at the time of EEG recording including seizures, interictal
abnormalities, and background activity such as posterior dominant rhythm.
However, due to a lack of standardization in the organization of information
presented and terminology in EEG reports, automatic extraction of those
information has proven to be a difficult task. As a result, manual information
retrieval has been the primary way of generating metadata for EEG-based ML
studies. Clearly, this is not scalable and warrants the development of natural
language processing-based automated tools.

The standardized computer-based organized reporting of EEG (SCORE) guidelines
seek to standardize reporting of EEG studies by specifying the content and
terminology of characteristics that are described in an EEG report [2, 3]. As
such, those guidelines provide an ideal target for automated information
retrieval and validation. In this work, we present a semi-automated tool for
extracting structured information from unstructured natural text EEG reports
with an eye towards converting past EEG reports to the standardized format
stipulated in the SCORE guidelines. Furthermore, we also leverage the
classification tasks made available through various labeled subsets of the TUH
dataset as a valuable proxy to evaluate the capabilities of our system.

The development of such a system poses several challenges: 1) data sparsity:
from an entire patient record consisting of dozens of sentences, often only a
single phrase is relevant to making a classification decision; 2) lack of
labeled data: the lack of clinical data available around many of the SCORE
attributes; and 3) accounting for varied clinician practices. We took a two-
step approach to address those challenges: first, we leverage a previously
developed named entity recognition approach using BERT Transformer models
trained on the National NLP Clinical Challenges (n2c2) dataset [4]; second,
hand-crafted rules are applied to these extracted entities, considering
factors such as the SCORE lexicon and numerical values to identify information
relevant to SCORE attributes. This hybrid approach allows us to leverage a)
existing well-trained ML models for relevant entity identification, and b)
domain knowledge to further classify them into SCORE attributes.

In the current work, we focus on identifying three SCORE attributes: (1) the
type of seizure if present (complex partial, simple partial, absence,
myoclonic, generalized tonic-clonic, other/none); (2) whether the recording
contained any abnormal events (e.g., seizures); and 3) whether the patient is
being evaluated for epilepsy. In addition, we designed and validated our
system on the Temple University Hospital (TUH) EEG dataset [5], containing
over 16,000 EEG recordings and reports. We utilized subsets of the TUH EEG
corpus consisting ground truth labels for the three tasks we focused on. We
achieved weighted F1 scores of 0.92 (on 171 test records), 0.82 (on 561
records), and 0.97 (on 2727 records) for the seizure classification,
normal/abnormal classification, and epilepsy classification tasks,
respectively.

## II. Related Work

Existing work on classification from patient medical records includes Track 1
of the 2018 National NLP Clinical Challenges (n2c2), which involves making
binary classification decisions about whether a patient meets some criteria,
such as alcoholism or history of myocardial infarction over past 6 months [6].
Existing strategies for this type of task include extraction of clinical
entities, hand-crafted features that are used to train simple machine learning
models, and creation of lexicons to feed into rules [7]. On the other hand,
there are related classification tasks in the clinical domain for which there
is ample labeled data available, e.g., clinical/biomedical named entity
recognition (NER) [4]. Prior work on performing NER has included the use of
classical Machine Learning models, such as Conditional Random Fields, as well
as deep neural networks, including LSTM networks [8] and, more recently, fine-
tuning transformer networks, such as BERT [9].

However, our work on automatically extracting useful metadata such as seizure
type, EEG classification, and diagnostic information from EEG reports, to our
knowledge, is the first attempt at developing automated information retrieval
approaches for EEG reports. Considering that the majority of the (past and
present) EEG reports are written in natural text format, our work offers the
potential to standardize EEG reports and to generate useful metadata for
subsequent ML-based analyses.

## III. Data

The TUH EEG Corpus (TUEG) is a collection of over 30,000 clinical EEG records
collected and made available by Temple University Hospital (TUH) [5].
Corresponding patient medical reports, in plaintext format, are also made
available alongside the EEG recordings. There are also subsets of the corpus,
containing labeled data for several tasks. In this work, we utilize the TUH
EEG Epilepsy Corpus (TUEP), TUH Abnormal EEG Corpus (TUAB), and TUH EEG
Seizure Corpus (TUSZ). Table 1 describes the number of labeled samples for
each of the subsets. (As the Epilepsy subset had no train and test partition,
unlabeled records from outside the subset were used to develop rules.)

TABLE 1: TUH Seizure Dataset Support Dataset |  Class |  Train Support |  Test Support
---|---|---|---
Seizure |  Absence |  10 |  6
|  Complex Partial |  45 |  13
|  Myoclonic |  1 |  12
|  Simple Partial |  2 |  0
|  Tonic-Clonic |  12 |  4
|  None |  913 |  97
Epilepsy |  Epilepsy |  428 |  428
|  No Epilepsy |  133 |  133
Abnormal |  Normal |  1371 |  150
|  Abnormal |  1346 |  126

## IV. Methodology

In this work, we present a system that, given a natural-language patient
report, predicts, 1) the type of seizure that was observed in the recording,
per physician impression; 2) whether the session recording was normal or
abnormal according to physician impression; and 3) whether a patient has
epilepsy or not.

To construct such a system, we provide a framework for classifying EEG records
that divides the process into two steps: a broad parsing step, for which tasks
are well-defined and considerable training data exists; and a narrow
parsing/classification step built on top of the broad parsing, for which tasks
are highly domain-specific and little training data is available. In this
manner, we address issues of data sparsity and support while simultaneously
leveraging effective data-driven, deep learning techniques that can be adapted
to this specific domain.

A high-level architecture of the system is detailed in Figure 1. In the
following subsections, we discuss each component.

###  IV-A Broad Parsing

The overall goal of this step is to reduce the data sparsity by only
extracting information from the record that will be potentially useful for
classification, while leveraging existing well-trained machine learning models
and methods that utilize the structure of the medical records.

First, section headers are identified, and each sentence in the report is
matched with its corresponding header. The headers are extracted using a
regular expression that matches the format of the medical records present in
the TUH corpus. Second, we perform Named Entity Recognition using BERT
Transformer models [9] trained on datasets released by national NLP clinical
challenges (n2c2). Specifically, we extract clinical entities, such as medical
problems, labs, and treatments, and medication entities, such as medication
name, dose, frequency, duration, and reason. The information extracted in this
step is then passed to the narrow parsing step.

###  IV-B Narrow Parsing

The narrow parsing step that is built on top of the broad parsing step. This
step is motivated by the fact that the classification tasks we address have a
small support of training data; therefore, it is not possible to train a
system only from the supervised data. Consequently, this phase consists of a
series of hand-crafted rules built around the extracted entities and sections
in the prior step, in effect performing domain adaptation from models trained
on related tasks in the broad parsing step to these three tasks with limited
data.

Figure 1: System Architecture.

Classification of each of the of three tasks (epilepsy, normal/abnormal,
seizure type) is performed using hand-crafted rules that are constructed using
the outputs of the section and entity extraction from the previous step. Thus,
the broad parsing step dramatically compresses the input dimensionality from
an unstructured, free-text medical report to a collection of named entities
and the section of the note they are identified in, thus making it possible to
create simpler and more generalizable rules.

For example, the rule used for binary classification in the Epilepsy
identification task consists of extracting all problem entities mentioned in
the “CLINICAL HISTORY" section of the patient record, then outputting a
positive classification if any of the entities match epilepsy (or synonyms and
related phrases).

## V. Experimental Design

The system was evaluated on three specific tasks that are subsets of the TUH
EEG dataset, and for which gold annotations are provided:

  1. 1.

Seizure type classification: given a record, determine the type of seizure
noted in the EEG record (complex partial, simple partial, absence, myoclonic,
tonic-clonic, other/none).

  2. 2.

Epilepsy classification: given a record, determine whether the patient is
being evaluated for epilepsy or not.

  3. 3.

Norma/abnormal classification: determine if the clinical impression in a given
record is normal or abnormal.

A detailed information for each of these tasks is provided in the following
subsections.

###  V-A Seizure Classification

The Seizure subset of the TUH dataset contains 1010 unique patient records
classified by the type of seizure that is presented in the corresponding EEG
recording. One important aspect to note is that, in this work, we focused on
specific types of seizures or NONE. Consequently, records labeled “GNSZ" or
“FNSZ" (for generalized or focal seizures respectively, without any further
specifics) were dropped from the dataset. The breakdown of support for each
type of seizures is described in Table 1.

###  V-B Epilepsy Classification

The Epilepsy subset of the TUH dataset contains 428 patient records labeled
epilepsy and 133 records labeled no epilepsy. One complication of this dataset
is that the labels are associated with the EEG recording findings, rather than
what is detailed int he patient records; in a manual review of the dataset, it
appears there are numerous instances of the gold label not being substantiated
within the clinical record alone.

###  V-C Normal/Abnormal Classification

The Abnormal subset of the TUH corpus consists of notes that are labeled
normal or abnormal. This classification is made based off physician comments
in the “CLINICAL IMPRESSIONS" section of medical records. There are 126 notes
with the abnormal label and 150 notes with the normal label in this subset.

## VI. Results

On the seizure classification task, our system performed with a weighted F1
score of 0.92 on 171 test records. On the epilepsy classification task, our
system performed with a weighted F1 score of 0.82 on 561 records. On the
abnormal classification task, our system performed with a weighted F1 of 0.97
on 2727 records. Notably, the system performed poorer on classes with a very
small support, due to difficulty in creating generalizable hand-crafted rules
from a very small dataset. More detailed results are provided in Table 2.

TABLE 2: Classification Results – Weighted Average Task |  Precision |  Recall |  F1-score |  Support
---|---|---|---|---
Seizure |  0.93 |  0.93 |  0.93 |  121
Abnormal |  0.98 |  0.97 |  0.97 |  276
Epilepsy |  0.82 |  0.82 |  0.82 |  561

## VII. Discussion

We proposed an automated approach for extracting clinically useful metadata
from natural text EEG reports. Major challenges addressed in our work include
1) designing strategies for integration of clinical domain knowledge, 2)
accounting for varied clinician practices, 3) establishing ground truth
standards, and 4) robust verification of the system. Applications of this
system include enabling better search and filtering of clinical reports, as
well as auto-labeling unstructured datasets for research purposes.

As mentioned previously, one complication with interpreting these results is
that for certain tasks, such as epilepsy classification, the labels are
derived from the EEG findings and not necessarily explicitly mentioned in the
patient medical record, making it difficult to fully accurately judge the
performance of the system.

Although the broad and narrow parsing architecture of the system helped in the
development of more general rules, the system still performed better on
classes with greater support. This can be attributed to the fact that having
access to a wider range of training samples allowed rules to be developed that
could capture more of the variance. On the other hand, the system performed
poorer on classes with few samples, as the rules developed with under 5
training samples were not able to be validated to ensure sufficient
generalizability.

## VIII. Future Work

The current iteration of the system still performs relatively poorly in
classification circumstances with low data support. Consequently, one
important area of focus is reducing the amount of hand-crafted rules needed,
while increasing the generalizability of the system, in order to be able to
apply the system across a wider range of use-cases. To that effect, a
prominent area of future focus remains using existing NLP tasks, such as
semantic textual similarity [10], as a means to perform zero-shot
classification in place of hand-crafting rules. Another area for future work
is implementing and validating our approach on additional datasets.

## IX. Conclusion

We described a semi-automated ML-based system to extract standardized
components from unstructured EEG reports. Our system can facilitate better
indexing, searching and organization of existing EEG report collections by
providing an semi-automated methodology of standardizing important data within
free-text EEG reports. Furthermore, our system can be used to generate labels
for large, unlabeled EEG corpora like the TUH dataset to motivate future
research.

## Acknowledgements

This research was supported by National Science Foundation’s Computer and
Information Science and Engineering Research Initiation Initiative Award
SCH-2105233. We would like to thank Dr. Susan Herman and her team from the
Barrow Neurological Institute, as well as Neeraj Wagh from the University of
Illinois at Urbana Champaign, for contributions to discussions on where this
work can be applied.

## References

  * [1] M. Golmohammadi, A. H. Harati Nejad Torbati, S. Lopez de Diego, I. Obeid, and J. Picone, “Automatic analysis of eegs using big data and hybrid deep learning architectures,” _Frontiers in human neuroscience_ , vol. 13, p. 76, 2019.
  * [2] S. Beniczky, H. Aurlien, J. C. Brøgger, A. Fuglsang-Frederiksen, A. Martins-da Silva, E. Trinka, G. Visser, G. Rubboli, H. Hjalgrim, H. Stefan _et al._ , “Standardized computer-based organized reporting of eeg: Score,” _Epilepsia_ , vol. 54, no. 6, pp. 1112–1124, 2013.
  * [3] S. Beniczky, H. Aurlien, J. C. Brøgger, L. J. Hirsch, D. L. Schomer, E. Trinka, R. M. Pressler, R. Wennberg, G. H. Visser, M. Eisermann _et al._ , “Standardized computer-based organized reporting of eeg: Score–second version,” _Clinical Neurophysiology_ , vol. 128, no. 11, pp. 2334–2346, 2017.
  * [4] J. Patrick and M. Li, “High accuracy information extraction of medication information from clinical notes: 2009 i2b2 medication extraction challenge,” _Journal of the American Medical Informatics Association_ , vol. 17, no. 5, pp. 524–527, 2010.
  * [5] A. Harati, S. Lopez, I. Obeid, J. Picone, M. Jacobson, and S. Tobochnik, “The tuh eeg corpus: A big data resource for automated eeg interpretation,” _2014 IEEE Signal Processing in Medicine and Biology Symposium (SPMB)_. IEEE, 2014, pp. 1–5.
  * [6] A. Stubbs, M. Filannino, E. Soysal, S. Henry, and Ö. Uzuner, “Cohort selection for clinical trials: n2c2 2018 shared task track 1,” _Journal of the American Medical Informatics Association_ , vol. 26, no. 11, pp. 1163–1171, 2019.
  * [7] S. Rawal, A. Prakash, S. Adhya, S. Kulkarni, S. Anwar, C. Baral, and M. Devarakonda, “Semi-automated clinical lexicon induction and its use in cohort selection from clinical notes,” _2020 IEEE International Conference on Healthcare Informatics (ICHI)_. IEEE, 2020, pp. 1–2.
  * [8] S. C. Rawal, “Prescription information extraction from electronic health records using bilstm-crf and word embeddings,” 2018.
  * [9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [10] Y. Wang, S. Fu, F. Shen, S. Henry, O. Uzuner, and H. Liu, “The 2019 n2c2/ohnlp track on clinical semantic textual similarity: overview,” _JMIR Medical Informatics_ , vol. 8, no. 11, p. e23375, 2020.